Mixed reality system and method for generating virtual content using the same

Abstract

A computer-implemented method for distorting virtual content from two sources includes generating, by a first source, first virtual content based on a first gesture. The method also includes generating, by a second source, second virtual content based on a second gesture. The method also includes processing, by a synthesizer, the first and second virtual contents in a single pass. Processing the first and second virtual contents includes generating distorted first virtual content by distorting the first virtual content based on a third gesture, generating distorted second virtual content by distorting the second virtual content based on the third gesture, and generating output content by synthesizing the distorted first and second virtual contents.

Description

[0001] This application is a divisional application of the Chinese patent application "Mixed reality system and method for generating virtual content using the system" with application number 201880018506.1 (filing date March 16, 2018). Technical Field

[0002] The present disclosure relates to a mixed reality system with virtual content synthesis, and a method of using the system to generate a mixed reality experience including synthesized virtual content. Background Art

[0003] Modern computing and display technologies have facilitated the development of "mixed reality (MR)" systems for so-called "virtual reality (VR)" or "augmented reality (AR)" experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears or may be perceived as real. VR scenarios typically involve presenting digital or virtual image information that is opaque to actual real-world visual input. AR scenarios typically involve presenting digital or virtual image information as an enhancement to a visualization of the real world around the user (i.e., transparent to real-world visual input). Accordingly, AR scenarios involve presenting digital or virtual image information that is transparent to real-world visual input.

[0004] Various optical systems generate images at various depths for displaying MR (VR and AR) scenes. Some such optical systems are described in U.S. Utility Patent Application Serial No. 14 / 555,585 (Attorney Docket No. ML.20011.00), filed on November 27, 2014, the contents of which are expressly and fully incorporated herein by reference as if fully set forth.

[0005] MR systems typically employ a wearable display device (e.g., a head-mounted display, helmet-mounted display, or smart glasses) that is at least loosely coupled to the user's head and thus moves when the user's head moves. If the display device detects movement of the user's head, the data being displayed may be updated to account for changes in head pose (i.e., the orientation and / or position of the user's head).

[0006] As an example, if a user wearing a head mounted display device views a virtual representation of a virtual object on the display device and walks around the area where the virtual object appears, the virtual object can be rendered for each viewpoint, giving the user the sensation that they are walking around the object occupying real space. If the head mounted display device is used to present multiple virtual objects, measurements of head pose can be used to render the scene to match the user's dynamically changing head pose and provide increased immersion.

[0007] AR-enabled head-mounted display devices provide simultaneous viewing of real and virtual objects. With an "optical see-through" display, a user can see through a transparent (or translucent) element in the display system to directly view light from real objects in the environment. The transparent element, often referred to as a "combiner," superimposes light from the display over the user's real-world view, where light from the display projects images of virtual content over see-through views of real objects in the environment. A camera may be mounted on the head-mounted display device to capture images or video of the scene the user is viewing.

[0008] Current optical systems, such as those in MR systems, render virtual content optically. The content is "virtual" because it does not correspond to a real physical object located at a corresponding location in space. Instead, the virtual content exists only in the brain (e.g., optical center) of the user of the head-mounted display device when stimulated by a beam of light directed to the user's eyes. In an application-rich environment, a 3D optical system can simultaneously render virtual objects generated by multiple sources / applications.

[0009] Some optical systems may include compositing software / systems that receive output from multiple sources / applications. The compositing system then "warps" (i.e., transforms the reference frame) and composites / combines the received outputs for display in a single reference frame (i.e., the reference frame of the display system / viewer; the "display reference frame"). The warping or transformation changes the way the virtual content is presented. The method takes the originally rendered virtual objects and transforms the way the virtual objects are presented in an attempt to display them from different angles.

[0010] Some compositing software / systems compose / combine the warped virtual content outputs in multiple processing passes corresponding to the multiple sources / applications providing the outputs to be composited. In each pass, the output from one source / application is transformed to a display reference frame. After the individual outputs are transformed to the same reference frame, the warped outputs can be composited / combined. However, such multi-pass systems for distorting and compositing outputs from multiple sources / applications can be computationally expensive (causing processor-related system limitations) and time consuming (causing system latency). Summary of the invention

[0011] In one embodiment, a computer-implemented method for distorting virtual content from two sources includes a first source generating first virtual content based on a first gesture. The method also includes a second source generating second virtual content based on a second gesture. The method also includes a synthesizer processing the first and second virtual content in a single pass. Processing the first and second virtual content includes generating a distorted first virtual content by distorting the first virtual content based on a third gesture, generating a distorted second virtual content by distorting the second virtual content based on the third gesture, and generating output content by synthesizing the distorted first and second virtual content.

[0012] In one or more embodiments, the first virtual content includes first image information at an X, Y position, a first brightness and / or a first color. The second virtual content may include second image information at an X, Y position, a second brightness and / or a second color.

[0013] In one or more embodiments, compositing the distorted first and second virtual contents includes performing a depth test on the distorted first and second virtual contents. The step of performing a depth test on the first and second virtual contents may include: determining a first depth of the distorted first virtual content based on a third posture, determining a second depth of the distorted second virtual content based on the third posture, and comparing the first depth to the second depth based on the third posture.

[0014] In one or more embodiments, distorting the first virtual content based on the third gesture includes applying a first transformation to the first virtual content to generate the distorted first virtual content. distorting the second virtual content based on the third gesture includes applying a second transformation to the second virtual content to generate the distorted second virtual content.

[0015] In one or more embodiments, the first source is a first application. The second source may be a second application different from the first application. The compositor may be a raster operation unit.

[0016] In one or more embodiments, the method further includes the synthesizer sending the output content to a display unit, and the display unit displays an image to a user based on the output content. The display unit may be a projector. The first virtual content may be pixels of a first virtual object. The second virtual content may be pixels of a second virtual object.

[0017] In another embodiment, a system for warping virtual content from two sources includes: a warping unit for receiving first and second virtual content from first and second sources, and generating corresponding first and second warped virtual content therefrom. The warping unit includes a pose estimator and a transformation unit. The system also includes a synthesis unit that processes the first and second warped virtual content in a single pass, the synthesis unit including a blending unit. The system also includes a database for temporarily storing the warped virtual content.

[0018] In yet another embodiment, a computer program product embodied in a non-transitory computer-readable medium having a sequence of instructions stored thereon that, when executed by a processor, causes the processor to perform a method for warping virtual content from two sources. The method includes a first source generating first virtual content based on a first gesture. The method also includes a second source generating second virtual content based on a second gesture. The method also includes a synthesizer processing the first and second virtual content in a single pass. Processing the first and second virtual content includes generating the warped first virtual content by warping the first virtual content based on a third gesture, generating the warped second virtual content by warping the second virtual content based on the third gesture, and generating output content by synthesizing the warped first and second virtual content.

[0019] Additional and other objects, features, and advantages of the present disclosure are described in the detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the drawings are not drawn to scale and that elements of similar structure or function are represented by the same reference numerals throughout the drawings. In order to better understand how to obtain the above and other advantages and objects of the various embodiments of the present disclosure, a more detailed description of the present disclosure briefly described above will be presented with reference to its specific embodiments shown in the accompanying drawings. It should be understood that these drawings depict only typical embodiments of the present disclosure and therefore should not be considered as limiting the scope thereof. The present disclosure will be described and explained with additional particularity and detail through the use of the accompanying drawings, in which:

[0021] Figure 1 Depicted is a user view of augmented reality (AR) through a wearable AR user device, according to some embodiments.

[0022] FIG. 2A to FIG. 2D An AR system and its subsystems according to some embodiments are schematically depicted.

[0023] Figure 3 A graphics processing unit (GPU) is schematically depicted in accordance with some embodiments.

[0024] Figure 4 Depicted are virtual objects stored as primitives in accordance with some embodiments.

[0025] Figure 5A and Figure 5B A method for warping virtual content from two sources is depicted in accordance with some embodiments.

[0026] Figure 5C , Figure 5D and Figure 5E A method for warping virtual content from two sources is depicted in accordance with some embodiments.

[0027] Figures 6 to 8 Various aspects of warping virtual content are shown in accordance with some embodiments.

[0028] Fig. 9 is a block diagram that schematically depicts an example computing system in accordance with some embodiments. DETAILED DESCRIPTION

[0029] Various embodiments of the present disclosure are directed to systems, methods, and articles of manufacture for warping and compositing virtual content from multiple applications in a single embodiment or multiple embodiments.Other objects, features, and advantages of the present disclosure are described in the detailed description, drawings, and claims.

[0030] Various embodiments will now be described in detail with reference to the accompanying drawings, and various embodiments are provided as illustrative examples of the present disclosure so that those skilled in the art can practice the present disclosure. It should be noted that the following drawings and examples are not meant to limit the scope of the present disclosure. In the case where known components (or methods or processes) can be used to partially or completely implement some elements of the present disclosure, only those parts of these known components (or methods or processes) necessary for understanding the present disclosure will be described, and the detailed description of other parts of these known components (or methods or processes) will be omitted, so as not to obscure the present disclosure. In addition, various embodiments include the current and future known equivalents of the components mentioned herein by way of illustration.

[0031] The virtual content warping and synthesis system can be implemented independently of the mixed reality system, but some of the following embodiments are described with respect to the AR system for illustrative purposes only. In addition, the virtual content warping and synthesis system described herein can also be used in the same manner as the VR system.

[0032] Summary of Problems and Solutions

[0033] When the optical system simultaneously renders virtual content generated by multiple sources / applications, the multiple applications may generate the corresponding virtual content using different postures specific to the corresponding applications (e.g., due to head movement between application arrivals or renderings, or different application reference frames). These postures may be different from each other and may be different from the posture of the display system that will display the synthesized virtual content. Therefore, the synthesis software / system of the optical system must first distort or transform the virtual content generated by the multiple applications into the display reference frame of the display system / viewer. Only after the transformation can the various distorted virtual content be synthesized into a single synthesized virtual content for display.

[0034] For example, a first application may output a first virtual content, which may be a chess game. A second application may output a second virtual content, which may be a movie. Some synthesis software / systems distort and synthesize / combine the first and second virtual contents in two processing rounds. In the first round, the first virtual content (e.g., a chess game) may be distorted / transformed into a display reference frame and stored in a buffer. In the second round, the second virtual content (e.g., a movie) may be distorted / transformed into the same display reference frame and stored in a buffer. After the first and second virtual contents are distorted to the display reference frame, the distorted first and second virtual contents may be read from the buffer and synthesized / combined. For example, if the corresponding pixels of the distorted first and second virtual contents will be displayed on the same pixel of the display (i.e., collide pixels), a depth test is performed on the corresponding portion and the closer virtual content is displayed (assuming the content is opaque).

[0035] Figure 5A and Figure 5B A method for compositing virtual content in two passes according to some embodiments is depicted. At step 502, a first source generates first virtual content 610 based on a first gesture. The first virtual content 610 is sent to the warping unit 280 of the block 290, which receives the first virtual content 610.

[0036] In step 506, the warping unit 280 (e.g., the pose estimator 282 and the transformation unit 284 of the warping unit 280 of the block 290) warps (e.g., transforms) the first virtual content 610 generated by the first application to generate a warped first virtual content 610' (note the difference in the positions of the first virtual content 610 and the warped first virtual content 610' in steps 502 and 506, respectively). The first warped virtual content 610' is sent to the synthesis unit 294 of the block 290.

[0037] In step 509 , the synthesis unit 294 synthesizes the distorted first virtual content 610 ′ to form synthesized first virtual content. Since no data of the synthesized virtual content has been previously written, the synthesis of the distorted first virtual content 610 ′ can be completed by writing the distorted first virtual content 610 ′ into the database 292 .

[0038] At step 504 , the second source generates second virtual content 612 based on the second gesture. The second virtual content 612 is sent to the warping unit 280 , which receives the second virtual content 612 .

[0039] In step 508, the warping unit 280 (e.g., the pose estimator 282 and the transform unit 284 of the warping unit 280 of the block 290) warps (e.g., transforms) the second virtual content 612 generated by the second application to generate a warped second virtual content 612' (note the difference in the positions of the second virtual content 612 and the warped second virtual content 612' in steps 504 and 508, respectively). The warped second virtual content 612' is sent to the synthesis unit 294.

[0040] At step 510, the synthesis unit 294 synthesizes the distorted second virtual content 612' and the synthesized first virtual content 610' to form synthesized virtual content 610' / 612'. Synthesizing the distorted second virtual content 612' and the synthesized first virtual content 610' may result in an overlapping area as shown in 510 (see the overlap between the distorted first virtual content 610' and the distorted second virtual content 612').

[0041] At step 514, a mixer (e.g., a blending unit 296 of a compositing unit 294 of a block 290) resolves the overlap between the warped first virtual content 610' and the warped second virtual content 612' (see the resolved overlap area between the warped first virtual content 610' and the warped second virtual content 612') to form a blended virtual content 610' / 612'. The blending unit 296 may use a depth test to resolve the overlap. The blending unit 296 may store the blended virtual content 610' / 612' in the database 292.

[0042] This two-pass system and method 500 for warping and compositing the outputs from two applications may be computationally expensive and time consuming. The computational expense and time required may increase as the number of applications for which the output / virtual content must be warped and composited for display increases. The increasing time requirements of current compositing software / systems as the number of applications increases may be incompatible with real-time systems such as some mixed reality systems. Furthermore, the increasing computational overhead of current compositing software / systems as the number of applications increases may manifest itself in size, power, heat, and other processing-related limitations that may be incompatible with portable systems such as some mixed reality systems.

[0043] To address these limitations, the system described herein warps and composites virtual content outputs from multiple applications in a single pass. Each virtual content output is warped based on the pose used to generate the virtual content output and the pose of a "display reference frame". Composite warped virtual content outputs include depth testing to resolve conflicting warped virtual content outputs from different sources to be displayed in a specific area in the display field of view.

[0044] Illustrative mixed reality scenarios and systems

[0045] The following description relates to an illustrative augmented reality system utilized to implement the synthesis system. However, it should be understood that the embodiments are also applicable to applications in other types of display systems (including other types of mixed reality systems), and therefore the embodiments are not limited to the illustrative system disclosed herein.

[0046] Mixed reality (e.g., VR or AR) scenarios typically include the presentation of virtual content (e.g., images and sounds) corresponding to virtual objects associated with real-world objects. Figure 1 , depicting an augmented reality (AR) scene 100 in which a user of AR technology sees a real-world physical park-like setting 102 featuring people, trees, and buildings in the background, as well as a real-world physical entity platform 104. In addition to these items, the user of the AR technology also feels that they "see" a virtual robot statue 106 standing on the physical entity platform 104, and a virtual cartoon-like avatar character 108 that looks like an avatar of a flying bee, even though these virtual objects 106, 108 do not exist in the real world. The virtual robot statue 106 and the virtual cartoon-like avatar character 108 can be generated by different applications that are operably coupled to the AR system. These applications can provide virtual content generated / rendered based on different poses. The AR system must warp and synthesize the distorted virtual content before displaying it. Synthesis includes depth testing to resolve conflicting distorted virtual content (e.g., if a portion of the virtual cartoon-like avatar character 108 occludes a portion of the virtual robot statue 106).

[0047] In a similar manner, VR scenes can also include virtual content from multiple sources / applications. Like AR scenes, VR scenes must also take into account the poses used to generate / render virtual content. Accurately warping and compositing virtual content to the AR / VR display reference frame and compositing the warped virtual content can improve the AR / VR scene, or at least not detract from the AR / VR scene.

[0048] The following description relates to an illustrative AR system with which the present disclosure may be implemented. However, it should be understood that the present disclosure is also applicable to applications in other types of augmented reality and virtual reality systems, and thus the present disclosure is not limited to the illustrative system disclosed herein.

[0049] Reference Figure 2A , an augmented reality (AR) system 200 is shown according to some embodiments. The AR system 200 can operate with a projection subsystem 208 to provide images of virtual objects mixed with physical objects in the field of view of a user 250. The method employs one or more at least partially transparent surfaces through which the surrounding environment including physical objects can be seen, and the AR system 200 generates images of virtual objects through the surface. The projection subsystem 208 is housed in a control subsystem 201, which is operably coupled to the display system / subsystem 204 via a link 207. The link 207 can be a wired or wireless communication link.

[0050] For AR applications, it may be desirable to spatially position various virtual objects relative to various physical objects in the field of view of the user 250. Virtual objects may take any of a variety of forms, with any kind of data, information, concept, or logical construct that can be represented as an image. Non-limiting examples of virtual objects may include: virtual text objects, virtual digital objects, virtual alphanumeric objects, virtual label objects, virtual field objects, virtual chart objects, virtual map objects, virtual instrument objects, or virtual visual representations of physical objects.

[0051] The AR system 200 includes a frame structure 202 worn by a user 250, a display system 204 carried by the frame structure 202 such that the display system 204 is positioned in front of the eyes of the user 250, and a speaker 206 incorporated into or connected to the display system 204. In the illustrated embodiment, the speaker 206 is carried by the frame structure 202 such that the speaker 206 (e.g., an earbud or earphone) is positioned near (in or around) the ear canal of the user 250.

[0052] The display system 204 is designed to present a photo-based radiation pattern to the eyes of the user 250 that can be comfortably perceived as an enhancement of the surrounding environment including two-dimensional and three-dimensional content. The display system 204 presents a sequence of frames at a high frequency that provides the perception of a single coherent scene. To this end, the display system 204 includes a projection subsystem 208 and a partially transparent display screen through which the projection subsystem 208 projects the image. The display screen is located in the field of view of the user 250 between the eyes of the user 250 and the surrounding environment.

[0053] In some embodiments, the projection subsystem 208 takes the form of a scanning-based projection device, and the display screen takes the form of a waveguide-based display into which the scanned light from the projection subsystem 208 is injected to produce, for example, an image at a single optical viewing distance that is closer than infinity (e.g., arm's length), an image at multiple discrete optical viewing distances or focal planes, and / or image layers stacked at multiple viewing distances or focal planes to represent a volumetric 3D object. The layers in the light field can be stacked closely enough together to appear continuous to the human visual subsystem (e.g., one layer is within the cone of confusion of an adjacent layer). Additionally or alternatively, image elements can be blended across two or more layers to increase the perceived continuity of transitions between layers in the light field, even if the layers are more sparsely stacked (e.g., one layer is outside the cone of confusion of an adjacent layer). The display system 204 can be monocular or binocular. The scanning component includes one or more light sources that generate light beams (e.g., emitting light of different colors in a defined pattern). The light source may take any of a variety of forms, such as a set of RGB sources (e.g., laser diodes capable of outputting red, green, and blue light) that are operable to produce red, green, and blue coherent collimated light, respectively, according to a defined pixel pattern specified in each frame of pixel information or data. Lasers provide high color saturation and are energy efficient. The optical coupling subsystem includes an optical waveguide input device, such as one or more reflective surfaces, diffraction gratings, mirrors, dichroic mirrors, or prisms, to optically couple light to the end of the display screen. The optical coupling subsystem also includes a collimating element for collimating light from the optical fiber. Optionally, the optical coupling subsystem includes an optical modulation device configured to converge light from the collimating element toward a focal point located at the center of the optical waveguide input device, thereby allowing the size of the optical waveguide input device to be minimized. Thus, the display subsystem 204 generates a series of synthetic image frames of pixel information that present an undistorted image of one or more virtual objects to the user. Further details describing the display subsystem are provided in U.S. Utility Patent Application No. 14 / 212,961, entitled “Display System and Method” (Attorney Docket No. ML.20006.00), and U.S. Utility Patent Application No. 14 / 331,218, entitled “Planar Waveguide Apparatus With Diffraction Element(s) and Subsystem Employing Same” (Attorney Docket No. ML.20020.00), the contents of which are expressly incorporated herein by reference in their entirety as if fully set forth.

[0054] The AR system 200 also includes one or more sensors mounted to the frame structure 202 for detecting the position (including orientation) and movement of the head of the user 250 and / or the eye position and interpupillary distance of the user 250. Such sensors may include image capture devices, microphones, inertial measurement units (IMUs), accelerometers, compasses, GPS units, radios, gyroscopes, and the like. For example, in one embodiment, the AR system 200 includes a head-mounted transducer subsystem that includes one or more inertial transducers to capture inertial measurements indicating the movement of the head of the user 250. These devices can be used to sense, measure, or collect information about the head movement of the user 250. For example, these devices can be used to detect / measure the movement, velocity, acceleration, and / or position of the head of the user 250. The position (including orientation) of the head of the user 250 is also referred to as the "head pose" of the user 250.

[0055] Figure 2A The AR system 200 may include one or more forward-facing cameras. The cameras may be used for any number of purposes, such as recording images / video from the forward direction of the system 200. Additionally, the cameras may be used to capture information about the environment in which the user 250 is located, such as information indicating the distance, orientation, and / or angular position of the user 250 relative to the environment and specific objects in the environment.

[0056] The AR system 200 may also include a rear-facing camera to track the angular position (direction of the eye or eyes looking), blinking, and depth of focus (by detecting eye convergence) of the user 250's eyes. Such eye tracking information may be discerned, for example, by projecting light at the end user's eyes and detecting the return or reflection of at least some of the projected light.

[0057] The augmented reality system 200 also includes a control subsystem 201, which can take any of a variety of forms. The control subsystem 201 includes a plurality of controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers, such as application specific integrated circuits (ASICs), programmable gate arrays (PGAs), such as field PGAs (FPGAs), and / or programmable logic controllers (PLUs). The control subsystem 201 may include a digital signal processor (DSP), a central processing unit (CPU) 251, a graphics processing unit (GPU) 252, and one or more frame buffers 254. The CPU 251 controls the overall operation of the system, while the GPU 252 renders frames (i.e., converts a three-dimensional scene into a two-dimensional image) and stores the frames in one or more frame buffers 254. Although not shown, one or more additional integrated circuits may control the reading of frames into and / or out of the frame buffers 254 and the operation of the display system 204. Reading into and / or out of the frame buffer 254 may employ dynamic addressing, for example, where a frame is over-rendered. The control subsystem 201 also includes a read-only memory (ROM) and a random access memory (RAM). The control subsystem 201 also includes a three-dimensional database 260 from which the GPU 252 may access three-dimensional data for one or more scenes used to render a frame, as well as synthesized sound data associated with virtual sound sources contained within the three-dimensional scene.

[0058] The augmented reality system 200 also includes a user position detection module 248. The user position module 248 detects the instantaneous position of the head of the user 250 and can predict the position of the head of the user 250 based on the position data received from the sensor. The user position module 248 also tracks the eyes of the user 250, and in particular tracks the direction and / or distance at which the user 250 is focusing based on the tracking data received from the sensor.

[0059] Figure 2B An AR system 200 ′ is depicted in accordance with some embodiments. Figure 2B The AR system 200′ shown in FIG. 1 is similar to Figure 2A The AR system 200 shown and described above includes, for example, a frame structure 202 , a display system 204 , a speaker 206 , and a control subsystem 201 ′ operably coupled to the display system 204 via a link 207 . Figure 2B The control subsystem 201' depicted in FIG. Figure 2AFor example, the control subsystem 201' includes the projection subsystem 208, the image / video database 271, the user position module 248, the CPU 251, the GPU 252, the 3D database 260, the ROM and the RAM.

[0060] Figure 2B The control subsystem 201' depicted in FIG. 1 is the AR system 200' and Figure 2A The corresponding systems / system components shown differ in that Figure 2B There is a block 290 in the control subsystem 201' shown in FIG. Block 290 is an independent warp block independent of GPU 252 or CPU 251. Figure 2C As shown in FIG. 2 , block 290 includes a warping unit 280, a database 292, and a synthesis unit 294. The synthesis unit 294 includes a mixing unit 296. Figure 2D As shown, the warping unit 280 includes a pose estimator 282 and a transformation unit 284 .

[0061] Various processing components of the AR system 200, 200' can be included in a distributed subsystem. For example, the AR system 200, 200' includes a local processing and data module (i.e., control subsystem 201, 201') operatively coupled to a portion of the display subsystem 204, such as by a wired conductor or a wireless connection 207. The local processing and data module can be mounted in various configurations, such as fixedly attached to the frame structure 202, fixedly attached to a helmet or hat, embedded in headphones, removably attached to the torso of the user 250, or removably attached to the hip of the user 250 in a belt-coupled configuration. The AR system 200, 200' may also include a remote processing module and a remote data repository operatively coupled to the local processing and data module, such as by a wired conductor or a wireless connection, so that these remote modules are operatively coupled to each other and can serve as resources for the local processing and data module. The local processing and data module may include a power-efficient processor or controller, and digital memory, such as flash memory, both of which may be used to assist in processing, caching, and storing data captured from the sensor and / or data acquired and / or processed using the remote processing module and / or remote data repository, possibly for delivery to the display system 204 after such processing or acquisition. The remote processing module may include one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. The remote data repository may include a relatively large-scale digital data storage facility that may be accessible through the Internet or other networking configurations in a "cloud" resource configuration. In one embodiment, all data is stored in the local processing and data module, and all calculations are performed, allowing for full autonomous use from the remote module. The coupling between the various components described above may include one or more wired interfaces or ports to provide wired or optical communications, or one or more wireless interfaces or ports, such as via RF, microwave, and IR to provide wireless communications. In some embodiments, all communications may be wired, while in some other embodiments, all communications may be wireless, except for optical fibers.

[0062] Illustrative Graphics Processing Unit

[0063] Figure 3 Schematically depicts an exemplary graphics processing unit (GPU) 252 that warps virtual content from one or more sources to a common reference frame and synthesizes the warped virtual content in accordance with some embodiments. The GPU 252 includes an input memory 310 for storing virtual content from one or more sources. In some embodiments, the virtual content is stored as primitives (e.g., Figure 4400 in the rendering pipeline). The GPU 252 also includes a command processor 312 that (1) receives / reads virtual content from the input memory 310, (2) divides the virtual content into dispatch units, and (3) sends the dispatch units in waves or warps along the rendering pipeline for parallel processing. The GPU 252 also includes a scheduler 314 that (1) receives dispatch units from the command processor 312, and (2) determines whether "new work" from the command processor 312 or "old work" returned from downstream in the rendering pipeline (described below) should be sent down the rendering pipeline at any particular time. In effect, the scheduler 914 determines the sequence in which the GPU 252 processes various input data.

[0064] The GPU 252 includes GPU cores 316, each of which has multiple parallel executable cores / units ("shader cores") 318 for parallel processing of the scheduling unit. The command processor 312 divides the virtual content into a number equal to the number of shader cores 318 (e.g., 32). The GPU 251 also includes a "first in, first out" ("FIFO") memory 320 to receive output from the GPU cores 316. The output can be routed back to the scheduler 314 from the FIFO memory 320 as "old work" to be inserted into the rendering pipeline additional processing performed by the GPU cores 316.

[0065] GPU 252 also includes a raster operations unit ("ROP") 322 that receives output from FIFO memory 320 and rasterizes the output for display. For example, primitives of virtual content may be stored as coordinates of triangle vertices. After being processed by GPU core 316 (during Figure 4 The three vertices 410, 412, 414 of the triangle 400 may be warped), the ROP 322 determines which pixels 416 are inside the triangle 400 defined by the three vertices 410, 412, 414, and fills those pixels 416 in the virtual content. The ROP 322 may also perform depth testing on the virtual content.

[0066] The GPU 252 also includes a buffer memory 324 for receiving the output (e.g., the warped virtual content) of the ROP 322. The output from the buffer memory 324 can be routed back to the scheduler 314 as "old work" for insertion into the rendering pipeline for additional processing by the GPU core 316, or for display in corresponding pixels of the display system. The GPU core 316 first processes the vertices 410, 412, 414 of the triangle 400, and then processes the pixels 416 within the triangle 400.

[0067] Virtual content synthesis system and method

[0068] In image processing without head pose changes and / or input from one or more sources, the processing results of GPU 252 are color / brightness values ​​and depth values ​​at various X, Y values ​​(e.g., at each pixel). However, as head pose changes and / or input from one or more sources, virtual content from various sources may overlap at pixels. In some methods for resolving conflicting virtual content, virtual content from each source is warped and stored. Any conflicting warped virtual content is then depth-tested against each other to determine the warped virtual content closest to the user, which is used in the synthesized virtual content to resolve the conflict. As described above, this multi-round warping and synthesis process can be computationally expensive and slow, making it difficult to use with portable display systems (such as mixed reality systems).

[0069] Figure 5C A method 520 for compositing virtual content in a single pass is depicted in accordance with some embodiments. At step 522, a first source generates first virtual content (e.g., first virtual content 610) based on a first pose corresponding to a first source reference frame. For example, first virtual content 610 may be a chess piece. At step 524, a second source generates second virtual content (e.g., second virtual content 612) based on a second pose corresponding to a second source reference frame. For example, second virtual content 612 may be a cube. Figure 6 Pawn 610 and cube 612 are shown in a common reference frame (e.g., a display reference frame), where pawn 610 and cube 612 are generated using different gestures corresponding to different sources (e.g., applications) that generated pawn 610 and cube 612 (e.g., pawn 610 generated using a first gesture and cube 612 generated using a second gesture).

[0070] At step 526, the compositor (e.g., the pose estimator 282 and the transform unit 284 of the warp unit 280 of the GPU 252 and / or the block 290) warps (e.g., transforms) the pixels of the first virtual content 610 generated by the first application to generate pixels of the warped first virtual content 610'. The compositor may select the pixels of the first virtual content 610 to correspond to a specific pixel (e.g., the upper left pixel) in the output content / display image. The compositor may select the pixels of the first virtual content to be warped by analyzing the target pixel in the output content and the transformation to be applied to the second virtual content 612 (based on the first pose corresponding to the first source reference frame and the third pose corresponding to the output / display reference frame). The compositor may warp the first virtual content (e.g., the pixels of the first virtual content 610) by performing a first Fourier transform on the first virtual content 610 using the first pose and the third pose.

[0071] The third pose may be predicted / estimated (e.g., by the pose estimator 282 of the warp unit 280) to be consistent with the pose when the synthesized warped virtual content is to be displayed. The third pose may be referred to as a viewer's pose, a unified pose, an aligned pose, a synthesized pose, and / or a coherent predicted pose. The third pose may be predicted / estimated so that when the first and second warped virtual contents are overlaid on top of each other and the resulting image is projected, the image is accurate relative to the viewer's actual head pose at the time of the projection. The third pose may correspond to an output / display reference frame.

[0072] At step 528, the compositor (e.g., the pose estimator 282 and the transform unit 284 of the warp unit 280 of the GPU 252 and / or the block 290) warps the pixels of the second virtual content 612 generated by the second application to generate pixels of the warped second virtual content 612'. For example, the compositor may select pixels of the second virtual content 612 to correspond to specific pixels in the output content / display image, for example, so that the pixels of the warped second virtual content 612' correspond to the pixels of the warped first virtual content 610' from step 526. The compositor may select the pixels of the second virtual content to be warped by analyzing the position of the warped first virtual content 610' from step 526 and the transformation to be applied to the second virtual content 612 (based on the second pose corresponding to the second source reference frame and the third pose corresponding to the output / display reference frame). The compositor may warp the second virtual content by performing a second Fourier transform on the second virtual content 610 using the second pose and the third pose. Due to the difference between the first and second poses, the first Fourier transform may be different from the second Fourier transform.

[0073] At step 530, a compositor (e.g., ROP 322 of GPU 252 and / or compositing unit 294 and blending unit 296 of block 290) composites the pixels of the warped first virtual content 610' and the second virtual content 612' to form output content for display. Figure 6 As shown in , if the pixels of the warped first virtual content 610 ′ and the second virtual content 612 ′ do not overlap, they are stored in a buffer (eg, buffer memory 924 and / or database 292 ) as temporary pixels of output content.

[0074] In some embodiments, during compositing, pixels of the warped first virtual content 610' and second virtual content 612' are assigned to the same pixels in the output content with each other, or with other pixels already stored in the buffer as temporary pixels of the output content. In other words, the pixels "collide", such as Figure 7 and Figure 8As shown. In this case, the compositor (e.g., ROP 322 of GPU 252 and / or synthesis unit 294 and blending unit 296 of block 290) compares the depth information of the conflicting pixels of the distorted first virtual content 610' and the second virtual content 612' to identify pixels that are closer to the user. After the pixels are transformed, they are in the same display reference frame and can be depth tested. The closer pixel is then written to a buffer (e.g., buffer memory 924 and / or database 292) as a temporary pixel at that location of the output content. Details regarding the blending of colliding pixels are described in U.S. Provisional Application Serial No. 62 / 472,985, which has been previously incorporated by reference. Figure 7 In the output content, the distorted first virtual content 610' is closer to the user than the distorted second virtual content 612'. Thus, when the pixels of the distorted first virtual content 610' and the distorted second virtual content 612' collide, the pixels of the distorted first virtual content 610' are displayed in the output content. Figure 8 In the output content, the distorted second virtual content 612' is closer to the user than the distorted first virtual content 610'. Thus, when the pixels of the distorted first virtual content 610' and the distorted second virtual content 612' collide, the pixels of the distorted second virtual content 612' are displayed in the output content.

[0075] At step 532, the pixel-by-pixel processing / coordination in steps 526-530 is repeated until the entire warped first virtual content 610' and second virtual content 612' have been composited into output content for display. Warping, compositing, and depth testing pixels of virtual content from multiple sources in a single pass reduces processor burden and the time required to composite virtual content to form output content for display.

[0076] Figure 5D and Figure 5E A method 540 for compositing virtual content in a single pass is depicted in accordance with some embodiments. At step 542, a first source generates first virtual content 610 based on a first pose corresponding to a first source reference frame. At step 544, a second source generates second virtual content 612 based on a second pose corresponding to a second source reference frame. The first virtual content 610 and the second virtual content 612 are sent to the warp unit 280 of the block 290, which receives the first virtual content 610 and the second virtual content 612.

[0077] In step 546, the warping unit (e.g., the pose estimator 282 and the transformation unit 284 of the warping unit 280 of the block 290) warps (e.g., transforms) the first virtual content 610 generated by the first application to generate a distorted first virtual content 610' (note the difference in the positions of the first virtual content 610 and the distorted first virtual content 610' in steps 542 and 546, respectively). In step 548, the warping unit (e.g., the pose estimator 282 and the transformation unit 284 of the warping unit 280 of the block 290) warps (e.g., transforms) the second virtual content 612 generated by the second application to generate a distorted second virtual content 612' (note the difference in the positions of the second virtual content 612 and the distorted second virtual content 612' in steps 544 and 548, respectively). The above description of the distorted second virtual content 612 can be used. Figure 5C The warped first virtual content 610' and the second virtual content 612' are generated by the pixel-by-pixel warping method described in the method 520 depicted in FIG. 2. In some embodiments, the warping unit performs steps 546 and 548 in parallel (substantially simultaneously). In some embodiments, the warping unit performs steps 546 and 548 serially (one by one) in any desired order. The pixels of the warped first virtual content 610' and the second virtual content 612' are sent to the synthesis unit 294 of the block 290.

[0078] At step 550, a synthesizer (e.g., synthesis unit 294 of block 290) synthesizes the distorted first virtual content 610' and the second virtual content 612', as described above with respect to method 520 depicted in Figure 5, thereby forming synthesized virtual content 610' / 612'. The synthesized virtual content 610' / 612' may have overlapping areas, as shown in 550 (see the overlap between the first virtual content 610' and the second virtual content 612').

[0079] At step 552, a mixer (e.g., a mixer unit 296 of a synthesis unit 294 of a block 290) resolves the overlap between the warped first virtual content 610' and the second virtual content 612' (see the resolved overlap region between the first virtual content 610' and the second virtual content 612') to form a mixed virtual content 610' / 612'. The mixer may resolve the overlap using the method described above with respect to the method 520 shown in FIG. 5. The mixer may store the mixed virtual content 610' / 612' in the database 292.

[0080] As mentioned above, FIG. 5C to FIG. 5E The methods 520, 540 depicted in FIG. 5 may also be performed on a separate warp block 290 that is independent of any GPU 252 or CPU 251. In some embodiments, FIG. 5C to FIG. 5EThe methods 520 and 540 described in the embodiment may be executed on the CPU 251. In some embodiments, FIG. 5C to FIG. 5E The methods 520 , 540 depicted in FIG. 5 may be performed on various combinations / sub-combinations of the GPU 252 , the CPU 251 , and a separate warp block 290 . FIG. 5C to FIG. 5E The methods 520 , 540 depicted in describe an image processing pipeline that can be executed using various execution models depending on the availability of system resources at a particular time.

[0081] The first virtual content (eg, chess piece) 610 and the second virtual content (eg, cube) 612 are displayed on the Figures 6 to 8 610 and the second virtual content 612 are shown in dashed lines, indicating that the first virtual content 610 and the second virtual content 612 generated by the first and second applications are not in the display reference frame. The distorted first virtual content 610' and the distorted second virtual content 612' are shown in solid lines only when the synthesis system has distorted the first virtual content 610 and the second virtual content 612.

[0082] Although the method 500 depicted in FIG. 5 composites the first virtual content 610 and the second virtual content 612 from two applications, in other embodiments, three or more applications may generate three or more corresponding virtual content that must be composited (and possibly depth tested) to form output content for display.

[0083] System Architecture Overview

[0084] Fig. 9 is a block diagram of an illustrative computing system 900 suitable for implementing embodiments of the present disclosure. Computer system 900 includes a bus 906 or other communication mechanism for passing information that interconnects subsystems and devices such as a processor 907, system memory 908 (e.g., RAM), static storage device 909 (e.g., ROM), (e.g., magnetic or optical) disk drive 910, communication interface 914 (e.g., a modem or Ethernet card), display 911 (e.g., CRT or LCD), input device 912 (e.g., keyboard), and cursor control.

[0085] According to one embodiment of the present disclosure, the computer system 900 performs specific operations by executing one or more sequences of one or more instructions contained in the system memory 908 by the processor 907. These instructions can be read into the system memory 908 from another computer readable / usable medium (e.g., static storage device 909 or disk drive 910). In alternative embodiments, hard-wired circuits can be used instead of software instructions or in combination with software instructions to implement the present disclosure. Therefore, the embodiments of the present disclosure are not limited to any specific combination of hardware circuits and / or software. In one embodiment, the term "logic" shall mean any combination of software or hardware used to implement all or part of the present disclosure.

[0086] As used herein, the term "computer-readable medium" or "computer-usable medium" refers to any medium that participates in providing instructions to processor 907 for execution. Such media may take many forms, including, but not limited to, non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks, such as disk drive 910. Volatile media include dynamic memory, such as system memory 908.

[0087] Common forms of computer-readable media include, for example, floppy disks, diskettes, hard disks, magnetic tape, any other magnetic medium, CD-ROMs, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM (e.g., NAND flash, NOR flash), any other memory chip or cassette, or any other medium from which a computer can read.

[0088] In an embodiment of the present disclosure, the execution of the instruction sequence for practicing the present disclosure is performed by a single computer system 900. According to other embodiments of the present disclosure, two or more computer systems 900 coupled via a communication link 915 (e.g., a LAN, PTSN, or wireless network) can coordinate with each other to execute the instruction sequence required to practice the present disclosure.

[0089] Computer system 900 can send and receive messages, data and instructions, including programs, i.e., application code, via communication link 915 and communication interface 914. Received program code can be executed by processor 907 as it is received, and / or stored in disk drive 910 or other non-volatile memory for later execution. Database 932 in storage medium 931 can be used to store data accessed by system 900 via data interface 933.

[0090] The present disclosure includes methods that can be performed using subject equipment. These methods can include the action of providing such appropriate equipment. Such provision can be performed by the user. In other words, the "providing" action only requires the user to obtain, access, approach, locate, set, activate, power on or otherwise act to provide the necessary equipment in the subject method. The methods described herein can be performed in any order of the events that are logically possible and the order of the events.

[0091] The exemplary aspects of the present disclosure and details on material selection and manufacturing have been set forth above. With regard to other details of the present disclosure, these can be understood in conjunction with the patents and publications cited above and what is generally known or understood by those skilled in the art. This can be the same for the method-based aspects of the present disclosure in terms of additional actions that are commonly or logically used.

[0092] In addition, although the present disclosure has been described with reference to several examples that optionally incorporate various features, the present disclosure is not limited to that which is desired to be described or indicated for each variation of the present disclosure. Various changes may be made to the described disclosure and equivalents (whether or not listed herein or not included for the sake of brevity) may be substituted without departing from the true spirit and scope of the present disclosure. In addition, where a numerical range is provided, it is understood that each intermediate value between the upper and lower limits of the range and any other specified value or intermediate value within the range are included in the present disclosure.

[0093] In addition, it is contemplated that any optional features of the described creative changes may be stated and claimed independently, or in combination with any one or more of the features described herein. Reference to singular items includes the possibility of multiple identical items. More specifically, as used herein and in the claims associated therewith, the singular forms "one," "an," "said," and "the" include plural referents unless otherwise specifically stated. In other words, the use of articles allows for "at least one" of the subject items in the above description and claims associated with the present disclosure. It is further noted that these claims may be drafted to exclude any optional elements. Therefore, this statement is intended to serve as an antecedent basis for the use of exclusive terms such as "only," "only," etc. in conjunction with the statement of claim elements or the use of "negative" limitations.

[0094] Without using such exclusive terms, the term "comprising" in the claims associated with the present disclosure shall allow for the inclusion of any additional elements - regardless of whether a given number of elements is recited in these claims, or the addition of features may be considered to modify the nature of the elements recited in these claims. Except as expressly defined herein, all technical and scientific terms used herein shall be interpreted as broadly as possible while maintaining the validity of the claims.

[0095] The breadth of the present disclosure is not limited to the examples provided and / or this description, but is only limited by the scope of the claim language associated with the present disclosure.

[0096] In the foregoing description, the present disclosure has been described with reference to specific embodiments thereof. However, it should be clear that various modifications and changes may be made without departing from the broader spirit and scope of the present disclosure. For example, the above process flow is described with reference to a specific order of process actions. However, the order of many of the described process actions may be changed without affecting the scope or operation of the present disclosure. These descriptions and drawings are accordingly to be considered in an illustrative rather than a restrictive sense.

Claims

1. A computer-implemented method for distorting virtual content, comprising: The synthesizer receives first virtual content based on a first gesture of the display system; The synthesizer receives second virtual content based on a second gesture of the display system; as well as The compositor processes the first virtual content and the second virtual content in a single pass in a single pipeline, wherein processing the first virtual content and the second virtual content comprises: The compositor selects a first pixel of the first virtual content to correspond to an output pixel position, the compositor distorting the first pixel of the first virtual content based on a third gesture of the display system to generate a first distorted pixel, The compositor selects a second pixel of the second virtual content to correspond to the output pixel position, The compositor distorts the second pixel of the second virtual content based on the third gesture of the display system to generate a second distorted pixel, and The synthesizer compares a first depth of the first warped pixel and a second depth of the second warped pixel to identify a display pixel at the output pixel location.

2. The method according to claim 1, wherein: The first virtual content includes first image information at the first pixel.

3. The method according to claim 2, wherein: The first image information includes a first brightness.

4. The method according to claim 2, wherein: The first image information includes a first color.

5. The method according to claim 2, wherein: The second virtual content includes second image information at the second pixel.

6. The method according to claim 5, wherein: The second image information includes a second brightness.

7. The method according to claim 5, wherein: The second image information includes a second color.

8. The method according to claim 1, wherein: Comparing the first depth of the first distorted pixel and the second depth of the second distorted pixel comprises: determining the first depth of the first distorted pixel based on the third pose of the display system; determining the second depth of the second distorted pixel based on the third pose of the display system; and Based on the third posture of the display system, the first depth and the second depth are compared.

9. The method according to claim 1, wherein: Warping the first pixel of the first virtual content based on the third gesture of the display system includes applying a first transformation to the first pixel of the first virtual content to generate the first distorted pixel.

10. The method according to claim 1, wherein: Warping the second pixel of the second virtual content based on the third gesture of the display system includes applying a second transformation to the second pixel of the second virtual content to generate the second distorted pixel.

11. The method according to claim 1, further comprising: The compositor receives the first virtual content from a first application.

12. The method according to claim 11, further comprising: The compositor receives the second virtual content from a second application different from the first application.

13. The method according to claim 1, wherein: The synthesizer includes a grid operation unit.

14. The method according to claim 1, further comprising: The synthesizer writes the display pixels into a memory; The synthesizer sends the display pixels to a display unit; as well as The display unit displays an image to a user based on the display pixels.

15. The method according to claim 14, wherein: The display unit is a projector.

16. The method according to claim 1, wherein: The first virtual content is pixels of a first virtual object.

17. The method according to claim 1, wherein: The second virtual content is pixels of a second virtual object.

18. A system for distorting virtual content, comprising: A distortion unit, configured to receive first virtual content and second virtual content generated based on a first posture and a second posture of a display device, and generate corresponding first distorted virtual content and second distorted virtual content therefrom, the distortion unit comprising: a posture estimator for predicting a third posture of the display device; and a transform unit; and Synthesis units, including mixing units, The system processes the first virtual content and the second virtual content in a single round in a single pipeline, comprising: The warping unit selects a first pixel of the first virtual content to correspond to an output pixel position, The distortion unit distorts the first pixel of the first virtual content based on the third posture of the display device to generate a first distorted pixel, The warping unit selects a second pixel of the second virtual content to correspond to the output pixel position, The distortion unit distorts the second pixel of the second virtual content based on the third posture of the display device to generate a second distorted pixel, The synthesis unit compares a first depth of the first distorted pixel and a second depth of the second distorted pixel to identify a display pixel at the output pixel location.

19. The system of claim 18, wherein: The warping unit also includes a database, The system processes the first virtual content and the second virtual content in the single round in the single pipeline, comprising: The synthesis unit writes the display pixels into the database, The synthesis unit sends the display pixels to the display device, and The display device displays an image to a user based on the display pixels.

20. A computer program product embodied in a non-transitory computer readable medium having stored thereon a sequence of instructions which, when executed by a processor, cause the processor to perform a method for warping virtual content, the method comprising: The synthesizer receives first virtual content based on a first gesture of the display system; The synthesizer receives second virtual content based on a second gesture of the display system; as well as The compositor processes the first virtual content and the second virtual content in a single pass in a single pipeline, wherein processing the first virtual content and the second virtual content comprises: The compositor selects a first pixel of the first virtual content to correspond to an output pixel position, the compositor distorting the first pixel of the first virtual content based on a third gesture of the display system to generate a first distorted pixel, The compositor selects a second pixel of the second virtual content to correspond to the output pixel position, The compositor distorts the second pixel of the second virtual content based on the third gesture of the display system to generate a second distorted pixel, and The synthesizer compares a first depth of the first warped pixel and a second depth of the second warped pixel to identify a display pixel at the output pixel location.

21. A computer-implemented method for distorting virtual content, comprising: The compositor processes the first virtual content and the second virtual content in a single pass in a single pipeline, wherein the first virtual content and the second virtual content are based on corresponding first gestures and second gestures, and Wherein, processing the first virtual content and the second virtual content includes: The compositor selects a first pixel of the first virtual content to correspond to an output pixel position, the compositor distorts the first pixel of the first virtual content based on a third gesture of the display system to generate a first distorted pixel, The compositor selects a second pixel of the second virtual content to correspond to the output pixel position, The compositor distorts the second pixel of the second virtual content based on the third gesture of the display system to generate a second distorted pixel, and The synthesizer compares a first depth of the first warped pixel and a second depth of the second warped pixel to identify a display pixel at the output pixel location.

22. The method according to claim 21, wherein: The first virtual content includes first image information at the first pixel.

23. The method according to claim 22, wherein: The first image information includes a first brightness.

24. The method according to claim 22, wherein: The first image information includes a first color.

25. The method according to claim 22, wherein: The second virtual content includes second image information at the second pixel.

26. The method according to claim 25, wherein: The second image information includes a second brightness.

27. The method according to claim 25, wherein: The second image information includes a second color.

28. The method of claim 21, wherein: Comparing the first depth of the first distorted pixel and the second depth of the second distorted pixel comprises: determining the first depth of the first distorted pixel based on the third pose of the display system; determining the second depth of the second distorted pixel based on the third pose of the display system; and Based on the third posture of the display system, the first depth and the second depth are compared.

29. The method according to claim 21, wherein: Warping the first pixel of the first virtual content based on the third gesture of the display system includes applying a first transformation to the first pixel of the first virtual content to generate the first distorted pixel.

30. The method of claim 21, wherein: Warping the second pixel of the second virtual content based on the third gesture of the display system includes applying a second transformation to the second pixel of the second virtual content to generate the second distorted pixel.

31. The method of claim 21, further comprising: The compositor receives the first virtual content from a first application.

32. The method of claim 31 further comprising: The compositor receives the second virtual content from a second application different from the first application.

33. The method of claim 21, wherein: The synthesizer includes a grid operation unit.

34. The method of claim 21, further comprising: The synthesizer writes the display pixels into a memory; The synthesizer sends the display pixels to a display unit; as well as The display unit displays an image to a user based on the display pixels.

35. The method of claim 34, wherein: The display unit is a projector.

36. The method of claim 21, wherein: The first virtual content is pixels of a first virtual object.

37. The method of claim 21, wherein: The second virtual content is pixels of a second virtual object.

38. A system for distorting virtual content, comprising: a distortion unit, configured to receive first virtual content and second virtual content generated based on a first posture and a second posture of a display device; as well as a synthesis unit operatively coupled to the twisting unit, The system processes the first virtual content and the second virtual content in a single round in a single pipeline, comprising: The warping unit selects a first pixel of the first virtual content to correspond to an output pixel position, The distortion unit distorts the first pixel of the first virtual content based on a third posture of the display device to generate a first distorted pixel, The warping unit selects a second pixel of the second virtual content to correspond to the output pixel position, The distortion unit distorts the second pixel of the second virtual content based on the third posture of the display device to generate a second distorted pixel, and The synthesis unit compares a first depth of the first distorted pixel and a second depth of the second distorted pixel to identify a display pixel at the output pixel location.

39. The system according to claim 38, wherein the warping unit further comprises a database, in, The system, processing the first virtual content and the second virtual content in the single pass in the single pipeline, comprises: The synthesis unit writes the display pixels into the database, The synthesis unit sends the display pixels to a display device, and The display device displays an image to a user based on the display pixels.

40. A computer program product embodied in a non-transitory computer readable medium having stored thereon a sequence of instructions which, when executed by a processor, cause the processor to perform a method for warping virtual content, the method comprising: The compositor processes the first virtual content and the second virtual content in a single pass in a single pipeline, wherein the first virtual content and the second virtual content are based on corresponding first gestures and second gestures, and Wherein, processing the first virtual content and the second virtual content includes: The compositor selects a first pixel of the first virtual content to correspond to an output pixel position, the compositor distorts the first pixel of the first virtual content based on a third gesture of the display system to generate a first distorted pixel, The compositor selects a second pixel of the second virtual content to correspond to the output pixel position, The compositor distorts the second pixel of the second virtual content based on the third gesture of the display system to generate a second distorted pixel, and The synthesizer compares a first depth of the first warped pixel and a second depth of the second warped pixel to identify a display pixel at the output pixel location.

Images