Method and system for creating virtual and augmented reality - Patents.com

Abstract

To provide methods and systems for creating virtual and augmented reality.SOLUTION: Configurations are disclosed for presenting virtual reality and augmented reality experiences to users. The systems may comprise an image capturing device to capture one or more images, the one or more images corresponding to a field of view of a user of a head-mounted augmented reality device, and a processor communicatively coupled to the image capturing device to extract a set of map points from the set of images, to identify a set of sparse points and a set of dense points from the extracted set of map points, and to perform normalization on the set of map points.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears or can be perceived as real. Virtual reality, or "VR," scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual input, while augmented reality, or "AR," scenarios typically involve the presentation of digital or virtual image information as an extension of the user's visualization of the real world around them. For example, an augmented reality scene may enable a user of AR technology to see one or more virtual objects superimposed on or in between real-world objects (e.g., a real-world park-like setting featuring people, trees, and buildings in the background, etc.).

[0002] The human visual perception system is highly complex, making it difficult to create VR or AR technologies that facilitate a comfortable, natural-feeling, and rich presentation of virtual image elements among other virtual or real-world image elements. Conventional stereoscopic wearable glasses generally feature two displays configured to display images with slightly different presentations of elements so that a three-dimensional perspective is perceived by the human visual system. Such configurations have been found to be uncomfortable for many users due to the mismatch between vergence and accommodation, which can be overcome to perceive images in three dimensions. In fact, some users cannot tolerate stereoscopic configurations.

[0003] Although several optical configurations (e.g., head-mounted glasses) are available (e.g., Google Glass®, Occulus Rift®, etc.), none of these configurations are optimally suited to presenting rich binocular three-dimensional augmented reality experiences in a manner that would be comfortable and maximally useful to the user, in part because conventional systems fail to address some of the fundamental aspects of the human perceptual system, including the photoreceptors of the retina and their interaction with the brain to provide the user with the perception of visualization.

[0004] The human eye is a highly complex organ, typically comprising the cornea, iris, lens, macula, retina, and optic nerve pathways to the brain. The macula is the central part of the retina and is used for seeing medium details. Within the central part of the macula is a part of the retina called the "fovea," which is used for seeing the finest details of a scene and contains more photoreceptors (approximately 120 cones per diopter) than any other part of the retina.

[0005] The human visual system is not a passive sensor-type system; it actively scans the environment. In a manner somewhat similar to using a flatbed scanner to capture an image or using a finger to read Braille from paper, the eye's photoreceptors emit signals in response to changes in stimuli rather than in a constant response to a constant stimulus state. Therefore, movement is required to present photoreceptor information to the brain.

[0006] Indeed, experiments using substances such as cobra venom used to paralyze eye muscles have shown that human subjects will go blind if positioned with their eyes open and viewing a static scene with the venom-induced paralysis of the eye. In other words, in the absence of a change in stimulus, the photoreceptor cells provide no input to the brain, resulting in blindness. This is thought to be at least one reason why normal human eyes have been observed to move back and forth, or jerk, in side-to-side movements also known as "microsaccades."

[0007] As previously mentioned, the fovea of ​​the retina contains the greatest density of photoreceptors. While humans are typically perceived as having high-resolution visual perception throughout the entire visual field, in reality, humans possess only a small, high-resolution central fovea, which provides near-constant mechanical vision, along with persistent storage of recently captured high-resolution information. In a somewhat similar fashion, the eye's focal length control mechanism (e.g., the ciliary muscle, which is operatively coupled to the crystalline lens in such a way that ciliary relaxation induces tension in the connective fibers, flattening the lens for more distant focal distances, and ciliary contraction induces relaxation in the connective fibers, allowing the lens to assume a rounder geometry for closer focal distances) oscillates back and forth by approximately 1 / 4 to 1 / 2 diopter to periodically induce small amounts of "refractive blur" at both the near and far sides of the target focal distance. This is utilized by the brain's accommodation control circuitry as a periodic negative feedback that constantly corrects heading and helps to keep the retinal image of a fixed object approximately in focus.

[0008] The brain's visualization centers also derive valuable perceptual information from the relative movement of the eyes and their components. Vergence of the eyes relative to one another (e.g., the rolling of the pupils toward or away from one another to converge the gaze of the eyes and fixate on an object) is closely linked to the focusing of the eye's lenses (or "accommodation"). Under normal conditions, changing the focus of the eye's lenses, i.e., accommodating the eyes to focus on objects at different distances, will automatically produce a corresponding change in vergence to the same distance, a relationship known as the "accommodation-vergence reflex." Similarly, a change in vergence will also induce a corresponding change in accommodation under normal conditions. Countering this reflex (as in most conventional stereoscopic AR or VR configurations) is known to cause eye strain, headaches, or other forms of discomfort to the user.

[0009] Head movement, which retracts the eyes, also has a significant impact on object visualization. Humans tend to move their heads to visualize the world around them, often repositioning and reorienting their heads relative to the object of interest in a fairly constant manner. Furthermore, most people prefer to move their heads when their line of sight needs to move more than about 20 degrees off-center to focus on a particular object (e.g., people typically do not like to look "out of the corner of their eyes"). Humans also typically scan or move their heads in sync with sound to improve audio signal capture and take advantage of the geometry of the ears relative to the head. The human visual system derives excellent depth cues from what is called "head motion parallax," which relates to the relative motion of objects at different distances as a function of head motion and eye vergence distance. In other words, if a person moves their head laterally and maintains fixation on an object, items farther from the object will move in the same direction as the head, while items directly in front of the object will move in the opposite direction to the head motion. These can be very salient cues to where an object is spatially located in the environment relative to a person. Head movement is also, of course, used to look around for objects.

[0010] Furthermore, head and eye movements are coordinated with the "vestibulo-ocular reflex," which stabilizes image information on the retina during head rotation, thus keeping object image information approximately centered on the retina. In response to head rotation, the eyes reflexively and proportionally rotate in the opposite direction, maintaining a stable fixation on the object. As a result of this compensation relationship, many people can read a book while oscillating their head back and forth. Interestingly, the same is generally not true when a book is turned back and forth at the same speed while the head remains nearly stationary. That is, a person is unlikely to be able to read a book that is being turned. The vestibulo-ocular reflex is one of the motor coordinations of the head and eye that is generally not developed for hand movements. This paradigm can be important for AR systems because a user's head movement can be relatively directly related to eye movement, and an ideal system would preferably be prepared to work with this relationship.

[0011] Indeed, given these various relationships, when placing digital content (e.g., 3-D content such as a virtual chandelier object presented to augment the real-world view of a room, or 2-D content such as a plane / flat virtual oil painting object presented to augment the real-world view of a room), design choices can be made to control the behavior of the object. For example, a 2-D oil painting object can be head-centered, in which case the object moves with the user's head (e.g., as in the Google Glass® approach). In another example, the object can be world-centered, in which case it can be presented as if it were part of a real-world coordinate system so that the user can move their head or eyes without moving the object's position relative to the real world.

[0012] Thus, when placing virtual content in an augmented reality world presented using an AR system, a choice is made as to whether the object should be presented world-centered, body-centered, head-centered, or eye-centered. In a head-centered approach, a virtual object remains in a fixed position in the real world so that a user can move their body, head, or eyes around it without changing its position relative to surrounding real-world objects, such as real-world walls. In a body-centered approach, a virtual element may be fixed relative to the user's torso so that it follows torso movement, allowing a user to move their head or eyes without moving the object. In a head-centered approach, the displayed object (and / or the display itself) may move with head movement, as described above with reference to Google Glass®. In an eye-centered approach, content is followed as a function of eye position, as in a "foveated display" configuration as described below.

[0013] In a world-centric configuration, it may be desirable to have inputs such as accurate head pose measurements, accurate representations and / or measurements of real-world objects and geometry around the user, low-latency dynamic rendering in an augmented reality display as a function of head pose, and generally low-latency displays.

[0014] The aforementioned U.S. patent applications present systems and techniques for working with typical human visual configurations and addressing various challenges in virtual reality and augmented reality applications. The design of these virtual reality and / or AR systems presents numerous challenges, including the speed of the system in delivering virtual content, the quality of the virtual content, the user's pupil distance, the size and portability of the system, and other system and optical challenges.

[0015] The systems and techniques described herein are designed to work with typical human visual configurations and address these challenges. Summary of the Invention [Means for solving the problem]

[0016]

[0003] Embodiments of the present invention are directed to devices, systems, and methods for facilitating virtual and / or augmented reality interaction for one or more users. In one aspect, a system for displaying virtual content is disclosed.

[0017] In one aspect, an augmented reality display system includes an image capture device for capturing one or more images, the one or more images corresponding to a field of view of a user of a head-mounted augmented reality device, and a processor communicatively coupled to the image capture device for extracting a set of map points from the set of images, identifying a set of sparse points and a set of dense points from the extracted set of map points, and performing normalization on the set of map points. In one or more embodiments, the processor may generate sparse and dense point descriptors for the set of sparse points and the set of dense points, respectively.

[0018] In one or more embodiments, the sparse point descriptors and the dense point descriptors are stored as map data. In one or more embodiments, the collection of sparse points corresponds to distinct features of one or more images. In one or more embodiments, the distinct features are selected from the group consisting of corners, circles, triangles, and text.

[0019] In one or more embodiments, the set of dense points corresponds to 3D points in the field of view. In one or more embodiments, the set of dense points further includes color values. In one or more embodiments, the normalization comprises scale normalization. In one or more embodiments, the normalization comprises coordinate normalization relative to a common origin. In one or more embodiments, the normalization utilizes machine learning. In one or more embodiments, sparse and dense point descriptors correspond to respective sparse and dense points of the set of sparse and dense points. In one or more embodiments, the sparse and dense point descriptors comprise information regarding at least one of scale, texture, orientation, and patch data.

[0020] In another aspect, a method for generating map data includes identifying sets of map points associated with one or more images, determining a set of sparse points and a set of dense points from the identified map points, and normalizing the respective sets of sparse points and dense points.

[0021] In one or more embodiments, the method further includes generating sparse and dense point descriptors for each set of sparse points and each set of dense points, and combining and storing the sparse and dense point descriptors as map data.

[0022] In one or more embodiments, the set of sparse points corresponds to distinct features. In one or more embodiments, the distinct features are selected from the group consisting of corners, circles, triangles, and text. In one or more embodiments, the set of dense points corresponds to 3D points in the field of view. In one or more embodiments, the set of dense points also includes color values.

[0023] In one or more embodiments, the normalization comprises scale normalization. In one or more embodiments, the normalization comprises coordinate normalization relative to a common origin. In one or more embodiments, the normalization is implemented using machine learning. In one or more embodiments, a sparse and dense point descriptor corresponds to each sparse and dense point of the respective sets of sparse and dense points. In one or more embodiments, each sparse and dense point descriptor includes information for the respective sparse and dense point selected from the group consisting of scale, orientation, patch data, and texture.

[0024] In another aspect, an augmented reality display system includes an image capture device for capturing one or more images, the one or more images corresponding to a user's field of view, the images capturing at least one gesture made by the user; and a processor communicatively coupled to the image capture device and configured to identify a set of points associated with the gesture, compare the set of points to a database of predetermined gestures, recognize the gesture based at least in part on the comparison, and determine a user input based at least in part on the recognized gesture.

[0025] In one or more embodiments, the processor generates a scoring value for the set of identified points based on the comparison. In one or more embodiments, the processor recognizes the gesture if the scoring value exceeds a threshold. In one or more embodiments, the augmented reality display system includes a database for storing the set of predefined gestures. In one or more embodiments, the system further includes a networked memory for accessing the database of predefined gestures.

[0026] In one or more embodiments, the gesture is a hand gesture. In one or more embodiments, the gesture is a finger gesture. In one or more embodiments, the gesture is a finger-to-finger interaction. In one or more embodiments, the gesture is selected from the group consisting of a finger-to-finger interaction, a pointing, a tap, and a rub.

[0027] In one or more embodiments, the augmented reality display system further comprises a spatial light modulator, the spatial light modulator communicatively coupled to the processor, the processor controlling the spatial light modulator such that one or more virtual objects are displayed to the user based at least in part on the determined user input, hi one or more embodiments, the one or more virtual objects comprise a virtual user interface.

[0028] In another aspect, a method for determining user input includes capturing an image of a user's field of view, the image including a gesture made by the user; analyzing the captured image to identify a set of points associated with the gesture; comparing the identified set of points with a set of points associated with a database of predetermined gestures; and determining the user input based on the recognized gesture.

[0029] In one or more embodiments, the method further includes generating a scored value for the set of identified points based on the comparison. In one or more embodiments, the method further includes recognizing the gesture if the scored value exceeds a threshold. In one or more embodiments, the method further includes storing the predetermined gesture in a database. In one or more embodiments, the method further includes accessing a networked memory and accessing the database of predetermined gestures.

[0030] In one or more embodiments, the gesture is a hand gesture. In one or more embodiments, the gesture is a finger gesture. In one or more embodiments, the gesture is a finger-to-finger interaction. In one or more embodiments, the gesture is selected from the group consisting of a finger-to-finger interaction, a pointing, a tap, and a rub.

[0031] In one or more embodiments, the method further includes displaying one or more virtual objects to the user based at least in part on the determined user input, hi one or more embodiments, the one or more virtual objects comprise a virtual user interface.

[0032] In another aspect, an augmented reality display system includes an image capture device for capturing one or more images and a processor communicatively coupled to the image capture device for analyzing the captured images and identifying one or more gestures made by a user, wherein identifying the one or more gestures includes utilizing a cascade mechanism having multiple stages.

[0033] In one or more embodiments, the cascading mechanism comprises a plurality of nodes, each node corresponding to a stage of the plurality of stages. In one or more embodiments, the cascading mechanism comprises a series of permissive analysis nodes. In one or more embodiments, earlier stages of the cascading mechanism are configured to consume less processing power than later stages of the cascading mechanism.

[0034] In one or more embodiments, non-gestures are removed based on an analysis of images captured in an earlier stage of the cascade mechanism, hi one or more embodiments, a later stage of the plurality of stages is configured to determine more complex gestures based at least in part on the captured images.

[0035] In one or more embodiments, analyzing the captured image includes determining whether the sharpness of contours of various shapes in the captured image is sufficiently sharp to constitute a gesture. In one or more embodiments, a later stage of the cascade mechanism is utilized to distinguish between different gestures. In one or more embodiments, the processor is further configured to generate a score based at least in part on the analysis. In one or more embodiments, the processor removes the candidate image from consideration if the generated score is below a minimum threshold. In one or more embodiments, the processor advances to a later stage of the cascade mechanism if the generated score is above a minimum threshold.

[0036] In another aspect, a method includes capturing one or more images corresponding to a field of view of a user and analyzing the captured one or more images to identify one or more gestures made by the user, wherein the analysis includes utilizing a cascading mechanism having multiple stages.

[0037] In one or more embodiments, the cascading mechanism comprises a plurality of nodes, each node corresponding to a stage of the plurality of stages. In one or more embodiments, the cascading mechanism comprises a series of permissive analysis nodes. In one or more embodiments, earlier stages of the cascading mechanism are configured to consume less processing power than later stages of the cascading mechanism.

[0038] In one or more embodiments, non-gestures are removed based on an analysis of images captured in an earlier stage of the cascade mechanism, hi one or more embodiments, a later stage of the plurality of stages is configured to determine more complex gestures based at least in part on the captured images.

[0039] In one or more embodiments, analyzing the captured image includes determining whether the sharpness of the contours of various shapes in the captured image are sufficiently sharp to constitute a gesture, hi one or more embodiments, later stages of a cascade mechanism are utilized to distinguish between different gestures.

[0040] In one or more embodiments, the method further includes generating a score based at least in part on the analysis. In one or more embodiments, the method further includes removing a candidate image from consideration if the generated score is below a minimum threshold. In one or more embodiments, the method further includes advancing to a later stage in a cascade mechanism if the generated score is above the minimum threshold.

[0041] In another aspect, an augmented reality system includes an image capture device for capturing a plurality of images of a user's respective fields of view, and a processor communicatively coupled to the image capture device for analyzing the plurality of images, generating a plurality of gesture candidates from the captured plurality of images, and generating an analysis value corresponding to the plurality of gesture candidates, wherein a gesture is recognized based at least in part on the analysis value.

[0042] In one or more embodiments, the processor is further configured to sort the gesture candidates based at least in part on the respective analysis values. In one or more embodiments, the processor is further configured to eliminate gesture candidates having analysis values ​​below a minimum threshold. In one or more embodiments, the processor is further configured to advance a gesture candidate to a next stage of processing if the analysis value is above the minimum threshold.

[0043] In another aspect, a method includes capturing a plurality of images of a user's respective fields of view, analyzing the plurality of images to generate a plurality of gesture candidates, and generating an analysis value corresponding to the plurality of analysis values, wherein the gesture is recognized based at least in part on the analysis value.

[0044] In one or more embodiments, the method further includes sorting the gesture candidates based at least in part on their respective analysis values. In one or more embodiments, the method further includes eliminating gesture candidates having analysis values ​​below a minimum threshold. In one or more embodiments, the method further includes advancing a gesture candidate to a next stage of processing if the analysis value is above a minimum threshold.

[0045] In yet another aspect, an augmented reality display system includes an image capture device for capturing an image of a user's field of view and a processor configured to generate a depth map corresponding to the captured image, analyze the depth map of the captured image, and identify a gesture.

[0046] In one or more embodiments, the processor is further configured to utilize a classification mechanism to identify hand portions corresponding to points in the generated depth map. In one or more embodiments, the processor is further configured to skeletonize the depth map based on the identification of the hand portions.

[0047] In one or more embodiments, the processor classifies the image as a gesture based at least in part on the skeletonized depth map. In one or more embodiments, the depth is generated at least in part by performing a depth partitioning process.

[0048] In one or more embodiments, the depth segmentation comprises a line search. In one or more embodiments, the processor performs a cascade analysis on the depth map to classify the image as a gesture. In one or more embodiments, the processor performs depth dilation on the depth map. In one or more embodiments, the processor performs surface normalization on the depth map.

[0049] In one or more embodiments, the processor performs orientation normalization on the depth map. In one or more embodiments, the processor performs background subtraction on the depth map. In one or more embodiments, the processor performs depth comparison on the depth map. In one or more embodiments, the processor classifies the image as a gesture based at least in part on the skeletonized depth map and previous information. In one or more embodiments, the classifier is a decision forest.

[0050] In another aspect, a method for classifying gestures includes capturing an image of a user's field of view, performing depth partitioning on the captured image to generate a depth map, and identifying a gesture based at least in part on the generated depth map.

[0051] In one or more embodiments, the method further includes analyzing the depth map using a classification mechanism to identify hand portions corresponding to points in the depth map. In one or more embodiments, the method further includes skeletonizing the depth map based on the identification of the hand portions. In one or more embodiments, the method further includes classifying the image as a gesture based on the skeletonized depth map.

[0052] In one or more embodiments, the depth segmentation comprises a line search. In one or more embodiments, the method further comprises performing a cascade analysis on the depth map to classify the image as a gesture. In one or more embodiments, the method further comprises performing depth dilation on the depth map.

[0053] In one or more embodiments, the method further comprises performing surface normalization on the depth map. In one or more embodiments, the method further comprises performing orientation normalization on the depth map. In one or more embodiments, the method further comprises performing background subtraction on the depth map.

[0054] In one or more embodiments, the method further includes performing a depth comparison to the depth map. In one or more embodiments, the method further includes classifying the image as a gesture based on the skeletonized depth map and prior information. In one or more embodiments, the classifier is a decision forest.

[0055] In another aspect, an augmented reality display system includes an image capture device for capturing a collection of images of a user's field of view, the collection of images capturing movement of the user relative to a predetermined physical object; and a processor communicatively coupled to the image capture device for analyzing the movement relative to the predetermined physical object and for determining user input based at least in part on the analyzed movement.

[0056] In one or more embodiments, the processor recognizes a predetermined physical object. In one or more embodiments, the predetermined physical object is recognized based at least in part on an image comparison between a captured image of the predetermined physical object and a database of predetermined physical objects. In one or more embodiments, the analyzed movement of the user relative to the predetermined physical object is used to generate a pattern. In one or more embodiments, the generated pattern is compared to a database of predetermined patterns. In one or more embodiments, the processor generates a scoring value for the recognized pattern based on the comparison.

[0057] In one or more embodiments, the processor determines a user input if the scored value exceeds a threshold. In one or more embodiments, the image capture device visually tracks movement relative to a predetermined physical object and generates a video recording. In one or more embodiments, the video recording is analyzed to determine a user input.

[0058] In one or more embodiments, the augmented reality display system further comprises a networked memory for accessing a database of predetermined patterns. In one or more embodiments, the predetermined physical objects are selected from the group consisting of existing structures in the field of view, actively marked totems, passively marked totems, camera / sensor integrated objects, and totem controller objects.

[0059] In one or more embodiments, the movement relative to the predetermined physical object is selected from the group consisting of a position, an orientation, and a movement of the predetermined physical object relative to a frame of reference. In one or more embodiments, the predetermined physical object comprises a first one of the user's hands, and the movement relative to the first hand comprises manipulating the first hand with a second one of the user's hands.

[0060] In one or more embodiments, the predetermined physics object has a soft hardness surface, and movement relating to the predetermined physics object comprises a user pressing the soft hardness surface. In one or more embodiments, the processor renders virtual interface elements associated with the predetermined physics object for the user, the virtual interface elements being viewed by the user through a display device.

[0061] In one or more embodiments, the virtual interface element is displayed relative to the predetermined physical object such that, when viewed by a user, the virtual interface element is modified based, at least in part, on the modification relative to the predetermined physical object. In one or more embodiments, the predetermined physical object comprises an electronic input device, and user input is determined and entered from the electronic input device based on the perceived movement of the predetermined physical object.

[0062] In one or more embodiments, the processor is configured to control a display device coupled to the user's eye such that virtual content displayed to the user is modified based at least in part on the determined user input.

[0063] In another aspect, a method for determining user input includes capturing images of one or more user's field of view, the images comprising at least a predetermined physical object; analyzing the images to detect user movement relative to the predetermined physical object; and determining user input based at least in part on the analysis of the movement relative to the predetermined physical object.

[0064] In one or more embodiments, the method further includes recognizing a predetermined physical object. In one or more embodiments, the predetermined physical object is recognized based at least in part on an image comparison between the captured image of the predetermined physical object and a database of predetermined physical objects. In one or more embodiments, movements of the user relative to the predetermined physical object are used to generate the pattern.

[0065] In one or more embodiments, the method further includes comparing the generated pattern to a database of predetermined patterns. In one or more embodiments, the method further includes generating a scoring value for the recognized pattern based on the comparison. In one or more embodiments, the method further includes determining a user input if the scoring value exceeds a threshold.

[0066] In one or more embodiments, the method further includes visually tracking movement relative to a predetermined physical object and generating a video recording. In one or more embodiments, the video recording is analyzed to determine user input. In one or more embodiments, the method further includes accessing a database of predetermined patterns through a networked memory. In one or more embodiments, the predetermined physical object is selected from the group consisting of existing structures in the field of view, actively marked totems, passively marked totems, camera / sensor integrated objects, and totem controller objects.

[0067] In one or more embodiments, the movement relative to the predetermined physical object is selected from the group consisting of a position, an orientation, and a movement of the predetermined physical object relative to a frame of reference. In one or more embodiments, the predetermined physical object comprises a first one of the user's hands, and the movement relative to the first hand comprises manipulating the first hand with a second one of the user's hands.

[0068] In one or more embodiments, the predetermined physics object has a soft hardness surface, and the movement relating to the predetermined physics object comprises a user pressing the soft hardness surface. In one or more embodiments, the method further includes rendering a virtual interface element associated with the predetermined physics object for the user.

[0069] In one or more embodiments, the virtual interface element is displayed relative to the predetermined physical object such that, when viewed by a user, the virtual interface element is modified based, at least in part, on the modification relative to the predetermined physical object. In one or more embodiments, the predetermined physical object comprises an electronic input device, and user input is determined and entered from the electronic input device based on the perceived movement of the predetermined physical object. In one or more embodiments, the method further includes modifying at least one characteristic of the virtual content displayed to the user based, at least in part, on the determined user input.

[0070] In another aspect, an augmented reality display system includes a display physically coupled to a user's eye, the display displaying a collection of virtual content related to one or more physical objects; and a processor communicatively coupled to the display and the image capture device, the processor identifying, based on user input, a virtual user interface to be displayed to the user via the display, obtaining the user's location relative to the world, determining a set of coordinate points at which to display the virtual user interface, and controlling the display such that the identified virtual user interface is displayed to the user.

[0071] In one or more embodiments, the user input is determined at least in part based on a recognized gesture. In one or more embodiments, the user input is determined at least in part based on a voice command. In one or more embodiments, the user input is determined at least in part based on an interaction with a predetermined physical object.

[0072] In one or more embodiments, the augmented reality display system further comprises a library of user interfaces, and the identified user interface is retrieved from the library of user interfaces. In one or more embodiments, the identified virtual user interface is associated with a frame of reference. In one or more embodiments, the frame of reference is a body-centered frame of reference.

[0073] In one or more embodiments, the reference frame is a head-centered reference frame. In one or more embodiments, the reference frame is a hand-centered reference frame. In one or more embodiments, the reference frame is a world-centered reference frame. In one or more embodiments, the processor performs a transformation between the reference frame associated with the identified virtual user interface and the user's resulting location relative to the world to determine a set of coordinate points of the virtual user interface.

[0074] In one or more embodiments, the user's location within the world is determined based on the user's GPS location. In one or more embodiments, the user's location within the world is determined based on a set of map points associated with the user. In one or more embodiments, the virtual user interface appears stationary as the user moves.

[0075] In one or more embodiments, the virtual user interface moves relative to the user's movements. In one or more embodiments, the processor determines a frame of reference associated with the identified virtual user interface, determines a location of the frame of reference relative to a world frame of reference, sets the determined location as an origin, and determines a set of coordinate points relative to the origin.

[0076] In another aspect, a method for generating a virtual user interface includes identifying a virtual user interface to be displayed to a user based on user input; obtaining a location associated with the user; determining, based at least in part on the obtained location, a set of coordinate points at which to display the identified virtual user interface; and displaying the virtual user interface to the user at the determined coordinate points.

[0077] In one or more embodiments, the user input is determined at least in part based on a recognized gesture. In one or more embodiments, the user input is determined at least in part based on a voice command. In one or more embodiments, the user input is determined at least in part based on an interaction with a predetermined physical object.

[0078] In one or more embodiments, the method further includes retrieving the identified user virtual user interface from a library of user interfaces. In one or more embodiments, the identified virtual user interface is associated with a reference frame. In one or more embodiments, the reference frame is a body-centered reference frame.

[0079] In one or more embodiments, the reference frame is a head-centered reference frame, in one or more embodiments, the reference frame is a hand-centered reference frame, in one or more embodiments, the reference frame is a world-centered reference frame.

[0080] In one or more embodiments, the method further includes transforming between a frame of reference associated with the identified virtual user interface and a resultant location of the user relative to the world to determine a set of coordinate points of the virtual user interface. In one or more embodiments, the location of the user within the world is determined based on the user's GPS location.

[0081] In one or more embodiments, the user's location within the world is determined based on a set of map points associated with the user. In one or more embodiments, the virtual user interface appears stationary as the user moves. In one or more embodiments, the virtual user interface moves in relation to the user's movements.

[0082] In one or more embodiments, the method further includes determining a reference frame associated with the identified virtual user interface, determining a location of the reference frame relative to a world reference frame, setting the determined location as an origin, and determining a set of coordinate points relative to the origin.

[0083] In one or more embodiments, the method further includes retrieving the set of map points from the networked memory. In one or more embodiments, the user input comprises a location in space where the virtual user interface is to be displayed. In one or more embodiments, the location in space is associated with a physical entity at the user's location. In one or more embodiments, the user input comprises a gesture including a throwing input indicating a wall. In one or more embodiments, the user input comprises a gesture signifying the end of an instruction to generate the virtual user interface.

[0084] In another aspect, a method for generating a virtual user interface includes detecting a manipulation of a predetermined physical object; recognizing a command for creating a virtual user interface based on the detected manipulation; determining, from a virtual world model, a set of map points associated with the position of the predetermined physical object; and rendering the virtual user interface in real time at the determined map points associated with the position of a totem such that the virtual user interface, when viewed by a user, appears to be stationary at the position of the predetermined physical object.

[0085] In one or more embodiments, the manipulation of the predetermined physical object includes a pinch-opening gesture by the user's hand on a surface of the predetermined physical object. In one or more embodiments, the virtual user interface, when viewed by the user, appears to cover a portion of the surface of the predetermined physical object, the portion corresponding to the location of the user's hand during the formation of the pinch-opening gesture. In one or more embodiments, the predetermined physical object is the user's hand.

[0086] In one or more embodiments, the manipulation of the predetermined physical object includes an action selected from the group consisting of a user opening their hand, a user showing an open palm, and a user raising their hand. In one or more embodiments, the virtual user interface, when viewed by the user, appears to cover a portion of the surface of the hand.

[0087] In one or more embodiments, the virtual user interface comprises a plurality of first level menu items selectable by a finger or a thumb of a hand. In one or more embodiments, the method further includes detecting a further manipulation of the hand, recognizing a command for creating a second virtual user interface based on the detected further manipulation, and rendering the second virtual user interface in real time at the determined map point associated with the location of the predetermined physical object such that the virtual user interface, when viewed by the user, appears to be stationary at the location of the predetermined physical object.

[0088] In one or more embodiments, the method further includes further manipulating the hand, including spreading the fingers of the hand apart. In one or more embodiments, the second virtual user interface comprises a plurality of second level menu items selectable by the fingers or thumb of the hand, the second level being lower than the first level.

[0089] In one or more embodiments, further manipulation of the totem includes using fingers from a second one of the user's hands to perform a circular motion on the palm of the hand. In one or more embodiments, the second virtual user interface comprises a plurality of menu items arranged in an arc, the menu items being scrollable and selectable by the fingers of the second hand.

[0090] In another aspect, an augmented reality display system includes an image capture device for capturing one or more images of a user's field of view, at least one image capturing manipulation of a predetermined physical object; a display device for displaying one or more virtual objects to the user; a database comprising a passable world model, the passable world model comprising a collection of map points associated with physical objects of the world; and a processor communicatively coupled to the image capture device, the processor recognizing commands to create a virtual user interface based on the one or more images, determining map points corresponding to the predetermined physical object based at least in part on the passable world model, and controlling the display in a manner such that the virtual user interface is generated at the determined map points corresponding to the predetermined physical object so that the virtual user interface appears to be stationary at the location of the predetermined physical object.

[0091] In one or more embodiments, the manipulation of the predetermined physical object includes a pinch-opening gesture by the user's hand on a surface of the predetermined physical object. In one or more embodiments, the virtual user interface, when viewed by the user, appears to cover a portion of the surface of the predetermined physical object, the portion corresponding to the location of the user's hand during the formation of the pinch-opening gesture. In one or more embodiments, the predetermined physical object is the user's hand.

[0092] In one or more embodiments, the manipulation of the predetermined physical object includes an action selected from the group consisting of a user opening their hand, a user showing an open palm, and a user raising their hand. In one or more embodiments, the virtual user interface, when viewed by the user, appears to cover a portion of the surface of the hand.

[0093] In one or more embodiments, the virtual user interface comprises a plurality of first level menu items selectable by a finger or thumb of a hand. In one or more embodiments, the predetermined physical object is a hand, and the processor detects further manipulation of the hand, recognizes a command to create a second virtual user interface, and controls the display such that the second virtual user interface is displayed at the determined map point.

[0094] In one or more embodiments, the further manipulation of the hand includes spreading the fingers of the hand apart. In one or more embodiments, the second virtual user interface comprises a plurality of second level menu items selectable by the fingers or thumb of the hand, the second level being lower than the first level. In one or more embodiments, the further manipulation of the totem includes using the fingers from a second of the user's hands to perform a circular motion on the palm of the hand.

[0095] In one or more embodiments, the second virtual user interface comprises a plurality of menu items arranged in an arc, the menu items being scrollable and selectable by the fingers of the second hand.

[0096] In another aspect, a method for updating a virtual world includes receiving a first input from a first device of a first user, the first input corresponding to a physical environment of the first user; updating a virtual world model based on the received first input, the virtual world model corresponding to the physical environment of the first user; and transmitting first updated information corresponding to a first portion of the virtual world model to a second user, the first updated information configured to indicate whether any portion of the first updated information needs to be displayed to the second user.

[0097] In one or more embodiments, the virtual world model resides on a networked memory. In one or more embodiments, the first user and the second user are located at different locations. In one or more embodiments, the first device of the first user is selected from the group consisting of an FOV camera, another camera, a sensor, a first device for eye tracking, and a first device for audio.

[0098] In one or more embodiments, the method further includes transmitting first updated information corresponding to the first portion of the virtual world model to the first user, the first updated information configured to indicate whether any portions of the first updated information should be displayed to the first user. In one or more embodiments, the method further includes transmitting first updated information corresponding to the first portion of the virtual world model to a plurality of other users, the first updated information configured to indicate whether any portions of the first updated information should be displayed to each of the plurality of other users.

[0099] In one or more embodiments, the method further includes receiving a plurality of inputs from a first device of each of a plurality of other users, the plurality of inputs corresponding to the physical environment of the first user; updating the virtual world model based on the received plurality of inputs; and transmitting additional updated information corresponding to each additional portion of the virtual world model to the second user, the additional updated information being configured to indicate whether any portion of the additional updated information needs to be displayed to the second user.

[0100] In one or more embodiments, the method further includes transmitting additional updated information corresponding to a portion of the virtual world model to the first user, the additional updated information configured to indicate whether any portion of the additional updated information needs to be displayed to the first user.

[0101] In one or more embodiments, the method further includes transmitting additional updated information corresponding to each additional portion of the virtual world model to the plurality of other users, the additional updated information being configured to indicate whether any portion of the additional updated information needs to be displayed to each of the plurality of other users.

[0102] In one or more embodiments, the method further includes receiving a second input from a second device of a second user, the second input corresponding to a physical environment of the second user, updating the virtual world model based on the received second input, the virtual world model corresponding to the physical environment of the first user, and transmitting second updated information to the first user corresponding to a second portion of the virtual world model, the second updated information being configured to indicate whether any portion of the second updated information needs to be displayed to the first user. In one or more embodiments, the second updated information corresponds to movement of an avatar of the second user within the second portion of the virtual world model.

[0103] In another aspect, a method includes projecting a light pattern onto a space; detecting, with a camera, first and second portions of the light pattern and generating first and second data points corresponding to the first and second portions of the pattern; and performing a triangulation analysis to determine locations of the first and second portions of the light pattern based at least in part on the first and second data points.

[0104] In one or more embodiments, the method further includes repeating the steps of projecting, detecting, and triangulating multiple portions of the pattern to obtain additional texture data for the space. In one or more embodiments, projecting light onto the space includes projecting light onto the space using a fiber-based projector.

[0105] In one or more embodiments, the method further comprises modifying the light with an optical element to form a bundle of light rays before the light is projected onto the space. In one or more embodiments, the light pattern corresponds to structured light, and the light pattern is dynamic. In one or more embodiments, the light pattern corresponds to patterned light, and the light pattern includes a plurality of points.

[0106] In one or more embodiments, the light pattern corresponds to a textured light, and the light pattern is irregular. In one or more embodiments, the first and second portions of the light pattern are respective pixels of the light pattern.

[0107] In another aspect, an augmented reality display system includes a light projector for projecting a light pattern into space; a camera coupled to a frame of the augmented reality display system for capturing an image of a user's field of view, the camera capturing first and second portions corresponding to the projected light pattern; and a processor communicatively coupled to the camera for detecting first and second data points corresponding to the first and second portions of the pattern based at least in part on data received from the camera, performing a triangulation analysis, and determining locations of the first and second portions of the light pattern based at least in part on the first and second data points.

[0108] In one or more embodiments, the processor repeats the steps of projecting, detecting, and triangulating with multiple portions of the pattern to obtain additional texture data for the space. In one or more embodiments, the light projector comprises a fiber-based projector. In one or more embodiments, the projected light is modified using optical elements to form a bundle of light rays before the light is projected onto the space.

[0109] In one or more embodiments, the light pattern corresponds to structured light, and the light pattern is dynamic. In one or more embodiments, the light pattern corresponds to patterned light, and the light pattern includes a plurality of points. In one or more embodiments, the light pattern corresponds to textured light, and the light pattern is irregular. In one or more embodiments, the first and second portions of the light pattern are respective pixels of the light pattern.

[0110] In another aspect, an augmented reality system includes a predetermined physical object having a surface configured to be manipulated such that the manipulation is detectable as user input by the augmented reality system, the predetermined physical object comprising a material configured to provide a tactile perception to a user when the user interacts with the predetermined physical object via touch.

[0111] In one or more embodiments, the predetermined physical object is configured to simulate a user input device, hi one or more embodiments, the user input device is selected from the group consisting of a computer keyboard, a computer mouse, a computer trackpad, and a handheld controller.

[0112] In one or more embodiments, the system further includes a display device for displaying one or more virtual objects to a user, and a processor for controlling the display device such that the virtual user interface is displayed on the surface of a predetermined physical object.

[0113] In one or more embodiments, the predetermined physics object has a spherical shape. In one or more embodiments, the system is configured to display the virtual user interface such that the virtual user interface appears to emerge from the predetermined physics object. In one or more embodiments, the predetermined physics object has a textured outer surface.

[0114] In one or more embodiments, the predetermined physical object comprises a user input element. In one or more embodiments, the user input element is selected from the group consisting of a key, a button, and a scroll wheel. In one or more embodiments, the user input element is not operably coupled to an electronic device. In one or more embodiments, the system further comprises a camera configured to detect user manipulation of the totem.

[0115] In one or more embodiments, the predetermined physical object comprises a depression, a cavity, or a protrusion. In one or more embodiments, the manipulation is selected from the group of finger movements consisting of: a swipe, a rocking, a rotating, a scrolling, a tap, a double tap, a short tap, and a long tap.

[0116] In one or more embodiments, the manipulation is selected from a group of finger movement characteristics consisting of a number of interactions, a type of interaction, and a duration of the interaction. In one or more embodiments, the manipulation is selected from a group of finger characteristics consisting of distance, direction, speed, and acceleration.

[0117] In yet another aspect, a method for generating a virtual room includes identifying a virtual room to be displayed to a user; obtaining a first set of map points associated with the virtual room, the first set of map points corresponding to a first physical room at the user's location; tethering the virtual room to the first set of map points; and displaying the virtual room to the user such that, when viewed by the user, the virtual room appears stationary in the set of the first set of map points.

[0118] In one or more embodiments, the method further includes identifying the virtual room based on user input, wherein the user input is selected from the group consisting of a gesture, visual data, auditory data, sensory data, a direct command, a voice command, eye tracking, and a physical button selection.

[0119] In one or more embodiments, the virtual room is provided by a user. In one or more embodiments, the method further includes retrieving the first set of map points from the networked memory. In one or more embodiments, the method further includes obtaining a second set of map points associated with the virtual room, the second set of map points corresponding to a second physical room at the user's location after the user moves from the first physical room; tethering the virtual room to the second set of map points; and displaying the virtual room to the user such that, when viewed by the user, the virtual room appears stationary in the second set of map points.

[0120] In one or more embodiments, the virtual room also includes first and second virtual objects, and the method further includes, when the virtual room is tethered to the second set of map points, preserving a relative spatial position between the first virtual object and the second virtual object corresponding to the first set of map points. In one or more embodiments, the method further includes displaying the virtual room to the user such that, when viewed by the user, the virtual room appears stationary relative to the portion of the first physical room. In one or more embodiments, the virtual room also includes selectable virtual decorations.

[0121] In another aspect, a method for creating a retail experience includes recognizing a location of a user within a retail store, retrieving data corresponding to the retail store, generating virtual content related to the retail store based on the retrieved data, creating a virtual user interface within a field of view of the user, and displaying the virtual content on the virtual user interface while the user engages in retail activities within the retail store.

[0122] In one or more embodiments, the retrieved data comprises a set of map points corresponding to retail stores, and the virtual user interface, when viewed by the user, appears stationary at the set of map points. In one or more embodiments, the user's location is recognized using a radio frequency identification transponder and communications. In one or more embodiments, the virtual content is selected from the group consisting of virtual characters, virtual coupons, games based on the user's location, promotional item listings, nutritional information, metadata associated with items, celebrity appearances, cross-selling advertisements, information from people known to the user, and e-books.

[0123] In one or more embodiments, the method further includes retrieving user data corresponding to the user and generating virtual content based on both the retrieved data and the retrieved user data. In one or more embodiments, the virtual content is selected from the group consisting of a virtual grocery list, a virtual coupon book, a virtual recipe book, a list of various ingredients in the user's home, and a virtual recipe builder. In one or more embodiments, the method further includes receiving user input, generating additional virtual content based on the user input, and displaying the additional virtual content on the virtual user interface while the user engages in retail activities in the retail store.

[0124] In one or more embodiments, the user input is selected from the group consisting of gestures, visual data, auditory data, sensory data, direct commands, voice commands, eye tracking, and physical button selection. In one or more embodiments, the virtual content is selected from the group consisting of a total operating cost, a smart virtual grocery list, an indicator of items in proximity to the user location, and a virtual payment system. In one or more embodiments, the method further includes transmitting the generated virtual content to the user device for display.

[0125] In another aspect, a method includes retrieving patient data related to a patient's medical history; generating virtual content based at least in part on the retrieved patient data; retrieving a first set of map points corresponding to a first location of a first user; creating a first virtual user interface within a field of view of the first user based at least in part on the retrieved set of map points; and displaying the virtual content on the first virtual user interface such that the first virtual user interface appears to be fixed at the first set of map points when viewed by the first user.

[0126] In one or more embodiments, the virtual content is selected from the group consisting of a three-dimensional image of a surgical target, patient identification information, medical images, patient vital sign information, and medical records. In one or more embodiments, the patient data is retrieved from a networked memory.

[0127] In one or more embodiments, the method further includes generating a second virtual user interface configured to facilitate communication between the first user and a second user, the second user being in a second location different from the first location of the first user. In one or more embodiments, the second virtual user interface is a visual representation of the second user. In one or more embodiments, the second user is selected from the group consisting of a senior specialist surgeon, a patient, an associate of the patient, and a medical student. In one or more embodiments, the method further includes displaying virtual content to the second user on the first virtual user interface such that the first virtual user interface, when viewed by the second user, appears to be fixed at the set of map points.

[0128] In one or more embodiments, the virtual content is displayed to the first user during the surgical procedure, and the method further includes receiving user input, generating additional virtual content based on the user input, and displaying the additional virtual content on the first virtual user interface while the first user is performing the surgical procedure.

[0129] In one or more embodiments, the user input is selected from the group consisting of a gesture, visual data, auditory data, sensory data, a direct command, a voice command, eye tracking, and a physical button selection. In one or more embodiments, the user input comprises an image of the first user's field of view, and the method further includes displaying additional virtual content on the first virtual user interface to the second user such that the first virtual user interface, when viewed by the second user, appears to be fixed at the set of map points.

[0130] In one or more embodiments, a method for facilitating medical rehabilitation includes receiving user input related to medical rehabilitation; sending a request related to medical rehabilitation to a cloud server based on the received user input; receiving data related to medical rehabilitation from a knowledge base connected to the cloud server based on the request; and displaying virtual content related to medical rehabilitation to a first user based on the received data.

[0131] In one or more embodiments, the user input is selected from the group consisting of visual, auditory, and sensory input. In one or more embodiments, the method further includes determining whether the user input is valid before sending the request to the cloud server. In one or more embodiments, the data is received from a plurality of knowledge bases connected to the cloud server.

[0132] In one or more embodiments, the request instructs the cloud server to retrieve data from a knowledge base. In one or more embodiments, the virtual content includes a relaxing environment. In one or more embodiments, the method further includes receiving additional data from the second user and displaying the additional virtual content to the first user based on the received additional data.

[0133] In one or more embodiments, the first and second users are located at different physical locations. In one or more embodiments, the additional virtual content is a visual representation of the second user. In one or more embodiments, the method further includes receiving first user data from the first user, receiving second user data from the second user, modifying the virtual content based on the received first and second user data, and displaying the modified virtual content to the first and second users.

[0134] In one or more embodiments, the first user engages in a physical activity for medical rehabilitation, and the virtual content relates to the physical activity. In one or more embodiments, the virtual content is selected from the group consisting of information about the physical activity, performance statistics of the first user, a virtual experience corresponding to the physical activity, and a virtual avatar. In one or more embodiments, the data is retrieved from a networked memory.

[0135] In another aspect, a method for improving task performance includes retrieving task data related to a task to be performed by a user, generating virtual content based on the task data, creating a virtual user interface within the user's field of view, retrieving a set of map points corresponding to the user's location, and displaying the virtual content on the virtual user interface while the user is performing the task such that the virtual user interface appears to be fixed at the set of map points when viewed by the user.

[0136] In one or more embodiments, the virtual content is a game having a virtual mapped pattern, the game configured to improve a user's performance of a task. In one or more embodiments, the game awards points for following the virtual mapped pattern. In one or more embodiments, the game doubles points for reaching a point in the virtual mapped pattern within a predetermined time.

[0137] In one or more embodiments, the game deducts points for straying from a virtual mapped pattern. In one or more embodiments, the game deducts points for moving adjacent to a physical object. In one or more embodiments, the task is selected from the group consisting of operating landscaping equipment, retrieving inventory items, stocking retail shelves, and sorting mail. In one or more embodiments, the task data is retrieved from a networked memory.

[0138] In one or more embodiments, the method further includes receiving user input, generating additional virtual content based on the user input, and displaying the additional virtual content on the virtual user interface while the user is performing the task. In one or more embodiments, the user input includes a user action related to performance of the task.

[0139] In yet another aspect, an augmented reality display system includes a display device configurable to display one or more virtual images to a user's eyes, and a processor communicatively coupled to the display device configured to provide the one or more virtual images to the display device, the processor including a prediction mechanism for predictively correcting timing discrepancies related to the display of the virtual images.

[0140] In one or more embodiments, the timing mismatch involves one or more sensor measurements made by one or more sensors, the one or more sensors being communicatively coupled to the processor, hi one or more embodiments, the timing mismatch involves processing delays when processing incoming data.

[0141] In one or more embodiments, the prediction mechanism utilizes a filter to compensate for the effects of timing mismatch. In one or more embodiments, the filter takes into account the relative speed of one or more sensor measurements, the sensor measurements being made by one or more sensors communicatively coupled to the processor. In one or more embodiments, the prediction mechanism utilizes a Kalman predictor.

[0142] In one or more embodiments, the Kalman predictor is utilized during the display processing stage. In one or more embodiments, the processor makes compensating changes to data associated with one or more virtual images to compensate for the timing mismatch. In one or more embodiments, the compensating changes include shifting the data associated with the virtual images.

[0143] In one or more embodiments, the compensation modification includes smoothing one or more visual artifacts associated with the virtual image. In one or more embodiments, the compensation modification includes correcting for negative effects of sensor measurements of one or more sensors, the one or more sensors being communicatively coupled to the processor.

[0144] In another aspect, a method for correcting delay in an augmented reality display system includes determining one or more timing mismatches in at least one stage of displaying a virtual image to a user, utilizing a prediction mechanism to predictively correct the timing mismatches, and compensating for at least one characteristic associated with data of the virtual image based at least in part on the prediction mechanism.

[0145] In one or more embodiments, the timing mismatch involves one or more sensor measurements made by one or more sensors, the one or more sensors being communicatively coupled to the processor, hi one or more embodiments, the timing mismatch involves processing delays when processing incoming data.

[0146] In one or more embodiments, the prediction mechanism utilizes a filter to compensate for the effects of timing mismatches. In one or more embodiments, the filter takes into account the relative speed of one or more sensor measurements, the sensor measurements being made by one or more sensors communicatively coupled to the processor.

[0147] In one or more embodiments, the prediction mechanism utilizes a Kalman predictor. In one or more embodiments, the Kalman predictor is utilized during a display processing stage. In one or more embodiments, the compensation includes shifting data associated with the virtual image. In one or more embodiments, the compensation includes smoothing one or more visual artifacts associated with the virtual image. In one or more embodiments, the compensation includes correcting for negative effects of sensor measurements of one or more sensors, the one or more sensors being communicatively coupled to the processor.

[0148] In another aspect, a method for calibrating an augmented reality system includes the steps of displaying a virtual image to a user, the virtual image configured to be displayed at a known focal length, the virtual image comprising pixel points; determining locations at which the pixel points will be displayed to the user, the pixel locations calculated based at least in part on locations of the pupils of the user's eyes; and aligning the pixel points of the virtual image with known points in space.

[0149] In one or more embodiments, the steps are repeated for multiple pixel points. In one or more embodiments, the locations at which the pixel points are displayed to the user are calculated based at least in part on the locations of known points in space. In one or more embodiments, the locations of the pupil, the locations of the pixel points, and the locations of the known points in space are collinear.

[0150] In one or more embodiments, the location where the pixel points are displayed to the user is modified based on user input. In one or more embodiments, the method further includes creating a gaming interface such that the plurality of pixel points are presented to the user. In one or more embodiments, the gaming interface includes firing a laser through the user's eye movements.

[0151] In one or more embodiments, the location of the known point is determined based at least in part on data received from one or more world cameras. In one or more embodiments, the steps are repeated for another eye of the user. In one or more embodiments, the location of the pixel point is determined based at least in part on a function of the location of the pupil and the location of the known point. In one or more embodiments, the function comprises a quadratic function.

[0152] In another aspect, an augmented reality display system includes a display device for displaying a virtual image to a user, the virtual image configured to be displayed at a known focal length, the virtual image comprising pixel points; and a processor communicatively coupled to the display device and configured to determine where the pixel points will be displayed to the user, calculate the locations of the pixel points based at least in part on the locations of the pupils of the user's eyes, and align the pixel points of the virtual image with the known points in space.

[0153] In one or more embodiments, the steps are repeated for multiple pixel points. In one or more embodiments, the location at which the pixel point is displayed to the user is calculated based at least in part on the location of the known point in space. In one or more embodiments, the location of the pupil, the location of the pixel point, and the location of the known point in space are collinear. In one or more embodiments, the location at which the pixel point is displayed to the user is modified based on user input.

[0154] In one or more embodiments, the system further includes a gaming interface configured to present the plurality of pixel points to a user. In one or more embodiments, the gaming interface includes emitting a laser through eye movements of the user. In one or more embodiments, the locations of the known points are determined based at least in part on data received from one or more world cameras.

[0155] In one or more embodiments, the steps are repeated for the user's other eye. In one or more embodiments, the location of the pixel point is determined based at least in part on a function of the location of the pupil and the location of the known point. In one or more embodiments, the function comprises a quadratic function.

[0156] In another aspect, a method for displaying a virtual interface includes identifying a user interface to be displayed to a user based on user input; displaying an avatar user interface in relation to at least one physical object, the avatar user interface presenting at least one virtual user interface element; and selecting the at least one virtual user interface element based at least in part on an interaction between the user and the avatar user interface.

[0157] In one or more embodiments, the user input comprises a gesture. In one or more embodiments, the method further comprises determining a location of the gesture relative to the user's augmented reality display system, and the virtual user interface is displayed at the determined location. In one or more embodiments, the avatar is a pre-selected avatar. In one or more embodiments, the avatar resembles the user.

[0158] In one or more embodiments, the avatar is displayed as if standing on the physical object. In one or more embodiments, the at least one virtual element comprises an application. In one or more embodiments, the method further includes selecting at least one virtual element based at least in part on another user input, the at least one virtual element comprising a user agent, and displaying another avatar representing the user agent in relation to the avatar virtual user interface and the physical object, wherein interaction between the user and the user agent is animated through virtual interaction between the avatar virtual user interface and the other avatar representing the user agent.

[0159] In one or more embodiments, the interaction includes transmitting data between the user's augmented reality system and a computing system corresponding to the user's intermediary. In one or more embodiments, the user input comprises a voice command.

[0160] In another aspect, a method for displaying a virtual interface includes identifying a user interface to be displayed to a user based on user input; and displaying a floating user interface in relation to at least one physical object, the floating user interface comprising one or more selectable virtual interface elements, the floating user interface appearing to be pushed out from a location where the user input was received.

[0161] In one or more embodiments, the user input comprises a user finger gesture. In one or more embodiments, the user finger gesture comprises touching a finger of one hand of the user with another finger of the other hand of the user. In one or more embodiments, the method further comprises determining a location where the finger of one hand of the user is touched with another finger of the other hand of the user, and the floating virtual user interface originates from the determined location.

[0162] In one or more embodiments, the floating user interface comprises a collection of three-dimensional blocks, a three-dimensional block of the collection of three-dimensional blocks representing at least one selectable virtual user interface element, hi one or more embodiments, the floating virtual user interface is created at least in part based on a world reference frame.

[0163] In one or more embodiments, the at least one selectable virtual user interface element exists as a stack of three-dimensional blocks, and the stack of three-dimensional blocks is rotated based at least in part on another user input. In one or more embodiments, the method further includes selecting a virtual user interface element of the at least one selectable virtual interface element based at least in part on another user input received from the user, the virtual interface element being associated with at least one other sub-virtual interface element, and displaying the at least one other sub-virtual interface element below the selected virtual interface element.

[0164] In one or more embodiments, the method further includes selecting a virtual user interface element of the at least one selectable virtual interface element based at least in part on another user input received from the user, and displaying content associated with the selected virtual interface element in a virtual box, the virtual box comprising the content displayed within the user's field of view.

[0165] In one or more embodiments, the method further includes identifying a second user input indicating that the virtual box should be closed, animating the virtual box in a manner resembling crumpling paper based at least in part on the second user input, and terminating content associated with the selected virtual interface element. In one or more embodiments, the second user input includes a gesture, the gesture resembling crumpling a piece of paper.

[0166] In yet another aspect, a method for displaying a virtual user interface includes identifying a user interface to be displayed to a user based on user input; and displaying a floating user interface in relation to at least one physical object, the floating user interface comprising one or more selectable virtual interface elements, wherein the floating user interface appears to be placed on the at least one physical object, and wherein rotation of the at least one physical object about a longitudinal axis of the at least one physical object causes additional selectable virtual interface elements to be displayed, the additional selectable virtual interface elements appearing to be resting on another side of the at least one physical object.

[0167] In one or more embodiments, the at least one physical object comprises a user's arm. In one or more embodiments, the user input comprises a gesture. In one or more embodiments, the gesture comprises a cupping of the user's hand over the user's arm, which causes a selectable virtual interface element to appear to be displayed.

[0168] In one or more embodiments, the method further includes determining a location on the user's arm of the cupped user's hand, and the floating user interface originates from the determined location. In one or more embodiments, the method further includes terminating the rendering of the floating virtual user interface based at least in part on another user input.

[0169] In one or more embodiments, the other user input includes a gesture, the gesture including dragging the user's cupped hand across a physical object. In one or more embodiments, the floating virtual user interface is created based at least in part on a hand-centered frame of reference. In one or more embodiments, the method further includes selecting a virtual user interface element of the at least one selectable virtual interface element based at least in part on another user input received from the user, and displaying content associated with the selected virtual interface element, the content being displayed in relation to the floating virtual interface.

[0170] In one or more embodiments, the displayed content corresponds to a hand-centered frame of reference. In one or more embodiments, the method further includes, based at least in part on input received from the user, moving the displayed content from the hand-centered frame of reference to a world-centered frame of reference when the floating virtual user interface is moving, such that the displayed content remains stationary.

[0171] In another aspect, a method for creating a virtual user interface includes identifying a user interface to be displayed to a user based on user input; and displaying a virtual user interface in relation to at least one physical object and arising from a location where the at least one physical object is touched, the virtual user interface appearing to protrude from a location where the at least one physical object is touched, the virtual user interface comprising at least one selectable virtual user interface element that appears to be attached to the virtual user interface.

[0172] In one or more embodiments, the at least one physical object comprises a horizontal surface. In one or more embodiments, the user input comprises a gesture. In one or more embodiments, the gesture comprises touching the at least one physical object with an outstretched finger for a predetermined period of time. In one or more embodiments, the method further comprises determining a location where the user's finger touched the at least one physical object, and the virtual user interface originates from the determined location.

[0173] In one or more embodiments, the method further includes terminating the rendering of the floating virtual user interface based at least in part on another user input. In one or more embodiments, the other user input includes a gesture, the gesture including a cutting movement with the user's hand across the rendered virtual user interface. In one or more embodiments, the virtual user interface is created at least in part based on a world-centered frame of reference such that the virtual user interface remains stationary with respect to any movements of the user.

[0174] In one or more embodiments, the method further includes selecting a virtual user interface element of the at least one selectable virtual interface element based at least in part on another user input received from the user, and generating a second virtual user interface at another location on the at least one physical object, the second virtual user interface comprising the additional selectable virtual interface element. In one or more embodiments, the virtual user interface resembles a tree growing from the location of the touch on the at least one physical object.

[0175] In another aspect, a method for displaying a virtual user interface includes identifying a user interface to be displayed to a user based on user input, rendering the virtual user interface to mirror the user's finger movements such that the virtual user interface mirrors the finger movements, and displaying one or more selectable user interface elements within the area in which the virtual user interface is rendered. In one or more embodiments, the finger gesture includes a pointing gesture using at least one finger of the user's hand for a predetermined period of time.

[0176] In one or more embodiments, the method further includes determining a location of the pointing gesture, and the virtual user interface originating from the determined location. In one or more embodiments, the method further includes detecting that the user is no longer drawing the virtual user interface, and the virtual user interface element is displayed based at least in part on the detection. In one or more embodiments, the drawing of the virtual user interface mirrors the dragging movement of the user's finger across space. In one or more embodiments, the method further includes terminating rendering of the floating virtual user interface based at least in part on another user input.

[0177] In one or more embodiments, the other user input includes a gesture, the gesture including the end of a continuous dragging motion of the user's finger. In one or more embodiments, the virtual user interface is created based at least in part on a world-centered frame of reference such that the virtual user interface remains stationary with respect to any movements of the user.

[0178] In one or more embodiments, the method further includes selecting a virtual user interface element of the at least one selectable virtual interface element based at least in part on another user input received from the user, and generating content associated with the selected virtual interface element related to the virtual user interface. In one or more embodiments, the shape of the virtual user interface resembles a shape traced by a user's finger.

[0179] In another aspect, a method for creating a virtual user interface includes identifying a user interface to be displayed to a user based on a hand gesture of the user, and displaying the virtual user interface having at least one selectable virtual interface element in response to a hand gesture on a vertical side of the physical object in a direction of the hand gesture such that the virtual user interface appears to be on the vertical side of the physical object.

[0180] In one or more embodiments, the method further includes displaying, in response to a hand gesture, a collection of virtual content resembling paint specks, the virtual content occurring at a point indicated by the hand gesture. In one or more embodiments, the hand gesture includes an outstretched finger raised toward a vertical side of the physical object. In one or more embodiments, the method further includes determining a location of the outstretched finger raised toward the vertical side of the physical object, the collection of virtual content resembling paint specks being displayed at the determined location.

[0181] In one or more embodiments, the method further includes displaying a virtual speck on a vertical side of the physical object, at least one characteristic of the virtual speck being displayed on the vertical side of the physical object based at least in part on a determined location of the outstretched finger raised in the direction of the vertical side of the physical object. In one or more embodiments, the physical object comprises a wall.

[0182] In one or more embodiments, the virtual user interface is created based at least in part on a world-centered frame of reference such that the virtual user interface remains stationary relative to any movement of the user. In one or more embodiments, the method further includes selecting a virtual user interface element of the at least one selectable virtual interface element based at least in part on another user input received from the user, and generating content associated with the selected user interface element at another location on the physical object.

[0183] In one or more embodiments, the method further includes storing content associated with the selected user interface element in a reduced form, the reduced form comprising a virtual band, the virtual band being displayed around the user's hand. In one or more embodiments, the virtual band is created at least in part based on a hand-centered frame of reference such that the virtual band moves at least in part based on movement of the hand.

[0184] In one or more embodiments, the method further includes displaying the content in its complete form based at least in part on another gesture of the user, in one or more embodiments, the other gesture includes a hand raising movement in which the virtual band is displayed.

[0185] In another aspect, a method for creating a virtual user interface includes identifying a user interface to be displayed to a user based on a hand gesture of the user; and displaying the virtual user interface in response to the hand gesture, the virtual user interface resembling a horizontal thread having at least one selectable virtual interface element such that at least one selectable virtual interface element moves to display an additional selectable virtual interface element based, at least in part, on another hand gesture of the user.

[0186] In one or more embodiments, the hand gesture includes touching a first finger of a first hand of a user with a second finger of a second hand. In one or more embodiments, the gesture further includes moving both the first finger and the second finger apart such that the first finger is moved away from the second finger. In one or more embodiments, the virtual thread mirrors the length of the movement of the first and second fingers, and the length of the virtual thread is based at least in part on the movement of the first and second fingers.

[0187] In one or more embodiments, the method further includes determining a touch movement of the first and second fingers, the virtual thread originating from the determined location. In one or more embodiments, the gesture of the other hand includes a turning movement of the user's hand, the turning movement moving at least one selectable virtual interface element in a direction of the turning movement.

[0188] In one or more embodiments, the virtual user interface is created based at least in part on a world-centered frame of reference such that the virtual user interface remains stationary with respect to any movements of the user. In one or more embodiments, the method further includes selecting a virtual user interface element of the at least one selectable virtual interface element based at least in part on a gesture of another hand of the user, and generating content associated with the selected user interface element at another location on the physical object. In one or more embodiments, the gesture of the other hand includes a movement of bringing one of the user's hands forward.

[0189] In another aspect, a method for creating a virtual user interface includes identifying a user interface to be displayed to a user based on a hand gesture of the user; and displaying the virtual user interface in response to the hand gesture, the virtual user interface comprising a set of vertical threads, each of the vertical threads being associated with at least one selectable virtual interface element.

[0190] In one or more embodiments, the hand gesture includes a gesture of spreading a first hand of the user for a predetermined period of time. In one or more embodiments, the method further includes selecting at least one selectable virtual interface element based at least in part on a gesture of another hand, the gesture of the other hand including a movement of two fingers of the user pulling at least one of the virtual threads of the set of virtual threads.

[0191] In one or more embodiments, the method further includes transmitting data associated with a first virtual element associated with the first virtual yarn to a second virtual element associated with the second virtual yarn based at least in part on the hand gesture. In one or more embodiments, the other hand gesture includes a squeezing movement of a first virtual element associated with the first virtual yarn with a second virtual element associated with the second virtual yarn, the squeezing movement being performed by a finger of the user.

[0192] In one or more embodiments, the virtual interface is displayed relative to at least one physical object. In one or more embodiments, the at least one physical object comprises a wall. In one or more embodiments, the method further includes determining a location of the hand gesture, the virtual thread originating from the determined location.

[0193] In one or more embodiments, the virtual user interface is created based at least in part on a world-centered frame of reference such that the virtual user interface remains stationary with respect to any movements of the user. In one or more embodiments, the method further includes selecting a virtual user interface element of the at least one selectable virtual interface element based at least in part on another hand gesture of the user, and generating virtual content associated with the selected user interface element at another location on the physical object.

[0194] In another aspect, a method for creating a virtual user interface includes identifying a user interface to be displayed to a user based on a hand gesture of the user, and displaying the virtual user interface in response to the hand gesture, the virtual user interface comprising a set of vertical threads, each of the vertical threads being associated with at least one selectable virtual interface element. In one or more embodiments, the hand gesture includes spreading a first hand of the user for a predetermined period of time.

[0195] In one or more embodiments, the method further includes selecting at least one selectable virtual interface element based at least in part on a gesture of another hand, the other hand gesture including a movement of two fingers of the user pulling at least one of the virtual threads of the collection of virtual threads. In one or more embodiments, the method further includes transmitting data associated with a first virtual element associated with the first virtual thread to a second virtual element associated with the second virtual thread based at least in part on the gesture of the hand.

[0196] In one or more embodiments, the other hand gesture includes a squeezing movement of a first virtual element associated with the first virtual thread with a second virtual element associated with the second virtual thread, the squeezing movement being performed by a finger of the user. In one or more embodiments, the virtual interface is displayed in relation to at least one physical object. In one or more embodiments, the at least one physical object comprises a wall.

[0197] In one or more embodiments, the method further includes determining a location of the hand gesture, and the virtual thread originates from the determined location. In one or more embodiments, the virtual user interface is created based at least in part on a world-centered frame of reference such that the virtual user interface remains stationary with respect to any movements of the user.

[0198] In one or more embodiments, the method further includes selecting a virtual user interface element of the at least one selectable virtual interface element based at least in part on another hand gesture of the user, and generating virtual content associated with the selected user interface element at another location on the physical object.

[0199] In yet another aspect, a method for creating a virtual user interface includes identifying a user interface to be displayed to a user based on a hand gesture of the user; and displaying the virtual user interface in response to the hand gesture, the virtual user interface resembling a virtual spider web, and the user pulling virtual threads of the virtual spider web to move the virtual interface closer to the user.

[0200] In one or more embodiments, the hand gesture comprises a pulling motion with a clenched fist of the user's hand. In one or more embodiments, the virtual threads of the virtual spider web comprise at least one selectable virtual element. In one or more embodiments, the method further comprises selecting at least one selectable virtual interface element based at least in part on a gesture of another hand, the gesture of the other hand comprising a pulling motion of at least one of the virtual threads toward the user.

[0201] In one or more embodiments, the method further includes transmitting data associated with a first virtual element associated with the first virtual yarn to a second virtual element associated with the second virtual yarn based at least in part on the hand gesture. In one or more embodiments, the other hand gesture includes a squeezing movement of a first virtual element associated with the first virtual yarn with a second virtual element associated with the second virtual yarn, the squeezing movement being performed by a finger of the user.

[0202] In one or more embodiments, the virtual interface is displayed in relation to at least one physical object. In one or more embodiments, the at least one physical object comprises a wall. In one or more embodiments, the method further includes determining a location of the hand gesture, and the virtual spider web emerges from the determined location.

[0203] In one or more embodiments, the virtual user interface is created based at least in part on a world-centered frame of reference such that the virtual user interface remains stationary with respect to any movements of the user. In one or more embodiments, the method further includes selecting a virtual user interface element of the at least one selectable virtual interface element based at least in part on another hand gesture of the user, and generating virtual content associated with the selected user interface element at another location on the physical object.

[0204] Additional and other objects, features, and advantages of the present invention are set forth in the detailed description, drawings, and claims. The present specification also provides, for example, the following items: (Item 1) 1. A method for generating map data, comprising: identifying a set of map points associated with one or more images; determining a set of sparse points and a set of dense points from the identified map points; normalizing said respective sets of sparse and dense points; A method comprising: (Item 2) generating sparse and dense point descriptors for the respective sets of sparse and dense points; combining the low-density point descriptors and the high-density point descriptors and storing them as map data; Item 1, the method of claim 1 further comprising: (Item 3) Item 10. The method of item 1, wherein the set of sparse points corresponds to a unique feature. (Item 4) Item 4. The method of item 3, wherein the unique feature is selected from the group consisting of a corner, a circle, a triangle, and text. (Item 5) Item 2. The method of item 1, wherein the set of high-density points corresponds to 3D points in the field of view. (Item 6) 6. The method of claim 5, wherein the set of high-density points also includes color values. (Item 7) Item 10. The method of item 1, wherein the normalization comprises scale normalization. (Item 8) Item 10. The method of item 1, wherein the normalization comprises coordinate normalization relative to a common origin. (Item 9) Item 10. The method of item 1, wherein the normalization is implemented using machine learning. (Item 10) Item 2. The method of item 1, wherein the sparse and dense point descriptors correspond to each sparse and dense point of the respective sets of sparse and dense points. (Item 11) Item 11. The method of item 10, wherein each sparse and dense point descriptor includes information about the respective sparse and dense point selected from the group consisting of scale, orientation, patch data, and texture. (Item 12) 1. A method for determining a user input, comprising: capturing an image of a user's field of view, the image including a gesture made by the user; analyzing the captured image to identify a set of points associated with the gesture; comparing the identified set of points to sets of points associated with a database of predetermined gestures; determining a user input based on the recognized gesture; A method comprising: (Item 13) Item 13. The method of item 12, further comprising generating a scoring value for the set of identified points based on the comparison. (Item 14) Item 14. The method of item 13, further comprising recognizing the gesture if the scored value exceeds a threshold. (Item 15) Item 13. The method of item 12, further comprising storing the predetermined gesture in a database. (Item 16) Item 16. The method of item 15, further comprising accessing a networked memory and accessing the database of predetermined gestures. (Item 17) Item 13. The method of item 12, wherein the gesture is a hand gesture. (Item 18) Item 13. The method of item 12, wherein the gesture is a finger gesture. (Item 19) Item 13. The method of item 12, wherein the gesture is a finger-to-finger interaction. (Item 20) Item 13. The method of item 12, wherein the gesture is selected from the group consisting of finger-to-finger interaction, pointing, tapping, and rubbing. (Item 21) Item 13. The method of item 12, further comprising displaying one or more virtual objects to the user based at least in part on the determined user input. (Item 22) 22. The method of claim 21, wherein the one or more virtual objects comprise a virtual user interface. (Item 23) 1. A method comprising: Capturing one or more images corresponding to a user's field of view; analyzing the captured one or more images to identify one or more gestures made by the user, the analysis including utilizing a cascading mechanism having multiple stages; A method comprising: (Item 24) 24. The method according to claim 23, wherein the cascade mechanism comprises a plurality of nodes, each node corresponding to a stage among the plurality of stages. (Item 25) Item 24. The method of item 23, wherein the cascade mechanism comprises a series of permissive analysis nodes. (Item 26) 24. The method of claim 23, wherein earlier stages of the plurality of stages in the cascade arrangement consume less processing power than later stages of the plurality of stages in the cascade arrangement. (Item 27) 27. The method of claim 26, wherein non-gestures are removed based on an analysis of images captured in the previous stage of the cascade mechanism. (Item 28) 27. The method of claim 26, wherein the later stages of the plurality of stages determine more complex gestures based at least in part on the captured images. (Item 29) 27. The method of claim 26, wherein analyzing the captured image includes determining whether the sharpness of contours of various shapes in the captured image is sufficiently sharp to constitute a gesture. (Item 30) 27. The method of claim 26, wherein the later stages of the cascade mechanism are utilized to distinguish between different gestures. (Item 31) 24. The method of claim 23, further comprising generating a score based at least in part on said analysis. (Item 32) 32. The method of claim 31, further comprising removing a candidate image from consideration if the generated score is below a minimum threshold. (Item 33) 32. The method of claim 31, further comprising proceeding to the subsequent stage of the cascade mechanism if the generated score is above a minimum threshold. (Item 34) 1. A method comprising: Capturing a plurality of images of each field of view of a user; analyzing the plurality of images to generate a plurality of gesture candidates; generating an analysis value corresponding to the plurality of analysis values, wherein a gesture is recognized based at least in part on the analysis value; A method comprising: (Item 35) 35. The method of claim 34, further comprising sorting the gesture candidates based at least in part on the respective analysis values. (Item 36) Item 35. The method of item 34, further comprising eliminating gesture candidates having an analysis value below a minimum threshold. (Item 37) 35. The method of claim 34, further comprising advancing the gesture candidate to a next stage of processing if the analysis value is above a minimum threshold. (Item 38) 1. A method of classifying gestures, comprising: Capturing an image of a user's field of view; performing depth segmentation on the captured image to generate a depth map; identifying a gesture based at least in part on the generated depth map; and A method comprising: (Item 39) Item 39. The method of item 38, further comprising using a classification mechanism to analyze the depth map and identify portions of a hand that correspond to points in the depth map. (Item 40) 40. The method of claim 39, further comprising: skeletonizing the depth map based on the identification of the hand portion. (Item 41) Item 41. The method of item 40, further comprising classifying the image as a gesture based on the skeletonized depth map. (Item 42) Item 39. The method of item 38, wherein the depth partitioning comprises a line search. (Item 43) 39. The method of claim 38, further comprising performing a cascade analysis on the depth map to classify the image as the gesture. (Item 44) Item 39. The method of item 38, further comprising performing depth dilation on the depth map. (Item 45) Item 39. The method of item 38, further comprising performing surface normalization on the depth map. (Item 46) Item 39. The method of item 38, further comprising performing orientation normalization on the depth map. (Item 47) Item 39. The method of item 38, further comprising performing background subtraction on the depth map. (Item 48) Item 39. The method of item 38, further comprising performing a depth comparison on the depth map. (Item 49) 39. The method of claim 38, further comprising classifying the image as a gesture based on the skeletonized depth map and prior information. (Item 50) 39. The method of claim 38, wherein the classification mechanism is a decision forest. (Item 51) 1. A method for determining a user input, comprising: capturing one or more images of a user's field of view, the images comprising at least a predetermined physical object; analyzing the image to detect movement of the user relative to the predetermined physical object; determining a user input based at least in part on an analysis of the movement relative to the given physical object; and A method comprising: (Item 52) Item 52. The method of item 51, further comprising recognizing the predetermined physical object. (Item 53) 53. The method of claim 52, wherein the predetermined physical object is recognized based at least in part on an image comparison between a captured image of the predetermined physical object and a database of predetermined physical objects. (Item 54) Item 52. The method of item 51, wherein the movements of the user relative to the predetermined physical object are used to generate a pattern. (Item 55) Item 55. The method of item 54, further comprising comparing the generated pattern to a database of predetermined patterns. (Item 56) 52. The method of claim 51, further comprising generating a scoring value for the recognized pattern based on the comparison. (Item 57) 57. The method of claim 56, further comprising determining the user input if the scored value exceeds a threshold. (Item 58) 52. The method of claim 51, further comprising visually tracking the movement relative to the predetermined physical object and generating a video recording. (Item 59) 59. The method of claim 58, wherein the video recording is analyzed to determine the user input. (Item 60) Item 56. The method of item 55, further comprising accessing the database of predetermined patterns through a networked memory. (Item 61) Item 52. The method of item 51, wherein the predetermined physical object is selected from the group consisting of an existing structure within the field of view, an actively marked totem, a passively marked totem, an object integrated into the camera / sensor, and a totem controller object. (Item 62) Item 52. The method of item 51, wherein the movement relative to the given physical object is selected from the group consisting of a position, an orientation, and a movement of the given physical object relative to a reference frame. (Item 63) Item 52. The method of item 51, wherein the predetermined physical object comprises a first one of the user's hands, and the movement involving the first hand comprises manipulating the first hand with a second one of the user's hands. (Item 64) Item 52. The method of item 51, wherein the predetermined physics object has a soft hardness surface, and movement relating to the predetermined physics object comprises the user pressing down on the soft hardness surface. (Item 65) Item 52. The method of item 51, further comprising rendering a virtual interface element associated with the given physical object for the user. (Item 66) Item 66. The method of item 65, wherein the virtual interface element is displayed relative to the predetermined physical object such that, when viewed by the user, the virtual interface element is modified based at least in part on modifications related to the predetermined physical object. (Item 67) 52. The method of claim 51, wherein the predetermined physical object comprises an electronic input device, and user input is determined based on the recognized movement of the predetermined physical object and is entered from the electronic input device. (Item 68) 52. The method of claim 51, further comprising modifying at least one characteristic of virtual content displayed to the user based at least in part on the determined user input. (Item 69) 1. A method for generating a virtual user interface, comprising: Identifying a virtual user interface to be displayed to the user based on the user input; obtaining a location associated with the user; determining a set of coordinate points at which to display the identified virtual user interface based at least in part on the obtained locations; displaying the virtual user interface to the user at the determined coordinate point; A method comprising: (Item 70) 70. The method of claim 69, wherein the user input is determined based at least in part on a recognized gesture. (Item 71) 70. The method of claim 69, wherein the user input is determined at least in part based on a voice command. (Item 72) 70. The method of claim 69, wherein the user input is determined at least in part based on an interaction with a predetermined physical object. (Item 73) Item 70. The method of item 69, further comprising retrieving the identified user virtual user interface from a library of user interfaces. (Item 74) Item 70. The method of item 69, wherein the identified virtual user interface is associated with a reference frame. (Item 75) Item 75. The method of item 74, wherein the reference frame is a body-centered reference frame. (Item 76) Item 75. The method of item 74, wherein the reference frame is a head-centered reference frame. (Item 77) Item 75. The method of item 74, wherein the reference frame is a hand-centered reference frame. (Item 78) Item 75. The method of item 74, wherein the reference frame is a world-centered reference frame. (Item 79) Item 70. The method of item 69, further comprising: performing a transformation between a reference frame associated with the identified virtual user interface and an obtained location of the user relative to the world to determine a set of coordinate points of the virtual user interface. (Item 80) 70. The method of claim 69, wherein the user's location in the world is determined based on the user's GPS location. (Item 81) Item 70. The method of item 69, wherein the user's location within the world is determined based on a set of map points associated with the user. (Item 82) Item 70. The method of item 69, wherein the virtual user interface appears stationary when the user moves. (Item 83) Item 70. The method of item 69, wherein the virtual user interface moves relative to the user's movements. (Item 84) determining a frame of reference associated with the identified virtual user interface; determining a location of the reference frame relative to the world reference frame; setting the determined location as an origin; determining the set of coordinate points relative to the origin; Item 70. The method of item 69, further comprising: (Item 85) 70. The method of claim 69, further comprising reading the set of map points from a networked memory. (Item 86) 70. The method of claim 69, wherein the user input comprises a location in space where the virtual user interface is to be displayed. (Item 87) 87. The method of claim 86, wherein the location in space is associated with a physical entity at the user's location. (Item 88) 70. The method of claim 69, wherein the user input includes a gesture including a throwing input indicating a wall. (Item 89) Item 70. The method of item 69, wherein the user input includes a gesture signifying the end of an instruction to generate the virtual user interface. (Item 90) 1. A method for generating a virtual user interface, comprising: Detecting a manipulation of a predetermined physical object; Recognizing a command for creating a virtual user interface based on the detected manipulation; determining a set of map points associated with the locations of the predetermined physical objects from a virtual world model; rendering the virtual user interface in real time at the determined map point associated with the location of the totem so that when the virtual user interface is viewed by the user, it appears to be stationary at the location of the predetermined physical object; A method comprising: (Item 91) Item 91. The method according to item 90, wherein the manipulation of the predetermined physical object includes a gesture of expanding a pinched state by the user's hand on the surface of the predetermined physical object. (Item 92) Item 92. The method of item 91, wherein the virtual user interface, when viewed by the user, appears to cover a portion of the surface of the predetermined physical object, the portion corresponding to the location of the user's hands during the formation of the gesture to unpinch the pinch. (Item 93) Item 91. The method according to item 90, wherein the predetermined physical object is the user's hand. (Item 94) Item 94. The method of item 93, wherein the manipulation of the predetermined physical object comprises an action selected from the group consisting of the user spreading their hand, the user showing an open palm, and the user raising their hand. (Item 95) Item 95. The method of item 94, wherein the virtual user interface, when viewed by the user, appears to cover a portion of the surface of the hand. (Item 96) Item 96. The method of item 95, wherein the virtual user interface comprises a plurality of first level menu items selectable by the fingers or thumb of the hand. (Item 97) detecting a further manipulation of the hand; recognizing a command to create a second virtual user interface based on the detected further manipulation; rendering the second virtual user interface in real time at the determined map point associated with the location of the predetermined physical object such that, when viewed by the user, the virtual user interface appears to be stationary at the location of the predetermined physical object; Item 97. The method of item 96, further comprising: (Item 98) Item 98. The method of item 97, wherein the further manipulation of the hand includes spreading the fingers of the hand apart. (Item 99) Item 99. The method of item 98, wherein the second virtual user interface comprises a plurality of second-level menu items selectable by the fingers or thumb of the hand, the second level being lower than the first level. (Item 100) Item 98. The method of item 97, wherein the further manipulation of the totem includes using fingers from a second one of the user's hands to perform a motion that traces a circle on the palm of the hand. (Item 101) Item 101. The method of item 100, wherein the second virtual user interface comprises a plurality of menu items arranged in an arc, the menu items being scrollable and selectable by the fingers of the second hand. (Item 102) 1. A method for updating a virtual world, comprising: receiving a first input from a first device of a first user, the first input corresponding to a physical environment of the first user; updating a virtual world model based on the received first input, wherein the virtual world model corresponds to a physical environment of the first user; and transmitting first updated information corresponding to a first portion of the virtual world model to a second user; Including, The method, wherein the first updated information indicates whether any portion of the first updated information needs to be displayed to the second user. (Item 103) Item 103. The method of item 102, wherein the virtual world model resides on a networked memory. (Item 104) Item 103. The method of item 102, wherein the first user and the second user are located at different locations. (Item 105) Item 103. The method of item 102, wherein the first device of the first user is selected from the group consisting of an FOV camera, another camera, a sensor, a first device for eye tracking, and a first device for audio. (Item 106) Item 103. The method of item 102, further comprising transmitting the first updated information corresponding to a first portion of the virtual world model to the first user, the first updated information indicating whether any portion of the first updated information needs to be displayed to the first user. (Item 107) Item 103. The method of item 102, further comprising transmitting the first updated information corresponding to a first portion of the virtual world model to a plurality of other users, the first updated information indicating whether any portion of the first updated information needs to be displayed to each of the plurality of other users. (Item 108) receiving a plurality of inputs from respective first devices of a plurality of other users, the plurality of inputs corresponding to a physical environment of the first user; updating the virtual world model based on the received inputs; transmitting additional updated information corresponding to each additional portion of the virtual world model to the second user; and further comprising Item 103. The method of item 102, wherein the additional updated information indicates whether any portion of the additional updated information needs to be displayed to the second user. (Item 109) Item 109. The method of item 108, further comprising transmitting the additional updated information corresponding to the portion of the virtual world model to the first user, the additional updated information indicating whether any portion of the additional updated information needs to be displayed to the first user. (Item 110) Item 109. The method of item 108, further comprising transmitting the additional updated information corresponding to each additional portion of the virtual world model to the plurality of other users, the additional updated information indicating whether any portion of the additional updated information needs to be displayed to each of the plurality of other users. (Item 111) receiving a second input from a second device of the second user, the second input corresponding to a physical environment of the second user; updating the virtual world model based on the received second input, wherein the virtual world model corresponds to a physical environment of the first user; and transmitting second updated information corresponding to a second portion of the virtual world model to the first user; and further comprising Item 103. The method of item 102, wherein the second updated information indicates whether any portion of the second updated information needs to be displayed to the first user. (Item 112) Item 112. The method of item 111, wherein the second updated information corresponds to movement of the second user's avatar within the second portion of the virtual world model. (Item 113) 1. A method comprising: projecting a light pattern onto a space; detecting first and second portions of the light pattern with a camera to generate first and second data points corresponding to the first and second portions of the pattern; performing a triangulation analysis to determine locations of the first and second portions of the light pattern based at least in part on the first and second data points; A method comprising: (Item 114) Item 114. The method of item 113, further comprising repeating the steps of projecting, detecting, and triangulating multiple portions of the pattern to obtain additional texture data for the space. (Item 115) Item 114. The method of item 113, wherein projecting the light onto the space includes projecting the light onto the space using a fiber-based projector. (Item 116) Item 114. The method of item 113, further comprising modifying the light using an optical element to form a bundle of light rays before the light is projected onto the space. (Item 117) Item 114. The method of item 113, wherein the light pattern corresponds to structured light and the light pattern is dynamic. (Item 118) Item 114. The method of item 113, wherein the light pattern corresponds to patterned light, and the light pattern includes a plurality of points. (Item 119) Item 114. The method of item 113, wherein the light pattern corresponds to textured light, and the light pattern is irregular. (Item 120) Item 114. The method of item 113, wherein the first and second portions of the light pattern are respective pixels of the light pattern. (Item 130) A method for generating a virtual room, comprising: Identifying a virtual room to be displayed to a user; obtaining a first set of map points associated with the virtual room, the first set of map points corresponding to a first physical room at the user's location; tethering the virtual room to the first set of map points; displaying the virtual room to the user such that, when viewed by the user, the virtual room appears stationary at the first set of map points; A method comprising: (Item 131) Item 131. The method of item 130, further comprising identifying the virtual room based on user input. (Item 132) Item 132. The method of item 131, wherein the user input is selected from the group consisting of gestures, visual data, auditory data, sensory data, direct commands, voice commands, eye tracking, and physical button selections. (Item 133) Item 132. The method of item 131, wherein the virtual room is predetermined by the user. (Item 134) Item 132. The method of item 131, further comprising reading the first set of map points from a networked memory. (Item 135) obtaining a second set of map points associated with the virtual room, the second set of map points corresponding to a second physical room at the user's location after the user moves from the first physical room; tethering the virtual room to the second set of map points; displaying the virtual room to the user such that, when viewed by the user, the virtual room appears stationary at the second set of map points; Item 132. The method of item 131, further comprising: (Item 136) Item 136. The method of item 135, wherein the virtual room also includes first and second virtual objects, and the method further includes, when the virtual room is tethered to the second set of map points, maintaining a relative spatial position between the first virtual object and the second virtual object corresponding to the first set of map points. (Item 137) Item 132. The method of item 131, further comprising displaying the virtual room to the user such that when viewed by the user, the virtual room appears stationary relative to a portion of the first physical room. (Item 138) Item 132. The method of item 131, wherein the virtual room also includes selectable virtual decorations. (Item 139) 1. A method of creating a retail experience, comprising: Recognizing a location of a user within a retail store; retrieving data corresponding to the retail store; generating virtual content related to the retail store based on the retrieved data; creating a virtual user interface within the user's field of view; displaying the virtual content on the virtual user interface while the user engages in retail activities within the retail store; and A method comprising: (Item 140) the retrieved data comprises a set of map points corresponding to the retail store; Item 139. The method of item 139, wherein the virtual user interface, when viewed by the user, appears to be stationary at the set of map points. (Item 141) Item 139. The method of item 139, wherein the user's location is recognized using a radio frequency identification transponder and communications. (Item 142) Item 139. The method of item 139, wherein the virtual content is selected from the group consisting of virtual characters, virtual coupons, games based on the user's location, promotional item listings, nutritional information, metadata associated with items, celebrity appearances, cross-selling advertisements, information from people known to the user, and e-books. (Item 143) reading user data corresponding to the user; generating the virtual content based on both the retrieved data and the retrieved user data; Item 139. The method of item 139, further comprising: (Item 144) Item 144. The method of item 143, wherein the virtual content is selected from the group consisting of a virtual grocery list, a virtual coupon book, a virtual recipe book, a list of various ingredients in the user's home, and a virtual recipe builder. (Item 145) receiving user input; generating additional virtual content based on the user input; displaying the additional virtual content on the virtual user interface while the user engages in retail activities within the retail store; and Item 139. The method of item 139, further comprising: (Item 146) Item 146. The method of item 145, wherein the user input is selected from the group consisting of gestures, visual data, auditory data, sensory data, direct commands, voice commands, eye tracking, and physical button selections. (Item 147) Item 146. The method of item 145, wherein the virtual content is selected from the group consisting of a total operating cost, a smart virtual grocery list, an indicator of items in proximity to the user location, and a virtual payment system. (Item 148) Item 139. The method of item 139, further comprising transmitting the generated virtual content to a user device for display. (Item 149) 1. A method comprising: Retrieving patient data relating to the patient's medical history; generating virtual content based at least in part on the retrieved patient data; Retrieving a first set of map points corresponding to a first location of the first user; creating a first virtual user interface within a field of view of a first user based at least in part on the retrieved set of map points; displaying the virtual content on the first virtual user interface such that, when viewed by the first user, the first virtual user interface appears to be fixed at the first set of map points; A method comprising: (Item 150) Item 149. The method of item 149, wherein the virtual content is selected from the group consisting of a three-dimensional image of a surgical target, patient identification information, medical images, patient vital sign information, and medical records. (Item 151) Item 149. The method of item 149, wherein the patient data is read from a networked memory. (Item 152) Item 149. The method of item 149, further comprising generating a second virtual user interface to facilitate communication between the first user and a second user, the second user being in a second location different from the first location of the first user. (Item 153) Item 153. The method of item 152, wherein the second virtual user interface is a visual representation of the second user. (Item 154) Item 153. The method of item 152, wherein the second user is selected from the group consisting of a senior specialist surgeon, a patient, a person related to the patient, and a medical student. (Item 155) Item 153. The method of item 152, further comprising displaying the virtual content to the second user on the first virtual user interface so that, when viewed by the second user, the first virtual user interface appears fixed at the set of map points. (Item 156) The virtual content is displayed to the first user during the surgical procedure, and the method includes: receiving user input; generating additional virtual content based on the user input; displaying the additional virtual content on the first virtual user interface while the first user is performing the surgical procedure; Item 149. The method of item 149, further comprising: (Item 157) Item 157. The method of item 156, wherein the user input is selected from the group consisting of gestures, visual data, auditory data, sensory data, direct commands, voice commands, eye tracking, and physical button selections. (Item 158) Item 157. The method of item 156, wherein the user input comprises an image of the first user's field of view, and the method further includes displaying the additional virtual content to the second user on the first virtual user interface such that, when viewed by the second user, the first virtual user interface appears to be fixed at the set of map points. (Item 159) 1. A method for facilitating medical rehabilitation, comprising: receiving user input related to medical rehabilitation; transmitting a request related to medical rehabilitation to a cloud server based on the received user input; receiving, based on the request, data related to medical rehabilitation from a knowledge base connected to the cloud server; displaying virtual content related to medical rehabilitation to the first user based on the received data; and A method comprising: (Item 160) Item 159. The method of item 159, wherein the user input is selected from the group consisting of visual, auditory, and sensory input. (Item 161) Item 159. The method of item 159, further comprising determining whether the user input is valid before sending the request to the cloud server. (Item 162) Item 159. The method of item 159, wherein the data is received from a plurality of knowledge bases connected to the cloud server. (Item 163) Item 159. The method of claim 159, wherein the request instructs the cloud server to retrieve data from the knowledge base. (Item 164) Item 159. The method of item 159, wherein the virtual content includes a relaxing environment. (Item 165) receiving additional data from a second user; displaying additional virtual content to the first user based on the received additional data; and Item 159. The method of item 159, further comprising: (Item 166) Item 166. The method of item 165, wherein the first and second users are located in different physical locations. (Item 167) Item 166. The method of item 165, wherein the additional virtual content is a visual representation of the second user. (Item 168) receiving first user data from the first user; receiving second user data from a second user; modifying the virtual content based on the received first and second user data; displaying the modified virtual content to the first and second users; and Item 159. The method of item 159, further comprising: (Item 169) Item 159. The method of item 159, wherein the first user is engaged in a physical activity for medical rehabilitation and the virtual content is related to the physical activity. (Item 170) Item 169. The method of item 169, wherein the virtual content is selected from the group consisting of information about the physical activity, performance statistics of the first user, a virtual experience corresponding to the physical activity, and a virtual avatar. (Item 171) Item 159. The method of item 159, wherein the data is read from a networked memory. (Item 172) 1. A method for improving performance of a task, comprising: Retrieving task data relating to a task to be performed by a user; generating virtual content based on the task data; creating a virtual user interface within the user's field of view; retrieving a set of map points corresponding to the user's location; displaying the virtual content on the virtual user interface while the user is performing the task such that, when viewed by the user, the virtual user interface appears fixed at the set of map points; A method comprising: (Item 173) Item 173. The method of item 172, wherein the virtual content is a game having a virtual mapped pattern, and the game improves the user's performance of the task. (Item 174) Item 174. The method of item 173, wherein the game awards points for following the virtual mapped pattern. (Item 175) Item 175. The method of item 174, wherein the game doubles points for reaching a point within the virtual mapped pattern within a predetermined time. (Item 176) Item 176. The method of claim 175, wherein the game deducts points for deviating from the virtual mapped pattern. (Item 177) Item 176. The method of item 175, wherein the game deducts points for moving adjacent to a physics object. (Item 178) Item 173. The method of item 172, wherein the task is selected from the group consisting of operating landscaping equipment, retrieving inventory items, stocking retail shelves, and sorting mail. (Item 179) Item 173. The method of item 172, wherein the task data is read from a networked memory. (Item 180) receiving user input; generating additional virtual content based on the user input; displaying the additional virtual content on the virtual user interface while the user is performing the task; and Item 173. The method of item 172, further comprising: (Item 181) Item 181. The method of item 180, wherein the user input comprises a user action related to performance of the task. (Item 182) 1. A method for compensating for delay in an augmented reality display system, comprising: determining one or more timing mismatches in at least one stage of displaying a virtual image to a user; utilizing a prediction mechanism to predictively correct the timing mismatch; Compensating for at least one characteristic associated with the virtual image data based at least in part on the prediction mechanism; and A method comprising: (Item 183) 183. The method of claim 182, wherein the timing mismatch involves one or more sensor measurements made by one or more sensors, the one or more sensors being communicatively coupled to the processor. (Item 184) Item 183. The method of item 182, wherein the timing mismatch relates to a processing delay when processing incoming data. (Item 185) 183. The method of claim 182, wherein the prediction mechanism utilizes a filter to correct for the effects of the timing mismatch. (Item 186) 186. The method of claim 185, wherein the filter considers the relative speed of one or more sensor measurements, the sensor measurements being made by one or more sensors communicatively coupled to the processor. (Item 187) Item 183. The method of item 182, wherein the prediction mechanism utilizes a Kalman predictor. (Item 188) Item 188. The method of item 187, wherein the Kalman predictor is utilized during a display processing stage. (Item 189) Item 183. The method of item 182, wherein the compensation includes shifting data associated with the virtual image. (Item 190) Item 183. The method of item 182, wherein the compensation includes smoothing one or more visual artifacts associated with the virtual image. (Item 191) Item 183. The method of item 182, wherein the compensation includes correcting for negative effects of sensor measurements of one or more sensors, the one or more sensors being communicatively coupled to the processor. (Item 192) 1. A method of calibrating an augmented reality system, comprising: Displaying a virtual image to a user, the virtual image being displayed at a known focal length, the virtual image comprising pixel points; determining a location where the pixel point is displayed to the user, the pixel location being calculated based at least in part on a location of a pupil of the user's eye; aligning pixel points of said virtual image with known points in space; A method comprising: (Item 193) Item 331. The method according to item 192, wherein the steps of item 331 are repeated for multiple pixel points. (Item 194) Item 193. The method of item 192, wherein the locations at which the pixel points are displayed to the user are calculated based at least in part on locations of known points in the space. (Item 195) Item 193. The method of item 192, wherein the pupil location, the pixel point location and the known point location in space are collinear. (Item 196) Item 193. The method of item 192, wherein the location at which the pixel point is displayed to the user is modified based on user input. (Item 197) Item 197. The method of item 196, further comprising creating a gaming interface such that a plurality of pixel points are presented to the user. (Item 198) Item 198. The method of item 197, wherein the gaming interface includes emitting a laser through the user's eye movements. (Item 199) Item 198. The method of item 197, wherein the location of the known point is determined based at least in part on data received from one or more world cameras. (Item 200) Item 193. The method of item 192, wherein the steps of item 1 are repeated for the user's other eye. (Item 201) Item 194. The method of item 193, wherein the location of the pixel point is determined based at least in part on a function of the location of the pupil and the location of the known point. (Item 202) Item 202. The method of item 201, wherein the function comprises a quadratic function. (Item 203) 1. A method for displaying a virtual interface, comprising: Identifying a user interface to be displayed to the user based on user input; displaying an avatar user interface in relation to at least one physical object, the avatar user interface presenting at least one virtual user interface element; selecting at least one virtual user interface element based at least in part on an interaction between the user and the avatar user interface; A method comprising: (Item 204) Item 204. The method of item 203, wherein the user input includes a gesture. (Item 205) Item 205. The method of item 204, further comprising determining a location of the gesture relative to the user's augmented reality display system, wherein the virtual user interface is displayed at the determined location. (Item 206) Item 204. The method of item 203, wherein the avatar is a pre-selected avatar. (Item 207) Item 204. The method of item 203, wherein the avatar resembles the user. (Item 208) Item 204. The method of item 203, wherein the avatar is displayed as if standing on the physics object. (Item 209) Item 204. The method of item 203, wherein the at least one virtual element comprises an application. (Item 210) selecting the at least one virtual element based at least in part on another user input, the at least one virtual element comprising a contact of the user; and displaying another avatar representing the user's contact in relation to the avatar virtual user interface and the physical object, wherein interactions between the user and the user's contact are animated through virtual interactions between the avatar virtual user interface and the other avatar representing the user's contact; Item 204. The method of item 203, further comprising: (Item 211) Item 211. The method of item 210, wherein the interaction includes transmitting data between the user's augmented reality system and a computing system corresponding to the user's contact. (Item 212) Item 204. The method of item 203, wherein the user input comprises a voice command. (Item 213) 1. A method for displaying a virtual interface, comprising: Identifying a user interface to be displayed to the user based on user input; displaying a floating user interface with one or more selectable virtual interface elements in relation to at least one physical object, the floating user interface appearing to be pushed out from a location where the user input is received; A method comprising: (Item 214) Item 214. The method of item 213, wherein the user input comprises a gesture of the user's finger. (Item 215) Item 215. The method of item 214, wherein the user's finger gesture includes touching a finger of one hand of the user with another finger of the other hand of the user. (Item 216) Item 216. The method of item 215, further comprising determining a location where a finger of one hand of the user is touched with another finger of the other hand of the user, and the floating virtual user interface results from the determined location. (Item 217) Item 214. The method of item 213, wherein the floating user interface comprises a collection of three-dimensional blocks, a three-dimensional block of the collection of three-dimensional blocks representing the at least one selectable virtual user interface element. (Item 218) Item 214. The method of item 213, wherein the floating virtual user interface is created based at least in part on a world frame of reference. (Item 219) Item 214. The method of item 213, wherein the at least one selectable virtual user interface element exists as a stack of three-dimensional blocks, and the stack of three-dimensional blocks is rotated based at least in part on another user input. (Item 220) selecting a virtual user interface element of the at least one selectable virtual interface element based at least in part on another user input received from the user, the virtual interface element being associated with at least one other sub-virtual interface element; and displaying the at least one other sub-virtual interface element below the selected virtual interface element; Item 214. The method of item 213, further comprising: (Item 221) selecting a virtual user interface element of the at least one selectable virtual interface element based at least in part on another user input received from the user; and displaying content associated with the selected virtual interface element in a virtual box, the virtual box comprising content displayed within the user's field of view; Item 214. The method of item 213, further comprising: (Item 222) identifying a second user input indicating that the virtual box should be closed; animating the virtual box in a manner similar to crumpling paper based at least in part on the second user input; Terminating content associated with the selected virtual interface element; Item 222. The method of item 221, further comprising: (Item 223) Item 223. The method of item 222, wherein the second user input includes a gesture, the gesture resembling crumpling a piece of paper. (Item 224) 1. A method for displaying a virtual user interface, comprising: Identifying a user interface to be displayed to the user based on user input; displaying a floating user interface in relation to at least one physical object, the floating user interface comprising one or more selectable virtual interface elements; Including, The floating user interface appears to be placed on the at least one physics object, and rotation of the at least one physics object about a longitudinal axis of the at least one physics object causes additional selectable virtual interface elements to be displayed, the additional selectable virtual interface elements appearing to be placed on another side of the at least one physics object. Item 225. The method of item 224, wherein the at least one physical object comprises an arm of the user. (Item 226) Item 225. The method of item 224, wherein the user input includes a gesture. (Item 227) Item 227. The method of item 226, wherein the gesture comprises a cupping movement of the user's hand over the user's arm, in which the selectable virtual interface element appears to be displayed. (Item 228) Item 228. The method of item 227, further comprising determining a location on the user's arm of the movement that cups the user's hand, and the floating user interface originates from the determined location. (Item 229) Item 225. The method of item 224, further comprising terminating rendering of the floating virtual user interface based at least in part on another user input. (Item 230) Item 229. The method of item 229, wherein the other user input comprises a gesture, the gesture comprising a dragging movement of the user's cupped hand across the physical object. (Item 231) Item 225. The method of item 224, wherein the floating virtual user interface is created based at least in part on a hand-centered frame of reference. (Item 232) selecting a virtual user interface element of the at least one selectable virtual interface element based at least in part on another user input received from the user; and displaying content associated with the selected virtual interface element, the content being displayed in relation to the floating virtual interface; Item 225. The method of item 224, further comprising: (Item 233) Item 232. The method of item 231, wherein the displayed content corresponds to the hand-centered frame of reference. (Item 234) Item 232. The method of item 231, further comprising, when the floating virtual user interface is moving, based at least in part on input received from the user, moving the displayed content from the hand-centered frame of reference to a world-centered frame of reference such that the displayed content remains stationary. (Item 235) 1. A method for creating a virtual user interface, comprising: Identifying a user interface to be displayed to the user based on user input; displaying a virtual user interface associated with at least one physical object and arising from a location of touch on the at least one physical object, the virtual user interface appearing to protrude from a location of touch on the at least one physical object, the virtual user interface comprising at least one selectable virtual user interface element appearing to be attached to the virtual user interface; A method comprising: (Item 236) Item 236. The method of item 235, wherein the at least one physical object comprises a horizontal surface. (Item 237) Item 236. The method of item 235, wherein the user input includes a gesture. (Item 238) Item 238. The method of item 237, wherein the gesture includes an outstretched finger touching the at least one physical object for a predetermined period of time. (Item 239) Item 236. The method of item 235, further comprising determining a location where the user's finger touches the at least one physical object, the virtual user interface arising from the determined location. (Item 240) 241. The method of claim 235, further comprising terminating rendering of the floating virtual user interface based at least in part on another user input. Item 241. The method of item 240, wherein the other user input comprises a gesture, the gesture comprising a cutting movement with the user's hand across the rendered virtual user interface. (Item 242) Item 236. The method of item 235, wherein the virtual user interface is created based at least in part on a world-centered frame of reference such that the virtual user interface remains stationary with respect to any movement of the user. (Item 243) selecting a virtual user interface element of the at least one selectable virtual interface element based at least in part on another user input received from the user; and generating a second virtual user interface at another location of the at least one physical object, the second virtual user interface comprising an additional selectable virtual interface element; and Item 236. The method of item 235, further comprising: (Item 244) Item 236. The method of item 235, wherein the virtual user interface resembles a tree growing from the location where the at least one physical object is touched. (Item 245) 1. A method for displaying a virtual user interface, comprising: Identifying a user interface to be displayed to the user based on user input; rendering a virtual user interface that mirrors the user's finger movements such that the virtual user interface mirrors the finger movements; displaying one or more selectable user interface elements within the area in which the virtual user interface is depicted; and A method comprising: (Item 246) Item 246. The method of item 245, wherein the finger gesture comprises a pointing gesture using at least one finger of the user's hand for a predetermined period of time. (Item 247) Item 247. The method of item 246, further comprising determining a location of the pointing gesture, wherein the virtual user interface results from the determined location. (Item 248) Item 246. The method of item 245, further comprising detecting that the user is no longer drawing the virtual user interface element, wherein the virtual user interface element is displayed based at least in part on the detection. (Item 249) Item 246. The method of item 245, wherein the representation of the virtual user interface mirrors the dragging movement of the user's finger across space. (Item 250) Item 246. The method of item 245, further comprising terminating rendering of the floating virtual user interface based at least in part on another user input. (Item 251) Item 251. The method of item 250, wherein the other user input includes a gesture, the gesture including the end of a continuous dragging motion of the user's finger. (Item 252) Item 246. The method of item 245, wherein the virtual user interface is created based at least in part on a world-centered frame of reference such that the virtual user interface remains stationary with respect to any movement of the user. (Item 253) selecting a virtual user interface element from the at least one selectable virtual interface element based at least in part on another user input received from the user; generating content related to the virtual user interface and associated with the selected virtual interface element; Item 246. The method of item 245, further comprising: (Item 254) Item 246. The method of item 245, wherein the shape of the virtual user interface resembles the shape drawn by the user's finger. (Item 255) 1. A method for creating a virtual user interface, comprising: Identifying a user interface to be displayed to a user based on a hand gesture of the user; in response to the hand gesture, displaying a virtual user interface having at least one selectable virtual interface element on a vertical side of a physical object in a direction of the hand gesture such that the virtual user interface appears to be on the vertical side of the physical object; A method comprising: (Item 256) Item 256. The method of claim 255, further comprising displaying a collection of virtual content resembling paint specks in response to the hand gesture, the virtual content occurring at a point indicated by the hand gesture. (Item 257) Item 256. The method of item 255, wherein the hand gesture includes outstretched fingers raised in the direction of the vertical side of the physical object. (Item 258) Item 258. The method of item 257, further comprising determining a location of an outstretched finger raised in the direction of the vertical side of the physical object, wherein a collection of virtual content resembling a paint speck is displayed at the determined location. (Item 259) Item 259. The method of item 258, further comprising displaying a virtual speck on the vertical side of the physical object, wherein at least one characteristic of the virtual speck is displayed on the vertical side of the physical object based at least in part on a determined location of an outstretched finger raised in the direction of the vertical side of the physical object. (Item 260) Item 256. The method of item 255, wherein the physical object comprises a wall. (Item 261) Item 256. The method of item 255, wherein the virtual user interface is created based at least in part on a world-centered frame of reference such that the virtual user interface remains stationary with respect to any movement of the user. (Item 262) selecting a virtual user interface element from the at least one selectable virtual interface element based at least in part on another user input received from the user; generating content associated with the selected user interface element at another location on the physical object; and 256. The method of claim 255, further comprising: (Item 263) Item 2622. The method of item 2622, further comprising storing content associated with the selected user interface element in a reduced form, the reduced form comprising a virtual band, the virtual band being displayed around the user's hand. (Item 264) Item 264. The method of item 263, wherein the virtual band is created at least in part based on a hand-centered frame of reference such that the virtual band moves at least in part based on movement of the hand. (Item 265) Item 264. The method of claim 263, further comprising displaying the content in its complete form based at least in part on another gesture of the user. (Item 266) Item 266. The method of item 265, wherein the other gesture includes a movement of raising the hand along which the virtual band is displayed. (Item 267) 1. A method for creating a virtual user interface, comprising: Identifying a user interface to be displayed to a user based on a hand gesture of the user; displaying a virtual user interface in response to the hand gesture; Including, The method, wherein the virtual user interface resembles a horizontal thread with at least one selectable virtual interface element such that the at least one selectable virtual interface element moves and displays additional selectable virtual interface elements based at least in part on different hand gestures of the user. (Item 268) Item 268. The method of item 267, wherein the hand gesture includes touching a first finger of a first hand of the user with a second finger of a second hand. (Item 269) Item 269. The method of item 268, wherein the gesture further includes a movement of pulling both the first finger and the second finger apart such that the first finger moves away from the second finger. (Item 270) Item 271. The method of item 269, wherein the virtual thread mirrors the length of the movement of the first and second fingers, and the length of the virtual thread is based at least in part on the movement of the first and second fingers. Item 268. The method of item 267, further comprising determining a location of the touch movement of the first finger and the second finger, wherein the virtual thread originates from the determined location. (Item 272) Item 268. The method of item 267, wherein the other hand gesture includes a turning movement of the user's hand, the turning movement causing the at least one selectable virtual interface element to move in the direction of the turning movement. (Item 273) Item 268. The method of item 267, wherein the virtual user interface is created based at least in part on a world-centered frame of reference such that the virtual user interface remains stationary with respect to any movement of the user. (Item 274) selecting a virtual user interface element from the at least one selectable virtual interface element based at least in part on another hand gesture of the user; generating content associated with the selected user interface element at another location on the physical object; and Item 268. The method of item 267, further comprising: (Item 275) Item 275. The method of item 274, wherein the other hand gesture includes a movement of bringing one hand of the user forward. (Item 276) 1. A method for creating a virtual user interface, comprising: Identifying a user interface to be displayed to a user based on a hand gesture of the user; displaying a virtual user interface in response to the hand gesture; Including, The method, wherein the virtual user interface comprises a collection of vertical threads, each of the vertical threads being associated with at least one selectable virtual interface element. (Item 277) Item 277. The method of item 276, wherein the hand gesture includes a gesture of spreading a first hand of the user for a predetermined period of time. (Item 278) Item 278. The method of item 277, further comprising selecting the at least one selectable virtual interface element based at least in part on a gesture of another hand, the gesture of the other hand comprising a movement of pulling at least one of the virtual threads of the set of virtual threads using two fingers of the user. (Item 279) Item 277. The method of item 276, further comprising transmitting data associated with a first virtual element associated with a first virtual yarn to a second virtual element associated with a second virtual yarn based at least in part on the hand gesture. (Item 280) Item 279. The method of item 279, wherein the other hand gesture includes a squeezing movement of the first virtual element associated with the first virtual yarn with the second virtual element associated with the second virtual yarn, the squeezing movement being performed by the user's fingers. (Item 281) Item 277. The method of item 276, wherein the virtual interface is displayed in relation to at least one physical object. (Item 282) Item 282. The method of item 281, wherein the at least one physical object comprises a wall. (Item 283) Item 277. The method of item 276, further comprising determining a location of a hand gesture, wherein the virtual thread originates from the determined location. (Item 284) Item 277. The method of item 276, wherein the virtual user interface is created based at least in part on a world-centered frame of reference such that the virtual user interface remains stationary with respect to any movement of the user. (Item 285) selecting a virtual user interface element from the at least one selectable virtual interface element based at least in part on another hand gesture of the user; generating virtual content associated with the selected user interface element at another location on the physical object; Item 277. The method of item 276, further comprising: (Item 286) 1. A method for creating a virtual user interface, comprising: Identifying a user interface to be displayed to a user based on a hand gesture of the user; displaying a virtual user interface in response to the hand gesture; Including, The method, wherein the virtual user interface comprises a collection of vertical threads, each of the vertical threads being associated with at least one selectable virtual interface element. (Item 287) Item 287. The method of item 286, wherein the hand gesture includes a gesture of spreading a first hand of the user for a predetermined period of time. (Item 288) Item 288. The method of item 287, further comprising selecting the at least one selectable virtual interface element based at least in part on a gesture of another hand, the gesture of the other hand comprising a movement of the user using two fingers to pull at least one of the virtual threads of the set of virtual threads. (Item 289) Item 287. The method of item 286, further comprising transmitting data associated with a first virtual element associated with a first virtual yarn to a second virtual element associated with a second virtual yarn based at least in part on the hand gesture. (Item 290) Item 289. The method of item 289, wherein the other hand gesture includes a squeezing movement of the first virtual element associated with the first virtual yarn with the second virtual element associated with the second virtual yarn, the squeezing movement being performed by the user's fingers. (Item 291) Item 287. The method of item 286, wherein the virtual interface is displayed in relation to at least one physical object. (Item 292) Item 292. The method of item 291, wherein the at least one physical object comprises a wall. (Item 293) Item 287. The method of item 286, further comprising determining a location of a hand gesture, wherein the virtual thread originates from the determined location. (Item 294) Item 287. The method of item 286, wherein the virtual user interface is created based at least in part on a world-centered frame of reference such that the virtual user interface remains stationary with respect to any movement of the user. (Item 295) selecting a virtual user interface element from the at least one selectable virtual interface element based at least in part on another hand gesture of the user; generating virtual content associated with the selected user interface element at another location on the physical object; 287. The method of claim 286, further comprising: (Item 296) 1. A method for creating a virtual user interface, comprising: Identifying a user interface to be displayed to a user based on a hand gesture of the user; displaying a virtual user interface in response to the hand gesture; Including, The method, wherein the virtual user interface resembles a virtual spider's web, and the user pulls virtual threads of the virtual spider's web to move the virtual interface closer to the user. (Item 297) Item 297. The method of item 296, wherein the hand gesture comprises a pulling motion with a clenched fist of the user's hand. (Item 298) Item 298. The method of item 297, wherein the virtual threads of the virtual spider web comprise at least one selectable virtual element. (Item 299) Item 299. The method of item 298, further comprising selecting the at least one selectable virtual interface element based at least in part on a gesture of another hand, the gesture of the other hand comprising a movement that pulls at least one of the virtual threads toward the user. (Item 300) Item 297. The method of item 296, further comprising transmitting data associated with a first virtual element associated with a first virtual yarn to a second virtual element associated with a second virtual yarn based at least in part on the hand gesture. (Item 301) Item 301. The method of item 300, wherein the other hand gesture includes a squeezing movement of the first virtual element associated with the first virtual yarn with the second virtual element associated with the second virtual yarn, the squeezing movement being performed by the user's fingers. (Item 302) Item 297. The method of item 296, wherein the virtual interface is displayed in relation to at least one physical object. (Item 303) Item 303. The method of item 302, wherein the at least one physical object comprises a wall. (Item 304) Item 297. The method of item 296, further comprising determining a location of a hand gesture, wherein the virtual spider web originates from the determined location. (Item 305) Item 297. The method of item 296, wherein the virtual user interface is created based at least in part on a world-centered frame of reference such that the virtual user interface remains stationary with respect to any movement of the user. (Item 306) selecting a virtual user interface element from the at least one selectable virtual interface element based at least in part on another hand gesture of the user; generating virtual content associated with the selected user interface element at another location on the physical object; 297. The method of claim 296, further comprising: (Item 307) The method of any one of items 1-306, implemented as a system having means for implementing the method steps. (Item 308) Item 1-306. The method of items 1-306, implemented as a computer program product comprising a computer usable storage medium having executable code for performing the method steps. [Brief explanation of the drawings]

[0205] The drawings illustrate the design and utility of various embodiments of the present invention. It should be noted that the drawings are not drawn to scale, and that elements of similar structure or function are represented by like reference numerals throughout the drawings. To better understand how the foregoing and other advantages and objects of the various embodiments of the present invention are obtained, a brief description of the preferred embodiments of the present invention will be provided by reference to specific embodiments thereof that are illustrated in the accompanying drawings. With the understanding that these drawings depict only exemplary embodiments of the invention and are not to be considered as limiting its scope, the present invention will be described and explained with additional specificity and detail through the use of the accompanying drawings. [Figure 1] FIG. 1 illustrates a system architecture of an augmented reality (AR) system interacting with one or more servers, according to one illustrated embodiment. [Figure 2] FIG. 2 illustrates a detailed view of a mobile phone used as an AR device interacting with one or more servers, according to one of the illustrated embodiments. [Figure 3] FIG. 3 illustrates a plan view of an exemplary AR device mounted on a user's head, according to one of the illustrated embodiments. [Figure 4A] 4A-4D illustrate one or more embodiments of various internal processing components of a wearable AR device. [Figure 4B] 4A-4D illustrate one or more embodiments of various internal processing components of a wearable AR device. [Figure 4C] 4A-4D illustrate one or more embodiments of various internal processing components of a wearable AR device. [Figure 4D] 4A-4D illustrate one or more embodiments of various internal processing components of a wearable AR device. [Figure 5A] 5A-5H illustrate an embodiment of the transmission of focused light to a user through a transmissive beam splitter substrate. [Figure 5B] 5A-5H illustrate an embodiment of the transmission of focused light to a user through a transmissive beam splitter substrate. [Figure 5C] 5A-5H illustrate an embodiment of the transmission of focused light to a user through a transmissive beam splitter substrate. [Figure 5D] 5A-5H illustrate an embodiment of the transmission of focused light to a user through a transmissive beam splitter substrate. [Figure 5E] 5A-5H illustrate an embodiment of the transmission of focused light to a user through a transmissive beam splitter substrate. [Figure 5F] 5A-5H illustrate an embodiment of the transmission of focused light to a user through a transmissive beam splitter substrate. [Figure 5G] 5A-5H illustrate an embodiment of the transmission of focused light to a user through a transmissive beam splitter substrate. [Figure 5H]5A-5H illustrate an embodiment of the transmission of focused light to a user through a transmissive beam splitter substrate. [Figure 6A] 6A and 6B illustrate embodiments of the combination of lens elements with the transmissive beam splitter substrates of FIGS. 5A-5H. [Figure 6B] 6A and 6B illustrate embodiments of the combination of lens elements with the transmissive beam splitter substrates of FIGS. 5A-5H. [Figure 7A] 7A and 7B illustrate an embodiment of the use of one or more waveguides to transmit light to a user. [Figure 7B] 7A and 7B illustrate an embodiment of the use of one or more waveguides to transmit light to a user. [Figure 8A] 8A-8Q illustrate possible diffractive optical element (ODE) embodiments. [Figure 8B] 8A-8Q illustrate possible diffractive optical element (ODE) embodiments. [Figure 8C] 8A-8Q illustrate possible diffractive optical element (ODE) embodiments. [Figure 8D] 8A-8Q illustrate possible diffractive optical element (ODE) embodiments. [Figure 8E] 8A-8Q illustrate possible diffractive optical element (ODE) embodiments. [Figure 8F] 8A-8Q illustrate possible diffractive optical element (ODE) embodiments. [Figure 8G] 8A-8Q illustrate possible diffractive optical element (ODE) embodiments. [Figure 8H] 8A-8Q illustrate possible diffractive optical element (ODE) embodiments. [Figure 8I] 8A-8Q illustrate possible diffractive optical element (ODE) embodiments. [Figure 8J]8A-8Q illustrate possible diffractive optical element (ODE) embodiments. [Figure 8K] 8A-8Q illustrate possible diffractive optical element (ODE) embodiments. [Figure 8L] 8A-8Q illustrate possible diffractive optical element (ODE) embodiments. [Figure 8M] 8A-8Q illustrate possible diffractive optical element (ODE) embodiments. [Figure 8N] 8A-8Q illustrate possible diffractive optical element (ODE) embodiments. [Figure 8O] 8A-8Q illustrate possible diffractive optical element (ODE) embodiments. [Figure 8P] 8A-8Q illustrate possible diffractive optical element (ODE) embodiments. [Figure 8Q] 8A-8Q illustrate possible diffractive optical element (ODE) embodiments. [Figure 9] 9A and 9B illustrate a wavefront generated from a light projector according to one of the illustrated embodiments. [Figure 10] FIG. 10 illustrates an embodiment of a stacked configuration of multiple transmissive beam splitter substrates coupled with optical elements, according to one of the illustrated embodiments. [Figure 11A] 11A-11C illustrate a collection of beamlets projected into a user's pupil, according to an illustrative embodiment. [Figure 11B] 11A-11C illustrate a collection of beamlets projected into a user's pupil, according to an illustrative embodiment. [Figure 11C] 11A-11C illustrate a collection of beamlets projected into a user's pupil, according to an illustrative embodiment. [Figure 12A] 12A and 12B illustrate the configuration of an array of microprojectors, according to an illustrative embodiment. [Figure 12B] 12A and 12B illustrate the configuration of an array of microprojectors, according to an illustrative embodiment. [Figure 13A] 13A-13M illustrate embodiments of the combination of a microprojector and an optical element, according to an illustrated embodiment. [Figure 13B] 13A-13M illustrate embodiments of the combination of a microprojector and an optical element, according to an illustrated embodiment. [Figure 13C] 13A-13M illustrate embodiments of the combination of a microprojector and an optical element, according to an illustrated embodiment. [Figure 13D] 13A-13M illustrate embodiments of the combination of a microprojector and an optical element, according to an illustrated embodiment. [Figure 13E] 13A-13M illustrate embodiments of the combination of a microprojector and an optical element, according to an illustrated embodiment. [Figure 13F] 13A-13M illustrate embodiments of the combination of a microprojector and an optical element, according to an illustrated embodiment. [Figure 13G] 13A-13M illustrate embodiments of the combination of a microprojector and an optical element, according to an illustrated embodiment. [Figure 13H] 13A-13M illustrate embodiments of the combination of a microprojector and an optical element, according to an illustrated embodiment. [Figure 13I] 13A-13M illustrate embodiments of the combination of a microprojector and an optical element, according to an illustrated embodiment. [Figure 13J] 13A-13M illustrate embodiments of the combination of a microprojector and an optical element, according to an illustrated embodiment. [Figure 13K] 13A-13M illustrate embodiments of the combination of a microprojector and an optical element, according to an illustrated embodiment. [Figure 13L] 13A-13M illustrate embodiments of the combination of a microprojector and an optical element, according to an illustrated embodiment. [Figure 13M] 13A-13M illustrate embodiments of the combination of a microprojector and an optical element, according to an illustrated embodiment. [Figure 14A] 14A-14F illustrate embodiments of a spatial light modulator combined with an optical element, according to an illustrated embodiment. [Figure 14B] 14A-14F illustrate embodiments of a spatial light modulator combined with an optical element, according to an illustrated embodiment. [Figure 14C] 14A-14F illustrate embodiments of a spatial light modulator combined with an optical element, according to an illustrated embodiment. [Figure 14D] 14A-14F illustrate embodiments of a spatial light modulator combined with an optical element, according to an illustrated embodiment. [Figure 14E] 14A-14F illustrate embodiments of a spatial light modulator combined with an optical element, according to an illustrated embodiment. [Figure 14F] 14A-14F illustrate embodiments of a spatial light modulator combined with an optical element, according to an illustrated embodiment. [Figure 15A] 15A-15C illustrate the use of a wedge-type waveguide in conjunction with multiple light sources, according to an illustrative embodiment. [Figure 15B] 15A-15C illustrate the use of a wedge-type waveguide in conjunction with multiple light sources, according to an illustrative embodiment. [Figure 15C] 15A-15C illustrate the use of a wedge-type waveguide in conjunction with multiple light sources, according to an illustrative embodiment. [Figure 16A] 16A-16O illustrate embodiments of coupling optical elements to optical fibers, according to illustrated embodiments. [Figure 16B] 16A-16O illustrate embodiments of coupling optical elements to optical fibers, according to illustrated embodiments. [Figure 16C] 16A-16O illustrate embodiments of coupling optical elements to optical fibers, according to illustrated embodiments. [Figure 16D]16A-16O illustrate embodiments of coupling optical elements to optical fibers, according to illustrated embodiments. [Figure 16E] 16A-16O illustrate embodiments of coupling optical elements to optical fibers, according to illustrated embodiments. [Figure 16F] 16A-16O illustrate embodiments of coupling optical elements to optical fibers, according to illustrated embodiments. [Figure 16G] 16A-16O illustrate embodiments of coupling optical elements to optical fibers, according to illustrated embodiments. [Figure 16H] 16A-16O illustrate embodiments of coupling optical elements to optical fibers, according to illustrated embodiments. [Figure 16I] 16A-16O illustrate embodiments of coupling optical elements to optical fibers, according to illustrated embodiments. [Figure 16J] 16A-16O illustrate embodiments of coupling optical elements to optical fibers, according to illustrated embodiments. [Figure 16K] 16A-16O illustrate embodiments of coupling optical elements to optical fibers, according to illustrated embodiments. [Figure 16L] 16A-16O illustrate embodiments of coupling optical elements to optical fibers, according to illustrated embodiments. [Figure 16M] 16A-16O illustrate embodiments of coupling optical elements to optical fibers, according to illustrated embodiments. [Figure 16N] 16A-16O illustrate embodiments of coupling optical elements to optical fibers, according to illustrated embodiments. [Figure 16O] 16A-16O illustrate embodiments of coupling optical elements to optical fibers, according to illustrated embodiments. [Figure 17] FIG. 17 illustrates a notch filter according to one of the illustrated embodiments. [Figure 18] FIG. 18 illustrates a spiral pattern of a fiber scanning display according to one of the illustrated embodiments. [Figure 19A] 19A-19N illustrate the occlusion effect when presenting a dark field to a user, according to an illustrated embodiment. [Figure 19B] 19A-19N illustrate the occlusion effect when presenting a dark field to a user, according to an illustrated embodiment. [Figure 19C] 19A-19N illustrate the occlusion effect when presenting a dark field to a user, according to an illustrated embodiment. [Figure 19D] 19A-19N illustrate the occlusion effect when presenting a dark field to a user, according to an illustrated embodiment. [Figure 19E] 19A-19N illustrate the occlusion effect when presenting a dark field to a user, according to an illustrated embodiment. [Figure 19F] 19A-19N illustrate the occlusion effect when presenting a dark field to a user, according to an illustrated embodiment. [Figure 19G] 19A-19N illustrate the occlusion effect when presenting a dark field to a user, according to an illustrated embodiment. [Figure 19H] 19A-19N illustrate the occlusion effect when presenting a dark field to a user, according to an illustrated embodiment. [Figure 19I] 19A-19N illustrate the occlusion effect when presenting a dark field to a user, according to an illustrated embodiment. [Figure 19J] 19A-19N illustrate the occlusion effect when presenting a dark field to a user, according to an illustrated embodiment. [Figure 19K] 19A-19N illustrate the occlusion effect when presenting a dark field to a user, according to an illustrated embodiment. [Figure 19L] 19A-19N illustrate the occlusion effect when presenting a dark field to a user, according to an illustrated embodiment. [Figure 19M] 19A-19N illustrate the occlusion effect when presenting a dark field to a user, according to an illustrated embodiment. [Figure 19N]19A-19N illustrate the occlusion effect when presenting a dark field to a user, according to an illustrated embodiment. [Figure 20A] 20A-20O illustrate various waveguide assembly embodiments, according to illustrative embodiments. [Figure 20B] 20A-20O illustrate various waveguide assembly embodiments, according to illustrative embodiments. [Figure 20C] 20A-20O illustrate various waveguide assembly embodiments, according to illustrative embodiments. [Figure 20D] 20A-20O illustrate various waveguide assembly embodiments, according to illustrative embodiments. [Figure 20E] 20A-20O illustrate various waveguide assembly embodiments, according to illustrative embodiments. [Figure 20F] 20A-20O illustrate various waveguide assembly embodiments, according to illustrative embodiments. [Figure 20G] 20A-20O illustrate various waveguide assembly embodiments, according to illustrative embodiments. [Figure 20H] 20A-20O illustrate various waveguide assembly embodiments, according to illustrative embodiments. [Figure 20I] 20A-20O illustrate various waveguide assembly embodiments, according to illustrative embodiments. [Figure 20J] 20A-20O illustrate various waveguide assembly embodiments, according to illustrative embodiments. [Figure 20K] 20A-20O illustrate various waveguide assembly embodiments, according to illustrative embodiments. [Figure 20L] 20A-20O illustrate various waveguide assembly embodiments, according to illustrative embodiments. [Figure 20M] 20A-20O illustrate various waveguide assembly embodiments, according to illustrative embodiments. [Figure 20N]20A-20O illustrate various waveguide assembly embodiments, according to illustrative embodiments. [Figure 20O] 20A-20O illustrate various waveguide assembly embodiments, according to illustrative embodiments. [Figure 21A] 21A-21N illustrate various configurations of DOEs coupled to other optical elements, according to illustrative embodiments. [Figure 21B] 21A-21N illustrate various configurations of DOEs coupled to other optical elements, according to illustrative embodiments. [Figure 21C] 21A-21N illustrate various configurations of DOEs coupled to other optical elements, according to illustrative embodiments. [Figure 21D] 21A-21N illustrate various configurations of DOEs coupled to other optical elements, according to illustrative embodiments. [Figure 21E] 21A-21N illustrate various configurations of DOEs coupled to other optical elements, according to illustrative embodiments. [Figure 21F] 21A-21N illustrate various configurations of DOEs coupled to other optical elements, according to illustrative embodiments. [Figure 21G] 21A-21N illustrate various configurations of DOEs coupled to other optical elements, according to illustrative embodiments. [Figure 21H] 21A-21N illustrate various configurations of DOEs coupled to other optical elements, according to illustrative embodiments. [Figure 21I] 21A-21N illustrate various configurations of DOEs coupled to other optical elements, according to illustrative embodiments. [Figure 21J] 21A-21N illustrate various configurations of DOEs coupled to other optical elements, according to illustrative embodiments. [Figure 21K] 21A-21N illustrate various configurations of DOEs coupled to other optical elements, according to illustrative embodiments. [Figure 21L]21A-21N illustrate various configurations of DOEs coupled to other optical elements, according to illustrative embodiments. [Figure 21M] 21A-21N illustrate various configurations of DOEs coupled to other optical elements, according to illustrative embodiments. [Figure 21N] 21A-21N illustrate various configurations of DOEs coupled to other optical elements, according to illustrative embodiments. [Figure 22A] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22B] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22C] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22D] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22E] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22F] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22G] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22H] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22I] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22J] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22K] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22L]22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22M] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22N] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22O] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22P] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22Q] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22R] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22S] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22T] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22U] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22V] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22W] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22X] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 22Y] 22A-22Y illustrate various configurations of freeform optics, according to illustrative embodiments. [Figure 23] FIG. 23 illustrates a top view of the components of a simplified individual AR device. [Figure 24]FIG. 24 illustrates an exemplary embodiment of the optics of an individual AR system. [Figure 25] FIG. 25 illustrates the system architecture of an individual AR system, according to one embodiment. [Figure 26] FIG. 26 illustrates a room-based sensor system, according to one embodiment. [Figure 27] FIG. 27 illustrates the communication architecture of the augmented reality system and the interaction of multiple users' augmented reality systems with the cloud. [Figure 28] FIG. 28 illustrates a simplified diagram of a passable world model, according to one embodiment. [Figure 29] FIG. 29 illustrates an exemplary method for rendering using a passable world model, according to one embodiment. [Figure 30] FIG. 30 illustrates a high-level flow diagram for the process of recognizing objects, according to one embodiment. [Figure 31] FIG. 31 illustrates a ring buffer approach employed by an object recognizer to recognize objects in a passable world, according to one embodiment. [Figure 32] FIG. 32 illustrates an exemplary topology map, according to one embodiment. [Figure 33] FIG. 33 illustrates a high-level flow diagram for the process of location determination using a topology map, according to one embodiment. [Figure 34] FIG. 34 illustrates a geometric map as a connection between various keyframes, according to one embodiment. [Figure 35] FIG. 35 illustrates an exemplary embodiment of a topology map layered on top of a geometric map, according to one embodiment. [Figure 36] FIG. 36 illustrates a high-level flow diagram for a process of performing wave propagation flux adjustment, according to one embodiment. [Figure 37] FIG. 37 illustrates a map point as seen through a virtual keyframe and a rendered line from the map point to the keyframe, according to one embodiment. [Figure 38] FIG. 38 illustrates a high-level flow diagram for a process of discovering map points based on rendering rather than searching, according to one embodiment. [Figure 39] FIG. 39 illustrates a high-level flow diagram for a process of rendering a virtual object based on a light map, according to one embodiment. [Figure 40] FIG. 40 illustrates a high-level flow diagram for the process of creating a light map, according to one embodiment. [Figure 41] FIG. 41 depicts a user-centric optical map, according to one embodiment. [Figure 42] FIG. 42 depicts an object-centric light map, according to one embodiment. [Figure 43] FIG. 43 illustrates a high-level flow diagram for the process of transforming a light map, according to one embodiment. [Figure 44] FIG. 44 illustrates a library of autonomous navigation definitions or objects, according to one embodiment. [Figure 45] FIG. 45 illustrates the interaction of various autonomous navigation objects, according to one embodiment. [Figure 46] FIG. 46 illustrates a stack of autonomous navigation definitions or objects, according to one embodiment. [Figure 47A] 47A-47B illustrate the use of autonomous navigation definitions to identify emotional states, according to one embodiment. [Figure 47B] 47A-47B illustrate the use of autonomous navigation definitions to identify emotional states, according to one embodiment. [Figure 48] FIG. 48 illustrates a correlation threshold graph to be used to define an autonomous navigation definition or object, according to one embodiment. [Figure 49] FIG. 49 illustrates a family tree of a passable world model, according to one embodiment. [Figure 50] FIG. 50 illustrates an exemplary method for displaying a virtual scene, according to one embodiment. [Figure 51] FIG. 51 illustrates a plan view of the various modules of the AR system according to one of the illustrated embodiments. [Figure 52] FIG. 52 illustrates an example of an object viewed by a user when an AR device is operated in an augmented reality mode, according to one of the illustrated embodiments. [Figure 53] FIG. 53 illustrates an example of an object viewed by a user when the AR device is operated in virtual mode, according to one of the illustrated embodiments. [Figure 54] FIG. 54 illustrates an example of an object viewed by a user when the AR device is operated in a hybrid virtual interface mode, according to one of the illustrated embodiments. [Figure 55] FIG. 55 illustrates an embodiment in which two users are located in different geographic locations and each interact with the other user and a common virtual world through their respective user devices, according to one embodiment. [Figure 56] FIG. 56 illustrates an embodiment in which the embodiment of FIG. 55 is extended to include the use of a tactile device, according to one embodiment. [Figure 57A] 57A-57B illustrate examples of mixed mode interfaces according to one or more embodiments. [Figure 57B] 57A-57B illustrate examples of mixed mode interfaces according to one or more embodiments. [Figure 58] FIG. 58 illustrates an exemplary illustration of a user's view when interfacing with an AR system, according to one embodiment. [Figure 59] FIG. 59 illustrates an exemplary illustration of a user's view showing virtual objects triggered by physical objects when the user interfaces with the system in augmented reality mode, according to one embodiment. [Figure 60]FIG. 60 illustrates one embodiment of an integrated augmented and virtual reality configuration in which one user in an augmented reality experience visualizes the presence of another user in a virtual reality experience. [Figure 61] FIG. 61 illustrates one embodiment of a time and / or contingent event based augmented reality experience configuration. [Figure 62] FIG. 62 illustrates one embodiment of a user display configuration suitable for virtual and / or augmented reality experiences. [Figure 63] FIG. 63 illustrates one embodiment of local and cloud-based computing collaboration. [Figure 64] FIG. 64 illustrates various aspects of an alignment configuration according to one of the illustrated embodiments. [Figure 65] FIG. 65 illustrates an example scenario of interacting with an AR system, according to one embodiment. [Figure 66] FIG. 66 illustrates another perspective view of the example scenario of FIG. 65 according to another embodiment. [Figure 67] FIG. 67 illustrates yet another perspective view of the example scenario of FIG. 65, according to another embodiment. [Figure 68] FIG. 68 illustrates a top view of an example scenario, according to one embodiment. [Figure 69] FIG. 69 illustrates a game view of the example scenario of FIGS. 65-68, according to one embodiment. [Figure 70] FIG. 70 illustrates a top view of the example scenario of FIGS. 65-68, according to one embodiment. [Figure 71] FIG. 71 illustrates an augmented reality scenario involving multiple users, according to one embodiment. [Figure 72A] 72A-72B illustrate the use of a smartphone or tablet as an AR device, according to one embodiment. [Figure 72B] 72A-72B illustrate the use of a smartphone or tablet as an AR device, according to one embodiment. [Figure 73]FIG. 73 illustrates an exemplary method of using location location to communicate between AR system users, according to one embodiment. [Figure 74A] 74A-74B illustrate an exemplary office scenario interacting with an AR system, according to one embodiment. [Figure 74B] 74A-74B illustrate an exemplary office scenario interacting with an AR system, according to one embodiment. [Figure 75] FIG. 75 illustrates an example scenario of interacting with an AR system in a home, according to one embodiment. [Figure 76] FIG. 76 illustrates another example scenario of interacting with an AR system in a home, according to one embodiment. [Figure 77] FIG. 77 illustrates another example scenario of interacting with an AR system in the home, according to one embodiment. [Figure 78A] 78A-78B illustrate yet another example scenario of interacting with an AR system in the home, according to one embodiment. [Figure 78B] 78A-78B illustrate yet another example scenario of interacting with an AR system in the home, according to one embodiment. [Figure 79A] 79A-79E illustrate another example scenario of interacting with an AR system in the home, according to one embodiment. [Figure 79B] 79A-79E illustrate another example scenario of interacting with an AR system in the home, according to one embodiment. [Figure 79C] 79A-79E illustrate another example scenario of interacting with an AR system in the home, according to one embodiment. [Figure 79D] 79A-79E illustrate another example scenario of interacting with an AR system in the home, according to one embodiment. [Figure 79E] 79A-79E illustrate another example scenario of interacting with an AR system in the home, according to one embodiment. [Figure 80A]80A-80C illustrate another example scenario of interacting with an AR system in a virtual room, according to one embodiment. [Figure 80B] 80A-80C illustrate another example scenario of interacting with an AR system in a virtual room, according to one embodiment. [Figure 80C] 80A-80C illustrate another example scenario of interacting with an AR system in a virtual room, according to one embodiment. [Figure 81] FIG. 81 illustrates another exemplary user interaction scenario, according to one embodiment. [Figure 82] FIG. 82 illustrates another exemplary user interaction scenario, according to one embodiment. [Figure 83A] 83A-83B illustrate yet another example user interaction scenario according to one or more embodiments. [Figure 83B] 83A-83B illustrate yet another example user interaction scenario according to one or more embodiments. [Figure 84A] 84A-84C illustrate a user interacting with an AR system in a virtual space, according to one or more embodiments. [Figure 84B] 84A-84C illustrate a user interacting with an AR system in a virtual space, according to one or more embodiments. [Figure 84C] 84A-84C illustrate a user interacting with an AR system in a virtual space, according to one or more embodiments. [Figure 85A] 85A-85C illustrate various user interface embodiments. [Figure 85B] 85A-85C illustrate various user interface embodiments. [Figure 85C] 85A-85C illustrate various user interface embodiments. [Figure 86A] 86A-86C illustrate other embodiments for creating a user interface according to one or more embodiments. [Figure 86B]86A-86C illustrate other embodiments for creating a user interface according to one or more embodiments. [Figure 86C] 86A-86C illustrate other embodiments for creating a user interface according to one or more embodiments. [Figure 87A] 87A-87C illustrate other embodiments for creating and navigating a user interface according to one or more embodiments. [Figure 87B] 87A-87C illustrate other embodiments for creating and navigating a user interface according to one or more embodiments. [Figure 87C] 87A-87C illustrate other embodiments for creating and navigating a user interface according to one or more embodiments. [Figure 88A] 88A-88C illustrate a user interface created on a user's hand in accordance with one or more embodiments. [Figure 88B] 88A-88C illustrate a user interface created on a user's hand in accordance with one or more embodiments. [Figure 88C] 88A-88C illustrate a user interface created on a user's hand in accordance with one or more embodiments. [Figure 89A] 89A-89J illustrate an exemplary user shopping experience using an AR system, according to one or more embodiments. [Figure 89B] 89A-89J illustrate an exemplary user shopping experience using an AR system, according to one or more embodiments. [Figure 89C] 89A-89J illustrate an exemplary user shopping experience using an AR system, according to one or more embodiments. [Figure 89D] 89A-89J illustrate an exemplary user shopping experience using an AR system, according to one or more embodiments. [Figure 89E] 89A-89J illustrate an exemplary user shopping experience using an AR system, according to one or more embodiments. [Figure 89F] 89A-89J illustrate an exemplary user shopping experience using an AR system, according to one or more embodiments. [Figure 89G] 89A-89J illustrate an exemplary user shopping experience using an AR system, according to one or more embodiments. [Figure 89H] 89A-89J illustrate an exemplary user shopping experience using an AR system, according to one or more embodiments. [Figure 89I] 89A-89J illustrate an exemplary user shopping experience using an AR system, according to one or more embodiments. [Figure 89J] 89A-89J illustrate an exemplary user shopping experience using an AR system, according to one or more embodiments. [Figure 90] FIG. 90 illustrates an exemplary library experience using an AR system, according to one or more embodiments. [Figure 91A] 91A-91F illustrate an example health care experience using an AR system, according to one or more embodiments. [Figure 91B] 91A-91F illustrate an example health care experience using an AR system, according to one or more embodiments. [Figure 91C] 91A-91F illustrate an example health care experience using an AR system, according to one or more embodiments. [Figure 91D] 91A-91F illustrate an example health care experience using an AR system, according to one or more embodiments. [Figure 91E] 91A-91F illustrate an example health care experience using an AR system, according to one or more embodiments. [Figure 91F] 91A-91F illustrate an example health care experience using an AR system, according to one or more embodiments. [Figure 92] FIG. 92 illustrates an example work experience using an AR system, according to one or more embodiments. [Figure 93A]93A-93L illustrate an example workspace experience using an AR system, according to one or more embodiments. [Figure 93B] 93A-93L illustrate an example workspace experience using an AR system, according to one or more embodiments. [Figure 93C] 93A-93L illustrate an example workspace experience using an AR system, according to one or more embodiments. [Figure 93D] 93A-93L illustrate an example workspace experience using an AR system, according to one or more embodiments. [Figure 93E] 93A-93L illustrate an example workspace experience using an AR system, according to one or more embodiments. [Figure 93F] 93A-93L illustrate an example workspace experience using an AR system, according to one or more embodiments. [Figure 93G] 93A-93L illustrate an example workspace experience using an AR system, according to one or more embodiments. [Figure 93H] 93A-93L illustrate an example workspace experience using an AR system, according to one or more embodiments. [Figure 93I] 93A-93L illustrate an example workspace experience using an AR system, according to one or more embodiments. [Figure 93J] 93A-93L illustrate an example workspace experience using an AR system, according to one or more embodiments. [Figure 93K] 93A-93L illustrate an example workspace experience using an AR system, according to one or more embodiments. [Figure 93L] 93A-93L illustrate an example workspace experience using an AR system, according to one or more embodiments. [Figure 94] FIG. 94 illustrates another example workspace experience using an AR system, in accordance with one or more embodiments. [Figure 95A] 95A-95E illustrate another AR experience according to one or more embodiments. [Figure 95B] 95A-95E illustrate another AR experience according to one or more embodiments. [Figure 95C] 95A-95E illustrate another AR experience according to one or more embodiments. [Figure 95D] 95A-95E illustrate another AR experience according to one or more embodiments. [Figure 95E] 95A-95E illustrate another AR experience according to one or more embodiments. [Figure 96A] 96A-96D illustrate yet another AR experience according to one or more embodiments. [Figure 96B] 96A-96D illustrate yet another AR experience according to one or more embodiments. [Figure 96C] 96A-96D illustrate yet another AR experience according to one or more embodiments. [Figure 96D] 96A-96D illustrate yet another AR experience according to one or more embodiments. [Figure 97A] 97A-97H illustrate a gaming experience using an AR system according to one or more embodiments. [Figure 97B] 97A-97H illustrate a gaming experience using an AR system according to one or more embodiments. [Figure 97C] 97A-97H illustrate a gaming experience using an AR system according to one or more embodiments. [Figure 97D] 97A-97H illustrate a gaming experience using an AR system according to one or more embodiments. [Figure 97E] 97A-97H illustrate a gaming experience using an AR system according to one or more embodiments. [Figure 97F] 97A-97H illustrate a gaming experience using an AR system according to one or more embodiments. [Figure 97G] 97A-97H illustrate a gaming experience using an AR system according to one or more embodiments. [Figure 97H] 97A-97H illustrate a gaming experience using an AR system according to one or more embodiments. [Figure 98A] 98A-98D illustrate a web shopping experience using an AR system according to one or more embodiments. [Figure 98B] 98A-98D illustrate a web shopping experience using an AR system according to one or more embodiments. [Figure 98C] 98A-98D illustrate a web shopping experience using an AR system according to one or more embodiments. [Figure 98D] 98A-98D illustrate a web shopping experience using an AR system according to one or more embodiments. [Figure 99] FIG. 99 illustrates a block diagram of various games within a gaming platform, according to one or more embodiments. [Figure 100] FIG. 100 illustrates various user inputs for communicating with an augmented reality system, according to one embodiment. [Figure 101] FIG. 101 illustrates LED lights and diodes that track the movement of a user's eyes, according to one embodiment. [Figure 102] FIG. 102 illustrates a Purkinje image, according to one embodiment. [Figure 103] FIG. 103 illustrates various hand gestures that may be used to communicate with an augmented reality system, according to one embodiment. [Figure 104] FIG. 104 illustrates an exemplary totem, according to one embodiment. [Figure 105A] 105A-105C illustrate other exemplary totems according to one or more embodiments. [Figure 105B] 105A-105C illustrate other exemplary totems according to one or more embodiments. [Figure 105C] 105A-105C illustrate other exemplary totems according to one or more embodiments. [Figure 106A]106A-106C illustrate other totems that can be used to communicate with an augmented reality system. [Figure 106B] 106A-106C illustrate other totems that can be used to communicate with an augmented reality system. [Figure 106C] 106A-106C illustrate other totems that can be used to communicate with an augmented reality system. [Figure 107A] 107A-107D illustrate other exemplary totems according to one or more embodiments. [Figure 107B] 107A-107D illustrate other exemplary totems according to one or more embodiments. [Figure 107C] 107A-107D illustrate other exemplary totems according to one or more embodiments. [Figure 107D] 107A-107D illustrate other exemplary totems according to one or more embodiments. [Figure 108A] 108A-108C illustrate exemplary embodiments of ring and bracelet totems, according to one or more embodiments. [Figure 108B] 108A-108C illustrate exemplary embodiments of ring and bracelet totems, according to one or more embodiments. [Figure 108C] 108A-108C illustrate exemplary embodiments of ring and bracelet totems, according to one or more embodiments. [Figure 109A] 109A-109C illustrate further exemplary totems according to one or more embodiments. [Figure 109B] 109A-109C illustrate further exemplary totems according to one or more embodiments. [Figure 109C] 109A-109C illustrate further exemplary totems according to one or more embodiments. [Figure 110A] 110A-110B illustrate a charm totem and a keychain totem, according to one or more embodiments. [Figure 110B] 110A-110B illustrate a charm totem and a keychain totem, according to one or more embodiments. [Figure 111] FIG. 111 illustrates a high-level flow diagram for the process of determining user input through a totem, according to one embodiment. [Figure 112] FIG. 112 illustrates a high-level flow diagram for a process of generating an acoustic wave front, according to one embodiment. [Figure 113] FIG. 113 is a block diagram of components used to generate an acoustic wave front, according to one embodiment. [Figure 114] FIG. 114 is an exemplary method for determining low density and high density points according to one embodiment. [Figure 115] FIG. 115 is a block diagram of projecting textured light according to one embodiment. [Figure 116] FIG. 116 is an exemplary block diagram of data processing, according to one embodiment. [Figure 117] FIG. 117 is a schematic diagram of an eye for tracking gaze, according to one embodiment. [Figure 118] FIG. 118 shows another perspective view of an eye and one or more cameras for tracking gaze, according to one embodiment. [Figure 119] FIG. 119 shows yet another perspective view of an eye and one or more cameras for tracking gaze, according to one embodiment. [Figure 120] FIG. 120 shows yet another perspective view of an eye and one or more cameras for tracking gaze, according to one embodiment. [Figure 121] FIG. 121 illustrates a translation matrix view for tracking gaze, according to one embodiment. [Figure 122] FIG. 122 illustrates an exemplary method for tracking gaze, according to one embodiment. [Figure 123A] 123A-123D illustrate a series of exemplary user interface flows using an avatar, according to one embodiment. [Figure 123B]123A-123D illustrate a series of exemplary user interface flows using an avatar, according to one embodiment. [Figure 123C] 123A-123D illustrate a series of exemplary user interface flows using an avatar, according to one embodiment. [Figure 123D] 123A-123D illustrate a series of exemplary user interface flows using an avatar, according to one embodiment. [Figure 124A] 124A-124M illustrate a series of exemplary user interface flows using extrusion, according to one embodiment. [Figure 124B] 124A-124M illustrate a series of exemplary user interface flows using extrusion, according to one embodiment. [Figure 124C] 124A-124M illustrate a series of exemplary user interface flows using extrusion, according to one embodiment. [Figure 124D] 124A-124M illustrate a series of exemplary user interface flows using extrusion, according to one embodiment. [Figure 124E] 124A-124M illustrate a series of exemplary user interface flows using extrusion, according to one embodiment. [Figure 124F] 124A-124M illustrate a series of exemplary user interface flows using extrusion, according to one embodiment. [Figure 124G] 124A-124M illustrate a series of exemplary user interface flows using extrusion, according to one embodiment. [Figure 124H] 124A-124M illustrate a series of exemplary user interface flows using extrusion, according to one embodiment. [Figure 124I] 124A-124M illustrate a series of exemplary user interface flows using extrusion, according to one embodiment. [Figure 124J]124A-124M illustrate a series of exemplary user interface flows using extrusion, according to one embodiment. [Figure 124K] 124A-124M illustrate a series of exemplary user interface flows using extrusion, according to one embodiment. [Figure 124L] 124A-124M illustrate a series of exemplary user interface flows using extrusion, according to one embodiment. [Figure 124M] 124A-124M illustrate a series of exemplary user interface flows using extrusion, according to one embodiment. [Figure 125A] 125A-125M illustrate a series of exemplary user interface flows using gauntlets, according to one embodiment. [Figure 125B] 125A-125M illustrate a series of exemplary user interface flows using gauntlets, according to one embodiment. [Figure 125C] 125A-125M illustrate a series of exemplary user interface flows using gauntlets, according to one embodiment. [Figure 125D] 125A-125M illustrate a series of exemplary user interface flows using gauntlets, according to one embodiment. [Figure 125E] 125A-125M illustrate a series of exemplary user interface flows using gauntlets, according to one embodiment. [Figure 125F] 125A-125M illustrate a series of exemplary user interface flows using gauntlets, according to one embodiment. [Figure 125G] 125A-125M illustrate a series of exemplary user interface flows using gauntlets, according to one embodiment. [Figure 125H] 125A-125M illustrate a series of exemplary user interface flows using gauntlets, according to one embodiment. [Figure 125I]125A-125M illustrate a series of exemplary user interface flows using gauntlets, according to one embodiment. [Figure 125J] 125A-125M illustrate a series of exemplary user interface flows using gauntlets, according to one embodiment. [Figure 125K] 125A-125M illustrate a series of exemplary user interface flows using gauntlets, according to one embodiment. [Figure 125L] 125A-125M illustrate a series of exemplary user interface flows using gauntlets, according to one embodiment. [Figure 125M] 125A-125M illustrate a series of exemplary user interface flows using gauntlets, according to one embodiment. [Figure 126A] 126A-126L illustrate a series of exemplary user interface flows using growth, according to one embodiment. [Figure 126B] 126A-126L illustrate a series of exemplary user interface flows using growth, according to one embodiment. [Figure 126C] 126A-126L illustrate a series of exemplary user interface flows using growth, according to one embodiment. [Figure 126D] 126A-126L illustrate a series of exemplary user interface flows using growth, according to one embodiment. [Figure 126E] 126A-126L illustrate a series of exemplary user interface flows using growth, according to one embodiment. [Figure 126F] 126A-126L illustrate a series of exemplary user interface flows using growth, according to one embodiment. [Figure 126G] 126A-126L illustrate a series of exemplary user interface flows using growth, according to one embodiment. [Figure 126H]126A-126L illustrate a series of exemplary user interface flows using growth, according to one embodiment. [Figure 126I] 126A-126L illustrate a series of exemplary user interface flows using growth, according to one embodiment. [Figure 126J] 126A-126L illustrate a series of exemplary user interface flows using growth, according to one embodiment. [Figure 126K] 126A-126L illustrate a series of exemplary user interface flows using growth, according to one embodiment. [Figure 126L] 126A-126L illustrate a series of exemplary user interface flows using growth, according to one embodiment. [Figure 127A] 127A-127E illustrate a series of exemplary user interface flows using brushes, according to one embodiment. [Figure 127B] 127A-127E illustrate a series of exemplary user interface flows using brushes, according to one embodiment. [Figure 127C] 127A-127E illustrate a series of exemplary user interface flows using brushes, according to one embodiment. [Figure 127D] 127A-127E illustrate a series of exemplary user interface flows using brushes, according to one embodiment. [Figure 127E] 127A-127E illustrate a series of exemplary user interface flows using brushes, according to one embodiment. [Figure 128A] 128A-128P illustrate a series of exemplary user interface flows using a finger brush, according to one embodiment. [Figure 128B] 128A-128P illustrate a series of exemplary user interface flows using a finger brush, according to one embodiment. [Figure 128C]128A-128P illustrate a series of exemplary user interface flows using a finger brush, according to one embodiment. [Figure 128D] 128A-128P illustrate a series of exemplary user interface flows using a finger brush, according to one embodiment. [Figure 128E] 128A-128P illustrate a series of exemplary user interface flows using a finger brush, according to one embodiment. [Figure 128F] 128A-128P illustrate a series of exemplary user interface flows using a finger brush, according to one embodiment. [Figure 128G] 128A-128P illustrate a series of exemplary user interface flows using a finger brush, according to one embodiment. [Figure 128H] 128A-128P illustrate a series of exemplary user interface flows using a finger brush, according to one embodiment. [Figure 128I] 128A-128P illustrate a series of exemplary user interface flows using a finger brush, according to one embodiment. [Figure 128J] 128A-128P illustrate a series of exemplary user interface flows using a finger brush, according to one embodiment. [Figure 128K] 128A-128P illustrate a series of exemplary user interface flows using a finger brush, according to one embodiment. [Figure 128L] 128A-128P illustrate a series of exemplary user interface flows using a finger brush, according to one embodiment. [Figure 128M] 128A-128P illustrate a series of exemplary user interface flows using a finger brush, according to one embodiment. [Figure 128N] 128A-128P illustrate a series of exemplary user interface flows using a finger brush, according to one embodiment. [Figure 128O]128A-128P illustrate a series of exemplary user interface flows using a finger brush, according to one embodiment. [Figure 128P] 128A-128P illustrate a series of exemplary user interface flows using a finger brush, according to one embodiment. [Figure 129A] 129A-129M illustrate a series of exemplary user interface flows using pivots, according to one embodiment. [Figure 129B] 129A-129M illustrate a series of exemplary user interface flows using pivots, according to one embodiment. [Figure 129C] 129A-129M illustrate a series of exemplary user interface flows using pivots, according to one embodiment. [Figure 129D] 129A-129M illustrate a series of exemplary user interface flows using pivots, according to one embodiment. [Figure 129E] 129A-129M illustrate a series of exemplary user interface flows using pivots, according to one embodiment. [Figure 129F] 129A-129M illustrate a series of exemplary user interface flows using pivots, according to one embodiment. [Figure 129G] 129A-129M illustrate a series of exemplary user interface flows using pivots, according to one embodiment. [Figure 129H] 129A-129M illustrate a series of exemplary user interface flows using pivots, according to one embodiment. [Figure 129I] 129A-129M illustrate a series of exemplary user interface flows using pivots, according to one embodiment. [Figure 129J] 129A-129M illustrate a series of exemplary user interface flows using pivots, according to one embodiment. [Figure 129K]129A-129M illustrate a series of exemplary user interface flows using pivots, according to one embodiment. [Figure 129L] 129A-129M illustrate a series of exemplary user interface flows using pivots, according to one embodiment. [Figure 129M] 129A-129M illustrate a series of exemplary user interface flows using pivots, according to one embodiment. [Figure 130A] 130A-130I illustrate a series of exemplary user interface flows using threads, according to one embodiment. [Figure 130B] 130A-130I illustrate a series of exemplary user interface flows using threads, according to one embodiment. [Figure 130C] 130A-130I illustrate a series of exemplary user interface flows using threads, according to one embodiment. [Figure 130D] 130A-130I illustrate a series of exemplary user interface flows using threads, according to one embodiment. [Figure 130E] 130A-130I illustrate a series of exemplary user interface flows using threads, according to one embodiment. [Figure 130F] 130A-130I illustrate a series of exemplary user interface flows using threads, according to one embodiment. [Figure 130G] 130A-130I illustrate a series of exemplary user interface flows using threads, according to one embodiment. [Figure 130H] 130A-130I illustrate a series of exemplary user interface flows using threads, according to one embodiment. [Figure 130I] 130A-130I illustrate a series of exemplary user interface flows using threads, according to one embodiment. [Figure 131A]131A-131I illustrate a series of exemplary user interface flows using spider web, according to one embodiment. [Figure 131B] 131A-131I illustrate a series of exemplary user interface flows using spider web, according to one embodiment. [Figure 131C] 131A-131I illustrate a series of exemplary user interface flows using spider web, according to one embodiment. [Figure 131D] 131A-131I illustrate a series of exemplary user interface flows using spider web, according to one embodiment. [Figure 131E] 131A-131I illustrate a series of exemplary user interface flows using spider web, according to one embodiment. [Figure 131F] 131A-131I illustrate a series of exemplary user interface flows using spider web, according to one embodiment. [Figure 131G] 131A-131I illustrate a series of exemplary user interface flows using spider web, according to one embodiment. [Figure 131H] 131A-131I illustrate a series of exemplary user interface flows using spider web, according to one embodiment. [Figure 131I] 131A-131I illustrate a series of exemplary user interface flows using spider web, according to one embodiment. [Figure 132] FIG. 132 is a plan view of various mechanisms by which virtual objects relate to one or more physical objects. [Figure 133] FIG. 133 is a plan view of various types of AR renderings, according to one or more embodiments. [Figure 134] FIG. 134 illustrates various types of user input in an AR system, according to one or more embodiments. [Figure 135A]135A-135J illustrate various embodiments relating to the use of gestures in an AR system, according to one or more embodiments. [Figure 135B] 135A-135J illustrate various embodiments relating to the use of gestures in an AR system, according to one or more embodiments. [Figure 135C] 135A-135J illustrate various embodiments relating to the use of gestures in an AR system, according to one or more embodiments. [Figure 135D] 135A-135J illustrate various embodiments relating to the use of gestures in an AR system, according to one or more embodiments. [Figure 135E] 135A-135J illustrate various embodiments relating to the use of gestures in an AR system, according to one or more embodiments. [Figure 135F] 135A-135J illustrate various embodiments relating to the use of gestures in an AR system, according to one or more embodiments. [Figure 135G] 135A-135J illustrate various embodiments relating to the use of gestures in an AR system, according to one or more embodiments. [Figure 135H] 135A-135J illustrate various embodiments relating to the use of gestures in an AR system, according to one or more embodiments. [Figure 135I] 135A-135J illustrate various embodiments relating to the use of gestures in an AR system, according to one or more embodiments. [Figure 135J] 135A-135J illustrate various embodiments relating to the use of gestures in an AR system, according to one or more embodiments. [Figure 136] FIG. 136 illustrates a plan view of various components for the calibration mechanism of an AR system, in accordance with one or more embodiments. [Figure 137] FIG. 137 illustrates a view of an AR device on a user's face, where the AR device has an eye-tracking camera, in accordance with one or more embodiments. [Figure 138] FIG. 138 illustrates an eye-identifying image of an AR system in accordance with one or more embodiments. [Figure 139] FIG. 139 illustrates a retinal image captured using an AR system, according to one or more embodiments. [Figure 140] FIG. 140 is a process flow diagram of an exemplary method for generating a virtual user interface, according to one of the illustrated embodiments. [Figure 141] FIG. 141 is another process flow diagram of an exemplary method for generating a virtual user interface based on a coordinate frame, according to one of the illustrated embodiments. [Figure 142] FIG. 142 is a process flow diagram of an exemplary method for building a customized user interface, according to one of the illustrated embodiments. [Figure 143] FIG. 143 is a process flow diagram of an exemplary method for retrieving information from a passable world model and interacting with other users of an AR system according to one of the illustrated embodiments. [Figure 144] FIG. 144 is a process flow diagram of an exemplary method for retrieving information from a knowledge base in the cloud based on received input, according to one of the illustrated embodiments. [Figure 145] FIG. 145 is a process flow diagram of an exemplary method for calibrating an AR system, according to one of the illustrated embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0206] Various embodiments will now be described in detail with reference to the drawings, which are provided as illustrative examples of the present invention to enable those skilled in the art to practice the present invention. It should be noted that the following figures and examples are not intended to limit the scope of the present invention. Where certain elements of the present invention can be partially or fully implemented using known components (or methods or processes), only those parts of such known components (or methods or processes) necessary for understanding the present invention will be described, and detailed descriptions of other parts of such known components (or methods or processes) will be omitted so as not to obscure the present invention. Furthermore, various embodiments encompass present and future known equivalents of the components referenced herein by way of example.

[0207] In the foregoing specification, the invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes can be made therein without departing from the broader spirit and scope of the invention. For example, the foregoing process flows are described with reference to a particular order of process actions. However, the order of many of the described process actions may be changed without affecting the scope or operation of the invention. The specification and drawings are, therefore, to be regarded in an illustrative rather than a restrictive sense.

[0208] Disclosed are methods and systems for generating virtual and / or augmented reality. To provide a realistic and enjoyable virtual reality (VR) or augmented reality (AR) experience, virtual content can be strategically delivered to a user's eyes in a manner that respects the physiology and limitations of the human eye. The following disclosure will provide various embodiments of such optical systems that can be integrated into AR systems. While much of the disclosure herein will be discussed in the context of AR systems, it should be understood that the same techniques may also be used for VR systems, and the following embodiments should not be read as limiting.

[0209] The following disclosure will provide details regarding various types of systems in which AR users can interact with each other through the creation of maps that provide comprehensive information about real-world physical objects in real time. The maps can be advantageously considered to project virtual images that relate to known real objects. The following disclosure will provide various approaches for understanding information about the real world and using this information to provide more realistic and enjoyable AR experiences. Additionally, the disclosure will provide various user scenarios and applications in which AR systems such as those described herein can be implemented.

[0210] (System Overview) In one or more embodiments, the AR system 10 includes a computing network 5 consisting of one or more computer servers 11 connected through one or more high-bandwidth interfaces 15. The servers 11 in the computing network may or may not be co-located. Each of the one or more servers 11 includes one or more processors for executing program instructions. The servers may also include memory for storing program instructions and data used and / or generated by processes performed by the servers 11 under the direction of the program instructions.

[0211] The computing network 5 communicates data between servers 11 and between the servers and one or more user devices 12 via one or more data network connections 13. Examples of such data networks include, but are not limited to, all types of public and private data networks, both mobile and wired, including, for example, the interconnection of many such networks commonly referred to as the Internet. No particular medium, topology, or protocol is intended to be implied by the diagram.

[0212] User devices are configured to communicate directly with either the computing network 5 or the server 11. Alternatively, user devices 12 communicate locally with the remote server 11, and optionally with other user devices, through a specially programmed local gateway 14 for processing and / or communicating data between the network 5 and one or more local user devices 12.

[0213] As shown, gateway 14 is implemented as a separate hardware component including a processor for executing software instructions and memory for storing software instructions and data. The gateway has its own wired and / or wireless connection to a data network for communicating with server 11 comprising computing network 5. Alternatively, gateway 14 can be integrated with user device 12 worn or carried by a user. For example, gateway 14 may be implemented as a downloadable software application that is installed and runs on a processor included within user device 12. Gateway 14, in one embodiment, provides one or more users access to computing network 5 via data network 13.

[0214] Each of the servers 11 includes, for example, working memory and storage for storing data and software programs, a microprocessor for executing program instructions, and graphics processors and other specialized processors for rendering and generating graphics, images, video, audio, and multimedia files. The computing network 5 may also include devices for storing data that is accessed, used, or created by the servers 11.

[0215] Software programs running on the server and, optionally, on the user device 12 and gateway 14 are used to generate a digital world (also referred to herein as a virtual world) in which a user interacts with the user device 12. The digital world (or map) (as will be described in more detail below) is represented by data and processes that describe and / or define virtual, non-existent entities, environments, and conditions that may be presented to a user through the user device 12 for the user to experience and interact with. For example, some types of objects, entities, or items that would appear to be physically present when instantiated within a scene viewed or experienced by a user may include descriptions of their appearance, their behavior, how a user is allowed to interact with them, and other characteristics.

[0216] Data used to create the environment of a virtual world (including virtual objects) may include, for example, atmospheric data, terrain data, weather data, temperature data, location data, and other data used to define and / or describe the virtual environment. In addition, data defining various conditions governing the operation of the virtual world may include, for example, laws of physics, time, spatial relationships, and other data that may be used to define and / or create various conditions governing the operation of the virtual world (including virtual objects).

[0217] Entities, objects, conditions, properties, behaviors, or other features of the digital world will be generally referred to herein as objects (e.g., digital objects, virtual objects, rendered physical objects, etc.) unless otherwise indicated by context. Objects may be any type of animate or inanimate object, including, but not limited to, buildings, plants, vehicles, people, animals, creations, machines, data, video, text, photographs, and other users. Objects may also be defined within the digital world to store information about items, behaviors, or conditions that actually exist in the physical world. Data that describes or defines an entity, object, or item, or that stores its current state, is generally referred to herein as object data. This data is processed by the server 11, or, depending on the implementation, by the gateway 14 or user device 12, to instantiate instances of the object and render the object in an appropriate manner for the user to experience through the user device.

[0218] Programmers who develop and / or oversee the digital world create or define objects and the conditions under which they are instantiated. However, the digital world may allow others to create or modify objects. Once an object is instantiated, the state of the object may be permitted to be altered, controlled, or manipulated by one or more users who experience the digital world.

[0219] For example, in one embodiment, the development, creation, and management of the digital world is generally provided by one or more system administration programmers. In some embodiments, this may include the development, design, and / or execution of storylines, themes, and events within the digital world, as well as the distribution of the stories through various forms of events and media, such as, for example, film, digital, network, mobile, augmented reality, and live entertainment. System administration programmers may also handle the technical management, coordination, and oversight of the digital world and its associated user community, as well as other tasks typically performed by network administrators.

[0220] Users interact with one or more digital worlds using some type of local computing device, generally designated as user device 12. Examples of such user devices include, but are not limited to, smartphones, tablet devices, head-mounted displays (HMDs), gaming consoles, or any other device capable of communicating data and providing an interface or display to a user, as well as combinations of such devices. In some embodiments, user device 12 may include or communicate with local peripherals or input / output components, such as, for example, a keyboard, a mouse, a joystick, a game controller, a tactile interface device, a motion capture controller, an optical tracking device, audio equipment, voice equipment, a projector system, a 3D display, and / or holographic 3D contact lenses.

[0221] An example of a user device 12 for interacting with system 10 is illustrated in Figure 2. In the exemplary embodiment shown in Figure 2, a user 21 may interface with one or more digital worlds through a smartphone 22. The gateway is implemented by a software application 23 stored on and running on the smartphone 22. In this particular example, the data network 13 includes a wireless mobile network that connects the user device (e.g., smartphone 22) to the computer network 5.

[0222] In one implementation of a preferred embodiment, the system 10 is capable of supporting a large number of concurrent users (e.g., millions of users), each using several types of user devices 12 to interface with the same digital world or multiple digital worlds.

[0223] The user device provides the user with an interface to enable visual, auditory, and / or physical interaction between the user and the digital world generated by the server 11, including other users and objects (real or virtual) presented to the user. The interface provides the user with a rendered scene that can be seen, heard, or otherwise sensed, and the ability to interact with the scene in real time. The manner in which the user interacts with the rendered scene may be determined by the capabilities of the user device. For example, if the user device is a smartphone, user interaction may be implemented by the user touching a touchscreen. In another example, if the user device is a computer or game console, user interaction may be implemented using a keyboard or game controller. The user device may include additional components that enable user interaction, such as sensors, and objects and information (including gestures) detected by the sensors may be provided as input representing the user's interaction with the virtual world using the user device.

[0224] The rendered scene can be presented in a variety of formats, such as, for example, two-dimensional or three-dimensional visual displays (including projections), sound, and tactile or haptic feedback. The rendered scene may be interfaced with by the user in one or more modes, including, for example, augmented reality, virtual reality, and combinations thereof. The format of the rendered scene and the interface mode may be determined by one or more of the user device, data processing power, user device connectivity, network capacity, and system workload. The simultaneous interaction of multiple users with the digital world and the real-time nature of the data exchange are made possible by the computing network 5, server 11, gateway component 14 (optional), and user devices 12.

[0225] In one example, computing network 5 consists of a large-scale computing system having single and / or multi-core servers (e.g., server 11) connected through high-speed connections (e.g., high-bandwidth interface 15). Computing network 5 may form a cloud or grid network. Each of the servers includes memory or is coupled with computer-readable memory for storing software to implement data for creating, designing, modifying, or manipulating objects in the digital world. These objects and their instantiations may be dynamic, appearing and disappearing, changing over time, and changing in response to other conditions. Examples of dynamic capabilities of objects are generally discussed herein with respect to various embodiments. In some embodiments, each user interfacing with system 10 may also be represented within one or more digital worlds as an object and / or collection of objects.

[0226] Servers 11 in computing network 5 also store computational state data for each digital world. Computational state data (also referred to herein as state data) may be a component of object data and generally defines the state of an instance of an object at a given instance of time. Thus, computational state data may change over time and may be affected by the actions of one or more users and / or programmers who maintain system 10. As users affect computational state data (or other data that comprises the digital world), they directly alter or otherwise manipulate the digital world. If the digital world is shared with or interfaced by other users, the user's actions may affect what is experienced by other users who interact with the digital world. Thus, in some embodiments, changes to the digital world made by a user will also be experienced by other users who interface with system 10.

[0227] Data stored on one or more servers 11 in the computing network 5 is transmitted or deployed, in one embodiment, with high speed and low latency, to one or more user devices 12 and / or gateway components 14. In one embodiment, the object data shared by the server may be complete or may be compressed and include instructions for recreating the complete object data at the user's end, rendered and visualized by the user's local computing device (e.g., gateway 14 and / or user device 12). Software running on the servers 11 of the computing network 5 may, in some embodiments, adapt the data it generates and sends to a particular user's device 12 for objects (or any other data) in the digital world exchanged by the computing network 5 as a function of the user's specific device and bandwidth.

[0228] For example, as a user interacts with the digital world or map through user device 12, server 11 may recognize the specific type of device being used by the user, the device's connectivity, and / or the available bandwidth between the user device and the server, and appropriately size and balance the data being sent to the device to optimize user interaction. An example of this may include reducing the size of transmitted data to low-resolution quality so that the data can be displayed on a particular user device with a low-resolution display. In preferred embodiments, computing network 5 and / or gateway component 14 deliver data to user device 12 at a rate sufficient to present an interface operating at or above 15 frames per second and at a resolution that is high-definition quality.

[0229] The gateway 14 provides local connectivity to the computing network 5 for one or more users. In some embodiments, it may be implemented by a downloadable software application running on the user device 12 or another local device, such as that shown in FIG. 2 . In other embodiments, it may be implemented by a hardware component (with appropriate software / firmware stored on the component, the component having a processor) that communicates with, but is either not integrated with, or not incorporated into, the user device 12. The gateway 14 communicates with the computing network 5 via the data network 13 and provides data exchange between the computing network 5 and one or more local user devices 12. As discussed in more detail below, the gateway component 14 may include software, firmware, memory, and processing circuitry and may be capable of processing data communicated between the network 5 and one or more local user devices 12.

[0230] In some embodiments, the gateway component 14 monitors and adjusts the rate of data exchanged between the user device 12 and the computer network 5, enabling optimal data throughput for a particular user device 12. For example, in some embodiments, the gateway 14 buffers and downloads both static and dynamic aspects of the digital world (even beyond the field of view presented to the user through the interface connected to the user device). In such embodiments, instances of static objects (structured data, software-implemented methods, or both) may be stored in memory (local to the gateway component 14, the user device 12, or both) and are referenced to the local user's current location, as indicated by data provided by the computing network 5 and / or the user's device 12.

[0231] Instances of dynamic objects, which may include, for example, intelligent software agents and objects controlled by other users and / or the local user, are stored in high-speed memory buffers. Dynamic objects, which represent two-dimensional or three-dimensional objects in a scene presented to a user, can be divided into component shapes, such as static shapes that move but do not change and dynamic shapes that change. The changing portions of dynamic objects can be updated by a real-time, threaded, high-priority data stream from server 11 through computing network 5 managed by gateway component 14.

[0232] As an example of prioritized threaded data streams, data within the 60-degree field of view of the user's eye may be given higher priority than data that is more peripheral. Another example is prioritizing dynamic characters and / or objects within the user's field of view over static objects in the background.

[0233] In addition to managing the data connection between the computing network 5 and the user device 12, the gateway component 14 may store and / or process data that may be presented to the user device 12. For example, the gateway component 14 may, in some embodiments, receive compressed data from the computing network 5 that describes graphical objects to be rendered for viewing by a user, and perform advanced rendering techniques to reduce the data load transmitted from the computing network 5 to the user device 12. In another example where the gateway 14 is a separate device, the gateway 14 may store and / or process data for local instances of objects rather than transmitting the data to the computing network 5 for processing.

[0234] 3, the virtual world may be experienced by one or more users in a variety of formats, which may depend on the capabilities of the user's device. In some embodiments, the user device 12 may include, for example, a smartphone, a tablet device, a head-mounted display (HMD), a gaming console, or a wearable device. Generally, the user device will include a processor for executing program code stored in memory on the device, coupled with a display, and a communications interface.

[0235] An exemplary embodiment of a user device is illustrated in Figure 3, which includes a mobile wearable device, i.e., head-mounted display system 30. According to an embodiment of the present disclosure, head-mounted display system 30 includes a user interface 37, a user sensing system 34, an environmental sensing system 36, and a processor 38. Although processor 38 is shown in Figure 3 as a separate and isolated component from head-mounted system 30, in alternative embodiments, processor 38 may be integrated with one or more components of head-mounted system 30 or may be integrated within other system 10 components, such as, for example, gateway 14, as shown in Figures 1 and 2.

[0236] User device 30 presents the user with an interface 37 for interacting with and experiencing the digital world. Such interaction may involve the user and the digital world, one or more other users interfacing with system 10, and objects within the digital world. Interface 37 generally provides visual and / or audio sensory input (and in some embodiments, physical sensory input) to the user. Accordingly, interface 37 may include speakers (not shown) and, in some embodiments, a display component 33 capable of providing stereoscopic 3D viewing and / or 3D viewing that embodies more natural characteristics of the human visual system.

[0237] In some embodiments, display component 33 may comprise a transparent interface (such as a clear OLED) that, when in an "off" setting, allows an optically correct view of the user's surrounding physical environment with little or no optical distortion or computing overlay. As discussed in more detail below, interface 37 may include additional settings that allow for various visual / interface capabilities and functionality.

[0238] User sensing system 34, in some embodiments, may include one or more sensors 31 operable to detect certain characteristics, properties, or information associated with an individual user wearing system 30. For example, in some embodiments, sensor 31 may include a camera or optical detection / scanning circuitry capable of detecting real-time optical properties / measurements of the user.

[0239] The user's real-time optical characteristics / measurements may be, for example, one or more of pupil constriction / dilation, angulation / positioning of each pupil, sphericity, eye shape (as eye shape changes over time), and other anatomical data. This data may be used to provide or calculate information (e.g., the user's visual focus) that can be used by head-mounted system 30 and / or interface system 10 to optimize the user's viewing experience. For example, in one embodiment, each of sensors 31 may measure the pupil constriction rate for each of the user's eyes. This data may be transmitted to processor 38 (or gateway component 14 or server 11), where it is used, for example, to determine the user's response to brightness settings of interface display 33.

[0240] The interface 37 may be adjusted according to the user's response, for example, by dimming the display 33 if the user's response indicates that the brightness level of the display 33 is too high. The user sensing system 34 may include other components other than those described above or illustrated in FIG. 3 . For example, in some embodiments, the user sensing system 34 may include a microphone for receiving audio input from the user. The user sensing system 34 may also include one or more infrared camera sensors, one or more visible spectrum camera sensors, structured light emitters and / or sensors, infrared light emitters, coherent light emitters and / or sensors, gyroscopes, accelerometers, magnetometers, proximity sensors, GPS sensors, ultrasonic emitters and detectors, and tactile interfaces.

[0241] The environmental sensing system 36 includes one or more sensors 32 for obtaining data from the physical environment surrounding the user. Objects or information detected by the sensors may be provided as input to a user device. In some embodiments, this input may represent user interaction with a virtual world. For example, a user viewing a virtual keyboard on a desk may gesture with their fingers as if they were typing on the virtual keyboard. Finger movement motions may be captured by the sensors 32 and provided as input to a user device or system, which may be used to modify the virtual world or create new virtual objects.

[0242] For example, finger movements may be recognized as typing (e.g., using a software program on a processor, etc.), and the recognized typing gestures may be combined with known locations of virtual keys on a virtual keyboard. The system may then render a virtual monitor that is displayed to the user (or other user interfacing with the system), where the virtual monitor displays the text being typed by the user.

[0243] The sensors 32 may include, for example, generally outward-facing cameras or scanners to interpret scene information, for example, through continuously and / or intermittently projected infrared structured light. The environmental sensing system (36) may be used to map one or more elements of the user's surrounding physical environment by detecting and locating the local environment, including static objects, dynamic objects, people, gestures, and various lighting, atmospheric, and acoustic conditions. Thus, in some embodiments, the environmental sensing system (36) may include image-based 3D reconstruction software embedded within a local computing system (e.g., gateway component 14 or processor 38) and operable to digitally reconstruct one or more objects or information detected by the sensors 32.

[0244] In one exemplary embodiment, the environmental sensing system 36 provides one or more of motion capture data (including gesture recognition), depth sensing, face recognition, object recognition, unique object feature recognition, voice / audio recognition and processing, acoustic source localization, noise reduction, infrared or similar laser projection, and monochrome and / or color CMOS sensors (or other similar sensors), field of view sensors, and various other optically enhanced sensors.

[0245] It should be understood that the environmental sensing system 36 may include other components than those described above or illustrated in Figure 3. For example, in some embodiments, the environmental sensing system 36 may include a microphone for receiving audio from the local environment. The user sensing system (36) may also include one or more infrared camera sensors, one or more visible spectrum camera sensors, structured light emitters and / or sensors, infrared light emitters, coherent light emitters and / or sensors, gyroscopes, accelerometers, magnetometers, proximity sensors, GPS sensors, ultrasonic emitters and detectors, and tactile interfaces.

[0246] As mentioned above, processor 38 may, in some embodiments, be integrated with other components of head-mounted system 30, integrated with other components of interface system 10, or may be a separate device (wearable or separate from the user) as shown in Figure 3. Processor 38 may be connected to various components of head-mounted system 30 and / or components of interface system 10 through a physical wired connection or through a wireless connection such as, for example, a mobile network connection (including cellular and data networks), Wi-Fi, or Bluetooth®.

[0247] In one or more embodiments, the processor 38 may include a memory module, an integrated and / or additional graphics processing unit, a wireless and / or wired Internet connection, and a codec and / or firmware capable of converting data from a source (e.g., the computing network 5, the user sensing system 34, the environmental sensing system 36, or the gateway component 14) into image and audio data, which may be presented to the user via the interface 37.

[0248] In one or more embodiments, processor 38 handles data processing for the various components of head-mounted system 30 and data exchange between head-mounted system 30 and gateway component 14, and in some embodiments, computing network 5. For example, processor 38 may be used to buffer and process data streams between a user and computing network 5, thereby enabling a smooth, continuous, and high-fidelity user experience.

[0249] In some embodiments, processor 38 may process data at rates sufficient to achieve anything from 8 frames per second at 320x240 resolution to 24 frames per second at high definition resolution (1280x720), or 60 to 120 frames per second at 4k resolution and even higher (50,000 frames per second at 10k+ resolution), etc. Additionally, processor 38 may store and / or process data that may be presented to a user rather than being streamed in real time from computing network 5.

[0250] For example, processor 38 may, in some embodiments, receive compressed data from computing network 5 and perform advanced rendering techniques (such as lighting or shading) to reduce the data load transmitted from computing network 5 to user device 12. In another example, processor 38 may store and / or process local object data rather than transmitting the data to gateway component 14 or computing network 5.

[0251] The head-mounted system 30, in some embodiments, may include various settings, or modes, to enable different visual / interface capabilities and functionality. The modes may be selected manually by the user or automatically by components of the head-mounted system 30 or gateway component 14. As previously mentioned, one exemplary mode of the head-mounted system 30 includes an “off” mode, in which the interface 37 provides substantially no digital or virtual content. In the “off” mode, the display component 33 may be transparent, thereby enabling an optically correct view of the physical environment around the user with little or no optical distortion or computing overlay.

[0252] In one exemplary embodiment, head-mounted system 30 includes an "augmented" mode, in which interface 37 provides an augmented reality interface. In the augmented mode, interface display 33 may be substantially transparent, thereby allowing the user to view the local physical environment. Simultaneously, virtual object data provided by computing network 5, processor 38, and / or gateway component 14 is provided on display 33 in combination with the local physical environment. The following sections will discuss various embodiments of exemplary head-mounted user systems that can be used for virtual and augmented reality purposes.

[0253] (user system) 4A-4D, some general component options are illustrated. In the detailed description portion that follows the discussion of Figures 4A-4D, various systems, subsystems, and components are presented to address the objective of providing a high-quality, comfortably perceived display system for human VR and / or AR.

[0254] 4A, a user 60 of a head-mounted augmented reality system ("AR system") is depicted wearing a frame 64 structure coupled to a display system 62 positioned directly in front of the user's eyes. A speaker 66, in the depicted configuration, is coupled to frame 64 and positioned adjacent to the user's 60 ear canal (in one embodiment, another speaker, not shown, is positioned adjacent the user's other ear canal to provide stereo / shapeable sound control). The display 62 is operably coupled 68 to a local processing and data module 70, which may be mounted in a variety of configurations, such as fixedly attached to the frame 64, by wired or wireless connection, fixedly attached to a helmet or hat 80 as shown in the embodiment of FIG. 4B, embedded within headphones, removably attached to the torso 82 of the user 60 in a configuration (e.g., placed in a backpack (not shown)) as shown in the embodiment of FIG. 4C, or removably attached to the waist 84 of the user 60 in a belt-coupled configuration as shown in the embodiment of FIG. 4D.

[0255] The local processing and data module 70 may include a power-saving processor or controller and digital memory, such as flash memory, both of which may be utilized to assist in processing, caching, and storing data (a) captured from sensors, such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes, which may be operatively coupled to the frame 64, and / or (b) acquired and / or processed using the remote processing module 72 and / or remote data repository 74, possibly for processing or retrieval and subsequent passage to the display 62.

[0256] The local processing and data module 70 may be operatively coupled to a remote processing module 72 and a remote data repository 74 (76, 78), such as via a wired or wireless communication link, such that these remote modules (72, 74) are operatively coupled to each other and available as resources to the local processing and data module 70. The processing module 70 may perform one or more computing tasks, including retrieving data from memory or one or more databases (e.g., cloud-based servers), to control the optical and other systems of the AR system and to provide virtual content to the user.

[0257] In one embodiment, remote processing module 72 may comprise one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. In one embodiment, remote data repository 74 may comprise a relatively large digital data storage facility that may be available over the Internet or other networking configuration in a "cloud" resource configuration. In one embodiment, all data is stored and all calculations are performed within the local processing and data module, allowing for fully autonomous use from any remote module.

[0258] (Optical embodiment) It should be understood that there are many approaches to presenting 3D virtual content to a user's eyes through the optical elements of a head-mounted user device. The following exemplary embodiments may be used in combination with other approaches and should not be read in a limiting sense. The following exemplary embodiments represent some example optical systems that may be integrated with a head-mounted user device (30) to enable a user to view virtual content in a comfortable, accommodative manner.

[0259] Referring to Figures 5A to 22Y, various display configurations are presented that are designed to present photon-based radiation patterns to the human eye that can be comfortably perceived as an extension to physical reality, with a high level of image quality and three-dimensional perception, and that are also capable of presenting two-dimensional content.

[0260] 5A, in a simplified example, a transmissive beam splitter substrate 104 with a 45-degree reflective surface 102 directs incident radiation 106, which may be output from a lens (not shown) through the pupil 45 of the eye 58 and onto the retina 54. The field of view for such a system is limited by the geometry of the beam splitter 104. To accommodate the desire for comfortable viewing with minimal hardware, in one embodiment, a larger field of view may be created by aggregating the output / reflections of various different reflective and / or diffractive surfaces. This may be achieved, for example, using a frame sequential configuration, in which the eye 58 is presented with a sequence of frames at a high frequency, providing the perception of a single coherent scene.

[0261] As an alternative to, or in addition to, presenting different image data in a time-sequential manner through different reflectors, the reflectors may separate content by other means, such as polarization selectivity or wavelength selectivity. In addition to being able to relay two-dimensional images, the reflectors may also relay three-dimensional wavefronts associated with true three-dimensional viewing of actual physical objects.

[0262] 5B, a substrate 108 is shown comprising multiple reflectors at multiple angles 110, each of which is actively reflective in the depicted configuration for illustrative purposes. The reflectors may comprise switchable elements to facilitate temporal selectivity. In one embodiment, the reflective surfaces may be intentionally and sequentially activated with frame-sequential input information 106, with each reflective surface presenting a narrow field-of-view sub-image that is tiled with other narrow field-of-view sub-images presented by other reflective surfaces to form a composite wide field-of-view image.

[0263] For example, referring to Figures 5C, 5D, and 5E, surface 110 (e.g., at the center of substrate 108) is switched to a reflective state "on" to reflect incident image information 106 and present a relatively narrow field of view sub-image in the center of a larger field of view, while other potentially reflective surfaces are in a transmissive state.

[0264] Referring to Figure 5C, incident image information 106 originating from the right side of the narrow FOV sub-image (as indicated by the angle of the incident beam 106 relative to the substrate 108 at the input interface 112 and the angle at which it exits the substrate 108) is reflected from the reflective surface 110 toward the eye 58. Figure 5D illustrates the same reflector 110 is active, with image information originating from the center of the narrow FOV sub-image, as indicated by the angle of the input information 106 at the input interface 112 and its angle as it exits the substrate 108.

[0265] Figure 5E illustrates the same reflector 110 being active, with image information originating from the left side of the field of view, as indicated by the angle of the input information 106 at the input interface 112 and the resulting exit angle at the surface of the substrate 108. Figure 5F illustrates a configuration in which the bottom reflector 110 is active, with image information 106 originating from the right-most side of the overall field of view. For example, Figures 5C, 5D, and 5E may illustrate one frame representing the center of a frame sequential tiled display image, and Figure 5F may illustrate a second frame representing the right-most side of that tiled display image.

[0266] In one embodiment, light carrying image information 106 may not first reflect off a surface of substrate 108 but may directly enter substrate 108 at input interface 112 before striking reflective surface 110. In one embodiment, light carrying image information 106 may reflect off one or more surfaces of substrate 108 after entering input interface 112 before striking reflective surface 110. For example, substrate 108 may act as a planar waveguide, propagating light carrying image information 106 by total internal reflection. Light may also reflect off one or more surfaces of substrate 108 through a partially reflective coating, a wavelength-selective coating, an angle-selective coating, and / or a polarization-selective coating.

[0267] In one embodiment, angled reflectors may be constructed using electro-active materials such that, in response to the application of a voltage and / or current to a particular reflector, the refractive index of the material comprising such reflector changes from a refractive index that is substantially matched to that of the rest of the substrate 108. When the refractive index of the reflector matches that of the rest of the substrate 108, the reflector is in a transmissive configuration. When the refractive index of the reflector is mismatched to that of the rest of the substrate 108, the reflector is in a reflective configuration such that a reflective effect is created. Exemplary electro-active materials include lithium niobate and electroactive polymers. Suitable substantially transparent electrodes for controlling multiple such reflectors may include materials such as indium tin oxide, as utilized in liquid crystal displays.

[0268] In one embodiment, the electro-active reflector 110 may comprise a liquid crystal embedded in a host medium of the substrate 108, such as glass or plastic. In some variations, a liquid crystal may be selected that changes its refractive index as a function of an applied electrical signal, so that a more analog transition, as opposed to a binary transition (from one transmissive state to one reflective state), can be achieved. In one embodiment, six sub-images are presented to the eye in a frame-sequential manner to form a large-scale tiled display image with an overall refresh rate of 60 frames per second, and it is desirable to have an electro-active reflector array that can sustain such a frequency, along with an input display that can refresh at a rate of approximately 360 Hz.

[0269] In one embodiment, lithium niobate may be utilized as the electroactive reflective material, as opposed to liquid crystal. Lithium niobate is utilized in the photonics industry for high-speed switches and fiber optic networks, and has the ability to switch its refractive index at very high frequencies in response to an applied voltage. This high frequency may be used to steer line-sequential or pixel-sequential sub-image information, particularly when the input display is a scanned light display, such as a fiber scanning display or a scanning mirror-based display.

[0270] In another embodiment, the variably switchable angled mirror configuration may comprise one or more high-speed mechanically repositionable reflective surfaces, such as a MEMS (microelectromechanical systems) device. The MEMS device may include what is known as a "digital mirror device," or "DMD," (often part of a "digital light processing," or "DLP," system, such as those available from Texas Instruments, Inc.). In another electromechanical embodiment, multiple air-separated (or vacuum-enclosed) reflective surfaces may be mechanically moved in and out of position at high frequencies. In another electromechanical embodiment, a single reflective surface may be moved up and down and re-pitched at very high frequencies.

[0271] 5G , it should be noted that the switchable variable angle reflector configurations described herein are capable of passing not only collimated or flat wavefront information to the retina 54 of the eye 58, but also image information with a curved wavefront 122, as shown in the illustration of FIG. 5G . This is generally not the case with other waveguide-based configurations, where total internal reflection of curved wavefront information creates undesirable complications, and therefore the input must generally be collimated. The ability to pass curved wavefront information facilitates the ability of configurations such as those shown in FIGS. 5B-5H to provide input to the retina 54 that is perceived as focused at various distances from the eye 58, rather than just at optical infinity (which would be the interpretation of collimated light in the absence of other cues).

[0272] 5H, in another embodiment, an array of static partially reflective surfaces 116 (e.g., always in a reflective mode; in another embodiment, they may be electro-active, as described above) may be embedded within substrate 114 along with a high-frequency gating layer 118 that controls output to eye 58. High-frequency gating layer 118 may only allow transmission through apertures 120, which are controllably movable. In other words, all but transmission through apertures 120 may be selectively blocked. Gating layer 118 may comprise a liquid crystal array, a lithium niobate array, an array of MEMS shutter elements, an array of DLP DMD elements, or an array of other MEMS devices configured to pass or transmit with relatively high frequency switching and high transmission in response to being switched into a transmission mode.

[0273] 6A-6B , another embodiment is depicted in which an array of optical elements is combined with an expanded exit pupil configuration to aid in the comfort of a user's virtual or augmented reality experience. The larger the “exit pupil” for the optical configuration, the less disruptive the positioning of the user's eyes relative to the display (which may be attached to the user's head in a glasses-like configuration, as in FIGS. 4A-4D ) is to the experience, since, due to the system's larger exit pupil, there is a larger acceptable area over which the user's anatomical pupil may still be positioned to receive information from the display system as desired. In other words, the larger the exit pupil, the less sensitive the system is to slight misalignment of the display relative to the user's anatomical pupil, and greater comfort for the user may be achieved due to fewer geometric constraints on its relationship with the display / glasses.

[0274] 6A and 6B, an alternative approach is illustrated. As shown in FIG. 6A, the display 140 on the left provides a collection of parallel light beams into the substrate 124. In one embodiment, the display may be a scanning fiber display that scans a narrow beam of light back and forth at an angle as shown and projects an image through a lens or other optical element 142 that may be utilized to collect the angle-scanned light and convert it into a parallel bundle of light beams. The light beams may be reflected from a series of reflective surfaces (126, 128, 130, 132, 134, 136) that may partially reflect and partially transmit the incident light so that the light may be shared approximately equally across the reflective surfaces (126, 128, 130, 132, 134, 136). A small lens 138 may be placed at each exit point from the waveguide 124, and the exiting light beam may be steered through the nodal points and scanned towards the eye 58, providing an array of exit pupils or the functional equivalent of one large exit pupil usable by the user when gazing at the display system.

[0275] For virtual reality configurations where it is desirable to also be able to see through the waveguide to the real world 144, a similar set of lenses 139 may be presented on the opposite side of the waveguide 124 to compensate for a subset of the lenses, thus creating the equivalent of a zero-magnification telescope. Each of the reflective surfaces (126, 128, 130, 132, 134, 136) may be aligned at approximately 45 degrees, as shown, or may be configured to have a different alignment (e.g., similar to the configurations of Figures 5B-5H). The reflective surfaces (126, 128, 130, 132, 134, 136) may comprise wavelength-selective reflectors, band-pass reflectors, semi-transparent mirrors, or other reflective configurations. The lenses (138, 139) shown are refractive lenses, but diffractive lens elements may also be utilized.

[0276] Referring to Figure 6B, a somewhat similar configuration is depicted in which multiple curved reflective surfaces (148, 150, 152, 154, 156, 158) may be utilized to effectively combine the functionality of the lens (element 138 in Figure 6A) and reflector (elements 126, 128, 130, 132, 134, 136 in Figure 6A) of the embodiment of Figure 6A, thereby eliminating the need for two lens groups (element 138 in Figure 6A).

[0277] The curved reflective surfaces (148, 150, 152, 154, 156, 158) may be of various curved configurations selected to both reflect and impart angular variation, such as parabolic or elliptical curved surfaces. In a parabolic configuration, a set of collimated incident light rays will be collected into a single output point. In an elliptical configuration, a set of diverging light rays from a single origin point will be collected into a single output point. Similar to the configuration of FIG. 6A, the curved reflective surfaces (148, 150, 152, 154, 156, 158) preferably partially reflect and partially transmit incident light so that it is shared across the length of the waveguide 146. The curved reflective surfaces (148, 150, 152, 154, 156, 158) may comprise wavelength-selective notch reflectors, semi-transparent mirrors, or other reflective configurations. In another embodiment, the curved reflective surfaces (148, 150, 152, 154, 156, 158) may be replaced with diffractive reflectors that are both reflective and polarizing.

[0278] 7A, the perception of Z-axis difference (e.g., linear distance from the eye along the optical axis) can be enhanced by using a waveguide in conjunction with a variable-focus optics configuration. As shown in FIG. 7A, image information from display 160 may be collimated, injected into waveguide 164, and distributed in a large-exit-pupil manner, for example, using a configuration such as that described with reference to FIGS. 6A and 6B, or other substrate-induced optics methods known to those skilled in the art; variable-focus optics capabilities may then be utilized to change the focus of the wavefront of the light emerging from the waveguide, providing the eye with the perception that the light emanating from waveguide 164 is from a particular focal distance.

[0279] In other words, the incoming light is collimated to avoid problems in total internal reflection waveguide configurations, and therefore exits in a collimated manner, requiring the viewer's eye to accommodate so that the far point focuses on the retina, which will necessarily be interpreted as being from optical infinity unless some other intervention causes the light to be refocused and perceived as being from a different viewing distance. One suitable such intervention is a variable focus lens.

[0280] In the embodiment of FIG. 7A , collimated image information from display 160 is launched into a piece of glass 162 or other material at an angle such that it undergoes total internal reflection and is passed into an adjacent waveguide 164. Waveguide 164 may be configured similarly to the waveguides of FIGS. 6A or 6B (124, 146, respectively) so that collimated light from the display is distributed to exit approximately uniformly across a distribution of reflectors or diffractive features along the length of the waveguide. In the depicted configuration, depending on the exit light toward eye 58, the exiting light is passed through variable-focus lens element 166, and depending on the controlled focus of variable-focus lens element 166, the light exiting variable-focus lens element 166 and entering eye 58 will have various levels of focus (a collimated, flat wavefront represents optical infinity, while greater beam divergence / wavefront curvature represents closer viewing distances relative to eye 58).

[0281] To compensate for the variable focus lens element 166 between the eye 58 and the waveguide 164, another similar variable focus lens element 167 is placed on the opposite side of the waveguide 164 to cancel out the optical effect of the lens 166 on the light emanating from the world 144 for augmented reality (e.g., one lens compensates for the other, creating the functional equivalent of a zero magnification telescope, as described above).

[0282] The variable focus lens element 166 may be a refractive element such as a liquid crystal lens, an electro-active lens, a conventional refractive lens with a movable element, a mechanical deformation-based lens (such as a fluid-filled membrane lens or a lens similar to the human lens in which a flexible element is flexed and relaxed by an actuator), an electrowetting lens, or multiple fluids with different refractive indices.

[0283] The variable focus lens element 166 may also comprise a switchable diffractive optical element (such as one featuring a polymer dispersed liquid crystal approach in which a host medium such as a polymer material has microdroplets of liquid crystal dispersed within the material, whereby when a voltage is applied, the molecules reorient so that their refractive index no longer matches that of the host medium, thereby creating a high frequency switchable diffraction pattern).

[0284] One embodiment includes a host medium in which microdroplets of a Kerr effect-based electro-active material, such as lithium niobate, are dispersed and, when coupled with a scanning light display, such as a fiber scanning display or a scanning mirror-based display, allow for refocusing of image information on a pixel-by-pixel or row-by-row basis. In variable focus lens element 166 configurations where liquid crystal, lithium niobate, or other technologies are utilized to represent the pattern, the pattern spacing can be modulated not only to change the focusing power of the variable focus lens element 166, but also to change the focusing power of the overall optical system for zoom lens-type functionality.

[0285] In one embodiment, lens 166 may be telecentric in that the focus of the display image can be altered while keeping the magnification constant, in the same way that a zoom lens for photography can be configured to decouple focus from zoom position. In another embodiment, lens 166 may be non-telecentric, so that focus changes will also follow zoom changes. In such a configuration, such magnification changes may be compensated for in software using dynamic scaling of the output from the graphics system that is synchronized with the focus change.

[0286] Referring back to the projector or other video display unit 160 in a "frame sequential" configuration and the issue of how images are fed into an optical display system, a stack of sequential two-dimensional images may be sequentially fed to the display in a manner similar to a computed tomography system, which uses stacked image slices to represent three-dimensional structures, creating a three-dimensional perception over time.

[0287] A series of two-dimensional image slices may be presented to the eye, each at a different focal distance relative to the eye, and the eye / brain will integrate such stacks into the perception of a coherent three-dimensional volume. Depending on the display type, row-by-row or even pixel-by-pixel sequential processing may be performed to create the perception of three-dimensional viewing. For example, in a scanning optical display (such as a scanning fiber display or a scanning mirror display), the display presents one line or one pixel at a time to the waveguide 164 in a sequential manner.

[0288] If the variable-focus lens element 166 is capable of supporting high-frequency pixel-by-pixel or row-by-row presentation, each line or pixel can be presented through the variable-focus lens element 166, dynamically focused, and perceived at a different focal distance from the eye 58. Per-pixel focus modulation generally requires an ultra-fast / high-frequency variable-focus lens element 166. For example, a 1080P resolution display with an overall frame rate of 60 frames per second typically represents approximately 125 million pixels per second. Such configurations may also be built using solid-state switchable lenses, such as those using electro-active materials, e.g., lithium niobate or electro-active polymers. In addition to its compatibility with the system illustrated in FIG. 7A, the frame sequential multifocal display driving approach may be used in conjunction with several display systems and optical embodiments described in this disclosure.

[0289] 7B, an electro-active layer 172 (such as comprising liquid crystal or lithium niobate) may be surrounded by functional electrodes (170, 174) (which may be made from indium tin oxide) and a waveguide 168 with a conventional transparent substrate 176. The waveguide, in one or more embodiments, may be made from glass or plastic with known total internal reflection properties and refractive index that match the on or off state of the electro-active layer 172. The electro-active layer 172 may be controlled such that the path of an incident beam can be dynamically altered to essentially create a time-varying bright field.

[0290] 8A , a stacked waveguide assembly 178 may be utilized to provide three-dimensional perception to the eye / brain by having multiple waveguides (182, 184, 186, 188, 190) and multiple weak lenses (198, 196, 194, 192) together configured to transmit image information to the eye with varying levels of wavefront curvature per waveguide level that indicates the focal length to be perceived for that waveguide level. Multiple displays (200, 202, 204, 206, 208), or in another embodiment, a single multiplexed display, may each be utilized to inject collimated image information into the waveguides (182, 184, 186, 188, 190), which may be configured to distribute incident light substantially equally across the length of each waveguide for emission to the eye, as previously described.

[0291] The waveguide 182 closest to the eye is configured to deliver collimated light to the eye as it is launched into such waveguide 182, which may represent an optical infinity focal plane. Another waveguide 184 is configured to send collimated light that passes through a first weak lens (192; e.g., a weak negative lens) and is delivered to the user's eye 58. The first weak lens 192 may be configured to create a slight convex wavefront curvature so that the eye / brain interprets the light emerging from the waveguide 184 as originating from a first focal plane closer inward from optical infinity toward the person. Similarly, the next waveguide 186 passes its output light through a first 192 and second 194 lens before reaching the eye 58. The combined refractive power of the first 192 and second 194 lenses may be configured to create another, increasing amount of wavefront divergence such that the eye / brain interprets light originating from waveguide 186 as originating from a second focal plane that is even closer to optical infinity and pointing inward toward the person than light from waveguide 184.

[0292] The other waveguide layers (188, 190) and weak lenses (196, 198) are similarly configured, with the highest waveguide 190 in the stack sending its output through all of the weak lenses between it and the eye for a collective focal power representing the focal plane closest to the person. To compensate for the stack of lenses (198, 196, 194, 192) when viewing / interpreting light originating from the world 144 on the other side of the stacked waveguide assembly 178, a compensating lens layer 180 is positioned on top of the stack to compensate for the collective refractive power of the lower lens stacks (198, 196, 194, 192).

[0293] Such a configuration, as described above, again provides a relatively large exit pupil configuration, with as many perceived focal planes as there are available waveguide / lens pairings. Both the reflective side of the waveguide and the focusing side of the lens may be static (e.g., not dynamic or electro-active). Alternative embodiments may be dynamic, using electro-active features as described above, allowing a small number of waveguides to be multiplexed in a time-sequential manner to generate a larger number of effective focal planes.

[0294] 8B-8N, various aspects of diffractive configurations for focusing and / or redirecting a collimated beam are depicted. Other aspects of diffractive systems for such purposes are disclosed in U.S. Patent Application Serial No. 14 / 331,218.

[0295] With reference to Figure 8B, it should be understood that passing a collimated beam through a linear diffraction pattern 210, such as a Bragg grating, will deflect, or "steer," the beam. It should also be understood that passing a collimated beam through a radially symmetric diffraction pattern 212, or "Fresnel zone plate," will change the focus of the beam. Figure 8C illustrates the deflection effect of passing a collimated beam through a linear diffraction pattern 210. Figure 8D illustrates the focusing effect of passing a collimated beam through a radially symmetric diffraction pattern 212.

[0296] 8E and 8F, a combined diffraction pattern having both linear and radial elements 214 produces both deflection and focusing of a collimated input beam. These deflection and focusing effects can be produced in reflective as well as transmissive modes. These principles may also be applied with waveguide configurations to allow additional optical control, for example, as shown in FIGS. 8G-8N.

[0297] As shown in Figures 8G-8N, a diffraction pattern (220), i.e., a "diffractive optical element" (or "DOE"), is embedded within the planar waveguide 216 such that as the collimated beam is totally internally reflected along the planar waveguide 216, it intersects the diffraction pattern 220 at multiple locations.

[0298] Preferably, DOE 220 has a relatively low diffraction efficiency so that only a portion of the beam's light is deflected toward eye 58 with each intersection of DOE 220, while the remainder continues traveling through planar waveguide 216 via total internal reflection. The light carrying the image information is thus split into several related light beams that exit the waveguide at multiple locations, resulting in a highly uniform pattern of output emission toward eye 58 for this particular collimated beam bouncing within planar waveguide 216, as shown in FIG. 8H. The output beam toward eye 58 is shown in FIG. 8H as substantially parallel because, in this case, DOE 220 has only a linear diffraction pattern. As shown in a comparison between FIGS. 8L, 8M, and 8N, variations to this linear diffraction pattern pitch may be utilized to controllably deflect the output collimated beam, thereby creating scanning or tiled display functionality.

[0299] Referring to Figure 8I, with the change in the radially symmetric diffraction pattern components of the embedded DOE 220, the output beam pattern becomes more divergent, requiring the eye to accommodate to closer distances and focus on the retina, and will be interpreted by the brain as light from a viewing distance closer to the eye than optical infinity. Referring to Figure 8J, with the addition of another waveguide 218 into which a beam may be injected (e.g., by a projector or display), a DOE 221 embedded within this other waveguide 218, such as a linear diffraction pattern, may function to spread light across the entire larger planar waveguide 216. This may provide the eye 58 with a very large incident field of incident light, e.g., a large eyebox, exiting the larger planar waveguide 216, depending on the particular DOE configuration in operation.

[0300] Although the DOEs (220, 221) are depicted as bisecting the associated waveguides (216, 218), this need not be the case. In one or more embodiments, they may be located closer to or on either side of the waveguides (216, 218) to have the same functionality. Thus, as shown in FIG. 8K, with the injection of a single collimated beam, the entire field of cloned collimated beams can be directed toward the eye 58. Additionally, in combined linear diffraction pattern / radially symmetric diffraction pattern scenarios such as those depicted at 214 in FIG. 8F and 220 in FIG. 8I, beam distribution waveguide optics with Z-axis focusing capability (due to functionality such as expanded exit pupil; with a configuration such as that in FIG. 8K, the exit pupil can be the same size as the optical element itself, which can be a very significant advantage for user comfort and ergonomics) are presented, where the divergence angles of the cloned beams and the wavefront curvature of each beam both represent light originating from a point closer than optical infinity.

[0301] In one embodiment, one or more DOEs are switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer-dispersed liquid crystal in which the microdroplets comprise a diffractive pattern within a host medium, and the refractive index of the microdroplets can be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0302] Additionally, with dynamic changes to diffraction terms, such as linear diffraction pitch terms, as in Figures 8L-8N, beam scanning or tiled display functionality can be achieved. As previously mentioned, it may be desirable to have a relatively low diffraction grating efficiency in each of the DOEs (220, 221) to facilitate light distribution. Also, a better view of the real world can be achieved through such a configuration because the light that is desirably transmitted through the waveguide (e.g., in an augmented reality configuration, light originating from the world 144 toward the eye 58) is less affected when the diffraction efficiency of the DOE 220 over which it resides is lower.

[0303] Configurations such as that shown in Figure 8K are preferably driven with the introduction of image information in a time-sequential approach, with frame sequential driving being the simplest to implement. For example, an image of the sky at optical infinity can be introduced at time 1, and a diffraction grating can be utilized to keep the light collimated. An image of a closer tree branch can then be introduced at time 2, while the DOE controllably applies a focus change, e.g., one diopter or one meter away, to provide the eye / brain with the perception that the branch's optical information is coming from a closer focal distance.

[0304] This type of paradigm may be repeated in a rapid time-series manner such that the eye / brain perceives the input as being all part of the same image. While this is only an example of two focal planes, it should be understood that preferably the system would be configured with more focal planes to provide smoother transitions between objects and their focal distances. This type of configuration generally assumes that the DOE is switched relatively slowly (e.g., in sync with the frame rate of the display throwing the image, in the range of tens to hundreds of cycles per second).

[0305] At the other extreme may be a configuration in which the DOE elements can shift focus at tens to hundreds of MHz or more, facilitating switching of the focus state of the DOE elements on a pixel-by-pixel basis as the pixels are scanned into the eye 58 using a scanning light display type approach. This is desirable because it means that the overall display frame rate can be kept very low (low enough to ensure that "flicker" is not an issue (in the range of about 60-120 frames per second)).

[0306] Between these ranges, if the DOE can be switched at a kHz rate, the focus on each scan line can be adjusted on a row-by-row basis, which can provide a visual advantage to the user in terms of temporal artifacts, for example, during eye movement relative to the display. For example, different focal planes within a scene can be interleaved in this way to minimize visual artifacts in response to head movement (as discussed in detail later in this disclosure). The row-by-row focus modulator can be operably coupled to a line-scan display, such as a grating light valve display, in which a linear array of pixels is swept to form an image, or to a scanning light display, such as a fiber scanning display and a mirror scanning light display.

[0307] A stacked configuration similar to that of FIG. 8A may provide simultaneous multi-plane focusing using a dynamic DOE (rather than the static waveguide and lens of the embodiment of FIG. 8A). For example, with three simultaneous focal planes, a primary focal plane (e.g., based on measured ocular accommodation) may be presented to the user, and + and - boundaries (e.g., one focal plane closer and one farther away) may be utilized to provide a large focal distance to which the user can accommodate before a plane update is required. This increased focal distance may provide a time advantage if the user switches to a closer or farther focus (e.g., as determined by accommodation measurements). The new plane of focus may then be brought to the central focal depth, and the + and - boundaries may again be ready for fast switching to either one while the system catches up.

[0308] Referring to FIG. 8O, a stack 222 of planar waveguides (244, 246, 248, 250, 252) is shown, each with a reflector (254, 256, 258, 260, 262) at its end, configured so that collimated image information launched into one end by a display (224, 226, 228, 230, 232) bounces back by total internal reflection to the reflector, at which point some or all of the light is reflected toward the eye or other target. Each of the reflectors may have a slightly different angle so that they all reflect the exiting light toward a common destination, such as the pupil. Such a configuration is somewhat similar to that of FIG. 5B, except that each differently angled reflector in the FIG. 8O embodiment has its own unique waveguide so that the projected light does not interfere much as it travels to the target reflector. Lenses (234, 236, 238, 240, 242) may be interposed between the display and the waveguide for beam steering and / or focusing.

[0309] 8P illustrates a geometrically staggered version in which reflectors (276, 278, 280, 282, 284) are positioned at alternating lengths within the waveguides (266, 268, 270, 272, 274) so ​​that the emitted beam can be relatively easily aligned with objects such as the anatomical pupil. Because the distance between the stack (264) and the eye (such as 28 mm between the cornea of ​​the eye and the spectacle lens, which is a typical comfortable geometry) is known, the geometries of the reflectors (276, 278, 280, 282, 284) and waveguides (266, 268, 270, 272, 274) may be set to fill the eye's pupil with the emitted light (typically about 8 mm wide or less).

[0310] By directing light into an eyebox that is larger than the diameter of the eye's pupil, the viewer is free to make any number of eye movements while still being able to see the displayed image. Referring back to the discussion relating to Figures 5A and 5B regarding field of view expansion and reflector size, an expanded field of view is similarly provided by the configuration of Figure 8P, without the complexity of the switchable reflective element of the embodiment of Figure 5B.

[0311] 8Q illustrates version 286, in which many reflectors 298 form a relatively continuous curved reflective surface in aggregate or discrete flat facets oriented to match the overall curve. The curve may be a parabolic or elliptical curve, and is shown cutting across multiple waveguides (288, 290, 292, 294, 296) to minimize any crosstalk issues, but may also be used with monolithic waveguide configurations.

[0312] In one implementation, a high frame rate and lower persistence display may be combined with a lower frame rate and higher persistence display and a variable focus element to form a relatively high frame rate sequential stereoscopic display. In one embodiment, the high frame rate display has a lower bit depth, and the lower frame rate display has a higher bit depth, combined to form an effective high frame rate and high bit depth display that is well suited to presenting image slices in a frame sequential manner. Using such an approach, the desired three-dimensional volume is functionally divided into a series of two-dimensional slices. Each of these two-dimensional slices is sequentially projected onto an eye frame, and the focus of the variable focus element is changed in synchronization with this presentation.

[0313] In one embodiment, to provide a sufficient frame rate to support such a configuration, two display elements may be integrated: a full-color, high-resolution liquid crystal display ("LCD"; a backlit ferroelectric panel display may also be utilized in another embodiment; in a further embodiment, a scanning fiber display may be utilized) operating at 60 frames per second, and aspects of a higher frequency DLP system. Instead of illuminating the rear of the LCD panel in a conventional manner (e.g., with a full-size fluorescent lamp or LED array), the conventional illumination configuration may be eliminated to accommodate a DLP projector and project a mask pattern onto the rear of the LCD. In one embodiment, the mask pattern may be binary (e.g., the DLP is either illuminated or not illuminated). In another embodiment described below, the DLP may be utilized to project a grayscale mask image.

[0314] It should be understood that DLP projection systems can be operated at very high frame rates. In one embodiment for six depth planes at 60 frames per second, the DLP projection system can be operated against the back of an LCD display at 360 frames per second. The DLP projector may then be utilized to selectively illuminate portions of the LCD panel in synchronization with a high-frequency variable focusing element (such as a deformable membrane mirror) positioned between the viewing side of the LCD panel and the user's eyes (the variable focusing element (VFE) is configured to vary the overall display focus on a frame-by-frame basis at 360 frames per second).

[0315] In one embodiment, the VFE is positioned optically conjugate to the exit pupil to allow adjustment of focus without simultaneously affecting image magnification or "zoom." In another embodiment, the VFE is not conjugated to the exit pupil so that image magnification changes accompany focus adjustments. In such an embodiment, software may be used to compensate for optical magnification changes and any distortion by pre-scaling and distorting the image to be presented.

[0316] In operation, it is useful to consider an example in which a three-dimensional scene is to be presented to a user, with sky in the background at a viewing distance of optical infinity, and branches attached to the tree extending from a tree trunk such that the tips of the branches are closer to the user than the proximal portions of the branches where they join the tree trunk. The tree may be closer than optical infinity, and the branches may be closer compared to the tree trunk.

[0317] In one embodiment, for a given overall frame, the system may be configured to present on the LCD a full-color, in-focus image of tree branches in front of a sky. Then, in sub-frame 1 within the overall frame, a DLP projector in a binary masking configuration (e.g., illuminated or not illuminated) may be used to illuminate only the portion of the LCD representing the cloudy sky, while functionally black-masking (e.g., not illuminating) the portions of the LCD representing the tree branches and other elements that should not be perceived at the same focal length as the sky, and a VFE (such as a deformable membrane mirror) may be utilized to position the focal plane at optical infinity so that the sub-image in sub-frame 1 appears to the eye as infinitely distant clouds.

[0318] Then, in subframe 2, the VFE may be switched to focus on a point approximately 1 meter away from the user's eye (e.g., 1 meter relative to the location of the branch). ...

Claims

1. 1. A method for updating a virtual world, comprising: receiving a first input from a first device of a first user, the first input corresponding to a physical environment of the first user; updating a virtual world model based on the received first input, the virtual world model corresponding to the physical environment of the first user; and transmitting first updated information corresponding to a first portion of the virtual world model to a second user; receiving a plurality of inputs from respective first devices of a plurality of other users, the plurality of inputs corresponding to a physical environment of the first user; updating the virtual world model based on the received inputs; transmitting additional updated information corresponding to each additional portion of the virtual world model to the second user; and Including, the first updated information indicating whether any portion of the first updated information needs to be displayed to the second user; The method, wherein the additional updated information indicates whether any portion of the additional updated information needs to be displayed to the second user.

2. The method of claim 1 , wherein the virtual world model resides on a networked memory.

3. The method of claim 1 , wherein the first user and the second user are located at different locations.

4. 10. The method of claim 1, wherein the first device of the first user is selected from the group consisting of an FOV camera, another camera, a sensor, a first device for eye tracking, and a first device for audio.

5. transmitting the first updated information corresponding to the first portion of the virtual world model to the first user; The method of claim 1 , wherein the first updated information indicates whether any portion of the first updated information needs to be displayed to the first user.

6. transmitting the first updated information corresponding to the first portion of the virtual world model to a plurality of other users; The method of claim 1 , wherein the first updated information indicates whether any portion of the first updated information should be displayed to each of the plurality of other users.

7. transmitting the additional updated information corresponding to the portion of the virtual world model to the first user; The method of claim 1 , wherein the additional updated information indicates whether any portion of the additional updated information needs to be displayed to the first user.

8. transmitting the additional updated information corresponding to each additional portion of the virtual world model to the plurality of other users; The method of claim 1 , wherein the additional updated information indicates whether any portion of the additional updated information should be displayed to each of the plurality of other users.

9. receiving a second input from a second device of the second user, the second input corresponding to a physical environment of the second user; updating the virtual world model based on the received second input, the virtual world model corresponding to the physical environment of the first user; and transmitting second updated information corresponding to a second portion of the virtual world model to the first user; and Further comprising: The method of claim 1 , wherein the second updated information indicates whether any portion of the second updated information needs to be displayed to the first user.

10. The method of claim 9 , wherein the second updated information corresponds to a movement of an avatar of the second user within the second portion of the virtual world model.

11. 1. An augmented reality display system, comprising: an image capture device of the augmented reality display system for capturing a first input corresponding to a physical environment of a first user, the image capture device comprising one or more image capture sensors; a processor coupled to the image capture device; Equipped with The processor, receiving the first input from the image capture device; updating a virtual world model based on the received first input, the virtual world model corresponding to the physical environment of the first user; and transmitting first updated information corresponding to a first portion of the virtual world model to a second user; receiving a plurality of inputs from respective augmented reality devices of a plurality of other users, the plurality of inputs corresponding to a physical environment of the first user; updating the virtual world model based on the received inputs; transmitting additional updated information corresponding to each additional portion of the virtual world model to the second user; and configured to run the first updated information indicating whether any portion of the first updated information needs to be displayed to the second user; The additional updated information indicates whether any portion of the additional updated information needs to be displayed to the second user.

12. The system of claim 11 , further comprising a networked memory, wherein the virtual world model resides on the networked memory.

13. The system of claim 11 , wherein the first user and the second user are located at different locations.

14. The system of claim 11 , wherein the image capture device is selected from the group consisting of an FOV camera, another camera, a sensor, a first device for eye tracking, and a first device for audio.

15. The processor also includes: configured to transmit the first updated information corresponding to the first portion of the virtual world model to the first user; The system of claim 11 , wherein the first updated information indicates whether any portion of the first updated information needs to be displayed to the first user.

16. The processor also includes: configured to transmit the first updated information corresponding to the first portion of the virtual world model to a plurality of other users; The system of claim 11 , wherein the first updated information indicates whether any portion of the first updated information should be displayed to each of the plurality of other users.

17. The processor also includes: configured to transmit the additional updated information corresponding to the portion of the virtual world model to the first user; The system of claim 11 , wherein the additional updated information indicates whether any portion of the additional updated information needs to be displayed to the first user.

18. The processor also includes: configured to transmit the additional updated information corresponding to respective additional portions of the virtual world model to the plurality of other users; The system of claim 11 , wherein the additional updated information indicates whether any portion of the additional updated information should be displayed to each of the plurality of other users.

Images