Methods and systems for creating virtual and augmented reality.

Abstract

To provide methods and systems for creating virtual and augmented reality.SOLUTION: Configurations are disclosed for presenting virtual reality and augmented reality experiences to users. The systems may comprise an image capturing device to capture one or more images, the one or more images corresponding to a field of view of a user of a head-mounted augmented reality device, and a processor communicatively coupled to the image capturing device to extract a set of map points from the set of images, to identify a set of sparse points and a set of dense points from the extracted set of map points, and to perform normalization on the set of map points.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] Modern computing and display technologies are driving the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or parts thereof are presented to the user in a manner that appears, or can be perceived, as real. Virtual reality, or "VR," scenarios typically involve the presentation of digital or virtual imagery without transparency to other real-world visual inputs, while augmented reality, or "AR," scenarios typically involve the presentation of digital or virtual imagery as an extension of the visualization of the real world around the user. For example, an augmented reality scene may allow a user of AR technology to see one or more virtual objects superimposed on or between real-world objects (e.g., a setting like a real-world park featuring people, trees, and buildings in the background).

[0002] The human visual perception system is extremely complex, making it difficult to generate VR or AR technologies that facilitate a comfortable, natural, and rich presentation of virtual image elements among other virtual or real-world image elements. Conventional stereoscopic wearable glasses generally feature two displays configured to show images with slightly different elements presented so that three-dimensional distance images are perceived by the human visual system. Such configurations have been found uncomfortable for many users due to the mismatch between binocular divergence and accommodation, which can be overcome to perceive images in three dimensions. In fact, some users cannot tolerate stereoscopic configurations.

[0003] While several optical configurations (e.g., head-mounted glasses) are available (e.g., Google Glass®, Occulus Rift®, etc.), none of these configurations are optimally suited to presenting a rich, binocular, three-dimensional augmented reality experience in a way that would be comfortable and most useful to the user, because conventional systems fail to address some of the fundamental aspects of the human perceptual system, including the interaction between retinal photoreceptor cells and the brain that brings about the perception of visualization in the user.

[0004] The human eye is a highly complex organ, typically comprising the cornea, iris, lens, macula, retina, and optic nerve pathway to the brain. The macula is the central part of the retina and is used for seeing moderate detail. At the center of the macula is a portion of the retina called the "fovea," which is used for seeing the finest details in a scene and contains more photoreceptor cells (approximately 120 cones per diopter) than any other part of the retina.

[0005] The human visual system is not a passive sensor-type system; rather, it actively scans its environment. In a manner somewhat similar to the use of a flatbed scanner to capture an image, or the use of a finger to read Braille from paper, the photoreceptor cells in the eye do not respond to a constant stimulus state, but rather emit signals in response to changes in the stimulus. Therefore, movement is required to present the photoreceptor information to the brain.

[0006] In fact, experiments using substances such as cobra venom to paralyze the eye muscles have shown that human subjects would go blind if their eyes were kept open and positioned to view a static scene with eyes paralyzed by the venom. In other words, without a change in stimulus, the photoreceptor cells would not provide input to the brain, leading to blindness. This is thought to be at least one reason why normal human eyes are observed to move back and forth, or make small movements, in lateral movements also known as "microsaccades."

[0007] As mentioned earlier, the fovea of ​​the retina contains the highest density of photoreceptor cells. While it is generally recognized that humans have perception with high-resolution visualization capabilities throughout the entire visual field, in reality, humans have only a small high-resolution center that mechanically scans almost constantly, along with a persistent memory of high-resolution information recently captured in the fovea. In a somewhat similar manner, the eye's focal length control mechanism (e.g., the ciliary muscle, which is operably connected to the lens in such a way that ciliary relaxation produces tension in the ciliary connective fibers, flattening the lens for farther focal lengths, and ciliary contraction produces relaxation in the ciliary connective fibers, allowing the lens to take on a rounder geometric shape for closer focal lengths) makes a reciprocal micro-movement of about 1 / 4 to 1 / 2 diopters to periodically induce a small amount of "refractory blur" on both the near and far sides of the target focal length. This is utilized by the brain's near-far accommodation control circuit as periodic negative feedback, which helps to constantly correct the trajectory and keep the retinal image of a fixed object nearly in focus.

[0008] The brain's visualization center also derives useful perceptual information from the relative movements of both eyes and their components. Binocular divergence (e.g., pupillary rolling, where the eyes converge their gaze towards or away from an object to fixate on it) is closely related to the focusing (or "accommodation") of the eye's lens. Under normal conditions, changing the focus of the eye's lens, i.e., adjusting the eye to focus on an object at a different distance, will automatically produce a consistent change in binocular divergence up to the same distance, under a relationship known as the "accommodation-binocular divergence reflex." Similarly, changes in binocular divergence will also induce a consistent change in accommodation under normal conditions. Actions that counteract this reflex (as in most conventional stereoscopic AR or VR configurations) are known to cause eye strain, headaches, or other forms of discomfort in the user.

[0009] Head movement, which involves retracting the eyes, also significantly impacts object visualization. Humans tend to move their heads to visualize the world around them, often repositioning and reorienting the head relative to the object of interest in a very constant state. Furthermore, most people prefer to move their heads when their line of sight needs to move more than about 20 degrees from the center to focus on a particular object (for example, people typically do not prefer to see things "out of the corner of their eye"). Humans also typically scan or move their heads in conjunction with sound to improve audio signal acquisition and take advantage of the geometry of the ears relative to the head. The human visual system derives excellent depth cues from what is called "head-motion parallax," which relates to the relative motion of objects at different distances as a function of head movement and binocular eccentricity distance. In other words, if a person moves their head laterally and maintains a fixed position relative to an object, items further away from the object will move in the same direction as the head, while items directly in front of the object will move in the opposite direction to the head movement. These can be very significant clues to the spatial location of an object within its environment relative to a person. Head movements are also, naturally, used to look around at objects.

[0010] Furthermore, head and eye movements are coordinated with the "vestibular-ocular reflex," which stabilizes the image information on the retina during head rotation, and thus keeps the object image information near the center of the retina. In response to head rotation, the eyes reflexively and proportionally rotate in the opposite direction to maintain a stable fixed state on the object. As a result of this compensatory relationship, many people can read a book while their head vibrates back and forth. Interestingly, the same is generally not true if the book is turned while the head remains nearly stationary and moves back and forth at the same speed; that is, a person is unlikely to be able to read the book being turned. The vestibular-ocular reflex is one of the head-eye movement coordinations and is generally not well-developed for hand movements. This paradigm can be important for AR systems because the user's head movements can be relatively directly correlated with eye movements, and an ideal system would preferably be ready to work with this relationship.

[0011] In fact, given these various relationships, when setting up digital content (e.g., 3-D content such as a virtual chandelier object presented to extend the real-world view of a room, or 2-D content such as a planar virtual oil painting object presented to extend the real-world view of a room), design choices can be made to control the behavior of the objects. For example, a 2-D oil painting object may be head-centered, in which case the object moves with the user's head (e.g., as in the GoogleGlass® approach). In another embodiment, the object may be world-centered, in which case it may be presented as if it were part of a real-world coordinate system, allowing the user to move their head or eyes without moving the object's position relative to the real world.

[0012] Therefore, when virtual content is placed in an augmented reality world presented using an AR system, a selection is made as to whether the object should be presented world-centered, body-centered, head-centered, or eye-centered. In the head-centered approach, the virtual object remains in a fixed position in the real world, allowing the user to move their body, head, and eyes around it without changing its position relative to real-world objects surrounding it, such as real-world walls. In the body-centered approach, the virtual element can be fixed to the user's torso so that it is driven by the movement of the torso, although the user can move their head or eyes without moving the object itself. In the head-centered approach, the displayed object (and / or the display itself) may move with the movement of the head, as described above with reference to Google Glass®. In the eye-centered approach, as in the “foveal display” configuration described below, the content is driven as a function of the position of the eyes.

[0013] In a world-centric configuration, it may be desirable to have inputs such as accurate head pose measurement, accurate representation and / or measurement of real-world objects and geometric shapes around the user, low-latency dynamic rendering on an augmented reality display as a function of head pose, and generally low-latency displays.

[0014] The aforementioned U.S. patent application presents systems and techniques for working with typical human visual configurations to address various challenges in virtual reality and augmented reality applications. The design of these virtual reality and / or AR systems presents numerous challenges, including the speed of the system in delivering virtual content, the quality of the virtual content, the user's eye distance, the size and portability of the system, and other system and optical challenges.

[0015] The systems and techniques described herein are configured to address these challenges in conjunction with typical human visual constructions. [Overview of the project] [Means for solving the problem]

[0016] Embodiments of the present invention relate to devices, systems, and methods for facilitating virtual reality and / or augmented reality interactions for one or more users. In one aspect, a system for displaying virtual content is disclosed.

[0017] In one aspect, the augmented reality display system comprises an image capture device for capturing one or more images, one or more images corresponding to the field of view of a user of a head-mounted augmented reality device, and a processor communicatively coupled to the image capture device, which extracts a set of map points from the set of images, identifies a set of low-density points and a set of high-density points from the extracted set of map points, and performs normalization on the set of map points. In one or more embodiments, the processor may generate low-density and high-density point descriptors for the set of low-density points and the set of high-density points, respectively.

[0018] In one or more embodiments, low-density point descriptors and high-density point descriptors are stored as map data. In one or more embodiments, a set of low-density points corresponds to a distinct feature of one or more images. In one or more embodiments, the distinct feature is selected from the group consisting of angles, circles, triangles, and text.

[0019] In one or more embodiments, the set of high-density points corresponds to 3D points in the field of view. In one or more embodiments, the set of high-density points further includes color values. In one or more embodiments, normalization includes scale normalization. In one or more embodiments, normalization includes coordinate normalization with respect to a common origin. In one or more embodiments, normalization utilizes machine learning. In one or more embodiments, low-density and high-density point descriptors correspond to the low-density and high-density points, respectively, in the set of low-density and high-density points. In one or more embodiments, the low-density and high-density point descriptors include information about at least one of scale, texture, orientation, and patch data.

[0020] In another respect, a method for generating map data includes the steps of identifying a set of map points associated with one or more images, determining a set of low-density points and a set of high-density points from the identified map points, and normalizing the respective sets of low-density points and high-density points.

[0021] In one or more embodiments, the method further includes the steps of generating low-density and high-density point descriptors for each set of low-density points and high-density points, and combining the low-density and high-density point descriptors and storing them as map data.

[0022] In one or more embodiments, a set of low-density points corresponds to a distinct feature. In one or more embodiments, the distinct feature is selected from the group consisting of angles, circles, triangles, and text. In one or more embodiments, a set of high-density points corresponds to 3D points in the field of view. In one or more embodiments, the set of high-density points also includes color values.

[0023] In one or more embodiments, the normalization includes scale normalization. In one or more embodiments, the normalization includes coordinate normalization relative to a common origin. In one or more embodiments, the normalization is implemented using machine learning. In one or more embodiments, low-density and high-density point descriptors correspond to each low-density and high-density point in their respective sets of low-density and high-density points. In one or more embodiments, each low-density and high-density point descriptor contains information about each low-density and high-density point, selected from the group consisting of scale, orientation, patch data, and texture.

[0024] In another aspect, the augmented reality display system comprises an image capture device for capturing one or more images, wherein the one or more images correspond to the user's field of view, and the images capture at least one gesture created by the user; and a processor communicatively coupled to the image capture device and configured to identify a set of points associated with a gesture, compare the set of points with a database of given gestures, recognize a gesture at least partially based on the comparison, and determine user input based at least partially recognized gestures.

[0025] In one or more embodiments, the processor generates a scored value for the set of identified points based on the comparison. In one or more embodiments, the processor recognizes a gesture if the scored value exceeds a threshold. In one or more embodiments, the augmented reality display system includes a database for storing a given set of gestures. In one or more embodiments, the system further includes networked memory for accessing the given database of gestures.

[0026] In one or more embodiments, the gesture is a hand gesture. In one or more embodiments, the gesture is a finger gesture. In one or more embodiments, the gesture is an interfinal interaction. In one or more embodiments, the gesture is selected from the group consisting of interfinal interactions, pointing, tapping, and friction.

[0027] In one or more embodiments, the augmented reality display system further comprises a spatial light modulator, which is communicably coupled to a processor, and the processor controls the spatial light modulator so that one or more virtual objects are displayed to the user based on at least partially determined user input. In one or more embodiments, one or more virtual objects comprises a virtual user interface.

[0028] In another respect, a method for determining user input includes the steps of: capturing an image of the user's field of view, the image including a gesture made by the user; analyzing the captured image and identifying a set of points associated with the gesture; comparing the identified set of points with a set of points associated with a given gesture database; and determining user input based on the recognized gesture.

[0029] In one or more embodiments, the method further includes the step of generating a scored value for a set of identified points based on a comparison. In one or more embodiments, the method further includes the step of recognizing a gesture if the scored value exceeds a threshold. In one or more embodiments, the method further includes the step of storing a given gesture in a database. In one or more embodiments, the method further includes the step of accessing networked memory and accessing a database of given gestures.

[0030] In one or more embodiments, the gesture is a hand gesture. In one or more embodiments, the gesture is a finger gesture. In one or more embodiments, the gesture is an interfinal interaction. In one or more embodiments, the gesture is selected from the group consisting of interfinal interactions, pointing, tapping, and friction.

[0031] In one or more embodiments, the method further includes the step of displaying one or more virtual objects to a user based on at least partially determined user input. In one or more embodiments, one or more virtual objects comprise a virtual user interface.

[0032] In another aspect, the augmented reality display system comprises an image capture device for capturing one or more images, and a processor communicatively coupled to the image capture device, which analyzes the captured images and identifies one or more gestures created by the user, the identification of one or more gestures including steps utilizing a cascading mechanism having multiple stages.

[0033] In one or more embodiments, the cascade mechanism comprises multiple nodes, each node corresponding to a stage among multiple stages. In one or more embodiments, the cascade mechanism comprises a series of permissive analysis nodes. In one or more embodiments, the earlier stages among the multiple stages of the cascade mechanism are configured to consume less processing power compared to the later stages among the multiple stages of the cascade mechanism.

[0034] In one or more embodiments, non-gestures are removed based on an analysis of images captured in a preceding step of the cascade mechanism. In one or more embodiments, later steps of the multiple steps are configured to determine more complex gestures based on at least partially captured images.

[0035] In one or more embodiments, the analysis of the captured image includes the step of determining whether the sharpness of the contours of various shapes within the captured image is sharp enough to constitute a gesture. In one or more embodiments, a later stage of the cascade mechanism is used to distinguish between different gestures. In one or more embodiments, the processor is further configured to generate a score based at least partially on the analysis. In one or more embodiments, the processor removes a candidate image from consideration if the generated score is below a minimum threshold. In one or more embodiments, the processor proceeds to a later stage of the cascade mechanism if the generated score is above the minimum threshold.

[0036] In another aspect, the method includes the steps of capturing one or more images corresponding to the user's field of view, and analyzing the captured one or more images to identify one or more gestures created by the user, wherein the analysis includes the step of utilizing a cascading mechanism having multiple stages.

[0037] In one or more embodiments, the cascade mechanism comprises multiple nodes, each node corresponding to a stage among multiple stages. In one or more embodiments, the cascade mechanism comprises a series of permissive analysis nodes. In one or more embodiments, the earlier stages among the multiple stages of the cascade mechanism are configured to consume less processing power compared to the later stages among the multiple stages of the cascade mechanism.

[0038] In one or more embodiments, non-gestures are removed based on an analysis of images captured in a preceding step of the cascade mechanism. In one or more embodiments, later steps of the multiple steps are configured to determine more complex gestures based on at least partially captured images.

[0039] In one or more embodiments, the analysis of the captured image includes the step of determining whether the sharpness of the contours of various shapes within the captured image is sharp enough to constitute a gesture. In one or more embodiments, a later stage of the cascade mechanism is used to distinguish between different gestures.

[0040] In one or more embodiments, the method further includes a step of generating a score based at least partially on the analysis. In one or more embodiments, the method further includes a step of removing a candidate image from consideration to determine whether the generated score is below a minimum threshold. In one or more embodiments, the method further includes a step of advancing to a later stage of the cascade mechanism if the generated score is above a minimum threshold.

[0041] In another aspect, the augmented reality system comprises an image capture device for capturing multiple images of each user's field of view, and a processor communicatively coupled to the image capture device, which analyzes the multiple images, generates multiple gesture candidates from the captured images, and generates analytical values ​​corresponding to the multiple gesture candidates, and the gestures are recognized, at least partially, based on the analytical values.

[0042] In one or more embodiments, the processor is further configured to sort the gesture candidates based at least partially on their respective analysis values. In one or more embodiments, the processor is further configured to eliminate gesture candidates having analysis values ​​lower than a minimum threshold. In one or more embodiments, the processor is further configured to advance gesture candidates to the next stage of processing if their analysis values ​​are higher than the minimum threshold.

[0043] In another aspect, the method includes the steps of capturing multiple images of each user's field of view, analyzing the multiple images to generate multiple gesture candidates, and generating analytical values ​​corresponding to the multiple analytical values, wherein the gesture is recognized at least partially based on the analytical values.

[0044] In one or more embodiments, the method further includes the step of sorting gesture candidates based at least partially on their respective analysis values. In one or more embodiments, the method further includes the step of eliminating gesture candidates having analysis values ​​lower than a minimum threshold. In one or more embodiments, the method further includes the step of advancing gesture candidates to the next stage of processing if their analysis values ​​are higher than the minimum threshold.

[0045] In yet another aspect, the augmented reality display system comprises an image capture device for capturing images of the user's field of view, and a processor configured to generate a depth map corresponding to the captured image, analyze the depth map of the captured image, and identify gestures.

[0046] In one or more embodiments, the processor is further configured to utilize a classification mechanism to identify parts of the hand corresponding to points in the generated depth map. In one or more embodiments, the processor is further configured to skeletonize the depth map based on the identification of the parts of the hand.

[0047] In one or more embodiments, the processor classifies an image as a gesture based on a depth map that is at least partially skeletonized. In one or more embodiments, the depth is generated by performing at least partially a depth partitioning process.

[0048] In one or more embodiments, depth partitioning includes line search. In one or more embodiments, the processor performs cascade analysis on the depth map and classifies the image as gestures. In one or more embodiments, the processor performs depth expansion on the depth map. In one or more embodiments, the processor performs surface normalization on the depth map.

[0049] In one or more embodiments, the processor performs orientation normalization on the depth map. In one or more embodiments, the processor performs background subtraction on the depth map. In one or more embodiments, the processor performs depth comparison on the depth map. In one or more embodiments, the processor classifies the image as a gesture based on at least a partially skeletonized depth map and prior information. In one or more embodiments, the classification mechanism is a decision forest.

[0050] In another respect, a method for classifying gestures includes the steps of capturing an image of the user's field of view, performing depth compartmentalization on the captured image to generate a depth map, and identifying the gesture based on the at least partially generated depth map.

[0051] In one or more embodiments, the method further includes the step of analyzing a depth map using a classification mechanism and identifying parts of a hand corresponding to points in the depth map. In one or more embodiments, the method further includes the step of skeletalizing the depth map based on the identification of parts of a hand. In one or more embodiments, the method further includes the step of classifying the image as a gesture based on the skeletal depth map.

[0052] In one or more embodiments, depth partitioning includes line search. In one or more embodiments, the method further includes the step of performing a cascade analysis on the depth map and classifying the images as gestures. In one or more embodiments, the method further includes the step of performing depth expansion on the depth map.

[0053] In one or more embodiments, the method further includes the step of performing surface normalization on the depth map. In one or more embodiments, the method further includes the step of performing orientation normalization on the depth map. In one or more embodiments, the method further includes the step of performing background subtraction on the depth map.

[0054] In one or more embodiments, the method further includes the step of performing depth comparisons on a depth map. In one or more embodiments, the method further includes the step of classifying an image as a gesture based on the skeletal depth map and prior information. In one or more embodiments, the classification mechanism is a decision forest.

[0055] In another aspect, the augmented reality display system comprises an image capture device for capturing a set of images in the user's field of view, the set of images capturing the user's movement related to a given physical object, and a processor communicatively coupled to the image capture device, which analyzes the movement related to the given physical object and determines user input based on the at least partially analyzed movement.

[0056] In one or more embodiments, the processor recognizes a given physical object. In one or more embodiments, the given physical object is recognized based at least partially on an image comparison between a captured image of the given physical object and a database of the given physical object. In one or more embodiments, the analyzed movement of a user related to the given physical object is used to generate a pattern. In one or more embodiments, the generated pattern is compared with a database of given patterns. In one or more embodiments, the processor generates a scored value for the recognized pattern based on the comparison.

[0057] In one or more embodiments, the processor determines user input if the scored value exceeds a threshold. In one or more embodiments, an image capture device visually tracks movement related to a given physical object and generates a video recording. In one or more embodiments, the video recording is analyzed to determine user input.

[0058] In one or more embodiments, the augmented reality display system further includes networked memory for accessing a database of predetermined patterns. In one or more embodiments, the predetermined physical object is selected from a group consisting of existing structures in the field of view, actively marked totems, passively marked totems, objects integrated into cameras / sensors, and totem controller objects.

[0059] In one or more embodiments, movement relating to a given physical object is selected from the group consisting of position, orientation, and movement of the given physical object relative to a reference frame. In one or more embodiments, the given physical object comprises a first hand of the user's hands, and movement relating to the first hand comprises manipulation of the first hand using a second hand of the user's hands.

[0060] In one or more embodiments, a given physical object has a soft-hard surface, and movement related to the given physical object involves pressing the soft-hard surface by the user. In one or more embodiments, a processor renders virtual interface elements associated with the given physical object for the user, and the virtual interface elements are visible to the user through a display device.

[0061] In one or more embodiments, the virtual interface element is displayed in relation to a predetermined physical object such that, when viewed by a user, the virtual interface element is modified based on modifications relating to the predetermined physical object, at least in part. In one or more embodiments, the predetermined physical object comprises an electronic input device, and user input is determined based on the perceived movement of the predetermined physical object and input from the electronic input device.

[0062] In one or more embodiments, the processor is configured to control a display device coupled to the user's eyes such that virtual content displayed to the user is modified based on at least partially determined user input.

[0063] In another respect, a method for determining user input includes the steps of capturing images of the field of view of one or more users, wherein the images include at least a predetermined physical object; analyzing the images and detecting user movement related to the predetermined physical object; and determining user input based on an analysis of the movement at least partially related to the predetermined physical object.

[0064] In one or more embodiments, the method further includes the step of recognizing a predetermined physical object. In one or more embodiments, the predetermined physical object is recognized based at least partially on an image comparison between a captured image of the predetermined physical object and a database of the predetermined physical object. In one or more embodiments, the movement of a user related to the predetermined physical object is used to generate a pattern.

[0065] In one or more embodiments, the method further includes the step of comparing the generated pattern with a database of predetermined patterns. In one or more embodiments, the method further includes the step of generating a scored value for the recognized pattern based on the comparison. In one or more embodiments, the method further includes the step of determining user input if the scored value exceeds a threshold.

[0066] In one or more embodiments, the method further includes the step of visually tracking movement related to a given physical object and generating a video recording. In one or more embodiments, the video recording is analyzed to determine user input. In one or more embodiments, the method further includes the step of accessing a database of given patterns through networked memory. In one or more embodiments, the given physical object is selected from the group consisting of existing structures in the field of view, actively marked totems, passively marked totems, objects integrated into a camera / sensor, and totem controller objects.

[0067] In one or more embodiments, movement relating to a given physical object is selected from the group consisting of position, orientation, and movement of the given physical object relative to a reference frame. In one or more embodiments, the given physical object comprises a first hand of the user's hands, and movement relating to the first hand comprises manipulation of the first hand using a second hand of the user's hands.

[0068] In one or more embodiments, a given physical object has a soft-hard surface, and movement related to the given physical object comprises pressing the soft-hard surface by the user. In one or more embodiments, the method further includes the step of rendering a virtual interface element associated with the given physical object for the user.

[0069] In one or more embodiments, a virtual interface element is displayed in relation to a predetermined physical object such that, when viewed by a user, the virtual interface element is modified based on modifications relating to the predetermined physical object. In one or more embodiments, the predetermined physical object comprises an electronic input device, and user input is determined based on the perceived movement of the predetermined physical object and input from the electronic input device. In one or more embodiments, the method further includes the step of modifying at least one characteristic of the virtual content displayed to the user based on at least partially determined user input.

[0070] In another aspect, an augmented reality display system comprises a display that is physically coupled to the user's eyes and displays a set of virtual content relating to one or more physical objects, and a processor that is communicatively coupled to the display and an image capture device and, based on user input, identifies a virtual user interface to be displayed to the user via the display, obtains the user's location relating to the world, determines a set of coordinate points on which the virtual user interface should be displayed, and controls the display so that the identified virtual user interface is displayed to the user.

[0071] In one or more embodiments, user input is determined based on at least partially recognized gestures. In one or more embodiments, user input is determined based at least partially on voice commands. In one or more embodiments, user input is determined based at least partially on interaction with a given physical object.

[0072] In one or more embodiments, the augmented reality display system further includes a library of user interfaces, from which identified user interfaces are retrieved. In one or more embodiments, identified virtual user interfaces are associated with a reference frame. In one or more embodiments, the reference frame is a body-centered reference frame.

[0073] In one or more embodiments, the reference frame is a head-centered reference frame. In one or more embodiments, the reference frame is a hand-centered reference frame. In one or more embodiments, the reference frame is a world-centered reference frame. In one or more embodiments, the processor performs a transformation between the reference frame associated with the identified virtual user interface and the obtained location of the user in relation to the world to determine the set of coordinate points of the virtual user interface.

[0074] In one or more embodiments, the user's location in the world is determined based on the user's GPS location. In one or more embodiments, the user's location in the world is determined based on a set of map points associated with the user. In one or more embodiments, the virtual user interface appears stationary as the user moves.

[0075] In one or more embodiments, the virtual user interface moves in relation to the user's movement. In one or more embodiments, the processor determines a reference frame associated with an identified virtual user interface, determines the location of the reference frame relative to a world reference frame, sets the determined location as the origin, and determines a set of coordinate points relative to the origin.

[0076] In another respect, a method for generating a virtual user interface includes the steps of: identifying a virtual user interface to be displayed to a user based on user input; obtaining a location associated with the user; determining a set of coordinate points on which the identified virtual user interface should be displayed, based at least partially on the obtained location; and displaying the virtual user interface to the user at the determined coordinate points.

[0077] In one or more embodiments, user input is determined based on at least partially recognized gestures. In one or more embodiments, user input is determined based at least partially on voice commands. In one or more embodiments, user input is determined based at least partially on interaction with a given physical object.

[0078] In one or more embodiments, the method further includes the step of retrieving an identified user virtual user interface from a user interface library. In one or more embodiments, the identified virtual user interface is associated with a reference frame. In one or more embodiments, the reference frame is a body-centered reference frame.

[0079] In one or more embodiments, the reference frame is a head-centered reference frame. In one or more embodiments, the reference frame is a hand-centered reference frame. In one or more embodiments, the reference frame is a world-centered reference frame.

[0080] In one or more embodiments, the method further includes the step of performing a transformation between a reference frame associated with an identified virtual user interface and the obtained location of the user in relation to the world, and determining a set of coordinate points for the virtual user interface. In one or more embodiments, the user's location in the world is determined based on the user's GPS location.

[0081] In one or more embodiments, the user's location in the world is determined based on a set of map points associated with the user. In one or more embodiments, the virtual user interface appears stationary as the user moves. In one or more embodiments, the virtual user interface moves in relation to the user's movement.

[0082] In one or more embodiments, the method further includes the steps of determining a reference frame associated with an identified virtual user interface, determining the location of the reference frame relating to a global reference frame, setting the determined location as the origin, and determining a set of coordinate points relating to the origin.

[0083] In one or more embodiments, the method further includes the step of reading a set of map points from networked memory. In one or more embodiments, user input comprises a location in space where the virtual user interface should be displayed. In one or more embodiments, the location in space is associated with a physical entity at the user's location. In one or more embodiments, user input comprises a gesture including a throwing input indicating a wall. In one or more embodiments, user input comprises a gesture signifying the end of an instruction for generating the virtual user interface.

[0084] In another respect, a method for generating a virtual user interface includes the steps of detecting operations on a given physical object, recognizing a command to create a virtual user interface based on the detected operations, determining a set of map points associated with the location of the given physical object from a virtual world model, and rendering the virtual user interface in real time to the determined map points associated with the location of a totem, such that when the virtual user interface is viewed by a user, it appears to be stationary at the location of the given physical object.

[0085] In one or more embodiments, the manipulation of a given physical object includes a gesture of releasing a state in which the user's hand is pinching the surface of the given physical object. In one or more embodiments, a virtual user interface, when viewed by the user, appears to cover a portion of the surface of the given physical object, the portion corresponding to the location of the user's hand during the formation of the gesture of releasing the pinching state. In one or more embodiments, the given physical object is the user's hand.

[0086] In one or more embodiments, the manipulation of a given physical object includes an action selected from the group consisting of the user spreading their hand, the user showing their palm, and the user raising their hand. In one or more embodiments, the virtual user interface appears to cover a portion of the surface of the hand when viewed by the user.

[0087] In one or more embodiments, the virtual user interface comprises a plurality of first-level menu items selectable by a finger or thumb. In one or more embodiments, the method further includes the steps of detecting further hand movements, recognizing a command to create a second virtual user interface based on the detected further movements, and rendering the second virtual user interface in real time at determined map points associated with the location of a given physical object, such that the virtual user interface appears to be stationary at the location of the given physical object when viewed by a user.

[0088] In one or more embodiments, the method further includes further hand operations, including spreading the fingers apart. In one or more embodiments, the second virtual user interface comprises a plurality of second-level menu items selectable by the fingers or thumb, the second level being lower than the first level.

[0089] In one or more embodiments, further manipulation of the totem includes performing a motion of drawing a circle on the palm using the fingers of a second hand of the user. In one or more embodiments, a second virtual user interface comprises a plurality of menu items arranged in an arc, the menu items being scrollable and selectable by the fingers of the second hand.

[0090] In another aspect, the augmented reality display system comprises: an image capture device for capturing one or more images of a user's field of view, the image capture device capturing at least one image capturing the operation of a given physical object; a display device for displaying one or more virtual objects to the user; a database comprising a passable world model, the passable world model comprising a set of map points associated with physical objects in the world; and a processor communicatively coupled to the image capture device, which recognizes commands for creating a virtual user interface based on one or more images, determines map points corresponding to a given physical object based at least partially on the passable world model, and controls the display in such a manner that the virtual user interface is generated at the determined map points corresponding to the given physical object so that the virtual user interface appears stationary at the location of the given physical object.

[0091] In one or more embodiments, the manipulation of a given physical object includes a gesture of releasing a state in which the user's hand is pinching the surface of the given physical object. In one or more embodiments, a virtual user interface, when viewed by the user, appears to cover a portion of the surface of the given physical object, the portion corresponding to the location of the user's hand during the formation of the gesture of releasing the pinching state. In one or more embodiments, the given physical object is the user's hand.

[0092] In one or more embodiments, the manipulation of a given physical object includes an action selected from the group consisting of the user spreading their hand, the user showing their palm, and the user raising their hand. In one or more embodiments, the virtual user interface appears to cover a portion of the surface of the hand when viewed by the user.

[0093] In one or more embodiments, the virtual user interface comprises a plurality of first-level menu items selectable by a finger or thumb. In one or more embodiments, a given physical object is a hand, and the processor detects further hand movements, recognizes commands to create a second virtual user interface, and controls the display so that the second virtual user interface is displayed at a determined map point.

[0094] In one or more embodiments, further hand manipulation includes spreading the fingers apart. In one or more embodiments, the second virtual user interface comprises a plurality of second-level menu items selectable by the fingers or thumb, the second level being lower than the first level. In one or more embodiments, further totem manipulation includes making a motion of drawing a circle on the palm using the fingers from the second hand of the user's hands.

[0095] In one or more embodiments, the second virtual user interface comprises a plurality of menu items arranged in an arc, the menu items being scrollable and selectable by the fingers of a second hand.

[0096] In another respect, a method for updating a virtual world includes the steps of: receiving a first input from a first device of a first user, wherein the first input corresponds to the physical environment of the first user; updating a virtual world model based on the received first input, wherein the virtual world model corresponds to the physical environment of the first user; and transmitting first updated information corresponding to a first part of the virtual world model to a second user, wherein the first updated information is configured to indicate whether any part of the first updated information should be displayed to the second user.

[0097] In one or more embodiments, the virtual world model resides on networked memory. In one or more embodiments, the first user and the second user are located in different locations. In one or more embodiments, the first device of the first user is selected from a group consisting of an FOV camera, other cameras, sensors, a first device for eye tracking, and a first device for audio.

[0098] In one or more embodiments, the method further includes the step of transmitting first updated information corresponding to a first part of a virtual world model to a first user, wherein the first updated information is configured to indicate whether any part of the first updated information should be displayed to the first user. In one or more embodiments, the method further includes the step of transmitting first updated information corresponding to a first part of a virtual world model to a plurality of other users, wherein the first updated information is configured to indicate whether any part of the first updated information should be displayed to each of the plurality of other users.

[0099] In one or more embodiments, the method further includes the steps of receiving a plurality of inputs from each of a plurality of other users' first devices, wherein the plurality of inputs correspond to the physical environment of the first user; updating a virtual world model based on the received plurality of inputs; and transmitting additional updated information corresponding to each additional part of the virtual world model to a second user, wherein the additional updated information is configured to indicate whether any part of the additional updated information should be displayed to the second user.

[0100] In one or more embodiments, the method further includes the step of transmitting additional updated information corresponding to a portion of a virtual world model to a first user, wherein the additional updated information is configured to indicate whether any portion of the additional updated information should be displayed to the first user.

[0101] In one or more embodiments, the method further includes the step of transmitting additional updated information corresponding to each additional part of the virtual world model to a plurality of other users, wherein the additional updated information is configured to indicate whether any part of the additional updated information should be displayed to each of the plurality of other users.

[0102] In one or more embodiments, the method further includes the steps of: receiving a second input from a second device of a second user, the second input corresponding to the physical environment of the second user; updating a virtual world model based on the received second input, the virtual world model corresponding to the physical environment of a first user; and transmitting second updated information corresponding to a second portion of the virtual world model to the first user, the second updated information being configured to indicate whether any portion of the second updated information should be displayed to the first user. In one or more embodiments, the second updated information corresponds to the movement of the second user's avatar within the second portion of the virtual world model.

[0103] In another aspect, the method includes the steps of projecting a light pattern into space, using a camera to detect first and second portions of the light pattern and generating first and second data points corresponding to the first and second portions of the pattern, and performing triangulation analysis to determine the locations of the first and second portions of the light pattern, at least partially based on the first and second data points.

[0104] In one or more embodiments, the method further includes the step of repeatedly projecting, detecting, and triangulating multiple portions of a pattern to obtain additional texture data of space. In one or more embodiments, the step of projecting light into space includes the step of projecting light into space using a fiber-based projector.

[0105] In one or more embodiments, the method further includes the step of modifying the light using optical elements to form a beam of light rays before the light is projected into space. In one or more embodiments, the light pattern corresponds to structured light and the light pattern is dynamic. In one or more embodiments, the light pattern corresponds to patterned light and the light pattern includes multiple points.

[0106] In one or more embodiments, the light pattern corresponds to textured light, and the light pattern is irregular. In one or more embodiments, the first and second portions of the light pattern are the respective pixels of the light pattern.

[0107] In another aspect, the augmented reality display system comprises a light projector for projecting a light pattern into space, a camera coupled to the frame of the augmented reality display system for capturing an image of the user's field of view, the camera for capturing first and second portions corresponding to the projected light pattern, and a processor communicatively coupled to the camera, which detects first and second data points corresponding to the first and second portions of the pattern based at least partially on data received from the camera, performs triangulation analysis, and determines the locations of the first and second portions of the light pattern based at least partially on the first and second data points.

[0108] In one or more embodiments, the processor repeatedly performs the steps of projection, detection, and triangulation using multiple parts of the pattern to acquire additional texture data of space. In one or more embodiments, the optical projector comprises a fiber-based projector. In one or more embodiments, the light to be projected is modified using optical elements so that the light forms a beam of light before it is projected into space.

[0109] In one or more embodiments, the light pattern corresponds to structured light and is dynamic. In one or more embodiments, the light pattern corresponds to patterned light and includes multiple points. In one or more embodiments, the light pattern corresponds to textured light and is irregular. In one or more embodiments, the first and second parts of the light pattern are the respective pixels of the light pattern.

[0110] In another aspect, the augmented reality system comprises a given physical object having a surface configured to be manipulated such that the manipulation is detectable by the augmented reality system as user input, the given physical object comprising a material configured to provide the user with tactile sensation when the user interacts with the given physical object via touch.

[0111] In one or more embodiments, a predetermined physical object is configured to simulate a user input device. In one or more embodiments, the user input device is selected from the group consisting of a computer keyboard, a computer mouse, a computer trackpad, and a handheld controller.

[0112] In one or more embodiments, the system further includes a display device for displaying one or more virtual objects to a user, and a processor for controlling the display device so that a virtual user interface is displayed on the surface of a predetermined physical object.

[0113] In one or more embodiments, a given physical object has a spherical shape. In one or more embodiments, the system is configured to display a virtual user interface such that it appears to emerge from the given physical object. In one or more embodiments, a given physical object has a textured outer surface.

[0114] In one or more embodiments, a given physical object includes a user input element. In one or more embodiments, the user input element is selected from the group consisting of keys, buttons, and scroll wheels. In one or more embodiments, the user input element is not operably coupled to an electronic device. In one or more embodiments, the system further includes a camera configured to detect user interaction with the totem.

[0115] In one or more embodiments, a given physical object includes a recess, cavity, or protrusion. In one or more embodiments, the operation is selected from a group of finger movements consisting of swiping, rocking, rotating, scrolling, tapping, double-tapping, short tapping, and long tapping.

[0116] In one or more embodiments, the operation is selected from a group of finger movement characteristics consisting of several interactions, types of interactions, and durations of interactions. In one or more embodiments, the operation is selected from a group of finger characteristics consisting of distance, direction, speed, and acceleration.

[0117] In yet another aspect, a method for generating a virtual room includes the steps of: identifying a virtual room to be displayed to a user; obtaining a first set of map points associated with the virtual room, the first set of map points corresponding to a first physical room at the user's location; tethering the virtual room to the first set of map points; and displaying the virtual room to the user such that, when the virtual room is viewed by the user, it appears to be stationary in the set of first map points.

[0118] In one or more embodiments, the method further includes the step of identifying a virtual room based on user input. In one or more embodiments, the user input is selected from the group consisting of gestures, visual data, auditory data, sensory data, direct commands, voice commands, eye tracking, and selection of physical buttons.

[0119] In one or more embodiments, the virtual room is provided by the user. In one or more embodiments, the method further includes the step of reading a first set of map points from networked memory. In one or more embodiments, the method further includes the step of obtaining a second set of map points associated with the virtual room, the second set of map points corresponding to a second physical room at the user's location after the user has moved from the first physical room; tethering the virtual room to the second set of map points; and displaying the virtual room to the user such that, when the virtual room is viewed by the user, it appears to be stationary in the second set of map points.

[0120] In one or more embodiments, the virtual room also includes first and second virtual objects, and the method further includes, if the virtual room is tethered to a second set of map points, the step of maintaining the relative spatial position between the first virtual object and the second virtual object corresponding to the first set of map points. In one or more embodiments, the method further includes the step of displaying the virtual room to the user such that, when viewed by the user, the virtual room appears to be stationary with respect to a portion of the first physical room. In one or more embodiments, the virtual room also includes selectable virtual decorations.

[0121] In another respect, the method for creating a retail experience includes the steps of recognizing the user's location within a retail store, retrieving data corresponding to the retail store, generating virtual content related to the retail store based on the retrieved data, creating a virtual user interface within the user's field of view, and displaying the virtual content on the virtual user interface while the user engages in retail activities within the retail store.

[0122] In one or more embodiments, the retrieved data comprises a set of map points corresponding to retail stores, and the virtual user interface appears stationary in the set of map points when viewed by the user. In one or more embodiments, the user's location is recognized using a radio frequency identification transponder and communication. In one or more embodiments, the virtual content is selected from a group consisting of virtual characters, virtual coupons, games based on the user's location, lists of promotional items, nutritional information, metadata related to items, celebrity appearances, cross-selling advertisements, information from people known to the user, and ebooks.

[0123] In one or more embodiments, the method further includes the steps of reading user data corresponding to a user, and generating virtual content based on both the read data and the read user data. In one or more embodiments, the virtual content is selected from a group consisting of a virtual grocery list, a virtual coupon book, a virtual recipe book, a list of various supplies in the user's home, and a virtual recipe builder. In one or more embodiments, the method further includes the steps of receiving user input, generating additional virtual content based on the user input, and displaying the additional virtual content on a virtual user interface while the user engages in retail activities in a retail store.

[0124] In one or more embodiments, user input is selected from a group consisting of gestures, visual data, auditory data, sensory data, direct commands, voice commands, eye tracking, and physical button selections. In one or more embodiments, virtual content is selected from a group consisting of total operating costs, a smart virtual grocery list, an indicator of items near the user's location, and a virtual payment system. In one or more embodiments, the method further includes the step of sending the generated virtual content to the user device for display.

[0125] In another aspect, the method includes the steps of: reading patient data relating to a patient's medical history; generating virtual content based on at least partially read patient data; reading a first set of map points corresponding to a first location of a first user; creating a first virtual user interface within the first user's field of view based on the at least partially read set of map points; and displaying the virtual content on the first virtual user interface such that, when viewed by the first user, the first virtual user interface appears to be fixed in the first set of map points.

[0126] In one or more embodiments, the virtual content is selected from a group consisting of a three-dimensional image of the surgical target, patient identification information, medical images, patient vital sign information, and medical records. In one or more embodiments, the patient data is read from networked memory.

[0127] In one or more embodiments, the method further includes the step of generating a second virtual user interface configured to facilitate communication between a first user and a second user, wherein the second user is in a second location different from the first user's first location. In one or more embodiments, the second virtual user interface is a visual representation of the second user. In one or more embodiments, the second user is selected from a group consisting of senior specialist surgeons, patients, persons associated with patients, and medical students. In one or more embodiments, the method further includes the step of displaying virtual content on the first virtual user interface to the second user such that, when the first virtual user interface is viewed by the second user, it appears to be fixed in a set of map points.

[0128] In one or more embodiments, virtual content is displayed to a first user during a surgical procedure, and the method further includes the steps of receiving user input, generating additional virtual content based on the user input, and displaying the additional virtual content on a first virtual user interface while the first user is performing a surgical procedure.

[0129] In one or more embodiments, user input is selected from the group consisting of gestures, visual data, auditory data, sensory data, direct commands, voice commands, eye tracking, and selection of physical buttons. In one or more embodiments, user input comprises an image of the first user's field of view, and the method further includes the step of displaying additional virtual content on the first virtual user interface to the second user such that the first virtual user interface appears to be fixed in a set of map points when viewed by the second user.

[0130] In one or more embodiments, a method for facilitating medical rehabilitation includes the steps of: receiving user input related to medical rehabilitation; sending a request related to medical rehabilitation to a cloud server based on the received user input; receiving data related to medical rehabilitation from a knowledge base connected to the cloud server based on the request; and displaying virtual content related to medical rehabilitation to a first user based on the received data.

[0131] In one or more embodiments, user input is selected from the group consisting of visual, auditory, and sensory input. In one or more embodiments, the method further includes the step of determining whether the user input is valid before sending the request to the cloud server. In one or more embodiments, data is received from multiple knowledge bases connected to the cloud server.

[0132] In one or more embodiments, the request instructs a cloud server to retrieve data from a knowledge base. In one or more embodiments, the virtual content includes a relaxing environment. In one or more embodiments, the method further includes the steps of receiving additional data from a second user and displaying additional virtual content to the first user based on the received additional data.

[0133] In one or more embodiments, the first and second users are located in different physical locations. In one or more embodiments, the additional virtual content is a visual representation of the second user. In one or more embodiments, the method further includes the steps of receiving first user data from the first user, receiving second user data from the second user, modifying the virtual content based on the received first and second user data, and displaying the modified virtual content to the first and second users.

[0134] In one or more embodiments, a first user engages in physical activities for medical rehabilitation, and the virtual content relates to the physical activities. In one or more embodiments, the virtual content is selected from a group consisting of information about the physical activities, performance statistics of the first user, a virtual experience corresponding to the physical activities, and a virtual avatar. In one or more embodiments, the data is read from networked memory.

[0135] In another respect, a method for improving task performance includes the steps of: reading task data related to the task to be performed by the user; generating virtual content based on the task data; creating a virtual user interface within the user's field of view; reading a set of map points corresponding to the user's location; and displaying the virtual content on the virtual user interface while the user is performing the task, such that the virtual user interface appears to be fixed in the set of map points when viewed by the user.

[0136] In one or more embodiments, the virtual content is a game having a virtual-mapped pattern, and the game is configured to improve the user's performance on a task. In one or more embodiments, the game awards points for following the virtual-mapped pattern. In one or more embodiments, the game doubles points for reaching a point in the virtual-mapped pattern within a given time.

[0137] In one or more embodiments, the game deducts points for deviating from a virtually mapped pattern. In one or more embodiments, the game deducts points for moving adjacent to a physical object. In one or more embodiments, tasks are selected from a group consisting of operating landscaping machinery, reading inventory items, displaying items on retail shelves, and sorting mail. In one or more embodiments, task data is read from networked memory.

[0138] In one or more embodiments, the method further includes the steps of receiving user input, generating additional virtual content based on the user input, and displaying the additional virtual content on a virtual user interface while the user is performing a task. In one or more embodiments, the user input includes user actions related to the performance of the task.

[0139] In yet another aspect, the augmented reality display system comprises a display device configurable to display one or more virtual images to the user's eyes, and a processor communicatively coupled to the display device configured to provide one or more virtual images to the display device, the processor having a predictive mechanism for predictively correcting timing discrepancies with respect to the display of virtual images.

[0140] In one or more embodiments, the timing mismatch relates to one or more sensor measurements performed by one or more sensors, which are communicatively coupled to a processor. In one or more embodiments, the timing mismatch relates to processing delays when processing incoming data.

[0141] In one or more embodiments, the prediction mechanism uses filters to compensate for the effects of timing mismatches. In one or more embodiments, the filters consider the relative speeds of one or more sensor measurements, and the sensor measurements are performed by one or more sensors that are communicatively coupled to a processor. In one or more embodiments, the prediction mechanism utilizes Kalman predictors.

[0142] In one or more embodiments, the Kalman predictor is used during the display processing stage. In one or more embodiments, the processor makes compensatory changes to the data associated with one or more virtual images to compensate for timing mismatches. In one or more embodiments, the compensatory changes include the step of shifting the data associated with the virtual images.

[0143] In one or more embodiments, the compensation modification includes the step of smoothing one or more visual artifacts associated with the virtual image. In one or more embodiments, the compensation modification includes the step of correcting the negative effects of sensor measurements of one or more sensors, which are communicatively coupled to the processor.

[0144] In another respect, a method for compensating for latency in an augmented reality display system includes the steps of determining one or more timing discrepancies in at least one stage of displaying a virtual image to a user, predictively compensating for the timing discrepancies using a predictive mechanism, and compensating for at least one characteristic related to the data of the virtual image, at least partially based on the predictive mechanism.

[0145] In one or more embodiments, the timing mismatch relates to one or more sensor measurements performed by one or more sensors, which are communicatively coupled to a processor. In one or more embodiments, the timing mismatch relates to processing delays when processing incoming data.

[0146] In one or more embodiments, the prediction mechanism uses a filter to compensate for the effects of timing mismatch. In one or more embodiments, the filter takes into account the relative speed of one or more sensor measurements, and the sensor measurements are performed by one or more sensors that are communicatively coupled to a processor.

[0147] In one or more embodiments, the prediction mechanism utilizes a Kalman predictor. In one or more embodiments, the Kalman predictor is utilized during the display processing stage. In one or more embodiments, compensation includes the step of shifting data associated with the virtual image. In one or more embodiments, compensation includes the step of smoothing one or more visual artifacts associated with the virtual image. In one or more embodiments, compensation includes the step of correcting the negative effects of sensor measurements of one or more sensors, one or more sensors being communicatively coupled to the processor.

[0148] In another respect, a method for calibrating an augmented reality system includes the steps of: displaying a virtual image to a user, wherein the virtual image is configured to be displayed at a known focal length, and the virtual image comprises pixel points; determining where the pixel points are displayed to the user, wherein the pixel locations are calculated, at least in part, based on the location of the user's pupils; and aligning the pixel points of the virtual image with known points in space.

[0149] In one or more embodiments, the step is repeated for multiple pixel points. In one or more embodiments, the location where a pixel point is displayed to the user is calculated based at least partially on the location of a known point in space. In one or more embodiments, the location of the pupil, the location of the pixel point, and the location of the known point in space are collinear.

[0150] In one or more embodiments, the location where pixel points are displayed to the user is modified based on user input. In one or more embodiments, the method further includes the step of creating a game interface such that multiple pixel points are presented to the user. In one or more embodiments, the game interface includes the step of emitting a laser through the user's eye movement.

[0151] In one or more embodiments, the location of a known point is determined based at least partially on data received from one or more world cameras. In one or more embodiments, the step is repeated for another eye of the user. In one or more embodiments, the location of a pixel point is determined based at least partially on a function of the pupil location and the location of the known point. In one or more embodiments, the function comprises a quadratic function.

[0152] In another aspect, an augmented reality display system comprises a display device for displaying a virtual image to a user, wherein the virtual image is configured to be displayed at a known focal length, and the virtual image comprises pixel points; and a processor communicatively coupled to the display device, configured to determine where the pixel points are displayed to the user, calculate the locations of the pixel points based at least partially on the location of the user's pupils, and align the pixel points of the virtual image to known points in space.

[0153] In one or more embodiments, the step is repeated for multiple pixel points. In one or more embodiments, the location where a pixel point is displayed to the user is calculated based at least partially on the location of a known point in space. In one or more embodiments, the location of the pupil, the location of the pixel point, and the location of the known point in space are collinear. In one or more embodiments, the location where a pixel point is displayed to the user is modified based on user input.

[0154] In one or more embodiments, the system further includes a game interface configured to present a plurality of pixel points to the user. In one or more embodiments, the game interface includes the step of emitting a laser through the user's eye movement. In one or more embodiments, the location of known points is determined based at least partially on data received from one or more world cameras.

[0155] In one or more embodiments, the step is repeated for another eye of the user. In one or more embodiments, the location of the pixel point is determined based at least partially on a function of the pupil location and the location of a known point. In one or more embodiments, the function comprises a quadratic function.

[0156] In another respect, a method for displaying a virtual interface includes the steps of: identifying a user interface to be displayed to a user based on user input; displaying an avatar user interface in relation to at least one physical object, wherein the avatar user interface presents at least one virtual user interface element; and selecting at least one virtual user interface element based at least partially on interaction between the user and the avatar user interface.

[0157] In one or more embodiments, user input includes gestures. In one or more embodiments, the method further includes the step of determining the location of a gesture related to the user's augmented reality display system, and the virtual user interface is displayed at the determined location. In one or more embodiments, the avatar is a pre-selected avatar. In one or more embodiments, the avatar resembles the user.

[0158] In one or more embodiments, the avatar is displayed as if standing on a physical object. In one or more embodiments, at least one virtual element comprises an application. In one or more embodiments, the method further includes the steps of: selecting at least one virtual element based at least in part on another user input, wherein at least one virtual element comprises a user intermediary; and displaying another avatar representing the user intermediary in relation to the avatar virtual user interface and a physical object, wherein the interaction between the user and the user intermediary is animated through a virtual interaction between the avatar virtual user interface and the other avatar representing the user intermediary.

[0159] In one or more embodiments, the interaction includes the step of transmitting data between the user's augmented reality system and a computing system corresponding to the user's intermediary. In one or more embodiments, the user input comprises voice commands.

[0160] In another aspect, a method for displaying a virtual interface includes the steps of: identifying a user interface to be displayed to a user based on user input; and displaying a floating user interface relating to at least one physical object, having one or more selectable virtual interface elements, wherein the floating user interface appears to be pushed out from where the user input was received.

[0161] In one or more embodiments, user input includes a user's finger gesture. In one or more embodiments, the user's finger gesture includes touching a finger of one hand with another finger of the user's other hand. In one or more embodiments, the method further includes the step of determining the location where the user's finger of one hand was touched with another finger of the user's other hand, and the floating virtual user interface arises from the determined location.

[0162] In one or more embodiments, the floating user interface comprises a set of three-dimensional blocks, where each three-dimensional block represents at least one selectable virtual user interface element. In one or more embodiments, the floating virtual user interface is created at least partially based on a global standard framework.

[0163] In one or more embodiments, at least one selectable virtual user interface element exists as a stack of three-dimensional blocks, and the stack of three-dimensional blocks is rotated at least partially based on another user input. In one or more embodiments, the method further includes the steps of selecting a virtual user interface element of at least one selectable virtual interface element based at least partially based on another user input received from the user, wherein the virtual interface element is associated with at least one other sub-virtual interface element, and displaying the at least one other sub-virtual interface element below the selected virtual interface element.

[0164] In one or more embodiments, the method further includes the steps of selecting a virtual user interface element of at least one selectable virtual interface element based on another user input received at least partially from the user, and displaying content associated with the selected virtual interface element within a virtual box, wherein the virtual box comprises content that is displayed within the user's field of view.

[0165] In one or more embodiments, the method further includes the steps of: identifying a second user input indicating that a virtual box should be closed; animating the virtual box in a manner similar to crumpling a piece of paper, at least partially based on the second user input; and terminating the content associated with the selected virtual interface element. In one or more embodiments, the second user input includes a gesture, the gesture being similar to crumpling a piece of paper.

[0166] In yet another aspect, a method for displaying a virtual user interface includes the steps of identifying a user interface to be displayed to the user based on user input, and displaying a floating user interface relating to at least one physical object, having one or more selectable virtual interface elements, the floating user interface appearing to be placed on at least one physical object, and the rotation of at least one physical object around the vertical axis of at least one physical object resulting in the display of additional selectable virtual interface elements, the additional selectable virtual interface elements appearing to be stationary on the other side of at least one physical object.

[0167] In one or more embodiments, at least one physical object comprises a user's arm. In one or more embodiments, user input includes a gesture. In one or more embodiments, the gesture includes movements such as forming a cup of the user's hand on the user's arm, which appears to display selectable virtual interface elements.

[0168] In one or more embodiments, the method further includes the step of determining the location of a movement such that it forms a cup of the user's hand on the user's arm, and the floating user interface arises from the determined location. In one or more embodiments, the method further includes the step of terminating the rendering of the floating virtual user interface based at least in part on another user input.

[0169] In one or more embodiments, other user input includes a gesture, which includes the movement of the user's cup-shaped hand dragging across a physical object. In one or more embodiments, a floating virtual user interface is created based at least partially on a hand-centered reference frame. In one or more embodiments, the method further includes the steps of selecting a virtual user interface element of at least one selectable virtual interface element based at least partially on another user input received from the user, and displaying content associated with the selected virtual interface element, wherein the content is displayed in relation to the floating virtual interface.

[0170] In one or more embodiments, the displayed content corresponds to a hand-centered reference frame. In one or more embodiments, the method further includes the step of moving the displayed content from a hand-centered reference frame to a world-centered reference frame so that the displayed content remains stationary when a floating virtual user interface is moving based at least partially on input received from the user.

[0171] In another respect, a method for creating a virtual user interface includes the steps of identifying a user interface to be displayed to the user based on user input, and displaying a virtual user interface relating to at least one physical object, resulting from a point where the user touches at least one physical object, the virtual user interface appears to be popping out from the point where the user touches at least one physical object, and the virtual user interface comprises at least one selectable virtual user interface element that appears to be attached to the virtual user interface.

[0172] In one or more embodiments, at least one physical object has a horizontal surface. In one or more embodiments, user input includes a gesture. In one or more embodiments, the gesture includes an extended finger touching at least one physical object for a predetermined period of time. In one or more embodiments, the method further includes the step of determining where the user's finger touched at least one physical object, and the virtual user interface arises from the determined location.

[0173] In one or more embodiments, the method further includes the step of terminating the rendering of a floating virtual user interface based at least in part on another user input. In one or more embodiments, the other user input includes a gesture, which includes a movement of cutting across the rendered virtual user interface using the user's hand. In one or more embodiments, the virtual user interface is created at least in part on a world-centered reference frame such that the virtual user interface remains stationary for any movement of the user.

[0174] In one or more embodiments, the method further includes the steps of selecting a virtual user interface element of at least one selectable virtual interface element based on another user input received at least partially from the user, and generating a second virtual user interface at another location on at least one physical object, the second virtual user interface comprising additional selectable virtual interface elements. In one or more embodiments, the virtual user interface is analogous to a tree growing from a point where the at least one physical object is touched.

[0175] In another aspect, a method for displaying a virtual user interface includes the steps of: identifying a user interface to be displayed to the user based on user input; depicting a virtual user interface that mirrors the movement of the user's fingers so that the virtual user interface mirrors the movement of the fingers; and displaying one or more selectable user interface elements within the area where the virtual user interface is depicted. In one or more embodiments, a finger gesture includes a gesture that is pointed out using at least one finger of the user's hand for a predetermined period of time.

[0176] In one or more embodiments, the method further includes the step of determining the location of an instruction gesture, and the virtual user interface arises from the determined location. In one or more embodiments, the method further includes the step of detecting that the user is no longer rendering the virtual user interface, and the virtual user interface elements are displayed, at least partially based on the detection. In one or more embodiments, the rendering of the virtual user interface mirrors the user's finger dragging movement across space. In one or more embodiments, the method further includes the step of terminating the rendering of a floating virtual user interface, at least partially based on another user input.

[0177] In one or more embodiments, other user input includes a gesture, which includes the termination of a continuous dragging motion of the user's finger. In one or more embodiments, the virtual user interface is created at least partially based on a world-centered reference frame such that the virtual user interface remains stationary for any movement of the user.

[0178] In one or more embodiments, the method further includes the steps of selecting a virtual user interface element of at least one selectable virtual interface element based on another user input received at least partially from the user, and generating content associated with the selected virtual interface element relating to the virtual user interface. In one or more embodiments, the shape of the virtual user interface is similar to the shape drawn by the user's fingers.

[0179] In another aspect, a method for creating a virtual user interface includes the steps of identifying a user interface to be displayed to the user based on the user's hand gestures, and displaying a virtual user interface having at least one selectable virtual interface element in response to the hand gestures on the vertical side of a physical object in the direction of the hand gestures, such that the virtual user interface appears to be on the vertical side of a physical object.

[0180] In one or more embodiments, the method further includes the step of displaying a collection of virtual content resembling paint splatters in response to a hand gesture, the virtual content occurring at a point indicated by the hand gesture. In one or more embodiments, the hand gesture includes an outstretched finger raised in the direction of the vertical side of a physical object. In one or more embodiments, the method further includes the step of determining the location of the outstretched finger raised in the direction of the vertical side of the physical object, and the collection of virtual content resembling paint splatters is displayed at the determined location.

[0181] In one or more embodiments, the method further includes the step of displaying a virtual spot on the vertical side of a physical object, wherein at least one characteristic of the virtual spot is displayed on the vertical side of the physical object based on a determined location of an outstretched finger swung up in the direction of the vertical side of the physical object, at least partially. In one or more embodiments, the physical object comprises a wall.

[0182] In one or more embodiments, the virtual user interface is created based at least partially on a world-centered reference frame such that the virtual user interface remains stationary for any movement of the user. In one or more embodiments, the method further includes the steps of selecting a virtual user interface element of at least one selectable virtual interface element based at least partially on another user input received from the user, and generating content associated with the selected user interface element in another location on a physical object.

[0183] In one or more embodiments, the method further includes the step of storing content associated with a selected user interface element in a reduced form, the reduced form comprising a virtual band, which is displayed around the user's hand. In one or more embodiments, the virtual band is created at least partially based on a hand-centered reference frame such that the virtual band moves at least partially based on hand movement.

[0184] In one or more embodiments, the method further includes the step of displaying content in its entirety based at least in part on another gesture of the user. In one or more embodiments, the other gesture includes a hand-raising motion on which a virtual band is displayed.

[0185] In another respect, a method for creating a virtual user interface includes the steps of identifying a user interface to be displayed to the user based on the user's hand gestures, and displaying the virtual user interface in response to the hand gestures, the virtual user interface being similar to a horizontal thread having at least one selectable virtual interface element such that at least one selectable virtual interface element moves at least partially based on another hand gesture of the user to display additional selectable virtual interface elements.

[0186] In one or more embodiments, a hand gesture includes a movement in which the first finger of the user's first hand is touched with the second finger of the second hand. In one or more embodiments, the gesture further includes a movement in which both the first and second fingers are pulled away from each other so that the first finger is pulled away from the second finger. In one or more embodiments, a virtual thread mirrors the length of the movement of the first and second fingers, and the length of the virtual thread is at least partially based on the movement of the first and second fingers.

[0187] In one or more embodiments, the method further includes the step of determining touch movements of first and second fingers, from which the virtual threads originate. In one or more embodiments, the gesture of the other hand includes a turning movement of the user's hand, and the turning movement causes at least one selectable virtual interface element to move in the direction of the turning movement.

[0188] In one or more embodiments, the virtual user interface is created based at least partially on a world-centered reference frame such that the virtual user interface remains stationary for any movement of the user. In one or more embodiments, the method further includes the steps of selecting a virtual user interface element of at least one selectable virtual interface element based at least partially on a gesture of the user's other hand, and generating content associated with the selected user interface element in another location on a physical object. In one or more embodiments, the gesture of the other hand includes the movement of bringing one of the user's hands forward.

[0189] In another aspect, a method for creating a virtual user interface includes the steps of: identifying a user interface to be displayed to a user based on a user's hand gesture; and displaying a virtual user interface in response to the hand gesture, wherein the virtual user interface comprises a set of vertical threads, each of which is associated with at least one selectable virtual interface element.

[0190] In one or more embodiments, a hand gesture includes a gesture of extending the user's first hand for a predetermined period of time. In one or more embodiments, the method further includes the step of selecting at least one selectable virtual interface element based at least in part on another hand gesture, the other hand gesture includes a movement of pulling at least one of the virtual threads of a set of virtual threads using two fingers of the user.

[0191] In one or more embodiments, the method further includes the step of transmitting data associated with a first virtual element associated with a first virtual thread to a second virtual element associated with a second virtual thread, at least in part, based on a hand gesture. In one or more embodiments, the gesture of the other hand includes a squeezing motion of the first virtual element associated with the first virtual thread using the second virtual element associated with the second virtual thread, the squeezing motion being performed by the user's fingers.

[0192] In one or more embodiments, the virtual interface is represented in relation to at least one physical object. In one or more embodiments, the at least one physical object comprises a wall. In one or more embodiments, the method further includes the step of determining the location of a hand gesture, from which the virtual thread arises.

[0193] In one or more embodiments, the virtual user interface is created based at least partially on a world-centered reference frame such that the virtual user interface remains stationary for any movement of the user. In one or more embodiments, the method further includes the steps of selecting a virtual user interface element of at least one selectable virtual interface element based at least partially on a gesture of the user's other hand, and generating virtual content associated with the selected user interface element in another location on a physical object.

[0194] In another aspect, a method for creating a virtual user interface includes the steps of: identifying a user interface to be displayed to a user based on a user's hand gesture; and displaying a virtual user interface in response to the hand gesture, wherein the virtual user interface comprises a set of vertical threads, each of which is associated with at least one selectable virtual interface element. In one or more embodiments, the hand gesture includes a gesture of extending the user's first hand for a predetermined period of time.

[0195] In one or more embodiments, the method further includes the step of selecting at least one selectable virtual interface element based at least in part on a gesture of another hand, the gesture of the other hand including a movement of pulling at least one of the virtual threads of a set of virtual threads using two fingers of the user. In one or more embodiments, the method further includes the step of transmitting data associated with a first virtual element associated with a first virtual thread to a second virtual element associated with a second virtual thread, based at least in part on a hand gesture.

[0196] In one or more embodiments, a gesture of the other hand includes a squeezing motion of a first virtual element associated with a first virtual thread using a second virtual element associated with a second virtual thread, the squeezing motion being performed by the user's fingers. In one or more embodiments, the virtual interface is displayed in relation to at least one physical object. In one or more embodiments, the at least one physical object comprises a wall.

[0197] In one or more embodiments, the method further includes the step of determining the location of a hand gesture, from which the virtual thread arises. In one or more embodiments, the virtual user interface is created at least in part on a world-centered reference frame such that the virtual user interface remains stationary for any movement of the user.

[0198] In one or more embodiments, the method further includes the steps of selecting a virtual user interface element of at least one selectable virtual interface element based at least partially on a gesture of the user's other hand, and generating virtual content associated with the selected user interface element in another location on a physical object.

[0199] In yet another aspect, the method for creating a virtual user interface includes the steps of identifying the user interface to be displayed to the user based on the user's hand gestures, and displaying the virtual user interface in response to the hand gestures, where the virtual user interface resembles a virtual spider web, and the user pulls on the virtual threads of the virtual spider web to bring the virtual interface closer to the user.

[0200] In one or more embodiments, a hand gesture includes a pulling motion using the user's fist. In one or more embodiments, the virtual threads of the virtual spider web comprise at least one selectable virtual element. In one or more embodiments, the method further includes the step of selecting at least one selectable virtual interface element at least in part based on another hand gesture, the other hand gesture including a pulling motion of at least one of the virtual threads toward the user.

[0201] In one or more embodiments, the method further includes the step of transmitting data associated with a first virtual element associated with a first virtual thread to a second virtual element associated with a second virtual thread, at least in part, based on a hand gesture. In one or more embodiments, the gesture of the other hand includes a squeezing motion of the first virtual element associated with the first virtual thread using the second virtual element associated with the second virtual thread, the squeezing motion being performed by the user's fingers.

[0202] In one or more embodiments, the virtual interface is displayed in relation to at least one physical object. In one or more embodiments, the at least one physical object comprises a wall. In one or more embodiments, the method further includes the step of determining the location of a hand gesture, from which the virtual spiderweb arises.

[0203] In one or more embodiments, the virtual user interface is created based at least partially on a world-centered reference frame such that the virtual user interface remains stationary for any movement of the user. In one or more embodiments, the method further includes the steps of selecting a virtual user interface element of at least one selectable virtual interface element based at least partially on a gesture of the user's other hand, and generating virtual content associated with the selected user interface element in another location on a physical object.

[0204] Additional and other purposes, features, and advantages of the present invention are described in the detailed description, figures, and claims. This specification also provides, for example, the following items: (Item 1) A method for generating map data, Identifying a set of map points associated with one or more images, Determining the set of low-density points and the set of high-density points from the identified map points, The normalization of the respective sets of low-density and high-density points mentioned above. Methods that include... (Item 2) To generate low-density and high-density point descriptors for each of the aforementioned sets of low-density and high-density points, The low-density and high-density point descriptors are combined and stored as map data. The method described in item 1, further including the method described in item 1. (Item 3) The set of low-density points corresponds to the method described in item 1, wherein the set of low-density points corresponds to an intrinsic feature. (Item 4) The method according to item 3, wherein the aforementioned unique feature is selected from the group consisting of corners, circles, triangles, and text. (Item 5) The method according to item 1, wherein the set of high-density points corresponds to 3D points in the field of view. (Item 6) The set of high-density points also includes color values, as described in item 5. (Item 7) The normalization described above is the method according to item 1, wherein the normalization includes scale normalization. (Item 8) The normalization method according to item 1, wherein the normalization includes coordinate normalization with respect to a common origin. (Item 9) The normalization described above is the method described in item 1, which is implemented using machine learning. (Item 10) The low-density and high-density point descriptors correspond to each low-density and high-density point in each set of low-density and high-density points, according to the method of item 1. (Item 11) The method according to item 10, wherein each low-density and high-density point descriptor includes information about the respective low-density and high-density point, selected from the group consisting of scale, orientation, patch data, and texture. (Item 12) A method for determining user input, The method involves capturing an image of the user's field of view, wherein the image includes gestures created by the user. The captured image is analyzed to identify the set of points associated with the gesture, The process involves comparing the identified set of points with a set of points associated with a predetermined gesture database, Determining user input based on recognized gestures and Methods that include... (Item 13) The method of item 12, further comprising generating a scored value for the set of identified points based on the comparison. (Item 14) The method according to item 13, further comprising recognizing the gesture if the scored value exceeds a threshold. (Item 15) The method of item 12, further comprising storing the predetermined gesture in a database. (Item 16) The method of item 15, further comprising accessing networked memory and accessing a database of predetermined gestures. (Item 17) The gesture is a hand gesture, as described in item 12. (Item 18) The method described in item 12, wherein the gesture is a finger gesture. (Item 19) The gesture is an interdigital interaction, as described in item 12. (Item 20) The method according to item 12, wherein the gesture is selected from the group consisting of interfinger interaction, pointing, tapping, and friction. (Item 21) The method according to item 12, further comprising displaying one or more virtual objects to the user based at least in part on the user input determined above. (Item 22) The method according to item 21, wherein the one or more virtual objects have a virtual user interface. (Item 23) It is a method, To capture one or more images corresponding to the user's field of view, The process involves analyzing one or more captured images and identifying one or more gestures created by the user, wherein the analysis includes utilizing a cascading mechanism having multiple stages. Methods that include... (Item 24) The cascade mechanism comprises a plurality of nodes, each node corresponding to one of the plurality of stages, according to the method of item 23. (Item 25) The cascade mechanism comprises a series of permissive analysis nodes, as described in item 23. (Item 26) The method according to item 23, wherein the earlier of the multiple stages of the cascade mechanism consumes less processing power than the later of the multiple stages of the cascade mechanism. (Item 27) The method of item 26, wherein non-gestures are removed based on an analysis of the images captured in the preceding stage of the cascade mechanism. (Item 28) The method according to item 26, wherein the later of the aforementioned steps determines a more complex gesture based at least partially on the captured image. (Item 29) The method according to item 26, wherein the analysis of the captured image includes determining whether the sharpness of the contours of various shapes in the captured image is sharp enough to constitute a gesture. (Item 30) The method described in item 26, wherein the later stages of the cascade mechanism are used to distinguish between different gestures. (Item 31) The method of item 23, further comprising generating a score based at least in part on the aforementioned analysis. (Item 32) The method according to item 31, further comprising removing a candidate image from consideration if the generated score is lower than a minimum threshold. (Item 33) The method according to item 31, further comprising advancing to the subsequent stage of the cascade mechanism if the generated score is higher than a minimum threshold. (Item 34) It is a method, Capturing multiple images from each user's field of view, The process involves analyzing the aforementioned multiple images and generating multiple gesture candidates. The process involves generating analytical values ​​corresponding to the aforementioned multiple analytical values, wherein the gesture is recognized at least partially based on the analytical values. Methods that include... (Item 35) The method of item 34, further comprising sorting the gesture candidates based at least in part on each of the aforementioned analysis values. (Item 36) The method of item 34, further comprising eliminating gesture candidates having analysis values ​​lower than a minimum threshold. (Item 37) The method according to item 34, further comprising advancing the gesture candidate to the next stage of processing if the analysis value is higher than a minimum threshold. (Item 38) A method for classifying gestures, Capturing images of the user's field of view, The captured image is subjected to depth compartmentalization to generate a depth map, Based at least partially on the generated depth map, the gesture is identified. Methods that include... (Item 39) The method of item 38, further comprising using a classification mechanism to analyze the depth map and to identify a portion of a hand corresponding to a point in the depth map. (Item 40) The method according to item 39, further comprising skeleting the depth map based on the identification of a portion of the hand. (Item 41) The method of item 40, further comprising classifying the image as a gesture based on the skeletal depth map. (Item 42) The depth compartmentalization is the method described in item 38, comprising line search. (Item 43) The method according to item 38, further comprising performing cascade analysis on the depth map and classifying the image as the gesture. (Item 44) The method according to item 38, further comprising performing depth expansion on the depth map. (Item 45) The method according to item 38, further comprising performing surface normalization on the depth map. (Item 46) The method according to item 38, further comprising performing orientation normalization on the depth map. (Item 47) The method according to item 38, further comprising performing background subtraction on the depth map. (Item 48) The method according to item 38, further comprising performing depth comparison on the depth map. (Item 49) The method according to item 38, further comprising classifying the image as a gesture based on the skeletonized depth map and previous information. (Item 50) The method according to item 38, wherein the classification mechanism is a decision forest. (Item 51) A method for determining user input, comprising: capturing an image of the field of view of one or more users, the image comprising at least a predetermined physical object; analyzing the image to detect the movement of the user related to the predetermined physical object; determining user input based at least in part on the analysis of the movement related to the predetermined physical object. The method includes. (Item 52) The method according to item 51, further comprising recognizing the predetermined physical object. (Item 53) The method according to item 52, wherein the predetermined physical object is recognized based at least in part on an image comparison between a captured image of the predetermined physical object and a database of the predetermined physical object. (Item 54) The method according to item 51, wherein the movement of the user related to the predetermined physical object is used to generate a pattern. (Item 55) The method according to item 54, further comprising comparing the generated pattern with a database of predetermined patterns. (Item 56) The method according to item 51, further comprising generating a scoring value for the recognized pattern based on the comparison. (Item 57) The method according to item 56, further comprising determining the user input if the scoring value exceeds a threshold. (Item 58) The method according to item 51, further comprising visually tracking the movement related to the predetermined physical object and generating a video recording. (Item 59) The method according to item 58, wherein the video recording is analyzed to determine the user input. (Item 60) The method according to item 55, further comprising accessing the database of the predetermined patterns through a networked memory. (Item 61) The method according to item 51, wherein the predetermined physical object is selected from the group consisting of an existing structure within the field of view, an actively marked totem, a passively marked totem, an object integrated into a camera / sensor, and a totem controller object. (Item 62) The method according to item 51, wherein the movement related to the predetermined physical object is selected from the group consisting of the position, orientation, and movement of the predetermined physical object with respect to a reference frame. (Item 63) The method according to item 51, wherein the predetermined physical object comprises a first hand of the user, and movement relating to the first hand comprises manipulation of the first hand using a second hand of the user. (Item 64) The method according to item 51, wherein the predetermined physical object has a soft-hard surface, and movement relating to the predetermined physical object comprises pressing the soft-hard surface by the user. (Item 65) The method according to item 51, further comprising rendering a virtual interface element associated with the predetermined physical object for the user. (Item 66) The method of item 65, wherein the virtual interface element is displayed in relation to the predetermined physical object such that, when viewed by the user, the virtual interface element is modified, at least in part, based on modifications relating to the predetermined physical object. (Item 67) The method according to item 51, wherein the predetermined physical object comprises an electronic input device, and user input is determined based on the recognized movement of the predetermined physical object and input from the electronic input device. (Item 68) The method according to item 51, further comprising modifying at least one characteristic of the virtual content displayed to the user based at least in part on the user input determined above. (Item 69) A method for generating a virtual user interface, Based on user input, identify the virtual user interface that should be displayed to the user, To obtain the location associated with the aforementioned user, Based at least partially on the locations obtained, determine the set of coordinate points on which the identified virtual user interface should be displayed. The virtual user interface is to be displayed to the user at the determined coordinate point. Methods that include... (Item 70) The method according to item 69, wherein the user input is determined at least in part based on a recognized gesture. (Item 71) The method according to item 69, wherein the user input is determined at least in part based on a voice command. (Item 72) The method according to item 69, wherein the user input is determined at least in part on the basis of interaction with a predetermined physical object. (Item 73) The method of item 69, further comprising reading the identified user virtual user interface from a user interface library. (Item 74) The identified virtual user interface is associated with a reference frame, as described in item 69. (Item 75) The aforementioned reference frame is a body-centered reference frame, as described in item 74. (Item 76) The aforementioned reference frame is a head-centered reference frame, as described in item 74. (Item 77) The aforementioned reference frame is a hand-centered reference frame, as described in item 74. (Item 78) The aforementioned standard frame is a globally central standard frame, as described in item 74. (Item 79) The method of item 69, further comprising performing a transformation between a reference frame associated with the identified virtual user interface and the user's obtained location relating to the world, and determining a set of coordinate points of the virtual user interface. (Item 80) The user's location within the world is determined based on the user's GPS location, as described in item 69. (Item 81) The method according to item 69, wherein the location of the user within the world is determined based on a set of map points associated with the user. (Item 82) The method according to item 69, wherein the virtual user interface appears to be stationary when the user moves. (Item 83) The method according to item 69, wherein the virtual user interface moves in relation to the movement of the user. (Item 84) Determining a reference frame associated with the identified virtual user interface; Determining the location of the reference frame in relation to the world reference frame; Setting the determined location as the origin; Determining a set of coordinate points in relation to the origin; The method according to item 69, further comprising the above steps. (Item 85) The method according to item 69, further comprising reading the set of map points from a networked memory. (Item 86) The method according to item 69, wherein the user input includes a location within the space where the virtual user interface is to be displayed. (Item 87) The method according to item 86, wherein the location within the space is associated with a physical entity at the location of the user. (Item 88) The method according to item 69, wherein the user input includes a gesture including a throwing input indicating a wall. (Item 89) The method according to item 69, wherein the user input includes a gesture meaning the end of an instruction for generating the virtual user interface. (Item 90) A method for generating a virtual user interface, comprising: Detecting an operation of a predetermined physical object; Based on the detected operation, recognize the command for creating a virtual user interface, From the virtual world model, determine the set of map points associated with the location of the predetermined physical object, When the virtual user interface is viewed by the user, it is rendered in real time to the determined map point associated with the totem's position so that it appears to be stationary at the location of the predetermined physical object. Methods that include... (Item 91) The method according to item 90, wherein the manipulation of the predetermined physical object includes a gesture of releasing the state in which the user's hand is holding the predetermined physical object on its surface. (Item 92) The method according to item 91, wherein the virtual user interface, when viewed by the user, appears to cover a portion of the surface of the predetermined physical object, the portion corresponding to the position of the user's hand while forming the gesture of unpinning the pinched state. (Item 93) The method according to item 90, wherein the predetermined physical object is the user's hand. (Item 94) The method according to item 93, wherein the operation of the predetermined physical object comprises an action selected from the group consisting of the user spreading their hand, the user showing their palm, and the user raising their hand. (Item 95) The method according to item 94, wherein the virtual user interface, when viewed by the user, appears to cover a portion of the surface of the hand. (Item 96) The method according to item 95, wherein the virtual user interface comprises a plurality of first-level menu items that can be selected by the fingers or thumb of the hand. (Item 97) To detect further movements of the hand, Based on the further operations detected, recognize the command for creating a second virtual user interface, When viewed by the user, the second virtual user interface is rendered in real time at the determined map point associated with the location of the predetermined physical object, so that the virtual user interface appears to be stationary at the location of the predetermined physical object. The method described in item 96, further including the method described in item 96. (Item 98) The method according to item 97, further manipulation of the hand, including spreading the fingers of the hand apart. (Item 99) The method according to item 98, wherein the second virtual user interface comprises a plurality of second-level menu items selectable by the fingers or thumb, the second level being lower than the first level. (Item 100) The method according to item 97, further manipulation of the totem includes performing a motion of drawing a circle on the palm using the fingers from the second hand of the user's hand. (Item 101) The method according to item 100, wherein the second virtual user interface comprises a plurality of menu items arranged in an arc, and the menu items are scrollable and selectable by the fingers of the second hand. (Item 102) A method for updating the virtual world, Receiving a first input from a first device of a first user, wherein the first input corresponds to the physical environment of the first user. The process involves updating a virtual world model based on the received first input, wherein the virtual world model corresponds to the physical environment of the first user. Transmitting first updated information corresponding to the first part of the virtual world model to a second user. Includes, A method for indicating whether the first updated information should be displayed to the second user. (Item 103) The aforementioned virtual world model resides on networked memory, as described in item 102. (Item 104) The first user and the second user are located in different locations, as described in item 102. (Item 105) The method according to item 102, wherein the first device of the first user is selected from the group consisting of an FOV camera, other cameras, sensors, a first device for eye tracking, and a first device for audio. (Item 106) The method according to item 102, further comprising transmitting the first updated information corresponding to a first portion of the virtual world model to the first user, wherein the first updated information indicates whether any portion of the first updated information should be displayed to the first user. (Item 107) The method according to item 102, further comprising transmitting the first updated information corresponding to a first portion of the virtual world model to a plurality of other users, wherein the first updated information indicates whether any portion of the first updated information should be displayed to each of the plurality of other users. (Item 108) Receiving multiple inputs from the respective first devices of multiple other users, wherein the multiple inputs correspond to the physical environment of the first user, The virtual world model is updated based on the multiple inputs received, Transmitting additional updated information corresponding to each additional part of the virtual world model to the second user It further includes, The method described in item 102, which indicates whether any part of the additional updated information needs to be displayed to the second user. (Item 109) The method of item 108, further comprising transmitting the additional updated information corresponding to the portion of the virtual world model to the first user, wherein the additional updated information indicates whether any portion of the additional updated information should be displayed to the first user. (Item 110) The method of item 108, further comprising transmitting the additional updated information corresponding to each additional part of the virtual world model to the plurality of other users, wherein the additional updated information indicates whether any part of the additional updated information should be displayed to each of the plurality of other users. (Item 111) Receiving a second input from the second user's second device, wherein the second input corresponds to the second user's physical environment, Updating the virtual world model based on the received second input, wherein the virtual world model corresponds to the physical environment of the first user. Transmitting to the first user second updated information corresponding to the second part of the virtual world model It further includes, The method described in item 102, which indicates whether any part of the second updated information should be displayed to the first user. (Item 112) The method according to item 111, wherein the second updated information corresponds to the movement of the second user's avatar within the second part of the virtual world model. (Item 113) It is a method, Projecting light patterns into space, Using a camera, first and second portions of the light pattern are detected, and first and second data points corresponding to the first and second portions of the pattern are generated. Perform triangulation analysis to determine the locations of the first and second portions of the light pattern, at least partially based on the first and second data points. Methods that include... (Item 114) The method according to item 113, further comprising the steps of projecting, detecting, and triangulating multiple parts of the pattern to obtain additional texture data of the space. (Item 115) Projecting the light into space is the method of item 113, which includes projecting the light into space using a fiber-based projector. (Item 116) The method according to item 113, further comprising modifying the light using an optical element to form a beam of light rays before the light is projected onto the space. (Item 117) The method according to item 113, wherein the light pattern corresponds to structured light, and the light pattern is dynamic. (Item 118) The method according to item 113, wherein the light pattern corresponds to patterned light, and the light pattern includes a plurality of points. (Item 119) The method according to item 113, wherein the light pattern corresponds to textured light, and the light pattern is irregular. (Item 120) The method according to item 113, wherein the first and second portions of the light pattern are the respective pixels of the light pattern. (Item 130) A method for generating a virtual room, Identifying the virtual room that should be displayed to the user, The objective is to obtain a set of first map points associated with the virtual room, wherein the set of first map points corresponds to a first physical room at the user's location. Tethering the aforementioned virtual room to the aforementioned set of map points, Displaying the virtual room to the user such that, when viewed by the user, the virtual room appears to be stationary in the first set of map points. Methods that include... (Item 131) The method according to item 130, further comprising identifying the virtual room based on user input. (Item 132) The method according to item 131, wherein the user input is selected from the group consisting of gestures, visual data, auditory data, sensory data, direct commands, voice commands, eye tracking, and selection of physical buttons. (Item 133) The virtual room is determined in advance by the user, according to the method described in item 131. (Item 134) The method according to item 131, further comprising reading the first set of map points from a networked memory. (Item 135) Obtaining a second set of map points associated with the virtual room, wherein the second set of map points corresponds to the second physical room at the user's location after the user has moved from the first physical room. Tethering the aforementioned virtual room to the aforementioned second set of map points, Displaying the virtual room to the user such that, when viewed by the user, the virtual room appears to be stationary in the second set of map points. The method described in item 131, further including the method described in item 131. (Item 136) The method according to item 135, wherein the virtual room also includes first and second virtual objects, and the method further includes maintaining the relative spatial positions between the first virtual object and the second virtual object corresponding to the first set of map points if the virtual room is tethered to the second set of map points. (Item 137) The method according to item 131, further comprising displaying the virtual room to the user such that, when viewed by the user, the virtual room appears to be stationary relative to a portion of the first physical room. (Item 138) The virtual room also includes selectable virtual decorations, as described in item 131. (Item 139) A method for creating a retail experience, Recognizing the user's location within a retail store, Retrieving data corresponding to the aforementioned retail store, Based on the data read out, generate virtual content related to the retail store, Creating a virtual user interface within the user's field of view, The virtual content is displayed on the virtual user interface while the user engages in retail activities within the retail store. Methods that include... (Item 140) The data read out comprises a set of map points corresponding to the retail store, The method according to item 139, wherein the virtual user interface appears to be stationary in the set of map points when viewed by the user. (Item 141) The user's location is identified using a radio frequency identification transponder and communication as described in item 139. (Item 142) The method according to item 139, wherein the virtual content is selected from the group consisting of virtual characters, virtual coupons, games based on the user's location, a list of promotional items, nutritional information, metadata related to the items, appearances by celebrities, cross-selling advertisements, information from people known to the user, and ebooks. (Item 143) Reading user data corresponding to the aforementioned user, The virtual content is generated based on both the read data and the read user data. The method described in item 139, which further includes the method described in item 139. (Item 144) The method according to item 143, wherein the virtual content is selected from a group consisting of a virtual grocery list, a virtual coupon book, a virtual recipe book, a list of various materials in the user's house, and a virtual recipe builder. (Item 145) Receiving user input and Based on the user input, additional virtual content is generated, The additional virtual content is displayed on the virtual user interface while the user engages in retail activities within the retail store. The method described in item 139, which further includes the method described in item 139. (Item 146) The method according to item 145, wherein the user input is selected from the group consisting of gestures, visual data, auditory data, sensory data, direct commands, voice commands, eye tracking, and selection of physical buttons. (Item 147) The method according to item 145, wherein the virtual content is selected from the group consisting of total operating costs, a smart virtual grocery list, an indicator of items close to the user's location, and a virtual payment system. (Item 148) The method according to item 139, further comprising sending the generated virtual content to a user device for display. (Item 149) It is a method, Retrieving patient data related to the patient's medical history, To generate virtual content based at least partially on the aforementioned retrieved patient data, Reading a set of first map points corresponding to the first location of the first user, Based at least partially on the set of map points read out, a first virtual user interface is created within the field of view of the first user, Displaying the virtual content on the first virtual user interface such that, when viewed by the first user, the first virtual user interface appears to be fixed in the set of first map points. Methods that include... (Item 150) The method according to item 149, wherein the virtual content is selected from the group consisting of a three-dimensional image of the surgical target, patient identification information, medical images, patient vital sign information, and medical records. (Item 151) The patient data is read from networked memory, as described in item 149. (Item 152) The method according to item 149, further comprising generating a second virtual user interface and facilitating communication between the first user and the second user, wherein the second user is in a second location different from the first location of the first user. (Item 153) The method described in item 152, wherein the second virtual user interface is a visual representation of the second user. (Item 154) The method according to item 152, wherein the second user is selected from a group consisting of senior specialist surgeons, patients, persons associated with the patient, and medical students. (Item 155) The method of item 152, further comprising displaying the virtual content on the first virtual user interface to the second user such that, when viewed by the second user, the first virtual user interface appears to be fixed in the set of map points. (Item 156) The virtual content is displayed to the first user during the surgical procedure, and the method is Receiving user input and Based on the user input, additional virtual content is generated, The additional virtual content is displayed on the first virtual user interface while the first user is performing the surgical procedure. The method described in item 149, further including the method described in item 149. (Item 157) The method according to item 156, wherein the user input is selected from the group consisting of gestures, visual data, auditory data, sensory data, direct commands, voice commands, eye tracking, and selection of physical buttons. (Item 158) The method according to item 156, wherein the user input comprises an image of the first user's field of view, and the method further includes displaying the additional virtual content on the first virtual user interface to the second user such that, when viewed by the second user, the first virtual user interface appears to be fixed in the set of map points. (Item 159) A method for promoting medical rehabilitation, Receiving user input related to medical rehabilitation, Based on the user input received, the system sends requests related to medical rehabilitation to the cloud server. Based on the aforementioned request, data related to medical rehabilitation is received from a knowledge base connected to the cloud server, Based on the received data, virtual content related to medical rehabilitation is displayed to the first user. Methods that include... (Item 160) The user input is selected from the group consisting of visual, auditory, and sensory inputs, as described in item 159. (Item 161) The method according to item 159, further comprising determining whether the user input is valid before sending the request to the cloud server. (Item 162) The data is received from multiple knowledge bases connected to the cloud server, as described in item 159. (Item 163) The method described in item 159, wherein the request instructs the cloud server to read data from the knowledge base. (Item 164) The virtual content includes a relaxing environment, as described in item 159. (Item 165) Receiving additional data from a second user, Based on the additional data received, additional virtual content is displayed to the first user. The method described in item 159, which further includes the method described in item 159. (Item 166) The first and second users are located in different physical locations, as described in item 165. (Item 167) The additional virtual content is the visual representation of the second user, as described in item 165. (Item 168) Receiving first user data from the first user, Receiving second user data from the second user, Based on the first and second user data received, the virtual content is modified, Displaying the modified virtual content to the first and second users. The method described in item 159, which further includes the method described in item 159. (Item 169) The first user engages in physical activities for medical rehabilitation, and the virtual content relates to the physical activities as described in item 159. (Item 170) The method according to item 169, wherein the virtual content is selected from the group consisting of information about the physical activity, performance statistics of the first user, a virtual experience corresponding to the physical activity, and a virtual avatar. (Item 171) The aforementioned data is read from networked memory, as described in item 159. (Item 172) A method for improving task performance, Retrieving task data related to tasks that should be performed by the user, Based on the aforementioned task data, generate virtual content, Creating a virtual user interface within the user's field of view, This involves reading the set of map points corresponding to the user's location, The virtual content is displayed on the virtual user interface while the user is performing the task, such that when viewed by the user, the virtual user interface appears to be fixed in the set of map points. Methods that include... (Item 173) The method according to item 172, wherein the virtual content is a game having a virtual mapped pattern, and the game improves the user's performance on the task. (Item 174) The game is as described in item 173, which awards points for following the virtual mapped pattern. (Item 175) The method according to item 174, wherein the game doubles points for reaching a point in the virtual mapped pattern within a predetermined time. (Item 176) The game is as described in item 175, wherein points are deducted for deviating from the virtual mapped pattern. (Item 177) The aforementioned game is the method described in item 175, which deducts points for moving adjacent to a physical object. (Item 178) The method according to item 172, wherein the task is selected from the group consisting of operating landscaping machinery, reading out inventory items, displaying items on retail shelves, and sorting mail. (Item 179) The task data is read from networked memory, as described in item 172. (Item 180) Receiving user input and Based on the user input, additional virtual content is generated, The additional virtual content is displayed on the virtual user interface while the user is performing the task. The method described in item 172, further including the method described in item 172. (Item 181) The method according to item 180, wherein the user input comprises user actions related to the performance of the task. (Item 182) A method for compensating for latency in an augmented reality display system, To determine one or more timing discrepancies in at least one stage of displaying a virtual image to the user, By using a prediction mechanism, the timing mismatch is corrected predictively, Compensating for at least one characteristic related to the data of the virtual image based at least in part on the prediction mechanism. Methods that include... (Item 183) The method according to item 182, wherein the timing mismatch involves one or more sensor measurements performed by one or more sensors, and the one or more sensors are communicatively coupled to the processor. (Item 184) The aforementioned timing mismatch relates to the processing delay when processing incoming data, as described in item 182. (Item 185) The prediction mechanism is the method described in item 182, which uses a filter to correct for the effects of the timing mismatch. (Item 186) The method according to item 185, wherein the filter takes into account the relative speed of one or more sensor measurements, and the sensor measurements are performed by one or more sensors that are communicably coupled to the processor. (Item 187) The aforementioned prediction mechanism is the method described in item 182, which utilizes a Kalman predictor. (Item 188) The Kalman predictor is used during the display processing stage, as described in item 187. (Item 189) The compensation is as described in item 182, which includes shifting the data associated with the virtual image. (Item 190) The compensation is as described in item 182, comprising smoothing one or more visual artifacts associated with the virtual image. (Item 191) The method according to item 182, wherein the compensation includes correcting for negative effects of sensor measurements of one or more sensors, the one or more sensors being communicably coupled to the processor. (Item 192) A method for calibrating an augmented reality system, Displaying a virtual image to the user, wherein the virtual image is displayed at a known focal length, and the virtual image comprises pixel points. The location of the pixel point displayed to the user is determined, and the location of the pixel is calculated based at least partially on the location of the user's pupil. Aligning the pixel points of the virtual image with known points in space Methods that include... (Item 193) The method described in item 192, wherein the steps in item 331 are repeated for multiple pixel points. (Item 194) The location of the pixel point displayed to the user is calculated based at least partially on the location of a known point in space, according to the method of item 192. (Item 195) The method according to item 192, wherein the location of the pupil, the location of the pixel point, and the location of the known point in space are collinear. (Item 196) The location of the pixel point displayed to the user is modified based on user input, as described in item 192. (Item 197) The method according to item 196, further comprising creating a game interface such that multiple pixel points are presented to the user. (Item 198) The method according to item 197, wherein the game interface includes emitting a laser through the user's eye movement. (Item 199) The location of the known point is determined at least in part based on data received from one or more world cameras, as described in item 197. (Item 200) The method according to item 192, wherein the step of item 1 is repeated for the other eye of the user. (Item 201) The method according to item 193, wherein the location of the pixel point is determined at least in part on a function of the location of the pupil and the location of the known point. (Item 202) The method according to item 201, wherein the function comprises a quadratic function. (Item 203) A method for displaying a virtual interface, Based on user input, identify the user interface that should be displayed to the user, Displaying an avatar user interface in relation to at least one physical object, wherein the avatar user interface presents at least one virtual user interface element, Selecting at least one virtual user interface element based at least partially on the interaction between the user and the avatar user interface. Methods that include... (Item 204) The user input is the method described in item 203, including gestures. (Item 205) The method according to item 204, further comprising determining the location of the gesture relating to the user's augmented reality display system, wherein the virtual user interface is displayed at the determined location. (Item 206) The method according to item 203, wherein the avatar is a pre-selected avatar. (Item 207) The avatar is similar to the user, as described in item 203. (Item 208) The method according to item 203, wherein the avatar is displayed as if standing on the physical object. (Item 209) The method according to item 203, wherein the at least one virtual element comprises an application. (Item 210) Selecting at least one virtual element based at least in part on another user input, wherein the at least one virtual element contains the user's contact information, In relation to the avatar virtual user interface and the physical object, another avatar representing the user's contacts is displayed, and the interaction between the user and the user's contacts is animated through virtual interaction between the avatar virtual user interface and the other avatar representing the user's contacts. The method described in item 203, further including the method described in item 203. (Item 211) The method according to item 210, wherein the interaction includes transmitting data between the user's augmented reality system and a computing system corresponding to the user's contacts. (Item 212) The user input is provided by the method described in item 203, wherein the user input includes voice commands. (Item 213) A method for displaying a virtual interface, Based on user input, identify the user interface that should be displayed to the user, Displaying a floating user interface relating to at least one physical object, comprising one or more selectable virtual interface elements, wherein the floating user interface appears to be pushed out from where the user input was received. Methods that include... (Item 214) The user input is the method described in item 213, including a finger gesture of the user. (Item 215) The method according to item 214, wherein the user's finger gesture includes touching a finger of one of the user's hands with another finger of the user's other hand. (Item 216) The method according to item 215, further comprising determining the location where a finger of one of the user's hands is touched by another finger of the user's other hand, wherein the floating virtual user interface arises from the determined location. (Item 217) The method according to item 213, wherein the floating user interface comprises a set of three-dimensional blocks, and each three-dimensional block in the set of three-dimensional blocks represents the at least one selectable virtual user interface element. (Item 218) The floating virtual user interface is created according to the method described in item 213, at least in part, based on a global standard framework. (Item 219) The method according to item 213, wherein the at least one selectable virtual user interface element exists as a stack of three-dimensional blocks, and the stack of three-dimensional blocks is rotated at least partially based on another user input. (Item 220) Selecting a virtual user interface element of at least one selectable virtual interface element based at least in part on another user input received from the user, wherein the virtual interface element is associated with at least one other sub-virtual interface element. Displaying at least one other sub-virtual interface element below the selected virtual interface element The method described in item 213, further including the method described in item 213. (Item 221) Selecting a virtual user interface element of at least one selectable virtual interface element based at least partially on another user input received from the user, Displaying content associated with the selected virtual interface element within a virtual box, wherein the virtual box contains content that is displayed within the user's field of view. The method described in item 213, further including the method described in item 213. (Item 222) Identifying a second user input indicating that the virtual box should be closed, Based at least in part on the second user input, the virtual box is animated in a manner similar to crumpling up a piece of paper, To terminate the content associated with the selected virtual interface element. The method described in item 221, further including the method described in item 221. (Item 223) The method described in item 222, wherein the second user input includes a gesture, the gesture being similar to crumpling a piece of paper into a ball. (Item 224) A method for displaying a virtual user interface, Based on user input, identify the user interface that should be displayed to the user, To display a floating user interface relating to at least one physical object, and having one or more selectable virtual interface elements. Includes, The floating user interface appears to be placed on the at least one physical object, and rotation of the at least one physical object around its vertical axis results in the appearance of additional selectable virtual interface elements, which appear to be placed on the other side of the at least one physical object. (Item 225) The method according to item 224, wherein the at least one physical object comprises the user's arm. (Item 226) The user input is the method described in item 224, including gestures. (Item 227) The method according to item 226, wherein the gesture includes an action such as forming a cup of the user's hand on the user's arm so that the selectable virtual interface element appears to be displayed. (Item 228) The method according to item 227, further comprising determining the location of a movement on the user's arm that forms a cup of the user's hand, the floating user interface arising from the determined location. (Item 229) The method of item 224, further comprising terminating the rendering of the floating virtual user interface based at least in part on another user input. (Item 230) The method according to item 229, wherein the other user input includes a gesture, the gesture includes the movement of the user's cup-shaped hand dragging across the physical object. (Item 231) The floating virtual user interface is created based at least in part on a hand-centered reference frame, as described in item 224. (Item 232) Selecting a virtual user interface element of at least one selectable virtual interface element based at least partially on another user input received from the user, Displaying content associated with the selected virtual interface element, wherein the content is displayed in relation to the floating virtual interface. The method described in item 224, further including the method described in item 224. (Item 233) The displayed content corresponds to the hand-centered reference frame, as described in item 231. (Item 234) The method according to item 231, further comprising moving the displayed content from the hand-centered reference frame to the world-centered reference frame so that the displayed content remains stationary if the floating virtual user interface is moving based at least in part on input received from the user. (Item 235) A method for creating a virtual user interface, Based on user input, identify the user interface that should be displayed to the user, Relating to at least one physical object, displaying a virtual user interface arising from a point of touch on the at least one physical object, wherein the virtual user interface appears to protrude from the point of touch on the at least one physical object, and the virtual user interface comprises at least one selectable virtual user interface element that appears to be attached to the virtual user interface. Methods that include... (Item 236) The method according to item 235, wherein the at least one physical object has a horizontal surface. (Item 237) The user input is the method described in item 235, including gestures. (Item 238) The method according to item 237, wherein the gesture includes touching the at least one physical object with an extended finger for a predetermined period of time. (Item 239) The method according to item 235, further comprising determining the location where the user's finger touched the at least one physical object, and the virtual user interface arising from the determined location. (Item 240) The method of item 235, further comprising terminating the rendering of the floating virtual user interface based at least in part on another user input. (Item 241) The method according to item 240, wherein the other user input includes a gesture, the gesture includes a cutting motion using the user's hand across the rendered virtual user interface. (Item 242) The method of item 235, wherein the virtual user interface is created at least in part based on a world-centered reference frame such that the virtual user interface remains stationary for any movement of the user. (Item 243) Selecting a virtual user interface element of at least one selectable virtual interface element based at least partially on another user input received from the user, The method involves generating a second virtual user interface in another location of the at least one physical object, wherein the second virtual user interface comprises additional selectable virtual interface elements. The method described in item 235, further including the method described in item 235. (Item 244) The method according to item 235, wherein the virtual user interface resembles a tree growing from the point where the user touches the at least one physical object. (Item 245) A method for displaying a virtual user interface, Based on user input, identify the user interface that should be displayed to the user, To depict a virtual user interface that mirrors the movement of the user's fingers, such that the virtual user interface mirrors the movement of the fingers, Displaying one or more selectable user interface elements within the area where the virtual user interface is depicted. Methods that include... (Item 246) The method according to item 245, wherein the finger gesture includes a gesture of pointing with at least one finger of the user's hand for a predetermined period of time. (Item 247) The method according to item 246, further comprising determining the location of the instruction gesture, wherein the virtual user interface arises from the determined location. (Item 248) The method according to item 245, further comprising detecting that the user is no longer depicting the virtual user interface, and the virtual user interface elements are displayed based at least in part on the detection. (Item 249) The depiction of the virtual user interface mirrors the dragging motion of the user's finger across space, as described in item 245. (Item 250) The method of item 245, further comprising terminating the rendering of a floating virtual user interface based at least in part on another user input. (Item 251) The method according to item 250, wherein the other user input includes a gesture, the gesture includes ending a continuous dragging motion of the user's finger. (Item 252) The method of item 245, wherein the virtual user interface is created at least in part based on a world-centered reference frame such that the virtual user interface remains stationary for any movement of the user. (Item 253) Selecting a virtual user interface element from the at least one selectable virtual interface element based at least partially on another user input received from the user, To generate content associated with the selected virtual interface element in relation to the virtual user interface. The method described in item 245, further including the method described in item 245. (Item 254) The method according to item 245, wherein the shape of the virtual user interface resembles the shape drawn by the user's fingers. (Item 255) A method for creating a virtual user interface, Identifying the user interface to be displayed to the user based on the user's hand gestures, In response to the hand gesture, a virtual user interface is displayed on the vertical side of the physical object in the direction of the hand gesture, such that the virtual user interface appears to be on the vertical side of the physical object. Methods that include... (Item 256) The method of item 255, further comprising displaying a set of virtual content resembling paint spots in response to the hand gesture, wherein the virtual content occurs at the point indicated by the hand gesture. (Item 257) The method according to item 255, wherein the hand gesture includes an extended finger swung upward in the direction of the vertical side of the physical object. (Item 258) The method according to item 257, further comprising determining the location of an outstretched finger raised in the direction of the vertical side of the physical object, wherein a collection of virtual content resembling paint spots is displayed at the determined location. (Item 259) The method according to item 258, further comprising displaying a virtual spot on the vertical side of the physical object, wherein at least one characteristic of the virtual spot is that it is displayed on the vertical side of the physical object at least partially based on a determined location of an outstretched finger raised in the direction of the vertical side of the physical object. (Item 260) The physical object is provided with walls, as described in item 255. (Item 261) The method of item 255, wherein the virtual user interface is created at least in part based on a world-centered reference frame such that the virtual user interface remains stationary for any movement of the user. (Item 262) Selecting a virtual user interface element from the at least one selectable virtual interface element based at least partially on another user input received from the user, To generate content associated with the selected user interface element in another location of the physical object. The method described in item 255, further including the method described in item 255. (Item 263) The method according to item 2622, further comprising storing the content associated with the selected user interface element in a reduced form, wherein the reduced form comprises a virtual band, and the virtual band is displayed around the user's hand. (Item 264) The method according to item 263, wherein the virtual band is created at least partially based on a hand-center reference frame such that the virtual band moves at least partially based on the movement of the hand. (Item 265) The method of item 263, further comprising displaying the content in its entirety based at least in part on another gesture of the user. (Item 266) The other gestures include the method of item 265, which includes the raising motion of the hand on which the virtual band is displayed. (Item 267) A method for creating a virtual user interface, Identifying the user interface to be displayed to the user based on the user's hand gestures, In response to the aforementioned hand gesture, a virtual user interface is displayed. Includes, The virtual user interface is similar to a horizontal thread having at least one selectable virtual interface element, such that at least one selectable virtual interface element moves and displays additional selectable virtual interface elements based at least partially on the user's other hand gesture. (Item 268) The method according to item 267, wherein the hand gesture includes the movement of touching the first finger of the user's first hand with the second finger of the second hand. (Item 269) The method according to item 268, wherein the gesture further includes the movement of pulling both the first finger and the second finger apart so that the first finger is separated from the second finger. (Item 270) The method according to item 269, wherein the virtual thread mirrors the length of the movement of the first and second fingers, and the length of the virtual thread is based at least in part on the movement of the first and second fingers. (Item 271) The method according to item 267, further comprising determining the location of the touch motion between the first finger and the second finger, wherein the virtual thread arises from the determined location. (Item 272) The method according to item 267, wherein the gesture of the other hand includes a repositioning movement of the user's hand, and the repositioning movement causes the at least one selectable virtual interface element to move in the direction of the repositioning movement. (Item 273) The method of item 267, wherein the virtual user interface is created at least in part based on a world-centered reference frame such that the virtual user interface remains stationary for any movement of the user. (Item 274) Selecting a virtual user interface element from the at least one selectable virtual interface element based at least partially on the user's other hand gesture, To generate content associated with the selected user interface element in another location of the physical object. The method described in item 267, further including the method described in item 267. (Item 275) The method according to item 274, wherein the gesture of the other hand includes the movement of bringing one of the user's hands forward. (Item 276) A method for creating a virtual user interface, Identifying the user interface to be displayed to the user based on the user's hand gestures, In response to the aforementioned hand gesture, a virtual user interface is displayed. Includes, The method wherein the virtual user interface comprises a set of vertical threads, each of which is associated with at least one selectable virtual interface element. (Item 277) The method according to item 276, wherein the hand gesture includes a gesture of extending the user's first hand for a predetermined period of time. (Item 278) The method according to item 277, further comprising selecting the at least one selectable virtual interface element at least in part on a gesture of another hand, wherein the gesture of the other hand includes a movement of pulling at least one of the virtual threads of the set of virtual threads using two fingers of the user. (Item 279) The method according to item 276, further comprising transmitting data associated with a first virtual element associated with a first virtual thread to a second virtual element associated with a second virtual thread, at least in part, based on a hand gesture. (Item 280) The method according to item 279, wherein the gesture of the other hand includes a squeezing motion of the first virtual element associated with the first virtual thread using the second virtual element associated with the second virtual thread, and the squeezing motion is performed by the user's fingers. (Item 281) The virtual interface is displayed in relation to at least one physical object, as described in item 276. (Item 282) The method according to item 281, wherein the at least one physical object comprises a wall. (Item 283) The method according to item 276, further comprising determining the location of a hand gesture, wherein the virtual thread arises from the determined location. (Item 284) The method of item 276, wherein the virtual user interface is created at least in part based on a world-centered reference frame such that the virtual user interface remains stationary for any movement of the user. (Item 285) Selecting a virtual user interface element from the at least one selectable virtual interface element based at least partially on the user's other hand gesture, To generate virtual content associated with the selected user interface element in another location of the physical object. The method described in item 276, further including the method described in item 276. (Item 286) A method for creating a virtual user interface, Identifying the user interface to be displayed to the user based on the user's hand gestures, In response to the aforementioned hand gesture, a virtual user interface is displayed. Includes, The method wherein the virtual user interface comprises a set of vertical threads, each of which is associated with at least one selectable virtual interface element. (Item 287) The method according to item 286, wherein the hand gesture includes a gesture of extending the user's first hand for a predetermined period of time. (Item 288) The method according to item 287, further comprising selecting the at least one selectable virtual interface element at least in part on a gesture of another hand, wherein the gesture of the other hand includes a movement of pulling at least one of the virtual threads of the set of virtual threads using two fingers of the user. (Item 289) The method according to item 286, further comprising transmitting data associated with a first virtual element associated with a first virtual thread to a second virtual element associated with a second virtual thread, at least in part, based on hand gestures. (Item 290) The method according to item 289, wherein the gesture of the other hand includes a squeezing motion of the first virtual element associated with the first virtual thread using the second virtual element associated with the second virtual thread, and the squeezing motion is performed by the user's fingers. (Item 291) The virtual interface is displayed in relation to at least one physical object, as described in item 286. (Item 292) The method according to item 291, wherein the at least one physical object comprises a wall. (Item 293) The method according to item 286, further comprising determining the location of a hand gesture, wherein the virtual thread arises from the determined location. (Item 294) The method of item 286, wherein the virtual user interface is created at least in part based on a world-centered reference frame such that the virtual user interface remains stationary for any movement of the user. (Item 295) Selecting a virtual user interface element from the at least one selectable virtual interface element based at least partially on the user's other hand gesture, To generate virtual content associated with the selected user interface element in another location of the physical object. The method described in item 286, further including the method described in item 286. (Item 296) A method for creating a virtual user interface, Identifying the user interface to be displayed to the user based on the user's hand gestures, In response to the aforementioned hand gesture, a virtual user interface is displayed. Includes, The method involves the virtual user interface resembling a virtual spider web, and the user pulling on the virtual threads of the virtual spider web to bring the virtual interface closer to the user. (Item 297) The hand gesture is the method described in item 296, which includes a pulling motion using the user's clenched fist. (Item 298) The method according to item 297, wherein the virtual thread of the virtual spider web comprises at least one selectable virtual element. (Item 299) The method according to item 298, further comprising selecting the at least one selectable virtual interface element at least in part on a gesture of another hand, wherein the gesture of the other hand includes a movement of pulling at least one of the virtual threads toward the user. (Item 300) The method according to item 296, further comprising transmitting data associated with a first virtual element associated with a first virtual thread to a second virtual element associated with a second virtual thread, at least in part, based on a hand gesture. (Item 301) The method according to item 300, wherein the gesture of the other hand includes a squeezing motion of the first virtual element associated with the first virtual thread using the second virtual element associated with the second virtual thread, and the squeezing motion is performed by the user's fingers. (Item 302) The virtual interface is displayed in relation to at least one physical object, as described in item 296. (Item 303) The method according to item 302, wherein the at least one physical object comprises a wall. (Item 304) The method according to item 296, further comprising determining the location of a hand gesture, wherein the virtual spider web arises from the determined location. (Item 305) The method of item 296, wherein the virtual user interface is created at least in part based on a world-centered reference frame such that the virtual user interface remains stationary for any movement of the user. (Item 306) Selecting a virtual user interface element from the at least one selectable virtual interface element based at least partially on the user's other hand gesture, To generate virtual content associated with the selected user interface element in another location of the physical object. The method described in item 296, further including the method described in item 296. (Item 307) The method according to item 1-306, implemented as a system having means for implementing the method steps described above. (Item 308) The method according to item 1-306, which is implemented as a computer program product comprising a computer-usable storage medium having executable code for performing the method steps described above. [Brief explanation of the drawing]

[0205] The drawings illustrate the design and availability of various embodiments of the present invention. Note that the drawings are not drawn to exact scale, and elements of similar structures or functions are represented throughout the drawings by similar reference numbers. To gain a deeper understanding of the aforementioned and other advantages and purposes of the various embodiments of the present invention, a brief mode for carrying out the aforementioned invention will be given by referring to the specific embodiments illustrated in the accompanying drawings. With the understanding that these drawings depict only typical embodiments of the present invention and should not be considered as limitations thereof, the present invention will be described and explained with additional specificity and detail through the use of the accompanying drawings. [Figure 1] Figure 1 illustrates the system architecture of an augmented reality (AR) system that interacts with one or more servers, according to one of the illustrated embodiments. [Figure 2] Figure 2 illustrates a detailed diagram of a mobile phone used as an AR device interacting with one or more servers, according to one of the illustrated embodiments. [Figure 3] Figure 3 shows a plan view of an exemplary AR device mounted on the user's head, according to one of the illustrated embodiments. [Figure 4A] Figures 4A-4D illustrate one or more embodiments of various internal processing components of a wearable AR device. [Figure 4B] Figures 4A-4D illustrate one or more embodiments of various internal processing components of a wearable AR device. [Figure 4C] Figures 4A-4D illustrate one or more embodiments of various internal processing components of a wearable AR device. [Figure 4D] Figures 4A-4D illustrate one or more embodiments of various internal processing components of a wearable AR device. [Figure 5A] Figures 5A-5H illustrate an embodiment of focused light transmission to the user through a transparent beam splitter substrate. [Figure 5B] Figures 5A-5H illustrate an embodiment of focused light transmission to the user through a transparent beam splitter substrate. [Figure 5C] Figures 5A-5H illustrate an embodiment of focused light transmission to the user through a transparent beam splitter substrate. [Figure 5D] Figures 5A-5H illustrate an embodiment of focused light transmission to the user through a transparent beam splitter substrate. [Figure 5E] Figures 5A-5H illustrate an embodiment of focused light transmission to the user through a transparent beam splitter substrate. [Figure 5F] Figures 5A-5H illustrate an embodiment of focused light transmission to the user through a transparent beam splitter substrate. [Figure 5G] Figures 5A-5H illustrate an embodiment of focused light transmission to the user through a transparent beam splitter substrate. [Figure 5H] Figures 5A-5H illustrate an embodiment of focused light transmission to the user through a transparent beam splitter substrate. [Figure 6A] Figures 6A and 6B illustrate embodiments of coupling the lens element with the transmissive beam splitter substrate shown in Figures 5A-5H. [Figure 6B] Figures 6A and 6B illustrate embodiments of coupling the lens element with the transmissive beam splitter substrate shown in Figures 5A-5H. [Figure 7A] Figures 7A and 7B illustrate embodiments of using one or more waveguides to transmit light to a user. [Figure 7B]Figures 7A and 7B illustrate embodiments of using one or more waveguides to transmit light to a user. [Figure 8A] Figures 8A-8Q illustrate embodiments of possible diffractive optical elements (ODEs). [Figure 8B] Figures 8A-8Q illustrate embodiments of possible diffractive optical elements (ODEs). [Figure 8C] Figures 8A-8Q illustrate embodiments of possible diffractive optical elements (ODEs). [Figure 8D] Figures 8A-8Q illustrate embodiments of possible diffractive optical elements (ODEs). [Figure 8E] Figures 8A-8Q illustrate embodiments of possible diffractive optical elements (ODEs). [Figure 8F] Figures 8A-8Q illustrate embodiments of possible diffractive optical elements (ODEs). [Figure 8G] Figures 8A-8Q illustrate embodiments of possible diffractive optical elements (ODEs). [Figure 8H] Figures 8A-8Q illustrate embodiments of possible diffractive optical elements (ODEs). [Figure 8I] Figures 8A-8Q illustrate embodiments of possible diffractive optical elements (ODEs). [Figure 8J] Figures 8A-8Q illustrate embodiments of possible diffractive optical elements (ODEs). [Figure 8K] Figures 8A-8Q illustrate embodiments of possible diffractive optical elements (ODEs). [Figure 8L] Figures 8A-8Q illustrate embodiments of possible diffractive optical elements (ODEs). [Figure 8M] Figures 8A-8Q illustrate embodiments of possible diffractive optical elements (ODEs). [Figure 8N] Figures 8A-8Q illustrate embodiments of possible diffractive optical elements (ODEs). [Figure 8O]Figures 8A-8Q illustrate embodiments of possible diffractive optical elements (ODEs). [Figure 8P] Figures 8A-8Q illustrate embodiments of possible diffractive optical elements (ODEs). [Figure 8Q] Figures 8A-8Q illustrate embodiments of possible diffractive optical elements (ODEs). [Figure 9] Figures 9A and 9B illustrate the wavefront generated from an optical projector according to one of the illustrated embodiments. [Figure 10] Figure 10 illustrates an embodiment of a stacked configuration of multiple transmissive beam splitter substrates coupled to an optical element, according to one of the illustrated embodiments. [Figure 11A] Figures 11A-11C illustrate the beamlet array projected into the user's pupil according to the illustrated embodiment. [Figure 11B] Figures 11A-11C illustrate the beamlet array projected into the user's pupil according to the illustrated embodiment. [Figure 11C] Figures 11A-11C illustrate the beamlet array projected into the user's pupil according to the illustrated embodiment. [Figure 12A] Figures 12A and 12B illustrate the configuration of a microprojector array according to the illustrated embodiment. [Figure 12B] Figures 12A and 12B illustrate the configuration of a microprojector array according to the illustrated embodiment. [Figure 13A] Figures 13A-13M illustrate embodiments of the coupling of a microprojector and optical elements according to the illustrated embodiments. [Figure 13B] Figures 13A-13M illustrate embodiments of the coupling of a microprojector and optical elements according to the illustrated embodiments. [Figure 13C] Figures 13A-13M illustrate embodiments of the coupling of a microprojector and optical elements according to the illustrated embodiments. [Figure 13D]Figures 13A-13M illustrate embodiments of the coupling of a microprojector and optical elements according to the illustrated embodiments. [Figure 13E] Figures 13A-13M illustrate embodiments of the coupling of a microprojector and optical elements according to the illustrated embodiments. [Figure 13F] Figures 13A-13M illustrate embodiments of the coupling of a microprojector and optical elements according to the illustrated embodiments. [Figure 13G] Figures 13A-13M illustrate embodiments of the coupling of a microprojector and optical elements according to the illustrated embodiments. [Figure 13H] Figures 13A-13M illustrate embodiments of the coupling of a microprojector and optical elements according to the illustrated embodiments. [Figure 13I] Figures 13A-13M illustrate embodiments of the coupling of a microprojector and optical elements according to the illustrated embodiments. [Figure 13J] Figures 13A-13M illustrate embodiments of the coupling of a microprojector and optical elements according to the illustrated embodiments. [Figure 13K] Figures 13A-13M illustrate embodiments of the coupling of a microprojector and optical elements according to the illustrated embodiments. [Figure 13L] Figures 13A-13M illustrate embodiments of the coupling of a microprojector and optical elements according to the illustrated embodiments. [Figure 13M] Figures 13A-13M illustrate embodiments of the coupling of a microprojector and optical elements according to the illustrated embodiments. [Figure 14A] Figures 14A-14F illustrate embodiments of a spatial light modulator coupled to an optical element, according to the illustrated embodiments. [Figure 14B] Figures 14A-14F illustrate embodiments of a spatial light modulator coupled to an optical element, according to the illustrated embodiments. [Figure 14C] Figures 14A-14F illustrate embodiments of a spatial light modulator coupled to an optical element, according to the illustrated embodiments. [Figure 14D] Figures 14A-14F illustrate embodiments of a spatial light modulator coupled to an optical element, according to the illustrated embodiments. [Figure 14E] Figures 14A-14F illustrate embodiments of a spatial light modulator coupled to an optical element, according to the illustrated embodiments. [Figure 14F] Figures 14A-14F illustrate embodiments of a spatial light modulator coupled to an optical element, according to the illustrated embodiments. [Figure 15A] Figures 15A-15C illustrate the use of a wedge-type waveguide with multiple light sources according to the illustrated embodiment. [Figure 15B] Figures 15A-15C illustrate the use of a wedge-type waveguide with multiple light sources according to the illustrated embodiment. [Figure 15C] Figures 15A-15C illustrate the use of a wedge-type waveguide with multiple light sources according to the illustrated embodiment. [Figure 16A] Figures 16A-16O illustrate embodiments of coupling between optical elements and optical fibers according to the illustrated embodiments. [Figure 16B] Figures 16A-16O illustrate embodiments of coupling between optical elements and optical fibers according to the illustrated embodiments. [Figure 16C] Figures 16A-16O illustrate embodiments of coupling between optical elements and optical fibers according to the illustrated embodiments. [Figure 16D] Figures 16A-16O illustrate embodiments of coupling between optical elements and optical fibers according to the illustrated embodiments. [Figure 16E] Figures 16A-16O illustrate embodiments of coupling between optical elements and optical fibers according to the illustrated embodiments. [Figure 16F] Figures 16A-16O illustrate embodiments of coupling between optical elements and optical fibers according to the illustrated embodiments. [Figure 16G] Figures 16A-16O illustrate embodiments of coupling between optical elements and optical fibers according to the illustrated embodiments. [Figure 16H]Figures 16A-16O illustrate embodiments of coupling between optical elements and optical fibers according to the illustrated embodiments. [Figure 16I] Figures 16A-16O illustrate embodiments of coupling between optical elements and optical fibers according to the illustrated embodiments. [Figure 16J] Figures 16A-16O illustrate embodiments of coupling between optical elements and optical fibers according to the illustrated embodiments. [Figure 16K] Figures 16A-16O illustrate embodiments of coupling between optical elements and optical fibers according to the illustrated embodiments. [Figure 16L] Figures 16A-16O illustrate embodiments of coupling between optical elements and optical fibers according to the illustrated embodiments. [Figure 16M] Figures 16A-16O illustrate embodiments of coupling between optical elements and optical fibers according to the illustrated embodiments. [Figure 16N] Figures 16A-16O illustrate embodiments of coupling between optical elements and optical fibers according to the illustrated embodiments. [Figure 16O] Figures 16A-16O illustrate embodiments of coupling between optical elements and optical fibers according to the illustrated embodiments. [Figure 17] Figure 17 illustrates a notch filter according to one of the illustrated embodiments. [Figure 18] Figure 18 illustrates a spiral pattern of a fiber scanning display according to one of the illustrated embodiments. [Figure 19A] Figures 19A-19N illustrate the occlusion effect when presenting a dark field to the user according to the illustrated embodiment. [Figure 19B] Figures 19A-19N illustrate the occlusion effect when presenting a dark field to the user according to the illustrated embodiment. [Figure 19C] Figures 19A-19N illustrate the occlusion effect when presenting a dark field to the user according to the illustrated embodiment. [Figure 19D] Figures 19A-19N illustrate the occlusion effect when presenting a dark field to the user according to the illustrated embodiment. [Figure 19E] Figures 19A-19N illustrate the occlusion effect when presenting a dark field to the user according to the illustrated embodiment. [Figure 19F] Figures 19A-19N illustrate the occlusion effect when presenting a dark field to the user according to the illustrated embodiment. [Figure 19G] Figures 19A-19N illustrate the occlusion effect when presenting a dark field to the user according to the illustrated embodiment. [Figure 19H] Figures 19A-19N illustrate the occlusion effect when presenting a dark field to the user according to the illustrated embodiment. [Figure 19I] Figures 19A-19N illustrate the occlusion effect when presenting a dark field to the user according to the illustrated embodiment. [Figure 19J] Figures 19A-19N illustrate the occlusion effect when presenting a dark field to the user according to the illustrated embodiment. [Figure 19K] Figures 19A-19N illustrate the occlusion effect when presenting a dark field to the user according to the illustrated embodiment. [Figure 19L] Figures 19A-19N illustrate the occlusion effect when presenting a dark field to the user according to the illustrated embodiment. [Figure 19M] Figures 19A-19N illustrate the occlusion effect when presenting a dark field to the user according to the illustrated embodiment. [Figure 19N] Figures 19A-19N illustrate the occlusion effect when presenting a dark field to the user according to the illustrated embodiment. [Figure 20A] Figures 20A-20O illustrate various embodiments of waveguide assemblies according to the illustrated embodiments. [Figure 20B] Figures 20A-20O illustrate various embodiments of waveguide assemblies according to the illustrated embodiments. [Figure 20C] Figures 20A-20O illustrate various embodiments of waveguide assemblies according to the illustrated embodiments. [Figure 20D] Figures 20A-20O illustrate various embodiments of waveguide assemblies according to the illustrated embodiments. [Figure 20E] Figures 20A-20O illustrate various embodiments of waveguide assemblies according to the illustrated embodiments. [Figure 20F] Figures 20A-20O illustrate various embodiments of waveguide assemblies according to the illustrated embodiments. [Figure 20G] Figures 20A-20O illustrate various embodiments of waveguide assemblies according to the illustrated embodiments. [Figure 20H] Figures 20A-20O illustrate various embodiments of waveguide assemblies according to the illustrated embodiments. [Figure 20I] Figures 20A-20O illustrate various embodiments of waveguide assemblies according to the illustrated embodiments. [Figure 20J] Figures 20A-20O illustrate various embodiments of waveguide assemblies according to the illustrated embodiments. [Figure 20K] Figures 20A-20O illustrate various embodiments of waveguide assemblies according to the illustrated embodiments. [Figure 20L] Figures 20A-20O illustrate various embodiments of waveguide assemblies according to the illustrated embodiments. [Figure 20M] Figures 20A-20O illustrate various embodiments of waveguide assemblies according to the illustrated embodiments. [Figure 20N] Figures 20A-20O illustrate various embodiments of waveguide assemblies according to the illustrated embodiments. [Figure 20O] Figures 20A-20O illustrate various embodiments of waveguide assemblies according to the illustrated embodiments. [Figure 21A] Figures 21A-21N illustrate various configurations of the DOE coupled to other optical elements according to the illustrated embodiments. [Figure 21B] Figures 21A-21N illustrate various configurations of the DOE coupled to other optical elements according to the illustrated embodiments. [Figure 21C]Figures 21A-21N illustrate various configurations of the DOE coupled to other optical elements according to the illustrated embodiments. [Figure 21D] Figures 21A-21N illustrate various configurations of the DOE coupled to other optical elements according to the illustrated embodiments. [Figure 21E] Figures 21A-21N illustrate various configurations of the DOE coupled to other optical elements according to the illustrated embodiments. [Figure 21F] Figures 21A-21N illustrate various configurations of the DOE coupled to other optical elements according to the illustrated embodiments. [Figure 21G] Figures 21A-21N illustrate various configurations of the DOE coupled to other optical elements according to the illustrated embodiments. [Figure 21H] Figures 21A-21N illustrate various configurations of the DOE coupled to other optical elements according to the illustrated embodiments. [Figure 21I] Figures 21A-21N illustrate various configurations of the DOE coupled to other optical elements according to the illustrated embodiments. [Figure 21J] Figures 21A-21N illustrate various configurations of the DOE coupled to other optical elements according to the illustrated embodiments. [Figure 21K] Figures 21A-21N illustrate various configurations of the DOE coupled to other optical elements according to the illustrated embodiments. [Figure 21L] Figures 21A-21N illustrate various configurations of the DOE coupled to other optical elements according to the illustrated embodiments. [Figure 21M] Figures 21A-21N illustrate various configurations of the DOE coupled to other optical elements according to the illustrated embodiments. [Figure 21N] Figures 21A-21N illustrate various configurations of the DOE coupled to other optical elements according to the illustrated embodiments. [Figure 22A] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22B]Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22C] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22D] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22E] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22F] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22G] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22H] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22I] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22J] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22K] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22L] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22M] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22N] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22O] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22P] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22Q] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22R] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22S] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22T] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22U] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22V] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22W] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22X] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 22Y] Figures 22A-22Y illustrate various configurations of free-form optics according to the illustrated embodiments. [Figure 23] Figure 23 shows a simplified top view of the components of an individual AR device. [Figure 24] Figure 24 illustrates exemplary optics of individual AR systems. [Figure 25] Figure 25 illustrates the system architecture of individual AR systems according to one embodiment. [Figure 26] Figure 26 illustrates a room-based sensor system according to one embodiment. [Figure 27] Figure 27 illustrates the communication architecture of an augmented reality system and the interaction between the augmented reality system and the cloud for multiple users. [Figure 28] Figure 28 illustrates a simplified diagram of a passable world model according to one embodiment. [Figure 29] Figure 29 illustrates an exemplary method of rendering using a passable world model according to one embodiment. [Figure 30] Figure 30 illustrates a high-level flowchart for an object recognition process according to one embodiment. [Figure 31] Figure 31 illustrates a ring buffer approach employed by an object recognizer that recognizes objects within a passable world, according to one embodiment. [Figure 32] Figure 32 illustrates an exemplary topology map according to one embodiment. [Figure 33] Figure 33 illustrates a high-level flow diagram for a localization process using a topology map according to one embodiment. [Figure 34] Figure 34 illustrates a geometric map as a connection between various keyframes according to one embodiment. [Figure 35] Figure 35 illustrates an exemplary embodiment of a topology map layered on top of a geometric map, according to one embodiment. [Figure 36] Figure 36 illustrates a high-level flow diagram for a wave propagation flux adjustment process according to one embodiment. [Figure 37] Figure 37 illustrates a map point and a rendering line from the map point to the key frame, as seen through a virtual keyframe, according to one embodiment. [Figure 38] Figure 38 illustrates a high-level flowchart for a process of discovering map points based on rendering rather than searching, according to one embodiment. [Figure 39] Figure 39 illustrates a high-level flowchart for a process of rendering virtual objects based on a light map, according to one embodiment. [Figure 40] Figure 40 illustrates a high-level flow diagram for the process of creating a light map according to one embodiment. [Figure 41] Figure 41 illustrates a user-centered light map according to one embodiment. [Figure 42]Figure 42 illustrates an object-centered light map according to one embodiment. [Figure 43] Figure 43 illustrates a high-level flow diagram for a process of converting an optical map according to one embodiment. [Figure 44] Figure 44 illustrates an autonomous navigation definition or object library according to one embodiment. [Figure 45] Figure 45 illustrates the interaction of various autonomous navigation objects according to one embodiment. [Figure 46] Figure 46 illustrates an autonomous navigation definition or object stack according to one embodiment. [Figure 47A] Figures 47A-47B illustrate the use of autonomous navigation definitions for identifying emotional states according to one embodiment. [Figure 47B] Figures 47A-47B illustrate the use of autonomous navigation definitions for identifying emotional states according to one embodiment. [Figure 48] Figure 48 illustrates a correlation threshold graph, according to one embodiment, which should be used to define an autonomous navigation definition or object. [Figure 49] Figure 49 illustrates a diagram of a passable world model according to one embodiment. [Figure 50] Figure 50 illustrates an exemplary method for displaying a virtual scene according to one embodiment. [Figure 51] Figure 51 illustrates plan views of various modules of the AR system according to one of the illustrated embodiments. [Figure 52] Figure 52 illustrates an example of an object visible to the user when the AR device is operating in augmented reality mode, according to one of the illustrated embodiments. [Figure 53] Figure 53 illustrates an example of an object visible to the user when the AR device is operating in virtual mode, according to one of the illustrated embodiments. [Figure 54]Figure 54 illustrates an example of an object visible to the user when the AR device is operating in a hybrid virtual interface mode, according to one of the illustrated embodiments. [Figure 55] Figure 55 illustrates an embodiment in which two users are located in different geographical locations and each interacts with the other user and a shared virtual world through their respective user devices. [Figure 56] Figure 56 illustrates an embodiment in which the embodiment of Figure 55 is extended to include the use of a tactile device. [Figure 57A] Figures 57A and 57B illustrate examples of mixed-mode interfaces according to one or more embodiments. [Figure 57B] Figures 57A and 57B illustrate examples of mixed-mode interfaces according to one or more embodiments. [Figure 58] Figure 58 illustrates an exemplary view of a user when interfaced with an AR system according to one embodiment. [Figure 59] Figure 59 illustrates an exemplary user view in one embodiment, showing virtual objects triggered by physical objects when a user interfaces with the system in augmented reality mode. [Figure 60] Figure 60 illustrates one embodiment of an augmented reality and virtual reality integrated configuration in which a user in an augmented reality experience visualizes the presence of another user in a virtual reality experience. [Figure 61] Figure 61 illustrates one embodiment of a time- and / or incidental event-based augmented reality experience configuration. [Figure 62] Figure 62 illustrates one embodiment of a user display configuration suitable for virtual and / or augmented reality experiences. [Figure 63] Figure 63 illustrates one embodiment of local and cloud-based computing collaboration. [Figure 64]Figure 64 illustrates various aspects of the alignment configuration according to one of the illustrated embodiments. [Figure 65] Figure 65 illustrates an exemplary scenario of interacting with an AR system according to one embodiment. [Figure 66] Figure 66 illustrates another perspective view of the exemplary scenario of Figure 65, according to another embodiment. [Figure 67] Figure 67 illustrates yet another perspective view of the exemplary scenario of Figure 65, according to another embodiment. [Figure 68] Figure 68 illustrates a top view of an exemplary scenario according to one embodiment. [Figure 69] Figure 69 illustrates a game view of the exemplary scenario shown in Figures 65-68 according to one embodiment. [Figure 70] Figure 70 illustrates a top view of the exemplary scenario shown in Figures 65-68 according to one embodiment. [Figure 71] Figure 71 illustrates an augmented reality scenario involving multiple users according to one embodiment. [Figure 72A] Figures 72A and 72B illustrate the use of a smartphone or tablet as an AR device according to one embodiment. [Figure 72B] Figures 72A and 72B illustrate the use of a smartphone or tablet as an AR device according to one embodiment. [Figure 73] Figure 73 illustrates an exemplary method, according to one embodiment, of using location information to communicate between AR system users. [Figure 74A] Figures 74A-74B illustrate an exemplary office scenario in which an AR system is interacted with, according to one embodiment. [Figure 74B] Figures 74A-74B illustrate an exemplary office scenario in which an AR system is interacted with, according to one embodiment. [Figure 75] Figure 75 illustrates an exemplary scenario of interacting with an AR system in a home, according to one embodiment. [Figure 76]Figure 76 illustrates another exemplary scenario of interacting with an AR system in a home, according to one embodiment. [Figure 77] Figure 77 illustrates another exemplary scenario of interacting with an AR system in a home, according to one embodiment. [Figure 78A] Figures 78A-78B illustrate yet another exemplary scenario of interacting with an AR system in a home, according to one embodiment. [Figure 78B] Figures 78A-78B illustrate yet another exemplary scenario of interacting with an AR system in a home, according to one embodiment. [Figure 79A] Figures 79A-79E illustrate another exemplary scenario of interacting with an AR system in a home, according to one embodiment. [Figure 79B] Figures 79A-79E illustrate another exemplary scenario of interacting with an AR system in a home, according to one embodiment. [Figure 79C] Figures 79A-79E illustrate another exemplary scenario of interacting with an AR system in a home, according to one embodiment. [Figure 79D] Figures 79A-79E illustrate another exemplary scenario of interacting with an AR system in a home, according to one embodiment. [Figure 79E] Figures 79A-79E illustrate another exemplary scenario of interacting with an AR system in a home, according to one embodiment. [Figure 80A] Figures 80A-80C illustrate another exemplary scenario in which an AR system interacts with a virtual room, according to one embodiment. [Figure 80B] Figures 80A-80C illustrate another exemplary scenario in which an AR system interacts with a virtual room, according to one embodiment. [Figure 80C] Figures 80A-80C illustrate another exemplary scenario in which an AR system interacts with a virtual room, according to one embodiment. [Figure 81] Figure 81 illustrates another exemplary user interaction scenario according to one embodiment. [Figure 82]Figure 82 illustrates another exemplary user interaction scenario according to one embodiment. [Figure 83A] Figures 83A-83B illustrate yet another exemplary user interaction scenario according to one or more embodiments. [Figure 83B] Figures 83A-83B illustrate yet another exemplary user interaction scenario according to one or more embodiments. [Figure 84A] Figures 84A-84C illustrate a user interacting with an AR system in a virtual space according to one or more embodiments. [Figure 84B] Figures 84A-84C illustrate a user interacting with an AR system in a virtual space according to one or more embodiments. [Figure 84C] Figures 84A-84C illustrate a user interacting with an AR system in a virtual space according to one or more embodiments. [Figure 85A] Figures 85A-85C illustrate various user interface embodiments. [Figure 85B] Figures 85A-85C illustrate various user interface embodiments. [Figure 85C] Figures 85A-85C illustrate various user interface embodiments. [Figure 86A] Figures 86A-86C illustrate other embodiments for creating a user interface according to one or more embodiments. [Figure 86B] Figures 86A-86C illustrate other embodiments for creating a user interface according to one or more embodiments. [Figure 86C] Figures 86A-86C illustrate other embodiments for creating a user interface according to one or more embodiments. [Figure 87A] Figures 87A-87C illustrate other embodiments for creating and navigating user interfaces according to one or more embodiments. [Figure 87B]Figures 87A-87C illustrate other embodiments for creating and navigating user interfaces according to one or more embodiments. [Figure 87C] Figures 87A-87C illustrate other embodiments for creating and navigating user interfaces according to one or more embodiments. [Figure 88A] Figures 88A-88C illustrate user interfaces created on the user's hand according to one or more embodiments. [Figure 88B] Figures 88A-88C illustrate user interfaces created on the user's hand according to one or more embodiments. [Figure 88C] Figures 88A-88C illustrate user interfaces created on the user's hand according to one or more embodiments. [Figure 89A] Figures 89A-89J illustrate exemplary user shopping experiences using an AR system according to one or more embodiments. [Figure 89B] Figures 89A-89J illustrate exemplary user shopping experiences using an AR system according to one or more embodiments. [Figure 89C] Figures 89A-89J illustrate exemplary user shopping experiences using an AR system according to one or more embodiments. [Figure 89D] Figures 89A-89J illustrate exemplary user shopping experiences using an AR system according to one or more embodiments. [Figure 89E] Figures 89A-89J illustrate exemplary user shopping experiences using an AR system according to one or more embodiments. [Figure 89F] Figures 89A-89J illustrate exemplary user shopping experiences using an AR system according to one or more embodiments. [Figure 89G] Figures 89A-89J illustrate exemplary user shopping experiences using an AR system according to one or more embodiments. [Figure 89H] Figures 89A-89J illustrate exemplary user shopping experiences using an AR system according to one or more embodiments. [Figure 89I] Figures 89A-89J illustrate exemplary user shopping experiences using an AR system according to one or more embodiments. [Figure 89J] Figures 89A-89J illustrate exemplary user shopping experiences using an AR system according to one or more embodiments. [Figure 90] Figure 90 illustrates an exemplary library experience using an AR system in one or more embodiments. [Figure 91A] Figures 91A-91F illustrate exemplary health management experiences using an AR system according to one or more embodiments. [Figure 91B] Figures 91A-91F illustrate exemplary health management experiences using an AR system according to one or more embodiments. [Figure 91C] Figures 91A-91F illustrate exemplary health management experiences using an AR system according to one or more embodiments. [Figure 91D] Figures 91A-91F illustrate exemplary health management experiences using an AR system according to one or more embodiments. [Figure 91E] Figures 91A-91F illustrate exemplary health management experiences using an AR system according to one or more embodiments. [Figure 91F] Figures 91A-91F illustrate exemplary health management experiences using an AR system according to one or more embodiments. [Figure 92] Figure 92 illustrates an exemplary work experience using an AR system in one or more embodiments. [Figure 93A] Figures 93A-93L illustrate exemplary workspace experiences using an AR system according to one or more embodiments. [Figure 93B] Figures 93A-93L illustrate exemplary workspace experiences using an AR system according to one or more embodiments. [Figure 93C] Figures 93A-93L illustrate exemplary workspace experiences using an AR system according to one or more embodiments. [Figure 93D]Figures 93A-93L illustrate exemplary workspace experiences using an AR system according to one or more embodiments. [Figure 93E] Figures 93A-93L illustrate exemplary workspace experiences using an AR system according to one or more embodiments. [Figure 93F] Figures 93A-93L illustrate exemplary workspace experiences using an AR system according to one or more embodiments. [Figure 93G] Figures 93A-93L illustrate exemplary workspace experiences using an AR system according to one or more embodiments. [Figure 93H] Figures 93A-93L illustrate exemplary workspace experiences using an AR system according to one or more embodiments. [Figure 93I] Figures 93A-93L illustrate exemplary workspace experiences using an AR system according to one or more embodiments. [Figure 93J] Figures 93A-93L illustrate exemplary workspace experiences using an AR system according to one or more embodiments. [Figure 93K] Figures 93A-93L illustrate exemplary workspace experiences using an AR system according to one or more embodiments. [Figure 93L] Figures 93A-93L illustrate exemplary workspace experiences using an AR system according to one or more embodiments. [Figure 94] Figure 94 illustrates another exemplary workspace experience using an AR system according to one or more embodiments. [Figure 95A] Figures 95A-95E illustrate different AR experiences according to one or more embodiments. [Figure 95B] Figures 95A-95E illustrate different AR experiences according to one or more embodiments. [Figure 95C] Figures 95A-95E illustrate different AR experiences according to one or more embodiments. [Figure 95D] Figures 95A-95E illustrate different AR experiences according to one or more embodiments. [Figure 95E]Figures 95A-95E illustrate different AR experiences according to one or more embodiments. [Figure 96A] Figures 96A-96D illustrate yet another AR experience according to one or more embodiments. [Figure 96B] Figures 96A-96D illustrate yet another AR experience according to one or more embodiments. [Figure 96C] Figures 96A-96D illustrate yet another AR experience according to one or more embodiments. [Figure 96D] Figures 96A-96D illustrate yet another AR experience according to one or more embodiments. [Figure 97A] Figures 97A-97H illustrate a game experience using an AR system according to one or more embodiments. [Figure 97B] Figures 97A-97H illustrate a game experience using an AR system according to one or more embodiments. [Figure 97C] Figures 97A-97H illustrate a game experience using an AR system according to one or more embodiments. [Figure 97D] Figures 97A-97H illustrate a game experience using an AR system according to one or more embodiments. [Figure 97E] Figures 97A-97H illustrate a game experience using an AR system according to one or more embodiments. [Figure 97F] Figures 97A-97H illustrate a game experience using an AR system according to one or more embodiments. [Figure 97G] Figures 97A-97H illustrate a game experience using an AR system according to one or more embodiments. [Figure 97H] Figures 97A-97H illustrate a game experience using an AR system according to one or more embodiments. [Figure 98A] Figures 98A-98D illustrate a web shopping experience using an AR system according to one or more embodiments. [Figure 98B]Figures 98A-98D illustrate a web shopping experience using an AR system according to one or more embodiments. [Figure 98C] Figures 98A-98D illustrate a web shopping experience using an AR system according to one or more embodiments. [Figure 98D] Figures 98A-98D illustrate a web shopping experience using an AR system according to one or more embodiments. [Figure 99] Figure 99 illustrates block diagrams of various games within a game platform according to one or more embodiments. [Figure 100] Figure 100 illustrates various user inputs for communicating with an augmented reality system according to one embodiment. [Figure 101] Figure 101 illustrates an LED light and diode that track the movement of a user's eyes according to one embodiment. [Figure 102] Figure 102 illustrates a Purkinje image according to one embodiment. [Figure 103] Figure 103 illustrates various hand gestures that may be used to communicate with an augmented reality system according to one embodiment. [Figure 104] Figure 104 illustrates an exemplary totem according to one embodiment. [Figure 105A] Figures 105A-105C illustrate other exemplary totems according to one or more embodiments. [Figure 105B] Figures 105A-105C illustrate other exemplary totems according to one or more embodiments. [Figure 105C] Figures 105A-105C illustrate other exemplary totems according to one or more embodiments. [Figure 106A] Figures 106A-106C illustrate other totems that may be used to communicate with augmented reality systems. [Figure 106B] Figures 106A-106C illustrate other totems that may be used to communicate with augmented reality systems. [Figure 106C]Figures 106A-106C illustrate other totems that may be used to communicate with augmented reality systems. [Figure 107A] Figures 107A-107D illustrate other exemplary totems according to one or more embodiments. [Figure 107B] Figures 107A-107D illustrate other exemplary totems according to one or more embodiments. [Figure 107C] Figures 107A-107D illustrate other exemplary totems according to one or more embodiments. [Figure 107D] Figures 107A-107D illustrate other exemplary totems according to one or more embodiments. [Figure 108A] Figures 108A-108C illustrate exemplary embodiments of ring and bracelet-type totems according to one or more embodiments. [Figure 108B] Figures 108A-108C illustrate exemplary embodiments of ring and bracelet-type totems according to one or more embodiments. [Figure 108C] Figures 108A-108C illustrate exemplary embodiments of ring and bracelet-type totems according to one or more embodiments. [Figure 109A] Figures 109A-109C illustrate further exemplary totems according to one or more embodiments. [Figure 109B] Figures 109A-109C illustrate further exemplary totems according to one or more embodiments. [Figure 109C] Figures 109A-109C illustrate further exemplary totems according to one or more embodiments. [Figure 110A] Figures 110A and 110B illustrate charm-type totems and keychain-type totems according to one or more embodiments. [Figure 110B] Figures 110A and 110B illustrate charm-type totems and keychain-type totems according to one or more embodiments. [Figure 111] Figure 111 illustrates a high-level flowchart for a process of determining user input through a totem, according to one embodiment. [Figure 112] Figure 112 illustrates a high-level flow diagram for a process for generating a sound wave front according to one embodiment. [Figure 113] Figure 113 is a block diagram of components used to generate a sound wave front according to one embodiment. [Figure 114] Figure 114 shows an exemplary method for determining low-density and high-density points according to one embodiment. [Figure 115] Figure 115 is a block diagram of a textured light projection according to one embodiment. [Figure 116] Figure 116 is an illustrative block diagram of data processing according to one embodiment. [Figure 117] Figure 117 is a schematic diagram of an eye for tracking eye movement according to one embodiment. [Figure 118] Figure 118 shows another perspective view of an eye and one or more cameras for tracking eye movement according to one embodiment. [Figure 119] Figure 119 shows yet another perspective view of an eye for tracking eye movement and one or more cameras according to one embodiment. [Figure 120] Figure 120 shows yet another perspective view of an eye for tracking eye movement and one or more cameras according to one embodiment. [Figure 121] Figure 121 shows a translation matrix view for tracking line of sight according to one embodiment. [Figure 122] Figure 122 illustrates an exemplary method for tracking eye movements according to one embodiment. [Figure 123A] Figures 123A-123D illustrate a series of exemplary user interface flows using avatars according to one embodiment. [Figure 123B] Figures 123A-123D illustrate a series of exemplary user interface flows using avatars according to one embodiment. [Figure 123C] Figures 123A-123D illustrate a series of exemplary user interface flows using avatars according to one embodiment. [Figure 123D] Figures 123A-123D illustrate a series of exemplary user interface flows using avatars according to one embodiment. [Figure 124A] Figures 124A-124M illustrate a series of exemplary user interface flows using extrusion according to one embodiment. [Figure 124B] Figures 124A-124M illustrate a series of exemplary user interface flows using extrusion according to one embodiment. [Figure 124C] Figures 124A-124M illustrate a series of exemplary user interface flows using extrusion according to one embodiment. [Figure 124D] Figures 124A-124M illustrate a series of exemplary user interface flows using extrusion according to one embodiment. [Figure 124E] Figures 124A-124M illustrate a series of exemplary user interface flows using extrusion according to one embodiment. [Figure 124F] Figures 124A-124M illustrate a series of exemplary user interface flows using extrusion according to one embodiment. [Figure 124G] Figures 124A-124M illustrate a series of exemplary user interface flows using extrusion according to one embodiment. [Figure 124H] Figures 124A-124M illustrate a series of exemplary user interface flows using extrusion according to one embodiment. [Figure 124I] Figures 124A-124M illustrate a series of exemplary user interface flows using extrusion according to one embodiment. [Figure 124J] Figures 124A-124M illustrate a series of exemplary user interface flows using extrusion according to one embodiment. [Figure 124K] Figures 124A-124M illustrate a series of exemplary user interface flows using extrusion according to one embodiment. [Figure 124L]Figures 124A-124M illustrate a series of exemplary user interface flows using extrusion according to one embodiment. [Figure 124M] Figures 124A-124M illustrate a series of exemplary user interface flows using extrusion according to one embodiment. [Figure 125A] Figures 125A-125M illustrate a series of exemplary user interface flows using long gloves according to one embodiment. [Figure 125B] Figures 125A-125M illustrate a series of exemplary user interface flows using long gloves according to one embodiment. [Figure 125C] Figures 125A-125M illustrate a series of exemplary user interface flows using long gloves according to one embodiment. [Figure 125D] Figures 125A-125M illustrate a series of exemplary user interface flows using long gloves according to one embodiment. [Figure 125E] Figures 125A-125M illustrate a series of exemplary user interface flows using long gloves according to one embodiment. [Figure 125F] Figures 125A-125M illustrate a series of exemplary user interface flows using long gloves according to one embodiment. [Figure 125G] Figures 125A-125M illustrate a series of exemplary user interface flows using long gloves according to one embodiment. [Figure 125H] Figures 125A-125M illustrate a series of exemplary user interface flows using long gloves according to one embodiment. [Figure 125I] Figures 125A-125M illustrate a series of exemplary user interface flows using long gloves according to one embodiment. [Figure 125J] Figures 125A-125M illustrate a series of exemplary user interface flows using long gloves according to one embodiment. [Figure 125K]Figures 125A-125M illustrate a series of exemplary user interface flows using long gloves according to one embodiment. [Figure 125L] Figures 125A-125M illustrate a series of exemplary user interface flows using long gloves according to one embodiment. [Figure 125M] Figures 125A-125M illustrate a series of exemplary user interface flows using long gloves according to one embodiment. [Figure 126A] Figures 126A-126L illustrate a series of exemplary user interface flows using growth according to one embodiment. [Figure 126B] Figures 126A-126L illustrate a series of exemplary user interface flows using growth according to one embodiment. [Figure 126C] Figures 126A-126L illustrate a series of exemplary user interface flows using growth according to one embodiment. [Figure 126D] Figures 126A-126L illustrate a series of exemplary user interface flows using growth according to one embodiment. [Figure 126E] Figures 126A-126L illustrate a series of exemplary user interface flows using growth according to one embodiment. [Figure 126F] Figures 126A-126L illustrate a series of exemplary user interface flows using growth according to one embodiment. [Figure 126G] Figures 126A-126L illustrate a series of exemplary user interface flows using growth according to one embodiment. [Figure 126H] Figures 126A-126L illustrate a series of exemplary user interface flows using growth according to one embodiment. [Figure 126I] Figures 126A-126L illustrate a series of exemplary user interface flows using growth according to one embodiment. [Figure 126J]Figures 126A-126L illustrate a series of exemplary user interface flows using growth according to one embodiment. [Figure 126K] Figures 126A-126L illustrate a series of exemplary user interface flows using growth according to one embodiment. [Figure 126L] Figures 126A-126L illustrate a series of exemplary user interface flows using growth according to one embodiment. [Figure 127A] Figures 127A-127E illustrate a series of exemplary user interface flows using a brush according to one embodiment. [Figure 127B] Figures 127A-127E illustrate a series of exemplary user interface flows using a brush according to one embodiment. [Figure 127C] Figures 127A-127E illustrate a series of exemplary user interface flows using a brush according to one embodiment. [Figure 127D] Figures 127A-127E illustrate a series of exemplary user interface flows using a brush according to one embodiment. [Figure 127E] Figures 127A-127E illustrate a series of exemplary user interface flows using a brush according to one embodiment. [Figure 128A] Figures 128A-128P illustrate a series of exemplary user interface flows using a finger brush according to one embodiment. [Figure 128B] Figures 128A-128P illustrate a series of exemplary user interface flows using a finger brush according to one embodiment. [Figure 128C] Figures 128A-128P illustrate a series of exemplary user interface flows using a finger brush according to one embodiment. [Figure 128D] Figures 128A-128P illustrate a series of exemplary user interface flows using a finger brush according to one embodiment. [Figure 128E]Figures 128A-128P illustrate a series of exemplary user interface flows using a finger brush according to one embodiment. [Figure 128F] Figures 128A-128P illustrate a series of exemplary user interface flows using a finger brush according to one embodiment. [Figure 128G] Figures 128A-128P illustrate a series of exemplary user interface flows using a finger brush according to one embodiment. [Figure 128H] Figures 128A-128P illustrate a series of exemplary user interface flows using a finger brush according to one embodiment. [Figure 128I] Figures 128A-128P illustrate a series of exemplary user interface flows using a finger brush according to one embodiment. [Figure 128J] Figures 128A-128P illustrate a series of exemplary user interface flows using a finger brush according to one embodiment. [Figure 128K] Figures 128A-128P illustrate a series of exemplary user interface flows using a finger brush according to one embodiment. [Figure 128L] Figures 128A-128P illustrate a series of exemplary user interface flows using a finger brush according to one embodiment. [Figure 128M] Figures 128A-128P illustrate a series of exemplary user interface flows using a finger brush according to one embodiment. [Figure 128N] Figures 128A-128P illustrate a series of exemplary user interface flows using a finger brush according to one embodiment. [Figure 128O] Figures 128A-128P illustrate a series of exemplary user interface flows using a finger brush according to one embodiment. [Figure 128P] Figures 128A-128P illustrate a series of exemplary user interface flows using a finger brush according to one embodiment. [Figure 129A]Figures 129A-129M illustrate a series of exemplary user interface flows using pivots according to one embodiment. [Figure 129B] Figures 129A-129M illustrate a series of exemplary user interface flows using pivots according to one embodiment. [Figure 129C] Figures 129A-129M illustrate a series of exemplary user interface flows using pivots according to one embodiment. [Figure 129D] Figures 129A-129M illustrate a series of exemplary user interface flows using pivots according to one embodiment. [Figure 129E] Figures 129A-129M illustrate a series of exemplary user interface flows using pivots according to one embodiment. [Figure 129F] Figures 129A-129M illustrate a series of exemplary user interface flows using pivots according to one embodiment. [Figure 129G] Figures 129A-129M illustrate a series of exemplary user interface flows using pivots according to one embodiment. [Figure 129H] Figures 129A-129M illustrate a series of exemplary user interface flows using pivots according to one embodiment. [Figure 129I] Figures 129A-129M illustrate a series of exemplary user interface flows using pivots according to one embodiment. [Figure 129J] Figures 129A-129M illustrate a series of exemplary user interface flows using pivots according to one embodiment. [Figure 129K] Figures 129A-129M illustrate a series of exemplary user interface flows using pivots according to one embodiment. [Figure 129L] Figures 129A-129M illustrate a series of exemplary user interface flows using pivots according to one embodiment. [Figure 129M]Figures 129A-129M illustrate a series of exemplary user interface flows using pivots according to one embodiment. [Figure 130A] Figures 130A-130I illustrate a series of exemplary user interface flows using thread according to one embodiment. [Figure 130B] Figures 130A-130I illustrate a series of exemplary user interface flows using thread according to one embodiment. [Figure 130C] Figures 130A-130I illustrate a series of exemplary user interface flows using thread according to one embodiment. [Figure 130D] Figures 130A-130I illustrate a series of exemplary user interface flows using thread according to one embodiment. [Figure 130E] Figures 130A-130I illustrate a series of exemplary user interface flows using thread according to one embodiment. [Figure 130F] Figures 130A-130I illustrate a series of exemplary user interface flows using thread according to one embodiment. [Figure 130G] Figures 130A-130I illustrate a series of exemplary user interface flows using thread according to one embodiment. [Figure 130H] Figures 130A-130I illustrate a series of exemplary user interface flows using thread according to one embodiment. [Figure 130I] Figures 130A-130I illustrate a series of exemplary user interface flows using thread according to one embodiment. [Figure 131A] Figures 131A-131I illustrate a series of exemplary user interface flows using a spiderweb according to one embodiment. [Figure 131B] Figures 131A-131I illustrate a series of exemplary user interface flows using a spiderweb according to one embodiment. [Figure 131C]Figures 131A-131I illustrate a series of exemplary user interface flows using a spiderweb according to one embodiment. [Figure 131D] Figures 131A-131I illustrate a series of exemplary user interface flows using a spiderweb according to one embodiment. [Figure 131E] Figures 131A-131I illustrate a series of exemplary user interface flows using a spiderweb according to one embodiment. [Figure 131F] Figures 131A-131I illustrate a series of exemplary user interface flows using a spiderweb according to one embodiment. [Figure 131G] Figures 131A-131I illustrate a series of exemplary user interface flows using a spiderweb according to one embodiment. [Figure 131H] Figures 131A-131I illustrate a series of exemplary user interface flows using a spiderweb according to one embodiment. [Figure 131I] Figures 131A-131I illustrate a series of exemplary user interface flows using a spiderweb according to one embodiment. [Figure 132] Figure 132 is a plan view of various mechanisms in which a virtual object is related to one or more physical objects. [Figure 133] Figure 133 is a plan view of various types of AR renderings according to one or more embodiments. [Figure 134] Figure 134 illustrates various types of user input in an AR system according to one or more embodiments. [Figure 135A] Figures 135A-135J illustrate various embodiments of the use of gestures in an AR system according to one or more embodiments. [Figure 135B] Figures 135A-135J illustrate various embodiments of the use of gestures in an AR system according to one or more embodiments. [Figure 135C]Figures 135A-135J illustrate various embodiments of the use of gestures in an AR system according to one or more embodiments. [Figure 135D] Figures 135A-135J illustrate various embodiments of the use of gestures in an AR system according to one or more embodiments. [Figure 135E] Figures 135A-135J illustrate various embodiments of the use of gestures in an AR system according to one or more embodiments. [Figure 135F] Figures 135A-135J illustrate various embodiments of the use of gestures in an AR system according to one or more embodiments. [Figure 135G] Figures 135A-135J illustrate various embodiments of the use of gestures in an AR system according to one or more embodiments. [Figure 135H] Figures 135A-135J illustrate various embodiments of the use of gestures in an AR system according to one or more embodiments. [Figure 135I] Figures 135A-135J illustrate various embodiments of the use of gestures in an AR system according to one or more embodiments. [Figure 135J] Figures 135A-135J illustrate various embodiments of the use of gestures in an AR system according to one or more embodiments. [Figure 136] Figure 136 illustrates plan views of various components for a calibration mechanism of an AR system according to one or more embodiments. [Figure 137] Figure 137 illustrates a diagram of an AR device on a user's face according to one or more embodiments, the AR device having an eye-tracking camera. [Figure 138] Figure 138 illustrates eye recognition images of an AR system according to one or more embodiments. [Figure 139] Figure 139 illustrates retinal images captured using an AR system according to one or more embodiments. [Figure 140]Figure 140 is a process flow diagram of an exemplary method for generating a virtual user interface according to one of the illustrated embodiments. [Figure 141] Figure 141 is another process flow diagram of an exemplary method for generating a virtual user interface based on a coordinate frame, according to one of the illustrated embodiments. [Figure 142] Figure 142 is a process flow diagram of an exemplary method for constructing a customized user interface according to one of the illustrated embodiments. [Figure 143] Figure 143 is a process flow diagram of an exemplary method, according to one of the illustrated embodiments, for reading information from a passable world model and interacting with other users of the AR system. [Figure 144] Figure 144 is a process flow diagram of an exemplary method for retrieving information from a knowledge base in the cloud based on received input, according to one of the illustrated embodiments. [Figure 145] Figure 145 is a process flow diagram of an exemplary method for calibrating an AR system according to one of the illustrated embodiments. [Modes for carrying out the invention]

[0206] Herein, various embodiments are described in detail with reference to the drawings, which are provided as illustrative examples of the invention, so as to enable those skilled in the art to practice the invention. It should be noted that the following drawings and embodiments are not intended to limit the scope of the invention. Where any element of the invention can be partially or completely implemented using known components (or methods or processes), only those parts of such known components (or methods or processes) necessary for understanding the invention will be described, and detailed descriptions of other parts of such known components (or methods or processes) will be omitted so as not to obscure the invention. Furthermore, the various embodiments include currently and future known equivalents of the components referenced herein as illustrations.

[0207] In the aforementioned specification, the present invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes can be made therein without departing from the broader spirit and scope of the invention. For example, the aforementioned process flow is described with reference to a specific sequence of process actions. However, many of the sequences of process actions described may be changed without affecting the scope or operation of the invention. The specification and drawings are therefore to be considered illustrative rather than restrictive.

[0208] Disclosed are methods and systems for generating virtual and / or augmented reality. To provide a realistic and enjoyable virtual reality (VR) or augmented reality (AR) experience, virtual content may be strategically delivered to the user's eyes in a manner that takes into account the physiology and limitations of the human eye. The following disclosure will provide various embodiments of such optical systems that may be integrated into AR systems. While much of this disclosure in this specification will be discussed in the context of AR systems, the same technology may also be used for VR systems, and the following embodiments should not be read as limiting.

[0209] The following disclosures will provide details of various types of systems that AR users can interact with through the creation of maps containing comprehensive information about real-world physical objects in real time. These maps may, advantageously, be considered for projecting virtual images relating to known real-world objects. The following disclosures will provide various approaches to understanding and using information about the real world to deliver more realistic and engaging AR experiences. In addition, the disclosures will provide various user scenarios and applications in which AR systems, such as those described herein, can be implemented.

[0210] (System Overview) In one or more embodiments, the AR system 10 comprises a computing network 5 consisting of one or more computer servers 11 connected through one or more high-bandwidth interfaces 15. The servers 11 in the computing network may or may not be located in the same location. Each of the one or more servers 11 comprises one or more processors for executing program instructions. The servers may also include memory for storing program instructions and data used and / or generated by processes executed by the servers 11 under the direction of program instructions.

[0211] The computing network 5 communicates data between servers 11 and between servers and one or more user devices 12 via one or more data network connections 13. Examples of such data networks include, but are not limited to, any type of public and private data network, both mobile and wired, including, for example, the internet, which is commonly referred to as the Internet and involves many interconnections of such networks. No particular medium, topology, or protocol is intended to be implied by the diagram.

[0212] A user device is configured to communicate directly with either the computing network 5 or the server 11. Alternatively, a user device 12 communicates locally with the remote server 11, and optionally with other user devices, and through a specially programmed local gateway 14 for processing and / or communicating data between the network 5 and one or more local user devices 12.

[0213] As illustrated, the gateway 14 is implemented as a separate hardware component including a processor for executing software instructions and memory for storing software instructions and data. The gateway has its own wired and / or wireless connectivity to the data network for communicating with the server 11, which has the computing network 5. Alternatively, the gateway 14 can be integrated with a user device 12, which is worn or carried by the user. For example, the gateway 14 may be implemented as a downloadable software application that is installed and launched on a processor contained within the user device 12. In one embodiment, the gateway 14 provides one or more user access to the computing network 5 via the data network 13.

[0214] Each of the servers 11 includes, for example, working memory and storage for storing data and software programs, a microprocessor for executing program instructions, and a graphics processor and other specialized processors for rendering and generating graphics, images, video, audio, and multimedia files. The computing network 5 may also include devices for storing data accessed, used, or created by the servers 11.

[0215] Software programs launched on the server and optionally on the user device 12 and gateway 14 are used to generate a digital world (also referred to herein as a virtual world) in which the user interacts with the user device 12. The digital world (or map) (as will be described in more detail below) is represented by data and processes that describe and / or define virtual, non-existent entities, environments, and conditions that may be presented to the user through the user device 12 for the user to experience and interact with. For example, certain types of objects, entities, or items that would appear to exist physically when instantiated within a scene being viewed or experienced by the user may include descriptions of their appearance, their behavior, how the user is permitted to interact with them, and other characteristics.

[0216] The data used to create the environment of a virtual world (including virtual objects) may include, for example, atmospheric data, topographic data, weather data, temperature data, location data, and other data used to define and / or describe the virtual environment. In addition, the data that defines the various conditions governing the behavior of the virtual world may include, for example, physical laws, time, spatial relationships, and other data that can be used to define and / or create the various conditions governing the behavior of the virtual world (including virtual objects).

[0217] Entities, objects, conditions, characteristics, behaviors, or other features of the digital world will be generally referred to as objects (e.g., digital objects, virtual objects, rendered physical objects, etc.) in this specification unless otherwise indicated by context. Objects may be any type of living or non-living object, including, but not limited to, buildings, plants, vehicles, people, animals, creations, machines, data, videos, text, photographs, and other users. Objects may also be defined in the digital world to store information about items, behaviors, or conditions that actually exist in the physical world. Data that describes or defines an entity, object, or item, or stores its current state, will generally be referred to as object data in this specification. This data is processed by the server 11, or, depending on the implementation, by the gateway 14 or user device 12, to instantiate instances of the object and render the object in an appropriate form for the user to experience through the user device.

[0218] Programmers who develop and / or oversee the digital world create or define objects and the conditions under which they are instantiated. However, the digital world can allow others to create or modify objects. Once an object is instantiated, its state may be modified, controlled, or manipulated by one or more users experiencing the digital world.

[0219] For example, in one embodiment, the development, creation, and management of a digital world are generally provided by one or more system administrator programmers. In some embodiments, this may include the development, design, and / or execution of storylines, themes, and events within the digital world, as well as the distribution of narratives through various forms of events and media, such as movies, digital, network, mobile, augmented reality, and live entertainment. The system administrator programmer may also handle the technical management, coordination, and oversight of the digital world and its associated user community, as well as other tasks typically performed by network administrators.

[0220] The user interacts with one or more digital worlds using several types of local computing devices, generally designated as user devices 12. Examples of such user devices include, but are not limited to, smartphones, tablet devices, head-mounted displays (HMDs), game consoles, or any other devices capable of communicating data and providing an interface or display to the user, as well as combinations of such devices. In some embodiments, user devices 12 may include, or communicate with, local peripherals or input / output components such as, for example, keyboards, mice, joysticks, game controllers, tactile interface devices, motion capture controllers, optical tracking devices, audio equipment, voice equipment, projector systems, 3D displays, and / or holographic 3D contact lenses.

[0221] An embodiment of a user device 12 for interacting with system 10 is illustrated in Figure 2. In the exemplary embodiment shown in Figure 2, user 21 may interface with one or more digital worlds through a smartphone 22. The gateway is implemented by a software application 23 that is stored on and launched on the smartphone 22. In this particular embodiment, the data network 13 includes a wireless mobile network that connects the user device (e.g., smartphone 22) to the computer network 5.

[0222] In one preferred embodiment, the system 10 can support a large number of simultaneous users (e.g., millions of users) who interface with the same digital world or multiple digital worlds using several types of user devices 12.

[0223] The user device provides the user with an interface to enable visual, auditory, and / or physical interaction between the user and the digital world generated by the server 11, including other users and objects (real or virtual) presented to the user. The interface provides the user with a rendered scene that can be seen, heard, or otherwise perceived, and the ability to interact with the scene in real time. The manner in which the user interacts with the rendered scene may be determined by the capabilities of the user device. For example, if the user device is a smartphone, the user interaction may be implemented by the user touching a touchscreen. In another embodiment, if the user device is a computer or a game console, the user interaction may be implemented using a keyboard or a game controller. The user device may include additional components that enable user interaction, such as sensors, and objects and information (including gestures) detected by the sensors may be provided as inputs representing user interaction with the virtual world using the user device.

[0224] The rendered scene can be presented in various formats, such as two-dimensional or three-dimensional visual displays (including projections), sound, and tactile or haptic feedback. The rendered scene may be interfaced by the user in one or more modes, including, for example, augmented reality, virtual reality, and combinations thereof. The format of the rendered scene and the interface mode may be determined by one or more of the following: user device, data processing capacity, user device connectivity, network capacity, and system workload. The simultaneous interaction of multiple users with the digital world and the real-time nature of data exchange are made possible by the computing network 5, server 11, gateway component 14 (optional), and user device 12.

[0225] In one embodiment, the computing network 5 comprises a large-scale computing system having single and / or multicore servers (e.g., server 11) connected through high-speed connections (e.g., high-bandwidth interface 15). The computing network 5 may form a cloud or grid network. Each server is coupled with computer-readable memory for storing software, including memory, to implement data for creating, designing, modifying, or processing objects in the digital world. These objects and their instantiations may be dynamic, appearing and disappearing, changing over time, and changing in response to other conditions. Embodiments of the dynamic capabilities of objects are generally discussed herein with respect to various embodiments. In some embodiments, each user interfaced with system 10 may also be represented in one or more digital worlds as an object and / or a collection of objects.

[0226] Server 11 within the computing network 5 also stores compute state data for each digital world. Computation state data (also referred to herein as state data) can be components of object data and generally define the state of an instance of an object in a given time instance. Computation state data can therefore change over time and be affected by the actions of one or more users and / or programmers maintaining system 10. As users influence compute state data (or other data comprising a digital world), they directly modify or otherwise manipulate the digital world. If a digital world is shared with other users or interfaced by them, a user's actions can affect what is experienced by other users interacting with the digital world. Thus, in some embodiments, changes to a digital world made by a user will also be experienced by other users interfaced with system 10.

[0227] In one embodiment, data stored in one or more servers 11 within the computing network 5 is transmitted or unpacked to one or more user devices 12 and / or gateway components 14 at high speed and with low latency. In one embodiment, object data shared by the servers may be complete or compressed and include instructions for recreating the complete object data on the user side, which may be rendered and visualized by the user's local computing device (e.g., gateway 14 and / or user device 12). In some embodiments, software running on the servers 11 of the computing network 5 may adapt the data it generates and transmits to a particular user's device 12 as a function of the user's specific device and bandwidth for objects (or any other data) in the digital world exchanged by the computing network 5.

[0228] For example, when a user interacts with the digital world or a map through a user device 12, the server 11 may recognize the specific type of device being used by the user, the device's connectivity, and / or the available bandwidth between the user device and the server, and appropriately size and balance the data being sent to the device to optimize the user interaction. An embodiment of this may involve reducing the size of the transmitted data to a low-resolution quality so that the data can be displayed on a particular user device having a low-resolution display. In a preferred embodiment, the computing network 5 and / or gateway component 14 deliver data to the user device 12 at a rate sufficient to present an interface operating at a resolution of 15 frames / second or higher and of high-definition quality or higher.

[0229] The gateway 14 provides local connectivity to the computing network 5 for one or more users. In some embodiments, it may be implemented by a downloadable software application that runs on a user device 12 or another local device, such as the one shown in Figure 2. In other embodiments, it may be implemented by a hardware component (with appropriate software / firmware stored on the component, the component having a processor) that communicates with the user device 12 but is either not incorporated with it or incorporated into it, or is incorporated with the user device 12. The gateway 14 communicates with the computing network 5 via the data network 13 and provides data exchange between the computing network 5 and one or more local user devices 12. As will be discussed in more detail below, the gateway component 14 may include software, firmware, memory, and processing circuitry that can process the data communicated between the network 5 and one or more local user devices 12.

[0230] In some embodiments, the gateway component 14 monitors and adjusts the rate of data exchanged between the user device 12 and the computer network 5, enabling optimal data processing capabilities for a particular user device 12. For example, in some embodiments, the gateway 14 buffers and downloads both static and dynamic aspects of the digital world (even beyond the field of view presented to the user through the interface connected to the user device). In such embodiments, instances of static objects (structured data, software implementations, or both) may be stored in memory (local to the gateway component 14, the user device 12, or both) and referenced relative to the local user's current location, indicated by the data provided by the computing network 5 and / or the user's device 12.

[0231] For example, instances of dynamic objects, which may include intelligent software agents and objects controlled by other users and / or local users, are stored in a high-speed memory buffer. Dynamic objects representing two-dimensional or three-dimensional objects in a scene presented to the user can be divided into constituent shapes, such as static shapes that move but do not change and dynamic shapes that change. The changing portions of dynamic objects can be updated by a real-time threaded high-priority data stream from the server 11 through a computing network 5 managed by the gateway component 14.

[0232] As one example of a prioritized threaded data stream, data within a 60-degree field of view of the user's eye may be given higher priority than more peripheral data. Another example involves prioritizing dynamic characters and / or objects within the user's field of view over static objects in the background.

[0233] In addition to managing data connections between the computing network 5 and the user device 12, the gateway component 14 may store and / or process data that may be presented to the user device 12. For example, in some embodiments, the gateway component 14 may receive compressed data from the computing network 5 describing graphical objects to be rendered for viewing by the user, and perform advanced rendering techniques to reduce the data load transmitted from the computing network 5 to the user device 12. In another embodiment, where the gateway 14 is a separate device, the gateway 14 may store and / or process data for local instances of objects, rather than transmitting the data to the computing network 5 for processing.

[0234] Referring here to Figure 3, the virtual world may be achieved by one or more users in various formats that may depend on the capabilities of the user's device. In some embodiments, the user device 12 may include, for example, a smartphone, a tablet device, a head-mounted display (HMD), a game console, or a wearable device. Generally, the user device would include a processor for executing program code stored in memory on the device, coupled with a display, and a communication interface.

[0235] An exemplary embodiment of a user device is shown in Figure 3, which comprises a mobile wearable device, i.e., a head-mounted display system 30. According to one embodiment of the present disclosure, the head-mounted display system 30 includes a user interface 37, a user sensing system 34, an environment sensing system 36, and a processor 38. The processor 38 is shown in Figure 3 as a separate, isolated component from the head-mounted system 30, but in alternative embodiments, the processor 38 may be integrated with one or more components of the head-mounted system 30, or may be integrated into other system 10 components, such as a gateway 14, as shown in Figures 1 and 2.

[0236] The user device 30 presents the user with an interface 37 for interacting with and experiencing the digital world. Such interaction may involve the user and the digital world, one or more other users interface with the system 10, and objects within the digital world. The interface 37 generally provides the user with image and / or audio sensory input (and, in some embodiments, physical sensory input). Thus, the interface 37 may include a speaker (not shown) and, in some embodiments, a display component 33 capable of providing stereoscopic 3D viewing and / or 3D viewing that embodies more natural characteristics of the human visual system.

[0237] In some embodiments, the display component 33 may include a transparent interface (such as a clear OLED) that, when set to "off," allows for an optically correct view of the physical environment surrounding the user with little to no optical distortion or computing overlay. As will be discussed in more detail below, the interface 37 may include additional settings that enable various visual / interface performance and functionality.

[0238] In some embodiments, the user sensing system 34 may include one or more sensors 31 capable of detecting certain features, characteristics, or information related to individual users wearing the system 30. For example, in some embodiments, the sensors 31 may include a camera or optical detection / scanning circuit capable of detecting the user's real-time optical characteristics / measurements.

[0239] The user's real-time optical characteristics / measurements may include, for example, one or more of the following: pupillary constriction / dilation, angle measurement / positioning of each pupil, sphericity, eye shape (as eye shape changes over time), and other anatomical data. This data may be used by the head-mounted system 30 and / or interface system 10 to provide or calculate information (e.g., the user's visual focus) to optimize the user's visual experience. For example, in one embodiment, each of the sensors 31 may measure the pupillary constriction rate for each of the user's eyes. This data may be transmitted to the processor 38 (or gateway component 14 or server 11), and the data may be used, for example, to determine the user's response to the brightness settings of the interface display 33.

[0240] Interface 37 may be adjusted according to the user's response, for example by dimming the display 33 if the user's response indicates that the brightness level of the display 33 is too high. User sensing system 34 may include other components other than those described above or illustrated in Figure 3. For example, in some embodiments, user sensing system 34 may include a microphone for receiving voice input from the user. User sensing system 34 may also include one or more infrared camera sensors, one or more visible spectrum camera sensors, structured light emitters and / or sensors, infrared light emitters, coherent light emitters and / or sensors, a gyroscope, an accelerometer, a magnetometer, a proximity sensor, a GPS sensor, an ultrasonic emitter and detector, and a tactile interface.

[0241] The environmental sensing system 36 includes one or more sensors 32 for obtaining data from the physical environment surrounding the user. Objects or information detected by the sensors may be provided as input to the user device. In some embodiments, this input may represent user interaction with a virtual world. For example, a user viewing a virtual keyboard on a desk may make gestures with their fingers as if typing on the virtual keyboard. The movement of the fingers may be captured by the sensors 32 and provided as input to the user device or system, and the input may be used to modify the virtual world or create new virtual objects.

[0242] For example, finger movements may be recognized as typing (e.g., using a processor software program), and the recognized typing gestures may be combined with known locations of virtual keys on a virtual keyboard. The system may then render a virtual monitor that is displayed to the user (or another user interfaced with the system), which displays the text being typed by the user.

[0243] Sensor 32 may include, for example, an outward-facing camera or scanner to interpret scene information through structured infrared light projected continuously and / or intermittently. The environment sensing system (36) may be used to map one or more elements of the physical environment around the user by detecting and aligning with the local environment, including static objects, dynamic objects, people, gestures, and various lighting, atmospheric, and acoustic conditions. Thus, in some embodiments, the environment sensing system (36) may include image-based 3D reconstruction software embedded in a local computing system (e.g., gateway component 14 or processor 38) and capable of digitally reconstructing one or more objects or information detected by Sensor 32.

[0244] In one exemplary embodiment, the environment sensing system 36 provides motion capture data (including gesture recognition), depth sensing, face recognition, object recognition, unique object feature recognition, voice / audio recognition and processing, sound source location, noise reduction, infrared or similar laser projection, and one or more of a monochromatic and / or color CMOS sensor (or other similar sensor), a field of view sensor, and various other optical enhancement sensors.

[0245] It should be understood that the environmental sensing system 36 may include other components other than those described above or illustrated in Figure 3. For example, in some embodiments, the environmental sensing system 36 may include a microphone for receiving audio from the local environment. The user sensing system (36) may also include one or more infrared camera sensors, one or more visible spectrum camera sensors, structural light emitters and / or sensors, infrared light emitters, coherent light emitters and / or sensors, a gyroscope, an accelerometer, a magnetometer, a proximity sensor, a GPS sensor, an ultrasonic emitter and detector, and a tactile interface.

[0246] As described above, the processor 38 may be integrated with other components of the interface system 10, which in some embodiments are integrated with other components of the head-mounted system 30, or it may be an isolated device (wearable or separate from the user), as shown in Figure 3. The processor 38 may be connected to various components of the head-mounted system 30 and / or components of the interface system 10 via a physical wired connection or via a wireless connection such as a mobile network connection (including cellular and data networks), Wi-Fi, or Bluetooth®.

[0247] In one or more embodiments, the processor 38 may include a memory module, an integrated and / or additional graphics processing unit, a wireless and / or wired internet connection, and a codec and / or firmware capable of converting data from a source (e.g., a computing network 5, a user sensing system 34, an environmental sensing system 36, or a gateway component 14) into image and audio data, which may be presented to the user via an interface 37.

[0248] In one or more embodiments, the processor 38 handles data processing for various components of the head-mounted system 30 and data exchange between the head-mounted system 30 and the gateway component 14, and in some embodiments, the computing network 5. For example, the processor 38 may be used to buffer and process data streams between the user and the computing network 5, thereby enabling a smooth, continuous, and high-fidelity user experience.

[0249] In some embodiments, the processor 38 may process data at a rate sufficient to achieve, such as 8 frames / second at 320x240 resolution to 24 frames / second at high-definition resolution (1280x720), or 60-120 frames / second at 4k resolution and higher (50,000 frames / second at 10k+ resolution), etc. In addition, the processor 38 may store and / or process the data that may be presented to the user, rather than having it streamed in real time from the computing network 5.

[0250] For example, in some embodiments, the processor 38 may receive compressed data from the computing network 5 and perform advanced rendering techniques (such as illumination or shading) to reduce the data load transmitted from the computing network 5 to the user device 12. In another embodiment, the processor 38 may store and / or process local object data instead of transmitting data to the gateway component 14 or the computing network 5.

[0251] In some embodiments, the head-mounted system 30 may include various settings, i.e., modes, to enable various visual / interface performance and functionality. The modes may be selected manually by the user or automatically by the components of the head-mounted system 30 or the gateway component 14. As described above, an exemplary mode of the head-mounted system 30 includes an "off" mode in which the interface 37 substantially does not provide digital or virtual content. In "off" mode, the display component 33 may be transparent, thereby enabling an optically correct view of the physical environment around the user with little or no optical distortion or computing overlay.

[0252] In one exemplary embodiment, the head-mounted system 30 includes an "augmented" mode, and the interface 37 provides an augmented reality interface. In augmented mode, the interface display 33 may be substantially transparent, thereby allowing the user to view the local physical environment. Simultaneously, virtual object data provided by the computing network 5, processor 38, and / or gateway component 14 is provided on the display 33 in combination with the physical local environment. The following sections will examine various embodiments of the exemplary head-mounted user system that may be used for virtual and augmented reality purposes.

[0253] (User System) Referring to Figures 4A-4D, several common component options are illustrated. In part of the detailed explanation following the discussion in Figures 4A-4D, various systems, subsystems, and components are presented to address the objective of providing a high-quality, comfortable-to-perceive display system for human VR and / or AR.

[0254] As shown in Figure 4A, a user 60 of a head-mounted augmented reality system ("AR system") is depicted wearing a frame 64 structure, which is coupled to a display system 62 positioned in front of the user's eyes. In the depicted configuration, a speaker 66 is coupled to the frame 64 and positioned adjacent to the user's ear canal (in one embodiment, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / shapeable sound control). The display 62 is operably coupled to a local processing and data module 70, and can be mounted in various configurations, such as being fixedly attached to a frame 64 by wired or wireless connections, being fixedly attached to a helmet or hat 80 as shown in the embodiment of Figure 4B, being embedded in headphones, being detachably attached to the torso 82 of the user 60 in a certain configuration (for example, installed in a backpack (not shown)) as shown in the embodiment of Figure 4C, or being detachably attached to the waist 84 of the user 60 in a belt-coupled configuration as shown in the embodiment of Figure 4D.

[0255] The local processing and data module 70 may comprise a low-power processor or controller and digital memory such as flash memory, both of which may be used to assist in processing, caching, and storing data acquired and / or processed using the remote processing module 72 and / or remote data repository 74 for passage to the display 62 after processing or reading, and / or from sensors that can be operably coupled to the frame 64, such as (a) an image acquisition device (such as a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope.

[0256] The local processing and data module 70 may be operably coupled to the remote processing module 72 and the remote data repository 74 via wired or wireless communication links, etc., so that these remote modules (72, 74) are operably coupled to each other and available as resources to the local processing and data module 70 (76, 78). The processing module 70 may perform one or more computing tasks, including reading data from memory or one or more databases (e.g., cloud-based servers), in order to control the optical and other systems of the AR system and to provide virtual content to the user.

[0257] In one embodiment, the remote processing module 72 may comprise one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. In one embodiment, the remote data repository 74 may comprise a relatively large digital data storage facility that may be available through the internet or other networking configurations in a “cloud” resource configuration. In one embodiment, all data is stored, and all calculations are performed within the local processing and data module, enabling fully autonomous use from any remote module.

[0258] (Optical Embodiment) It should be understood that there are many approaches to presenting 3D virtual content to a user's eyes through the optical elements of a head-mounted user device. The following exemplary embodiments may be used in combination with other approaches and should not be read restrictively. The following exemplary embodiments represent several exemplary optical systems that may be integrated with a head-mounted user device (30) to enable the user to view virtual content in a comfortable and easily adjustable manner.

[0259] Referring to Figures 5A to 5Y, various display configurations are presented that are designed to present photon-based emission patterns to the human eye that can be comfortably perceived as an extension of physical reality, with high levels of image quality and three-dimensional perception, and capable of presenting two-dimensional content.

[0260] Referring to Figure 5A, in a simplified embodiment, a transmissive beam splitter substrate 104 with a 45-degree reflective surface 102 directs incident radiation 106, which can be output from a lens (not shown) through the pupil 45 of an eye 58 to the retina 54. The field of view for such a system is limited by the geometry of the beam splitter 104. To address the desire for comfortable viewing with minimal hardware, in one embodiment a larger field of view can be created by aggregating the outputs / reflections of various different reflective and / or diffracting surfaces. This can be achieved, for example, by using a frame-sequenced configuration in which a sequence of frames is presented to the eye 58 at high frequencies, providing the perception of a single coherent scene.

[0261] As an alternative to, or in addition to, presenting different image data in a time-series manner via different reflectors, the reflectors may separate the content by other means, such as polarization selectivity or wavelength selectivity. In addition to being able to relay two-dimensional images, the reflectors may also relay three-dimensional wavefronts associated with the true three-dimensional perception of actual physical objects.

[0262] Referring to Figure 5B, a substrate 108 is shown having multiple reflectors at multiple angles 110, each reflector actively reflecting in the configuration depicted for illustrative purposes. The reflectors may have switchable elements to facilitate temporal selectivity. In one embodiment, the reflective surfaces may be activated intentionally and sequentially with frame-sequential input information 106, and each reflective surface presents a narrow-field subimage tiled together with other narrow-field subimages presented by other reflective surfaces to form a composite wide-field image.

[0263] For example, referring to Figures 5C, 5D, and 5E, surface 110 (e.g., located in the center of substrate 108) is switched to a reflective state "on" to reflect incident image information 106 and display a relatively narrow field of view subimage in the center of a larger field of view, while other potential reflective surfaces are in a transmissive state.

[0264] Referring to Figure 5C, incident image information 106 originating from the right side of the narrow-field subimage (as indicated by the angle of the incident beam 106 relative to the substrate 108 at the input interface 112 and the angle at which it exits the substrate 108) is reflected from the reflective surface 110 toward the eye 58. Figure 5D illustrates that the same reflector 110 is active, and the image information originates from the center of the narrow-field subimage, as indicated by the angle of the input information 106 at the input interface 112 and the angle at which it exits the substrate 108.

[0265] Figure 5E illustrates that the same reflector 110 is active, and the image information originates from the left side of the field of view, as indicated by the angle of the input information 106 at the input interface 112 and the emission angle obtained on the surface of the substrate 108. Figure 5F illustrates a configuration in which the bottom reflector 110 is active, and the image information 106 originates from the far right of the overall field of view. For example, Figures 5C, 5D, and 5E may illustrate one frame representing the center of a frame-sequential tile display image, and Figure 5F may illustrate a second frame representing the far right of that tile display image.

[0266] In one embodiment, the light carrying the image information 106 may first enter the substrate 108 directly at the input interface 112 without being reflected from the surface of the substrate 108, and then strike the reflective surface 110. In one embodiment, the light carrying the image information 106 may be reflected from one or more surfaces of the substrate 108 after being entered at the input interface 112, before striking the reflective surface 110. For example, the substrate 108 may act as a planar waveguide, allowing the light carrying the image information 106 to propagate by total internal reflection. The light may also be reflected from one or more surfaces of the substrate 108 through a partial reflection coating, a wavelength-selective coating, an angle-selective coating, and / or a polarization-selective coating.

[0267] In one embodiment, an angled reflector may be constructed using an electroactive material such that, in response to the application of a voltage and / or current to a particular reflector, the refractive index of the material comprising such a reflector changes from a refractive index substantially matching that of the rest of the substrate 108. When the refractive index of the reflector matches that of the rest of the substrate 108, the reflector is in a transmissive configuration. When the refractive index of the reflector is mismatched with that of the rest of the substrate 108, the reflector is in a reflective configuration such that a reflective effect is created. Exemplary electroactive materials include lithium niobate and electroactive polymers. Suitable substantially transparent electrodes for controlling multiple such reflectors may include materials such as indium tin oxide, which are used in liquid crystal displays.

[0268] In one embodiment, the electroactive reflector 110 may comprise a liquid crystal embedded in a host medium such as a substrate 108, which may be glass or plastic. In some modifications, a liquid crystal may be selected that changes its refractive index as a function of an applied electrical signal, allowing for more analog changes to be performed, as opposed to binary changes (from one transparent state to one reflective state). In one embodiment, it is desirable to have an electroactive reflector array that can maintain such a frequency, along with an input display that can refresh at a rate of about 360 Hz, where six sub-images are presented to the eye in a frame-sequential manner, forming a large tiled display image at an overall refresh rate of 60 frames / second.

[0269] In one embodiment, lithium niobate may be used as an electroactive reflective material, in contrast to liquid crystal. Lithium niobate is used in the optical engineering industry for high-speed switches and optical fiber networks and has the ability to switch refractive indices at ultra-high frequencies in response to an applied voltage. These high frequencies may be used, in particular, to manipulate line-sequential or pixel-sequential sub-image information when the input display is a scanning optical display such as a fiber scanning display or a scanning mirror-based display.

[0270] In another embodiment, a variable-switchable angled mirror configuration may comprise one or more high-speed mechanically repositionable reflective surfaces, such as a MEMS (Micro-Electro-Mechanical Systems) device. The MEMS device may include what is known as a “digital mirror device,” i.e., a “DMD,” (often part of a “digital photoprocessing,” i.e., “DLP” system, such as those available from Texas Instruments, Inc.). In another electromechanical embodiment, multiple air-separated (or vacuum-sealed) reflective surfaces may be mechanically moved in and out of position at high frequencies. In yet another electromechanical embodiment, a single reflective surface may be moved up and down and re-pitched at ultra-high frequencies.

[0271] Referring to Figure 5G, it should be noted that the switchable variable-angle reflector configuration described herein allows collimated or flat wavefront information to pass not only to the retina 54 of the eye 58, but also to curved wavefront 122 image information, as illustrated in Figure 5G. This is not the case with other waveguide-based configurations, where the total internal reflection of curved wavefront information generally leads to undesirable complexity, and therefore the input generally must be collimated. The ability to pass curved wavefront information facilitates the ability of configurations such as those shown in Figures 5B-5H to provide the retina 54 with an input that is perceived as it is focused not only at optical infinity (which would be an interpretation of collimated light in the absence of other cues), but also at various distances from the eye 58.

[0272] Referring to Figure 5H, in another embodiment, an array of static partial reflective surfaces 116 (e.g., always in reflective mode; in another embodiment, as previously stated, they may be electroactive) may be embedded in the substrate 114 together with a high-frequency gating layer 118 that controls the output of the eye 58. The high-frequency gating layer 118 may be controllably movable and allow only transmission through the aperture 120. In other words, all transmission other than through the aperture 120 may be selectively blocked. The gating layer 118 may comprise a liquid crystal array, a lithium niobate array, an array of MEMS shutter elements, an array of DLP DMD elements, or an array of other MEMS devices configured to pass or transmit with relatively high frequency switching and high transmittance depending on the transmission mode.

[0273] Referring to Figures 6A-6B, another embodiment is depicted in which array-like optical elements are combined with an exit pupil enlargement configuration to enhance the user's comfort in the virtual or augmented reality experience. The larger the "exit pupil" for the optical configuration, the less the user's eye positioning relative to the display (which may be mounted on the user's head in a spectacle-like configuration, as shown in Figures 4A-4D) interferes with the experience, because, due to the larger exit pupil of the system, there is still a larger acceptable area in which the user's anatomical pupil can still be positioned to receive information from the display system as desired. In other words, the larger the exit pupil, the less sensitive the system is to slight misalignments of the display with the user's anatomical pupil, and the fewer geometric constraints on its relationship with the display / spectacles, thus allowing for greater user comfort.

[0274] Referring here to Figures 6A and 6B, alternative approaches are illustrated. As shown in Figure 6A, the left-hand display 140 supplies a set of parallel rays into the substrate 124. In one embodiment, the display may be a scanning fiber display that projects an image through a lens or other optical element 142 which can be used to back-scan a narrow beam of light at an angle as shown, collect the angle-scanned light, and convert it into a parallel beam of rays. The rays may be reflected from a set of reflective surfaces (126, 128, 130, 132, 134, 136) which can partially reflect and partially transmit the incident light so that the light can be shared substantially equally across the group of reflective surfaces (126, 128, 130, 132, 134, 136). Small lenses 138 are positioned at each exit point from the waveguide 124, and the exit rays are directed through the nodes and scanned toward the eye 58, which can provide a functional equivalence of an array of exit pupils or one large exit pupil usable by the user when gazing at the display system.

[0275] For a virtual reality configuration where it is desirable that the real world 144 is also visible through the waveguide, an analogous set of lenses 139 may be presented on the opposite side of the waveguide 124 to compensate for the subset of lenses and thus create an equivalent of the zero-magnification telescope. Each of the reflective surfaces (126, 128, 130, 132, 134, 136) may be matched at approximately 45 degrees as shown, or may be configured to have different matches (for example, analogous to the configuration in Figures 5B-5H). The reflective surfaces (126, 128, 130, 132, 134, 136) may comprise wavelength-selective reflectors, band-pass reflectors, semi-transparent mirrors, or other reflective configurations. The shown lenses (138, 139) are refractive lenses, but diffractive lens elements may also be utilized.

[0276] Referring to Figure 6B, a somewhat similar configuration is depicted in which multiple curved reflective surfaces (148, 150, 152, 154, 156, 158) can be used to effectively combine the functionality of the lens (element 138 in Figure 6A) and reflector (elements 126, 128, 130, 132, 134, 136 in Figure 6A) in the embodiment of Figure 6A, thereby eliminating the need for two lens groups (element 138 in Figure 6A).

[0277] The curved reflective surfaces (148, 150, 152, 154, 156, 158) may have various curvature configurations, such as parabolic or elliptical curved surfaces, selected to perform both reflection and angle change. In a parabolic shape, a parallel set of incident rays will be collected into a single output point. In an elliptical configuration, a set of rays diverging from a single generation point will be collected into a single output point. Similar to the configuration in Figure 6A, the curved reflective surfaces (148, 150, 152, 154, 156, 158) preferably partially reflect and partially transmit incident light so that it is shared across the length of the waveguide 146. The curved reflective surfaces (148, 150, 152, 154, 156, 158) may comprise wavelength-selective notch reflectors, semi-transparent mirrors, or other reflective configurations. In another embodiment, the curved reflective surfaces (148, 150, 152, 154, 156, 158) may be replaced with diffractive reflectors that reflect and deflect light.

[0278] Referring to Figure 7A, the perception of the Z-axis difference (e.g., the straight-line distance from the eye along the optical axis) can be enhanced by using a waveguide and a variable-focus optical element configuration in combination. As shown in Figure 7A, image information from the display 160 may be collimated and fed into the waveguide 164 and distributed in a large-exit pupil manner, for example, using a configuration such as that described with reference to Figures 6A and 6B, or other substrate-guided optics methods known to those skilled in the art, and then the variable-focus optical element capability may be used to change the focus of the wavefront of the light emerging from the waveguide, providing the eye with the perception that the light emanating from the waveguide 164 is from a specific focal length.

[0279] In other words, the incident light is collimated to avoid the challenges in an all-internal reflective waveguide configuration, and therefore exits in a collimated manner, requiring the viewer's eye to adjust for distance so that the far point is in focus on the retina, and consequently will be interpreted as being from optical infinity unless some other intervention refocuses the light and causes it to be perceived as being from a different viewing distance. One suitable such intervention is a variable focus lens.

[0280] In the embodiment of Figure 7A, collimated image information from the display 160 is introduced into the glass 162 or other material piece at an angle such that it undergoes total internal reflection and passes into the adjacent waveguide 164. The waveguide 164 may be configured similarly to the waveguides in Figures 6A or 6B (124, 146, respectively) such that the collimated light from the display is distributed to exit substantially uniformly along the length of the waveguide, across the distribution of reflectors or diffraction features. Depending on the exit toward the eye 58, in the configuration described, the exiting light passes through the variable focus lens element 166, and depending on the controlled focus of the variable focus lens element 166, the light exiting the variable focus lens element 166 and incident on the eye 58 will have varying levels of focus (a collimated flat wavefront represents optical infinity, and a larger beam divergence / wavefront curvature represents a closer viewing distance to the eye 58).

[0281] To compensate for the variable focus lens element 166 between the eye 58 and the waveguide 164, another similar variable focus lens element 167 is placed on the opposite side of the waveguide 164 to cancel out the optical effect of lens 166 on light originating from the world 144 for augmented reality (for example, as previously mentioned, one lens compensates for the other, creating a functional equivalent of a zero-magnification telescope).

[0282] The variable focus lens element 166 may be a liquid crystal lens, an electroactive lens, a conventional refractive lens with a movable element, a mechanically deformable lens (such as a fluid-filled film lens, or a lens similar to the human crystalline lens in which a flexible element is bent and relaxed by an actuator), an electrowetting lens, or a refractive element such as a plurality of fluids with different refractive indices.

[0283] The variable focus lens element 166 may also include a switchable diffractive optical element (such as one characterized by a polymer-dispersed liquid crystal approach in which a host medium such as a polymer material has microdroplets of liquid crystal dispersed within the material, and when a voltage is applied, the molecules are reoriented such that their refractive index no longer matches that of the host medium, thereby creating a high-frequency switchable diffraction pattern).

[0284] One embodiment includes a host medium in which microdroplets of a Kerr effect-based electroactive material, such as lithium niobate, are dispersed and coupled with a scanning light display, such as a fiber scanning display or a scanning mirror-based display, enabling refocusing of image information on a pixel-by-pixel or row-by-row basis. In a variable focus lens element 166 configuration in which liquid crystal, lithium niobate, or other techniques are used to represent a pattern, the pattern spacing can be modulated not only to change the focal force of the variable focus lens element 166 but also to change the focal force of the overall optical system for zoom lens-type functionality.

[0285] In one embodiment, the lens 166 may be telecentric in that the focus of the display image can be altered while maintaining a constant magnification, in the same way that a zoom lens for photography may be configured to decouple the focus from the zoom position. In another embodiment, the lens 166 may be non-telecentric so that the focus change will also follow the zoom change. In such a configuration, such magnification changes may be compensated in software using dynamic scaling of the output from a graphics system synchronized with the focus change.

[0286] Returning to the issue of how to supply images into the optical display system with the projector or other video display unit 160 in a "frame-sequential" configuration, a stack of sequential two-dimensional images can be sequentially supplied to the display in a manner similar to a computed tomography system, where the stacked image slices represent a three-dimensional structure, thereby generating three-dimensional perception over time.

[0287] A series of two-dimensional image slices may be presented to the eye, each at a different focal length relative to the eye, and the eye / brain will integrate such a stack into the perception of a coherent three-dimensional volume. Depending on the display type, row-by-row or even pixel-by-pixel sequence processing may be performed to generate the perception of three-dimensional viewing. For example, in a scanning light display (such as a scanning fiber display or scanning mirror display), the display presents one line or one pixel at a time in a sequential manner to the waveguide 164.

[0288] If the variable focus lens element 166 is capable of maintaining high-frequency pixel-by-pixel or row-by-row presentation, each row or pixel may be presented through the variable focus lens element 166, dynamically focused, and perceived by the eye 58 at different focal lengths. Pixel-by-pixel focus modulation generally requires an ultrafast / high-frequency variable focus lens element 166. For example, a 1080P resolution display with an overall frame rate of 60 frames / second typically represents about 125 million pixels / second. Such configurations may also be constructed using solid switchable lenses, such as those using electroactive materials, e.g., lithium niobate or electroactive polymers. In addition to its compatibility with the system illustrated in Figure 7A, the frame-sequential multifocal display driving approach may be used in conjunction with several display systems and optical embodiments described herein.

[0289] Referring to Figure 7B, the electroactive layer 172 (which may comprise liquid crystal or lithium niobate, for example) may be surrounded by a waveguide 168, which includes functional electrodes (170, 174) (which may be made from indium tin oxide) and a conventional transparent substrate 176. In one or more embodiments, the waveguide may be made of glass or plastic, having known total internal reflection properties and refractive index that match the on or off state of the electroactive layer 172. The electroactive layer 172 may be controlled so that the path of the incident beam can be dynamically modified to essentially create a time-varying brightfield.

[0290] Referring to Figure 8A, a stacked waveguide assembly 178 may be used to provide three-dimensional perception to the eye / brain by having multiple waveguides (182, 184, 186, 188, 190) and multiple weak lenses (198, 196, 194, 192) configured together to transmit image information to the eye at various levels of wavefront curvature for each waveguide level, indicating the focal length to be perceived relative to that waveguide level. Multiple displays (200, 202, 204, 206, 208), or in another embodiment, a single multiplexed display, may be used to load collimated image information into waveguides (182, 184, 186, 188, 190), each of which may be configured to distribute incident light substantially equally across the length of each waveguide for emission to the eye, as described above.

[0291] The waveguide 182 closest to the eye is configured to deliver collimated light to the eye as it is introduced into such waveguide 182, which may represent an optical infinity focal plane. Another waveguide 184 is configured to transmit collimated light that passes through a first weak lens (192; e.g., a weak negative lens) and is delivered to the user's eye 58. The first weak lens 192 may be configured to create a slight convex wavefront curvature so that the eye / brain interprets the light originating from waveguide 184 as originating from a first focal plane closer to the person and inward from optical infinity. Similarly, the next waveguide 186 passes its output light through the first 192 and second 194 lenses before reaching the eye 58. The combined refractive power of the first 192 and second 194 lenses may be configured to create another gradually increasing wavefront divergence so that the eye / brain interprets the light originating from waveguide 186 as originating from a second focal plane that is further inward toward the person and closer to optical infinity than the light originating from waveguide 184.

[0292] Other waveguide layers (188, 190) and weak lenses (196, 198) are configured similarly, and the highest waveguide 190 in the stack transmits its output through all the weak lenses between it and the eye for a cohesive focal force representing the focal plane closest to the person. When viewing / interpreting light originating from the other side world 144 of the stacked waveguide assembly 178, a compensating lens layer 180 is positioned on top of the stack to compensate for the stack of lenses (198, 196, 194, 192) and to compensate for the cohesive refractive force of the lower lens stack (198, 196, 194, 192).

[0293] Such a configuration, as described above, again provides a relatively large exit pupil configuration for the same number of perceived focal planes as the available waveguide / lens pairings. Both the reflective side of the waveguide and the focusing side of the lens may be static (e.g., not dynamic or electroactive). In alternative embodiments, they may be dynamic and, as described above, use electroactive features, allowing a small number of waveguides to be multiplexed in a time-series manner to generate a larger number of effective focal planes.

[0294] Referring to Figures 8B-8N, various aspects of diffraction configurations for focusing and / or redirecting a collimated beam are depicted. Other aspects of diffraction systems for such purposes are disclosed in U.S. Patent Application No. 14 / 331,218.

[0295] Referring to Figure 8B, it should be understood that passing a collimated beam, such as one made with a Bragg grating, through a linear diffraction pattern 210 will deflect, or "steer," the beam. Furthermore, it should be understood that passing a collimated beam through a radially symmetric diffraction pattern 212, or a "Fresnel zone plate," will change the beam's focus. Figure 8C illustrates the deflection effect of passing a collimated beam through a linear diffraction pattern 210. Figure 8D illustrates the focusing effect of passing a collimated beam through a radially symmetric diffraction pattern 212.

[0296] Referring to Figures 8E and 8F, a combined diffraction pattern having both linear and radial elements 214 generates both deflection and focusing of the collimated input beam. These deflection and focusing effects can be generated in both reflective and transmissive modes. These principles may be applied, for example, with waveguide configurations to enable additional optical system control, as shown in Figures 8G-8N.

[0297] As shown in Figures 8G-8N, the diffraction pattern (220), i.e., the “diffractive optical element” (or “DOE”), is embedded within the plane waveguide 216 so as the collimated beam undergoes total internal reflection along the plane waveguide 216, intersecting the diffraction pattern 220 at numerous locations.

[0298] Preferably, the DOE220 has relatively low diffraction efficiency such that only a portion of the beam light is deflected toward the eye 58 using each intersection of the DOE220, while the remainder continues to travel through the planar waveguide 216 via total internal reflection. The light carrying the image information is therefore split into several related light beams that exit the waveguide at numerous locations, resulting in a very uniform pattern of exit emission toward the eye 58 for this particular collimated beam bouncing within the planar waveguide 216, as shown in Figure 8H. The exit beam toward the eye 58 is shown in Figure 8H as substantially parallel, since the DOE220 in this case has only a linear diffraction pattern. As shown in the comparison between Figures 8L, 8M, and 8N, a change in the pitch of this linear diffraction pattern may be used to controllly deflect the exit parallel beam, thereby generating scanning or tile display functionality.

[0299] Referring to Figure 8I, as the radially symmetric diffraction pattern components of the embedded DOE 220 change, the emitted beam pattern becomes more divergent, requiring the eye to adjust for closer distances and focus on the retina, and will be interpreted by the brain as light from a viewing distance closer to the eye than optical infinity. Referring to Figure 8J, with the addition of another waveguide 218 into which a beam may be injected (e.g., by a projector or display), the DOE 221 embedded in this other waveguide 218, such as a linear diffraction pattern, may function to diffuse the light across the entire larger planar waveguide 216. This may provide the eye 58 with a very large incident field of incident light emitting from the larger planar waveguide 216, e.g., a large eyebox, according to the specific DOE configuration in operation.

[0300] The DOE (220, 221) is depicted as bisecting the associated waveguide (216, 218), although this is not required. In one or more embodiments, they may be positioned closer to or on either side of the waveguide (216, 218) so as to have identical functionality. Thus, as shown in Figure 8K, with the input of a single collimated beam, the entire field of cloned collimated beams can be directed toward eye 58. In addition, in combined linear diffraction pattern / radially symmetric diffraction pattern scenarios such as those depicted in 214 of Figure 8F and 220 of 8I, beam distribution waveguide optics are presented with Z-axis focusing capability (for functionality such as functional extension of the exit pupil; using configurations such as those in Figure 8K, the exit pupil can be the same size as the optical element itself, which can be a very significant advantage for user comfort and ergonomics), and both the divergence angle of the cloned beam and the wavefront curvature of each beam represent light originating from a point closer than optical infinity.

[0301] In one embodiment, one or more DOEs are switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer-dispersed liquid crystal where microdroplets have a diffraction pattern within the host medium, and the refractive index of the microdroplets can be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light). Alternatively, the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0302] Furthermore, as shown in Figures 8L-8N, beam scanning or tile display functionality can be achieved through dynamic changes to diffraction terms such as the linear diffraction pitch term. As mentioned above, it may be desirable for each of the DOEs (220, 221) to have relatively low diffraction grating efficiencies in order to facilitate the distribution of light. Also, since the light generated through the waveguide, which is desirable to be transmitted (for example, the light generated from world 144 toward eye 58 in an augmented reality configuration), is less affected when the diffraction efficiency of the DOE 220 it crosses is lower, a better view of the real world can be achieved through such a configuration.

[0303] Configurations such as those illustrated in Figure 8K are preferably driven in conjunction with the input of image information in a time-series approach, while frame-sequential driving is the easiest to implement. For example, an image of the sky at optical infinity may be input at time 1, and a diffraction grating that preserves light collimation may be used. Then, an image of a closer tree branch may be input at time 2, while the DOE controls the change in focus, for example, at a distance of 1 diopter or 1 meter, providing the eye / brain with the perception that the light information of the branch is coming from a closer focal distance.

[0304] This type of paradigm may be repeated in a fast time-series manner such that the eye / brain perceives the input as the entirety of the same image. While this is only an embodiment of two focal planes, it should be understood that preferably the system would have more focal planes and be configured to provide smoother transitions between objects and their focal lengths. This type of configuration generally assumes that the DOE is switched at a relatively slow rate (e.g., synchronized with the frame rate of a display that feeds images in the range of tens to hundreds of cycles / second).

[0305] The opposite configuration may involve a DOE element that can shift focus by tens to hundreds of MHz or more, facilitating the switching of the DOE element's focus state on a per-pixel basis as the pixels are scanned into the eye 58 using a scanning light display type approach. This is desirable because it means the overall display frame rate can be kept very low (low enough to ensure that "flicker" is not an issue (it is in the range of about 60 to 120 frames / second)).

[0306] Between these ranges, if the DOE can be switched at a kHz rate, the focus on each scan line may be adjusted on a line-by-line basis, which may give the user a visual advantage in terms of temporal artifacts, for example, during eye movements to the display. For example, different focal planes in a scene can thus be interleaved and minimize visual artifacts in response to head movements (as will be discussed in detail later in this disclosure). The line-by-line focus modulator may be operably coupled to a line-scanning display, such as a grating-valve display, where a linear array of pixels is swept to form an image, or to a scanning light display, such as a fiber-optic scanning display and a mirror-scanning light display.

[0307] A stacked configuration similar to that in Figure 8A may simultaneously provide multi-plane focusing using dynamic DOE (rather than the static waveguide and lens in the embodiment of Figure 8A). For example, with three simultaneous focal planes, a primary focal plane (e.g., based on measured ocular accommodation) may be presented to the user, and + and - boundaries (e.g., one focal plane is closer and the other is further away) may be utilized to provide a large focal length that the user can adjust to before a plane update is required. This increased focal length can provide a time advantage when the user switches to a closer or further focal (e.g., as determined by accommodation measurement). The new focal plane may then be the central depth of focus, and the + and - boundaries are again ready for a fast switch to either one while the system catches up.

[0308] Referring to Figure 8O, a stack of planar waveguides (244, 246, 248, 250, 252) is shown, each having reflectors (254, 256, 258, 260, 262) at one end, configured such that collimated image information fed into one end by a display (224, 226, 228, 230, 232) bounces back by all internal reflections up to the reflectors, at which point some or all of the light is reflected toward the eye or another target. Each of the reflectors may have a slightly different angle so that all of the outgoing light is reflected toward a common destination such as the pupil. Such a configuration is somewhat similar to that in Figure 5B, but each of the differently angled reflectors in the embodiment of Figure 8O has its own waveguide so that the projected light hardly interferes when it is traveling toward the target reflector. Lenses (234, 236, 238, 240, 242) may be interposed between the display and the waveguide for beam steering and / or focusing.

[0309] Figure 8P illustrates a geometrically alternating version in which the reflectors (276, 278, 280, 282, 284) are positioned at alternating lengths within the waveguides (266, 268, 270, 272, 274) so ​​that the emitted beam can be relatively easily aligned with an object such as an anatomical pupil. Since the distance between the stack (264) and the eye (typically a comfortable geometry, such as 28 mm between the cornea of ​​the eye and the spectacle lens) is known, the geometry of the reflectors (276, 278, 280, 282, 284) and waveguides (266, 268, 270, 272, 274) may be set to fill the pupil of the eye with emitted light (typically about 8 mm or less in width).

[0310] By directing light to an eyebox larger than the diameter of the pupil, the viewer is free to move their eyes any number of times w...

Claims

1. An augmented reality (AR) device, wherein the AR device is A first image generator for displaying a first image to the user's first eye, A first inwardly directed light source for illuminating the user's first eye in a first pattern to facilitate tracking of the user's first eye, Processor and Equipped with, The aforementioned processor, A first driver for providing first image information and first control signals to the first image generator, A first controller is communicably coupled to the first inwardly directed light source. Equipped with, An AR device in which the first controller is configured to randomly vary the operating parameters of the first light source directed inward, thereby randomly varying the first pattern.

2. The AR device according to claim 1, wherein the first inwardly directed light source is a first light-emitting diode (LED).

3. The AR device according to claim 1, wherein the first pattern is a spatial pattern.

4. The AR device according to claim 1, wherein the first pattern is a temporal pattern.

5. The AR device according to claim 1, wherein illuminating the user's first eye in the first pattern facilitates the detection of the user's first eye.

6. The AR device according to claim 1, wherein illuminating the user's first eye in the first pattern facilitates monitoring the orientation of the user's first eye.

7. The AR device according to claim 1, wherein illuminating the user's first eye in the first pattern facilitates monitoring of the user's first eye movement.

8. A second image generator for displaying a second image to the user's second eye, A second inwardly directed light source for illuminating the user's second eye in a second pattern to facilitate tracking of the user's second eye, Furthermore, The AR device according to claim 1, further comprising a second controller communicably coupled to the second inwardly directed light source, wherein the processor further comprises a second controller.

9. The AR device according to claim 8, wherein the second inwardly directed light source is a second LED.

10. The AR device according to claim 8, wherein the second pattern is a spatial pattern.

11. The AR device according to claim 8, wherein the second pattern is a temporal pattern.

12. The AR device according to claim 8, wherein illuminating the user's second eye in the second pattern facilitates the detection of the user's second eye.

13. The AR device according to claim 8, wherein illuminating the user's second eye in the second pattern facilitates monitoring of the orientation of the user's second eye.

14. The AR device according to claim 8, wherein illuminating the user's second eye in the second pattern facilitates monitoring of the user's second eye movement.

Images