Methods and systems for creating virtual and augmented reality

Abstract

To provide methods and systems for creating virtual and augmented reality.SOLUTION: Configurations are disclosed for presenting virtual reality and augmented reality experiences to users. The systems may comprise an image capturing device to capture one or more images, the one or more images corresponding to a field of view of a user of a head-mounted augmented reality device, and a processor communicatively coupled to the image capturing device to extract a set of map points from the set of images, to identify a set of sparse points and a set of dense points from the extracted set of map points, and to perform normalization on the set of map points.SELECTED DRAWING: Figure 1

Description

Background Art

[0001] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences in which digitally reproduced images or portions thereof are presented to a user in a manner that appears or is perceived to be real. Virtual reality, i.e., "VR" scenarios, typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs, and augmented reality, i.e., "AR" scenarios, typically involve the presentation of digital or virtual image information as an extension of the visualization of the actual world surrounding the user. For example, an augmented reality scene may enable a user of AR technology to see one or more virtual objects superimposed on or in between real-world objects (e.g., a setting such as a real-world park featuring people, trees, buildings in the background, etc.).

[0002] The human visual perception system is very complex, and it is difficult to generate VR or AR technologies that facilitate a comfortable, natural, and rich presentation of virtual image elements among other virtual or real-world image elements. Conventional stereoscopic head-mounted glasses typically feature two displays configured to display an image with the presentation of slightly different elements such that a three-dimensional perspective is perceived by the human visual perception system. Such a configuration has been found to be uncomfortable for many users due to an inconsistency between vergence movement and accommodation that can be overcome to perceive an image in three dimensions. In fact, some users are unable to tolerate a stereoscopic configuration.

[0003] Several optical configurations (e.g., head-mounted glasses) are available (e.g., Google Glass™, Occulus Rift™, etc.), but none of these configurations are optimally suited to presenting a rich binocular three-dimensional augmented reality experience in a manner that would be comfortable and maximally useful to the user, because they all, in part, fail to address some of the fundamental aspects of the human perceptual system, including the retinal photoreceptor cells and their interaction with the brain for providing the perception of visualization to the user.

[0004] The human eye is a very complex organ and typically includes a cornea, iris, lens, macula, retina, and the optic nerve pathway to the brain. The macula is the center of the retina and is used for viewing medium-detail. At the center of the macula is a part of the retina called the "fovea", which is used for viewing the finest details of a scene and contains more photoreceptor cells (about 120 cones per degree of vision) than any other part of the retina.

[0005] The human visual system is not a passive sensor type of system. That is, it actively scans the environment. In a manner somewhat similar to the use of a flatbed scanner to capture an image or the use of a finger to read Braille from paper, the photoreceptor cells of the eye do not respond uniformly to a constant state of stimulation, but rather emit signals in response to changes in stimulation. Therefore, movement is required to present photoreceptor cell information to the brain.

[0006] In fact, experiments using substances such as cobra venom, which are used to paralyze the eye muscles, have shown that a human subject will become blind if positioned while their eyes are open and viewing a static scene with eyes paralyzed by the venom. In other words, in the absence of changes in stimulation, the photoreceptor cells do not provide input to the brain and blindness occurs. This is thought to be at least one reason why normal human eyes are observed to move back and forth, i.e., to make small movements, in a lateral movement also known as "microsaccades".

[0007] As described above, the fovea of the retina contains the maximum density of photoreceptor cells. Typically, humans are recognized to have a perception with high-resolution visualization ability throughout the entire visual field. In reality, however, humans have only a small high-resolution center that mechanically scans almost constantly, along with a persistent memory of the high-resolution information most recently captured at the fovea. In a somewhat similar manner, the eye's focal length control mechanism (e.g., the ciliary muscle operably coupled to the lens in such a way that ciliary muscle relaxation causes tension in the ciliary zonular fibers, flattening the lens for a more distant focal length, and ciliary muscle contraction causes relaxation of the ciliary zonular fibers, enabling the lens to assume a more spherical geometry for a closer focal length) reciprocates by approximately 1 / 4 to 1 / 2 diopter to periodically induce a small amount of "optical refractive blur" both proximal and distal to the target focal length. This is utilized by the brain's depth perception control circuit as periodic negative feedback that helps to constantly correct the line of sight and keep the retinal image of a stationary object approximately in focus.

[0008] The brain's visualization center also obtains useful perceptual information from the relative motion of both eyes and their components. The relative binocular divergence motion of both eyes (e.g., the rolling of the pupils towards or away from each other to converge the line of sight of the eyes and fix on an object) is closely associated with the focusing (or "accommodation") of the eye's lens. Under normal conditions, changing the focus of the eye's lens, i.e., accommodating the eye and focusing on objects at different distances, will automatically cause a coordinated change in the binocular divergence motion to the same distance under the relationship known as the "accommodation-binocular divergence reflex." Similarly, a change in binocular divergence motion will, under normal conditions, also induce a coordinated change in accommodation. Actions that oppose this reflex are known to cause eye fatigue, headache, or other forms of discomfort to the user (as in most conventional stereoscopic AR or VR configurations).

[0009] The storage of the eyes, and also the movement of the head, also have a significant impact on the visualization of objects. Humans tend to move their heads and visualize the world around them, and often, in a very steady state, reposition and reorient their heads with respect to the object of interest. Furthermore, most people prefer to move their heads when their line of sight needs to move more than about 20 degrees from the center in order to focus on a particular object (for example, people typically do not prefer to look at an object "from the corner of their eye"). Humans also typically scan or move their heads in conjunction with sound, to improve audio signal capture and utilize the geometry of the ears relative to the head. The human visual system obtains excellent depth cues from what is called "head motion parallax", which is related to the relative motion of objects at different distances as a function of head movement and the vergence distance of the eyes. In other words, when a person moves their head laterally and maintains a fixed position with respect to an object, items that are farther away from that object will move in the same direction as the head, and items in front of that object will move in the opposite direction of the head movement. These can be very significant cues as to where an object is spatially located within the environment relative to the person. Head movement is also, of course, utilized to look around objects.

[0010] Furthermore, the movements of the head and eyes are coordinated with the "vestibulo-ocular reflex," which stabilizes image information on the retina during head rotation and thus keeps object image information near the center of the retina. In response to head rotation, the eyes rotate reflexively and proportionally in the opposite direction to maintain a stable fixation on the object. As a result of this compensatory relationship, many humans can read a book while oscillating their head back and forth. Interestingly, the same generally does not hold true when the book is flipped back and forth at a constant speed with the head remaining mostly stationary. That is, a person is likely unable to read the flipping book. The vestibulo-ocular reflex is one of the head and eye movement coordinations and is generally not developed for hand movement. This paradigm can be important for an AR system because the user's head movement can be relatively directly associated with eye movement, and an ideal system would preferably be prepared to cooperate with this relationship.

[0011] Indeed, assuming these various relationships, when installing digital content (e.g., 3-D content such as a virtual chandelier object presented to expand the real-world view of a room, or 2-D content such as a flat / virtual oil painting object presented to expand the real-world view of a room), design choices can be made to control the behavior of the object. For example, a 2-D oil painting object can be centered on the head, in which case the object moves with the user's head (e.g., as in the GoogleGlass (registered trademark) approach). In another embodiment, the object can be centered on the world, in which case it can be presented as if it were part of the real-world coordinate system such that the user can move their head or eyes without moving the object's position relative to the real world.

[0012] Therefore, when placing virtual content within the augmented reality world presented using an AR system, a selection is made as to whether the object should be presented centered on the world, centered on the body, centered on the head, or centered on the eyes. In the head-centered approach, the virtual object remains at a fixed position within the real world such that the user can move their body, head, and eyes around it without changing its position relative to real-world objects surrounding it, such as real-world walls. In the body-centered approach, the virtual element can be fixed relative to the user's torso such that the user can move their head or eyes without moving the object, but is driven by the movement of the torso. In the head-centered approach, the object being displayed (and / or the display itself) can be moved with the movement of the head, as previously described with reference to Google Glass™. In the eye-centered approach, as in the "foveated display" configuration described below, the content is driven as a function of the position of the eyes.

[0013] In a world-centered configuration, it may be desirable to have inputs such as accurate head pose measurement, accurate representation and / or measurement of real-world objects and geometries around the user, dynamic rendering with low latency in the augmented reality display as a function of head pose, and generally low-latency displays.

[0014] The aforementioned U.S. patent application presents systems and techniques for cooperating with the typical human visual configuration and addressing various challenges in virtual reality and augmented reality applications. The design of these virtual reality and / or AR systems presents a number of challenges, including the speed of the system in delivering virtual content, the quality of the virtual content, the user's interpupillary distance, the size and portability of the system, and other system and optical challenges.

[0015] The systems and techniques described herein are configured to work in conjunction with the typical human visual configuration and address these challenges. SUMMARY OF THE INVENTION

Means for Solving the Problem

[0016] Embodiments of the present invention are directed to devices, systems, and methods for facilitating virtual reality and / or augmented reality interactions for one or more users. In one aspect, a system for displaying virtual content is disclosed.

[0017] In one aspect, an augmented reality display system includes an image capture device for capturing one or more images, one or more images corresponding to a user's field of view of a head-mounted augmented reality device, and a processor communicatively coupled to the image capture device that extracts a set of map points from the set of images, identifies a set of low-density points and a set of high-density points from the extracted set of map points, and performs normalization on the set of map points. In one or more embodiments, the processor may generate low-density and high-density point descriptors for the set of low-density points and the set of high-density points, respectively.

[0018] In one or more embodiments, the low-density point descriptor and the high-density point descriptor are stored as map data. In one or more embodiments, the set of low-density points corresponds to distinct features of one or more images. In one or more embodiments, the distinct features are selected from the group consisting of corners, circles, triangles, and text.

[0019] In one or more embodiments, the set of high-density points corresponds to 3D points within the field of view. In one or more embodiments, the set of high-density points further includes color values. In one or more embodiments, the normalization includes scale normalization. In one or more embodiments, the normalization includes coordinate normalization with respect to a common origin. In one or more embodiments, the normalization utilizes machine learning. In one or more embodiments, the low-density and high-density point descriptors correspond to the low-density and high-density points of the set of low-density points and the set of high-density points, respectively. In one or more embodiments, the low-density and high-density point descriptors include information regarding at least one of scale, texture, orientation, and patch data.

[0020] In another aspect, a method of generating map data includes identifying a set of map points associated with one or more images, determining from the identified map points a set of low-density points and a set of high-density points, and normalizing each of the set of low-density points and the set of high-density points.

[0021] In one or more embodiments, the method further includes generating low-density and high-density point descriptors for each of the set of low-density points and the set of high-density points, and combining the low-density point descriptors and the high-density point descriptors and storing them as map data.

[0022] In one or more embodiments, the set of low-density points corresponds to distinct features. In one or more embodiments, the distinct features are selected from the group consisting of corners, circles, triangles, and text. In one or more embodiments, the set of high-density points corresponds to 3D points within the field of view. In one or more embodiments, the set of high-density points also includes color values.

[0023] In one or more embodiments, the normalization comprises scale normalization. In one or more embodiments, the normalization comprises coordinate normalization with respect to a common origin. In one or more embodiments, the normalization is implemented using machine learning. In one or more embodiments, the low-density and high-density point descriptors correspond to each of the low-density and high-density points of each of the set of low-density points and the set of high-density points. In one or more embodiments, each low-density and high-density point descriptor includes information regarding each of the low-density and high-density points selected from the group consisting of scale, orientation, patch data, and texture.

[0024] In another aspect, the augmented reality display system is an image capture device for capturing one or more images, where the one or more images correspond to the user's field of view and the images capture at least one gesture created by the user, an image capture device, communicatively coupled to the image capture device, identifying a set of points associated with the gesture, comparing the set of points with a database of predetermined gestures, recognizing the gesture at least in part based on the comparison, and determining user input at least in part based on the recognized gesture, a processor configured to do so.

[0025] In one or more embodiments, the processor generates a scoring value for the identified set of points based on the comparison. In one or more embodiments, the processor recognizes the gesture if the scoring value exceeds a threshold. In one or more embodiments, the augmented reality display system includes a database for storing a set of predetermined gestures. In one or more embodiments, the system further includes a networked memory for accessing the database of predetermined gestures.

[0026] In one or more embodiments, the gesture is a hand gesture. In one or more embodiments, the gesture is a finger gesture. In one or more embodiments, the gesture is an interaction between fingers. In one or more embodiments, the gesture is selected from the group consisting of interaction between fingers, pointing, tapping, and rubbing.

[0027] In one or more embodiments, the augmented reality display system further includes a spatial light modulator communicatively coupled to the processor, and the processor controls the spatial light modulator such that one or more virtual objects are displayed to the user based at least in part on the determined user input. In one or more embodiments, the one or more virtual objects include a virtual user interface.

[0028] In another aspect, a method of determining user input includes capturing an image of the user's field of view, the image including a gesture created by the user, analyzing the captured image to identify a set of points associated with the gesture, comparing the identified set of points with a set of points associated with a database of predetermined gestures, and determining user input based on the recognized gesture.

[0029] In one or more embodiments, the method further includes generating a scoring value for the identified set of points based on the comparison. In one or more embodiments, the method further includes recognizing the gesture if the scoring value exceeds a threshold. In one or more embodiments, the method further includes storing a predetermined gesture in the database. In one or more embodiments, the method further includes accessing a networked memory and accessing a database of predetermined gestures.

[0030] In one or more embodiments, the gesture is a hand gesture. In one or more embodiments, the gesture is a finger gesture. In one or more embodiments, the gesture is an interaction between fingers. In one or more embodiments, the gesture is selected from the group consisting of an interaction between fingers, an indication, a tap, and a friction.

[0031] In one or more embodiments, the method further includes displaying one or more virtual objects to the user based at least in part on the determined user input. In one or more embodiments, the one or more virtual objects comprise a virtual user interface.

[0032] In another aspect, the augmented reality display system includes an image capture device for capturing one or more images, and a processor communicatively coupled to the image capture device, analyzing the captured images and identifying one or more gestures made by a user, the identification of the one or more gestures including the step of utilizing a cascading mechanism having a plurality of stages.

[0033] In one or more embodiments, the cascading mechanism comprises a plurality of nodes, each node corresponding to a certain stage among the plurality of stages. In one or more embodiments, the cascading mechanism comprises a series of permissive analysis nodes. In one or more embodiments, a previous stage among the plurality of stages of the cascading mechanism is configured to consume less processing power compared to a later stage among the plurality of stages of the cascading mechanism.

[0034] In one or more embodiments, non-gestures are removed based on the analysis of the captured images captured at a previous stage of the cascading mechanism. In one or more embodiments, a later stage among the plurality of stages is configured to determine more complex gestures based at least in part on the captured images.

[0035] In one or more embodiments, the analysis of the captured images includes the step of determining whether the sharpness of the contours of various shapes within the captured images is sharp enough to constitute a gesture. In one or more embodiments, a later stage of the cascading mechanism is utilized to distinguish between different gestures. In one or more embodiments, the processor is further configured to generate a score based at least in part on the analysis. In one or more embodiments, the processor removes a candidate image from consideration if the generated score is lower than a minimum threshold. In one or more embodiments, the processor proceeds to a later stage of the cascading mechanism if the generated score is higher than the minimum threshold.

[0036] In another aspect, the method includes the steps of capturing one or more images corresponding to the user's field of view and analyzing the captured one or more images to identify one or more gestures created by the user, the analysis including utilizing a cascade mechanism having multiple stages.

[0037] In one or more embodiments, the cascade mechanism comprises a plurality of nodes, each node corresponding to a certain stage of the plurality of stages. In one or more embodiments, the cascade mechanism comprises a series of permissive analysis nodes. In one or more embodiments, a previous stage of the plurality of stages of the cascade mechanism is configured to consume less processing power compared to a later stage of the plurality of stages of the cascade mechanism.

[0038] In one or more embodiments, non-gestures are removed based on the analysis of the images captured at a previous stage of the cascade mechanism. In one or more embodiments, a later stage of the plurality of stages is configured to determine more complex gestures based at least in part on the captured images.

[0039] In one or more embodiments, the analysis of the captured images includes determining whether the sharpness of the contours of various shapes within the captured images is sharp enough to constitute a gesture. In one or more embodiments, a later stage of the cascade mechanism is utilized to distinguish between different gestures.

[0040] In one or more embodiments, the method further includes generating a score based at least in part on the analysis. In one or more embodiments, the method further includes removing candidate images from consideration if the generated score is lower than a minimum threshold. In one or more embodiments, the method further includes advancing to a later stage of the cascade mechanism if the generated score is higher than the minimum threshold.

[0041] In another aspect, the augmented reality system includes an image capture device for capturing a plurality of images of each user's field of view, and a processor communicatively coupled to the image capture device, the processor analyzing the plurality of images, generating a plurality of gesture candidates from the captured plurality of images, and generating analysis values corresponding to the plurality of gesture candidates, wherein the gesture is recognized based at least in part on the analysis values.

[0042] In one or more embodiments, the processor is further configured to sort the gesture candidates based at least in part on their respective analysis values. In one or more embodiments, the processor is further configured to exclude gesture candidates having analysis values lower than a minimum threshold. In one or more embodiments, the processor is further configured to advance a gesture candidate to the next stage of processing if the analysis value is higher than the minimum threshold.

[0043] In another aspect, the method includes capturing a plurality of images of each user's field of view, analyzing the plurality of images to generate a plurality of gesture candidates, and generating analysis values corresponding to the plurality of gesture candidates, wherein the gesture is recognized based at least in part on the analysis values.

[0044] In one or more embodiments, the method further includes sorting the gesture candidates based at least in part on their respective analysis values. In one or more embodiments, the method further includes excluding gesture candidates having analysis values lower than a minimum threshold. In one or more embodiments, the method further includes advancing a gesture candidate to the next stage of processing if the analysis value is higher than the minimum threshold.

[0045] In yet another aspect, the augmented reality display system includes an image capture device for capturing an image of the user's field of view, and a processor configured to generate a depth map corresponding to the captured image, analyze the depth map of the captured image, and identify a gesture.

[0046] In one or more embodiments, the processor is further configured to utilize a classification mechanism to identify a part of a hand corresponding to a point in the generated depth map. In one or more embodiments, the processor is further configured to skeletonize the depth map based on the identification of the part of the hand.

[0047] In one or more embodiments, the processor classifies an image as a gesture based on at least a partially skeletonized depth map. In one or more embodiments, the depth is generated at least in part by performing a depth partitioning process.

[0048] In one or more embodiments, the depth partitioning comprises a line search. In one or more embodiments, the processor performs cascade analysis on the depth map and classifies the image as a gesture. In one or more embodiments, the processor performs depth expansion on the depth map. In one or more embodiments, the processor performs surface normalization on the depth map.

[0049] In one or more embodiments, the processor performs orientation normalization on the depth map. In one or more embodiments, the processor performs background subtraction on the depth map. In one or more embodiments, the processor performs depth comparison on the depth map. In one or more embodiments, the processor classifies an image as a gesture based on at least a partially skeletonized depth map and previous information. In one or more embodiments, the classification mechanism is a decision forest.

[0050] In another aspect, a method of classifying a gesture includes capturing an image of a user's field of view, performing depth partitioning on the captured image to generate a depth map, and identifying the gesture based on at least the partially generated depth map.

[0051] In one or more embodiments, the method further includes using a classification mechanism to analyze the depth map and identify a part of a hand corresponding to a point in the depth map. In one or more embodiments, the method further includes skeletonsizing the depth map based on the identification of the part of the hand. In one or more embodiments, the method further includes classifying the image as a gesture based on the skeletonsized depth map.

[0052] In one or more embodiments, the depth partitioning has a line search. In one or more embodiments, the method further includes performing cascade analysis on the depth map and classifying the image as a gesture. In one or more embodiments, the method further includes performing depth expansion on the depth map.

[0053] In one or more embodiments, the method further includes performing surface normalization on the depth map. In one or more embodiments, the method further includes performing orientation normalization on the depth map. In one or more embodiments, the method further includes performing background subtraction on the depth map.

[0054] In one or more embodiments, the method further includes performing depth comparison on the depth map. In one or more embodiments, the method further includes classifying the image as a gesture based on the skeletonsized depth map and previous information. In one or more embodiments, the classification mechanism is a decision forest.

[0055] In another aspect, an augmented reality display system includes an image capture device for capturing a set of images of a user's field of view, the set of images capturing the user's movement in relation to a predetermined physical object, and a processor communicatively coupled to the image capture device, the processor analyzing the movement in relation to the predetermined physical object and determining user input based at least in part on the analyzed movement.

[0056] In one or more embodiments, the processor recognizes a predetermined physical object. In one or more embodiments, the predetermined physical object is recognized based at least in part on an image comparison between a captured image of the predetermined physical object and a database of the predetermined physical object. In one or more embodiments, the analyzed movement of the user related to the predetermined physical object is used to generate a pattern. In one or more embodiments, the generated pattern is compared to a database of predetermined patterns. In one or more embodiments, the processor generates a scoring value for the recognized pattern based on the comparison.

[0057] In one or more embodiments, the processor determines user input if the scoring value exceeds a threshold. In one or more embodiments, the image capture device visually tracks movement related to the predetermined physical object and generates a video recording. In one or more embodiments, the video recording is analyzed to determine user input.

[0058] In one or more embodiments, the augmented reality display system further comprises a networked memory for accessing a database of predetermined patterns. In one or more embodiments, the predetermined physical object is selected from the group consisting of an existing structure within the field of view, an actively marked totem, a passively marked totem, an object integrated with a camera / sensor, and a totem controller object.

[0059] In one or more embodiments, the movement related to the predetermined physical object is selected from the group consisting of the position, orientation, and movement of the predetermined physical object relative to a reference frame. In one or more embodiments, the predetermined physical object comprises a first hand of the user, and the movement related to the first hand comprises an operation of the first hand using a second hand of the user.

[0060] In one or more embodiments, a given physical object has a soft-hardness surface, and movement associated with the given physical object comprises pressing of the soft-hardness surface by a user. In one or more embodiments, a processor renders a virtual interface element associated with a given physical object for a user, and the virtual interface element is visually perceived by the user through a display device.

[0061] In one or more embodiments, when the virtual interface element is visually perceived by the user, the virtual interface element is displayed in relation to the given physical object such that the virtual interface element is modified based at least in part on a modification associated with the given physical object. In one or more embodiments, the given physical object comprises an electronic input device, and user input is determined based on a recognized movement of the given physical object and input from the electronic input device.

[0062] In one or more embodiments, the processor is configured to control a display device coupled to a user's eye such that virtual content displayed to the user is modified based at least in part on determined user input.

[0063] In another aspect, a method of determining user input includes capturing an image of the field of view of one or more users, the image comprising at least a given physical object, analyzing the image to detect movement of the user associated with the given physical object, and determining user input based at least in part on an analysis of the movement associated with the given physical object.

[0064] In one or more embodiments, the method further includes the step of recognizing one or more predetermined physical objects. In one or more embodiments, the one or more predetermined physical objects are recognized based at least in part on an image comparison between a captured image of the one or more predetermined physical objects and a database of the one or more predetermined physical objects. In one or more embodiments, the movement of a user associated with the one or more predetermined physical objects is used to generate a pattern.

[0065] In one or more embodiments, the method further includes the step of comparing the generated pattern with a database of predetermined patterns. In one or more embodiments, the method further includes the step of generating a scoring value for the recognized pattern based on the comparison. In one or more embodiments, the method further includes the step of determining user input if the scoring value exceeds a threshold.

[0066] In one or more embodiments, the method further includes the step of visually tracking movement associated with the one or more predetermined physical objects and generating a video recording. In one or more embodiments, the video recording is analyzed to determine user input. In one or more embodiments, the method further includes the step of accessing a database of predetermined patterns through a networked memory. In one or more embodiments, the one or more predetermined physical objects are selected from the group consisting of an existing structure within the field of view, an actively marked totem, a passively marked totem, an object integrated into a camera / sensor, and a totem controller object.

[0067] In one or more embodiments, the movement associated with the one or more predetermined physical objects is selected from the group consisting of the position, orientation, and movement of the one or more predetermined physical objects relative to a reference frame. In one or more embodiments, the one or more predetermined physical objects comprise a first hand of a user, and the movement associated with the first hand comprises an operation of the first hand using a second hand of the user.

[0068] In one or more embodiments, a given physical object has a soft hardness surface, and movement related to the given physical object comprises depressing the soft hardness surface by a user. In one or more embodiments, the method further includes rendering a virtual interface element associated with the given physical object for the user.

[0069] In one or more embodiments, the virtual interface element is displayed in relation to the given physical object such that when viewed by the user, the virtual interface element is modified based at least in part on a modification related to the given physical object. In one or more embodiments, the given physical object comprises an electronic input device, and the user input is determined based on the recognized movement of the given physical object and input from the electronic input device. In one or more embodiments, the method further includes modifying at least one characteristic of the virtual content displayed to the user based at least in part on the determined user input.

[0070] In another aspect, an augmented reality display system comprises a display physically coupled to a user's eye and displaying a set of virtual content related to one or more physical objects, and a processor communicatively coupled to the display and an image capture device, identifying a virtual user interface to be displayed to the user via the display based on user input, obtaining a location of the user in relation to the world, determining a set of coordinate points at which the virtual user interface is to be displayed, and controlling the display such that the identified virtual user interface is displayed to the user.

[0071] In one or more embodiments, the user input is determined based at least in part on a recognized gesture. In one or more embodiments, the user input is determined based at least in part on a voice command. In one or more embodiments, the user input is determined based at least in part on an interaction with a given physical object.

[0072] In one or more embodiments, the augmented reality display system further comprises a library of user interfaces, and the identified user interface is read from the library of user interfaces. In one or more embodiments, the identified virtual user interface is associated with a reference frame. In one or more embodiments, the reference frame is a body-centered reference frame.

[0073] In one or more embodiments, the reference frame is a head-centered reference frame. In one or more embodiments, the reference frame is a hand-centered reference frame. In one or more embodiments, the reference frame is a world-centered reference frame. In one or more embodiments, the processor performs a transformation between the reference frame associated with the identified virtual user interface and the obtained location of the user relative to the world, and determines a set of coordinate points of the virtual user interface.

[0074] In one or more embodiments, the location of the user within the world is determined based on the GPS location of the user. In one or more embodiments, the location of the user within the world is determined based on a set of map points associated with the user. In one or more embodiments, the virtual user interface appears to be stationary as the user moves.

[0075] In one or more embodiments, the virtual user interface moves in relation to the movement of the user. In one or more embodiments, the processor determines the reference frame associated with the identified virtual user interface, determines the location of the reference frame relative to the world reference frame, sets the determined location as the origin, and determines a set of coordinate points relative to the origin.

[0076] In another aspect, a method of generating a virtual user interface includes identifying, based on user input, a virtual user interface to be presented to the user; obtaining a location associated with the user; determining, based at least in part on the obtained location, a set of coordinate points at which the identified virtual user interface is to be presented; and presenting the virtual user interface to the user at the determined coordinate points.

[0077] In one or more embodiments, the user input is determined based at least in part on a recognized gesture. In one or more embodiments, the user input is determined based at least in part on a voice command. In one or more embodiments, the user input is determined based at least in part on an interaction with a predetermined physical object.

[0078] In one or more embodiments, the method further includes reading the identified virtual user interface from a library of user interfaces. In one or more embodiments, the identified virtual user interface is associated with a reference frame. In one or more embodiments, the reference frame is a body-centered reference frame.

[0079] In one or more embodiments, the reference frame is a head-centered reference frame. In one or more embodiments, the reference frame is a hand-centered reference frame. In one or more embodiments, the reference frame is a world-centered reference frame.

[0080] In one or more embodiments, the method further includes performing a transformation between the reference frame associated with the identified virtual user interface and the obtained location of the user relative to the world to determine the set of coordinate points of the virtual user interface. In one or more embodiments, the location of the user within the world is determined based on the user's GPS location.

[0081] In one or more embodiments, the location of a user worldwide is determined based on a set of map points associated with the user. In one or more embodiments, the virtual user interface appears to be stationary as the user moves. In one or more embodiments, the virtual user interface moves in relation to the movement of the user.

[0082] In one or more embodiments, the method further includes determining a reference frame associated with the identified virtual user interface, determining the location of the reference frame relative to the world reference frame, setting the determined location as the origin, and determining a set of coordinate points related to the origin.

[0083] In one or more embodiments, the method further includes reading a set of map points from a networked memory. In one or more embodiments, the user input comprises a location within the space where the virtual user interface is to be displayed. In one or more embodiments, the location within the space is associated with a physical entity at the location of the user. In one or more embodiments, the user input includes a gesture including a throwing input indicating a wall. In one or more embodiments, the user input includes a gesture indicating the end of an instruction for generating the virtual user interface.

[0084] In another aspect, a method for generating a virtual user interface includes detecting an operation of a predetermined physical object, recognizing a command for creating the virtual user interface based on the detected operation, determining a set of map points associated with the location of the predetermined physical object from a virtual world model, and rendering the virtual user interface in real time to the determined map points associated with the location of the totem such that the virtual user interface appears to be stationary at the location of the predetermined physical object when viewed by the user.

[0085] In one or more embodiments, the operation of a predetermined physical object includes a gesture of spreading the state of being pinched by the user's hand on the surface of the predetermined physical object. In one or more embodiments, when viewed by the user, the virtual user interface appears to cover a part of the surface of the predetermined physical object, and the part corresponds to the location of the user's hand during the formation of the gesture of spreading the pinched state. In one or more embodiments, the predetermined physical object is the user's hand.

[0086] In one or more embodiments, the operation of a predetermined physical object includes an action selected from the group consisting of the user spreading the hand, the user spreading the palm to show, and the user raising the hand. In one or more embodiments, when viewed by the user, the virtual user interface appears to cover a part of the surface of the hand.

[0087] In one or more embodiments, the virtual user interface comprises a plurality of first-level menu items selectable by the fingers or thumb of the hand. In one or more embodiments, the method further comprises detecting a further operation of the hand, recognizing a command for creating a second virtual user interface based on the detected further operation, and rendering the second virtual user interface in real time at a determined map point associated with the location of the predetermined physical object such that the virtual user interface appears to be stationary at the location of the predetermined physical object when viewed by the user.

[0088] In one or more embodiments, the method further includes a further operation of the hand, including spreading the fingers of the hand apart. In one or more embodiments, the second virtual user interface comprises a plurality of second-level menu items selectable by the fingers or thumb of the hand, and the second level is lower than the first level.

[0089] In one or more embodiments, further manipulation of the totem includes using a finger from a second hand of the user's hand to perform a motion of drawing a circle on the palm. In one or more embodiments, the second virtual user interface comprises a plurality of menu items arranged in an arc, and the menu items are scrollable and selectable by a finger of the second hand.

[0090] In another aspect, an augmented reality display system includes an image capture device for capturing one or more images of a user's field of view, wherein at least one image captures an operation of a predetermined physical object, a display device for displaying one or more virtual objects to the user, and a database having a passable world model, the passable world model comprising a set of map points associated with physical objects in the world, a processor communicatively coupled to the image capture device, recognizing commands for creating a virtual user interface based on the one or more images, determining a map point corresponding to a predetermined physical object based at least in part on the passable world model, and controlling the display in a manner such that the virtual user interface is generated at the determined map point corresponding to the predetermined physical object so as to appear stationary at the location of the predetermined physical object.

[0091] In one or more embodiments, the operation of the predetermined physical object includes a gesture of spreading a pinched state by the user's hand on the surface of the predetermined physical object. In one or more embodiments, the virtual user interface appears to cover a portion of the surface of the predetermined physical object when viewed by the user, and the portion corresponds to the location of the user's hand during the formation of the gesture of spreading the pinched state. In one or more embodiments, the predetermined physical object is the user's hand.

[0092] In one or more embodiments, the operation of a predetermined physical object includes an action selected from the group consisting of the user spreading their hand, the user showing the palm of their hand, and the user raising their hand. In one or more embodiments, the virtual user interface appears to cover a portion of the surface of the hand when viewed by the user.

[0093] In one or more embodiments, the virtual user interface comprises a plurality of first-level menu items selectable by the fingers or thumb of the hand. In one or more embodiments, the predetermined physical object is a hand, and the processor detects further operations of the hand, recognizes commands for creating a second virtual user interface, and controls the display so that the second virtual user interface is displayed at the determined map point.

[0094] In one or more embodiments, further operations of the hand include spreading the fingers of the hand apart. In one or more embodiments, the second virtual user interface comprises a plurality of second-level menu items selectable by the fingers or thumb of the hand, and the second level is lower than the first level. In one or more embodiments, further operations of the totem include using the fingers from a second hand of the user's hand to perform a motion of drawing a circle on the palm.

[0095] In one or more embodiments, the second virtual user interface comprises a plurality of menu items arranged in an arc, and the menu items are scrollable and selectable by the fingers of the second hand.

[0096] In another aspect, a method of updating a virtual world includes receiving a first input from a first device of a first user, the first input corresponding to the physical environment of the first user; updating a virtual world model based on the received first input, the virtual world model corresponding to the physical environment of the first user; and transmitting first updated information corresponding to a first portion of the virtual world model to a second user, the first updated information being configured to indicate whether any portion of the first updated information needs to be displayed to the second user.

[0097] In one or more embodiments, the virtual world model resides on a networked memory. In one or more embodiments, the first user and the second user are located in different respective locations. In one or more embodiments, the first device of the first user is selected from the group consisting of a FOV camera, other cameras, sensors, a first device for eye tracking, and a first device for audio.

[0098] In one or more embodiments, the method further includes transmitting first updated information corresponding to a first portion of the virtual world model to the first user, the first updated information being configured to indicate whether any portion of the first updated information needs to be displayed to the first user. In one or more embodiments, the method further includes transmitting first updated information corresponding to a first portion of the virtual world model to a plurality of other users, the first updated information being configured to indicate whether any portion of the first updated information needs to be displayed to each of the plurality of other users.

[0099] In one or more embodiments, the method further comprises receiving a plurality of inputs from respective first devices of a plurality of other users, the plurality of inputs corresponding to the physical environment of a first user, updating a virtual world model based on the received plurality of inputs, and transmitting additional updated information corresponding to respective additional portions of the virtual world model to a second user, the additional updated information being configured to indicate whether any portion of the additional updated information needs to be displayed to the second user.

[0100] In one or more embodiments, the method further comprises transmitting additional updated information corresponding to a portion of the virtual world model to a first user, the additional updated information being configured to indicate whether any portion of the additional updated information needs to be displayed to the first user.

[0101] In one or more embodiments, the method further comprises transmitting additional updated information corresponding to respective additional portions of the virtual world model to a plurality of other users, the additional updated information being configured to indicate whether any portion of the additional updated information needs to be displayed to each of the plurality of other users.

[0102] In one or more embodiments, the method further comprises receiving a second input from a second device of a second user, the second input corresponding to the physical environment of the second user, updating the virtual world model based on the received second input, the virtual world model corresponding to the physical environment of the first user, and transmitting second updated information corresponding to a second portion of the virtual world model to the first user, the second updated information being configured to indicate whether any portion of the second updated information needs to be displayed to the first user. In one or more embodiments, the second updated information corresponds to the movement of an avatar of the second user within the second portion of the virtual world model.

[0103] In another aspect, the method includes projecting a light pattern spatially, detecting, using a camera, first and second portions of the light pattern and generating first and second data points corresponding to the first and second portions of the pattern, and performing triangulation analysis to determine the locations of the first and second portions of the light pattern based at least in part on the first and second data points.

[0104] In one or more embodiments, the method further includes repeating the steps of projecting, detecting, and triangulating multiple portions of the pattern to obtain additional texture data for the space. In one or more embodiments, the step of projecting light spatially includes projecting light spatially using a fiber-based projector.

[0105] In one or more embodiments, the method further includes modifying the light using an optical element to form a beam of light rays before the light is projected spatially. In one or more embodiments, the light pattern corresponds to structured light and the light pattern is dynamic. In one or more embodiments, the light pattern corresponds to patterned light and the light pattern includes multiple points.

[0106] In one or more embodiments, the light pattern corresponds to textured light and the light pattern is irregular. In one or more embodiments, the first and second portions of the light pattern are respective pixels of the light pattern.

[0107] In another aspect, an augmented reality display system includes an optical projector for projecting an optical pattern into a space, a camera coupled to a frame of the augmented reality display system and configured to capture an image of the user's field of view, the camera being configured to capture first and second portions corresponding to the projected optical pattern, and a processor communicatively coupled to the camera and configured to detect first and second data points corresponding to the first and second portions of the pattern, perform triangulation analysis, and determine locations of the first and second portions of the optical pattern based at least in part on the first and second data points.

[0108] In one or more embodiments, the processor repeats the steps of projecting, detecting, and triangulating using multiple portions of the pattern to obtain additional texture data for the space. In one or more embodiments, the optical projector comprises a fiber-based projector. In one or more embodiments, the light to be projected is modified using an optical element to form a bundle of light rays before the light is projected onto the space.

[0109] In one or more embodiments, the optical pattern corresponds to structured light and the optical pattern is dynamic. In one or more embodiments, the optical pattern corresponds to patterned light and the optical pattern includes multiple points. In one or more embodiments, the optical pattern corresponds to textured light and the optical pattern is irregular. In one or more embodiments, the first and second portions of the optical pattern are respective pixels of the optical pattern.

[0110] In another aspect, an augmented reality system includes a predetermined physical object having a surface configured to be operated such that an operation is detectable as user input by the augmented reality system, the predetermined physical object comprising a material configured to provide a tactile perception to the user when the user interacts with the predetermined physical object via touch.

[0111] In one or more embodiments, a given physical object is configured to simulate a user input device. In one or more embodiments, the user input device is selected from the group consisting of a computer keyboard, a computer mouse, a computer trackpad, and a handheld controller.

[0112] In one or more embodiments, the system further includes a display device for displaying one or more virtual objects to the user, and a processor for controlling the display device such that a virtual user interface is displayed on the surface of the given physical object.

[0113] In one or more embodiments, the given physical object has a spherical shape. In one or more embodiments, the system is configured to display a virtual user interface such that the virtual user interface appears to emerge from the given physical object. In one or more embodiments, the given physical object has a textured outer surface.

[0114] In one or more embodiments, the given physical object includes a user input element. In one or more embodiments, the user input element is selected from the group consisting of keys, buttons, and scroll wheels. In one or more embodiments, the user input element is not operably coupled to an electronic device. In one or more embodiments, the system further includes a camera configured to detect the operation of the user of the totem.

[0115] In one or more embodiments, the given physical object includes a depression, a cavity, or a protrusion. In one or more embodiments, the operation is selected from the group of finger movements consisting of swipes, swings, rotations, scrolls, taps, double taps, short taps, and long taps.

[0116] In one or more embodiments, the operation is selected from a group of finger movement characteristics consisting of several interactions, interaction types, and interaction durations. In one or more embodiments, the operation is selected from a group of finger characteristics consisting of distance, direction, speed, and acceleration.

[0117] In yet another aspect, a method of generating a virtual room includes identifying a virtual room to be displayed to a user, obtaining a first set of map points associated with the virtual room, the first set of map points corresponding to a first physical room at the location of the user, tethering the virtual room to the first set of map points, and displaying the virtual room to the user such that the virtual room appears to be stationary in the set of the first set of map points when viewed by the user.

[0118] In one or more embodiments, the method further includes identifying the virtual room based on user input. In one or more embodiments, the user input is selected from the group consisting of gestures, visual data, auditory data, sensory data, direct commands, voice commands, eye tracking, and selection of physical buttons.

[0119] In one or more embodiments, the virtual room is provided by the user. In one or more embodiments, the method further includes reading the first set of map points from a networked memory. In one or more embodiments, the method further includes obtaining a second set of map points associated with the virtual room, the second set of map points corresponding to a second physical room at the location of the user after the user has moved from the first physical room, tethering the virtual room to the second set of map points, and displaying the virtual room to the user such that the virtual room appears to be stationary in the second set of map points when viewed by the user.

[0120] In one or more embodiments, the virtual room also includes first and second virtual objects, and the method further includes, when the virtual room is tethered to a second set of map points, maintaining a relative spatial position between the first virtual object corresponding to the first set of map points and the second virtual object. In one or more embodiments, the method further includes displaying the virtual room to the user such that, when viewed by the user, the virtual room appears stationary relative to a portion of the first physical room. In one or more embodiments, the virtual room also includes selectable virtual decorations.

[0121] In another aspect, a method of creating a retail experience includes recognizing a user's location within a retail store, reading data corresponding to the retail store, generating virtual content related to the retail store based on the read data, creating a virtual user interface within the user's field of view, and displaying the virtual content on the virtual user interface while the user engages in retail activity within the retail store.

[0122] In one or more embodiments, the read data comprises a set of map points corresponding to the retail store, and the virtual user interface appears stationary within the set of map points when viewed by the user. In one or more embodiments, the user's location is recognized using radio frequency identification transponders and communication. In one or more embodiments, the virtual content is selected from the group consisting of virtual characters, virtual coupons, games based on the user location, lists of promotional items, nutritional information, metadata related to items, appearances of famous people, cross-selling advertisements, information from people known to the user, and e-books.

[0123] In one or more embodiments, the method further includes reading user data corresponding to the user, and generating virtual content based on both the read data and the read user data. In one or more embodiments, the virtual content is selected from the group consisting of a virtual grocery list, a virtual coupon book, a virtual recipe book, a list of various materials in the user's home, and a virtual recipe builder. In one or more embodiments, the method further includes receiving user input, generating additional virtual content based on the user input, and displaying the additional virtual content on a virtual user interface while the user is engaged in retail activities within a retail store.

[0124] In one or more embodiments, the user input is selected from the group consisting of gestures, visual data, auditory data, sensory data, direct commands, voice commands, eye tracking, and selection of physical buttons. In one or more embodiments, the virtual content is selected from the group consisting of total operating costs, a smart virtual grocery list, an indicator of items proximate to the user's location, and a virtual payment system. In one or more embodiments, the method further includes transmitting the generated virtual content to a user device for display.

[0125] In another aspect, the method includes reading patient data related to a patient's medical history, generating virtual content based at least in part on the read patient data, reading a set of first map points corresponding to a first location of a first user, creating a first virtual user interface within the field of view of the first user based at least in part on the read set of map points, and displaying the virtual content on the first virtual user interface such that the first virtual user interface appears to be fixed at the set of first map points when the first virtual user interface is viewed by the first user.

[0126] In one or more embodiments, the virtual content is selected from the group consisting of a three-dimensional image of a surgical target, patient identification information, medical images, the patient's vital sign information, and a medical record. In one or more embodiments, the patient data is read from a networked memory.

[0127] In one or more embodiments, the method further includes generating a second virtual user interface configured to facilitate communication between a first user and a second user, where the second user is located in a second location different from the first location of the first user. In one or more embodiments, the second virtual user interface is a visual representation of the second user. In one or more embodiments, the second user is selected from the group consisting of a senior specialist surgeon, a patient, a person related to the patient, and a medical student. In one or more embodiments, the method further includes displaying the virtual content on the first virtual user interface for the second user such that when the first virtual user interface is viewed by the second user, it appears to be fixed at a set of map points.

[0128] In one or more embodiments, the virtual content is displayed to the first user during a surgical procedure, and the method further includes receiving user input, generating additional virtual content based on the user input, and displaying the additional virtual content on the first virtual user interface while the first user is performing the surgical procedure.

[0129] In one or more embodiments, the user input is selected from the group consisting of gestures, visual data, auditory data, sensory data, direct commands, voice commands, eye tracking, and selection of physical buttons. In one or more embodiments, the user input includes an image of the first user's field of view, and the method further includes displaying the additional virtual content on the first virtual user interface for the second user such that when the first virtual user interface is viewed by the second user, it appears to be fixed at a set of map points.

[0130] In one or more embodiments, a method for promoting medical rehabilitation includes receiving user input related to medical rehabilitation, sending a request related to medical rehabilitation to a cloud server based on the received user input, receiving data related to medical rehabilitation from a knowledge base connected to the cloud server based on the request, and displaying virtual content related to medical rehabilitation to a first user based on the received data.

[0131] In one or more embodiments, the user input is selected from the group consisting of visual, auditory, and sensory inputs. In one or more embodiments, the method further includes determining whether the user input is valid before sending the request to the cloud server. In one or more embodiments, the data is received from a plurality of knowledge bases connected to the cloud server.

[0132] In one or more embodiments, the request instructs the cloud server to read data from the knowledge base. In one or more embodiments, the virtual content includes a relaxing environment. In one or more embodiments, the method further includes receiving additional data from a second user and displaying additional virtual content to the first user based on the received additional data.

[0133] In one or more embodiments, the first and second users are located in different physical locations. In one or more embodiments, the additional virtual content is a visual representation of the second user. In one or more embodiments, the method further includes receiving first user data from the first user, receiving second user data from the second user, modifying the virtual content based on the received first and second user data, and displaying the modified virtual content to the first and second users.

[0134] In one or more embodiments, the first user engages in a physical activity for medical rehabilitation, and the virtual content relates to the physical activity. In one or more embodiments, the virtual content is selected from the group consisting of information about the physical activity, the first user's performance statistics, a virtual experience corresponding to the physical activity, and a virtual avatar. In one or more embodiments, the data is read from a networked memory.

[0135] In another aspect, a method of improving the performance of a task includes the steps of reading task data related to the task to be performed by a user, generating virtual content based on the task data, creating a virtual user interface within the user's field of view, reading a set of map points corresponding to the user's location, and displaying the virtual content on the virtual user interface while the user is performing the task such that the virtual user interface appears to be fixed at the set of map points when viewed by the user.

[0136] In one or more embodiments, the virtual content is a game having a virtual mapped pattern, and the game is configured to improve the performance of a task by the user. In one or more embodiments, the game awards points for following the virtual mapped pattern. In one or more embodiments, the game doubles the points for reaching a location within the virtual mapped pattern within a predetermined time.

[0137] In one or more embodiments, the game deducts points for deviating from the virtual mapped pattern. In one or more embodiments, the game deducts points for moving adjacent to a physical object. In one or more embodiments, the task is selected from the group consisting of operating a gardening machine, reading inventory items, stocking shelves in a retail store, and sorting mail. In one or more embodiments, the task data is read from a networked memory.

[0138] In one or more embodiments, the method further includes receiving user input, generating additional virtual content based on the user input, and displaying the additional virtual content on a virtual user interface while the user is performing a task. In one or more embodiments, the user input includes user actions related to the performance of the task.

[0139] In yet another aspect, an augmented reality display system includes a display device configurable to display one or more virtual images to a user's eyes, and a processor communicatively coupled to the display device to provide the one or more virtual images to the display device, the processor having a prediction mechanism for predictively correcting a timing mismatch related to the display of the virtual images.

[0140] In one or more embodiments, the timing mismatch is related to one or more sensor measurements made by one or more sensors, the one or more sensors being communicatively coupled to the processor. In one or more embodiments, the timing mismatch is related to a processing delay when processing incoming data.

[0141] In one or more embodiments, the prediction mechanism utilizes a filter to correct the effect of the timing mismatch. In one or more embodiments, the filter takes into account the relative speed of one or more sensor measurements, the sensor measurements being made by one or more sensors communicatively coupled to the processor. In one or more embodiments, the prediction mechanism utilizes a Kalman predictor.

[0142] In one or more embodiments, the Kalman predictor is utilized during a display processing stage. In one or more embodiments, the processor makes compensation changes to the data associated with one or more virtual images to compensate for the timing mismatch. In one or more embodiments, the compensation changes include shifting the data associated with the virtual images.

[0143] In one or more embodiments, the compensation modification includes the step of smoothing one or more visual artifacts associated with the virtual image. In one or more embodiments, the compensation modification includes the step of correcting the negative impact of sensor measurements of one or more sensors, and the one or more sensors are communicatively coupled to the processor.

[0144] In another aspect, a method for correcting latency in an augmented reality display system includes determining one or more timing mismatches in at least one stage of displaying a virtual image to a user, using a prediction mechanism to predictively correct the timing mismatch, and compensating for at least one characteristic associated with the data of the virtual image, at least in part, based on the prediction mechanism.

[0145] In one or more embodiments, the timing mismatch is related to one or more sensor measurements performed by one or more sensors, and the one or more sensors are communicatively coupled to the processor. In one or more embodiments, the timing mismatch is related to a processing delay when processing incoming data.

[0146] In one or more embodiments, the prediction mechanism uses a filter to correct the impact of the timing mismatch. In one or more embodiments, the filter takes into account the relative speed of one or more sensor measurements, and the sensor measurements are performed by one or more sensors communicatively coupled to the processor.

[0147] In one or more embodiments, the prediction mechanism uses a Kalman predictor. In one or more embodiments, the Kalman predictor is utilized during the display processing stage. In one or more embodiments, the compensation includes the step of shifting the data associated with the virtual image. In one or more embodiments, the compensation includes the step of smoothing one or more visual artifacts associated with the virtual image. In one or more embodiments, the compensation includes the step of correcting the negative impact of sensor measurements of one or more sensors, and the one or more sensors are communicatively coupled to the processor.

[0148] In another aspect, a method of calibrating an augmented reality system includes displaying a virtual image to a user, the virtual image being configured to be displayed at a known focal length and the virtual image comprising pixel points; determining a location at which the pixel points are displayed to the user, the location of the pixel being calculated based at least in part on the location of the pupil of the user's eye; and aligning the pixel points of the virtual image with known points in space.

[0149] In one or more embodiments, the steps are repeated for a plurality of pixel points. In one or more embodiments, the location at which the pixel points are displayed to the user is calculated based at least in part on the location of known points in space. In one or more embodiments, the location of the pupil, the location of the pixel points, and the location of the known points in space are collinear.

[0150] In one or more embodiments, the location at which the pixel points are displayed to the user is modified based on user input. In one or more embodiments, the method further includes creating a game interface such that a plurality of pixel points are presented to the user. In one or more embodiments, the game interface includes firing a laser through eye movement of the user.

[0151] In one or more embodiments, the location of the known points is determined based at least in part on data received from one or more world cameras. In one or more embodiments, the steps are repeated for the other eye of the user. In one or more embodiments, the location of the pixel points is determined based at least in part on a function of the location of the pupil and the location of the known points. In one or more embodiments, the function comprises a quadratic function.

[0152] In another aspect, an augmented reality display system includes a display device for presenting virtual images to a user, the virtual images being configured to be presented at a known focal length and having pixel points, and a processor communicatively coupled to the display device, the processor determining locations at which the pixel points are presented to the user, calculating locations of the pixel points based at least in part on a location of a pupil of the user's eye, and configured to align pixel points of the virtual image to known points in space.

[0153] In one or more embodiments, the step is repeated for a plurality of pixel points. In one or more embodiments, the locations at which the pixel points are presented to the user are calculated based at least in part on locations of known points in space. In one or more embodiments, the location of the pupil, the location of the pixel points, and the location of the known points in space are collinear. In one or more embodiments, the locations at which the pixel points are presented to the user are modified based on user input.

[0154] In one or more embodiments, the system further includes a game interface configured to present a plurality of pixel points to the user. In one or more embodiments, the game interface includes the step of emitting a laser through the movement of the user's eyes. In one or more embodiments, the location of the known points is determined based at least in part on data received from one or more world cameras.

[0155] In one or more embodiments, the step is repeated for the other eye of the user. In one or more embodiments, the location of the pixel points is determined based at least in part on a function of the location of the pupil and the location of the known points. In one or more embodiments, the function includes a quadratic function.

[0156] In another aspect, a method of displaying a virtual interface includes identifying, based on user input, a user interface to be presented to a user, presenting an avatar user interface in relation to at least one physical object, the avatar user interface presenting at least one virtual user interface element, and selecting at least one virtual user interface element based at least in part on an interaction between the user and the avatar user interface.

[0157] In one or more embodiments, the user input includes a gesture. In one or more embodiments, the method further includes determining a location of the gesture in relation to the user's augmented reality display system, and the virtual user interface is presented at the determined location. In one or more embodiments, the avatar is a preselected avatar. In one or more embodiments, the avatar resembles the user.

[0158] In one or more embodiments, the avatar is presented as standing on the physical object. In one or more embodiments, at least one virtual element comprises an application. In one or more embodiments, the method further includes selecting at least one virtual element based at least in part on another user input, the at least one virtual element comprising a user mediator, and presenting another avatar representing the user mediator in relation to the avatar virtual user interface and the physical object, the interaction between the user and the user mediator being animated through a virtual interaction between the avatar virtual user interface and the other avatar representing the user mediator.

[0159] In one or more embodiments, the interaction includes transmitting data between the user's augmented reality system and a computing system corresponding to the user mediator. In one or more embodiments, the user input includes a voice command.

[0160] In another aspect, a method of displaying a virtual interface includes identifying, based on user input, a user interface to be presented to the user, and presenting a floating user interface having one or more selectable virtual interface elements associated with at least one physical object, the floating user interface appearing to be pushed out from where the user input was received.

[0161] In one or more embodiments, the user input includes a gesture of the user's finger. In one or more embodiments, the gesture of the user's finger includes touching a finger of one hand of the user with another finger of the user's other hand. In one or more embodiments, the method further includes determining where a finger of one hand of the user touches another finger of the user's other hand, and the floating virtual user interface originates from the determined location.

[0162] In one or more embodiments, the floating user interface comprises a set of three-dimensional blocks, and the three-dimensional blocks of the set of three-dimensional blocks represent at least one selectable virtual user interface element. In one or more embodiments, the floating virtual user interface is created based at least in part on a world reference frame.

[0163] In one or more embodiments, at least one selectable virtual user interface element exists as a stack of three-dimensional blocks, and the stack of three-dimensional blocks is rotated based at least in part on another user input. In one or more embodiments, the method further includes selecting a virtual user interface element of at least one selectable virtual interface element based at least in part on another user input received from a user, the virtual interface element being associated with at least one other sub-virtual interface element; and displaying at least one other sub-virtual interface element under the selected virtual interface element.

[0164] In one or more embodiments, the method further includes selecting a virtual user interface element of at least one selectable virtual interface element based at least in part on another user input received from a user; and displaying, within a virtual box, content associated with the selected virtual interface element, the virtual box comprising content that is displayed within the user's field of view.

[0165] In one or more embodiments, the method further includes identifying a second user input indicating that the virtual box is to be closed; animating the virtual box in a manner similar to crumpling a piece of paper at least in part based on the second user input; and ending the content associated with the selected virtual interface element. In one or more embodiments, the second user input includes a gesture that is similar to crumpling a single piece of paper.

[0166] In yet another aspect, a method of displaying a virtual user interface includes identifying, based on user input, a user interface to be presented to the user, and presenting a floating user interface having one or more selectable virtual interface elements associated with at least one physical object, the floating user interface appearing to be placed on the at least one physical object, rotation of the at least one physical object about a longitudinal axis of the at least one physical object resulting in additional selectable virtual interface elements being presented, the additional selectable virtual interface elements appearing to be stationary on another side of the at least one physical object.

[0167] In one or more embodiments, the at least one physical object comprises a user's arm. In one or more embodiments, the user input includes a gesture. In one or more embodiments, the gesture includes a movement that forms a cup of the user's hand on the user's arm such that selectable virtual interface elements appear to be presented.

[0168] In one or more embodiments, the method further includes determining a location of a movement that forms a cup of the user's hand on the user's arm, the floating user interface originating from the determined location. In one or more embodiments, the method further includes ending the rendering of the floating virtual user interface, at least in part, based on another user input.

[0169] In one or more embodiments, other user inputs include gestures, which include the movement of dragging the user's cupped hand across a physical object. In one or more embodiments, the floating virtual user interface is created at least in part based on a hand-centered reference frame. In one or more embodiments, the method further includes selecting a virtual user interface element of at least one selectable virtual interface element based at least in part on another user input received from the user, and displaying content associated with the selected virtual interface element, the content being displayed in relation to the floating virtual interface.

[0170] In one or more embodiments, the displayed content corresponds to a hand-centered reference frame. In one or more embodiments, the method further includes moving the displayed content from the hand-centered reference frame to a world-centered reference frame such that the displayed content remains stationary when the floating virtual user interface is moving, based at least in part on input received from the user.

[0171] In another aspect, a method of creating a virtual user interface includes identifying a user interface to be presented to a user based on user input, and presenting a virtual user interface that originates from a location where at least one physical object has been touched, in relation to the at least one physical object, the virtual user interface appearing to protrude from the location where the at least one physical object has been touched and comprising at least one selectable virtual user interface element that appears to be attached to the virtual user interface.

[0172] In one or more embodiments, at least one physical object comprises a horizontal surface. In one or more embodiments, user input includes gestures. In one or more embodiments, a gesture includes an extended finger touching at least one physical object for a predetermined period of time. In one or more embodiments, the method further includes determining a location where the user's finger touches at least one physical object, and the virtual user interface originates from the determined location.

[0173] In one or more embodiments, the method further includes ending the rendering of the floating virtual user interface, at least in part based on another user input. In one or more embodiments, the other user input includes a gesture, and the gesture includes a motion of cutting across the rendered virtual user interface with the user's hand. In one or more embodiments, the virtual user interface is created based at least in part on a world-centered reference frame such that the virtual user interface remains stationary with respect to any movement of the user.

[0174] In one or more embodiments, the method further includes selecting a virtual user interface element of at least one selectable virtual interface element based at least in part on another user input received from the user, and generating a second virtual user interface at another location of at least one physical object, the second virtual user interface comprising additional selectable virtual interface elements. In one or more embodiments, the virtual user interface resembles a tree growing from a location where it touches at least one physical object.

[0175] In another aspect, a method of displaying a virtual user interface includes identifying, based on user input, a user interface to be presented to the user, depicting a virtual user interface that mirrors the movement of the user's finger such that the virtual user interface mirrors the movement of the finger, and displaying one or more selectable user interface elements within an area in which the virtual user interface is depicted. In one or more embodiments, a finger gesture includes a gesture indicated using at least one finger of the user's hand for a predetermined period of time.

[0176] In one or more embodiments, the method further includes determining a location of the indicating gesture, and the virtual user interface originates from the determined location. In one or more embodiments, the method further includes detecting that the user is no longer depicting the virtual user interface, and the virtual user interface elements are presented, at least in part, based on the detection. In one or more embodiments, the depiction of the virtual user interface mirrors a dragging movement of the user's finger across space. In one or more embodiments, the method further includes ending the rendering of the floating virtual user interface, at least in part, based on another user input.

[0177] In one or more embodiments, the other user input includes a gesture, and the gesture includes the end of a continuous dragging movement of the user's finger. In one or more embodiments, the virtual user interface is created, at least in part, based on a world-centered reference frame such that the virtual user interface remains stationary relative to any movement of the user.

[0178] In one or more embodiments, the method further includes selecting a virtual user interface element of at least one selectable virtual interface element based at least in part on another user input received from the user, and generating content associated with the selected virtual interface element related to the virtual user interface. In one or more embodiments, the shape of the virtual user interface is similar to the shape depicted by the user's finger.

[0179] In another aspect, a method of creating a virtual user interface includes identifying a user interface to be presented to the user based on the user's hand gesture, and presenting a virtual user interface having at least one selectable virtual interface element in response to the hand gesture of the vertical side of the physical object such that the virtual user interface appears to be on the vertical side of the physical object.

[0180] In one or more embodiments, the method further includes presenting a set of virtual content similar to paint spots in response to the hand gesture, the virtual content occurring at the points indicated by the hand gesture. In one or more embodiments, the hand gesture includes an extended finger lifted in the direction of the vertical side of the physical object. In one or more embodiments, the method further includes determining the location of the extended finger lifted in the direction of the vertical side of the physical object, and the set of virtual content similar to paint spots is presented at the determined location.

[0181] In one or more embodiments, the method further includes presenting the virtual spots on the vertical side of the physical object, and at least one characteristic of the virtual spots is presented on the vertical side of the physical object based at least in part on the determined location of the extended finger lifted in the direction of the vertical side of the physical object. In one or more embodiments, the physical object includes a wall.

[0182] In one or more embodiments, the virtual user interface is created based at least in part on a world-centered reference frame such that the virtual user interface remains stationary with respect to any movement of the user. In one or more embodiments, the method further includes selecting a virtual user interface element of at least one selectable virtual interface element based at least in part on another user input received from the user, and generating content associated with the selected user interface element at another location of the physical object.

[0183] In one or more embodiments, the method further includes storing the content associated with the selected user interface element in a reduced form, the reduced form comprising a virtual band, the virtual band being displayed around the user's hand. In one or more embodiments, the virtual band is created based at least in part on a hand-centered reference frame such that the virtual band moves based at least in part on the movement of the hand.

[0184] In one or more embodiments, the method further includes displaying the content in a full form based at least in part on another gesture of the user. In one or more embodiments, the other gesture includes a lifting motion of the hand in which the virtual band is displayed.

[0185] In another aspect, a method of creating a virtual user interface includes identifying a user interface to be presented to the user based on a gesture of the user's hand, and presenting the virtual user interface in response to the gesture of the hand, the virtual user interface being such that at least one selectable virtual interface element moves based at least in part on another gesture of the user's other hand and resembles a horizontal thread having at least one selectable virtual interface element for displaying additional selectable virtual interface elements.

[0186] In one or more embodiments, the hand gesture includes an action of touching the first finger of the user's first hand with the second finger of the second hand. In one or more embodiments, the gesture further includes an action of separating both the first finger and the second finger such that the first finger is separated from the second finger. In one or more embodiments, the virtual thread mirrors the length of the movement of the first and second fingers, and the length of the virtual thread is at least partially based on the movement of the first and second fingers.

[0187] In one or more embodiments, the method further includes determining the touch movement of the first and second fingers, and the virtual thread originates from the determined location. In one or more embodiments, the gesture of the other hand includes the user's hand turning movement, and the turning movement moves at least one selectable virtual interface element in the direction of the turning movement.

[0188] In one or more embodiments, the virtual user interface is created based at least in part on a world-centered reference frame such that the virtual user interface remains stationary with respect to any movement of the user. In one or more embodiments, the method further includes selecting a virtual user interface element of at least one selectable virtual interface element based at least in part on the gesture of the user's other hand, and generating content associated with the selected user interface element at another location of the physical object. In one or more embodiments, the gesture of the other hand includes an action of bringing one of the user's hands forward.

[0189] In another aspect, a method of creating a virtual user interface includes identifying a user interface to be presented to the user based on the user's hand gesture, and presenting the virtual user interface in response to the hand gesture, the virtual user interface comprising a set of vertical threads, each of the vertical threads being associated with at least one selectable virtual interface element.

[0190] In one or more embodiments, the hand gesture includes a gesture of spreading the user's first hand during a predetermined period. In one or more embodiments, the method further includes selecting at least one selectable virtual interface element based at least in part on a gesture of another hand, wherein the gesture of the other hand includes a movement of pulling at least one of the virtual threads of a set of virtual threads using two fingers of the user.

[0191] In one or more embodiments, the method further includes transmitting data associated with a first virtual element associated with a first virtual thread to a second virtual element associated with a second virtual thread based at least in part on a hand gesture, wherein the gesture of the other hand includes a squeezing movement of the first virtual element associated with the first virtual thread using the second virtual element associated with the second virtual thread, and the squeezing movement is performed by the user's finger.

[0192] In one or more embodiments, the virtual interface is displayed in relation to at least one physical object. In one or more embodiments, the at least one physical object includes a wall. In one or more embodiments, the method further includes determining a location of the hand gesture, and the virtual thread originates from the determined location.

[0193] In one or more embodiments, the virtual user interface is created based at least in part on a world-centered reference frame such that the virtual user interface remains stationary with respect to any movement of the user. In one or more embodiments, the method further includes selecting a virtual user interface element of at least one selectable virtual interface element based at least in part on a gesture of another hand of the user, and generating virtual content associated with the selected user interface element at another location of the physical object.

[0194] In another aspect, a method of creating a virtual user interface includes identifying a user interface to be presented to a user based on the user's hand gesture, and presenting the virtual user interface in response to the hand gesture, the virtual user interface including a set of vertical threads, each vertical thread being associated with at least one selectable virtual interface element. In one or more embodiments, the hand gesture includes spreading the user's first hand during a predetermined period.

[0195] In one or more embodiments, the method further includes selecting at least one selectable virtual interface element based at least in part on another hand gesture, the other hand gesture including a motion of pulling at least one of the virtual threads of the set of virtual threads using two fingers of the user. In one or more embodiments, the method further includes transmitting data associated with a first virtual element associated with a first virtual thread to a second virtual element associated with a second virtual thread based at least in part on the hand gesture.

[0196] In one or more embodiments, the other hand gesture includes a squeezing motion of a first virtual element associated with a first virtual thread using a second virtual element associated with a second virtual thread, the squeezing motion being performed by the user's finger. In one or more embodiments, the virtual interface is presented in relation to at least one physical object. In one or more embodiments, the at least one physical object includes a wall.

[0197] In one or more embodiments, the method further includes determining a location of the hand gesture, the virtual threads originating from the determined location. In one or more embodiments, the virtual user interface is created based at least in part on a world-centered reference frame such that the virtual user interface remains stationary with respect to any movement of the user.

[0198] In one or more embodiments, the method further includes selecting, at least in part based on a gesture of another hand of the user, a virtual user interface element of at least one selectable virtual interface element; and generating virtual content associated with the selected user interface element at another location of the physical object.

[0199] In yet another aspect, a method of creating a virtual user interface includes identifying a user interface to be presented to a user based on a gesture of the user's hand; and presenting the virtual user interface in response to the gesture of the hand, the virtual user interface resembling a virtual spider web, and the user pulling a virtual thread of the virtual spider web to bring the virtual interface closer to the user.

[0200] In one or more embodiments, the gesture of the hand includes a pulling motion using a clenched fist of the user's hand. In one or more embodiments, the virtual threads of the virtual spider web comprise at least one selectable virtual element. In one or more embodiments, the method further includes selecting, at least in part based on another gesture of the hand, at least one selectable virtual interface element, the gesture of the other hand including a pulling motion of at least one of the virtual threads toward the user.

[0201] In one or more embodiments, the method further includes transmitting data associated with a first virtual element associated with a first virtual thread to a second virtual element associated with a second virtual thread, at least in part based on a gesture of the hand. In one or more embodiments, the gesture of the other hand includes a squeezing motion of the first virtual element associated with the first virtual thread using the second virtual element associated with the second virtual thread, the squeezing motion being performed by the user's fingers.

[0202] In one or more embodiments, the virtual interface is displayed in relation to at least one physical object. In one or more embodiments, the at least one physical object comprises a wall. In one or more embodiments, the method further includes determining a location of a hand gesture, and the virtual spider web originates from the determined location.

[0203] In one or more embodiments, the virtual user interface is created based at least in part on a world-centered reference frame such that the virtual user interface remains stationary with respect to any movement of the user. In one or more embodiments, the method further includes selecting a virtual user interface element of at least one selectable virtual interface element based at least in part on another hand gesture of the user, and generating virtual content associated with the selected user interface element at another location of the physical object.

[0204] Additional and other objects, features, and advantages of the present invention are set forth in the detailed description, the drawings, and the claims. This specification also provides, for example, the following items. (Item 1) A method for generating map data, comprising: identifying a set of map points associated with one or more images; determining a set of low-density points and a set of high-density points from the identified map points; normalizing each of the sets of low-density points and high-density points; and a method. (Item 2) generating low-density and high-density point descriptors for each of the sets of low-density points and high-density points; combining the low-density point descriptor and the high-density point descriptor and storing them as map data; The method according to Item 1, further comprising. (Item 3) The set of the low-density points corresponds to the unique features, the method according to item 1. (Item 4) The unique features are selected from the group consisting of a corner, a circle, a triangle, and text, the method according to item 3. (Item 5) The set of the high-density points corresponds to the 3D points in the field of view, the method according to item 1. (Item 6) The set of the high-density points also includes color values, the method according to item 5. (Item 7) The normalization includes scale normalization, the method according to item 1. (Item 8) The normalization includes coordinate normalization with respect to a common origin, the method according to item 1. (Item 9) The normalization is implemented using machine learning, the method according to item 1. (Item 10) The low-density and high-density point descriptors correspond to each of the low-density and high-density points of the respective sets of the low-density and high-density points, the method according to item 1. (Item 11) Each of the low-density and high-density point descriptors includes information regarding the respective low-density and high-density points selected from the group consisting of scale, orientation, patch data, and texture, the method according to item 10. (Item 12) A method for determining user input, capturing an image of the user's field of view, the image including a gesture created by the user, analyzing the captured image to identify a set of points associated with the gesture, comparing the identified set of points with a set of points associated with a database of predetermined gestures, determining user input based on the recognized gesture and including, the method. (Item 13) The method according to item 12, further comprising generating a scoring value for the set of identified points based on the comparison. (Item 14) The method according to item 13, further comprising recognizing the gesture when the scoring value exceeds a threshold. (Item 15) The method according to item 12, further comprising storing the predetermined gesture in a database. (Item 16) The method according to item 15, further comprising accessing a networked memory and accessing the database of the predetermined gesture. (Item 17) The method according to item 12, wherein the gesture is a hand gesture. (Item 18) The method according to item 12, wherein the gesture is a finger gesture. (Item 19) The method according to item 12, wherein the gesture is an interaction between fingers. (Item 20) The method according to item 12, wherein the gesture is selected from the group consisting of finger-to-finger interaction, pointing, tapping, and friction. (Item 21) The method according to item 12, further comprising displaying one or more virtual objects to the user based at least in part on the determined user input. (Item 22) The method according to item 21, wherein the one or more virtual objects comprise a virtual user interface. (Item 23) A method comprising: capturing one or more images corresponding to the user's field of view; analyzing the one or more captured images to identify one or more gestures made by the user, the analysis including using a cascade mechanism having a plurality of stages; A method comprising the above. (Item 24) The cascade mechanism comprises a plurality of nodes, each node corresponding to a certain stage among the plurality of stages, the method according to item 23. (Item 25) The cascade mechanism comprises a series of permissive analysis nodes, the method according to item 23. (Item 26) Among the plurality of stages of the cascade mechanism, the previous stage consumes less processing power compared to the subsequent stage among the plurality of stages of the cascade mechanism, the method according to item 23. (Item 27) Non-gestures are removed based on the analysis of the images captured at the previous stage of the cascade mechanism, the method according to item 26. (Item 28) Among the plurality of stages, the subsequent stage determines more complex gestures based at least in part on the captured images, the method according to item 26. (Item 29) The analysis of the captured images includes determining whether the sharpness of the contours of various shapes in the captured images is sharp enough to constitute a gesture, the method according to item 26. (Item 30) The subsequent stage of the cascade mechanism is used to distinguish between different gestures, the method according to item 26. (Item 31) The method according to item 23 further includes generating a score based at least in part on the analysis. (Item 32) The method according to item 31 further includes excluding candidate images from consideration if the generated score is lower than a minimum threshold. (Item 33) The method according to item 31 further includes proceeding to the subsequent stage of the cascade mechanism if the generated score is higher than a minimum threshold. (Item 34) A method comprising: capturing a plurality of images of each user's field of view; Analyzing the plurality of images to generate a plurality of gesture candidates; Generating an analysis value corresponding to the plurality of analysis values, wherein the gesture is recognized based at least in part on the analysis value; A method comprising the above. (Item 35) The method according to item 34, further comprising sorting the gesture candidates based at least in part on each of the analysis values. (Item 36) The method according to item 34, further comprising eliminating gesture candidates having an analysis value lower than a minimum threshold value. (Item 37) The method according to item 34, further comprising advancing the gesture candidate to the next stage of processing when the analysis value is higher than a minimum threshold value. (Item 38) A method for classifying a gesture, comprising: Capturing an image of a user's field of view; Performing depth partitioning on the captured image to generate a depth map; Identifying a gesture based at least in part on the generated depth map; A method comprising the above. (Item 39) The method according to item 38, further comprising using a classification mechanism to analyze the depth map and identify a part of a hand corresponding to a point in the depth map. (Item 40) The method according to item 39, further comprising skeletonizing the depth map based on the identification of the part of the hand. (Item 41) The method according to item 40, further comprising classifying the image as a gesture based on the skeletonized depth map. (Item 42) The method according to item 38, wherein the depth partitioning comprises a line search. (Item 43) The method according to item 38, further comprising performing cascade analysis on the depth map and classifying the image as the gesture. (Item 44) The method according to item 38, further comprising performing depth expansion on the depth map. (Item 45) The method according to item 38, further comprising performing surface normalization on the depth map. (Item 46) The method according to item 38, further comprising performing orientation normalization on the depth map. (Item 47) The method according to item 38, further comprising performing background subtraction on the depth map. (Item 48) The method according to item 38, further comprising performing depth comparison on the depth map. (Item 49) The method according to item 38, further comprising classifying the image as a gesture based on the skeletonized depth map and previous information. (Item 50) The method according to item 38, wherein the classification mechanism is a decision forest. (Item 51) A method for determining user input, comprising: capturing an image of the field of view of one or more users, the image comprising at least a predetermined physical object; analyzing the image to detect movement of the user related to the predetermined physical object; determining user input based at least in part on an analysis of the movement related to the predetermined physical object. Including the method. (Item 52) The method according to item 51, further comprising recognizing the predetermined physical object. (Item 53) The method according to item 52, wherein the predetermined physical object is recognized based at least in part on an image comparison between a captured image of the predetermined physical object and a database of the predetermined physical object. (Item 54) The method according to item 51, wherein the movement of the user related to the predetermined physical object is used to generate a pattern. (Item 55) The method according to item 54, further comprising comparing the generated pattern with a database of predetermined patterns. (Item 56) The method according to item 51, further comprising generating a scoring value for the recognized pattern based on the comparison. (Item 57) The method according to item 56, further comprising determining the user input if the scoring value exceeds a threshold. (Item 58) The method according to item 51, further comprising visually tracking the movement related to the predetermined physical object and generating a video recording. (Item 59) The method according to item 58, wherein the video recording is analyzed to determine the user input. (Item 60) The method according to item 55, further comprising accessing the database of the predetermined patterns through a networked memory. (Item 61) The method according to item 51, wherein the predetermined physical object is selected from the group consisting of an existing structure within the field of view, an actively marked totem, a passively marked totem, an object integrated into a camera / sensor, and a totem controller object. (Item 62) The method according to item 51, wherein the movement related to the predetermined physical object is selected from the group consisting of the position, orientation, and movement of the predetermined physical object with respect to a reference frame. (Item 63) The method according to item 51, wherein the predetermined physical object comprises a first hand of the user's hands, and the movement related to the first hand comprises an operation of the first hand using a second hand of the user's hands. (Item 64) The method according to item 51, wherein the predetermined physical object has a soft hardness surface, and the movement related to the predetermined physical object comprises pressing of the soft hardness surface by the user. (Item 65) The method according to item 51, further comprising rendering a virtual interface element associated with the predetermined physical object for the user. (Item 66) The method according to item 65, wherein when the virtual interface element is viewed by the user, the virtual interface element is displayed in relation to the predetermined physical object such that the virtual interface element is modified based at least in part on a modification related to the predetermined physical object. (Item 67) The method according to item 51, wherein the predetermined physical object comprises an electronic input device, and user input is determined based on a recognized movement of the predetermined physical object and is input from the electronic input device. (Item 68) The method according to item 51, further comprising modifying at least one characteristic of virtual content displayed to the user based at least in part on the determined user input. (Item 69) A method of generating a virtual user interface, comprising: identifying a virtual user interface to be displayed to a user based on user input; obtaining a location associated with the user; determining a set of coordinate points at which the identified virtual user interface is to be displayed based at least in part on the obtained location; and displaying the virtual user interface to the user at the determined coordinate points. A method including (Item 70) The method according to item 69, wherein the user input is determined based at least in part on a recognized gesture. (Item 71) The method according to item 69, wherein the user input is determined based at least in part on a voice command. (Item 72) The method according to item 69, wherein the user input is determined based at least in part on an interaction with a predetermined physical object. (Item 73) The method according to item 69, further comprising reading the identified user virtual user interface from a library of user interfaces. (Item 74) The method according to item 69, wherein the identified virtual user interface is associated with a reference frame. (Item 75) The method according to item 74, wherein the reference frame is a body-centered reference frame. (Item 76) The method according to item 74, wherein the reference frame is a head-centered reference frame. (Item 77) The method according to item 74, wherein the reference frame is a hand-centered reference frame. (Item 78) The method according to item 74, wherein the reference frame is a world-centered reference frame. (Item 79) The method according to item 69, further comprising performing a conversion between the reference frame associated with the identified virtual user interface and the obtained location of the user relative to the world to determine a set of coordinate points of the virtual user interface. (Item 80) The method according to item 69, wherein the location of the user within the world is determined based on the GPS location of the user. (Item 81) The method according to item 69, wherein the location of the user within the world is determined based on a set of map points associated with the user. (Item 82) The method according to item 69, wherein the virtual user interface appears to be stationary when the user moves. (Item 83) The method according to item 69, wherein the virtual user interface moves in relation to the movement of the user. (Item 84) Determining a reference frame associated with the identified virtual user interface; Determining the location of the reference frame in relation to the world reference frame; Setting the determined location as the origin; Determining a set of coordinate points in relation to the origin; The method according to item 69, further comprising: (Item 85) The method according to item 69, further comprising reading the set of map points from a networked memory. (Item 86) The method according to item 69, wherein the user input comprises a location within the space where the virtual user interface is to be displayed. (Item 87) The method according to item 86, wherein the location within the space is associated with a physical entity at the location of the user. (Item 88) The method according to item 69, wherein the user input comprises a gesture including a throwing input indicating a wall. (Item 89) The method according to item 69, wherein the user input comprises a gesture indicating the end of an instruction for generating the virtual user interface. (Item 90) A method for generating a virtual user interface, comprising: Detecting an operation of a predetermined physical object; Recognizing a command for creating a virtual user interface based on the detected operation; Determining a set of map points associated with the position of the predetermined physical object from the virtual world model; When the virtual user interface is viewed by the user, rendering the virtual user interface in real time at the determined map points associated with the position of the totem so as to appear to be stationary at the position of the predetermined physical object; A method comprising: (Item 91) The method according to item 90, wherein the operation of the predetermined physical object includes a gesture of spreading the state pinched by the user's hand on the surface of the predetermined physical object. (Item 92) The method according to item 91, wherein when the virtual user interface is viewed by the user, it appears to cover a part of the surface of the predetermined physical object, and the part corresponds to the location of the user's hand during the formation of the gesture of spreading the pinched state. (Item 93) The method according to item 90, wherein the predetermined physical object is the user's hand. (Item 94) The method according to item 93, wherein the operation of the predetermined physical object includes an action selected from the group consisting of the user spreading the hand, the user showing the palm, and the user raising the hand. (Item 95) The method according to item 94, wherein when the virtual user interface is viewed by the user, it appears to cover a part of the surface of the hand. (Item 96) The method according to item 95, wherein the virtual user interface includes a plurality of first-level menu items selectable by the fingers or thumb of the hand. (Item 97) Detecting a further operation of the hand; Recognizing a command for creating a second virtual user interface based on the detected further operation; When viewed by the user, rendering the second virtual user interface in real time at the determined map point associated with the position of the predetermined physical object such that the virtual user interface appears to be stationary at the position of the predetermined physical object; The method according to item 96, further comprising: (Item 98) The method according to item 97, wherein the further operation of the hand includes spreading the fingers of the hand apart. (Item 99) The method according to item 98, wherein the second virtual user interface comprises a plurality of second-level menu items selectable by a finger or thumb of the hand, and the second level is lower than the first level. (Item 100) The method according to item 97, wherein the further operation of the totem includes making a motion of drawing a circle on the palm of the hand using a finger from a second hand of the user's hands. (Item 101) The method according to item 100, wherein the second virtual user interface comprises a plurality of menu items arranged in an arc, and the menu items are scrollable and selectable by a finger of the second hand. (Item 102) A method for updating a virtual world, comprising: Receiving a first input from a first device of a first user, the first input corresponding to the physical environment of the first user; Updating a virtual world model based on the received first input, the virtual world model corresponding to the physical environment of the first user; Transmitting first updated information corresponding to a first portion of the virtual world model to a second user; and comprising: The method for indicating whether any part of the first updated information needs to be displayed to the second user. (Item 103) The virtual world model resides on a networked memory, the method according to item 102. (Item 104) The first user and the second user are located at different locations, the method according to item 102. (Item 105) The first device of the first user is selected from the group consisting of a FOV camera, other cameras, sensors, a first device for eye tracking, and a first device for audio, the method according to item 102. (Item 106) Further including transmitting the first updated information corresponding to the first part of the virtual world model to the first user, and the first updated information indicates whether any part of the first updated information needs to be displayed to the first user, the method according to item 102. (Item 107) Further including transmitting the first updated information corresponding to the first part of the virtual world model to a plurality of other users, and the first updated information indicates whether any part of the first updated information needs to be displayed to each of the plurality of other users, the method according to item 102. (Item 108) Receiving a plurality of inputs from respective first devices of a plurality of other users, wherein the plurality of inputs correspond to the physical environment of the first user, and Updating the virtual world model based on the received plurality of inputs, and Transmitting additional updated information corresponding to each additional part of the virtual world model to the second user And further including The additional updated information indicates whether any part of the additional updated information needs to be displayed to the second user, the method according to item 102. (Item 109) Further comprising transmitting to the first user the additional updated information corresponding to the portion of the virtual world model, wherein the additional updated information indicates whether any portion of the additional updated information needs to be displayed to the first user, the method according to item 108. (Item 110) Further comprising transmitting to the plurality of other users the additional updated information corresponding to each additional portion of the virtual world model, wherein the additional updated information indicates whether any portion of the additional updated information needs to be displayed to each of the plurality of other users, the method according to item 108. (Item 111) Receiving a second input from a second device of the second user, the second input corresponding to the physical environment of the second user, and Updating the virtual world model based on the received second input, the virtual world model corresponding to the physical environment of the first user, and Transmitting to the first user second updated information corresponding to a second portion of the virtual world model and Further comprising The second updated information indicates whether any portion of the second updated information needs to be displayed to the first user, the method according to item 102. (Item 112) The second updated information corresponds to the movement of the avatar of the second user within the second portion of the virtual world model, the method according to item 111. (Item 113) A method comprising Projecting a light pattern spatially Using a camera to detect first and second portions of the light pattern and generating first and second data points corresponding to the first and second portions of the pattern Performing triangulation analysis to determine the locations of the first and second portions of the light pattern based at least in part on the first and second data points A method comprising (Item 114) The method according to item 113, further comprising repeating the steps of projecting, detecting, and triangulating a plurality of portions of the pattern to obtain additional texture data for the space. (Item 115) The method according to item 113, wherein projecting the light onto the space includes projecting the light onto the space using a fiber-based projector (Item 116) The method according to item 113, further comprising modifying the light using an optical element to form a beam of light rays before the light is projected onto the space (Item 117) The method according to item 113, wherein the light pattern corresponds to structured light and the light pattern is dynamic (Item 118) The method according to item 113, wherein the light pattern corresponds to patterned light and the light pattern includes a plurality of points (Item 119) The method according to item 113, wherein the light pattern corresponds to textured light and the light pattern is irregular (Item 120) The method according to item 113, wherein the first and second portions of the light pattern are respective pixels of the light pattern (Item 130) A method of generating a virtual room, comprising Identifying a virtual room to be displayed to a user Obtaining a first set of map points associated with the virtual room, the first set of map points corresponding to a first physical room at the location of the user Tethering the virtual room to the first set of map points When viewed by the user, displaying the virtual room to the user such that the virtual room appears to be stationary in the set of the first map points A method comprising the above. (Item 131) The method according to item 130, further comprising identifying the virtual room based on user input. (Item 132) The method according to item 131, wherein the user input is selected from the group consisting of gestures, visual data, auditory data, sensory data, direct commands, voice commands, eye tracking, and selection of physical buttons. (Item 133) The method according to item 131, wherein the virtual room is predetermined by the user. (Item 134) The method according to item 131, further comprising reading the set of the first map points from a networked memory. (Item 135) Obtaining a set of second map points associated with the virtual room, wherein the set of second map points corresponds to a second physical room at the location of the user after the user has moved from the first physical room, Tethering the virtual room to the set of the second map points, When viewed by the user, displaying the virtual room to the user such that the virtual room appears to be stationary in the set of the second map points The method according to item 131, further comprising the above. (Item 136) The virtual room also includes first and second virtual objects, and the method further comprises maintaining a relative spatial position between the first virtual object corresponding to the set of the first map points and the second virtual object when the virtual room is tethered to the set of the second map points. The method according to item 135. (Item 137) The method according to item 131, further comprising displaying the virtual room to the user such that when viewed by the user, the virtual room appears to be stationary with respect to a part of the first physical room. (Item 138) The method according to item 131, wherein the virtual room also includes selectable virtual decorations. (Item 139) A method for creating a retail experience, comprising: recognizing the location of a user within a retail store; reading data corresponding to the retail store; generating virtual content related to the retail store based on the read data; creating a virtual user interface within the user's field of view; displaying the virtual content on the virtual user interface while the user is engaged in retail activities within the retail store. A method comprising the above steps. (Item 140) The read data comprises a set of map points corresponding to the retail store, The method according to item 139, wherein the virtual user interface appears to be stationary within the set of map points when viewed by the user. (Item 141) The method according to item 139, wherein the location of the user is recognized using a radio frequency identification transponder and communication. (Item 142) The virtual content is selected from the group consisting of virtual characters, virtual coupons, games based on the user location, lists of promotional items, nutritional information, metadata related to items, appearances of famous people, cross-selling advertisements, information from people known to the user, and e-books. The method according to item 139. (Item 143) reading user data corresponding to the user; generating the virtual content based on both the read data and the read user data The method according to item 139, further comprising (Item 144) The virtual content is selected from the group consisting of a virtual grocery list, a virtual coupon book, a virtual recipe book, a list of various materials in the user's home, and a virtual recipe builder, according to the method of item 143. (Item 145) receiving user input; generating additional virtual content based on the user input; displaying the additional virtual content on the virtual user interface while the user is engaged in retail activities within the retail store The method according to item 139, further comprising (Item 146) The user input is selected from the group consisting of gestures, visual data, auditory data, sensory data, direct commands, voice commands, eye tracking, and selection of physical buttons, according to the method of item 145. (Item 147) The virtual content is selected from the group consisting of total operating costs, a smart virtual grocery list, an indicator of items proximate to the user's location, and a virtual payment system, according to the method of item 145. (Item 148) The method according to item 139, further comprising transmitting the generated virtual content to a user device for display. (Item 149) A method comprising: reading patient data related to a patient's medical history; generating virtual content based at least in part on the read patient data; reading a set of first map points corresponding to a first location of the first user; Creating a first virtual user interface within the field of view of a first user based at least in part on the set of read map points; Displaying the virtual content on the first virtual user interface such that when viewed by the first user, the first virtual user interface appears to be fixed at the set of first map points; A method comprising the above. (Item 150) The method according to item 149, wherein the virtual content is selected from the group consisting of a three-dimensional image of a surgical target, patient identification information, medical images, patient vital sign information, and a medical chart. (Item 151) The method according to item 149, wherein the patient data is read from a networked memory. (Item 152) The method according to item 149, further comprising generating a second virtual user interface and facilitating communication between the first user and a second user, wherein the second user is located in a second location different from the first location of the first user. (Item 153) The method according to item 152, wherein the second virtual user interface is a visual representation of the second user. (Item 154) The method according to item 152, wherein the second user is selected from the group consisting of a senior specialist surgeon, a patient, a person associated with the patient, and a medical student. (Item 155) The method according to item 152, further comprising displaying the virtual content on the first virtual user interface for the second user such that when viewed by the second user, the first virtual user interface appears to be fixed at the set of map points. (Item 156) The virtual content is displayed to the first user during the surgical procedure, and the method comprises: Receiving user input; generating additional virtual content based on the user input; displaying the additional virtual content on the first virtual user interface while the first user is performing the surgical procedure; The method according to item 149, further comprising. (Item 157) The method according to item 156, wherein the user input is selected from the group consisting of gestures, visual data, auditory data, sensory data, direct commands, voice commands, eye tracking, and selection of physical buttons. (Item 158) The method according to item 156, wherein the user input comprises an image of the field of view of the first user, and the method further comprises displaying the additional virtual content on the first virtual user interface for the second user such that when viewed by the second user, the first virtual user interface appears to be fixed at the set of map points. (Item 159) A method for promoting medical rehabilitation, comprising: receiving user input related to medical rehabilitation; sending a request related to medical rehabilitation to a cloud server based on the received user input; receiving data related to medical rehabilitation from a knowledge base connected to the cloud server based on the request; displaying virtual content related to medical rehabilitation to a first user based on the received data; A method comprising. (Item 160) The method according to item 159, wherein the user input is selected from the group consisting of visual, auditory, and sensory inputs. (Item 161) The method according to item 159, further comprising determining whether the user input is valid before sending the request to the cloud server. (Item 162) The method according to item 159, wherein the data is received from a plurality of knowledge bases connected to the cloud server. (Item 163) The method according to item 159, wherein the request instructs the cloud server to read data from the knowledge base. (Item 164) The method according to item 159, wherein the virtual content includes an environment for relaxation. (Item 165) Receiving additional data from a second user, and displaying additional virtual content to the first user based on the received additional data The method according to item 159, further comprising. (Item 166) The method according to item 165, wherein the first and second users are located in different physical locations. (Item 167) The method according to item 165, wherein the additional virtual content is a visual representation of the second user. (Item 168) Receiving first user data from the first user, Receiving second user data from a second user, modifying the virtual content based on the received first and second user data, and displaying the modified virtual content to the first and second users The method according to item 159, further comprising. (Item 169) The method according to item 159, wherein the first user is engaged in a physical activity for medical rehabilitation, and the virtual content is related to the physical activity. (Item 170) The method according to item 169, wherein the virtual content is selected from the group consisting of information about the physical activity, performance statistics of the first user, virtual experiences corresponding to the physical activity, and virtual avatars. (Item 171) The data is read from a networked memory, and the method described in item 159. (Item 172) A method for improving the performance of a task, reading task data related to the task to be performed by the user, generating virtual content based on the task data, creating a virtual user interface within the user's field of view, reading a set of map points corresponding to the location of the user, displaying the virtual content on the virtual user interface while the user is performing the task such that when viewed by the user, the virtual user interface appears to be fixed within the set of map points comprising a method. (Item 173) The virtual content is a game having a virtual mapped pattern, and the game improves the performance of the task by the user, the method described in item 172. (Item 174) The game gives points for following the virtual mapped pattern, the method described in item 173. (Item 175) The game doubles the points for reaching a location within the virtual mapped pattern within a predetermined time, the method described in item 174. (Item 176) The game deducts points for deviating from the virtual mapped pattern, the method described in item 175. (Item 177) The game deducts points for moving adjacent to a physical object, the method described in item 175. (Item 178) The method according to item 172, wherein the task is selected from the group consisting of operating a gardening machine, reading an inventory item, displaying on a retail shelf, and sorting mail. (Item 179) The method according to item 172, wherein the task data is read from a networked memory. (Item 180) Receiving user input; Generating additional virtual content based on the user input; Displaying the additional virtual content on the virtual user interface while the user is performing the task The method according to item 172, further comprising: (Item 181) The method according to item 180, wherein the user input comprises user actions related to performance of the task. (Item 182) A method for correcting latency in an augmented reality display system, comprising: Determining one or more timing mismatches in at least one stage of displaying a virtual image to a user; Predictively correcting the timing mismatch using a prediction mechanism; Compensating for at least one characteristic related to the data of the virtual image based at least in part on the prediction mechanism The method comprising: (Item 183) The method according to item 182, wherein the timing mismatch involves one or more sensor measurements performed by one or more sensors communicatively coupled to the processor. (Item 184) The method according to item 182, wherein the timing mismatch involves a processing delay when processing incoming data. (Item 185) The method according to item 182, wherein the prediction mechanism uses a filter to correct the effect of the timing mismatch. (Item 186) The method according to item 185, wherein the filter takes into account the relative speed of one or more sensor measurements, and the sensor measurements are performed by one or more sensors communicatively coupled to the processor. (Item 187) The method according to item 182, wherein the prediction mechanism utilizes a Kalman predictor. (Item 188) The method according to item 187, wherein the Kalman predictor is utilized during the display processing stage. (Item 189) The method according to item 182, wherein the compensation includes performing a shift of data associated with the virtual image. (Item 190) The method according to item 182, wherein the compensation includes smoothing one or more visual artifacts associated with the virtual image. (Item 191) The method according to item 182, wherein the compensation includes correcting for the negative impact of sensor measurements of one or more sensors, and the one or more sensors are communicatively coupled to the processor. (Item 192) A method for calibrating an augmented reality system, comprising: displaying a virtual image to a user, the virtual image being displayed at a known focal length and the virtual image comprising pixel points; determining a location where the pixel points are being displayed to the user, the location of the pixels being calculated based at least in part on the location of the user's eye pupils; aligning the pixel points of the virtual image with known points in space; and. (Item 193) The method according to item 192, wherein the step of item 331 is repeated for a plurality of pixel points. (Item 194) The method according to item 192, wherein the location where the pixel points are being displayed to the user is calculated based at least in part on the location of known points in the space. (Item 195) The method according to item 192, wherein the location of the pupil, the location of the pixel point, and the location of a known point in the space are on the same line. (Item 196) The method according to item 192, wherein the location where the pixel point is displayed to the user is corrected based on user input. (Item 197) The method according to item 196, further comprising creating a game interface such that a plurality of pixel points are presented to the user. (Item 198) The method according to item 197, wherein the game interface includes emitting a laser through the eye movement of the user. (Item 199) The method according to item 197, wherein the location of the known point is determined based at least in part on data received from one or more world cameras. (Item 200) The method according to item 192, wherein the step of item 1 is repeated for the other eye of the user. (Item 201) The method according to item 193, wherein the location of the pixel point is determined based at least in part on a function of the location of the pupil and the location of the known point. (Item 202) The method according to item 201, wherein the function comprises a quadratic function. (Item 203) A method of displaying a virtual interface, comprising: identifying, based on user input, a user interface to be displayed to the user; displaying an avatar user interface in relation to at least one physical object, the avatar user interface presenting at least one virtual user interface element; selecting at least one virtual user interface element based at least in part on an interaction between the user and the avatar user interface A method comprising (Item 204) The user input is the method according to item 203, including gestures. (Item 205) The method according to item 204, further comprising determining a location of the gesture related to the user's augmented reality display system, wherein the virtual user interface is displayed at the determined location. (Item 206) The method according to item 203, wherein the avatar is a pre-selected avatar. (Item 207) The method according to item 203, wherein the avatar resembles the user. (Item 208) The method according to item 203, wherein the avatar is displayed as if standing on the physical object. (Item 209) The method according to item 203, wherein the at least one virtual element has an application. (Item 210) Selecting the at least one virtual element based at least in part on another user input, wherein the at least one virtual element has the user's contact information, and Displaying another avatar representing the user's contact information in relation to the avatar virtual user interface and the physical object, wherein the interaction between the user and the user's contact information is animated through a virtual interaction between the avatar virtual user interface and another avatar representing the user's contact information, and The method according to item 203, further comprising (Item 211) The method according to item 210, wherein the interaction includes transmitting data between the user's augmented reality system and a computing system corresponding to the user's contact information. (Item 212) The method according to item 203, wherein the user input has a voice command. (Item 213) A method for displaying a virtual interface, comprising: identifying, based on user input, a user interface to be displayed to the user; displaying a floating user interface having one or more selectable virtual interface elements in relation to at least one physical object, the floating user interface appearing to be pushed out from the location where the user input was received; A method as described above. (Item 214) The method according to item 213, wherein the user input includes a gesture of the user's finger. (Item 215) The method according to item 214, wherein the gesture of the user's finger includes touching a finger of one hand of the user with another finger of the other hand of the user. (Item 216) The method according to item 215, further comprising determining a location where a finger of one hand of the user touches another finger of the other hand of the user, and the floating virtual user interface originates from the determined location. (Item 217) The method according to item 213, wherein the floating user interface comprises a set of three-dimensional blocks, and the three-dimensional blocks in the set of three-dimensional blocks represent the at least one selectable virtual user interface element. (Item 218) The method according to item 213, wherein the floating virtual user interface is created based at least in part on a world reference frame. (Item 219) The method according to item 213, wherein the at least one selectable virtual user interface element exists as a stack of three-dimensional blocks, and the stack of three-dimensional blocks is rotated based at least in part on another user input. (Item 220) Selecting a virtual user interface element of the at least one selectable virtual interface element based at least in part on another user input received from the user, wherein the virtual interface element is associated with at least one other sub-virtual interface element; Displaying the at least one other sub-virtual interface element below the selected virtual interface element; The method according to item 213, further comprising. (Item 221) Selecting a virtual user interface element of the at least one selectable virtual interface element based at least in part on another user input received from the user; Displaying content associated with the selected virtual interface element within a virtual box, the virtual box comprising content displayed within the user's field of view; The method according to item 213, further comprising. (Item 222) Identifying a second user input indicating that the virtual box is to be closed; Animating the virtual box in a manner similar to crumpling a piece of paper based at least in part on the second user input; Ending the content associated with the selected virtual interface element; The method according to item 221, further comprising. (Item 223) The method according to item 222, wherein the second user input includes a gesture that resembles crumpling a single piece of paper. (Item 224) A method of displaying a virtual user interface, comprising: Identifying a user interface to be displayed to the user based on a user input; Displaying a floating user interface having one or more selectable virtual interface elements in relation to at least one physical object including wherein the floating user interface appears to be placed on the at least one physical object, and rotation of the at least one physical object about a vertical axis thereof results in additional selectable virtual interface elements being displayed, the additional selectable virtual interface elements appearing to be placed on another side of the at least one physical object. (Item 225) The method according to item 224, wherein the at least one physical object comprises the user's arm. (Item 226) The method according to item 224, wherein the user input includes a gesture. (Item 227) The method according to item 226, wherein the gesture includes a movement that forms a cup of the user's hand on the user's arm where the selectable virtual interface element appears to be displayed. (Item 228) The method according to item 227, further comprising determining a location of a movement that forms a cup of the user's hand on the user's arm, the floating user interface originating from the determined location. (Item 229) The method according to item 224, further comprising ending rendering of the floating virtual user interface based at least in part on another user input. (Item 230) The other user input includes a gesture, the gesture including a movement of dragging the user's cupped hand across the physical object. The method according to item 229. (Item 231) The floating virtual user interface is the method according to item 224, created at least partially based on the hand-centered reference frame. (Item 232) Selecting a virtual user interface element of the at least one selectable virtual interface element at least partially based on another user input received from the user; Displaying content associated with the selected virtual interface element, the content being displayed in relation to the floating virtual interface; The method according to item 224, further comprising. (Item 233) The method according to item 231, wherein the displayed content corresponds to the hand-centered reference frame. (Item 234) When the floating virtual user interface is moving at least partially based on the input received from the user, further moving the displayed content from the hand-centered reference frame to the world-centered reference frame so that the displayed content remains stationary. The method according to item 231. (Item 235) A method of creating a virtual user interface, Identifying a user interface to be displayed to the user based on user input; Displaying a virtual user interface originating from the location where at least one physical object is touched in relation to the at least one physical object, the virtual user interface appearing to pop out from the location where the at least one physical object is touched, the virtual user interface comprising at least one selectable virtual user interface element that appears to be attached to the virtual user interface; A method comprising. (Item 236) The method according to item 235, wherein the at least one physical object comprises a horizontal surface. (Item 237) The method according to item 235, wherein the user input includes a gesture. (Item 238) The method according to item 237, wherein the gesture includes a stretched finger touching the at least one physical object for a predetermined period of time. (Item 239) The method according to item 235, further comprising determining a location where the user's finger touches the at least one physical object, and wherein the virtual user interface originates from the determined location. (Item 240) The method according to item 235, further comprising ending the rendering of the floating virtual user interface at least partially based on another user input. (Item 241) The method according to item 240, wherein the other user input includes a gesture, and the gesture includes a motion of cutting across the rendered virtual user interface using the user's hand. (Item 242) The method according to item 235, wherein the virtual user interface is created at least partially based on a world-centered reference frame such that the virtual user interface remains stationary with respect to any movement of the user. (Item 243) Selecting a virtual user interface element of the at least one selectable virtual interface element at least partially based on another user input received from the user, and Generating a second virtual user interface at another location of the at least one physical object, the second virtual user interface comprising additional selectable virtual interface elements, The method according to item 235, further comprising. (Item 244) The method according to item 235, wherein the virtual user interface resembles a tree growing from the location where the at least one physical object is touched. (Item 245) A method of displaying a virtual user interface, identifying, based on user input, a user interface to be displayed to the user; depicting a virtual user interface that mirrors the movement of the user's finger such that the virtual user interface mirrors the movement of the finger; displaying one or more selectable user interface elements within the area where the virtual user interface is depicted and including. (Item 246) The method according to item 245, wherein the finger gesture includes a gesture indicated using at least one finger of the user's hand for a predetermined period. (Item 247) The method according to item 246, further including determining the location of the pointing gesture, wherein the virtual user interface originates from the determined location. (Item 248) The method according to item 245, further including detecting that the user is no longer depicting the virtual user interface, wherein the virtual user interface element is displayed based at least in part on the detection. (Item 249) The method according to item 245, wherein the depiction of the virtual user interface mirrors a dragging motion of the user's finger across space. (Item 250) The method according to item 245, further including ending the rendering of the floating virtual user interface based at least in part on another user input. (Item 251) The method according to item 250, wherein the other user input includes a gesture, and the gesture includes the end of a continuous dragging motion of the user's finger. (Item 252) The method according to item 245, wherein the virtual user interface is created based at least in part on a world-centered reference frame such that the virtual user interface remains stationary with respect to any movement of the user. (Item 253) Selecting a virtual user interface element among the at least one selectable virtual interface elements based at least in part on another user input received from the user; Generating content associated with the selected virtual interface element in relation to the virtual user interface; The method according to item 245, further comprising: (Item 254) The method according to item 245, wherein the shape of the virtual user interface resembles the shape depicted by the user's finger. (Item 255) A method of creating a virtual user interface, comprising: Identifying a user interface to be displayed to the user based on a gesture of the user's hand; Displaying a virtual user interface having at least one selectable virtual interface element on a vertical side of the physical object in the direction of the hand gesture such that the virtual user interface appears to be on the vertical side of the physical object in response to the hand gesture; A method comprising: (Item 256) The method according to item 255, further comprising displaying a set of virtual content resembling paint spots in response to the hand gesture, the virtual content occurring at points indicated by the hand gesture. (Item 257) The method according to item 255, wherein the hand gesture includes an extended finger lifted in the direction of the vertical side of the physical object. (Item 258) The method according to item 257, further comprising determining a location of an extended finger lifted in a direction of the vertical side surface of the physical object, and a set of virtual contents similar to paint spots is displayed at the determined location. (Item 259) The method according to item 258, further comprising displaying a virtual spot on the vertical side surface of the physical object, wherein at least one characteristic of the virtual spot is at least partially based on a determined location of an extended finger lifted in a direction of the vertical side surface of the physical object and is displayed on the vertical side surface of the physical object. (Item 260) The method according to item 255, wherein the physical object comprises a wall. (Item 261) The method according to item 255, wherein the virtual user interface is created at least partially based on a world-centered reference frame such that the virtual user interface remains stationary with respect to any movement of the user. (Item 262) selecting a virtual user interface element among the at least one selectable virtual interface elements at least partially based on another user input received from the user; generating content associated with the selected user interface element at another location of the physical object; The method according to item 255, further comprising the above. (Item 263) The method according to item 263, further comprising storing the content associated with the selected user interface element in a reduced form, the reduced form comprising a virtual band, and the virtual band is displayed around the user's hand. (Item 264) The method according to item 263, wherein the virtual band is created at least partially based on a hand-centered reference frame such that the virtual band moves at least partially based on the movement of the hand. (Item 265) The method according to item 263, further comprising displaying the content in a complete form based at least in part on another gesture of the user. (Item 266) The method according to item 265, wherein the other gesture includes a lifting motion of the hand where the virtual band is displayed. (Item 267) A method of creating a virtual user interface, comprising: identifying a user interface to be displayed to the user based on a gesture of the user's hand; displaying a virtual user interface in response to the gesture of the hand; and the virtual user interface is similar to a horizontal thread having at least one selectable virtual interface element, wherein the at least one selectable virtual interface element moves and additional selectable virtual interface elements are displayed based at least in part on another gesture of the user's other hand. (Item 268) The method according to item 267, wherein the gesture of the hand includes a motion of touching a first finger of the user's first hand with a second finger of a second hand. (Item 269) The method according to item 268, wherein the gesture further includes a motion of pulling both the first finger and the second finger away from each other such that the first finger moves away from the second finger. (Item 270) The method according to item 269, wherein the virtual thread mirrors the lengths of the movements of the first and second fingers, and the length of the virtual thread is based at least in part on the movements of the first and second fingers. (Item 271) The method according to item 267, further comprising determining a location of the touch motion between the first finger and the second finger, and the virtual thread originates from the determined location. (Item 272) The gesture of the other hand of the user includes a movement of changing the direction of the user's hand, and the method according to item 267, wherein the movement of changing the direction causes the at least one selectable virtual interface element to move in the direction of the movement of changing the direction. (Item 273) The method according to item 267, wherein the virtual user interface is created based at least in part on a world-centered reference frame such that the virtual user interface remains stationary with respect to any movement of the user. (Item 274) Selecting a virtual user interface element among the at least one selectable virtual interface elements based at least in part on a gesture of another hand of the user; Generating content associated with the selected user interface element at another location of the physical object; The method according to item 267, further comprising: (Item 275) The method according to item 274, wherein the gesture of the other hand includes a movement of bringing one hand of the user forward. (Item 276) A method of creating a virtual user interface, comprising: Identifying a user interface to be displayed to the user based on a gesture of the user's hand; Displaying a virtual user interface in response to the gesture of the hand; Comprising: The virtual user interface comprises a set of vertical threads, each of the vertical threads being associated with at least one selectable virtual interface element. (Item 277) The method according to item 276, wherein the gesture of the hand includes a gesture of spreading the first hand of the user for a predetermined period of time. (Item 278) Further comprising selecting the at least one selectable virtual interface element based at least in part on a gesture of the other hand, the gesture of the other hand comprising a movement of pulling at least one of the virtual threads of the set of virtual threads using two fingers of the user, the method of claim 277. (Item 279) Further comprising transmitting data associated with a first virtual element associated with a first virtual thread to a second virtual element associated with a second virtual thread based at least in part on a hand gesture, the method of claim 276. (Item 280) The gesture of the other hand comprises a squeezing movement of the first virtual element associated with the first virtual thread using the second virtual element associated with the second virtual thread, the squeezing movement being performed by a finger of the user, the method of claim 279. (Item 281) The virtual interface is displayed in relation to at least one physical object, the method of claim 276. (Item 282) The at least one physical object comprises a wall, the method of claim 281. (Item 283) Further comprising determining a location of a hand gesture, the virtual thread originating from the determined location, the method of claim 276. (Item 284) The virtual user interface is created based at least in part on a world-centered reference frame such that the virtual user interface remains stationary with respect to any movement of the user, the method of claim 276. (Item 285) Selecting a virtual user interface element of the at least one selectable virtual interface element based at least in part on a gesture of another hand of the user; Generating virtual content associated with the selected user interface element at another location of the physical object The method according to item 276, further comprising (Item 286) A method for creating a virtual user interface, comprising: identifying a user interface to be displayed to the user based on a gesture of the user's hand; displaying a virtual user interface in response to the gesture of the hand; and wherein the virtual user interface comprises a set of vertical threads, each of the vertical threads being associated with at least one selectable virtual interface element. (Item 287) The method according to item 286, wherein the gesture of the hand includes a gesture of spreading the first hand of the user for a predetermined period. (Item 288) The method according to item 287, further comprising selecting the at least one selectable virtual interface element based at least in part on another hand gesture, the other hand gesture including a movement of pulling at least one of the virtual threads of the set of virtual threads using two fingers of the user. (Item 289) The method according to item 286, further comprising transmitting data associated with a first virtual element associated with a first virtual thread to a second virtual element associated with a second virtual thread based at least in part on a hand gesture. (Item 290) The method according to item 289, wherein the other hand gesture includes a squeezing movement of the first virtual element associated with the first virtual thread using the second virtual element associated with the second virtual thread, the squeezing movement being performed by a finger of the user. (Item 291) The method according to item 286, wherein the virtual interface is displayed in relation to at least one physical object. (Item 292) The method according to item 291, wherein the at least one physical object comprises a wall. (Item 293) The method according to item 286, further comprising determining a location of a hand gesture, wherein the virtual thread originates from the determined location. (Item 294) The method according to item 286, wherein the virtual user interface is created based at least in part on a world - centered reference frame such that the virtual user interface remains stationary with respect to any movement of the user. (Item 295) Selecting a virtual user interface element among the at least one selectable virtual interface elements based at least in part on another hand gesture of the user; Generating virtual content associated with the selected user interface element at another location of the physical object The method according to item 286, further comprising the above. (Item 296) A method of creating a virtual user interface, comprising: Identifying a user interface to be presented to the user based on a hand gesture of the user; Displaying a virtual user interface in response to the hand gesture Including The virtual user interface resembles a virtual spider web, and the user pulls a virtual thread of the virtual spider web to bring the virtual interface closer to the user. (Item 297) The method according to item 296, wherein the hand gesture includes a pulling motion using a clenched fist of the user's hand. (Item 298) The method according to item 297, wherein the virtual thread of the virtual spider web comprises at least one selectable virtual element. (Item 299) Further comprising selecting the at least one selectable virtual interface element based at least in part on a gesture of the other hand, the gesture of the other hand including a movement of pulling at least one of the virtual threads toward the user, the method according to item 298. (Item 300) Further comprising transmitting data associated with a first virtual element associated with a first virtual thread to a second virtual element associated with a second virtual thread based at least in part on a hand gesture, the method according to item 296. (Item 301) The gesture of the other hand includes a squeezing movement of the first virtual element associated with the first virtual thread using the second virtual element associated with the second virtual thread, the squeezing movement being performed by a finger of the user, the method according to item 300. (Item 302) The virtual interface is displayed in relation to at least one physical object, the method according to item 296. (Item 303) The at least one physical object comprises a wall, the method according to item 302. (Item 304) Further comprising determining a location of a hand gesture, the virtual spider web originating from the determined location, the method according to item 296. (Item 305) The virtual user interface is created based at least in part on a world-centered reference frame such that the virtual user interface remains stationary with respect to any movement of the user, the method according to item 296. (Item 306) Selecting a virtual user interface element among the at least one selectable virtual interface element based at least in part on a gesture of another hand of the user; Generating virtual content associated with the selected user interface element at another location of the physical object The method according to item 296, further comprising (Item 307) The method according to items 1-306, implemented as a system having means for implementing said method steps. (Item 308) The method according to items 1-306, implemented as a computer program product comprising a computer-usable storage medium having executable code for executing said method steps.

Brief Description of the Drawings

[0205] The drawings illustrate the design and usability of various embodiments of the present invention. Note that the figures are not drawn to scale and elements of similar structure or function are represented by like reference numerals throughout the figures. In order to gain a deeper understanding of the foregoing and other advantages and objects of various embodiments of the present invention, the manner of implementing the invention briefly described above will be given by reference to the specific embodiments illustrated in the accompanying drawings. It is understood that these drawings depict only typical embodiments of the present invention and are not to be considered as limiting of its scope, and that the present invention will be described and explained with additional specificity and detail through the use of the accompanying drawings.

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 4C

Figure 4D

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 5E

Figure 5F

Figure 5G

Figure 5H

Figure 6A

Figure 6B

Figure 7A

Figure 7B

Figure 8A

Figure 8B

Figure 8C

Figure 8D

Figure 8E

Figure 8F

Figure 8G

Figure 8H

Figure 8I

Figure 8J

Figure 8K

Figure 8L

Figure 8M

Figure 8N

Figure 8O

Figure 8P

Figure 8Q

Figure 9

Figure 10

Figure 11A

Figure 11B

Figure 11C

Figure 12A

Figure 12B

Figure 13A

Figure 13B

Figure 13C

Figure 13D

Figure 13E

Figure 13F

Figure 13G

Figure 13H

Figure 13I

Figure 13J

Figure 13K

Figure 13L

Figure 13M

Figure 14A

Figure 14B

Figure 14C

Figure 14D

Figure 14E

Figure 14F

Figure 15A

Figure 15B

Figure 15C

Figure 16A

Figure 16B

Figure 16C

Figure 16D

Figure 16E

Figure 16F

Figure 16G

Figure 16H

Figure 16I

Figure 16J

Figure 16K

Figure 16L

Figure 16M

Figure 16N

Figure 16O

Figure 17

Figure 18

Figure 19A

Figure 19B

Figure 19C

Figure 19D

Figure 19E

Figure 19F

Figure 19G

Figure 19H

Figure 19I

Figure 19J

Figure 19K

Figure 19L

Figure 19M

Figure 19N

Figure 20A

Figure 20B

Figure 20C

Figure 20D

Figure 20E

Figure 20F

Figure 20G

Figure 20H

Figure 20I

Figure 20J

Figure 20K

Figure 20L

Figure 20M

Figure 20N

Figure 20O

Figure 21A

Figure 21B

Figure 21C

Figure 21D

Figure 21E

Figure 21F

Figure 21G

Figure 21H

Figure 21I

Figure 21J

Figure 21K

Figure 21L

Figure 21M

Figure 21N

Figure 22A

Figure 22B

Figure 22C

Figure 22D

Figure 22E

Figure 22F

Figure 22G

Figure 22H

Figure 22I

Figure 22J

Figure 22K

Figure 22L

Figure 22M

Figure 22N

Figure 22O

Figure 22P

Figure 22Q

Figure 22R

Figure 22S

Figure 22T

Figure 22U

Figure 22V

Figure 22W

Figure 22X

Figure 22Y

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Figure 43

Figure 44

Figure 45

Figure 46

Figure 47A

Figure 47B

Figure 48

Figure 49

Figure 50

Figure 51

Figure 52

Figure 53

Figure 54

Figure 55

Figure 56

Figure 57A

Figure 57B

Figure 58

Figure 59

Figure 60

Figure 61

Figure 62

Figure 63

Figure 64

Figure 65

Figure 66

Figure 67

Figure 68

Figure 69

Figure 70

Figure 71

Figure 72A

Figure 72B

Figure 73

Figure 74A

Figure 74B

Figure 75

Figure 76

Figure 77

Figure 78A

Figure 78B

Figure 79A

Figure 79B

Figure 79C

Figure 79D

Figure 79E

Figure 80A

Figure 80B

Figure 80C

Figure 81

Figure 82

Figure 83A

Figure 83B

Figure 84A

Figure 84B

Figure 84C

Figure 85A

Figure 85B

Figure 85C

Figure 86A

Figure 86B

Figure 86C

Figure 87A

Figure 87B

Figure 87C

Figure 88A

Figure 88B

Figure 88C

Figure 89A

Figure 89B

Figure 89C

Figure 89D

Figure 89E

Figure 89F

Figure 89G

Figure 89H

Figure 89I

Figure 89J

Figure 90

Figure 91A

Figure 91B

Figure 91C

Figure 91D

Figure 91E

Figure 91F

Figure 92

Figure 93A

Figure 93B

Figure 93C

Figure 93D

Figure 93E

Figure 93F

Figure 93G

Figure 93H

Figure 93I

Figure 93J

Figure 93K

Figure 93L

Figure 94

Figure 95A

Figure 95B

Figure 95C

Figure 95D

Figure 95E

Figure 96A

Figure 96B

Figure 96C

Figure 96D

Figure 97A

Figure 97B

Figure 97C

Figure 97D

Figure 97E

Figure 97F

Figure 97G

Figure 97H

Figure 98A

Figure 98B

Figure 98C

Figure 98D

Figure 99

Figure 100

Figure 101

Figure 102

Figure 103

Figure 104

Figure 105A

Figure 105B

Figure 105C

Figure 106A

Figure 106B

Figure 106C

Figure 107A

Figure 107B

Figure 107C

Figure 107D

Figure 108A

Figure 108B

Figure 108C

Figure 109A

Figure 109B

Figure 109C

Figure 110A

Figure 110B

Figure 111

Figure 112

Figure 113

Figure 114

Figure 115

Figure 116

Figure 117

Figure 118

Figure 119

Figure 120

Figure 121

Figure 122

Figure 123A

Figure 123B

Figure 123C

Figure 123D

Figure 124A

Figure 124B

Figure 124C

Figure 124D

Figure 124E

Figure 124F

Figure 124G

Figure 124H

Figure 124I

Figure 124J

Figure 124K

Figure 124L

Figure 124M

Figure 125A

Figure 125B

Figure 125C

Figure 125D

Figure 125E

Figure 125F

Figure 125G

Figure 125H

Figure 125I

Figure 125J

Figure 125K

Figure 125L

Figure 125M

Figure 126A

Figure 126B

Figure 126C

Figure 126D

Figure 126E

Figure 126F

Figure 126G

Figure 126H

Figure 126I

Figure 126J

Figure 126K

Figure 126L

Figure 127A

Figure 127B

Figure 127C

Figure 127D

Figure 127E

Figure 128A

Figure 128B

Figure 128C

Figure 128D

Figure 128E

Figure 128F

Figure 128G

Figure 128H

Figure 128I

Figure 128J

Figure 128K

Figure 128L

Figure 128M

Figure 128N

Figure 128O

Figure 128P

Figure 129A

Figure 129B

Figure 129C

Figure 129D

Figure 129E

Figure 129F

Figure 129G

Figure 129H

Figure 129I

Figure 129J

Figure 129K

Figure 129L

Figure 129M

Figure 130A

Figure 130B

Figure 130C

Figure 130D

Figure 130E

Figure 130F

Figure 130G

Figure 130H

Figure 130I

Figure 131A

Figure 131B

Figure 131C

Figure 131D

Figure 131E

Figure 131F

Figure 131G

Figure 131H

Figure 131I

Figure 132

Figure 133

Figure 134

Figure 135A

Figure 135B

Figure 135C

Figure 135D

Figure 135E

Figure 135F

Figure 135G

Figure 135H

Figure 135I

Figure 135J

Figure 136

Figure 137

Figure 138

Figure 139

Figure 140

Figure 141

Figure 142

Figure 143

Figure 144

Figure 145

[0206] Here, various embodiments will be described in detail with reference to the drawings, which are provided as illustrative examples of the present invention to enable those skilled in the art to practice the present invention. It should be noted that the following figures and examples are not meant to limit the scope of the present invention. If an element of the present invention can be partially or fully implemented using known components (or methods or processes), only those parts of such known components (or methods or processes) necessary for understanding the present invention will be described, and detailed descriptions of the other parts of such known components (or methods or processes) will be omitted so as not to obscure the present invention. Further, various embodiments include present and future known equivalents of the components referred to herein by way of illustration.

[0207] In the foregoing specification, the present invention has been described with reference to its specific embodiments. However, it will be apparent that various modifications and changes can be made thereto without departing from the broader spirit and scope of the present invention. For example, the foregoing process flow is described with reference to a particular order of process actions. However, many of the orders of the process actions described may be changed without affecting the scope or operation of the present invention. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a limiting sense.

[0208] Disclosed are methods and systems for generating virtual and / or augmented reality. In order to provide a realistic and enjoyable virtual reality (VR) or augmented reality (AR) experience, virtual content can be strategically delivered to a user's eyes in a manner that takes into account the physiology and limitations of the human eye. The following disclosure will provide various embodiments of such optical systems that can be integrated within an AR system. Most of the present disclosure herein will be discussed in the context of an AR system, but the same techniques may also be used for a VR system, and it should be understood that the following embodiments are not to be read as limiting.

[0209] The following disclosure will provide details regarding various types of systems through which AR users can interact with each other through the creation of a map that provides real-time comprehensive information about physical objects in the real world. The map can advantageously be considered for projecting virtual images related to known real objects. The following disclosure will provide various approaches for understanding information about the real world and using this information to provide a more realistic and enjoyable AR experience. In addition, the present disclosure will provide various user scenarios and applications in which an AR system such as that described herein can be implemented.

[0210] (System Overview) In one or more embodiments, the AR system 10 comprises a computing network 5 consisting of one or more computer servers 11 connected through one or more high-bandwidth interfaces 15. The servers 11 within the computing network may or may not be located in the same place. Each of the one or more servers 11 comprises one or more processors for executing program instructions. The server may also include a memory for storing program instructions and data used and / or generated by a process implemented by the server 11 under the direction of the program instructions.

[0211] The computing network 5 communicates data between the servers 11 and between the servers and one or more user devices 12 via one or more data network connections 13. Examples of such data networks include, but are not limited to, any type of public and private data networks, both mobile and wired, including many interconnections of such networks, commonly referred to as the Internet in general. It is not intended that a particular medium, topology, or protocol be implied by the figures.

[0212] The user device is configured to communicate directly with either the computing network 5 or the server 11. Alternatively, the user device 12 communicates locally through a specially programmed local gateway 14 for processing and / or communicating data between the remote server 11, optionally with other user devices, and the network 5 and one or more local user devices 12.

[0213] As shown, the gateway 14 is implemented as a separate hardware component that includes a processor for executing software instructions and a memory for storing software instructions and data. The gateway has its own wired and / or wireless connection to the data network to communicate with the server 11 with the computing network 5. Alternatively, the gateway 14 can be integrated with the user device 12, which is worn or carried by the user. For example, the gateway 14 may be implemented as a downloadable software application that is installed and launched on a processor included within the user device 12. In one embodiment, the gateway 14 provides one or more user accesses to the computing network 5 via the data network 13.

[0214] Each of the servers 11 includes, for example, a working memory and storage for storing data and software programs, a microprocessor for executing program instructions, and a graphics processor and other special processors for rendering and generating graphics, images, videos, audio, and multimedia files. The computing network 5 may also include a device for storing data accessed, used, or created by the server 11.

[0215] A software program that runs on a server and optionally on user device 12 and gateway 14 is used to generate a digital world (also referred to herein as a virtual world) in which a user interacts with user device 12. The digital world (or map), as will be described in more detail below, is represented by data and processes that describe and / or define virtual non-existent entities, environments, and conditions that can be presented to the user through user device 12 for the user to experience and interact with. For example, when instantiated within a scene viewed or experienced by the user, some types of objects, entities, or items that would appear to physically exist may include descriptions of their appearance, their behavior, the ways in which the user is permitted to interact with them, and other characteristics.

[0216] The data used to create the environment of the virtual world (including virtual objects) may include, for example, atmospheric data, terrain data, weather data, temperature data, location data, and other data used to define and / or describe the virtual environment. In addition, the data that defines various conditions that control the operation of the virtual world may include, for example, the laws of physics, time, spatial relationships, and other data that can be used to define and / or create the various conditions that control the operation of the virtual world (including virtual objects).

[0217] Unless otherwise indicated by context, entities, objects, conditions, properties, behaviors, or other features in the digital world are generally referred to herein as objects (e.g., digital objects, virtual objects, rendered physical objects, etc.). An object may be any type of living or non-living object, including but not limited to buildings, plants, vehicles, people, animals, creations, machines, data, videos, text, photos, and other users. An object may also be defined within the digital world to store information about items, behaviors, or conditions that actually exist in the physical world. Data that describes or defines an entity, object, or item, or stores its current state, is generally referred to herein as object data. This data is processed by server 11, or depending on the implementation, by gateway 14 or user device 12, to instantiate an instance of the object and render the object in an appropriate manner for a user to experience through the user device.

[0218] Programmers who develop and / or oversee the digital world create or define objects and the conditions under which they are instantiated. However, the digital world can be made accessible to others to enable them to create or modify objects. Once an object is instantiated, its state may be permitted to be altered, controlled, or manipulated by one or more users experiencing the digital world.

[0219] For example, in one embodiment, the development, production, and management of the digital world are generally provided by one or more system administrators. In some embodiments, this may include the development, design, and / or execution of the storyline, themes, and events within the digital world, as well as the distribution of the storyline through various forms of events and media such as movies, digital, network, mobile, augmented reality, and live entertainment. The system administrator may also handle the technical management, coordination, and supervision of the digital world and the associated user community, as well as other tasks typically performed by a network administrator.

[0220] Users generally interact with one or more digital worlds using some type of local computing device designated as user device 12. Examples of such user devices include, but are not limited to, smartphones, tablet devices, head-mounted displays (HMDs), gaming consoles, or any other device capable of communicating data and providing an interface or display to the user, as well as combinations of such devices. In some embodiments, user device 12 may include or communicate with local peripheral devices or input / output components such as, for example, a keyboard, mouse, joystick, game controller, touch interface device, motion capture controller, optical tracking device, audio device, voice device, projector system, 3D display, and / or holographic 3D contact lens.

[0221] An example of user device 12 for interacting with system 10 is illustrated in FIG. 2. In the exemplary embodiment shown in FIG. 2, user 21 may interface with one or more digital worlds through smartphone 22. The gateway is implemented by software application 23 stored on and launched on smartphone 22. In this particular example, data network 13 includes a wireless mobile network that connects the user device (e.g., smartphone 22) to computer network 5.

[0222] In one implementation of the preferred embodiment, system 10 can support a number of simultaneous users (e.g., millions of users) each interfacing with the same digital world or multiple digital worlds using several types of user devices 12.

[0223] The user device provides an interface for the user to enable visual, auditory, and / or physical interaction between the user and the digital world generated by server 11, including other users and objects (real or virtual) presented to the user. The interface provides the user with a rendered scene that can be visually recognized, listened to, or otherwise sensed, and the ability to interact with the scene in real time. The manner in which the user interacts with the rendered scene can be determined by the capabilities of the user device. For example, if the user device is a smartphone, the user interaction may be implemented by the user touching the touch screen. In another example, if the user device is a computer or a game console, the user interaction may be implemented using a keyboard or a game controller. The user device may include additional components that enable user interaction, such as sensors, and the objects and information (including gestures) detected by the sensors may be provided as input representing the user interaction with the virtual world using the user device.

[0224] The rendered scene can be presented in various formats such as, for example, 2D or 3D visual displays (including projections), sound, and haptic or tactile feedback. The rendered scene may be interfaced by a user in one or more modes including, for example, augmented reality, virtual reality, and combinations thereof. The format of the scene to be rendered as well as the interface mode may be determined by one or more of a user device, data processing capabilities, user device connection, network capacity, and system workload. The ability to interact with the digital world and the real-time nature of data exchange for a number of users simultaneously are enabled by a computing network 5, a server 11, a gateway component 14 (optional), and a user device 12.

[0225] In one embodiment, the computing network 5 consists of a large-scale computing system having single and / or multi-core servers (such as server 11) connected through high-speed connections (such as high-bandwidth interface 15). The computing network 5 may form a cloud or grid network. Each of the servers is coupled with a computer-readable memory containing memory or storing software for implementing data for creating, designing, modifying, or processing objects of the digital world. These objects and their instantiations are dynamic, may appear or disappear, change over time, and change in response to other conditions. Examples of the dynamic capabilities of the objects are generally discussed herein with respect to various embodiments. In some embodiments, each user interfacing with the system 10 may also be represented within one or more digital worlds as an object and / or a collection of objects.

[0226] Server 11 within computing network 5 also stores computing state data for each digital world. The computing state data (also referred to herein as state data) can be a component of object data and generally defines the state of an instance of an object at a given time instance. Thus, the computing state data can change over time and can be affected by the actions of one or more users and / or programmers maintaining system 10. As the user affects the computing state data (or other data comprising the digital world), the user directly modifies or otherwise manipulates the digital world. When the digital world is shared with or interfaced by other users, the user's actions can affect what is experienced by other users interacting with the digital world. Thus, in some embodiments, changes to the digital world made by a user will also be experienced by other users interfacing with system 10.

[0227] The data stored in one or more servers 11 within computing network 5 is, in one embodiment, transmitted or deployed to one or more user devices 12 and / or gateway components 14 quickly and with low latency. In one embodiment, the object data shared by the server may be complete or compressed and include instructions for reconstructing the complete object data on the user side and may be rendered and visualized by the user's local computing device (e.g., gateway 14 and / or user device 12). The software running on server 11 of computing network 5 may, in some embodiments, adapt the data it generates and sends to a particular user's device 12 as a function of the user's specific device and bandwidth for objects (or any other data) within the digital world exchanged by computing network 5.

[0228] For example, when a user interacts with the digital world or map through user device 12, server 11 recognizes the specific type of device being used by the user, the device connectivity, and / or the bandwidth available between the user device and the server, and may appropriately size and balance the data being sent to the device to optimize the user interaction. As an example of this, the size of the data being transmitted may be reduced to low-resolution quality so that the data can be displayed on a specific user device having a low-resolution display. In a preferred embodiment, computing network 5 and / or gateway component 14 delivers data to user device 12 at a rate sufficient to present an interface that operates at 15 frames per second or greater and at a resolution or greater than high-definition quality.

[0229] Gateway 14 provides a local connection to computing network 5 for one or more users. In some embodiments, it may be implemented by a downloadable software application that launches on user device 12 or another local device such as that shown in FIG. 2. In other embodiments, it may be implemented by a hardware component (accompanied by appropriate software / firmware stored on the component, the component having a processor) that communicates with user device 12 but is either not incorporated with it, or not taken in by it, or is either incorporated with user device 12. Gateway 14 communicates with computing network 5 via data network 13 and provides for data exchange between computing network 5 and one or more local user devices 12. As will be discussed in more detail below, gateway component 14 may include software, firmware, memory, and processing circuitry and may be capable of processing data communicated between network 5 and one or more local user devices 12.

[0230] In some embodiments, the gateway component 14 monitors and adjusts the rate of data exchanged between the user device 12 and the computer network 5, enabling optimal data processing capabilities for a particular user device 12. For example, in some embodiments, the gateway 14 buffers and downloads both static and dynamic aspects of the digital world, even those that extend beyond the view presented to the user through the interface connected to the user device. In such embodiments, instances of static objects (structured data, software implementation methods, or both) may be stored in memory (local to the gateway component 14, the user device 12, or both) and referenced against the current location of the local user as indicated by the data provided by the computing network 5 and / or the user's device 12.

[0231] For example, instances of dynamic objects, which may include intelligent software agents and objects controlled by other users and / or the local user, are stored in a high-speed memory buffer. Dynamic objects representing two-dimensional or three-dimensional objects within the scene presented to the user can be divided into component shapes such as static shapes that move but do not change and dynamic shapes that change. The changing portions of the dynamic objects can be updated by a real-time threaded high-priority data stream from the server 11 through the computing network 5 managed by the gateway component 14.

[0232] As an example of a prioritized threaded data stream, data within the 60-degree field of view of the user's eye can be given a higher priority than more peripheral data. Another example is to prioritize dynamic characters and / or objects within the user's field of view over static objects in the background.

[0233] In addition to managing the data connection between the computing network 5 and the user device 12, the gateway component 14 may store and / or process data that can be presented to the user device 12. For example, in some embodiments, the gateway component 14 may receive, from the computing network 5, compressed data that describes graphical objects to be rendered, for example, for viewing by a user, and may perform advanced rendering techniques to reduce the data load transmitted from the computing network 5 to the user device 12. In another example where the gateway 14 is a separate device, the gateway 14 may store and / or process data for local instances of objects, rather than transmitting the data to the computing network 5 for processing.

[0234] Referring now to FIG. 3, the virtual world may be achieved by one or more users in a variety of formats that may depend on the capabilities of the user's device. In some embodiments, the user device 12 may include, for example, a smartphone, a tablet device, a head-mounted display (HMD), a gaming console, or a wearable device. Generally, the user device will include a processor for executing program code stored in memory on the device, coupled to a display, and a communication interface.

[0235] Exemplary embodiments of the user device are illustrated in FIG. 3, and the user device includes a mobile wearable device, namely, a head-mounted display system 30. According to certain embodiments of the present disclosure, the head-mounted display system 30 includes a user interface 37, a user sensing system 34, an environmental sensing system 36, and a processor 38. Although the processor 38 is shown in FIG. 3 as a separate and isolated component from the head-mounted system 30, in alternative embodiments, the processor 38 may be integrated with one or more components of the head-mounted system 30, or may be integrated within other system 10 components, such as gateway 14, as shown, for example, in FIGS. 1 and 2.

[0236] The user device 30 presents an interface 37 to the user for interacting with and experiencing the digital world. Such interactions can involve the user and the digital world, one or more other users interfacing with the system 10, and objects within the digital world. The interface 37 generally provides the user with image and / or audio sensory inputs (and in some embodiments, physical sensory inputs). Accordingly, the interface 37 may include speakers (not shown) and, in some embodiments, a display component 33 capable of providing stereoscopic 3D viewing and / or 3D viewing that embodies more natural characteristics of the human visual system.

[0237] In some embodiments, the display component 33 may comprise a transparent interface (such as a clear OLED) that, when in the "off" setting, allows an optically correct view of the physical environment around the user with little or no optical distortion or computing overlay. As discussed in more detail below, the interface 37 may include additional settings that enable various visual / interface performance and functionality.

[0238] In some embodiments, the user perception system 34 may include one or more sensors 31 operable to detect certain features, characteristics, or information related to the individual users wearing the system 30. For example, in some embodiments, the sensor 31 may include a camera or optical detection / scanning circuitry capable of detecting the real-time optical characteristics / measurements of the user.

[0239] The real-time optical characteristics / measurements of the user may be, for example, one or more of pupil constriction / dilation, angular measurement / positioning of each pupil, sphericity, eye shape (as the eye shape changes over time), and other anatomical data. This data may be used by the head-mounted system 30 and / or the interface system 10 to optimize the user's viewing experience, provide information (e.g., the user's visual focus), or be used to calculate it. For example, in one embodiment, each of the sensors 31 may measure the pupil constriction rate for each eye of the user. This data may be transmitted to the processor 38 (or the gateway component 14 or the server 11), and the data may be used, for example, to determine the user's reaction to the brightness setting of the interface display 33.

[0240] The interface 37 may be adjusted according to the user's reaction, for example, by dimming the display 33 if the user's reaction indicates that the brightness level of the display 33 is too high. The user perception system 34 may include other components other than those described above or illustrated in FIG. 3. For example, in some embodiments, the user perception system 34 may include a microphone for receiving voice input from the user. The user perception system 34 may also include one or more infrared camera sensors, one or more visible spectrum camera sensors, structured light emitters and / or sensors, infrared light emitters, coherent light emitters and / or sensors, gyroscopes, accelerometers, magnetometers, proximity sensors, GPS sensors, ultrasonic emitters and detectors, and tactile interfaces.

[0241] The environmental perception system 36 includes one or more sensors 32 for obtaining data from the physical environment around the user. The objects or information detected by the sensors may be provided as input to the user device. In some embodiments, this input may represent user interaction with the virtual world. For example, a user viewing a virtual keyboard on an airplane may gesture with a finger as if typing on the virtual keyboard. The movement of the finger may be captured by the sensor 32 and provided as input to the user device or system, and the input may be used to change the virtual world or create a new virtual object.

[0242] For example, the movement of the finger may be recognized as typing (e.g., using a software program of a processor, etc.), and the recognized typing gesture may be combined with the known locations of the virtual keys on the virtual keyboard. The system may then render a virtual monitor to be displayed to the user (or other users interfacing with the system), and the virtual monitor displays the text being typed by the user.

[0243] The sensor 32 may include, for example, a generally outward-facing camera or scanner for interpreting scene information through structured light of infrared rays projected continuously and / or intermittently. The environmental perception system (36) may be used to map one or more elements of the physical environment around the user by detecting and aligning the local environment, including static objects, dynamic objects, people, gestures, and various lighting, atmospheric, and acoustic conditions. Thus, in some embodiments, the environmental perception system (36) is embedded within a local computing system (e.g., the gateway component 14 or the processor 38) and may include image-based 3D reconstruction software operable to digitally reconstruct one or more objects or information detected by the sensor 32.

[0244] In one exemplary embodiment, the environmental perception system 36 provides one or more of motion capture data (including gesture recognition), depth perception, face recognition, object recognition, unique object feature recognition, voice / audio recognition and processing, acoustic source localization, noise reduction, infrared or similar laser projection, and monochromatic and / or color CMOS sensors (or other similar sensors), field of view sensors, and various other optical enhancement sensors.

[0245] It should be understood that the environmental perception system 36 may include other components other than those described above or illustrated in FIG. 3. For example, in some embodiments, the environmental perception system 36 may include a microphone for receiving audio from the local environment. The user perception system (36) may also include one or more infrared camera sensors, one or more visible spectrum camera sensors, a structured light emitter and / or sensor, an infrared light emitter, a coherent light emitter and / or sensor, a gyroscope, an accelerometer, a magnetometer, a proximity sensor, a GPS sensor, an ultrasonic emitter and detector, and a haptic interface.

[0246] As described above, in some embodiments, the processor 38 may be integrated with other components of the head-mounted system 30, integrated with other components of the interface system 10, or may be a separate device (wearable or separate from the user) as shown in FIG. 3. The processor 38 may be connected to various components of the head-mounted system 30 and / or components of the interface system 10 through a physical wired connection or through a wireless connection such as, for example, a mobile network connection (including cellular phones and data networks), Wi-Fi, or Bluetooth®.

[0247] In one or more embodiments, the processor 38 may include a memory module, an integrated and / or additional graphics processing unit, a wireless and / or wired Internet connection, and a codec and / or firmware capable of converting data from a source (e.g., computing network 5, user sensing system 34, environment sensing system 36, or gateway component 14) into image and auditory data, and the image / video and audio may be presented to the user via interface 37.

[0248] In one or more embodiments, the processor 38 handles data processing for the various components of the head-mounted system 30 and data exchange between the head-mounted system 30 and the gateway component 14, and in some embodiments, the computing network 5. For example, the processor 38 may buffer and process a data stream between the user and the computing network 5 and thereby be used to enable a smooth, continuous, and high-fidelity user experience.

[0249] In some embodiments, the processor 38 may process data at a rate sufficient to achieve, such as 8 frames per second at 320×240 resolution to 24 frames per second at high-definition resolution (1280×720), or 60 to 120 frames per second and higher at 4k resolution (50,000 frames per second at 10k+ resolution) and above. Additionally, the processor 38 may store and / or process data presented to the user rather than streaming it in real time from the computing network 5.

[0250] For example, in some embodiments, the processor 38 may receive compressed data from the computing network 5 and perform advanced rendering techniques (such as lighting or shading) to reduce the data load transmitted from the computing network 5 to the user device 12. In another example, instead of transmitting data to the gateway component 14 or the computing network 5, the processor 38 may store and / or process local object data.

[0251] The head-mounted system 30 may include various settings, i.e., modes, in some embodiments to enable various visual / interface performance and functionality. The modes may be selected manually by the user or automatically by components of the head-mounted system 30 or the gateway component 14. As described above, an exemplary mode of the head-mounted system 30 includes an "off" mode, in which the interface 37 provides substantially no digital or virtual content. In the "off" mode, the display component 33 is transparent, thereby enabling an optically correct view of the physical environment around the user with little / no optical distortion or computing overlay.

[0252] In one exemplary embodiment, the head-mounted system 30 includes an "extended" mode, in which the interface 37 provides an augmented reality interface. In the extended mode, the interface display 33 is substantially transparent, thereby enabling the user to view the local physical environment. At the same time, virtual object data provided by the computing network 5, the processor 38, and / or the gateway component 14 is provided on the display 33 in combination with the physical local environment. The following sections will consider various embodiments of exemplary head-mounted user systems that may be used for virtual and augmented reality purposes.

[0253] (User System) Referring to FIGS. 4A-4D, some general component options are illustrated. In part of the detailed description that follows the discussion of FIGS. 4A-4D, various systems, subsystems, and components are presented to address the purpose of providing a high-quality and comfortably perceivable display system for human VR and / or AR.

[0254] As shown in FIG. 4A, a user 60 of a head-mounted augmented reality system (“AR system”) is depicted while wearing a frame 64 structure coupled to a display system 62 positioned in front of the user's eyes. A speaker 66 is coupled to the frame 64 and positioned adjacent to the user's outer ear canal in the depicted configuration (in one embodiment, another speaker, not shown, is positioned adjacent to the user's other outer ear canal to provide stereo / formable sound control). The display 62 can be mounted in various configurations, such as fixed to the frame 64 by a wired conductor or wireless connection, fixed to a helmet or hat 80 as shown in the embodiment of FIG. 4B, embedded within headphones, fixed to the user's torso 82 removably (e.g., installed within a backpack (not shown)) in a certain configuration as shown in the embodiment of FIG. 4C, or removably attached to the user's waist 84 in a belt-coupled configuration as shown in the embodiment of FIG. 4D, and is operably coupled to a local processing and data module 70.

[0255] The local processing and data module 70 may comprise a power-saving processor or controller and digital memory such as flash memory, both of which may be used to assist in the processing, caching, and storing of data captured from sensors (such as an image capture device (e.g., a camera), microphone, inertial measurement unit, accelerometer, compass, GPS unit, wireless device, and / or gyroscope, etc.) that can be operably coupled to frame 64 and / or (b) data that may potentially be acquired and / or processed using the remote processing module 72 and / or remote data repository 74 for passage to the display 62 after processing or reading.

[0256] The local processing and data module 70 may be operably coupled to the remote processing module 72 and the remote data repository 74 via a wired or wireless communication link etc. (76, 78) such that these remote modules (72, 74) are operably coupled to each other and available as resources to the local processing and data module 70. The processing module 70 may perform one or more computing tasks including reading data from a memory or one or more databases (e.g., a cloud-based server) to control the optical and other systems of the AR system and provide virtual content to the user.

[0257] In one embodiment, the remote processing module 72 may comprise one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. In one embodiment, the remote data repository 74 may comprise a relatively large-scale digital data storage facility that may be available through the Internet or other networking configuration in a "cloud" resource configuration. In one embodiment, all data is stored and all computations are performed within the local processing and data module, allowing for fully autonomous use from any remote module.

[0258] (Optical embodiment) It should be understood that there are many approaches to presenting 3D virtual content to a user's eyes through the optical elements of a head-mounted user device. The following exemplary embodiments may be used in combination with other approaches and should not be read in a limiting sense. The following exemplary embodiments represent several example optical systems that may be integrated with a head-mounted user device (30) to enable a user to view virtual content in a comfortable and easily focusable manner.

[0259] Referring to FIGS. 5A through 22Y, various display configurations are presented that are designed to present a photon-based radiation pattern to a human eye that can be comfortably perceived as an augmentation to physical reality, with a high level of image quality and three-dimensional perception, and that can also present two-dimensional content.

[0260] Referring to FIG. 5A, in a simplified example, a transmissive beam splitter substrate 104 with a 45-degree reflective surface 102 directs incident radiation 106 that can be output from a lens (not shown) through the pupil 45 of the eye 58 to the retina 54. The field of view for such a system is limited by the geometry of the beam splitter 104. To meet the desire to obtain comfortable viewing with minimal hardware, in one embodiment, a larger field of view can be created by aggregating the outputs / reflections of various different reflective and / or diffractive surfaces. This can be achieved, for example, using a frame sequential configuration such that a sequence of frames is presented to the eye 58 at a high frequency, providing the perception of a single coherent scene.

[0261] As an alternative to, or in addition to, presenting different image data in a time-sequential manner through different reflectors, the reflectors may separate the content by other means such as polarization selectivity or wavelength selectivity. In addition to being able to relay two-dimensional images, the reflectors may also relay three-dimensional wavefronts associated with a true three-dimensional view of an actual physical object.

[0262] Referring to FIG. 5B, a substrate 108 is shown having a plurality of reflectors at a plurality of angles 110, each reflector being actively reflective in the depicted configuration for illustrative purposes. The reflectors may comprise switchable elements to facilitate temporal selectivity. In one embodiment, the reflective surfaces may be intentionally and sequentially activated with frame sequential input information 106, each reflective surface presenting a narrow field of view sub-image that is tiled with other narrow field of view sub-images presented by other reflective surfaces to form a composite wide field of view image.

[0263] For example, referring to FIGS. 5C, 5D, and 5E, a surface 110 (e.g., at the center of substrate 108) reflects the incident image information 106 and is switched to the reflective state “on” to represent a relatively narrow field of view sub-image at the center of a larger field of view, while other potential reflective surfaces are in a transmissive state.

[0264] Referring to FIG. 5C, incident image information 106 (as indicated by the angle of the incident beam 106 on substrate 108 at input interface 112 and the angle at which it exits substrate 108) originating from the right side of the narrow field of view sub-image is reflected from the reflective surface 110 towards the eye 58. FIG. 5D illustrates that the same reflector 110 is active, and the image information originates from the center of the narrow field of view sub-image as indicated by the angle of the input information 106 at input interface 112 and its angle when exiting substrate 108.

[0265] FIG. 5E illustrates that the same reflector 110 is active, and the image information originates from the left side of the field of view as indicated by the angle of the input information 106 at input interface 112 and the exit angle obtained at the surface of substrate 108. FIG. 5F illustrates a configuration where the bottom reflector 110 is active and the image information 106 originates from the far right side of the overall field of view. For example, FIGS. 5C, 5D, and 5E may illustrate one frame representing the center of a frame sequential tiled display image, and FIG. 5F may illustrate a second frame representing the far right side of that tiled display image.

[0266] In one embodiment, the light that conveys the image information 106 may first impinge on the reflective surface 110 after directly entering the substrate 108 at the input interface 112 without being reflected from the surface of the substrate 108. In one embodiment, the light that conveys the image information 106 may be reflected from one or more surfaces of the substrate 108 after entering the input interface 112 and before impinging on the reflective surface 110. For example, the substrate 108 may act as a planar waveguide and convey the light that conveys the image information 106 by total internal reflection. The light may also be reflected from one or more surfaces of the substrate 108 through a partial reflection coating, a wavelength selection coating, an angle selection coating, and / or a polarization selection coating.

[0267] In one embodiment, the angled reflector may be constructed using an electroactive material such that, in response to the application of a voltage and / or current to a particular reflector, the refractive index of the material comprising such reflector changes from a refractive index that substantially matches that of the remainder of the substrate 108. When the refractive index of the reflector matches that of the remainder of the substrate 108, the reflector is in a transmissive configuration. When the refractive index of the reflector is mismatched with that of the remainder of the substrate 108, the reflector is in a reflective configuration such that a reflection effect is created. Exemplary electroactive materials include lithium niobate and electroactive polymers. Suitable substantially transparent electrodes for controlling a plurality of such reflectors may include materials such as indium tin oxide, which are utilized in liquid crystal displays.

[0268] In one embodiment, the electroactive reflector 110 may comprise liquid crystal embedded in a host medium of a substrate 108 such as glass or plastic. In some variations, a liquid crystal may be selected that changes the refractive index as a function of an applied electrical signal so that a more analog change can be effected, as opposed to binary (from one transmissive state to one reflective state). In one embodiment, with an input display that can refresh at a rate of about 360 Hz, it is desirable to have an electroactive reflector array that can maintain such a frequency, with six sub-images presented to the eye in a frame sequential manner to form a large tiled display image at an overall refresh rate of 60 frames per second.

[0269] In one embodiment, lithium niobate may be utilized as the electroactive reflective material, as opposed to liquid crystal. Lithium niobate is utilized in the photonics industry for high-speed switches and optical fiber networks and has the ability to switch the refractive index at ultra-high frequencies in response to an applied voltage. This high frequency may be used, in particular, to steer line sequential or pixel sequential sub-image information when the input display is a scanned light display such as a fiber-scanned display or a scanning mirror-based display.

[0270] In another embodiment, a variable-switchable angled mirror configuration may comprise one or more rapidly mechanically repositionable reflective surfaces such as those of a MEMS (Micro-Electro-Mechanical System) device. MEMS devices may include those known as "digital mirror devices", i.e., "DMDs", (part of "digital light processing", i.e., "DLP" systems, such as those available from Texas Instruments, Inc. in many cases). In another electromechanical embodiment, a plurality of air-spaced (or in a vacuum) reflective surfaces may be mechanically moved in and out of position at high frequencies. In another electromechanical embodiment, a single reflective surface may be moved up and down and re-pitched at ultra-high frequencies.

[0271] Referring to FIG. 5G, it should be noted that the switchable variable angle reflector configuration described herein can pass collimated or flat wavefront information not only to the retina 54 of the eye 58, but also, as shown in the illustration of FIG. 5G, image information of the curved wavefront 122. This generally does not apply to other waveguide-based configurations where total internal reflection of the curved wavefront information causes undesirable complexity and thus the input generally must be collimated. The ability to pass curved wavefront information facilitates the ability of configurations such as those shown in FIGS. 5B - 5H to provide inputs that are perceived as focused at various distances from the eye 58, not just at optical infinity (which would be the interpretation of collimated light in the absence of other cues) to the retina 54.

[0272] Referring to FIG. 5H, in another embodiment, an array of static partial reflection surfaces 116 (e.g., always in reflection mode; in another embodiment, may be electroactive as described above) may be embedded within the substrate 114, along with a high-frequency gating layer 118 that controls the output of the eye 58. The high-frequency gating layer 118 may be controllably movable and may allow transmission only through the aperture 120. In other words, all transmission except through the aperture 120 may be selectively blocked. The gating layer 118 may comprise an array of liquid crystals, an array of lithium niobate, an array of MEMS shutter elements, an array of DLP DMD elements, or an array of other MEMS devices configured to switch to a transmission mode and pass or transmit at relatively high frequencies and high transmittance in response thereto.

[0273] Referring to FIGS. 6A-6B, other embodiments are depicted in which an array of optical elements is combined with an exit pupil expansion configuration to assist in the comfort of a user's virtual or augmented reality experience. The larger the "exit pupil" for an optical configuration, the larger the acceptable area exists such that due to the larger exit pupil of the system, the user's anatomical pupil can still be positioned to receive information from the display system as desired, and thus the positioning of the user's eyes relative to the display (which can be attached to the user's head in a glasses-like configuration as in FIGS. 4A-4D) is less likely to interfere with that experience. In other words, the larger the exit pupil, the less sensitive the system is to some misalignment of the display relative to the user's anatomical pupil, and fewer geometric constraints are imposed on its relationship with the display / glasses, and thus greater comfort can be achieved for the user.

[0274] Referring now to FIGS. 6A and 6B, an alternative approach is illustrated. As shown in FIG. 6A, the left display 140 supplies a collection of parallel light rays into the substrate 124. In one embodiment, the display may be a scanning fiber optic display that projects an image through a lens or other optical element 142 that can be utilized to reciprocally scan a narrow beam of light at the angles shown, collect the angularly scanned light, and convert it into a parallel bundle of light rays. The light rays may be reflected from a series of reflective surfaces (126, 128, 130, 132, 134, 136) that partially reflect and partially transmit the incident light such that the light is approximately equally shared as it traverses the group of reflective surfaces (126, 128, 130, 132, 134, 136). Small lenses 138 are placed at each exit point from the waveguide 124, and the exiting light rays are steered through the nodes and scanned towards the eye 58, providing an array of exit pupils or a functional equivalent of one large exit pupil that can be used by the user when viewing the display system.

[0275] For a virtual reality configuration where it is desirable for the real world 144 to also be visible through the waveguide, a similar set of lenses 139 is presented on the opposite side of the waveguide 124 to compensate for the subset of lenses and thus may create the equivalent of a zero magnification telescope. Each of the reflective surfaces (126, 128, 130, 132, 134, 136) may be aligned at approximately 45 degrees as shown or may be configured to have a different alignment (e.g., similar to the configurations of FIGS. 5B - 5H). The reflective surfaces (126, 128, 130, 132, 134, 136) may comprise wavelength selective reflectors, band - pass reflectors, beam splitters, or other reflective configurations. The lenses (138, 139) shown are refractive lenses, although diffractive lens elements may also be utilized.

[0276] Referring to FIG. 6B, a plurality of curved reflective surfaces (148, 150, 152, 154, 156, 158) effectively combine the functionality of the lenses (element 138 of FIG. 6A) and reflectors (elements 126, 128, 130, 132, 134, 136 of FIG. 6A) of the embodiment of FIG. 6A, thereby depicting a somewhat similar configuration that can be utilized to eliminate the need for two lens groups (element 138 of FIG. 6A).

[0277] The curved reflective surfaces (148, 150, 152, 154, 156, 158) may be of various curved configurations selected to perform both reflection and angular change, such as parabolic or elliptical curved surfaces. In a parabolic shape, a parallel set of incident light rays will be collected into a single output point. In an elliptical configuration, a set of light rays diverging from a single generation point is collected into a single output point. Similar to the configuration of FIG. 6A, the curved reflective surfaces (148, 150, 152, 154, 156, 158) preferably reflect partially and transmit partially such that the incident light is shared across the length of the waveguide 146. The curved reflective surfaces (148, 150, 152, 154, 156, 158) may comprise wavelength selective notch reflectors, beam splitters, or other reflective configurations. In another embodiment, the curved reflective surfaces (148, 150, 152, 154, 156, 158) may be replaced with diffractive reflectors that reflect and also deflect.

[0278] Referring to FIG. 7A, the perception of the Z-axis difference (e.g., the straight-line distance from the eye along the optical axis) can be facilitated by using a combination of a waveguide and a variable-focus optical element configuration. As shown in FIG. 7A, the image information from the display 160 is collimated, for example, using a configuration such as that described with reference to FIGS. 6A and 6B, or other substrate-guided optical methods known to those skilled in the art, and introduced into the waveguide 164, where it may be distributed in a large exit pupil pattern. Then, the variable-focus optical element capability may be utilized to change the focus of the wavefront of the light emerging from the waveguide and provide the eye with the perception that the light emerging from the waveguide 164 is from a specific focal distance.

[0279] In other words, since the incident light is collimated to avoid issues in a total internal reflection waveguide configuration, it exits in a collimated manner, requiring the viewer's eye to accommodate such that the far point is in focus on the retina and will necessarily be interpreted as being from optical infinity unless some other intervention recombines the light and causes it to be perceived as being from different viewing distances. One such suitable intervention is a variable-focus lens.

[0280] In the embodiment of FIG. 7A, the collimated image information from the display 160 is introduced into the glass 162 or other piece of material at an angle such that it undergoes total internal reflection and passes into the adjacent waveguide 164. The waveguide 164 may be configured similarly to the waveguides (124, 146, respectively) of FIGS. 6A or 6B such that the collimated light from the display exits substantially uniformly across the distribution of reflectors or diffractive features along the length of the waveguide. In response to the exit towards the eye 58, in the configuration depicted, the exiting light passes through the variable-focus lens element 166 and exits the variable-focus lens element 166 according to the controlled focus of the variable-focus lens element 166, and the light incident on the eye 58 will have various levels of focus (a collimated flat wavefront represents optical infinity, and the greater the beam divergence / wavefront curvature, the closer the viewing distance to the eye 58).

[0281] To compensate for the variable focus lens element 166 between the eye 58 and the waveguide 164, another similar variable focus lens element 167 is installed on the opposite side of the waveguide 164, and for augmented reality, it cancels out the optical effect of the lens 166 on the light originating from the world 144 (e.g., as described above, one lens compensates for the other lens and generates a functional equivalent of a zero magnification telescope).

[0282] The variable focus lens element 166 may be a liquid crystal lens, an electroactive lens, a conventional refractive lens with a movable element, a mechanically deformable based lens (such as a fluid-filled membrane lens, or a lens similar to the human crystalline lens where a flexible element is flexed and relaxed by an actuator), an electrowetting lens, or a refractive element such as a plurality of fluids with different refractive indices.

[0283] The variable focus lens element 166 may also comprise a switchable diffractive optical element (such as a polymer dispersed liquid crystal approach characterized by having microdroplets of a liquid crystal in a host medium where, when a voltage is applied, the molecules reorient so that their refractive index no longer matches that of the host medium, thereby creating a high frequency switchable diffraction pattern).

[0284] One embodiment includes a host medium in which microdroplets of a Kerr effect based electroactive material such as lithium niobate are dispersed and when combined with a scanning light display such as a fiber scanning display or a scanning mirror based display, enables refocusing of image information on a pixel-by-pixel or row-by-row basis. In the variable focus lens element 166 configuration where liquid crystal, lithium niobate, or other techniques are utilized to represent a pattern, the pattern pitch can be modulated not only to change the focal power of the variable focus lens element 166 but also to change the focal power of the overall optical system for zoom lens type functionality.

[0285] In one embodiment, the lens 166 can be telecentric in that the focus of the display image can be modified while keeping the magnification constant in the same way as a photographic zoom lens is configured to disengage the focus from the zoom position. In another embodiment, the lens 166 may be non - telecentric such that the focus change will also follow the zoom change. In such a configuration, such magnification changes may be compensated for in software using dynamic scaling of the output from a graphics system that is synchronized with the focus change.

[0286] Returning to the problem of the method of supplying an image into the optical display system with the projector or other video display unit 160 in a "frame - sequential" configuration, a stack of sequential two - dimensional images is sequentially supplied to the display in a manner similar to a computed tomography system where the stacked image slices are used to represent a three - dimensional structure and can generate a three - dimensional perception over time.

[0287] A series of two - dimensional image slices can be presented to the eye at different focal distances respectively, and the eye / brain will integrate such a stack into the perception of a coherent three - dimensional volume. Depending on the display type, row - by - row or even pixel - by - pixel sequence processing can be performed to generate the perception of three - dimensional viewing. For example, in a scanning light display (such as a scanning fiber display or a scanning mirror display), the display presents one line or one pixel at a time to the waveguide 164 in a sequential manner.

[0288] If the variable focus lens element 166 is capable of maintaining high-frequency per-pixel or per-row presentation, each line or pixel is presented through the variable focus lens element 166, dynamically focused, and can be perceived at different focal distances from the eye 58. Per-pixel focus modulation generally requires an ultra-fast / high-frequency variable focus lens element 166. For example, a 1080P resolution display with an overall frame rate of 60 frames / second typically represents approximately 125 million pixels / second. Such a configuration may also be constructed using a solid-state switchable lens, such as those using electroactive materials, e.g., lithium niobate or electroactive polymers. In addition to its compatibility with the system illustrated in FIG. 7A, the frame sequential multi-focus display driving approach may be used in combination with several of the display systems and optical embodiments described in the present disclosure.

[0289] Referring to FIG. 7B, an electroactive layer 172 (such as one comprising liquid crystal or lithium niobate) may be surrounded by a waveguide 168 with functional electrodes (170, 174) (which may be made from indium tin oxide) and a conventional transmissive substrate 176. The waveguide may be made of glass or plastic with known total internal reflection characteristics and refractive indices that match the on or off state of the electroactive layer 172 in one or more embodiments. The electroactive layer 172 can be controlled such that the path of the incident beam can be dynamically modified to essentially create a time-varying bright field of view.

[0290] Referring to FIG. 8A, a stacked waveguide assembly 178 may be utilized to provide three-dimensional perception to the eye / brain by having a plurality of waveguides (182, 184, 186, 188, 190) and a plurality of weak lenses (198, 196, 194, 192) that are configured together to transmit image information to the eye with various levels of wavefront curvature for each waveguide level that indicates a focal distance to be perceived for that waveguide level. A plurality of displays (200, 202, 204, 206, 208), or in another embodiment, a single multiplexed display, may each be utilized to input collimated image information into the waveguides (182, 184, 186, 188, 190) that are configured to substantially equally distribute the incident light across the length of each waveguide for output to the eye as described above.

[0291] The waveguide 182 closest to the eye is configured to deliver collimated light to the eye as it is input into such waveguide 182 which may represent an optically infinite focal plane. Another waveguide 184 is configured to transmit collimated light that passes through a first weak lens (192; e.g., a weak negative lens) and is delivered to the user's eye 58. The first weak lens 192 may be configured to create a slightly convex wavefront curvature such that the eye / brain interprets the light arising from waveguide 184 as arising from a first focal plane that is closer in an inward direction from the optically infinite distance towards the person. Similarly, the next waveguide 186 has its output light passed through the first 192 and second 194 lenses before reaching the eye 58. The combined refractive power of the first 192 and second 194 lenses may be configured to create another incremental amount of wavefront divergence such that the eye / brain interprets the light arising from waveguide 186 as arising from a second focal plane that is closer in an inward direction from the optically infinite distance towards the person than the light from waveguide 184.

[0292] The other waveguide layers (188, 190) and weak lenses (196, 198) are similarly configured, and the highest waveguide 190 in the stack transmits its output through all of the weak lenses between it and the eye for the focusing force of the focal plane closest to the person, representing the focal plane closest to the person. When viewing / interpreting the light originating from the world 144 on the other side of the stacked waveguide assembly 178, a compensation lens layer 180 is disposed on top of the stack to compensate for the focusing refractive power of the lower lens stack (198, 196, 194, 192) in order to compensate for the stack of lenses (198, 196, 194, 192).

[0293] Such a configuration, as described above, again provides a relatively large exit pupil configuration for the same number of perceived focal planes as the available waveguide / lens pairings. Both the reflective side of the waveguide and the focusing side of the lens may be static (e.g., not dynamic or electroactive). In an alternative embodiment, it may be dynamic and use electroactive features as described above, allowing a small number of waveguides to be multiplexed in a time series manner to generate a larger number of effective focal planes.

[0294] Referring to FIGS. 8B - 8N, various aspects of diffractive configurations for focusing and / or redirecting a collimated beam are depicted. Other aspects of diffractive systems for such purposes are disclosed in U.S. Patent Application No. 14 / 331,218.

[0295] Referring to FIG. 8B, it should be understood that passing a collimated beam, such as a Bragg grating, through a linear diffraction pattern 210 will deflect, i.e., "steer," the beam. Also, it should be understood that passing a collimated beam through a radially symmetric diffraction pattern 212, i.e., a "Fresnel zone plate," will change the focus of the beam. FIG. 8C illustrates the deflecting effect of passing a collimated beam through a linear diffraction pattern 210. FIG. 8D illustrates the focusing effect of passing a collimated beam through a radially symmetric diffraction pattern 212.

[0296] Referring to FIGS. 8E and 8F, a combined diffraction pattern having both linear and radial elements 214 generates both deflection and focusing of the collimated input beam. These deflection and focusing effects can be generated in both reflection and transmission modes. These principles may be applied, for example, with a waveguide configuration to enable additional optical system control, as shown in FIGS. 8G-8N.

[0297] As shown in FIGS. 8G-8N, a diffraction pattern (220), i.e., a “diffractive optical element” (or “DOE”), is embedded within the planar waveguide 216 such that as the collimated beam is totally internally reflected along the planar waveguide 216, it intersects the diffraction pattern 220 at multiple locations.

[0298] Preferably, the DOE 220 has a relatively low diffraction efficiency such that only a portion of the light of the beam is deflected towards the eye 58 using each intersection of the DOE 220, while the remainder continues to travel through the planar waveguide 216 via total internal reflection. The light carrying the image information is thus split into several associated light beams that exit the waveguide at multiple locations, resulting in a very uniform pattern of emission towards the eye 58 for this particular collimated beam that bounces within the planar waveguide 216 as shown in FIG. 8H. The output beam towards the eye 58 is shown in FIG. 8H as being substantially parallel in this case since the DOE 220 has only a linear diffraction pattern. As shown in the comparison between FIGS. 8L, 8M, and 8N, a change to this linear diffraction pattern pitch may be utilized to controllably deflect the output parallel beam, thereby generating scanning or tiling display functionality.

[0299] Referring to FIG. 8I, as the radially symmetric diffractive pattern components of the DOE 220 to be embedded change, the output beam pattern becomes more divergent, requiring the eye to perform focusing at a closer distance and be interpreted by the brain as light from a viewing distance closer to the eye than optical infinity. Referring to FIG. 8J, with the addition of another waveguide 218 into which a beam can be input (e.g., by a projector or display), the DOE 221 embedded within this other waveguide 218, such as a linear diffractive pattern, can function to spread the light across the larger planar waveguide 216. This can provide the eye 58 with a very large incident field of the incident light exiting from the larger planar waveguide 216, e.g., a large eyebox, according to a particular DOE configuration in operation.

[0300] The DOE (220, 221) is depicted as bisecting the associated waveguide (216, 218), but this need not be the case. In one or more embodiments, they may be placed closer to or on either side of any of the waveguides (216, 218) to have the same functionality. Thus, as shown in FIG. 8K, with the input of a single collimated beam, the entire field of the cloned collimated beams can be directed towards the eye 58. Additionally, in a combined linear diffractive pattern / radially symmetric diffractive pattern scenario such as those depicted in 214 of FIG. 8F and 220 of FIG. 8I, beam distribution waveguide optics with Z-axis focusing capabilities (for functionality such as the functional expansion of the exit pupil; using a configuration such as that of FIG. 8K, the exit pupil can be the same size as the optical element itself, which can be a very significant advantage for user comfort and ergonomics) is presented, and both the divergence angle of the cloned beams and the wavefront curvature of each beam represent light originating from a point closer to the eye than optical infinity.

[0301] In one embodiment, one or more DOEs are switchable between an “on” state where they actively diffract and an “off” state where they do not significantly diffract. For example, a switchable DOE may comprise a polymer dispersed liquid crystal layer that can be switched such that the microdroplets have a diffraction pattern within the host medium and the refractive index of the microdroplets substantially matches that of the host material (in which case the pattern does not significantly diffract the incident light), or the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts the incident light).

[0302] Furthermore, as shown in FIGS. 8L - 8N, beam scanning or tiled display functionality can be achieved with dynamic changes to diffraction terms such as the linear diffraction pitch term. As described above, it may be desirable for each of the DOEs (220, 221) to have a relatively low diffraction grating efficiency to facilitate the distribution of light. Also, the light that is preferably transmitted through the waveguide (e.g., in an augmented reality configuration, the light that travels from the world 144 to the eye 58) is less affected when the diffraction efficiency of the DOE220 that it crosses is lower, so a better view of the real world can be achieved through such a configuration.

[0303] Configurations such as those illustrated in FIG. 8K are preferably driven with the input of image information in a time - series approach, and frame - sequential driving is the easiest to implement. For example, an empty image at optical infinity can be input at time 1, and a diffraction grating that maintains the collimation of light may be utilized. Then, an image of a closer tree branch can be input at time 2, while the DOE is given the ability to controllably impart a focus change, for example, 1 diopter or 1 meter away, providing the eye / brain with the perception that the light information of the branch is originating from a closer focal distance.

[0304] This type of paradigm may be repeated in a high-speed time series manner such that the eye / brain perceives that the input is the entire part of the same image. This is only an example of two focal planes, but it should be understood that preferably the system will be configured to have more focal planes and provide a smoother transition between the object and its focal distance. The configuration of this type generally assumes that the DOE is switched relatively slowly (e.g., synchronized with the frame rate of a display that inputs images in the range of tens to hundreds of cycles per second).

[0305] Conversely, a configuration may be such that the DOE element can shift the focus at tens to hundreds of MHz or more, and as the pixels are scanned into the eye 58 using a scanning light display type approach, it facilitates switching of the focus state of the DOE element on a per-pixel basis. This is desirable because it means that the overall display frame rate can be kept very low (low enough to ensure that "flicker" is not a problem (within the range of about 60 to 120 frames per second)).

[0306] Between these ranges, if the DOE can be switched at a KHz rate, on a per-row basis, the focus on each scan line may be adjusted, which can give the user a visible advantage from the perspective of temporal artifacts, for example, during eye movement with respect to the display. For example, different focal planes within the scene may be interleaved in this way to minimize visible artifacts in response to head movement (as discussed in detail later in this disclosure). The per-row focus modulator may be operably coupled to a line-scanning display such as a grating light valve display where a linear array of pixels is swept to form an image, or may be operably coupled to a scanning light display such as a fiber scanning display and a mirror scanning light display.

[0307] A stacked configuration similar to that of FIG. 8A may provide multi-plane focusing simultaneously using a dynamic DOE (rather than the static waveguides and lenses of the embodiment of FIG. 8A). For example, in three simultaneous focal planes, a primary focal plane (e.g., based on the measured eye accommodation) may be presented to the user, and the + and - boundaries (e.g., one focal plane is closer and one focal plane is farther away) may be utilized to provide a large focal distance that the user can adjust in focus before a plane update is required. This increased focal distance can provide a time advantage when the user switches to a closer or farther focus (as determined by the focus adjustment measurement). Then, the new plane of focus can be brought to the central depth of focus, and the + and - boundaries are again ready for high-speed switching to either side while the system catches up.

[0308] Referring to FIG. 8O, a stack 222 of planar waveguides (244, 246, 248, 250, 252) is shown, each having a reflector (254, 256, 258, 260, 262) at an end, such that collimated image information input at one end by a display (224, 226, 228, 230, 232) bounces back by total internal reflection to the reflector, at which point some or all of the light is reflected towards the eye or other target. Each of the reflectors may have a slightly different angle such that all reflect the outgoing light towards a common destination such as the pupil. Such a configuration is somewhat similar to that of FIG. 5B, but each differently angled reflector in the embodiment of FIG. 8O has its own waveguide so that the projected light interferes little as it travels towards the target reflector. Lenses (234, 236, 238, 240, 242) ...

Claims

1. An augmented reality (AR) device, comprising: a first image generator for displaying a first image to a first eye of a user; a first inwardly directed light source for illuminating the first eye of the user in a first pattern to facilitate tracking of the first eye of the user; Processor and Equipped with The processor, a first driver for providing first image information and a first control signal to the first image generator; a first controller communicatively coupled to the first inwardly directed light source; An AR device comprising:

2. The AR device of claim 1, wherein the first inwardly directed light source is a first light emitting diode (LED).

3. The AR device of claim 1, wherein the first pattern is a spatial pattern.

4. The AR device of claim 1, wherein the first pattern is a temporal pattern.

5. The AR device of claim 1, wherein illuminating the first eye of the user with the first pattern facilitates detection of the first eye of the user.

6. The AR device of claim 1, wherein illuminating the first eye of the user with the first pattern facilitates monitoring the orientation of the first eye of the user.

7. The AR device of claim 1, wherein illuminating the first eye of the user with the first pattern facilitates monitoring movements of the first eye of the user.

8. A second image generator for displaying a second image to a second eye of the user. a second inwardly directed light source for illuminating the second eye of the user in a second pattern to facilitate tracking of the second eye of the user; and Further equipped with The AR device of claim 1 , wherein the processor further comprises a second controller communicatively coupled to the second inwardly directed light source.

9. The AR device of claim 8, wherein the second inwardly directed light source is a second LED.

10. The AR device of claim 8, wherein the second pattern is a spatial pattern.

11. The AR device of claim 8, wherein the second pattern is a temporal pattern.

12. The AR device of claim 8, wherein illuminating the second eye of the user with the second pattern facilitates detection of the second eye of the user.

13. The AR device of claim 8, wherein illuminating the user's second eye with the second pattern facilitates monitoring the orientation of the user's second eye.

14. The AR device of claim 8, wherein illuminating the second eye of the user with the second pattern facilitates monitoring movements of the second eye of the user.

Images