Selection of virtual object within three-dimensional space

The wearable system addresses the challenge of precise object selection in VR, AR, and MR by dynamically switching input modes based on user posture and context, enhancing interaction comfort and precision.

JP2024159884AActive Publication Date: 2024-11-08MAGIC LEAP INC
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
JP2024139591
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2016-03-31
Filing Date
2024-08-21
Publication Date
2024-11-08
Estimated Expiration
2036-10-18

AI Technical Summary

Technical Problem

Existing VR, AR, and MR technologies face challenges in providing a comfortable and natural-feeling interaction with virtual objects in 3D space, as precise selection and control can be tiring and difficult due to the need for maintaining specific postures.

Method used

A wearable system that switches user input modes based on user posture and input device interactions, allowing for precise selection of virtual objects using head gestures for large objects and hand gestures for small objects, and adjusts input modes based on context information such as object density and layout.

Benefits of technology

Enhances user interaction comfort by reducing fatigue and improving precision in selecting virtual objects, allowing for more natural and efficient interaction with virtual environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024159884000001_ABST
    Figure 2024159884000001_ABST
Patent Text Reader

Abstract

To provide selection of a virtual object within a suitable, three-dimensional space.SOLUTION: A system and a method for interacting with a virtual object within a three-dimensional space by using a wearable system are disclosed. The wearable system can be programmed so as to allow a user to interact with a virtual object by using a user input device and posture. The wearable system also automatically determines context information, such as the layout of a virtual object in a user's environment, and can switch a user input mode on the basis of the context information.SELECTED DRAWING: Figure 12
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority under 35 USC § 119(e) to U.S. Provisional Application No. 62 / 244,115, filed October 20, 2015, entitled "USER APPLICATIONS, INTERFACES, AND EXPERIENCES WITH AUGMENTED REALITY DISPLAY DEVICES," U.S. Provisional Application No. 62 / 301,422, filed February 29, 2016, entitled "SELECTING VIRTUAL OBJECTS IN 3D SPACE," and U.S. Provisional Application No. 62 / 316,179, filed March 31, 2016, entitled "SELECTING VIRTUAL OBJECTS IN 3D SPACE," all of which are incorporated herein by reference in their entireties.

[0002] The present disclosure relates to virtual reality, augmented reality, and mixed reality imaging and visualization systems, and in particular to systems for interacting with virtual objects in three-dimensional (3D) space. [Background technology]

[0003] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality", "augmented reality", or "mixed reality" experiences in which digitally reproduced images or portions thereof are presented to a user in a manner that appears or can be perceived as real. Virtual reality or "VR" scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs. Augmented reality or "AR" scenarios typically involve digital or virtual image information as an extension of the visualization of the real world around the user. Mixed reality or "MR" relates to the merging of real and virtual worlds to generate new environments in which physical and virtual objects coexist and interact in real time. In conclusion, the human visual perception system is highly complex, making it difficult to generate VR, AR, or MR technologies that promote a comfortable and natural-feeling rich presentation of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies. Summary of the Invention [Means for solving the problem]

[0004] In some embodiments, a system for changing a user input mode for a wearable device is disclosed. The system can include a display system of the wearable device configured to present a three-dimensional (3D) view to a user, the 3D view comprising an interactable object, a user input device configured to receive user input, a sensor configured to acquire data associated with a user's posture, and a hardware processor in communication with the user input device. The hardware processor can be programmed to determine whether a current user input mode for interacting with the interactable object is a first user input mode or a second user input mode, the first user input mode being based, at least in part, on a posture of the user and the second user input mode being based, at least in part, on a user input from the user input device. In response to determining that the current user input mode is the first user input mode, the hardware processor can monitor a user's posture using the sensor, present via the display system a focus indicator in a first shape associated with the first user input mode based at least in part on the monitored posture in a direction associated with the user's posture, receive the first indication, switch to a second user input mode, and in response to the first indication, switch the current user input mode to the second user input mode. In response to determining that the current user input mode is the second user input mode, the hardware processor can monitor user input from the user input device, present via the display system a focus indicator in a second shape associated with the second user input mode based at least in part on the monitored input, receive the second indication, switch to the first user input mode, and in response to the second indication, switch the current user input mode to the first user input mode.

[0005] In an embodiment, a method for changing a user input mode for a wearable device is disclosed. The method may be performed under the control of a wearable device comprising a computer processor. The wearable device may be configured to enable user interaction with an interactable object within a user's ocular field of view (FOR), the FOR comprising a portion of the user's surrounding environment perceivable by the user via a display system of the wearable device. The method includes determining a user's posture, displaying via the display system a first focus indicator associated with a target interactable object in a direction associated with the user's posture, the target interactable object comprising a plurality of virtual objects, receiving a selection of the target interactable object, presenting the user with an option to switch a user input mode from posture to hand gestures on a user input device, displaying via the display system the plurality of virtual objects, and in response to determining that the user has switched the user input mode from posture to hand gestures on the user input device, displaying via the display system a second focus indicator associated with a target virtual object of the plurality of virtual objects, and updating the second focus indicator based at least in part on the user input from the user input device.

[0006] In some embodiments, a wearable system and method for selecting a virtual object located in a three-dimensional (3D) space is disclosed. The wearable system can include a display system configured to present the virtual object in the 3D space, a non-transient data store configured to store the interactable object in the 3D space, a sensor configured to determine a user's posture, and a hardware processor programmed to communicate with the display system, the data store, and the sensor. The wearable system and method can determine the user's posture based at least in part on data received from the sensor, determine the user's field of view (FOV) based at least in part on the user's posture, the FOV comprising a portion of the user's environment perceived by the user at a given time, identify a group of interactable objects within the FOV, identify a target interactable object within the FOV based at least in part on the user's posture, and initiate a selection event associated with the target interactable object.

[0007] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Neither this Summary nor the following Detailed Description purport to define or limit the scope of the inventive subject matter. The present invention provides, for example, the following: (Item 1) 1. A system for changing a user input mode for a wearable device, comprising: a display system of a wearable device configured to present a three-dimensional (3D) view to a user, the 3D view comprising interactable objects; a user input device configured to receive user input; a sensor configured to acquire data associated with a posture of the user; and a hardware processor in communication with the user input device, determining whether a current user input mode for interacting with the interactable object is a first user input mode or a second user input mode; the first user input mode is based, at least in part, on a posture of the user; the second user input mode is based, at least in part, on user input from the user input device; and in response to determining that the current user input mode is the first user input mode, monitoring a posture of the user using the sensor; presenting, via the display system, a focus indicator in a first shape associated with the first user input mode in an orientation related to a pose of the user based, at least in part, on the monitored pose; receiving a first indication and switching to the second user input mode; in response to the first indication, switching the current user input mode to the second user input mode; in response to determining that the current user input mode is the second user input mode, monitoring user input from the user input device; presenting, via the display system, a focus indicator in a second shape associated with the second user input mode based, at least in part, on the monitored input; and receiving a second indication and switching to the first user input mode; in response to the second indication, switching the current user input mode to the first user input mode; a hardware processor programmed to A system comprising: (Item 2) The processor further comprises: determining a field of view (FOV) of the user based, at least in part, on a posture of the user, the FOV comprising a portion of the user's environment as perceived by the user at a given time; determining contextual information associated with the FOV, the contextual information comprising at least one of a layout of the interactable objects within the FOV, a size of the FOV, a size of one or more of the interactable objects within the user's FOV; and presenting an option for switching from the first user input mode to the second user input mode or from the second user input mode to the first user input mode based at least in part on the context information; 2. The system according to item 1, programmed to: (Item 3) 4. The system of claim 3, wherein to present the options, the processor is programmed to present the focus indicator in the second shape when the current user input mode is in the first user input mode, and to present the focus indicator in the first shape when the current user input mode is in the second user input mode. (Item 4) 2. The system of claim 1, wherein the first indication comprises a change in the user's posture and the second indication comprises an actuation of the user input device. (Item 5) The system of any one of items 1-4, wherein the user's pose comprises at least one of a head pose, an eye pose, a foot pose, or a body pose, or the sensor comprises at least one of an inward-facing imaging system, an outward-facing imaging system, or an inertial measurement unit, or the display system comprises a light field display configured to display one or more of the interactable objects in multiple depth planes. (Item 6) 1. A method for changing a user input mode for a wearable device, comprising: Under control of a wearable device comprising a computer processor, the wearable device configured to enable user interaction with interactable objects within a user's field of view (FOR), the FOR comprising a portion of the user's surrounding environment perceivable by the user via a display system of the wearable device; determining a posture of a user; displaying, via the display system, a first focus indicator associated with a target interactable object in a direction related to a posture of the user, the target interactable object comprising a plurality of virtual objects; receiving a selection of the target interactable object; presenting the user with an option to switch the user input mode from posture to hand gestures on a user input device; displaying the plurality of virtual objects via the display system; displaying, via the display system, a second focus indicator associated with a target virtual object of the plurality of virtual objects in response to determining that the user has switched the user input mode from a posture to a hand gesture on the user input device; updating the second focus indicator based at least in part on user input from the user input device. A method comprising: (Item 7) 7. The method of claim 6, wherein the options are presented in response to a selection of the target interactable object or contextual information associated with the plurality of virtual objects or the target interactable object. (Item 8) 8. The method of claim 7, wherein the contextual information comprises a density of the plurality of virtual objects, and wherein an option to switch the user input mode from a posture to a hand gesture on a user input device is presented in response to determining that the density of the plurality of virtual objects exceeds a threshold density. (Item 9) 7. The method of claim 6, wherein updating the second focus indicator includes transitioning the second focus indicator from the target virtual object to another virtual object of the plurality of virtual objects. (Item 10) 7. The method of claim 6, further comprising initiating a selection event on the target virtual object, the selection event comprising at least one of opening a menu associated with the target virtual object or receiving an indication to select the target virtual object. (Item 11) 11. The method of any one of items 6-10, wherein the target virtual object is identified based at least in part on a pose of the user. (Item 12) 7. The method of claim 6, wherein the plurality of virtual objects comprises at least one of a weather application or an astronomy application, and in response to a selection of the weather application, the wearable device is programmed to display virtual weather information superimposed on objects in the user's environment, and in response to a selection of the astronomy application, the wearable device is programmed to display an interactive planetary system comprising a three-dimensional virtual planet superimposed on the user's environment. (Item 13) 1. A wearable system for selecting a virtual object located in three-dimensional (3D) space, comprising: A display system configured to present virtual objects in a 3D space; and a non-transitory data store configured to store interactable objects within said 3D space; A sensor configured to determine a posture of a user; a hardware processor programmed to communicate with the display system, the data store, and the sensor, determining a posture of the user based at least in part on the data received from the sensor; and determining a field of view (FOV) of the user based, at least in part, on a posture of the user, the FOV comprising a portion of the user's environment as perceived by the user at a given time; identifying a group of interactable objects within the FOV; Identifying a target interactable object within the FOV based, at least in part, on a pose of the user; and Initiating a selection event associated with the target interactable object; A hardware processor and A wearable system comprising: (Item 14) Item 14. The wearable system of item 13, wherein the group of interactable objects is stored in a data structure and an index associated with each interactable object is determined, at least in part, based on a position of the interactable object in the 3D space. (Item 15) Item 14. The wearable system of item 13, wherein in response to initiation of a selection event on the target interactable object, the processor of the wearable system is programmed to present, within a two-dimensional (2D) interface, a virtual object within a threshold range of the target interactable object in the 3D space. (Item 16) Item 16. The wearable system of item 15, wherein the 2D interface is interactable via a user input device. (Item 17) To identify a target interactable object within the FOV, the processor: determining a gaze path of the user based on a pose of the user and selecting an interactable object that intersects with the gaze path as the target interactable object; or selecting a left-most or right-most interactable object within the user's FOV as the target interactable object, the left-most or right-most interactable object being selected based at least in part on an index associated with the group of interactable objects; Item 14. The wearable system of item 13, configured to perform at least one of the following: (Item 18) Item 14. The wearable system of item 13, wherein the processor is configured to initiate the selection event in response to at least one of receiving an input from a user input device or detecting a change in the user's posture. (Item 19) The wearable system of any one of items 13-19, wherein the processor is further configured to present a focus indicator associated with the target interactable object. (Item 20) and a geolocation sensor configured to obtain data associated with the user's location, the target interactable object comprising a weather application, the weather application comprising: determining a location of the user based on data acquired by the geolocation sensor; communicating with a remote computing device to obtain weather data based on a location of the user; generating a virtual element associated with the weather data; superimposing said virtual element in the user's 3D space; Item 14. The wearable system of item 13, programmed to: [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 depicts an illustration of a mixed reality scenario involving a virtual reality object and a physical object viewed by a person.

[0009] [Diagram 2] FIG. 2 illustrates diagrammatically an embodiment of a wearable system.

[0010] [Diagram 3] FIG. 3 diagrammatically illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes.

[0011] [Figure 4] FIG. 4 diagrammatically illustrates an embodiment of a waveguide stack for outputting image information to a user.

[0012] [Diagram 5]FIG. 5 shows an exemplary output beam that may be output by a waveguide.

[0013] [Figure 6] FIG. 6 is a schematic diagram showing an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem for use in generating a multifocal volumetric display, image, or light field.

[0014] [Figure 7] FIG. 7 is a block diagram of an embodiment of a wearable system.

[0015] [Figure 8] FIG. 8 is a process flow diagram of an embodiment of a method for rendering virtual content in relation to recognized objects.

[0016] [Figure 9] FIG. 9 is a block diagram of another embodiment of a wearable system.

[0017] [Figure 10] FIG. 10 is a process flow diagram of an example method for determining user input to a wearable system.

[0018] [Figure 11] FIG. 11 is a process flow diagram of an embodiment of a method for interacting with a virtual user interface.

[0019] [Figure 12] FIG. 12 illustrates an example of a virtual object in the field of view and a virtual object in the kinetic field of view.

[0020] [Figure 13A] FIG. 13A illustrates an example of selection of an interactable object using touch gestures on a touch screen of a user input device.

[0021] [Figure 13B] FIG. 13B illustrates an example of filtering selectable objects using hand gestures of a user input device.

[0022] [Figure 14] FIG. 14 is an example of a coordinate system for head pose.

[0023] [Figure 15] FIG. 15 illustrates an example of interaction with an interactable object using head pose.

[0024] [Figure 16] FIG. 16 illustrates an example of interaction with an interactable object using hand gestures.

[0025] [Figure 17] FIG. 17 illustrates example interaction events with a weather application.

[0026] [Figure 18] FIG. 18 illustrates an example user experience of interacting with a 3D virtual object.

[0027] [Figure 19] FIG. 19 illustrates an example process for selection of a virtual object using a combination of posture and hand gestures on a user input device.

[0028] [Figure 20] FIG. 20 illustrates an example process for interaction with a virtual object using a combination of postures and hand gestures on a user input device.

[0029] [Figure 21] FIG. 21 illustrates an example process for switching input control from head poses to hand gestures based on context information.

[0030] [Figure 22] FIG. 22 illustrates an example process for switching modes of user interaction based on context information.

[0031] [Figure 23] FIG. 23 illustrates an example process of interaction with an interactable object that comprises a group of virtual objects.

[0032] Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] (Overview) With the use of AR / VR / MR devices, a user may wish to target and select objects in three-dimensional (3D) space using a virtual user interface. For example, a user may select a virtual object using a body posture, such as physically approaching an item, grasping it, or touching it. A user may also select a virtual object by pointing to and clicking on the object using a virtual ray or beam. However, these techniques may require the user to hold their posture still to achieve the selection, which may cause fatigue and make it difficult to precisely select an object.

[0034] This disclosure provides an example of a wearable system that addresses some or all of these issues. As an example, a user may move their head and look at a group of objects. The objects closest to the center of the user's field of view may be highlighted as potential target objects, and the user may actuate a user input device (e.g., by swiping on a touch screen) to shift the highlight from one object to another. The user may confirm selection of the target object by actuating the user input device again (e.g., by touching the touch screen). Once selected, the AR user interface may allow the user to perform additional actions on the selected target object (e.g., displaying or selecting from a menu associated with the object, performing an action associated with a game in which the target object appears, etc.). This technique may be particularly advantageous for a user to select an object of interest while reducing fatigue, since head pose is difficult to precisely control. The wearable system may preliminarily identify an object of interest based on the user's head pose, while allowing the user to precisely select the object using hand gestures.

[0035] In some implementations, the interactable object may include multiple virtual objects. For example, a virtual user interface plane may include multiple virtual applications, such as, for example, a video streaming application, a virtual classroom application, a weather application, a gaming application, an astronomy application, and the like. The wearable system may support different user input modes based on the characteristics of the interactable object. For example, when the interactable object is a user interface plane (which may be large in size), the wearable system may allow the user to interact with it using posture. On the other hand, when the interactable object is relatively small, the wearable system may instead set the user input device as the default input mode and allow the user to precisely interact with the virtual object. These implementations may be advantageous because moving and targeting a large object may require less precision in the user's movement, while moving and selecting a small object may require the user to precisely target.

[0036] The wearable system can also determine the user input mode based on the context information. For example, the wearable system can determine the layout of virtual objects in the user's environment. When the wearable system detects a dense cluster of virtual objects in the user's line of sight, the wearable system may give the user the option to switch input control from head control to hand control. In this way, the user can interact with the virtual objects more precisely. As another example, the AR system may detect the orientation of the object (e.g., vertical or horizontal) and provide the appropriate interaction for the user (e.g., volume control for a television application that appears vertically in front of the user or typing control for a virtual keyboard that appears horizontally on the user's desk).

[0037] The wearable system may enable the user to share virtual content with others (who are also wearing a wearable system), for example, by passing a world map of the user's environment over a network, or by communicating virtual content (or updating virtual content) between wearable systems.

[0038] (Example of 3D display) Figure 1 depicts an illustration of a mixed reality scenario with certain virtual reality objects and certain physical objects viewed by a person. In Figure 1, an MR scene 100 is depicted in which a user of the MR technology sees a real-world park-like setting 110 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of the MR technology also perceives that they "see" a robotic figure 130 standing on the real-world platform 120, and a flying cartoon-like avatar character 140 that appears to be an anthropomorphic bumblebee, although these elements do not exist in the real world.

[0039] In order for a 3D display to produce a true depth sensation, and more specifically a simulated sensation of surface depth, it may be desirable to generate, for each point in the display's field of view, an accommodation response that corresponds to that point's virtual depth. If the accommodation response to a display point does not correspond to that point's virtual depth as determined by the binocular depth cues of convergence and stereopsis, the human eye may experience accommodation conflicts, resulting in unstable imaging, deleterious eye strain, headaches, and, in the absence of accommodative information, a near-complete lack of surface depth.

[0040] VR, AR, and MR experiences can be provided by a display system having a display in which images corresponding to multiple depth planes are provided to the viewer. The images may be different for each depth plane (e.g., providing slightly different presentations of a scene or object) and may be focused separately by the viewer's eyes, thereby serving to provide depth cues to the user based on the ocular accommodation required to focus on different image features of a scene located on different depth planes, or based on observing different image features on different depth planes that are out of focus. As discussed elsewhere herein, such depth cues provide a believable perception of depth.

[0041] FIG. 2 illustrates an example of a wearable system 200. The wearable system 200 includes a display 220 and various mechanical and electronic modules and systems to support the functionality of the display 220. The display 220 may be coupled to a frame 230 that is wearable by a user, wearer, or viewer 210. The display 220 can be positioned in front of the eyes of the user 210. The display 220 can present AR / VR / MR content to the user. The display 220 can present a head mounted display (HMD) that is worn on the user's head. In some embodiments, a speaker 240 is coupled to the frame 230 and positioned adjacent to the user's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / shapeable sound control).

[0042] The wearable system 200 may include an outwardly facing imaging system 464 (shown in FIG. 4) that observes the world in the user's surrounding environment. The wearable system 200 may also include an inwardly facing imaging system 462 (shown in FIG. 4) that may track the user's eye movements. The inwardly facing imaging system may track either one eye's movements or both eyes' movements. The inwardly facing imaging system 462 may be mounted to the frame 230 and may be in electrical communication with a processing module 260 or 270 that may process image information acquired by the inwardly facing imaging system and determine, for example, pupil diameter or orientation of the user's 210 eyes, eye movements, or eye pose.

[0043] As an example, the wearable system 200 can capture images of the user's posture using the outward facing imaging system 464 or the inward facing imaging system 462. The images may be still images, frames of video or videos, combinations thereof, or the like.

[0044] The display 220 is operably coupled (250) to a local data processing module 260, which may be mounted in a variety of configurations, such as fixedly attached to the frame 230, by wired leads or a wireless connection, fixedly attached to a helmet or hat worn by the user, built into headphones, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, in a belt-connected configuration).

[0045] The local processing and data module 260 may comprise a hardware processor and digital memory such as non-volatile memory (e.g., flash memory), both of which may be utilized to aid in processing, caching, and storing data. The data may include a) data captured from sensors (e.g., which may be operatively coupled to the frame 230 or otherwise attached to the user 210), such as image capture devices (e.g., cameras in the inwardly facing imaging system and / or the outwardly facing imaging system), microphones, inertial measurement units (IMUs), accelerometers, compasses, global positioning systems (GPS), wireless devices, or gyroscopes), or b) data acquired or processed using the remote processing module 270 and / or remote data repository 280, possibly for transmission to the display 220 after such processing or readout. The local processing and data module 260 may be operatively coupled to a remote processing module 270 or a remote data repository 280 by a communication link 262 or 264, such as via a wired or wireless communication link, such that these remote modules are available as resources to the local processing and data module 260. In addition, the remote processing module 280 and the remote data repository 280 may be operatively coupled to each other.

[0046] In some embodiments, remote processing module 270 may comprise one or more processors configured to analyze and process the data and / or image information. In some embodiments, remote data repository 280 may comprise a digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module, allowing for fully autonomous use from the remote module.

[0047] The human visual system is complex and difficult to provide a realistic perception of depth. Without being limited by theory, it is believed that a viewer of an object may perceive the object as three-dimensional due to a combination of vergence and accommodation. Vergence of the two eyes relative to one another (i.e., the rotation of the pupils toward or away from one another to cause the gaze of the eyes to converge and fixate on an object) is closely coupled to the focusing (or "accommodation") of the eye's lenses. Under normal conditions, changing the focus of the eye's lenses, or accommodating the eyes and changing focus from one object to another at a different distance, will automatically produce a corresponding change in vergence at the same distance, in a relationship known as the "accommodation-vergence reflex." Similarly, a change in vergence will induce a corresponding change in accommodation under normal conditions. Display systems that provide a better match between accommodation and convergence-divergence movements may produce more realistic and comfortable simulations of three-dimensional images.

[0048] FIG. 3 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes. With reference to FIG. 3, objects at various distances from the eyes 302 and 304 on the z-axis are accommodated by the eyes 302 and 304 such that the objects are in focus. The eyes 302 and 304 assume particular accommodated states that focus objects at different distances along the z-axis. As a result, a particular accommodated state may be said to be associated with a particular one of the depth planes 306 with an associated focal length such that an object or part of an object at a particular depth plane is in focus when the eye is in an accommodated state relative to that depth plane. In some embodiments, a three-dimensional image may be simulated by providing different presentations of images for each eye 302 and 304, and by providing different presentations of images corresponding to each of the depth planes. Although shown as separate for clarity of illustration, it should be understood that the fields of view of the eyes 302 and 304 may overlap, for example, as the distance along the z-axis increases. In addition, while shown to be flat for ease of illustration, it should be understood that the contour of the depth plane may be curved in physical space such that all features within the depth plane are in focus with the eye in a particular accommodation state. Without being limited by theory, it is believed that the human eye may typically interpret a finite number of depth planes to provide depth perception. As a result, a highly realistic simulation of perceived depth may be achieved by providing the eye with a different representation of an image corresponding to each of these limited number of depth planes.

[0049] (Waveguide Stack Assembly) FIG. 4 illustrates an example of a waveguide stack for outputting image information to a user. The wearable system 400 includes a stack of waveguides or a stacked waveguide assembly 480 that can be utilized to provide a three-dimensional perception to the eye / brain using multiple waveguides 432b, 434b, 436b, 438b, 400b. In some embodiments, the wearable system 400 may correspond to the wearable system 200 of FIG. 2, and FIG. 4 diagrammatically illustrates some portions of the wearable system 200 in more detail. For example, in some embodiments, the waveguide assembly 480 may be integrated into the display 220 of FIG. 2.

[0050] 4, the waveguide assembly 480 may also include a number of features 458, 456, 454, 452 between the waveguides. In some embodiments, the features 458, 456, 454, 452 may be lenses. In other embodiments, the features 458, 456, 454, 452 may not be lenses. Rather, they may simply be spacers (e.g., cladding layers or structures to form air gaps).

[0051] The waveguides 432b, 434b, 436b, 438b, 440b or multiple lenses 458, 456, 454, 452 may be configured to transmit image information to the eye with various levels of wavefront curvature or light beam divergence. Each waveguide level may be associated with a particular depth plane and configured to output image information corresponding to that depth plane. Image injection devices 420, 422, 424, 426, 428 may be utilized to inject image information into the waveguides 440b, 438b, 436b, 434b, 432b, respectively, which may be configured to disperse incident light across each individual waveguide for output toward the eye 410. Light exits output surfaces of image launch devices 420, 422, 424, 426, 428 and is launched into corresponding input edges of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam of light (e.g., a collimated beam) may be launched into each waveguide to output an entire field of cloned collimated beams that are directed towards the eye 410 at a particular angle (and divergence) that corresponds to the depth plane associated with the particular waveguide.

[0052] In some embodiments, the image input devices 420, 422, 424, 426, 428 are discrete displays that generate image information for input into each corresponding waveguide 440b, 438b, 436b, 434b, 432b, respectively. In some other embodiments, the image input devices 420, 422, 424, 426, 428 are outputs of a single multiplexed display that may pipe image information to each of the image input devices 420, 422, 424, 426, 428, for example, via one or more optical conduits (such as fiber optic cables).

[0053] A controller 460 controls the operation of the stacked waveguide assembly 480 and the image input devices 420, 422, 424, 426, 428. The controller 460 includes programming (e.g., instructions in a non-transitory computer-readable medium) that coordinates the timing and provision of image information to the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the controller 460 may be a single integrated device or a distributed system connected by a wired or wireless communication channel. The controller 460 may be part of the processing module 260 or 270 (shown in FIG. 2) in some embodiments.

[0054] The waveguides 440b, 438b, 436b, 434b, 432b may be configured to propagate light within each individual waveguide by total internal reflection (TIR). The waveguides 440b, 438b, 436b, 434b, 432b may each be planar or have another shape (e.g., curved) with major top and bottom surfaces and edges extending between the major top and bottom surfaces. In the illustrated configuration, the waveguides 440b, 438b, 436b, 434b, 432b may each include a light extraction optical element 440a, 438a, 436a, 434a, 432a configured to extract light from the waveguide by redirecting the light to propagate within each individual waveguide and outputting image information from the waveguide to the eye 410. The extracted light may also be referred to as out-coupled light, and the light extraction optical element may also be referred to as out-coupling optical element. The extracted light beam is output by the waveguide to where the light propagating in the waveguide strikes the light redirecting element. The light extraction optical element (440a, 438a, 436a, 434a, 432a) may be, for example, a reflective or diffractive optical feature. Although shown disposed on the bottom major surface of the waveguides 440b, 438b, 436b, 434b, 432b for ease of explanation and clarity of drawing, in some embodiments the light extraction optical element 440a, 438a, 436a, 434a, 432a may be disposed on the top or bottom major surface, or directly within the volume of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed in a layer of material that is attached to a transparent substrate and forms the waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, 432b may be a monolithic piece of material and the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed on and / or within that piece of material.

[0055] Continuing with reference to FIG. 4, as discussed herein, each waveguide 440b, 438b, 436b, 434b, 432b is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 as it is launched into such waveguide 432b. The collimated light may represent an optical infinity focal plane. The next waveguide 434b may be configured to send collimated light that passes through a first lens 452 (e.g., a negative lens) before it can reach the eye 410. The first lens 452 may be configured to generate a slight convex wavefront curvature so that the eye / brain interprets the light emerging from that next waveguide 434b as emerging from a first focal plane closer inward from optical infinity toward the eye 410. Similarly, the third waveguide 436b passes its output light through both a first lens 452 and a second lens 454 before reaching the eye 410. The combined refractive power of the first and second lenses 452 and 454 may be configured to produce another incremental amount of wavefront curvature such that the eye / brain interprets the light emerging from the third waveguide 436b as emerging from a second focal plane even closer inward toward the person from optical infinity where the light from the next waveguide 434b was.

[0056] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured with the highest waveguide 440b in the stack sending its output through all of the lenses between it and the eye for an aggregate focal power that represents the focal plane closest to the person. To compensate the stack of lenses 458, 456, 454, 452 when viewing / interpreting light originating from the world 470 on the other side of the stacked waveguide assembly 480, a compensating lens layer 430 may be placed on top of the stack to compensate for the aggregate power of the lens stacks 458, 456, 454, 452 below. Such a configuration provides as many perceived focal planes as there are waveguide / lens pairs available. Both the light extraction optical elements of the waveguides and the focusing sides of the lenses may be static (e.g., not dynamic or electro-active). In some alternative embodiments, either or both may be dynamic using electro-active features.

[0057] Continuing with reference to FIG. 4, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be configured to redirect light from its respective waveguide and output this light with the appropriate amount of divergence or collimation for the particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes may have different configurations of light extraction optical elements that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be solid or surface features that may be configured to output light at specific angles. For example, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be volume holograms, surface holograms, and / or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published June 25, 2015, which is incorporated by reference in its entirety.

[0058] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffractive features, i.e., "diffractive optical elements" (also referred to herein as "DOEs"), that form a diffraction pattern. Preferably, the DOEs have a relatively low diffraction efficiency, such that only a portion of the light of the beam is deflected toward the eye 410 with each intersection point of the DOE, while the remainder continues traveling through the waveguide via total internal reflection. The light carrying the image information is thus split into several related exit beams that exit the waveguide at multiple locations, which can result in a very uniform exit emission pattern toward the eye 304 for this particular collimated beam bouncing within the waveguide.

[0059] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer dispersed liquid crystal in which the microdroplets form a diffractive pattern in a host medium, and the refractive index of the microdroplets can be switched to one that substantially matches the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets can be switched to one that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0060] In some embodiments, the number and distribution of depth planes or depths of field may be dynamically varied based on the size or orientation of the pupil of the viewer's eye. The depth of field may vary inversely with the size of the viewer's pupil. As a result, as the size of the viewer's eye pupil decreases, the depth of field increases so that one plane that is indistinguishable because its location exceeds the focal depth of the eye may become distinguishable and appear more focused with a reduction in pupil size and a corresponding increase in depth of field. Similarly, the number of spaced depth planes used to present different images to the viewer may be reduced with a reduced pupil size. For example, the viewer may not be able to clearly perceive the details of both the first and second depth planes at one pupil size without adjusting the accommodation of the eye from one depth plane to the other. However, these two depth planes may be sufficient to simultaneously focus the user at another pupil size without changing accommodation.

[0061] In some embodiments, the display system may vary the number of waveguides that receive image information based on a determination of pupil size and / or orientation, or in response to receiving an electrical signal indicative of a particular pupil size and / or orientation. For example, if the user's eye is unable to distinguish between two depth planes associated with two waveguides, the controller 460 may be configured or programmed to stop providing image information to one of those waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments in which the DOE for a waveguide is switchable between on and off states, the DOE may be switched to the off state when the waveguide receives image information.

[0062] In some embodiments, it may be desirable to have the exit beam meet a condition that it has a diameter less than the diameter of the viewer's eye. However, meeting this condition may be difficult in light of the variability in the size of the viewer's pupil. In some embodiments, this condition is met over a wide range of pupil sizes by varying the size of the exit beam in response to a determination of the size of the viewer's pupil. For example, as the pupil size decreases, the size of the exit beam may also decrease. In some embodiments, the exit beam size may be varied using a variable aperture.

[0063] The wearable system 400 may include an outwardly facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the field of view (FOV), and the imaging system 464 is sometimes referred to as the FOV camera. The entire area available for viewing or imaging by a viewer may be referred to as the ocular field of view (FOR). The FOR may include a solid angle of 4π steradians surrounding the wearable system 400 as the wearer moves their body, head, or eyes to perceive virtually any direction in space. In other situations, the wearer's movements may be more constrained, and accordingly, the wearer's FOR may subtend a smaller solid angle. Images obtained from the outwardly facing imaging system 464 may be used to track gestures (e.g., hand or finger gestures) made by the user, detect objects in the world 470 in front of the user, etc.

[0064] The wearable system 400 may also include an inwardly facing imaging system 466 (e.g., a digital camera) that observes user movements, such as eye and face movements. The inwardly facing imaging system 466 may be used to capture images of the eye 410 and determine the size or orientation of the pupil of the eye 304. The inwardly facing imaging system 466 may be used to obtain images for use in determining the direction the user is looking (e.g., eye pose) or for biometric identification of the user (e.g., via iris identification). In some embodiments, at least one camera may be utilized for each eye independently to separately determine the pupil size or eye pose of each eye, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye. In some other embodiments, the pupil diameter or orientation of only a single eye 410 (e.g., using only a single camera per pair of eyes) is determined and assumed to be similar for both eyes of the user. Images obtained by the inward-facing imaging system 466 may be analyzed to determine the user's eye posture or mood, which may be used by the wearable system 400 to determine audio or visual content to be presented to the user. The wearable system 400 may also determine head pose (e.g., head position or head orientation) using sensors such as an IMU, accelerometer, gyroscope, etc.

[0065] The wearable system 400 may include a user input device 466 through which a user may input commands into the controller 460 and interact with the wearable system 400. For example, the user input device 466 may include a track pad, a touch screen, a joystick, a multiple degree of freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a wand, a tactile device, a totem (e.g., serving as a virtual user input device), and the like. In some cases, a user may use a finger (e.g., a thumb) to press or swipe on a touch-sensitive input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held by the user's hand during use of the wearable system 400. The user input device 466 may communicate with the wearable system 400 via wired or wireless communication.

[0066] 5 shows an example of an exit beam output by a waveguide. Although one waveguide is shown, it should be understood that other waveguides in the waveguide assembly 480 may function similarly and that the waveguide assembly 480 includes multiple waveguides. Light 520 is launched into the waveguide 432b at the input edge 432c of the waveguide 432b and propagates within the waveguide 432b by TIR. At the point where the light 520 impinges on the DOE 432a, a portion of the light exits the waveguide as an exit beam 510. Although the exit beams 510 are shown as approximately parallel, they may also be redirected to propagate to the eye 410 at an angle (e.g., forming a diverging exit beam) depending on the depth plane associated with the waveguide 432b. It should be understood that a nearly collimated exit beam may refer to a waveguide, with light extraction optics that outcouples light and forms an image that appears to be set on the depth plane at a long distance (e.g., optical infinity) from the eye 410. Other waveguides or other sets of light extraction optics may output a more divergent exit beam pattern that would require the eye 410 to accommodate to closer distances and focus on the retina, and would be interpreted by the brain as light from a distance closer to the eye 410 than optical infinity.

[0067] FIG. 6 is a schematic diagram showing an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem used in generating a multifocal volumetric display, image, or light field. The optical system can include a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem. The optical system can be used to generate a multifocal volumetric display, image, or light field. The optical system can include one or more primary planar waveguides 632a (only one is shown in FIG. 6) and one or more DOEs 632b associated with each of at least some of the primary waveguides 632a. The planar waveguides 632b can be similar to the waveguides 432b, 434b, 436b, 438b, 440b discussed with reference to FIG. 4. The optical system may employ a dispersive waveguide device to relay light along a first axis (vertical or Y-axis in the illustration of FIG. 6) and expand the effective exit pupil of the light along the first axis (e.g., Y-axis). The dispersive waveguide device may include, for example, a dispersive planar waveguide 622b and at least one DOE 622a (illustrated by a double dashed line) associated with the dispersive planar waveguide 622b. The dispersive planar waveguide 622b may be similar or the same in at least some respects as the primary planar waveguide 632b, which has a different orientation therefrom. Similarly, the at least one DOE 622a may be similar or the same in at least some respects as the DOE 632a. For example, the dispersive planar waveguide 622b or the DOE 622a may be made of the same material as the primary planar waveguide 632b or the DOE 632a, respectively. The embodiment of the optical display system 600 shown in FIG. 6 can be integrated into the wearable system 200 shown in FIG.

[0068] The relayed and exit pupil expanded light may be optically coupled from the dispersive waveguide device into one or more primary planar waveguides 632b. The primary planar waveguides 632b may relay the light along a second axis (e.g., the horizontal or X-axis in the diagram of FIG. 6) that is preferably orthogonal to the first axis. It should be noted that the second axis may be a non-orthogonal axis to the first axis. The primary planar waveguides 632b may expand the effective exit pupil of the light along its second axis (e.g., the X-axis). For example, the dispersive planar waveguide 622b may relay and expand the light along the vertical or Y-axis and pass the light to the primary planar waveguide 632b, which may relay and expand the light along the horizontal or X-axis.

[0069] The optical system may include one or more colored light sources (e.g., red, green, and blue laser light) 610, which may be optically coupled into a proximal end of a single mode optical fiber 640. The distal end of the optical fiber 640 may be threaded or received through a hollow tube 642 of piezoelectric material. The distal end protrudes from the tube 642 as an unfixed flexible cantilever 644. The piezoelectric tube 642 may be associated with four quadrant electrodes (not shown). The electrodes may be, for example, plated on the outside, outer surface or outer periphery, or diameter of the tube 642. A core electrode (not shown) may also be located in the core, center, inner periphery, or inner diameter of the tube 642.

[0070] For example, drive electronics 650, electrically coupled via wires 660, drive opposing pairs of electrodes to bend the piezoelectric tube 642 independently in two axes. The protruding distal tip of the optical fiber 644 has a mechanical resonant mode. The frequency of the resonance may depend on the diameter, length, and material properties of the optical fiber 644. By oscillating the piezoelectric tube 642 near the first mechanical resonant mode of the fiber cantilever 644, the fiber cantilever 644 may be caused to oscillate and sweep through a large deflection.

[0071] By stimulating resonant vibrations in two axes, the tip of the fiber cantilever 644 is scanned biaxially within an area filling a two-dimensional (2-D) scan. By modulating the intensity of the light source 610 synchronously with the scanning of the fiber cantilever 644, the light emitted from the fiber cantilever 644 forms an image. A description of such a setup is provided in U.S. Patent Publication No. 2014 / 0003762, which is incorporated herein by reference in its entirety.

[0072] The optical coupler subsystem components can collimate the light emitted from the scanning fiber cantilever 644. The collimated light can be reflected by a mirrored surface 648 into a narrow dispersive planar waveguide 622b, which contains at least one diffractive optical element (DOE) 622a. The collimated light propagates perpendicularly (with respect to the view of FIG. 6) along the dispersive planar waveguide 622b by TIR, and in so doing repeatedly intersects with the DOE 622a. The DOE 622a preferably has a low diffraction efficiency. This causes a portion of the light (e.g., 10%) to diffract toward the edge of the larger primary planar waveguide 632b at each point of intersection with the DOE 622a, and a portion of the light to continue on its original trajectory down the length of the dispersive planar waveguide 622b via TIR.

[0073] At each point of intersection with DOE 622a, additional light can be diffracted towards the entrance of primary planar waveguide 632b. By splitting the incoming light into multiple outcoupled sets, the exit pupil of the light can be vertically expanded by DOE 4 in dispersive planar waveguide 622b. This vertically expanded light outcoupled from dispersive planar waveguide 622b can enter the edge of primary planar waveguide 632b.

[0074] Light entering the primary waveguide 632b can propagate horizontally (with respect to the view of FIG. 6) along the primary waveguide 632b via TIR. The light propagates horizontally along at least a portion of the length of the primary waveguide 632b via TIR as it intersects with the DOE 632a at multiple points. The DOE 632a can be designed or configured to have a phase profile that is the sum of a linear diffraction pattern and a radially symmetric diffraction pattern, producing both deflection and focusing of the light. The DOE 632a can advantageously have a low diffraction efficiency (e.g., 10%) such that only a portion of the light of the beam is deflected toward the viewer's eye at each intersection of the DOE 632a, while the remainder of the light continues to propagate through the primary waveguide 632b via TIR.

[0075] At each point of intersection between the propagating light and the DOE 632a, a portion of the light is diffracted toward the adjacent face of the primary waveguide 632b, allowing the light to escape the TIR and emerge from the face of the primary waveguide 632b. In some embodiments, the radially symmetric diffraction pattern of the DOE 632a additionally imparts a focal level to the diffracted light, both shaping (e.g., imparting curvature) the optical wavefronts of the individual beams and steering the beams to angles that match the designed focal level.

[0076] These different paths can accordingly couple light out of the primary planar waveguide 632b by resulting in different fill patterns at the exit pupil, focus levels, and / or multiplicities of the DOE 632a at different angles. Different fill patterns at the exit pupil can be advantageously used to generate light field displays with multiple depth planes. Each layer or set of layers (e.g., three layers) in the waveguide assembly or stack may be employed to generate individual colors (e.g., red, blue, green). Thus, for example, a first set of three adjacent layers may be employed to generate red, blue, and green light at a first focal depth. A second set of three adjacent layers may be employed to generate red, blue, and green light at a second focal depth, respectively. Multiple sets may be employed to generate full 3D or 4D color image light fields with various focal depths.

[0077] (Other components of the wearable system) In many implementations, the wearable system may include other components in addition to or as an alternative to the components of the wearable system described above. The wearable system may include, for example, one or more tactile devices or components. The tactile devices or components may be operable to provide a sense of touch to the user. For example, the tactile device or component may provide a sense of pressure and / or texture upon touching the virtual content (e.g., a virtual object, virtual tool, other virtual structure). The haptics may replicate the sensation of a physical object that the virtual object represents, or may replicate the sensation of an imaginary object or character (e.g., a dragon) that the virtual content represents. In some implementations, the tactile device or component may be worn by the user (e.g., a user-wearable glove). In some implementations, the tactile device or component may be held by the user.

[0078] A wearable system may include one or more physical objects that can be manipulated by a user, for example, to enable input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as, for example, pieces of metal or plastic, walls, table surfaces, etc. In some implementations, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, a totem may simply provide a physical surface, and the wearable system may render a user interface to appear to the user on one or more surfaces of the totem. For example, the wearable system may render an image of a computer keyboard and trackpad to appear to reside on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and virtual trackpad to appear on the surface of a thin rectangular plate of aluminum that serves as the totem. The rectangular plate itself does not have any physical keys or trackpads or sensors. However, the wearable system may detect user manipulation or interaction or touch with the rectangular plate as a selection or input made via a virtual keyboard or virtual trackpad. User input device 466 (shown in FIG. 4) may be an embodiment of the totem, which may include a trackpad, touchpad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. A user may use the totem alone or in combination with posture to interact with the wearable system and / or other users.

[0079] Examples of tactile devices and totems usable with the wearable devices, HMDs, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated by reference in its entirety.

[0080] Exemplary Wearable Systems, Environments, and Interfaces The wearable system may employ various mapping-related techniques to achieve a high depth of field in the rendered light field. When mapping a virtual world, it is advantageous to capture all features and points in the real world and accurately depict the virtual objects in relation to the real world. To this end, FOV images captured from a user of the wearable system can be added to the world model by including new photos that convey information about various points and features of the real world. For example, the wearable system can collect a set of map points (such as 2D or 3D points), find new map points, and render a more accurate version of the world model. The world model of the first user can be communicated (e.g., via a network such as a cloud network) to a second user so that the second user can experience the world surrounding the first user.

[0081] 7 is a block diagram of an example of a MR environment 700. The MR environment 700 may be configured to receive inputs (e.g., visual inputs 702 from a user's wearable system, stationary inputs 704 such as a room camera, sensory inputs 706 from various sensors, gestures, totems, eye tracking, user inputs from user input devices 504, etc.) from one or more user wearable systems (e.g., wearable system 200 or display system 220) or stationary room systems (e.g., room cameras, etc.). The wearable systems can use various sensors (e.g., accelerometers, gyroscopes, temperature sensors, movement sensors, depth sensors, GPS sensors, inward-facing imaging systems, outward-facing imaging systems, etc.) to determine the location and various other attributes of the user's environment. This information may be further supplemented with information from stationary cameras in the room, which may provide images from different perspectives or various cues. Image data acquired by cameras (e.g., room cameras or cameras of an outward-facing imaging system) may be reduced to a set of mapping points.

[0082] One or more object recognizers 708 can crawl through the received data (e.g., a collection of points), recognize or map the points, tag the images, and associate semantic information with the objects using a map database 710. The map database 710 may comprise various points and their corresponding objects collected over time. The various devices and the map database may be interconnected through a network (e.g., LAN, WAN, etc.) and accessible to the cloud.

[0083] Based on this information and the set of points in the map database, the object recognizer 708a-708n may recognize objects, complement the objects with semantic information, and bring them to life. For example, if the object recognizer recognizes that the set of points is a door, the system may attach some semantic information (e.g., a door has a hinge and has 90 degree movement around the hinge). If the object recognizer recognizes that the set of points is a mirror, the system may attach the semantic information that a mirror has a reflective surface that can reflect an image of an object in a room. Over time, the map database grows as the system (which may reside locally or be accessible through a wireless network) accumulates more data from the world. Once an object is recognized, the information may be transmitted to one or more wearable systems. For example, the MR environment 700 may include information about a scene generating in California. The environment 700 may be transmitted to one or more users in New York. Based on the data received from the FOV camera and other inputs, an object recognizer and other software components can map points collected from various images, recognize objects, etc., so that the scene can be accurately "passed" to a second user who may be in a different part of the world. The environment 700 may also use a topology map for localization purposes.

[0084] 8 is a process flow diagram of an example embodiment of a method 800 for rendering virtual content in relation to recognized objects. Method 800 illustrates how a virtual scene may be presented to a user of a wearable system. The user may be geographically remote from the scene. For example, a user may be in New York but may wish to view a scene currently occurring in California, or may wish to go for a walk with a friend who is in California.

[0085] In block 810, the AR system may receive input from the user and other users regarding the user's environment. This may be accomplished through various input devices and knowledge already held in the map database. The user's FOV camera, sensors, GPS, eye tracking, etc., communicate information to the system in block 810. The system may determine general points based on this information in block 820. The general points may be used in determining pose data (e.g., head pose, eye pose, body pose, or hand gestures) that may be used in displaying and understanding the orientation and position of various objects in the user's surroundings. The object recognizers 708a-708n may crawl through these collected points and recognize one or more objects using the map database in block 830. This information may then be communicated to the user's individual wearable system in block 840, and the desired virtual scene may be displayed to the user in block 850 as appropriate. For example, a desired virtual scene (eg, a user in CA) may be displayed in the proper orientation, position, etc., relative to various objects and other surroundings of the user in New York.

[0086] FIG. 9 is a block diagram of another example of a wearable system. In this example, the wearable system 900 comprises a map, which may include map data about the world. The map may reside in part locally on the wearable system, or in part in a networked storage location (e.g., in a cloud system) accessible by a wired or wireless network. An attitude process 910 may run on the wearable computing architecture (e.g., processing module 260 or controller 460) and utilize data from the map to determine the position and orientation of the wearable computing hardware or the user. The attitude data may be calculated from data collected on the fly as the user experiences the system and operates within the world. The data may comprise images, data from sensors (such as inertial measurement units, which generally include accelerometer and gyroscope components), and surface information about objects in the real or virtual environment.

[0087] The rough point representation may be the output of a simultaneous localization and mapping (SLAM or V-SLAM, which refers to configurations where the input is image / vision only) process. The system can be configured to find what the world is made of, not just the locations of various components within the world. Poses may be building blocks that accomplish many goals, including filling in maps and using data from maps.

[0088] In one embodiment, the rough point locations may not be entirely adequate by themselves, and further information may be required to generate a multifocal AR, VR, or MR experience. A dense representation, generally referring to depth map information, may be utilized to fill this gap, at least in part. Such information may be calculated from a process called stereoscopic vision 940, where depth information is determined using techniques such as triangulation or time-of-flight sensing. Image information and active patterns (such as infrared patterns generated using an active projector) may serve as inputs to the stereoscopic vision process 940. A significant amount of depth map information may be fused together, some of which may be summarized using a surface representation. For example, a mathematically definable surface may be an efficient (e.g., compared to a large-scale point cloud) and applicable input to other processing devices such as a game engine. Thus, the outputs of the stereoscopic vision process (e.g., depth map) 940 may be combined in a fusion process 930. The pose may also be input to this fusion process 930, whose output becomes the input to populate the map process 920. Sub-surfaces may connect to each other to form larger surfaces, such as in topographical mapping, and the map becomes a large-scale hybrid of points and surfaces.

[0089] Various inputs may be utilized to resolve various aspects in the mixed reality process 960. For example, in the embodiment depicted in FIG. 9, game parameters may be inputs to determine that a user of the system is playing a monster battle game with one or more monsters in various locations, whether the monster is dead, fleeing under various conditions (such as when the user shoots the monster), walls or other objects in various locations, and the like. A world map may contain information about where such objects are relative to each other, which is another useful input for mixed reality. Orientation relative to the world is an input as well, and plays an important role for nearly any interactive system.

[0090] Controls or inputs from the user are another input to the wearable system 900. As described herein, user inputs can include visual inputs, gestures, totems, audio inputs, sensory inputs, and the like. To move around or play a game, for example, the user may need to command the wearable system 900 as to what they want to do. There are various forms of user control that can be utilized beyond just moving around in space themselves. In one embodiment, a totem (e.g., a user input device), or an object such as a toy gun, may be held by the user and tracked by the system. The system would preferably be configured to know that the user is holding an item and understand the type of interaction the user is having with the item (e.g., if the totem or object is a gun, the system may be configured to understand not only the location and orientation, but also whether the user is clicking a trigger or other sensitive button or element, which may be equipped with sensors such as an IMU that can help determine what is happening even when such activity is not within the field of view of any of the cameras).

[0091] Hand gesture tracking or recognition may also provide input information. The wearable system 900 may be configured to track and interpret hand gestures to gesture left or right, stop, grab, hold, etc. for a button press. For example, in one configuration, a user may want to flip through email or calendar in a non-gaming environment, or to do a "fist bump" with another person or performer. The wearable system 900 may be configured to utilize a minimal amount of hand gestures, which may or may not be dynamic. For example, gestures may be simple static gestures, such as spreading the hand to indicate stop, thumbs up to indicate OK, thumbs down to indicate not OK, or flipping the hand left and right or up and down to indicate a directional command.

[0092] Eye tracking is another input (e.g., to track where the user is looking, control display technology, render at a specific depth or range). In one embodiment, eye vergence may be determined using triangulation, and then accommodation may be determined using a vergence / accommodation model developed for that particular person.

[0093] With regard to the camera system, the exemplary wearable system 900 shown in FIG. 9 may include three pairs of cameras: a relative wide FOV or passive SLAM pair of cameras arranged on either side of the user's face, and a different pair of cameras oriented in front of the user to handle the stereoscopic imaging process 940 and also capture hand gestures and totem / object trajectories in front of the user's face. The FOV cameras and pair of cameras for the stereo process 940 may be part of the outward facing imaging system 464 (shown in FIG. 4). The wearable system 900 may include an eye tracking camera (which may be part of the inward facing imaging system 462 shown in FIG. 4) oriented towards the user's eyes to triangulate eye vectors and other information. The wearable system 900 may also include one or more textured light projectors (such as infrared (IR) projectors) to inject texture into the scene.

[0094] FIG. 10 is a process flow diagram of an example of a method 1000 for determining user input to a wearable system. In this example, a user may interact with a totem. A user may have multiple totems. For example, a user may have one totem designated for social media applications, another totem for playing games, etc. In block 1010, the wearable system may detect the movement of the totem. The movement of the totem may be recognized through an outward facing system or may be detected through sensors (e.g., tactile gloves, image sensors, hand tracking devices, eye tracking cameras, head pose sensors, etc.).

[0095] Based at least in part on the detected gestures, eye poses, head poses, or inputs through the totem, the wearable system detects the position, orientation, and / or movement of the totem (or the user's eyes or head or gestures) relative to a reference frame at block 1020. The reference frame may be a set of map points based on which the wearable system translates the totem's (or user's) movements into actions or commands. At block 1030, the user's interactions with the totem are mapped. Based on the mapping of the user interactions to the reference frame 1020, the system determines the user input at block 1040.

[0096] For example, a user may move a totem or physical object back and forth, turn a virtual page, indicate moving to the next page, or move from one user interface (UI) display screen to another UI screen. As another example, a user may move their head or eyes to look at different real or virtual objects in the user's FOR. If the user's gaze at a particular real or virtual object is longer than a threshold time, that real or virtual object may be selected as the user input. In some implementations, the vergence-divergence movement of the user's eyes can be tracked and an accommodation / vergence-divergence movement model can be used to determine the accommodation state of the user's eyes, which provides information about the depth plane the user is focusing on. In some implementations, the wearable system can use ray casting techniques to determine real or virtual objects that are in line with the direction of the user's head pose or eye pose. In various implementations, the ray casting techniques can include casting a thin bundle of rays with substantially little lateral width, or casting a ray (e.g., a cone or a truncated cone) with substantial lateral width.

[0097] The user interface may be projected by a display system as described herein (such as display 220 in FIG. 2). It may also be displayed using a variety of other techniques, such as one or more projectors. A projector may project an image onto a physical object, such as a canvas or a sphere. Interactions with the user interface may be tracked using one or more cameras on or part of the system (e.g., using inwardly facing imaging system 462 or outwardly facing imaging system 464).

[0098] 11 is a process flow diagram of an example method 1100 for interacting with a virtual user interface. Method 1100 may be performed by a wearable system described herein.

[0099] In block 1110, the wearable system may identify a particular UI. The type of UI may be provided by the user. The wearable system may identify that a particular UI needs to be populated based on user input (e.g., gestures, visual data, audio data, sensory data, direct commands, etc.). In block 1120, the wearable system may generate data for a virtual UI. For example, data associated with the UI's boundaries, general structure, shape, etc. may be generated. In addition, the wearable system may determine map coordinates of the user's physical location so that the wearable system may display the UI in relation to the user's physical location. For example, if the UI is body-centered, the wearable system may determine coordinates of the user's physical standing position, head pose, or eye pose so that a ring UI may be displayed around the user or a planar UI may be displayed on a wall or in front of the user. If the UI is hand-centered, map coordinates of the user's hand may be determined. These map points may be derived through data received through the FOV camera, sensory input, or any other type of collected data.

[0100] In block 1130, the wearable system may send data from the cloud to the display, or data may be sent from a local database to the display component. In block 1140, a UI is displayed to the user based on the sent data. For example, a light field display may project the virtual UI into one or both of the user's eyes. Once the virtual UI is generated, the wearable system may simply wait for commands from the user and generate more virtual content on the virtual UI in block 1150. For example, the UI may be a body-centered ring around the user's body. The wearable system may then wait for a command (gesture, head or eye movement, input from a user input device, etc.) and if recognized (block 1160), the virtual content associated with the command may be displayed to the user (block 1170). As an example, the wearable system may wait for a user's hand gesture before mixing multiple stem tracks.

[0101] Additional examples of AR systems, UIs, and user experiences (UX) are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated by reference in its entirety.

[0102] (Example objects in the FOR and FOV) FIG. 12 diagrammatically illustrates an example of a virtual object in a field of view (FOV) and a virtual object in a field of view (FOR). As discussed with reference to FIG. 4, the FOR comprises a portion of the user's surrounding environment perceivable by the user via the wearable system. In FIG. 12, the FOR 1200 may contain a group of objects (e.g., 1210, 1220, 1230, 1242, and 1244) that may be perceived by the user via the wearable system. The objects in the user's FOR 1200 may be virtual and / or physical objects. For example, the user's FOR 1200 may include physical objects such as chairs, sofas, walls, etc. The virtual objects may include operating system objects such as, for example, a trash can for deleted files, a terminal for entering commands, a file manager for accessing files or directories, icons, menus, applications for audio or video streaming, notifications from the operating system, etc. The virtual objects may also include objects in applications such as, for example, avatars, virtual objects in games, graphics or images, etc. Some virtual objects can be both operating system objects and objects within an application. In some embodiments, the wearable system can add virtual elements to existing physical objects. For example, the wearable system may add a virtual menu associated with a television in a room, and the virtual menu may give the user options to turn on the television or change the channel using the wearable system.

[0103] A virtual object may be a three-dimensional (3D), two-dimensional (2D), or one-dimensional (1D) object. For example, as illustrated diagrammatically in FIG. 16, a virtual object may be a 3D coffee mug 1636 (which may represent a virtual control for a physical coffee maker). The virtual object may also be a 2D graphical representation of a wall clock 1634 (which displays the user's current time). In some implementations, one or more virtual objects may be displayed within (or associated with) another virtual object. For example, with reference to FIG. 13, a virtual coffee mug 1636 is shown inside the user interface plane 1514, but the virtual coffee mug appears to be 3D within this 2D planar virtual space.

[0104] The objects in the user's FOR can be part of a world map as described with reference to FIG. 9. Data associated with the objects (e.g., location, semantic information, properties, etc.) can be stored in various data structures, such as, for example, arrays, lists, trees, hashes, graphs, etc. An index for each stored object may be determined, for example, by the location of the object, if applicable. For example, the data structure may index the object by a single coordinate, such as the distance of the object from a reference position (e.g., distance to the left or right of the reference position, distance from the top or bottom of the reference position, or depth from the reference position). The reference position may be determined based on the position of the user (such as the position of the user's head). The reference position may also be determined based on the position of a virtual or physical object (such as a target interactable object) in the user's environment. In this way, the 3D space in the user's environment may be collapsed into a 2D user interface, with the virtual objects arranged according to the distance of the objects from the reference position.

[0105] Within the FOR 1200, the portion of the world that the user perceives at a given time is referred to as the FOV 1250 (e.g., the FOV 1250 may encompass the portion of the FOR that the user is currently looking at). In FIG. 12, the FOV 1250 is diagrammatically illustrated by a dashed line 1252. A user of the wearable system may perceive multiple objects within the FOV 1250, such as object 1242, object 1244, and a portion of object 1230. The FOV may depend on the size or optical properties of the display of the wearable device. For example, an AR display may include optics that only provide AR functionality when the user looks through a particular portion of the display. The FOV 1250 may correspond to a solid angle perceivable by the user when looking through an AR display, such as, for example, a stacked waveguide assembly 480 (FIG. 4) or a planar waveguide 600 (FIG. 6).

[0106] As the user's pose (e.g., head pose or eye pose) changes, the FOV 1250 changes correspondingly, and objects within the FOV 1250 may also change. For example, the map 1210 is initially outside the user's FOV in FIG. 12. When the user looks toward the map 1210, the map 1210 may move into the user's FOV 1250 and (for example) the object 1230 may move outside the user's FOV 1250. As will be described herein, the wearable system may track objects within the FOR 1200 and objects within the FOV 1250.

[0107] (Example of an Interactive Object) 12, a user can interact with a subset of objects in the user's FOR 1200. This subset of objects may sometimes be referred to as interactable objects. In some implementations, the interactable objects may encompass all objects (virtual and physical) in the user's environment, while in other implementations, the interactable objects may include only a portion of the objects in the user's environment.

[0108] The wearable system can identify a subgroup of interactable objects (e.g., 1242, 1244, and 1230) that are within the user's FOV 1250. The subgroup of interactable objects within the FOV are sometimes also referred to as selectable objects because the user currently perceives them and can select them (e.g., move them, activate them, get information about them, etc.). As discussed herein, the user's FOV can change as the user moves their body, head, or eyes. In general, some objects will remain within the FOV, some objects will move from inside the FOV to outside (and will no longer be selectable), and other objects that were outside the FOV will move into the FOV (and become selectable). In response, the wearable system can update the subgroup of interactable objects within the FOV based on the user's body, head, or eye pose.

[0109] The wearable system may identify a target interactable object within the user's FOV. The target interactable object may be an object that the user desires to interact with or an object that the wearable system expects the user will interact with (e.g., the interactable object the user is looking at or the interactable object closest to the center of the user's FOV). The target interactable object may be identified using various rules, such as object location, user preference, or user posture. For example, the wearable system may select the object closest to the center of the FOV to be the target interactable object. The wearable system may also select the left-most or right-most object in the user's FOV to be the target interactable object. As another example, the wearable system may determine the direction of the user's gaze using an inward-facing imaging system 462 (shown in FIG. 4 ), alone or in combination with an IMU. The wearable system may identify an object that collides with the direction of the user's gaze as the target interactable object.

[0110] In some implementations, the AR system can automatically orient the target interactable object so that the normal of the interactable object faces the user. For example, the virtual TV screen may initially face upward toward the ceiling of the room. Once the AR system determines that the user is looking toward the virtual TV screen, the AR system can automatically rotate the virtual TV screen so that it faces the user.

[0111] (Example of focus indicator) The wearable system may assign a focus indicator to the target interactable object so that the user can more easily perceive the target interactable object. The focus indicator can be displayed to the user. For example, the focus indicator can comprise a halo, a color, a change in perceived size or depth (e.g., causing the target object to appear closer and / or larger when selected), or other visual effects that draw the user's attention. The focus indicator can also include audible or tactile effects, such as vibrations, ring tones, beeps, etc.

[0112] In some embodiments, the wearable system may initially identify an object as the target interactable object based on the rules described herein, and then change the target interactable object to another object based on a change in the user's posture, such that the focus indicator may move from one object to the other as the user changes their posture.

[0113] The wearable system may also display a cursor that corresponds to the user's current position. The cursor may take various shapes, such as a geometric cone, a beam of light, a crosshair, an arrow, an oval, a circle, a polygon, or other 1D, 2D, or 3D shapes. The cursor may be presented in the same form as the focus indicator. For example, the cursor may have the same visual, audio, or tactile effects as the focus indicator. As an example, the cursor may be a crosshair that corresponds to the user's head position. As another example, the cursor may have an arrow shape that corresponds to the current position associated with the user input device. As the user changes its posture or activates the user input device, the cursor may move accordingly. The cursor may point to one or more objects or empty spaces in the user's environment as the user moves. For example, referring to FIG. 16, the AR system may move a cursor on the virtual user interface 1514, such as from position 1620 to position 1624 or from position 1622 to position 1620.

[0114] The wearable system may present a cursor in addition to or as an alternative to the focus indicator. For example, in FIG. 15, the wearable system may either display crosshairs (which may correspond to the user's direction of gaze) or provide a light blue halo as a focus indicator, or both, on the virtual object 1514. In some implementations, the cursor is an embodiment of the focus indicator. For example, when the user is gazing at a virtual object, the wearable system may present a halo around the virtual object, a crosshair object, or an arrow on the object. These visual indications may represent both the target object the user is interested in interacting with and the user's current location.

[0115] (Example interactions with interactable objects) Through the wearable system, the user can interact with interactable objects in the user's FOR 1200, and in particular, interactable objects in the user's current FOV 1250. For example, virtual object 1230 may be a graph showing changes in stock prices over time. By selecting virtual object 1230, the user may interact with virtual object 1230 to, for example, get stock quotes, buy or sell stocks, get information about companies, etc. To perform these interactions, the wearable system may display menus, toolbars, etc. associated with the virtual object that may enable the user to perform various actions (e.g., get stock quotes).

[0116] A user can interact with objects within its FOV using various techniques, for example, by selecting an object, moving an object, opening a menu or toolbar associated with the object, or choosing a new set of selectable objects. A user may use hand gestures to interact with objects and actuate a user input device (e.g., see user input device 466 in FIG. 4 ), for example, by clicking a mouse, tapping on a touchpad, swiping on a touchscreen, hovering over or touching a capacitive button, pressing a key on a keyboard or game controller (e.g., a 5-way D-pad), pointing a joystick, wand, or totem toward an object, pressing a button on a remote control, or other interaction with a user input device. A user may also interact with interactable objects using head, eye, hand, foot, or other body postures, for example, by gazing at or pointing with an arm at an object for a period of time, tapping a foot, blinking an eye a number of times for a threshold time interval, etc. These hand gestures and user postures on the user input device can cause the AR system to initiate a selection event, for example, causing a user interface action to be taken (such as displaying a menu associated with the target interactable object, or performing a game action on an avatar in a game).

[0117] In response to initiation of a selection event, the AR system can assign a focus indicator to a target interactable object within the user's FOV using the rules described herein. For example, in FIG. 12, the AR system may assign a focus indicator to object 1244 because it is closest to the center point of the FOV.

[0118] During a selection event, the user can change the target interactable object using various hand gestures described herein. For example, in FIG. 12, the user can swipe left on the touch screen, which can cause the AR system to change the target interactable object from object 1244 to object 1230. The AR system can also transition the visible focus indicator from object 1244 to object 1230 in response.

[0119] In some implementations, the hand gesture may cause the AR system to update the list of selectable objects within the user's FOV. For example, in FIG. 12, when the user swipes right, the AR system may move object 1210 into the user's FOV and move object 1230 out of the user's FOV. The AR system may also update the target interactable objects based on the new group of selectable objects. For example, after object 1210 is moved into the FOV, the system may change the target interactable object from object 1244 to object 1242.

[0120] The user may confirm selection of the target interactable object using hand gestures or postures as discussed herein. The user's action confirming selection of the target interactable object may be the same or a different action than that used to initiate the selection event. The AR system may change the focus indicator when the user confirms the selection, for example, by changing the color, brightness, or shape of the focus indicator.

[0121] A user may perform a selection of a series of user interface actions on the target interactable object. These actions may sometimes be referred to as interaction events. Interaction events may include, for example, resizing an interactable object, displaying a menu of interactable objects, browsing a menu, selecting an item on a menu, searching for an item, playing a game, watching a video, conducting a conference call, previewing the target interactable object, etc. Interaction events may occur in parallel or sequentially with a selection event. In some implementations, an interaction event may be part of a selection event.

[0122] In some embodiments, once a selection event is initiated, the wearable system may "lock" the user's FOV such that the wearable system will stop updating the group of selectable objects in the user's FOV even if the user's FOV changes after the selection event is initiated. In some implementations, the user can still transition the focus indicator between selectable objects in the user's FOV through actuation or pose changes of the user input device.

[0123] A selection event can be terminated by user input or other interaction with the wearable system. For example, a selection event can be terminated by confirming selection of a target interactable object, initiating an interaction event, actuating a user input device to end the selection event, determining a change in head or body posture that has the effect of ending the selection event, etc.

[0124] (Example of selecting a virtual object in 3D space using hand gestures) A user may target and select an interactable object by actuating a user input device. FIG. 13A is an example of selecting an interactable object using a touch gesture on a touch screen 1310 of a user input device 1300. The user input device may be an embodiment of the user input device 466 shown in FIG. 4. The touch gesture can trigger the wearable system to assign a focus indicator to the target interactable object within the user's FOV. The touch gestures, alone or in combination, can also cause the wearable system to initiate a selection event, initiate an interaction event, end a selection event, end an interaction event, confirm selection of the target interactable object, etc.

[0125] 13B is an example of filtering selectable objects using hand gestures on a user input device. A user may swipe along a path on a user input device 1300. For example, as indicated by arrow 1314 in FIG. 13B, a user may swipe along a path toward the right on a touch screen 1310. Any type of path can be used (e.g., horizontal, vertical, diagonal, or other trajectory relative to the input device) or any type of direction can be used (e.g., left or right, up or down, etc.).

[0126] A swipe gesture can cause the wearable system to move a visible focus indicator from one object to another. With reference to the example shown in FIG. 12, when a user swipes rightward (as shown in the example in FIG. 13B), the AR system can transition the focus indicator from object 1244 to object 1242. In some embodiments, a swipe gesture can cause the AR system to update a list of selectable objects within the user's FOV. For example, when a user swipes rightward, the AR system can move object 1210 into the user's FOV and move object 1230 out of the user's FOV. Objects that receive a visible focus indicator can also be updated accordingly (e.g., from object 1244 to 1242).

[0127] Swipe gestures may be used in combination with touch gestures (described with reference to FIG. 13A) and head poses (described with reference to FIG. 14) to filter and select 3D virtual objects within the FOR or FOV. As discussed herein, the user may also use other gestures or movements to change the set of selectable objects and change the target interactable object.

[0128] (Example of selection of virtual objects in 3D space using head pose) FIG. 14 is an example of a coordinate system for head pose. The head 1410 may have multiple degrees of freedom. As the head 1410 moves towards different directions, the head pose will change relative to the natural rest direction 1420. The coordinate system in FIG. 14 shows three angular degrees of freedom (e.g., yaw, pitch, and roll) that can be used to measure the head pose relative to the natural rest state 1420 of the head. As illustrated in FIG. 14, the head 1410 can tilt forward and backward (e.g., pitch), turn left and right (e.g., yaw), and tilt laterally (e.g., roll). In other implementations, other techniques or angle representations for measuring head pose can also be used, for example, any other type of Euler angle system.

[0129] The wearable system can determine the user's head pose using various sensors described herein (e.g., with reference to Figures 2, 4, and 7). For example, the wearable system can use an IMU or an inward-facing imaging system to calculate the user's head pose or eye pose. The wearable system can use data acquired by these sensors to identify a target interactable object. For example, the target interactable object may be an object that collides with the direction of the user's gaze. The wearable system can identify the target interactable object and assign a visible focus indicator to the target interactable object based on the gaze extended toward a certain direction (e.g., a focus indicator is assigned to the target object if the user looks at the object for longer than a threshold time).

[0130] The wearable system can determine and update the object that becomes the target interactable object based on changes in head pose, such as roll, yaw, or pitch during a selection event. For example, with reference to Figure 12, when a user pivots their head 1410 to the left, the target interactable object may be updated from object 1244 to its neighboring object, such as object 1230. In some implementations, the wearable system can transition the focus indicator from object 1244 to object 1230 to reflect this update.

[0131] The wearable system can also update the list of selectable objects within the user's FOV based on changes in head pose. For example, the user may turn head 1410 to the right, which can cause the wearable system to move object 1230 out of the user's FOV and move object 1210 into the user's FOV 1250. The target interactable objects can also be updated accordingly, using the rules described herein.

[0132] A user may use various changes in head or eye pose to switch between multiple planes. A user may also use head or eye pose variations to initiate a selection event, confirm selection of a target interactable object, initiate an interaction event, interact with a target interactable object during an interaction event, terminate a selection event, or otherwise interact with the user interface.

[0133] In some implementations, the wearable system can associate a depth plane with a user's head movement so that the user can interact with virtual objects only in that depth plane. For example, the wearable system can set a cursor (e.g., crosshairs) corresponding to the user's head position in a depth plane. As a result, as the user moves his / her head, the cursor is translated within the set depth plane, and the user can select between objects in the set depth plane even if other virtual objects exist in different depth planes. In some implementations, the depth plane may accompany a virtual user interface. The wearable system can set the crosshairs in a target virtual user interface depth plane so that the user can interact with virtual objects in the target user interface plane.

[0134] (Example interactions using a combination of head poses and hand gestures on a user input device) The user can also select a virtual object using a combination of the user's posture and the user's hand gestures. As an example, the user can perceive a group of virtual objects in the user's FOV. This group of virtual objects may be an embodiment of the selectable objects described herein. The group of virtual objects may be presented using a light field display as described with reference to Figs. 2, 4-6. The light field display can project virtual objects in different depth planes to the user, such that some virtual objects may appear in front of another virtual object.

[0135] The wearable system may maintain an array of virtual objects that are currently within the user's FOV. The wearable system may use the position of the virtual object in the user's environment as an array index for the virtual object. For example, the wearable system may use the y-value of the virtual object in xyz coordinates (e.g., the xyz coordinates shown in FIG. 6) as the array index for the object. In other embodiments, the wearable system may use the x-value or z-value, alone or in combination with the y-value, to determine the array index of the virtual object.

[0136] As the user moves their head or the direction of their gaze, the group of virtual objects that appear within the user's FOV may change. The wearable system may also update the array accordingly. In some implementations, the wearable system may use the array to keep the virtual objects within the user's FOR and to identify groups of virtual objects within the FOV when the user initiates a selection event.

[0137] A user can initiate a selection event on a virtual object in the user's FOV by actuating a user input device. In response to initiation of the selection event, the wearable system can present a user interface that includes all (or a portion) of the virtual object in the user's FOV. In some implementations, the wearable system can display a "hidden virtual object," such as a virtual user interface menu, or some information about the virtual object, etc. A "hidden virtual object" may become perceptible in response to initiation of the selection event, but is hidden prior to or after initiation of the selection event.

[0138] In some embodiments, in response to initiation of a selection event, the wearable system may change the position of the virtual object within the user's FOV. For example, the wearable system may move a distant object closer to the user or a nearby object away from the user so that all virtual objects appear in substantially the same depth plane. In some implementations, the wearable system may change (increase or decrease) the size of the virtual object so that it may fit the size of the user's FOV. Additionally or alternatively, the wearable system may show a portion of the virtual object to the user (such as showing an icon instead of the content of a virtual email application).

[0139] The wearable system may also cluster virtual objects within the user's FOV into multiple depth planes in response to initiation of a selection event, where each depth plane may be associated with a virtual user interface.

[0140] The wearable system can use the array index of the virtual objects to arrange the virtual objects on one or more user interfaces. For example, the wearable system can present virtual objects whose y-axis values ​​are within a certain range together on the same user interface. Additionally or alternatively, the wearable system can arrange the virtual objects on a depth plane based on the user's pose. For example, if there may be multiple virtual objects in the direction of the user's gaze because they are in different depth planes in the user's environment, the wearable system can present these virtual objects inside the user's FOV while placing other virtual objects outside the user's FOV in response to the initiation of a selection event. The user can move virtual objects in and out of the FOV using the techniques described with reference to Figures 12 and 13.

[0141] The wearable system can identify a target interactable object and present a focus indicator indicating the target interactable object. The wearable system can rearrange the virtual objects in the vicinity of the target interactable object and present the rearranged virtual objects in the user's FOV. For example, referring to FIG. 12, the wearable system can identify a group of virtual objects in the vicinity of the target interactable object based on the distance of the virtual objects from a reference position (such as the position of the target interactable object). The wearable system can rearrange the positions of these virtual objects based on values ​​in the array index xyz coordinates (shown in FIG. 6), or distance from the target interactable object, etc. As shown in FIG. 12, objects 1242, 1244, and 1230 may have different initial positions in the user's FOR. For example, object 1242 may be positioned higher (e.g., closer to the ceiling of the user's room) than object 1244 and may be further away from the user than object 1244. The initial position of object 1230 may be lower (e.g., closer to the floor of the user's room) than object 1244 in the user's FOR. When the wearable system identifies object 1244 as the target interactable object, the wearable system may "fold" the user's 3D space into a 2D user interface based on the y-axis value, with the object with the larger y-axis value being positioned to the left of the user's FOV. Accordingly, in FOV 1250, virtual object 1242 appears to the left of virtual object 1244, which is to the left of virtual object 1230. In other embodiments, different techniques for reorganization of virtual objects may be used. For example, FOV 1250 may show a 2D projection of the virtual objects in 3D space. As illustrated in FIG. 12, the user may use hand gestures to transition the focus indicator between objects in the FOV.The user may also use hand gestures to change the target interactable object, which may cause the wearable system to present another set of virtual objects in the FOV based on a reorganization of the virtual objects in the vicinity of the new target interactable object. If the wearable system is configured to present multiple user interfaces in different depth planes, the user may also use hand gestures to transition the focus indicator between the multiple depth planes to switch between user interfaces. In some embodiments, the wearable system may restore the position of the virtual object to its original position prior to the reorganization when the virtual object is no longer within the user's FOV. In some circumstances, a virtual user interface presenting the reorganized virtual objects may be generated after the user initiates a selection event on the target interactable object. Additional details regarding interaction with multiple user interfaces are further described with reference to FIG. 15.

[0142] Example of user interaction based on context information The wearable system can automatically select or recommend a mode of user interaction (e.g., posture or hand gestures on a user input device) based on the context information. The context information can include object type (e.g., physical or virtual), object layout (e.g., object density, object location and size, etc.), user characteristics, or the user's current interaction with objects in the environment, combinations, or the like. For example, during ray casting (described with reference to FIG. 10 ), the wearable system may detect that the user is looking at multiple virtual objects located in close proximity to each other. The wearable system can calculate the density of the virtual objects in the user's FOV. When the density exceeds a certain threshold, the wearable system can recommend to the user to switch the mode of user interaction. For example, when the density exceeds a certain threshold (indicating that the objects are located very close to each other), the wearable system can switch the mode of user interaction from head posture to hand gestures on a user input device to allow for more precise interaction with the objects. As another example, when the density drops below a certain threshold (indicating that objects are far away from one another), the wearable system can switch the mode of user interaction from hand gestures to head pose on the user input device. These implementations can be particularly advantageous because head position can be difficult to control precisely and can tire a user when trying to precisely position their head and interact with densely clustered objects. On the other hand, hand gestures on the user input device can provide finer control of the user's position, but can tire a user when the user needs to move their hands over long distances to select sparsely located objects.

[0143] Figures 15 and 16 provide examples of changing modes of user interaction based on context information. Figure 15 illustrates an example of interaction with an interactable object using head pose. Figure 16 illustrates an example of interaction with an interactable object using hand gestures on a user input device. The user input device 1610 shown in Figure 16 may be an embodiment of the user input device 466 described in Figure 4.

[0144] In FIG. 15, the user's FOR includes interactable objects such as virtual user interfaces 1512, 1514, and 1516. In some implementations, virtual user interfaces 1512, 1514, and 1516 may be planar objects that include other virtual objects (1D, 2D, or 3D) within the virtual user interface plane. Virtual user interfaces 1512, 1514, and 1516 have large sizes and are not densely located adjacent to each other. As a result, the wearable system may determine that head movement may be the optimal mode of user interaction because the user does not need to move a long distance on the user input device to select a virtual user interface. The wearable system may use ray casting techniques to identify that the user is currently looking at virtual user interface 1514. The user may initiate a selection event on virtual user interface 1514 and interact with an object within user interface 1514. As shown in FIG. 16, the virtual user interface 1514 can include multiple virtual objects, such as, for example, a virtual TV screen 1632, a virtual coffee cup 1636, a virtual clock 1634, a camera application 1652, a weather application 1638, and a music application.

[0145] Example of Recommendation of Modes of User Interaction Based on Context Information 16, the wearable system may determine the relative positions between a virtual TV 1632, a virtual clock 1634, a virtual coffee mug 1636, and a weather application 1638. These four objects are in close proximity to one another, so when the wearable system determines that the user's current position is at position 1620, the wearable system may present the user with the option of whether they wish to switch from head pose to hand control.

[0146] On the other hand, object 1652 (a camera application) does not have other objects in its vicinity. Accordingly, the wearable system may not provide an option to interact with object 1652 using a user input device. However, the user may still interact with object 1652 using poses such as head pose and eye pose.

[0147] The user may confirm the switching of the user interaction mode using the user input device or by changing the body posture (such as nodding the head). Once the user interaction mode is switched to the user input device, the user may actuate the user input device 1610 to interact with the virtual object. For example, the user may swipe along a path on the user input device 1610, which transitions the cursor from position 1620 to position 1624. Similarly, the user may actuate the user input device 1610, which transitions the cursor (which may be in the shape of an arrow) from position 1620 to 1622. In addition to these examples, the user may swipe along any type of path on the user input device 1610 (e.g., horizontal, vertical, or diagonal with respect to the input device) or any type of direction (e.g., left or right, up or down, etc.).

[0148] When the cursor is at position 1624, a portion of the cursor overlaps the virtual TV screen 1632. The user may actuate the user input device 1610 (e.g., by clicking a touch screen) to select the virtual TV screen 1632. When the wearable system receives the selection of the virtual TV screen 1632, the wearable system may display one or more virtual menus (e.g., virtual menus 1642a and 1642b) associated with the TV screen. For example, the virtual menus may include options such as, for example, adjusting the volume, selecting a video application (e.g., a movie or television streaming service), selecting to start a conference call, etc.

[0149] (Example of automatic switching of user interaction modes based on context information) The wearable system can also automatically switch the mode of user interaction. For example, the mode of user interaction may be set to head pose when the user selects between the virtual user interface planes 1512, 1514, 1516. Once the user selects the virtual user interface plane, the mode of user interaction may automatically change to hand gestures on the user input device (as shown in FIG. 16). As another example, the mode of user interaction can be set to body pose when the objects are sufficiently sparse or when the layout of the objects meets certain criteria (such as when there is no occlusion between the objects). The wearable system can automatically change the mode of user interaction to hand gestures on the user input device when the objects are located at a high density or when the layout of the objects no longer meets the criteria (such as when one object occludes another object).

[0150] The wearable system may consider the relative positions of objects in 3D space in addition to or as an alternative to the relative positions of objects in 2D space. For example, the user interface 1514 may be a 3D user interface instead of a 2D user interface. As shown in FIG. 16, a weather application 1638 may be located in the depth plane farther from the user than a coffee mug application 1636. When the user's direction of gaze is at position 1620, the wearable system may detect two virtual objects (the virtual clock 1634 and the music application 1654) that intersect with the user's direction of gaze, even though the music application 1654 may appear farther away than the virtual clock 1634. Based on this information, the wearable system may determine that the objects are sufficiently close to each other. In response, the wearable system may automatically switch the mode of user interaction to hand gestures or prompt the user with an option to switch to hand gestures.

[0151] Other Exemplary User Interface Features Based on Context Information In some implementations, the wearable system may reposition the dense cluster of virtual objects to one or more fixed depth planes when the user switches to hand gestures. As described with reference to Figures 12, 13A-13B, the user can select a virtual object in a fixed depth plane or switch depth planes using hand gestures. This implementation may be particularly advantageous because it reduces clutter on the user input device caused by navigating between virtual objects that are each located at slightly different depth planes in 3D space.

[0152] The wearable system may change the focus indicator or cursor, either alone or in combination, when the mode of user interaction changes from one way to another. For example, the wearable system may change the color of the focus indicator when the user changes the mode of user interaction from head pose to hand gestures on the user input device (and vice versa). In another example, the wearable system may change the appearance of the focus indicator from a crosshair shape to an arrow shape (as shown in FIG. 16) and indicate an option to change input control from head pose to hand gestures on the user input device.

[0153] In some implementations, changes in the focus indicator or cursor may be used, alone or in combination, to indicate that an option to change modes of user interaction has become available. For example, while a user interacts with a group of sparsely positioned objects using hand gestures on a user input device, the wearable system may change the appearance of the focus indicator from an arrow to a crosshair to indicate that an option to interact using head pose is available. The user may confirm the change from hand gestures to head pose by, for example, actuating the user input device (such as tapping the user input device) or changing a body posture (such as nodding their head). As another example, the wearable system may provide a vibration on the user input device as a focus indicator to indicate that an alternative mode of user interaction has become available.

[0154] Although the embodiments are described with reference to selecting one object, the wearable system may be configured to identify multiple target objects and select multiple target objects. The AR system may be configured to perform the selection event recursively on a subgroup of target interactable objects. For example, the AR system may identify several target objects that collide with a virtual cone in ray casting (described in FIG. 10). The target objects may include interactable objects. The AR system may magnify these target objects and enable the user to select within these target objects using postures and / or hand gestures described herein. Furthermore, although the embodiments are described with reference to changing between head poses and hand gestures on a user input device, similar techniques can also be used to switch between other modes of user interaction. For example, the wearable system may employ similar techniques to change the mode of user interaction between body postures, hand gestures, head postures, foot postures, eye postures, etc., alone or in combination.

[0155] (Example of an interaction event) A user can initiate an interaction event on an interactable object in its FOV after the user selects the interactable object. In some implementations, the virtual object may correspond to a physical object. As a result, when a user performs an interaction event on a virtual object, the virtual object may communicate with the physical object, thereby allowing the user to interact with the physical object through the virtual user interface. For example, the 3D coffee mug 1636 in FIG. 16 may communicate with a coffee machine in the user's environment. The water level shown in the 3D coffee mug 1636 may represent the coffee making progress. As an example, the water level may initially be invisible because the coffee machine is idle. The user can select the 3D coffee mug 1636 and initiate an interaction event, which causes the wearable system to send a command to the coffee machine in the user's kitchen to start brewing coffee. During the brewing process, the water level in the 3D coffee mug 1636 may gradually increase as the coffee machine produces more coffee. Once the coffee machine has finished brewing, the wearable system may indicate that the 3D coffee mug 1636 is full. In some implementations, the wearable system may also provide a focus indicator (such as a sound or a halo) on the 3D coffee mug to indicate that the coffee has finished brewing.

[0156] As another example, the wearable system may present virtual content associated with physical objects in the user's environment during an interaction event. In FIG. 16, the user may select a weather application 1638 and initiate an interaction event on the weather application 1638. FIG. 17 illustrates an example interaction event with the weather application 1638. In this example, a user of the wearable system may be outside their home and may perceive physical objects such as storm clouds 1710 and tornadoes 1720 in the distance. The user may also perceive other physical objects such as roads 1730, vehicles 1732, and buildings 1734.

[0157] The wearable system can identify the user's location by analyzing images of the user's environment. When the user initiates an interaction event on the weather application 1638 (shown in FIG. 16 ), the wearable system can present an AR / MR scene 1700 superimposed on physical objects in the user's FOR. The wearable system can use geolocation sensors (e.g., Global Positioning System (GPS) sensors) to determine the user's location and information about the weather in the user's vicinity (e.g., the presence of storm clouds 1710 and tornadoes 1720 that the user is viewing). In some implementations, the wearable system can also obtain images of the user's environment using an outward-facing imaging system. The wearable system can use sensors such as the outward-facing imaging system 464, the inward-facing imaging system 462, or an IMU (described in FIG. 4 ), alone or in combination, to determine that the user is looking toward the storms 1710 and tornadoes 1720.

[0158] The wearable system can communicate with a network (wired or wireless) and access information about the storm and display such information to the user as virtual content, such as the presence and duration of a storm watch 1750, the speed of the tornado 1754, the weather forecast 1762 (e.g., temperature, precipitation probability as a function of time), the direction of the storm 1756 (e.g., storm location as a function of time), expected rainfall 1766, etc. This information can be presented to the user via text or graphics (e.g., the virtual content can appear superimposed on the physical content) such that at least a portion of the information is perceived at or near the location of the storm cloud 1710 or tornado 1720. For example, as shown in FIG. 17, arrows 1752 indicate the direction of the storm (e.g., tornado) and appear as if they are 3D and are superimposed on or around the tornado 1720 from the perspective of the user of the wearable system. Wind speed 1754 can be presented near the directional arrows 1752. The wearable system may also present the wearer with other information about the storm, such as, for example, a temperature forecast 1762, where rain is currently falling or is expected to fall (e.g., as shown via a dashed diagonal line 1764 below the storm cloud 1710), the amount of precipitation expected (as shown using reference numeral 1766), and wind direction 1752 and speed 1754 within the storm (e.g., at different altitudes of the tornado 1720).

[0159] The wearable system can present realistic 3D virtual content 1750, 1752, 1754, 1756, 1762, 1764, 1766 to a user of the wearable system using a light field display (as shown in Figures 4-6) so that it appears at an appropriate distance from the user (e.g., on or near the storm 1710 and tornado 1720) and is appropriately sized, shaped, or scaled (e.g., as shown diagrammatically in Figure 17).

[0160] The wearable system may also allow two or more users to interact with the interactable object. Both users may wear their respective wearable systems (such as their head-mounted devices). FIG. 18 illustrates an exemplary user experience of multiple users interacting with a 3D virtual object. In this example, the users are wearing wearable devices 1852. The users can perceive virtual content 1800 through the wearable devices 1852. In this example, the virtual content may include astronomical objects 1810 (e.g., stars, or otherwise galaxies, planets, nebulae, or solar systems). Information about the virtual content 1800 may be displayed to appear on or adjacent to the virtual content. For example, orbits 1820 of stars or planets 1830, constellations, nearby stars, etc. may be displayed adjacent to the astronomical objects 1810. The wearable system can present a user interface to the wearer who may access a virtual menu 1832 or virtual input features 1840 from which different actions may be selected (e.g., by gesture) using posture or a user input device. For example, as shown in FIG. 18, the virtual menu 1832 may allow the wearer to edit / delete or save a profile of the displayed virtual content 1800. The profile may allow the wearer (or another authorized user) to access the virtual content at different times or places. Another virtual menu may allow the user to interact with and modify the displayed virtual content. For example, as shown in FIG. 18, the user input element 1840 may allow the wearer to "add a planet" (e.g., by selecting the virtual button 1840 with a gesture such as "pressing" the wearer's finger at the location of the virtual button 1840). After selection, the wearer may be able to access functionality (e.g., via a virtual menu, a virtual drop-down box, etc.) and create a planet profile with information such as the planet's name, diameter, temperature, or distance.After selection, additional virtual content (in this example, additional planets) can be displayed to the wearer.

[0161] The wearable system may enable a user to share virtual content with others, for example, by passing a world map of the user's environment over a network or by communicating virtual content between wearable systems (or updating virtual content). For example, as shown in FIG. 18, another user 1850 wearing a wearable system 200 may view the virtual content shared and manipulated by the first user. Both users may enjoy the experience of interacting with each other and the virtual content. Although an astronomical object 1810 is used in this example, the virtual content 1800 may be any type of content. For example, an encyclopedia may be accessed and content on one or more subjects may be displayed and shared (with virtual text, virtual images, sounds, etc.). A group of users sharing virtual content need not be physically present in the same location to view the virtual content, and many users (e.g., 2, 3, 4, 5, 10, 100, or more) may view the shared virtual content substantially simultaneously.

[0162] Although examples are described with reference to virtual planetary systems and weather applications, these examples are not limiting. The techniques described herein can also be applied to other applications that present interactive virtual content within an AR / MR / VR environment. For example, a wearable system can be programmed to include a clothing shopping application. While a user is in a department store, the application can access a database associated with the department store and identify clothing information or racks. The application can present the accessed information on virtual content superimposed on physical clothing within the department store.

[0163] (Example process of selecting a virtual object using a combination of pose and user input device) 19 illustrates an example process for selecting a virtual object using a combination of posture and hand gestures on a user input device. Process 1900 can be performed (e.g., by one or both of processing modules 260, 270) by a wearable system (e.g., HMD) described herein. The wearable system may include a user input device (e.g., see user input device 466 in FIG. 4) and various imaging systems, such as an outwardly facing imaging system (e.g., see outwardly facing imaging system 464 in FIG. 4) and an inwardly facing imaging system (e.g., see inwardly facing imaging system 462 in FIG. 4).

[0164] In block 1910, the wearable system can identify interactable objects in the user's FOR. The interactable objects may be stored, for example, in the remote data repository 280 (shown in FIG. 2) using a data structure such as an array. The interactable objects in the user's FOR can be a subset of all objects in the user's FOR. The wearable system can use the outward-facing imaging system 462 and other sensors (such as IMU and GPS) to determine the user's location and use this location information to determine interactable objects in the user's environment.

[0165] In block 1920, the wearable system can measure the user's posture using sensors described herein. The wearable system can use the inward-facing imaging system 462 to identify the user's orientation.

[0166] In block 1930, the wearable system can identify interactable objects within the user's FOV based on the user's direction of gaze. The FOV can also use ray casting to determine interactable objects that intersect with the user's direction of gaze. In some implementations, the interactable objects within the user's FOV can be referred to as selectable objects. The wearable system can use the positions of the selectable objects to store the selectable objects in an array. For example, the wearable system can index the selectable objects (and / or interactable objects) based on x-axis values ​​(e.g., see xyz coordinates in FIG. 6). The wearable system can sort the objects based on the x-axis values ​​and present the selectable objects from left to right within the user's FOV in a 1D or 2D view.

[0167] In some implementations, the wearable system may not have a separate array just for selectable objects, in which the wearable system can, for example, use the array index of the selectable object (determined based on the user's FOV) to identify and retrieve the selectable object from the array of interactable objects.

[0168] However, in some embodiments, the wearable system may maintain two arrays, one for interactable objects in the user's FOR, while another array is for selectable objects in the user's FOV. For example, the array for interactable objects may be kept in the remote data repository 280 (shown in FIG. 2 ), while the array for selectable objects may be kept in the wearable system's local data storage. This implementation may be advantageous because the wearable system's local storage and data processing capacity may be limited, while the remote data repository may have larger data storage and more powerful data processing capacity. Keeping only a subset of all interactable objects in the wearable system's local storage may reduce the hardware requirements of the wearable system and reduce the possibility of data overflow in the wearable system.

[0169] If no interactable objects are present within the user's FOV, the wearable system may return to block 1910. The wearable system may continuously monitor the user's posture, update the user's FOV, and continuously determine the list of interactable objects within the user's FOV.

[0170] In block 1940, the wearable system may determine whether the user desires to initiate a selection event on an object within the user's FOV. The wearable system may make such a determination based on various indications, alone or in combination, such as, for example, prolonged gaze at the object, a user's head pose such as nodding, or input from a user input device. If the wearable system receives an indication that the user desires to select an object, the wearable system may enable the user to interact with the selectable object using various hand gestures described herein. For example, the user may swipe along a trajectory on the user input device to browse the selectable object or click the user input device to select the selectable object.

[0171] Initiation of the selection event may cause the wearable system to identify a target interactable object within the user's FOV (at block 1940) and assign a focus indicator to the target interactable object (at block 1950). The target interactable object may be an object on the left or right side of the FOV. A visible focus indicator may also be placed on an object at the center of the FOV when the object at the center is the target interactable object. The wearable system may also use 3D eye tracking to determine the direction of the user's eye pose and place a visible focus indicator on an object that intersects with the direction of the user's eye pose.

[0172] In block 1960, the user can transition the focus indicator to another object using various gestures described herein. The target interactable object can be updated to the other object accordingly. For example, the user can move the focus indicator from its current location to its neighboring object, resulting in the target interactable object being updated to the neighboring object.

[0173] In some embodiments, these gestures may cause the wearable system to move the virtual object along a path within the user's FOV (instead of shifting the visible focus indicator itself). For example, with reference to Figure 12, a user swiping left on a touchpad may have the effect of moving object 1230 completely into the FOV while shifting the position of object 1242 leftward.

[0174] 20 illustrates an example process for interacting with a virtual object using a combination of postures and hand gestures on a user input device. Process 2000 can be performed by a wearable system described herein.

[0175] In block 2010, the wearable system can determine a group of interactable objects in the user's FOR. The group of interactable objects can be a subset of objects in the user's environment. The interactable objects can be virtual objects and / or physical objects. In some embodiments, the AR system can add virtual elements to existing physical objects. For example, the AR system may add a virtual menu to a television in a room, and the virtual menu may give the user the option to turn on the television using the AR system.

[0176] The AR system may store interactable objects and information associated with the interactable objects in various data structures, as described with reference to Figure 12. The location of the object may be used as an index for storing the information associated with the object.

[0177] In block 2020, the wearable system can determine the user's posture. The user's posture may be head, eye, foot, or other body posture, either alone or in combination. The wearable system can determine the user's posture using various sensors shown in FIG. 4, such as, for example, the inwardly facing imaging system 462, input received on the user input device 466, or the outwardly facing imaging system 464.

[0178] In block 2030, the wearable system can determine the user's FOV based on the user's posture. The FOV can comprise a portion of the FOR perceived by the user at a given time. Based on the user's FOV, in block 2040, the AR system can determine a subgroup of interactable objects within the user's FOV. This subgroup of interactable objects is sometimes also referred to as selectable objects. As the user's FOV changes, the selectable objects within the user's FOV also change.

[0179] In block 2050, the wearable system receives a selection of a target interactable object from the subgroup of interactable objects. The wearable system may first choose a target interactable object based on various rules (such as the location of the target interactable object relative to the user's FOV), as described with reference to FIG. 12. The AR system may assign a focus indicator to the target interactable object. The visible focus indicator may move from one object to another as the user's FOV changes.

[0180] In some embodiments, the wearable system can identify the target interactable object after the user activates the user input device. The user can activate the user input device using various hand gestures described with reference to FIGS. 12-13. These hand gestures can trigger the wearable system to assign a focus indicator to the target interactable object within the user's FOV. In some implementations, once the AR system receives a selection of the target interactable object from the user, the AR system may stop updating the group of selectable objects within the user's FOV while the user's FOV changes. The user can still browse the interactable objects or transition the visible focus indicator from one object to another within the FOV.

[0181] At block 2070, the user may decide to initiate a selection event on the target interactable object. The selection event can be initiated using postures and gestures described herein. In some embodiments, the initiation of the selection event may trigger the wearable system to assign a visible focus indicator to the target interactable object. The wearable system may stop updating the group of selectable objects in the user's FOV, even though the user's FOV may change with changes in the user's posture. The user may transition the visible focus indicator from one object to another object in the user's FOV using gestures described herein. For example, the user may swipe along a trajectory (such as left and right) on a touchpad, which may cause the wearable system to transition the visible focus indicator from one object to its nearest neighbor.

[0182] In some implementations, a user may initiate an interaction event during or after a selection event. An interaction event may also be part of a selection event. For example, as described with reference to FIG. 12, an interaction event may comprise resizing an interactable object, displaying a menu of interactable objects, browsing a menu, selecting an item on a menu, searching for an item, playing a game, watching a video, conducting a conference call, etc. A user may participate in an interaction event using various postures and gestures described herein. A user may also confirm selection of a target interactable object using the postures and gestures discussed herein.

[0183] (An exemplary process of interaction with an object based on context information) 21 illustrates an example process for switching input control from head pose to hand gestures based on context information. Process 2100 can be performed by a wearable system described herein (e.g., by one or both of processing modules 260, 270).

[0184] The wearable system may display a cursor indicating the user's current location. The cursor may be a crosshair that corresponds to the user's head position. As the user moves, the cursor may be moved to the target interactable object. The user may select the target interactable object using posture, hand gestures on the user input device, alone or in combination. In process 2100, the user may first interact with the object using head posture. In block 2110, the wearable system may determine whether the user has selected the target interactable object. The target interactable object may be a 2D planar virtual user interface. If the user has not selected the target interactable object, the process ends at block 2190. In some embodiments, the wearable system may continually determine the user's current location as the user moves. The wearable system may also identify other target interactable objects within the user's FOV based on the user's head posture.

[0185] As shown in block 2120, when the wearable system receives a selection of a target interactable object, such as a 2D planar virtual user interface, the wearable system may assign a focus indicator to the target interactable object. For example, the wearable system may display a halo around the target interactable object and move the target interactable object closer to the user so that the target interactable object appears to be floating in 3D space. The wearable system may also set the normal and depth of the cursor (corresponding to the head position) to be the same as the normal and depth of the target interactable object. As a result, the user may continue to interact with virtual objects within the target interactable object using head pose.

[0186] In block 2130, the wearable system may identify a context associated with the user's interaction. For example, the wearable system may determine the layout of virtual objects (or physical objects) within the user's FOV.

[0187] In block 2140, if the wearable system determines that the layout satisfies a certain pattern (such as one virtual object being occluded by another virtual object) or the density exceeds a certain threshold, the wearable system may provide the user with an option to switch modes of input control. As an example, the wearable system may provide the user with an option to switch interaction modes from head pose to user input device.

[0188] If the user chooses not to switch, the user can still use head pose to target and select interactable objects, as shown in block 2142. The process ends at block 2190.

[0189] If the user elects to switch input control to hand gestures, in block 2150, the user may actuate the user input device to interact with the virtual object. In block 2160, the wearable system may receive the user's selection of a target virtual object, such as a UI element on a 2D planar virtual user interface. For example, referring to FIG. 16, the user may select a weather application 1638, a coffee making application 1636, etc. If the user does not select a target virtual object in block 2160, the user may continue to actuate the user input device, as shown in block 6150.

[0190] The wearable system can initiate a selection or interaction event on the selected virtual object. For example, the wearable system can provide a focus indicator on the selected virtual object. The wearable system can also present a VR / AR / MR scene associated with the selected virtual object. For example, the wearable system can present scenes 1700 (shown in FIG. 17) and 1800 (shown in FIG. 18) with which the user can interact.

[0191] In block 2180, the wearable system may determine whether the user has completed the selection or interaction event. If the wearable system determines that the user has finished interacting with the target virtual object, process 2100 ends in block 2190. In some embodiments, in block 2190, the wearable system may switch the mode of user interface control from hand gestures back to head poses.

[0192] 22 illustrates an example process for switching modes of user interaction based on context information. Process 2200 can be performed by a wearable system described herein (e.g., by one or both of processing modules 260, 270).

[0193] In block 2210, the wearable system may determine a current input mode used by the user to interact with the interactable object. The current input mode may be a posture or a hand gesture on the user input device.

[0194] In block 2220, the wearable system may determine context information such as the layout of objects in the user's FOV, the density of objects in the user's FOV, characteristics of objects in the user's FOV (size, position, object type, etc.).

[0195] Based on the context information, the wearable system can present an option to change the current user input mode in block 2230. For example, while the user interacts with objects using head pose, the wearable system can present an option to change the user input mode to user input device when the wearable system identifies a high density group of objects in the user's line of sight direction. As another example, the wearable system can provide an option to change the current user input mode from user input device to head pose when the wearable system determines that objects are located in a sparse density.

[0196] As another example, the wearable system may allow a user to use a pose to interact with a user interface plane (which may include other virtual objects) while using a user input device to interact with user interface elements (such as applications in the user interface plane). As a result, when the wearable system detects that the user has selected the user interface plane, the wearable system may change the user input mode from head pose to user input device. On the other hand, when the user finishes interacting with the user interface elements, the wearable system may change the user input mode from user input device to head pose.

[0197] In some implementations, the appearance of the focus indicator (including the cursor) may be changed based on different user input modes. For example, the wearable system may use crosshairs to indicate that the user interacts with the object using a head pose, while using an arrow to indicate that the user interacts with the object using a user input device. In some implementations, the wearable system may change the appearance of the focus indicator to indicate that an option to switch user input modes is available. For example, the wearable system may initially display crosshairs when the user interacts with a head pose. When the wearable system detects a dense group of objects, the wearable system may display an arrow (instead of crosshairs) to indicate that the user can switch user input modes to a user input device. However, when the user moves away from the dense group of objects, the wearable system may change the focus indicator from an arrow back to a crosshair.

[0198] In block 2240, the user may select an option for a new user input mode. For example, the user may actuate the user input device and confirm that they wish to change the user input mode from head pose to hand gestures on the user input device. The wearable system may accordingly update the current user input mode to the newly selected mode. The wearable system may also update the focus indicator to one associated with the newly selected mode.

[0199] 23 illustrates an example process for interacting with an interactable object comprising a group of virtual objects. Process 2300 can be performed by a wearable system described herein (e.g., by one or both of processing modules 260, 270). The wearable system may include various sensors, such as a user input device 466 (shown in FIG. 4), a light field display (described with reference to FIGS. 2 or 4-6), an outwardly facing imaging system 464 (shown in FIG. 4) and an inwardly facing imaging system 462 (shown in FIG. 4).

[0200] In block 2310, the wearable system determines the user's posture. The posture may be head, eye, foot, or other body posture, etc. The wearable system can determine the user's posture using various sensors, such as, for example, an inward-facing imaging system 462, an outward-facing imaging system 464 (such as an FOV camera described with reference to FIG. 10), an IMU, etc.

[0201] The wearable system may determine a group of interactable objects in the user's FOR. For example, the wearable system may access a map of the user's environment, comprising information of objects in the user's environment. In block 6230, the wearable system may determine a target interactable object from the group of interactable objects. The target interactable object may be determined based on the user's posture. For example, the target interactable object may be an object that intersects with the user's direction of gaze. As the user's posture changes, the target interactable object may also change.

[0202] In block 2330, the wearable system may receive a selection of a target interactable object. A user may select the target interactable object by actuating a user input device, changing posture, alone or in combination. As shown in block 2340, after the wearable system receives the selection of the target interactable object (as shown in block 6250), it may assign a focus indicator to the target interactable object.

[0203] In some implementations, one or more virtual objects may further be displayed within the target interactable object. For example, the virtual user interface may include user interface elements of a weather application, a video streaming application, etc. In block 2350, the wearable system may determine a group of virtual objects associated with the selected target interactable object. The user may interact with the group of virtual objects using head poses and hand gestures. For example, the user may select a virtual object within the group of virtual objects using a pose or hand gesture. The user may also initiate an interaction event, such as playing a video game, viewing a VR / AR / MR scene, or other user interface interaction with the selected virtual object.

[0204] In some embodiments, the wearable system may change the mode of interaction from pose to user input device when the user selects the target interactable object. For example, the user may initially use head pose to target and select an interactable object in the environment. Once the user selects the target interactable object, the wearable system may change the input mode from head pose to user input device so that the user may use the user input device to interact with virtual objects in the target interactable object. Once the user has finished interacting with the virtual object, the wearable system may change the input mode back to head pose so that the user may continue to target and interact with other interactable objects in the environment. In some implementations, the wearable system may provide an option for the user to switch modes of input control. For example, if the user decides not to switch from head pose to hand gestures after selecting the target interactable object, the user may continue to interact with virtual objects in the target interactable object using head pose.

[0205] (Additional User Interface Experience) (Additional Examples of AR and MR Visual Experiences) As described above, a wearable system (such as a head mounted display) can be configured to present 3D virtual objects superimposed on the physical world. For example, a user of a wearable device may be in a school gym and not only perceive the local physical environment and physical objects (e.g., the gym and students sitting or standing in the gym), but also perceive virtual objects superimposed on the physical world (e.g., the school gym). The virtual object may include a leaping whale surrounded by splashes of water. For example, the user may perceive the experience of a whale emerging from the school gym floor, leaping partially across the gym, landing in a large splash on the floor, and then disappearing. In this example, the wearable system can determine the dimensions of the area in the outside world viewed by the wearer (e.g., in this example, the size of the gym) to display an image of the leaping whale as perceived by the wearer as emanating from a sub-area of ​​the area (e.g., from the gym floor) and use the light field display described in Figures 4-6 so that the leaping whale and splashes appear realistic and lifelike to the user. In some implementations, the wearable system can present to the user (e.g., via the speaker 240 shown in Figure 2) the sound of the leaping whale that accompanies the image displayed to the user. The AR system can (in addition or as an alternative) display additional virtual content such as text or graphics on the scene viewed by the user. For example, the AR system can display information about the whale (such as its type, age, behavior, etc.) to the wearer before, during, or after the virtual whale appears to leap from the gym floor.

[0206] As another example, a user of a wearable system may be in a retail market. The user may view virtual content that includes images of actual physical people in the environment and astronauts walking around the market. The virtual content may be superimposed within the FOV of the wearable system's display.

[0207] The wearable system can modify the image of the physical world to provide the user with an MR experience. For example, the user may see a flock of physical birds flying in a v-formation. An outward-facing imaging system (e.g., as shown in FIG. 4) can capture this physical content, and the wearable system can process it and identify the flock of birds flying in a v-formation. The wearable system can add to the flock a virtual object (e.g., in this example, a dragon) flying in or near the formation, or replace one (or more) birds with it. As another example, the wearable system can add a virtual object (e.g., a virtual whale) floating on or flying above the beach to the user's view of a physical beach. The ability of light field displays to present realistic images as they appear at different distances allows the wearable display system to present an image of a whale near or away from the wearer. In some implementations, the wearable system may use shadow mapping techniques so that the virtual content appears to have virtual shadows, which may also make the virtual content displayed by the wearable system more realistic.

[0208] In some implementations, a user of the wearable system may perceive a sequence of virtual images within the AR / MR environment. For example, the user may view their cupped hands. A virtual object, such as a miniature elephant, may be displayed by the wearable system's display such that the user perceives the virtual object to be present within the user's cupped hands. The wearable system may use an outward-facing imaging system to image an area of ​​the environment, such as an area containing the wearer's hand (and any background behind the hand). The wearable system may determine the distance to the wearer's hand such that the virtual content (e.g., an elephant) may be scaled and appear at an appropriate size and distance within a particular sub-area (e.g., the hand) of the overall area in which the content is viewed. The wearable system may make the scene appear as if the wearer is holding the elephant in the wearer's hand. The position of the elephant may change from image to image such that the elephant may appear closer to the user in the temporal sequence compared to earlier in the temporal sequence. The image of the prancing elephant may be accompanied by sound (e.g., voice, music).

[0209] Additional Examples of Interaction with Virtual Objects As an example of user interaction with a virtual user interface, a user of the wearable system can perceive and interact with virtual objects in a physical room where people are dancing. In this example, the user may be a disc jockey (DJ) and the wearable system can display to the DJ a virtual UI that can be manipulated by the DJ's hand movements (e.g., gestures). The virtual UI can include virtual content that allows the DJ to interact with the UI. In this example, the virtual UI can be configured to be a DJ audio control system that can control the sound played to the dancing people. The UI can include user input features such as dials (e.g., jog shuttle dials), switches, sliders, buttons, or turntables that can be adjusted by the DJ via gestures. The virtual UI can include output features such as sound level graphics or equalizers. The output features can respond in real time as the sound levels or audio mix are altered by the DJ. An outwardly facing imaging system of the wearable system can image the DJ's hands and arms and determine the DJ's gestures (e.g., hand or finger movements). In response to the determined gesture, the wearable system may adjust the audio, for example by increasing or decreasing the volume, fading or panning the music, mixing the music, etc.

[0210] As another example, a user of a wearable system may view an operating room along with a doctor performing surgery on a patient. The wearable system may present the user with virtual content displaying the anatomy or organ (such as the heart) of the patient undergoing surgery. The orientation and position of the virtual organ may be adjusted via gestures (e.g., via the wearer reaching over and grasping or moving a virtual image of the heart) or a user input device. The virtual organ may represent an idealization of the heart (e.g., a textbook image) or an actual image of the patient's heart (e.g., taken during surgery or pre-mapped prior to surgery). The light field display capabilities of the wearable system (illustrated in Figures 4-6) allow the wearer to view a 3D image of the organ. In some implementations, the user does not need to be physically present in the environment (e.g., an operating room) to interact with objects (virtual and / or physical objects) in the environment. The user can interact with the doctor's avatar or the appearance of the operating room, such as by communicating with the doctor (eg, via speakers) or by interacting with a virtual image of a virtual organ.

[0211] The user can also use the wearable system to view and interact with educational virtual content. In this example, the educational virtual content can include an avatar (e.g., a creation designed to be attractive and not intimidating to a student) that holds a pointer and points to graphics (e.g., numbers) that are displayed to the wearer as part of an educational lesson. An educational system in communication with the wearable system can generate and deliver educational virtual content for presentation to the wearer as part of an educational lesson. The virtual content can include text, images, videos, graphics, and sounds. For example, an avatar can explain a math lesson to a student (e.g., 4×5=?). In some cases, the wearable system includes a microphone that can receive sounds in the surrounding environment, e.g., the student's voice. The student can ask a question, the wearable system (or educational system) can convert the question into an electronic format using voice recognition technology, and the educational system can send a response back to the wearable system. For example, the avatar may respond to a student's questions by answering the questions, pointing to different parts of the virtual content (eg, with a wand), explaining the answers, etc.

[0212] In another AR experience, a user of a wearable system (such as an HMD) can view physical objects (e.g., another person and a house) and a virtual entertainment display. The virtual entertainment display shows a presentation of a sporting event (in this example, a basketball game). The virtual entertainment display can present the user of the wearable system with information about the game being watched or other games (e.g., scores, play-by-play, game replays, player statistics, standings, etc.). The virtual entertainment display can appear as 3D, allowing the user to move around the display and see different sides of the virtual entertainment display (different content, different games, different sports, or even different entertainment genres (e.g., movies) can be displayed on different sides of the display). The sporting event may be presented to the user in real time as the sporting event occurs. This provides the user with the ability to interact with the sporting event even if the user is not physically present at the sporting event. A particular user may control the virtual entertainment display by using gestures or a user input device. In some cases, a single user has control of a virtual entertainment display while other users can watch the action on the display but cannot alter the content being displayed. In other implementations, the virtual entertainment display can present content that is created or updated as a result of the actions of multiple users.

[0213] (Additional Example Interactions Between Multiple Users) When each person in a group wears a wearable device, the user can interact with another user in the AR / VR / MR environment. For example, people in the group can enjoy, interact with, share, or manipulate (e.g., via gestures) virtual content (which can include images, videos, sounds, text, etc.) via the wearable devices they are wearing.

[0214] Such user interactions may occur within the virtual game. As players interact with each other in the game, the wearable device may present virtual content and sounds to the user of the wearable device. In this example, the user may perceive a room with physical objects (e.g., a bed, a table, a window, and another player 404 of the game). The user may also perceive a fire-breathing dragon flying around the other player. One or both of the players may control the dragon's position, movement, and actions (e.g., whether it breathes fire) by gestures (e.g., hand or arm gestures), totems (e.g., wands), or controllers or interfaces (physical or virtual). In some cases, the other players are not physically present in the room but are displayed to the user of the wearable device as if they were physically present (e.g., via telepresence). For example, the wearable device may present avatars (such as in the form of virtual children) of the other players as they play the game. The child avatar (and dragon) can be generated by the wearable system or another gaming system and communicated to the wearable system for display to the wearer. The child avatar can be a representation of the actual appearance of the other player, or can be selected as the other player chooses to be perceived by the user of the wearable. The other player can be a human player or a machine player. In other embodiments, more than one player can appear in the wearer's environment (e.g., in a room, outside the room, looking through a window, etc.). The ability of a light field display (e.g., such as the light field display described with reference to Figures 4-6) to present images as being at different distances (e.g., different depth planes) from the wearer can greatly enhance the realism and playability of the game. In addition to the dragon, the wearable device can optionally display other information (e.g., text or graphics) to the user (e.g., game play statistics or status).

[0215] As described with reference to Figure 7, a wearable device can pass a map of a physical or virtual environment and objects in the environment to another wearable device. For example, a wearable device can pass a map of a user's room and virtual objects in the room to another user's wearable device. In response, the users can interact with the virtual objects as if they were in the same environment.

[0216] (Additional embodiments) In a first aspect, a method for viewing virtual content comprising: accessing region data relating to a region within a user's field of view; analysing the region data to identify a sub-region of the region in which the virtual content is to be displayed; accessing or generating the virtual content based in part on the region data and the sub-region data; and displaying the virtual content such that when viewed by the user, the virtual content appears to be located within or on the sub-region of the region.

[0217] In a second aspect, the method of aspect 1, wherein the identified sub-region comprises a physical object within the region.

[0218] In a third aspect, the method of aspect 1 or aspect 2, wherein the region data is obtained from analysis of an image of a region within the user's field of view.

[0219] In a fourth aspect, the method of any one of aspects 1 to 3, wherein the virtual content is modified based in part on newly received region or sub-region data.

[0220] In a fifth aspect, the method of any one of aspects 1 to 4 further includes receiving user input; accessing or generating additional virtual content based, at least in part, on the user input; and displaying the additional virtual content.

[0221] In a sixth aspect, the method of any one of aspects 1 to 5, wherein the sub-region comprises a portion of a region in the vicinity of the user.

[0222] In a seventh aspect, a method for interacting with virtual content includes the steps of accessing or generating the virtual content, displaying the virtual content within a user's field of view, displaying a virtual user interface associated with the virtual content, receiving user input, associating the user input with a function associated with a user input feature of the virtual user interface, and performing the function.

[0223] In an eighth aspect, the method of aspect 7 further comprises manipulating or modifying the virtual content based in part on the received user input.

[0224] In a ninth aspect, the method of any one of aspects 7 to 8, wherein the user input is a gesture.

[0225] In a tenth aspect, the method of any one of aspects 7 to 9, wherein the user input is a movement of the user's eyes.

[0226] In an eleventh aspect, the method of any one of aspects 7 to 10, wherein the virtual content comprises educational content or entertainment content.

[0227] In a twelfth aspect, the method of any one of aspects 7 to 11, wherein the virtual content or virtual user interface appears to be three-dimensional when perceived by the user.

[0228] In a thirteenth aspect, the method of any one of aspects 7 to 12, wherein the user input feature of the virtual user interface comprises a dial, a switch, a slider, or a button.

[0229] In a fourteenth aspect, the method of any one of aspects 7 to 13, wherein the virtual user interface comprises an output feature configured to display information related to the functionality of the virtual user interface.

[0230] In a fifteenth aspect, the method of aspect 14, wherein the output features are updated in real time.

[0231] In a sixteenth aspect, a method for displaying information within an augmented reality environment includes determining a user's location; identifying a physical object within the user's field of view; accessing or generating the location or the identified physical object based, at least in part, on information related to the identified physical object; generating virtual content based on the information; and displaying the virtual content such that the information appears to the user in association with the physical object.

[0232] In a seventeenth aspect, the method of aspect 16, wherein the virtual content comprises text, an image, a graphic, or a video.

[0233] In an eighteenth aspect, the method of aspect 16 or aspect 17 further includes playing audio associated with the displayed virtual content.

[0234] In a nineteenth aspect, the method of any one of aspects 16 to 18 further comprises displaying additional virtual content based, at least in part, on user input.

[0235] In a twentieth aspect, a hardware processor is programmed with executable instructions that, when executed, cause the hardware processor to perform a method according to any one of aspects 1 to 19.

[0236] In a twenty-first aspect, a wearable display system comprising the hardware processor of aspect 20 and a display configured to provide an image to an eye of a user of the wearable display system.

[0237] In a twenty-second aspect, the wearable display system of aspect 21, wherein the display comprises a light field display.

[0238] In a twenty-third aspect, the wearable display system of aspect 21 or aspect 22, wherein the display is configured to display images at multiple depth planes.

[0239] In a twenty-fourth aspect, a wearable display system as described in any one of aspects 21 to 23, further comprising an image capture device configured to capture an image of an area comprising physical objects, and the wearable display system configured to display virtual content perceived by the user to be associated with one or more of the physical objects.

[0240] In a twenty-fifth aspect, a method for selecting a virtual object located in three-dimensional (3D) space, the method comprising: under control of an augmented reality (AR) system comprising computer hardware, the AR system configured to enable user interaction with interactable objects within a user's field of view (FOR), the FOR comprising a portion of an environment surrounding the user perceivable by the user via the AR system; determining a group of interactable objects within a user's FOR under control of the AR system; determining a user's posture; and determining a user's field of view (FOV) based, at least in part, on the user's posture, the FOV comprising a portion of the FOR perceived by the user at a given time; updating a subgroup of interactable objects located within the user's FOV based on a change in the user's posture or FOV; receiving a selection of a target interactable object from the subgroup of interactable objects; and initiating a selection event on the target interactable object.

[0241] In a twenty-sixth aspect, the method of aspect 25 further comprises storing the group of interactable objects in a data structure.

[0242] In a twenty-seventh aspect, the method of aspect 26, wherein each interactable object in the group of interactable objects is represented in the data structure based, at least in part, on the location of the interactable object within the user's FOV.

[0243] In a twenty-eighth aspect, the method of aspect 27, wherein the location comprises a distance from an edge of the user's FOV.

[0244] In a twenty-ninth aspect, the method of aspect 25, wherein the step of receiving a selection of a target interactable object from the subgroup of interactable objects includes the steps of receiving a first input from a user device and identifying the target interactable object from the subgroup of interactable objects in response to the first input.

[0245] In a 30th aspect, the method of aspect 25 further includes the steps of receiving a second input from the user device and initiating an interaction event on the target interactable object in response to the second input.

[0246] In a thirty-first aspect, the method of aspect 30, wherein the target interactable object is an interactable object within a subgroup of interactable objects closest to a center point of the user's FOV.

[0247] In a thirty-second aspect, the method of aspect 30, wherein the target interactable object is the leftmost or rightmost interactable object within a subgroup of interactable objects within the user's FOV.

[0248] In a thirty-third aspect, the method of any one of aspects 30-32, wherein the step of initiating the interaction event includes one or more of: adjusting a size of the target interactable object, displaying a menu of the target interactable object, browsing a menu of the target interactable object, selecting a first item on the menu, searching for a second item in a database, playing a video game associated with the target interactable object, watching a video, or conducting a conference call.

[0249] In a thirty-fourth aspect, the method of aspect 25, wherein the step of receiving the selection of the target interactable object is performed by determining a user's gaze path based on the user's posture and selecting an object that intersects with the user's gaze path as the target interactable object.

[0250] In a thirty-fifth aspect, the method of any one of aspects 25-34 further comprises the step of assigning a visible focus indicator to the target interactable object.

[0251] In a thirty-sixth aspect, the method of aspect 35, wherein the visible focus indicator comprises a highlight, a halo, a color change, a size change, or a change in the perceived depth of the target interactable object.

[0252] In a thirty-seventh aspect, the method of aspect 25, wherein the step of initiating the selection event includes one or more of the steps of changing the target interactable object to a different interactable object that becomes the target interactable object, opening a menu associated with the target interactable object, or receiving confirmation of selecting the target interactable object from the user.

[0253] In a thirty-eighth aspect, the method of any one of aspects 25-37, wherein the group of interactable objects comprises a virtual object.

[0254] In a thirty-ninth aspect, the method of any one of aspects 25-38, wherein the user's posture comprises an eye posture.

[0255] In a 40th aspect, the method of any one of aspects 25-39, wherein the user's posture comprises a head posture.

[0256] In a forty-first aspect, an augmented reality (AR) system for selecting a virtual object located in three-dimensional (3D) space comprises: a display system; a network interface; and a computer processor configured to communicate with the network interface and the display system, determine a group of interactable objects in a user's FOR, determine a user's posture, and determine a user's field of view (FOV) based at least in part on the user's posture, the FOV comprising a portion of the FOR perceived by the user at a given time, update a subgroup of interactable objects located within the user's FOV based on a change in the user's posture or FOV, receive a selection of a target interactable object from the subgroup of interactable objects, and initiate a selection event on the target interactable object.

[0257] In a forty-second aspect, the system of aspect 41, wherein the computer processor is further configured to store the group of interactable objects in a data structure.

[0258] In a forty-third aspect, the system described in aspect 42, wherein one or more interactable objects in the group of interactable objects are represented in the data structure based, at least in part, on the location of the interactable objects within the user's FOV.

[0259] In a forty-fourth aspect, the system of aspect 43, wherein the location comprises a distance from an edge of the user's FOV.

[0260] In aspect 45, the system described in any one of aspects 41-44 is configured to receive a selection of a target interactable object from a subgroup of interactable objects, the computer processor including receiving a first input from a user device and identifying the target interactable object from the subgroup of interactable objects in response to the first input.

[0261] In a 46th aspect, the system described in any one of aspects 41-45, wherein the target interactable object is an interactable object within a subgroup of interactable objects closest to a center point of the user's FOV.

[0262] In a 47th aspect, the system described in any one of aspects 41-45, wherein the target interactable object is the leftmost or rightmost interactable object in a subgroup of interactable objects within the user's FOV.

[0263] In a forty-eighth aspect, the system described in aspects 41-47, wherein the computer processor is configured to receive a selection of a target interactable object from a subgroup of interactable objects, the system including determining a user's gaze path based on the user's posture and selecting an object that intersects with the user's gaze path as the target interactable object.

[0264] In a forty-ninth aspect, the system of any one of aspects 41-48, wherein the computer processor is further configured to assign a visible focus indicator to the target interactable object.

[0265] In a 50th aspect, the system of aspect 49, wherein the visible focus indicator comprises a highlight, a halo, a color change, a size change, or a change in the perceived depth of the target interactable object.

[0266] In a fifty-first aspect, the system described in aspects 41-50, wherein the computer processor is configured to initiate a selection event, including one or more of changing the target interactable object to a different interactable object that becomes the target interactable object, opening a menu associated with the target interactable object, or receiving confirmation of selecting the target interactable object from the user.

[0267] In a 52nd aspect, the method of any one of aspects 41-51, wherein the group of interactable objects comprises a virtual object.

[0268] In a 53rd aspect, the method of any one of aspects 41-52, wherein the user's posture comprises an eye posture.

[0269] In a 54th aspect, the method of any one of aspects 51-53, wherein the user's posture comprises a head posture.

[0270] In a fifty-fifth aspect, a system for interacting with virtual objects in three-dimensional (3D) space includes an augmented reality display for displaying a plurality of interactable objects; a user input device; one or more sensors configured to determine a posture of a user; and one or more processors configured to switch between a first user input mode and a second user input mode to select a virtual object, the first user input mode being based, at least in part, on a posture of the user and the second user input mode being based, at least in part, on a signal from the user input device; and further comprising: and one or more processors configured to monitor a user's posture while operating in the first user input mode, and display a first focus indicator associated with the first user input mode in a direction associated with the user's posture based, at least in part, on the monitored posture, and receive an indication to switch to a second user input mode, the indication further indicating selection of the target interactable object; and monitor user input from the user input device while the system is operating in the second user input mode, and display a second focus indicator associated with the second user input mode based, at least in part, on the monitored user input.

[0271] In a fifty-sixth aspect, the first focus indicator comprises a cursor in a first shape and the second focus indicator comprises a cursor in a second shape, the system described in aspect 55.

[0272] In aspect 57, the system of aspect 56, wherein the first shape comprises a crosshair and the second shape comprises an arrow.

[0273] In a fifty-eighth aspect, the one or more processors are further configured to determine contextual information associated with a location of the first focus indicator or the second focus indicator, and to display options for switching to different user input modes based, at least in part, on the contextual information, in the system described in aspect fifty-fifth aspect.

[0274] In a fifty-ninth aspect, the system of any one of aspects 55-58, wherein the target interactable object comprises one or more virtual objects.

[0275] In a 60th aspect, the system of any one of aspects 55-59, wherein the user's posture comprises at least one of a head posture, an eye posture, or a body posture.

[0276] In a 61st aspect, the system of any one of aspects 55-60, wherein the user input device is a thumb pad, a track pad, a D-pad, or a touch screen.

[0277] In a sixty-second aspect, a method for selecting an interactable object located in three-dimensional (3D) space, under control of an augmented reality (AR) system comprising computer hardware, the AR system configured to enable user interaction with an interactable object within a user's field of view (FOR), the FOR comprising a portion of an environment surrounding the user perceivable by the user via the AR system, and the AR system further comprising a user input device, the method comprising: under control of the AR system, determining a group of interactable objects associated with a virtual object in the user's FOR; determining a posture of the user; determining a target interactable object from the group of interactable objects based, at least in part, on the user's posture; associating a focus indicator with the selected target interactable object; and initiating a selection event on the selected target surface based, at least in part, on input of the user input device.

[0278] In a 63rd aspect, the method of aspect 62, wherein the step of determining the target interactable object is performed by ray casting.

[0279] In a 64th aspect, the method described in aspect 63 includes a step of casting a bundle of rays.

[0280] In a 65th aspect, the method described in aspect 63, wherein the ray casting includes a step of casting a ray with a substantial lateral width.

[0281] In a 66th aspect, the method of aspect 62, wherein the focus indicator is visible to the user.

[0282] In a 67th aspect, the method of aspect 66 further includes the step of displaying a focus indicator to a user.

[0283] In a 68th aspect, the method of aspect 62 further includes a step of determining a user's field of view (FOV) based, at least in part, on the user's posture, the FOV comprising a portion of the FOR perceived by the user at a given time; a step of accessing contextual information of one or more interactable objects within the field of view (FOV); and a step of providing, at least in part, one or more options for a user input mode based on the contextual information.

[0284] In a sixty-ninth aspect, the method of aspect 68 further includes the steps of receiving a selection of an option for a user input mode, determining a current user input mode, and, in response to determining that the current user input mode is different from the selected option, updating the current user input mode to the selected option.

[0285] In a seventieth aspect, the method of aspect 69 further comprises updating a focus indicator to indicate that a current user input mode has changed.

[0286] In a seventy-first aspect, the method of aspect 70, wherein the user input mode includes one or more of a user posture or a user input device.

[0287] In a 72nd aspect, the method of any one of aspects 62-71, wherein the group of interactable objects comprises one or more virtual user interface planes.

[0288] In a 73rd aspect, the method of any one of aspects 62-72, wherein the user's posture comprises at least one of an eye posture, a head posture, or a body posture.

[0289] In a seventy-fourth aspect, a method for interacting with virtual objects in a three-dimensional (3D) environment, under control of an augmented reality (AR) system comprising computer hardware, the AR system configured to enable user interaction with interactable objects within a user's field of view (FOR), the FOR comprising a portion of an environment surrounding the user perceivable by the user via the AR system, and the AR system further comprising a user input device, the method including: determining a posture of a user under control of the AR system; displaying a first focus indicator associated with a target interactable object in a direction related to the user's posture, the target interactable object comprising a plurality of virtual objects; receiving a selection of the target interactable object; displaying the plurality of virtual objects; displaying a second focus indicator associated with the target virtual object; and updating the second focus indicator based at least in part on input from the user input device.

[0290] In a seventy-fifth aspect, the method of aspect 74, wherein the first focus indicator comprises a cursor in a first shape and the second focus indicator comprises a cursor in a second shape.

[0291] In a 76th aspect, the method of aspect 75, wherein the first shape comprises a crosshair and the second shape comprises an arrow.

[0292] In a 77th aspect, the method of any one of aspects 74-76, wherein the user's posture comprises at least one of a head posture, an eye posture, or a body posture.

[0293] In a 78th aspect, the method of any one of aspects 74-77, wherein the user input device is a thumb pad, a track pad, a D-pad, or a touch screen.

[0294] In a seventy-ninth aspect, a method for interacting with virtual objects in three-dimensional (3D) space, under control of an augmented reality (AR) system comprising computer hardware, the AR system configured to enable user interaction with interactable objects within a user's field of view (FOR), the FOR comprising a portion of an environment surrounding the user perceivable by the user via the AR system, the AR system further comprising a user input device, comprising the steps of: monitoring, under control of the AR system, a posture of the user while the AR system is operating in a first user input mode, the first user input mode being based, at least in part, on a posture of the user; and detecting, at least in part, the monitored posture. and displaying a first focus indicator associated with the first user input mode in a direction associated with the user's posture based on the selected user's posture; receiving an indication to switch to a second user input mode, the indication further indicating selection of the target interactable object; monitoring user input from the user input device while the AR system is operating in the second user input mode, the second user input mode being based, at least in part, on a signal from the user input device; and displaying a second focus indicator associated with the second user input mode based, at least in part, on the monitored user input.

[0295] In an 80th aspect, the method of aspect 79, wherein the first focus indicator comprises a cursor in a first shape and the second focus indicator comprises a cursor in a second shape.

[0296] In an 81st aspect, the method of aspect 80, wherein the first shape comprises a crosshair and the second shape comprises an arrow.

[0297] In an 82nd aspect, the method of aspect 79 further includes a step of determining contextual information associated with a location of the first focus indicator or the second focus indicator, and a step of displaying options for switching to different user input modes based, at least in part, on the contextual information.

[0298] In an 83rd aspect, the method of any one of aspects 79-82, wherein the target interactable object comprises one or more virtual objects.

[0299] In an 84th aspect, the method of any one of aspects 79-83, wherein the user's posture comprises at least one of a head posture, an eye posture, or a body posture.

[0300] In an 85th aspect, the method of any one of aspects 79-84, wherein the user input device is a thumb pad, a track pad, a D-pad, or a touch screen.

[0301] In an 86th aspect, an augmented reality device comprises computer hardware programmed to perform a method according to any one of aspects 62-85.

[0302] In an 87th aspect, a system for changing a user input mode for a wearable device includes a display system of the wearable device configured to present a three-dimensional (3D) view to a user, the 3D view comprising an interactable object, a user input device configured to receive user input, and a sensor configured to acquire data associated with a posture of the user, and a hardware processor in communication with the user input device, the sensor determining whether a current user input mode for interacting with the interactable object is a first user input mode or a second user input mode, the first user input mode being based, at least in part, on a posture of the user, and the second user input mode being based, at least in part, on user input from the user input device, and in response to determining that the current user input mode is the first user input mode, and a processor programmed to: monitor a posture of a user using the sensor, present, via the display system, a focus indicator in a first shape associated with a first user input mode in a direction associated with the user's posture based at least in part on the monitored posture, receive the first indication, and switch to a second user input mode, and in response to the first indication, switch the current user input mode to the second user input mode, and in response to determining that the current user input mode is the second user input mode, monitor user input from the user input device, present, via the display system, a focus indicator in a second shape associated with the second user input mode based at least in part on the monitored input, receive the second indication, and switch to the first user input mode, and in response to the second indication, switch the current user input mode to the first user input mode.

[0303] In aspect 88, the system of aspect 87, wherein the user's posture comprises at least one of a head posture, an eye posture, a foot posture, or a body posture, or the sensor comprises at least one of an inward-facing imaging system, an outward-facing imaging system, or an inertial measurement unit, or the display system comprises a light field display configured to display one or more of the interactable objects in multiple depth planes.

[0304] In an 89th aspect, the processor is further programmed to determine a user's field of view (FOV) based, at least in part, on the user's posture, the FOV comprising a portion of the user's environment perceived by the user at a given time, determine contextual information associated with the FOV, the contextual information comprising at least one of a layout of interactable objects within the FOV, a size of the FOV, and a size of one or more of the interactable objects in the user's FOV, and present an option for switching from the first user input mode to the second user input mode or from the second user input mode to the first user input mode based, at least in part, on the contextual information.

[0305] In a 90th aspect, the system described in aspect 89, wherein to present the options, the processor is programmed to present a focus indicator in the second shape when the current user input mode is in the first user input mode, and to present a focus indicator in the first shape when the current user input mode is in the second user input mode.

[0306] In a 91st aspect, the system of any one of aspects 87-90, wherein the first indication comprises a change in the user's posture and the second indication comprises actuation of a user input device.

[0307] In a ninety-second aspect, a method for changing a user input mode for a wearable device, comprising: determining, under control of a wearable device comprising a computer processor, the wearable device configured to enable user interaction with an interactable object within a user's ocular field of view (FOR), the FOR comprising a portion of the user's surrounding environment perceivable by the user via a display system of the wearable device; and displaying, under control of the wearable device, a posture of the user; and displaying, via the display system, a first focus indicator associated with a target interactable object in a direction related to the user's posture, the target interactable object comprising a plurality of receiving a selection of a target interactable object; presenting to a user an option for switching a user input mode from posture to hand gestures on a user input device; displaying via a display system a plurality of virtual objects; in response to determining that the user has switched the user input mode from posture to hand gestures on the user input device, displaying via the display system a second focus indicator associated with a target virtual object of the plurality of virtual objects; and updating the second focus indicator based, at least in part, on user input from the user input device.

[0308] In a 93rd aspect, the method of aspect 92, wherein the option is presented in response to a selection of a target interactable object or multiple virtual objects or contextual information associated with the target interactable object.

[0309] In a 94th aspect, the method of aspect 93, wherein the context information comprises a density of the plurality of virtual objects, and an option for switching a user input mode from a posture to a hand gesture on the user input device is presented in response to determining that the density of the plurality of virtual objects exceeds a threshold density.

[0310] In a 95th aspect, the method of any one of aspects 92-94, wherein the target virtual object is identified based, at least in part, on the user's posture.

[0311] In aspect 96, the method of any one of aspects 92-95, wherein the step of updating the second focus indicator includes a step of transitioning the second focus indicator from the target virtual object to another virtual object of the plurality of virtual objects.

[0312] In a 97th aspect, the method of any one of aspects 92-96 further comprises initiating a selection event on the target virtual object, the selection event further comprising at least one of opening a menu associated with the target virtual object or receiving an indication to select the target virtual object.

[0313] In a 98th aspect, the method of any one of aspects 92-97, wherein the plurality of virtual objects comprises at least one of a weather application or an astronomy application, and in response to a selection of the weather application, the wearable device is programmed to display virtual weather information superimposed on objects in the user's environment, and in response to a selection of the astronomy application, the wearable device is programmed to display an interactive planetary system comprising a three-dimensional virtual planet superimposed on the user's environment.

[0314] In a ninety-ninth aspect, a wearable system for selecting a virtual object located in three-dimensional (3D) space comprises: a display system configured to present the virtual object in the 3D space; a non-transitory data store configured to store interactable objects in the 3D space; a sensor configured to determine a user's posture; and a hardware processor in communication with the display system, the data store, and the sensor, and programmed to determine the user's posture based at least in part on data received from the sensor, determine a user's field of view (FOV) based at least in part on the user's posture, the FOV comprising a portion of the user's environment perceived by the user at a given time, identify a group of interactable objects within the FOV, identify a target interactable object within the FOV based at least in part on the user's posture, and initiate a selection event associated with the target interactable object.

[0315] In aspect 100, a wearable system as described in aspect 99, wherein a group of interactable objects is stored in a data structure and an index associated with each interactable object is determined, at least in part, based on the position of the interactable object in 3D space.

[0316] In aspect 101, a wearable system as described in any one of aspects 99-100, wherein in response to initiation of a selection event on a target interactable object, a processor of the wearable system is programmed to present, within a two-dimensional (2D) interface, a virtual object within a threshold range of the target interactable object in 3D space.

[0317] In aspect 102, the wearable system of aspect 101, wherein the 2D interface is interactable via a user input device.

[0318] In aspect 103, to identify a target interactable object within the FOV, the processor is configured to perform at least one of the following steps: determining a user's gaze path based on the user's posture and selecting an interactable object that intersects with the gaze path as the target interactable object, or selecting a leftmost or rightmost interactable object within the user's FOV as the target interactable object, wherein the leftmost or rightmost interactable object is selected based, at least in part, on an index associated with a group of interactable objects, the wearable system of any one of aspects 99-102.

[0319] In aspect 104, a wearable system described in any one of aspects 99-103, wherein the processor is configured to initiate a selection event in response to at least one of receiving input from a user input device or detecting a change in the user's posture.

[0320] In aspect 105, the wearable system of any one of aspects 99-104, wherein the processor is further configured to present a focus indicator associated with the target interactable object.

[0321] In aspect 106, a wearable system as described in any one of aspects 99-105, further comprising a geolocation sensor configured to acquire data associated with a user's location, and the target interactable object comprises a weather application programmed to determine a user's location based on data acquired by the geolocation sensor, communicate with a remote computing device, acquire weather data based on the user's location, generate virtual elements associated with the weather data, and superimpose the virtual elements within the user's 3D space.

[0322] (Conclusion) Each of the processes, methods, and algorithms described herein and / or depicted in the accompanying figures may be embodied in code modules executed by one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, and may be fully or partially automated thereby. For example, the computing system may include a general-purpose computer (e.g., a server) or a special-purpose computer programmed with specific computer instructions, special-purpose circuits, etc. The code modules may be written in a programming language that may be compiled and linked into an executable program, installed in a dynamic link library, or interpreted. In some implementations, certain operations and methods may be performed by circuitry specific to a given function.

[0323] Furthermore, certain implementations of the functionality of the present disclosure may be sufficiently mathematically, computationally, or technically complex that special purpose hardware (utilizing appropriate specialized executable instructions) or one or more physical computing devices may be required to implement the functionality, e.g., due to the amount or complexity of the calculations involved, or to provide results in substantially real-time. For example, a video may contain many frames, each frame may have millions of pixels, and specifically programmed computer hardware may be required to process the video data to provide the desired image processing task or application in a commercially reasonable amount of time.

[0324] The code modules or any type of data may be stored on any type of non-transitory computer readable medium, such as physical computer storage devices, including hard drives, solid state memory, random access memory (RAM), read only memory (ROM), optical disks, volatile or non-volatile storage devices, combinations of the same, and / or the like. The methods and modules (or data) may also be transmitted as data signals (e.g., as part of a carrier wave or other analog or digital propagated signal) generated over a variety of computer readable transmission media, including wireless-based and wired / cable-based media, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored persistently or otherwise in any type of non-transitory tangible computer storage device, or communicated via a computer readable transmission medium.

[0325] Any process, block, state, step, or functionality in the flow diagrams described herein and / or depicted in the accompanying figures should be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in the process. Various processes, blocks, states, steps, or functionality can be combined, rearranged, added, deleted, modified, or otherwise altered from the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules may perform some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps, or states associated therewith can be performed in other sequences as appropriate, e.g., serially, in parallel, or in some other manner. Tasks or events may be added to or removed from the disclosed exemplary embodiments. Furthermore, the separation of various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods, and systems may generally be integrated together in a single computer product or packaged into multiple computer products. Many implementation variations are possible.

[0326] The process, method, and system may be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowdsourced computing networks, the Internet, and the World Wide Web. The network may be a wired or wireless network or any other type of communication network.

[0327] The systems and methods of the present disclosure each have several innovative aspects, none of which is solely responsible for or required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of one another or may be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of the present disclosure. Various modifications of the implementations described in the present disclosure may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations without departing from the spirit or scope of the present disclosure. Thus, the claims are not intended to be limited to the implementations shown herein, but should be accorded the widest scope consistent with the present disclosure, the principles, and novel features disclosed herein.

[0328] Certain features described herein in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation can also be implemented separately in multiple implementations or in any suitable subcombination. Furthermore, although features may be described above as acting in a combination and may be initially claimed as such, one or more features from the claimed combination may in some cases be deleted from the combination and the claimed combination may be directed to a subcombination or a variation of the subcombination. No single feature or group of features is required or essential to every embodiment.

[0329] In particular, conditional statements used herein, such as "can," "could," "might," "may," "eg," and the like, are generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not, unless specifically stated otherwise or understood otherwise within the context as used. Thus, such conditional statements are generally not intended to suggest that features, elements, and / or steps are in any way required for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps should be included or performed in any particular embodiment, with or without authorial input or prompting. The terms "comprise," "include," "have," and the like, are synonymous and used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. Also, the term "or" is used in its inclusive sense (and not its exclusive sense), thus, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. In addition, the articles "a," "an," and "the," as used in this application and the appended claims, unless otherwise specified, should be interpreted to mean "one or more" or "at least one."

[0330] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single elements. As an example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Transitive phrases such as "at least one of X, Y, and Z" are generally understood differently in the context in which they are used to convey that an item, term, etc. may be at least one of X, Y, or Z, unless specifically stated otherwise. Thus, such transitive phrases are generally not intended to suggest that an embodiment requires that at least one of X, at least one of Y, and at least one of Z, respectively, be present.

[0331] Similarly, although operations may be depicted in the figures in a particular order, it should be appreciated that such operations need not be performed in the particular order depicted, or in sequential order, or that all of the depicted operations need not be performed, to achieve desirable results. Additionally, the figures may diagrammatically depict one or more exemplary processes in the form of a flow chart. However, other operations not depicted may also be incorporated within the diagrammatically depicted exemplary methods and processes. For example, one or more additional operations may be performed before, after, simultaneously with, or during any of the depicted operations. Additionally, operations may be rearranged or reordered in other implementations. In some circumstances, multitasking and parallel processing may be advantageous. Additionally, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results.

Claims

[Claim 1] The invention described in this specification.

Citation Information

Patent Citations

  • Information processing apparatus, display control method, and program

    JP2013218597A

  • Systems and methods for augmented and virtual reality

    JP2015503141A

  • Contextual User Interface

    JP2015524110A

  • Calibration free, motion tolerent eye-gaze direction detector with contextually aware computer interaction and communication methods

    US20120105486A1

  • System and method for augmented and virtual reality

    US20140306866A1