Selecting a virtual object in 3D space

Abstract

To provide selection of a virtual object within a suitable, three-dimensional space.SOLUTION: A system and a method for interacting with a virtual object within a three-dimensional space by using a wearable system are disclosed. The wearable system can be programmed so as to allow a user to interact with a virtual object by using a user input device and posture. The wearable system also automatically determines context information, such as the layout of a virtual object in a user's environment, and can switch a user input mode on the basis of the context information.SELECTED DRAWING: Figure 12

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims priority under 35 U.S.C.§119(e) to U.S. Provisional Application No. 62 / 244,115, filed October 20, 2015, entitled "USER APPLICATIONS, INTERFACES, AND EXPERIENCES WITH AUGMENTED REALITY DISPLAY DEVICES", U.S. Provisional Application No. 62 / 301,422, filed February 29, 2016, entitled "SELECTING VIRTUAL OBJECTS IN 3D SPACE", and U.S. Provisional Application No. 62 / 316,179, filed March 31, 2016, entitled "SELECTING VIRTUAL OBJECTS IN 3D SPACE". The entire disclosures of the above provisional applications are hereby incorporated by reference into this specification in their entirety.

[0002] This disclosure relates to virtual reality, augmented reality, and mixed reality imaging and visualization systems, and more particularly to systems for interacting with virtual objects within a three - dimensional (3D) space.

Background Art

[0003] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality", "augmented reality", or "mixed reality" experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner that appears or can be perceived as being real. A virtual reality or "VR" scenario typically involves the presentation of digital or virtual image information without transparency to other actual real-world visual inputs. An augmented reality or "AR" scenario typically involves digital or virtual image information as an augmentation of the visualization of the actual world around the user. Mixed reality or "MR" is related to the fusion of the real and virtual worlds to create a new environment in which physical and virtual objects coexist and interact in real time. In summary, the human visual perception system is very complex, and it is difficult to generate VR, AR, or MR technologies that facilitate a comfortable and natural-feeling rich presentation of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies.

SUMMARY OF THE INVENTION

MEANS FOR SOLVING THE PROBLEM

[0004] In some embodiments, a system for changing a user input mode for a wearable device is disclosed. The system can include a display system of the wearable device configured to present a three-dimensional (3D) view to a user, the 3D view including interactive objects, a user input device configured to receive user input, a sensor configured to acquire data associated with the user's pose, and a hardware processor communicatively coupled to the user input device. The hardware processor can be programmed to determine whether a current user input mode for interacting with the interactive objects is a first user input mode or a second user input mode, the first user input mode being at least partially based on the user's pose and the second user input mode being at least partially based on user input from the user input device. In response to determining that the current user input mode is the first user input mode, the hardware processor can use the sensor to monitor the user's pose and, via the display system, present a focus indicator in a first shape associated with the first user input mode in a direction related to the monitored pose. The hardware processor can receive a first indication and switch to the second user input mode, and in response to the first indication, switch the current user input mode to the second user input mode. In response to determining that the current user input mode is the second user input mode, the hardware processor can monitor user input from the user input device and, via the display system, present a focus indicator in a second shape associated with the second user input mode based at least in part on the monitored input. The hardware processor can receive a second indication and switch to the first user input mode, and in response to the second indication, switch the current user input mode to the first user input mode.

[0005] In one embodiment, a method for changing a user input mode for a wearable device is disclosed. The method may be performed under the control of a wearable device comprising a computer processor. The wearable device can be configured to enable user interaction with an interactive object within the user's field of regard (FOR), where the FOR comprises a portion of the user's surrounding environment perceptible to the user via a display system of the wearable device. The method includes determining the user's posture, and displaying, via the display system, a first focus indicator associated with a target interactive object in a direction related to the user's posture, wherein the target interactive object comprises a plurality of virtual objects, receiving a selection of the target interactive object, presenting an option to the user to switch the user input mode from the posture to a hand gesture on a user input device, displaying, via the display system, the plurality of virtual objects, in response to determining that the user has switched the user input mode from the posture to a hand gesture on the user input device, displaying, via the display system, a second focus indicator associated with a target virtual object among the plurality of virtual objects, and updating the second focus indicator based at least in part on user input from the user input device.

[0006] In some embodiments, a wearable system and method for selecting a virtual object located in a three-dimensional (3D) space are disclosed. The wearable system can include a display system configured to present a virtual object in the 3D space, a non-transitory data storage configured to store interactive objects in the 3D space, a sensor configured to determine a user's pose, and a hardware processor programmed to communicate with the display system, the data storage device, and the sensor. The wearable system and method can determine the user's pose based at least in part on data received from the sensor, determine the user's field of view (FOV) based at least in part on the user's pose, where the FOV comprises a portion of the user's environment perceived by the user at a given time, identify a group of interactive objects within the FOV, identify a target interactive object within the FOV based at least in part on the user's pose, and initiate a selection event associated with the target interactive object.

[0007] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. Neither this summary nor the following detailed description purports to define or limit the scope of the subject matter of the invention. The present invention provides, for example, the following. (Item 1) A system for changing a user input mode for a wearable device, A display system of a wearable device configured to present a three-dimensional (3D) view to a user, the 3D view comprising interactive objects, a display system of the wearable device, A user input device configured to receive user input, A sensor configured to obtain data associated with the pose of the user, A hardware processor that communicates with the user input device, determining whether the current user input mode for interacting with the interactive object is a first user input mode or a second user input mode, wherein the first user input mode is at least partially based on the user's posture, wherein the second user input mode is at least partially based on user input from the user input device, in response to determining that the current user input mode is the first user input mode, using the sensor to monitor the user's posture, presenting, via the display system, a focus indicator in a first shape associated with the first user input mode in a direction related to the monitored posture, at least partially based on the monitored posture, receiving a first indication and switching to the second user input mode, in response to the first indication, switching the current user input mode to the second user input mode, in response to determining that the current user input mode is the second user input mode, monitoring user input from the user input device, presenting, via the display system, a focus indicator in a second shape associated with the second user input mode based on the monitored input, at least partially based on the monitored input, receiving a second indication and switching to the first user input mode, in response to the second indication, switching the current user input mode to the first user input mode A hardware processor programmed to perform the above, A system comprising the above. (Item 2) The processor further at least partially determine the user's field of view (FOV) based on the user's posture, the FOV comprising a portion of the user's environment perceived by the user at a given time, and determine context information associated with the FOV, the context information comprising at least one of the layout of the interactive objects within the FOV, the size of the FOV, and the size of one or more of the interactive objects within the user's FOV, and at least partially based on the context information, present an option to switch from the first user input mode to the second user input mode or from the second user input mode to the first user input mode and is programmed to perform the system of claim 1. (Item 3) To present the option, the processor is programmed to present the focus indicator in the second shape when the current user input mode is the first user input mode and to present the focus indicator in the first shape when the current user input mode is the second user input mode, the system of claim 3. (Item 4) The first indication comprises a change in the user's posture and the second indication comprises an operation of the user input device, the system of claim 1. (Item 5) The posture of the user includes at least one of a head posture, an eye posture, a foot posture, or a body posture, or the sensor includes at least one of an imaging system facing inward, an imaging system facing outward, or an inertial measurement unit, or the display system includes a light field display configured to display one or more of the interactive objects in a plurality of depth planes, the system according to any one of items 1-4. (Item 6) A method for changing a user input mode for a wearable device, under the control of a wearable device comprising a computer processor, the wearable device being configured to enable user interaction with an interactive object within a user's field of regard (FOR), the FOR comprising a portion of the user's surrounding environment perceptible to the user via a display system of the wearable device, under the control of the wearable device, determining a posture of the user; displaying, via the display system, a first focus indicator associated with a target interactive object in a direction related to the posture of the user, the target interactive object comprising a plurality of virtual objects; receiving a selection of the target interactive object; presenting to the user an option to switch the user input mode from posture to a hand gesture on a user input device; displaying, via the display system, the plurality of virtual objects; in response to determining that the user has switched the user input mode from posture to a hand gesture on the user input device, displaying, via the display system, a second focus indicator associated with a target virtual object among the plurality of virtual objects; Updating the second focus indicator at least partially based on user input from the user input device A method comprising (Item 7) The method according to item 6, wherein the option is presented in response to selection of the target interactable object or context information associated with the plurality of virtual objects or the target interactable object (Item 8) The context information comprises a density of the plurality of virtual objects, and the option for switching the user input mode from a posture to a hand gesture on the user input device is presented in response to a determination that the density of the plurality of virtual objects exceeds a threshold density, the method according to item 7 (Item 9) The step of updating the second focus indicator comprises moving the second focus indicator from the target virtual object to another virtual object among the plurality of virtual objects, the method according to item 6 (Item 10) The method according to item 6, further comprising starting a selection event on the target virtual object, the selection event comprising at least one of opening a menu associated with the target virtual object or receiving an indication of selecting the target virtual object (Item 11) The method according to any one of items 6-10, wherein the target virtual object is identified at least partially based on the user's posture (Item 12) The plurality of virtual objects includes at least one of a weather application or an astronomy application, and in response to selection of the weather application, the wearable device is programmed to display virtual weather information superimposed on objects in the user's environment, and in response to selection of the astronomy application, the wearable device is programmed to display an interactive galaxy including a three-dimensional virtual planet superimposed on the user's environment, the method according to item 6. (Item 13) A wearable system for selecting a virtual object located within a three-dimensional (3D) space, A display system configured to present virtual objects within the 3D space, A non-transitory data storage configured to store interactive objects within the 3D space, A sensor configured to determine a user's pose, A hardware processor programmed to communicate with the display system, the data storage, and the sensor, Determining the user's pose based at least in part on data received from the sensor; Determining the user's field of view (FOV) based at least in part on the user's pose, the FOV comprising a portion of the user's environment perceived by the user at a given time; Identifying a group of interactive objects within the FOV; Identifying a target interactive object within the FOV based at least in part on the user's pose; Initiating a selection event associated with the target interactive object; and performing, a hardware processor A wearable system comprising. (Item 14) The group of the interaction - capable objects is stored in a data structure, and an index associated with each interaction - capable object is determined, at least in part, based on the position of the interaction - capable object in the 3D space, for the wearable system according to item 13. (Item 15) In response to the start of a selection event on the target interaction - capable object, the processor of the wearable system is programmed to present virtual objects within a threshold range of the target interaction - capable object in the 3D space in a two - dimensional (2D) interface, for the wearable system according to item 13. (Item 16) The 2D interface is capable of interacting via a user input device, for the wearable system according to item 15. (Item 17) To identify a target interaction - capable object within the FOV, the processor determines a path of the user's line of sight based on the user's pose and selects an interaction - capable object that intersects the path of the line of sight as the target interaction - capable object, or selects the left - most or right - most interaction - capable object within the user's FOV as the target interaction - capable object, where the left - most or right - most interaction - capable object is selected, at least in part, based on an index associated with the group of the interaction - capable objects, in a step, is configured to perform at least one of the above for the wearable system according to item 13. (Item 18) The processor is configured to start the selection event in response to at least one of receiving an input from a user input device or detecting a change in the user's pose, for the wearable system according to item 13. (Item 19) The wearable system according to any one of items 13-19, wherein the processor is further configured to present a focus indicator associated with the target-interactable object. (Item 20) The wearable system further includes a geolocation sensor configured to obtain data associated with the location of the user, the target-interactable object includes a weather application, and the weather application determines the location of the user based on the data obtained by the geolocation sensor, communicates with a remote computing device, obtains weather data based on the location of the user, generates virtual elements associated with the weather data, and superimposes the virtual elements within the 3D space of the user. The wearable system according to item 13, which is programmed to perform the above.

Brief Description of Drawings

[0008]

Figure 1

[0009]

Figure 2

[0010]

Figure 3

[0011]

Figure 4

[0012]

Figure 5

[0013]

Figure 6

[0014]

Figure 7

[0015]

Figure 8

[0016]

Figure 9

[0017]

Figure 10

[0018]

Figure 11

[0019]

Figure 12

[0020]

Figure 13A

[0021]

Figure 13B

[0022]

Figure 14

[0023]

Figure 15

[0024]

Figure 16

[0025]

Figure 17

[0026]

Figure 18

[0027]

Figure 19

[0028]

Figure 20

[0029]

Figure 21

[0030]

Figure 22

[0031]

Figure 23

[0032] Throughout the drawings, reference numerals may be reused to indicate the correspondence between the elements being referenced. The drawings are provided to illustrate the exemplary embodiments described herein and are not intended to limit the scope of the present disclosure.

DETAILED DESCRIPTION

[0033] (Overview) By using the use of AR / VR / MR devices, a user may desire to target and select an object in a three-dimensional (3D) space using a virtual user interface. For example, the user may select a virtual object using a body posture such as physically approaching an item, grasping it, or touching it. The user may also select a virtual object by pointing at the object with a virtual ray or beam and clicking on it. However, these techniques can cause fatigue and can be difficult to precisely select an object because the user may be required to hold their posture still to achieve the selection.

[0034] The present disclosure provides an example of a wearable system that addresses some or all of these problems. As an example, a user may move their head and direct their eyes towards a group of objects. The object nearest to the center of the user's field of view can be highlighted as a potential target object, and the user can operate a user input device (e.g., by swiping on a touch screen) to move the highlight from one object to another. The user can confirm the selection of the target object by operating the user input device again (e.g., by touching the touch screen). Once selected, the AR user interface may enable the user to perform additional actions on the selected target object (e.g., display or select from a menu associated with the object, perform an action associated with a game in which the target object appears, etc.). This technique can be particularly advantageous for a user to select an object of interest while reducing fatigue. This is because it is difficult to precisely control the head pose. The wearable system can preliminarily identify the object of interest based on the user's head pose, while enabling the user to precisely select the object using hand gestures.

[0035] In some implementations, the interactive object may include a plurality of virtual objects. For example, the virtual user interface plane may include a plurality of virtual applications, such as a video streaming application, a virtual class application, a weather application, a game application, an astronomy application, etc. The wearable system may support different user input modes based on the characteristics of the interactive object. For example, when the interactive object is a user interface plane (which may be large in size), the wearable system may allow the user to interact with it using gestures. On the other hand, when the interactive object is relatively small, the wearable system may instead set the user input device as the default input mode and allow the user to interact precisely with the virtual object. These implementations may be advantageous because the movement and targeting of large objects may not require much accuracy in the user's movement, while the movement and selection of small objects may require the user to target precisely.

[0036] The wearable system can also determine the user input mode based on context information. For example, the wearable system can determine the layout of virtual objects in the user's environment. When the wearable system detects a high-density cluster of virtual objects in the user's line of sight, the wearable system may give the user the option to switch input control from head control to hand control. In this way, the user can interact more precisely with the virtual object. As another example, the AR system can detect the orientation of an object (e.g., vertical or horizontal) and provide appropriate interaction for the user (e.g., volume control of a TV application that appears vertically in front of the user or typing control of a virtual keyboard that appears horizontally on the user's desk).

[0037] A wearable system can enable a user to share virtual content with others (also wearing a wearable system) by, for example, passing a world map of the user's environment via a network or communicating (or updating) virtual content between wearable systems.

[0038] (Example of 3D display) Figure 1 depicts an illustration of a composite reality scenario with a virtual reality object and a physical object as viewed by a person. In Figure 1, the MR scene 100 is depicted, and to a user of MR technology, a real-world park-like setting 110 featuring people, trees, buildings in the background, and a concrete platform 120 can be seen. In addition to these items, a user of MR technology also "sees" a robot figure 130 standing on the real-world platform 120 and a flying comic-like avatar character 140 that appears anthropomorphic like a bumblebee, but these elements do not exist in the real world.

[0039] It may be desirable for a 3D display to generate a perspective adjustment response corresponding to its virtual depth for each point within the display's field of view in order to generate a true sense of depth, more specifically, a simulated sense of surface depth. If the perspective adjustment response for a display point does not correspond to the virtual depth of that point as determined by both binocular depth cues of convergence and stereopsis, the human eye experiences a perspective conflict, resulting in unstable imaging, harmful eye strain, headaches, and in the absence of perspective adjustment information, a near-complete lack of surface depth.

[0040] VR, AR, and MR experiences can be provided by a display system having a display that provides an image corresponding to a plurality of depth planes to a viewer. The images may differ for each depth plane (e.g., providing a somewhat different presentation of a scene or object), and are separately focused by the viewer's eyes, thereby based on the eye accommodation required to focus on different image features regarding scenes located on different depth planes, or based on observing different image features on different depth planes that are out of focus, which can help provide depth cues to the user. As discussed anywhere in this specification, such depth cues provide a believable perception of depth.

[0041] FIG. 2 illustrates an embodiment of a wearable system 200. The wearable system 200 includes a display 220 and various mechanical and electronic modules and systems to support the functions of the display 220. The display 220 may be coupled to a frame 230 that is wearable by a user, wearer, or viewer 210. The display 220 can be positioned in front of the eyes of the user 210. The display 220 can present AR / VR / MR content to the user. The display 220 can present a head-mounted display (HMD) that is worn on the user's head. In some embodiments, a speaker 240 is coupled to the frame 230 and positioned adjacent to the user's outer ear canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other outer ear canal to provide stereo / formable acoustic control).

[0042] The wearable system 200 can include an outward-facing imaging system 464 (shown in FIG. 4) that observes the world in the environment around the user. The wearable system 200 can also include an inward-facing imaging system 462 (shown in FIG. 4) that can track the user's eye movements. The inward-facing imaging system can track either the movement of one eye or the movement of both eyes. The inward-facing imaging system 462 may be attached to the frame 230 and may communicate electrically with a processing module 260 or 270 that processes the image information acquired by the inward-facing imaging system and can determine, for example, the pupil diameter or orientation of the user 210's eyes, eye movements, or eye posture.

[0043] As an example, the wearable system 200 can use the outward-facing imaging system 464 or the inward-facing imaging system 462 to acquire an image of the user's posture. The image may be a still image, a video frame or video, a combination thereof, or the like.

[0044] The display 220 is operably coupled (250) to a local data processing module 260 and can be mounted in various configurations, such as fixed to the frame 230 by a wired conductor or wireless connection, fixed to a helmet or hat worn by the user, built into headphones, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, in a belt attachment configuration).

[0045] The local processing and data module 260 may include a digital memory such as a hardware processor and a non-volatile memory (e.g., flash memory), both of which can be used to assist in data processing, caching, and storage. The data may include a) data captured from sensors such as an image capture device (e.g., a camera within an inward-facing imaging system and / or an outward-facing imaging system), a microphone, an inertial measurement unit (IMU), an accelerometer, a compass, a global positioning system (GPS), a wireless device, or a gyroscope (e.g., operably coupled to frame 230 or otherwise attachable to user 210), or b) data obtained or processed using remote processing module 270 and / or remote data repository 280, optionally for transmission to display 220 after such processing or reading. The local processing and data module 260 may be operably coupled to the remote processing module 270 or the remote data repository 280 via communication links 262 or 264, such as via a wired or wireless communication link, so that these remote modules are available as resources to the local processing and data module 260. Additionally, the remote processing module 280 and the remote data repository 280 may be operably coupled to each other.

[0046] In some embodiments, the remote processing module 270 may include one or more processors configured to analyze and process data and / or image information. In some embodiments, the remote data repository 280 may include a digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module, enabling complete autonomy from the remote modules.

[0047] The human visual system is complex and it is difficult to provide a realistic perception of depth. Although not limited by theory, it is thought that an object viewer can perceive an object in three dimensions due to a combination of vergence and accommodation. The vergence of the two eyes relative to each other (i.e., the rotational movement of the pupils towards or away from each other to converge the lines of sight of the eyes and fix them on an object) is closely associated with the focusing (or "accommodation") of the eye's lens. Under normal conditions, changing the focus of the eye's lens, or accommodating the eye, to change the focus from one object to another at a different distance will automatically cause a corresponding change in vergence at the same distance under a relationship known as the "accommodation-vergence reflex". Similarly, a change in vergence will, under normal conditions, induce a corresponding change in accommodation. A display system that provides better correspondence between accommodation and vergence can form a more realistic and comfortable simulation of a three-dimensional image.

[0048] FIG. 3 illustrates a side view of an approach for simulating a three-dimensional image using a plurality of depth planes. Referring to FIG. 3, objects at various distances from eyes 302 and 304 on the z-axis are focused by eyes 302 and 304 such that those objects are in focus. Eyes 302 and 304 take on a particular focused state and focus objects at different distances along the z-axis. As a result, a particular focused state can be said to be associated with a particular one of depth planes 306 having an associated focal length such that an object or a portion of an object in a particular depth plane is in focus when the eye is in the focused state with respect to that depth plane. In some embodiments, the three-dimensional image may be simulated by providing different presentations of the image for each of eyes 302 and 304 and also by providing different presentations of the image corresponding to each of the depth planes. Although shown as being separate for clarity of illustration, it should be understood that the fields of view of eyes 302 and 304 may overlap, for example, as the distance along the z-axis increases. Additionally, although shown as being flat for ease of illustration, it should be understood that the contours of the depth planes may be curved in physical space such that all features within the depth plane are in focus with the eye in a particular focused state. Without being limited by theory, it is believed that the human eye can typically interpret a finite number of depth planes to provide depth perception. As a result, a highly realistic simulation of the perceived depth can be achieved by providing the eye with different representations of the image corresponding to each of these limited number of depth planes.

[0049] (Waveguide stack assembly) FIG. 4 illustrates an example of a waveguide stack for outputting image information to a user. Wearable system 400 includes a stack of waveguides or a stacked waveguide assembly 480 that can be utilized to provide three-dimensional perception to the eye / brain using a plurality of waveguides 432b, 434b, 436b, 438b, 400b. In some embodiments, wearable system 400 may correspond to wearable system 200 of FIG. 2, and FIG. 4 schematically shows some parts of wearable system 200 in more detail. For example, in some embodiments, waveguide assembly 480 may be integrated within display 220 of FIG. 2.

[0050] Continuing to refer to FIG. 4, waveguide assembly 480 may also include a plurality of features 458, 456, 454, 452 between the waveguides. In some embodiments, features 458, 456, 454, 452 may be lenses. In other embodiments, features 458, 456, 454, 452 may not be lenses. Rather, they may simply be spacers (e.g., a cladding layer or structure for forming an air gap).

[0051] Waveguides 432b, 434b, 436b, 438b, 440b or a plurality of lenses 458, 456, 454, 452 may be configured to transmit image information to the eye with various levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a specific depth plane and may be configured to output image information corresponding to that depth plane. Image input devices 420, 422, 424, 426, 428 may be utilized to input image information into waveguides 440b, 438b, 436b, 434b, 432b, respectively, and may be configured to disperse incident light across each individual waveguide for output toward the eye 410. Light exits from the output surfaces of image input devices 420, 422, 424, 426, 428 and is input into the corresponding input edges of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam of light (e.g., a collimated beam) may be input into each waveguide and output an entire field of cloned collimated beams directed toward the eye 410 at a specific angle (and divergence amount) corresponding to the depth plane associated with the specific waveguide.

[0052] In some embodiments, image input devices 420, 422, 424, 426, 428 are discrete displays that each generate image information for input into their respective corresponding waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, image input devices 420, 422, 424, 426, 428 are the output ends of a single multiplexed display that can pipe image information to each of image input devices 420, 422, 424, 426, 428, for example, via one or more optical conduits (such as an optical fiber cable).

[0053] Controller 460 controls the operation of the stacked waveguide assemblies 480 and the image input devices 420, 422, 424, 426, 428. Controller 460 includes programming (e.g., instructions in a non-transitory computer-readable medium) that adjusts the timing and provides the image information to waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, controller 460 may be a single integrated device or a distributed system connected by wired or wireless communication channels. Controller 460 may, in some embodiments, be part of processing module 260 or 270 (shown in FIG. 2).

[0054] The waveguides 440b, 438b, 436b, 434b, 432b may be configured to propagate light within each individual waveguide by total internal reflection (TIR). The waveguides 440b, 438b, 436b, 434b, 432b may each be planar, or have another shape (e.g., curved), with major top and bottom surfaces and an edge extending between those major top and bottom surfaces. In the illustrated configuration, the waveguides 440b, 438b, 436b, 434b, 432b each include light extraction optical elements 440a, 438a, 436a, 434a, 432a configured to redirect light, propagate it within each individual waveguide, and extract the light from the waveguide by outputting image information from the waveguide to the eye 410. The extracted light may also be referred to as external coupled light, and the light extraction optical elements may also be referred to as external coupling optical elements. The beam of the extracted light is output by the waveguide at the location where the light propagating within the waveguide impinges on the light redirecting element. The light extraction optical elements (440a, 438a, 436a, 434a, 432a) may be, for example, reflective or diffractive optical features. For ease of explanation and clarity of the drawings, they are shown disposed on the bottom major surfaces of the waveguides 440b, 438b, 436b, 434b, 432b, but in some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be disposed on the top or bottom major surfaces, or directly within the volume of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be attached to a transparent substrate and formed within a layer of the material forming the waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, 432b may be a monolithic piece of material, and the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed on and / or within the surface of that piece of material.

[0055] Continuing to refer to FIG. 4, as discussed herein, each of the waveguides 440b, 438b, 436b, 434b, 432b is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 as it is input into such waveguide 432b. The collimated light may represent an optically infinite focal plane. The next waveguide 434b may be configured to output collimated light that passes through a first lens 452 (e.g., a negative lens) before reaching the eye 410. The first lens 452 may be configured to generate a slight convex wavefront curvature such that the eye / brain interprets the light emerging from the next waveguide 434b as originating from a first focal plane that is closer inwardly from the optically infinite towards the eye 410. Similarly, the third waveguide 436b passes its output light through both the first lens 452 and the second lens 454 before reaching the eye 410. The combined refractive power of the first and second lenses 452 and 454 may be configured to generate another increment of wavefront curvature such that the eye / brain interprets the light emerging from the third waveguide 436b as originating from a second focal plane that is even closer inwardly from the optically infinite towards the person than the light from the next waveguide 434b was.

[0056] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured to send their output through all of the lenses between them and the eye for the aggregated focus power representing the focal plane closest to the person using the highest waveguide 440b in the stack. When viewing / interpreting light originating from the world 470 on the other side of the stacked waveguide assembly 480, a compensation lens layer 430 may be disposed on top of the stack to compensate for the stack of lenses 458, 456, 454, 452 to compensate for the aggregated power of the lower lens stack 458, 456, 454, 452. Such a configuration provides the same number of perceived focal planes as there are available waveguide / lens pairs. Both the light extraction optical elements of the waveguides and the focusing sides of the lenses may be static (e.g., not dynamic or electrically active). In some alternative embodiments, one or both may be dynamic using electrically active features.

[0057] Continuing to refer to FIG. 4, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be configured to redirect light from their respective waveguides and output the light using an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides having different associated depth planes may have light extraction optical elements of different configurations that output light using different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be three-dimensional or surface features configured to output light at a specific angle. For example, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be volume holograms, surface holograms, and / or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published Jun. 25, 2015, which is incorporated herein by reference in its entirety.

[0058] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffraction features that form a diffraction pattern, i.e., a "diffractive optical element" (also referred to herein as a "DOE"). Preferably, the DOE has a relatively low diffraction efficiency such that only a portion of the light of the beam is deflected toward the eye 410 using each intersection of the DOE, while the remainder continues to travel through the waveguide via total internal reflection. The light carrying the image information is thus split into several associated output beams that exit the waveguide at multiple locations, resulting in a very uniform output emission pattern toward the eye 304 with respect to this particular collimated beam that bounces within the waveguide.

[0059] In some embodiments, one or more DOEs may be switchable between an "on" state where they actively diffract and an "off" state where they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer dispersed liquid crystal in which microdroplets form a diffraction pattern, and the refractive index of the microdroplets can be switched to a refractive index that substantially matches that of the host material (in which case the pattern does not significantly diffract the incident light), or the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts the incident light).

[0060] In some embodiments, the number and distribution of depth planes or depth of field may vary dynamically based on the pupil size or orientation of the viewer's eye. The depth of field may vary inversely with the pupil size of the viewer. As a result, as the pupil size of the viewer's eye decreases, one plane that was indistinguishable because its location in the plane exceeded the depth of focus of the eye becomes distinguishable, and with the reduction in pupil size and the corresponding increase in depth of field, it may increase to appear more in focus. Similarly, the number of spaced-apart depth planes used to present different images to the viewer may be decreased with a decreased pupil size. For example, a viewer may not be able to clearly perceive the details of both a first depth plane and a second depth plane at one pupil size without adjusting the eye's focusing from one depth plane to the other. However, these two depth planes may be sufficient to come into focus for the user at another pupil size without changing the focusing adjustment simultaneously.

[0061] In some embodiments, the display system may vary the number of waveguides that receive image information based on a determination of pupil size and / or orientation, or in response to receiving an electrical signal indicating a particular pupil size / orientation. For example, if the user's eye is indistinguishable between two depth planes associated with two waveguides, the controller 460 may be configured or programmed to stop providing image information to one of these waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments where the DOE for the waveguide is switchable between on and off states, the DOE may be switched to the off state when the waveguide receives image information.

[0062] In some embodiments, it may be desirable to satisfy the condition that the emitted beam has a diameter less than the diameter of the viewer's eye. However, satisfying this condition can be difficult in light of the variability of the size of the viewer's pupil. In some embodiments, this condition is satisfied over a wide range of pupil sizes by varying the size of the emitted beam in response to a determination of the size of the viewer's pupil. For example, as the pupil size decreases, the size of the emitted beam may also decrease. In some embodiments, the emitted beam size may be varied using a variable aperture.

[0063] The wearable system 400 can include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 can be referred to as the field of view (FOV), and the imaging system 464 is sometimes also referred to as the FOV camera. The entire area available for viewing or imaging by the viewer can be referred to as the field of regard (FOR). The FOR may include a solid angle of 4π steradians surrounding the wearable system 400 so that the wearer can move their body, head, or eyes and perceive substantially any direction within the space. In other situations, the movement of the wearer may be more restricted, and accordingly, the wearer's FOR may contact a smaller solid angle. The image obtained from the outward-facing imaging system 464 can be used, for example, to track gestures made by the user (e.g., hand or finger gestures) and to detect objects within the world 470 in front of the user.

[0064] The wearable system 400 can also include an inward-facing imaging system 466 (e.g., a digital camera) that observes user movements such as eye movements and face movements. The inward-facing imaging system 466 can capture an image of the eye 410 and may be used to determine the size or orientation of the pupil of the eye 304. The inward-facing imaging system 466 can be used to obtain an image for determining the direction in which the user is looking (e.g., eye posture), or for biometric identification of the user (e.g., via iris identification). In some embodiments, at least one camera can be utilized to separately determine the pupil size or eye posture of each eye independently for each eye, thereby enabling the presentation of image information to each eye to be dynamically adjusted with respect to that eye. In some other embodiments, only the pupil diameter or orientation of a single eye 410 (e.g., using only a single camera per pair of eyes) is determined and assumed to be similar for both eyes of the user. The images obtained by the inward-facing imaging system 466 may be analyzed to determine the user's eye posture or mood, which can be used by the wearable system 400 to determine the audio or visual content to be presented to the user. The wearable system 400 can also use sensors such as an IMU, accelerometer, gyroscope, etc. to determine the head posture (e.g., head position or head orientation).

[0065] The wearable system 400 can include a user input device 466 through which a user can input commands to the controller 460 and interact with the wearable system 400. For example, the user input device 466 can include a trackpad, a touch screen, a joystick, a multi-degree-of-freedom (DOF) controller, a capacitance sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, a totem (e.g., functioning as a virtual user input device), etc. In some cases, the user may use a finger (e.g., the thumb) to press or swipe on a touch sensor-based input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held by the user's hand during use of the wearable system 400. The user input device 466 can communicate with the wearable system 400 either wired or wirelessly.

[0066] FIG. 5 shows an embodiment of an output beam output by a waveguide. One waveguide is shown, but other waveguides within waveguide assembly 480 may function similarly, and it should be understood that waveguide assembly 480 includes a plurality of waveguides. Light 520 is introduced into waveguide 432b at input edge 432c of waveguide 432b and propagates within waveguide 432b by TIR. At the point where light 520 impinges on DOE 432a, a portion of the light exits the waveguide as output beam 510. Output beams 510 are shown as being substantially parallel, but they may also be redirected to propagate at an angle to eye 410 depending on the depth plane associated with waveguide 432b (e.g., to form a diverging output beam). It should be understood that a substantially parallel output beam may represent a waveguide with an optical extraction optical element that externally couples the light and forms an image that appears to be set on the depth plane at a long distance (e.g., optical infinity) from eye 410. Other waveguides or other sets of optical extraction optical elements may output a more diverging output beam pattern that requires the eye 410 to focus at a closer distance and be interpreted by the brain as light from a distance closer to the eye 410 than optical infinity.

[0067] FIG. 6 is a schematic diagram showing an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem used in the generation of a multi-focus stereo display, an image, or a light field. The optical system can include a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem. The optical system can be used to generate a multi-focus stereo, an image, or a light field. The optical system can include one or more primary planar waveguides 632a (only one is shown in FIG. 6) and one or more DOEs 632b associated with at least some of the respective primary waveguides 632a. The planar waveguide 632b can be similar to the waveguides 432b, 434b, 436b, 438b, 440b discussed with reference to FIG. 4. The optical system can employ a diffractive waveguide device to relay light along a first axis (vertical or Y-axis in the view of FIG. 6) and expand the effective exit pupil of the light along the first axis (e.g., the Y-axis). The diffractive waveguide device can include, for example, a diffractive planar waveguide 622b and at least one DOE 622a (illustrated by the dashed line) associated with the diffractive planar waveguide 622b. The diffractive planar waveguide 622b can be similar or identical to the primary planar waveguide 632b having a different orientation at at least some points. Similarly, at least one DOE 622a can be similar or identical to the DOE 632a at at least some points. For example, the diffractive planar waveguide 622b or the DOE 622a can each be made of the same material as the primary planar waveguide 632b or the DOE 632a. The embodiment of the optical display system 600 shown in FIG. 6 can be integrated into the wearable system 200 shown in FIG. 2.

[0068] The relayed light with an expanded exit pupil can be optically coupled into one or more primary planar waveguides 632b from the diffractive waveguide device. The primary planar waveguide 632b can preferably relay light along a second axis (e.g., horizontal or X-axis in the view of FIG. 6) that is orthogonal to the first axis. It should be noted that the second axis can be a non-orthogonal axis with respect to the first axis. The primary planar waveguide 632b expands the effective exit pupil of the light along its second axis (e.g., X-axis). For example, the diffractive planar waveguide 622b can pass the light through the primary planar waveguide 632b that can relay and expand the light along the vertical or Y-axis and can also relay and expand the light along the horizontal or X-axis.

[0069] The optical system may include one or more colored light sources (e.g., red, green, and blue laser light) 610 that can be optically coupled into the proximal end of the single-mode optical fiber 640. The distal end of the optical fiber 640 may be screwed or received through the hollow tube 642 of the piezoelectric material. The distal end projects from the tube 642 as a flexible cantilever 644 that is not fixed. The piezoelectric tube 642 can be associated with four quadrant electrodes (not shown). The electrodes may be deposited, for example, on the outside, outer surface, or outer periphery of the tube 642, or on the diameter. A core electrode (not shown) may also be located at the core, center, inner periphery, or inner diameter of the tube 642.

[0070] For example, the drive electronics 650, which are electrically coupled via the wire 660, drive the opposing pair of electrodes and independently bend the piezoelectric tube 642 in two axes. The protruding distal tip of the optical fiber 644 has a mechanical resonance mode. The resonance frequency can depend on the diameter, length, and material properties of the optical fiber 644. By vibrating the piezoelectric tube 642 near the first mechanical resonance mode of the fiber cantilever 644, the fiber cantilever 644 can be vibrated and swept through a large deflection.

[0071] By stimulating resonant vibrations along two axes, the tip of the fiber cantilever 644 is scanned in two axial directions within the area filling the 2-D scan. By modulating the intensity of the light source 610 in synchronization with the scan of the fiber cantilever 644, the light emitted from the fiber cantilever 644 forms an image. An explanation of such a setup is provided in U.S. Patent Publication No. 2014 / 0003762, which is incorporated herein by reference in its entirety.

[0072] The components of the optical coupler subsystem can collimate the light emitted from the scanning fiber cantilever 644. The collimated light can be reflected by the mirrored surface 648 into a narrow-dispersion planar waveguide 622b containing at least one diffractive optical element (DOE) 622a. The collimated light propagates vertically (with respect to the view in FIG. 6) along the dispersion planar waveguide 622b by TIR, thereby repeatedly intersecting the DOE 622a. The DOE 622a preferably has a low diffraction efficiency. This diffracts a portion of the light (e.g., 10%) towards the edge of the larger primary planar waveguide 632b at each point of intersection with the DOE 622a and allows a portion of the light to continue on its original trajectory along the length of the dispersion planar waveguide 622b via TIR.

[0073] At each point of intersection with the DOE 622a, additional light can be diffracted towards the entrance of the primary waveguide 632b. By splitting the incident light into a plurality of external coupling sets, the exit pupil of the light can be vertically expanded by the DOE 4 within the dispersion planar waveguide 622b. The vertically expanded light externally coupled from the dispersion planar waveguide 622b can enter the edge of the primary planar waveguide 632b.

[0074] Light entering the first waveguide 632b can propagate horizontally along the first waveguide 632b (with respect to the figure in FIG. 6) via TIR. As the light intersects the DOE 632a at multiple points, it propagates horizontally along at least a portion of the length of the first waveguide 632b via TIR. The DOE 632a preferably has a phase profile that is advantageously the sum of a linear diffraction pattern and a radially symmetric diffraction pattern and can be designed or configured to generate both light deflection and focusing. The DOE 632a preferably has a low diffraction efficiency (e.g., 10%) such that only a portion of the light of the beam is deflected towards the viewer's eye at each intersection of the DOE 632a, while the remainder of the light continues to propagate through the first waveguide 632b via TIR.

[0075] At each point of intersection between the propagating light and the DOE 632a, a portion of the light is diffracted towards the adjacent surface of the first waveguide 632b, allowing the light to escape from TIR and be emitted from the surface of the first waveguide 632b. In some embodiments, the radially symmetric diffraction pattern of the DOE 632a additionally imparts a certain focal level to the diffracted light and shapes (e.g., imparts curvature to) the wavefronts of the individual beams and steers the beams to an angle that matches the designed focal level.

[0076] Accordingly, these different paths can couple light outside the primary planar waveguide 632b by resulting in different multiplicity of the DOE632a, focus levels, and / or filling patterns at different angles in the exit pupil. The different filling patterns in the exit pupil can advantageously be used to generate a light field display with multiple depth planes. Each layer within the waveguide assembly or a set of layers within a stack (e.g., three layers) may be employed to generate an individual color (e.g., red, blue, green). Thus, for example, a first set of three adjacent layers may be employed to generate red, blue, and green light at a first focal depth. A second set of three adjacent layers may each be employed to generate red, blue, and green light at a second focal depth. Multiple sets may be employed to generate a full 3D or 4D color image light field with various focal depths.

[0077] (Other components of the wearable system) In many implementations, the wearable system may include other components in addition to or instead of the components of the wearable system described above. The wearable system may include, for example, one or more haptic devices or components. The haptic device or component may be operable to provide a haptic sensation to the user. For example, the haptic device or component may provide a sensation of pressure and / or texture when touching virtual content (e.g., virtual objects, virtual tools, other virtual structures). The haptic sensation may reproduce the sensation of a physical object represented by the virtual object, or may reproduce the sensation of an imaginary object or character (e.g., a dragon) represented by the virtual content. In some implementations, the haptic device or component may be worn by the user (e.g., a user wearable glove). In some implementations, the haptic device or component may be held by the user.

[0078] A wearable system may include one or more physical objects that are, for example, operable by a user and enable input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as, for example, pieces of metal or plastic, walls, table surfaces, etc. In some implementations, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, a totem may simply provide a physical surface, and the wearable system may render a user interface so as to appear to the user to be on one or more surfaces of the totem. For example, the wearable system may render an image of a computer keyboard and trackpad so as to appear to be resident on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and virtual trackpad so as to appear to be on the surface of a thin rectangular plate of aluminum that serves as a totem. The rectangular plate itself does not have any physical keys or trackpad or sensors. However, the wearable system may detect user operations or interactions or touches using the rectangular plate as selections or inputs made via the virtual keyboard or virtual trackpad. The user input device 466 (shown in FIG. 4) may be an embodiment of a totem that may include a trackpad, touchpad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. The user may use the totem alone or in combination with a gesture to interact with the wearable system and / or other users.

[0079] Examples of haptic devices and totems that can be used with the wearable devices, HMDS, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated herein by reference in its entirety.

[0080] (Exemplary Wearable Systems, Environments, and Interfaces) The wearable system may employ various mapping-related techniques to achieve within a light field rendered with a high depth of field. When mapping the virtual world, it is advantageous to capture all features and points within the real world and accurately depict virtual objects in relation to the real world. To achieve this goal, the FOV images captured from the user of the wearable system can be added to the world model by including new photos that convey information about various points and features of the real world. For example, the wearable system can collect a set of map points (such as 2D points or 3D points), find new map points, and render a more accurate version of the world model. The world model of the first user can be communicated to the second user (e.g., via a network such as a cloud network) so that the second user can experience the world surrounding the first user.

[0081] FIG. 7 is a block diagram of an example of an MR environment 700. The MR environment 700 may be configured to receive inputs (e.g., visual input 702 from a user's wearable system, stationary input 704 such as an indoor camera, sensory input 706 from various sensors, gestures, totems, eye tracking, user input, etc. from user input device 504) from one or more user wearable systems (e.g., wearable system 200 or display system 220) or stationary indoor systems (e.g., indoor cameras, etc.). The wearable system may use various sensors (e.g., accelerometers, gyroscopes, temperature sensors, motion sensors, depth sensors, GPS sensors, inward-facing imaging systems, outward-facing imaging systems, etc.) to determine the location and various other attributes of the user's environment. This information may further be supplemented with information from stationary cameras in the room that may provide images or various cues from different viewpoints. Image data acquired by a camera (such as an indoor camera or a camera of an outward-facing imaging system) may be reduced to a set of mapping points.

[0082] One or more object recognition devices 708 may crawl through the received data (e.g., a set of points), recognize or map the points, tag the images, and associate semantic information with the objects using map database 710. The map database 710 may comprise various points collected over time and their corresponding objects. The various devices and map databases may be interconnected with each other through a network (e.g., LAN, WAN, etc.) and may access the cloud.

[0083] Based on the book information and set of points within the map database, the object recognition devices 708a - 708n may recognize an object, complement the object with semantic information, and assign a life to the object. For example, when the object recognition device recognizes that a set of points is a door, the system may associate some semantic information (e.g., the door has hinges and has a 90 - degree movement around the hinges). When the object recognition device recognizes that a set of points is a mirror, the system may associate the semantic information that the mirror has a reflective surface that can reflect images of objects within a room. Over time, the map database grows as the system (which may be resident locally or accessible through a wireless network) accumulates more data from the world. Once an object is recognized, the information may be transmitted to one or more wearable systems. For example, the MR environment 700 may include information about a scene generated in California. The environment 700 may be transmitted to one or more users in New York. Based on data received from the FOV camera and other inputs, the object recognition device and other software components can map points collected from various images so that the scene can be accurately "passed" to a second user who may be located in a different part of the world, and can recognize objects, etc. The environment 700 may also use a topological map for location - specific purposes.

[0084] FIG. 8 is a process flow diagram of an embodiment of a method 800 for rendering virtual content in relation to a recognized object. The method 800 describes a way in which a virtual scene can be presented to a user of a wearable system. The user may be geographically remote from the scene. For example, the user may be present in New York but may desire to view a scene currently taking place in California, or may desire to take a walk with a friend present in California.

[0085] In block 810, the AR system may receive input regarding the user's environment from the user and other users. This may be accomplished through various input devices and knowledge already held within the map database. The user's FOV camera, sensors, GPS, eye tracking, etc. communicate information to the system in block 810. The system may determine rough points based on this information in block 820. The rough points may be used when determining pose data (e.g., head pose, eye pose, body pose, or hand gesture) that can be used to display and understand the orientation and position of various objects in the user's surroundings. Object recognition devices 708a - 708n may crawl through these collected points in block 830 and use the map database to recognize one or more objects. This information may then be communicated to the user's individual wearable system in block 840, and a desired virtual scene may be appropriately displayed to the user in block 850. For example, a desired virtual scene (e.g., for a user in CA) may be displayed in appropriate orientations, positions, etc. in relation to various objects and other surroundings of a user in New York.

[0086] FIG. 9 is a block diagram of another embodiment of a wearable system. In this embodiment, the wearable system 900 comprises a map that may include map data regarding the world. The map may be resident locally on the wearable system in part and may be resident in part in a networked storage location (e.g., within a cloud system) accessible by a wired or wireless network. A pose process 910 is executed on a wearable computing architecture (e.g., processing module 260 or controller 460) and may utilize data from the map to determine the position and orientation of the wearable computing hardware or the user. The pose data may be calculated from data collected on-the-fly as the user experiences the system and operates within its world. The data may comprise images regarding objects in the real or virtual environment, data from sensors (e.g., an inertial measurement unit generally comprising accelerometer and gyroscope components), and surface information.

[0087] Coarse point representation may be the output of a simultaneous localization and mapping (SLAM or V-SLAM, referring to a configuration where the input is only image / vision) process. The system can be configured to find not only the locations of various components within the world but also what the world consists of. Pose may be a building block that achieves many goals, including filling in the map and using data from the map.

[0088] In one embodiment, the approximate point positions may not be entirely appropriate by themselves, and additional information may be required to generate a multi-focus AR, VR, or MR experience. A dense representation, generally referring to depth map information, may be utilized, at least in part, to fill this gap. Such information may be calculated from a process called stereoscopy 940, and the depth information may be determined using techniques such as triangulation or time-of-flight sensing. Image information and active patterns (such as infrared patterns generated using an active projector) may serve as inputs to the stereoscopy process 940. A significant amount of depth map information may be fused together, some of which may be summarized using surface representations. For example, a mathematically definable surface may be an efficient (e.g., compared to a large-scale point cloud) and summarizable input to other processing devices such as a game engine. Thus, the output of the stereoscopy process (e.g., depth map) 940 may be combined in the fusion process 930. Pose may similarly be an input to this fusion process 930, and the output of the fusion 930 serves as an input to fill the map process 920. Sub-surfaces may be interconnected in topographic mapping, etc., to form larger surfaces, and the map becomes a large-scale hybrid of points and surfaces.

[0089] To address various aspects in the composite reality process 960, various inputs may be utilized. For example, in the embodiment depicted in FIG. 9, game parameters may be inputs for determining that the user of the system is playing a monster battle game with one or more monsters in various locations, that the monster is dead, is fleeing under various conditions (such as when the user shoots the monster), walls or other objects in various locations, and the like. The world map may include information regarding where such objects exist relative to each other, serving as another useful input for composite reality. The pose with respect to the world may similarly be an input and plays an important role for almost any two-way system.

[0090] Control or input from the user is another input to the wearable system 900. As described herein, user input can include visual input, gestures, totems, audio input, sensory input, etc. For example, for moving around or playing a game, the user may need to instruct the wearable system 900 as to what they want to do. There are various forms of user control that can be utilized, not only for moving oneself within a space. In one embodiment, a totem (e.g., a user input device), or an object such as a toy gun, may be held by the user and tracked by the system. The system will preferably be configured to sense that the user is holding the item and understand the type of interaction the user is having with the item (e.g., if the totem or object is a gun, the system may be equipped with sensors such as an IMU that can assist in determining what is happening, not only the location and orientation, but also if the user is clicking a trigger or other sensing button or element, even when such an activity is not within the field of view of any of the cameras).

[0091] Hand gesture tracking or recognition may also provide input information. The wearable system 900 may be configured to track and interpret hand gestures for button presses, gestures such as left or right, stop, grip, hold, etc. For example, in one configuration, the user may desire to flip through an email or calendar in a non-game environment, or perform a "fist bump" with another person or performer. The wearable system 900 may be configured to utilize a minimal amount of hand gestures, which may be dynamic or not. For example, the gestures may be simple static gestures such as spreading the hand to indicate stop, raising the thumb to indicate okay, lowering the thumb to indicate not okay, or flipping the hand left or right or up or down to indicate a directional command.

[0092] Eye tracking is another input (e.g., tracking where the user is looking, controlling display technology, and rendering at a specific depth or range). In one embodiment, the convergence / divergence movement of the eyes may be determined using triangulation, and then the focus adjustment may be determined using a convergence / divergence movement / focus adjustment model developed for that particular person.

[0093] Regarding the camera system, the exemplary wearable system 900 shown in FIG. 9 can include three pairs of cameras, namely, a relatively wide FOV or passive SLAM pair of cameras arranged on both sides of the user's face, and a different pair of cameras oriented in front of the user to handle the stereoscopic imaging process 940 and capture hand gestures and the trajectory of totems / objects in front of the user's face. The FOV cameras and the pair of cameras for the stereo process 940 may be part of the outward-facing imaging system 464 (shown in FIG. 4). The wearable system 900 can include an eye-tracking camera (which may be part of the inward-facing imaging system 462 shown in FIG. 4) oriented towards the user's eyes for triangulating eye vectors and other information. The wearable system 900 may also include one or more textured light projectors (such as an infrared (IR) projector), and may project textures into the scene.

[0094] FIG. 10 is a process flow diagram of an example of a method 1000 for determining user input to a wearable system. In this example, the user may interact with totems. The user may have multiple totems. For example, the user may have one designated totem for a social media application, another totem for playing games, etc. In block 1010, the wearable system may detect the movement of the totem. The movement of the totem may be recognized through the outward-facing system, or may be detected through sensors (such as a tactile glove, an image sensor, a hand tracking device, an eye-tracking camera, a head pose sensor, etc.).

[0095] At least in part, based on input through a detected gesture, eye gesture, head gesture, or totem, the wearable system detects, at block 1020, the position, orientation, and / or movement of the totem (or the user's eyes or head or gesture) relative to a reference frame. The reference frame may be a set of map points based on which the wearable system converts the movement of the totem (or the user) into an action or command. At block 1030, the user's interaction with the totem is mapped. Based on the mapping of the user interaction relative to the reference frame 1020, the system determines, at block 1040, the user input.

[0096] For example, the user may move a totem or physical object back and forth to scroll a virtual page, move to the next page, or move from one user interface (UI) display screen to another. As another example, the user may move their head or eyes to view different real or virtual objects within the user's FOR. If the user's line of sight at a particular real or virtual object is longer than a threshold time, that real or virtual object may be selected as the user input. In some implementations, the user's eye convergence / divergence movement can be tracked, and a focus adjustment / convergence / divergence movement model can be used to determine the user's eye focus adjustment state that provides information about the depth plane at which the user is in focus. In some implementations, the wearable system can use a raycasting technique to determine real or virtual objects along the direction of the user's head gesture or eye gesture. In various implementations, the raycasting technique can include casting a thin beam of light with substantially little lateral width or a beam of light with a substantial lateral width (e.g., a cone or frustum of a cone).

[0097] The user interface may be projected by a display system (such as display 220 in FIG. 2) as described herein. It may also be presented using a variety of other techniques such as one or more projectors. The projector may project the image onto a physical object such as a canvas or a sphere. Interaction with the user interface may be tracked using one or more cameras external to the system or part of the system (e.g., using the imaging system 462 facing inward or the imaging system 464 facing outward).

[0098] FIG. 11 is a process flow diagram of an embodiment of a method 1100 for interacting with a virtual user interface. The method 1100 may be performed by a wearable system as described herein.

[0099] In block 1110, the wearable system may identify a particular UI. The type of UI may be provided by the user. The wearable system may identify that a particular UI needs to be populated based on user input (e.g., gestures, visual data, audio data, sensory data, direct commands, etc.). In block 1120, the wearable system may generate data for the virtual UI. For example, data associated with the boundaries, general structure, shape, etc. of the UI may be generated. Additionally, the wearable system may determine the map coordinates of the user's physical location so that the wearable system can display the UI in relation to the user's physical location. For example, if the UI is body-centered, the wearable system may determine the coordinates of the user's physical standing position, head pose, or eye pose so that a ring UI can be displayed around the user or a flat UI can be displayed on a wall or in front of the user. If the UI is hand-centered, the map coordinates of the user's hand may be determined. These map points may be derived through a FOV camera, data received through sensory input, or any other type of collected data.

[0100] In block 1130, the wearable system may send data from the cloud to the display, or the data may be sent from the local database to the display component. In block 1140, the UI is presented to the user based on the sent data. For example, a light field display can project a virtual UI into one or both of the user's eyes. Once the virtual UI is generated, the wearable system may, in block 1150, simply wait for commands from the user and generate more virtual content on the virtual UI. For example, the UI may be a body-centered ring around the user's body. The wearable system may then wait for commands (such as gestures, head or eye movements, input from a user input device, etc.), and if recognized (block 1160), the virtual content associated with the command may be presented to the user (block 1170). As an example, the wearable system may wait for a gesture of the user's hand before mixing multiple stem tracks.

[0101] Additional examples of AR systems, UIs, and user experiences (UX) are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated herein by reference in its entirety.

[0102] (Exemplary Objects within the Field of Ocular Rotation (FOR) and Field of View (FOV)) FIG. 12 schematically illustrates an example of virtual objects within the field of view (FOV) and virtual objects within the field of regard (FOR). As discussed with reference to FIG. 4, the FOR comprises a portion of the user's surrounding environment that is perceivable by the user via a wearable system. In FIG. 12, the FOR 1200 can contain a group of objects (e.g., 1210, 1220, 1230, 1242, and 1244) that are perceivable by the user via the wearable system. The objects within the user's FOR 1200 may be virtual and / or physical objects. For example, the user's FOR 1200 may include physical objects such as a chair, a sofa, a wall, etc. Virtual objects may include, for example, operating system objects such as a trash can for deleted files, a terminal for entering commands, a file manager for accessing files or directories, icons, menus, applications for audio or video streaming, notifications from the operating system, etc. Virtual objects may also include, for example, objects within an application such as an avatar, virtual objects within a game, graphics or images. Some virtual objects can be both operating system objects and objects within an application. In some embodiments, the wearable system can add virtual elements to existing physical objects. For example, the wearable system may add a virtual menu associated with a television in a room, and the virtual menu may provide the user with options to turn the television on or change channels using the wearable system.

[0103] The virtual object may be a three-dimensional (3D), two-dimensional (2D), or one-dimensional (1D) object. For example, as schematically illustrated in FIG. 16, the virtual object may be a 3D coffee mug 1636 (which may represent virtual control for a physical coffee maker). The virtual object may also be a 2D graphical representation of a wall clock 1634 (which displays the user's current time). In some implementations, one or more virtual objects may be displayed within (or associated with) another virtual object. For example, referring to FIG. 13, the virtual coffee mug 1636 is shown inside the user interface plane 1514, but the virtual coffee mug appears to be 3D within this 2D planar virtual space.

[0104] An object within the user's FOR can be part of a world map as described with reference to FIG. 9. Data associated with the object (e.g., location, semantic information, properties, etc.) can be stored within various data structures such as, for example, arrays, lists, trees, hashes, graphs, etc. The index of each stored object may be determined, where applicable, for example, by the location of the object. For example, the data structure may index the object by a single coordinate such as the distance of the object from a reference position (e.g., the distance to the left or right of the reference position, the distance from the top or bottom of the reference position, or the depth from the reference position). The reference position may be determined based on the user's position (such as the position of the user's head). The reference position may also be determined based on the position of a virtual or physical object (such as a target interaction-capable object) within the user's environment. Thus, the 3D space within the user's environment may be folded into the 2D user interface, and the virtual objects are arranged according to the distance of the object from the reference position.

[0105] Within the FOR 1200, the portion of the world that the user perceives at a given time is referred to as the FOV 1250 (e.g., the FOV 1250 may encompass the portion of the FOR that the user is currently looking at). In FIG. 12, the FOV 1250 is schematically illustrated by the dashed line 1252. The user of the wearable system can perceive a plurality of objects within the FOV 1250, such as the object 1242, the object 1244, and a portion of the object 1230. The FOV can depend on the size or optical characteristics of the display of the wearable device. For example, an AR display may include optics that provide only AR functionality when the user looks through a particular portion of the display. The FOV 1250 may correspond to the solid angle perceivable by the user when looking through an AR display, such as the stacked waveguide assembly 480 (FIG. 4) or the planar waveguide 600 (FIG. 6).

[0106] As the user's posture (e.g., head posture or eye posture) changes, the FOV 1250 also correspondingly changes, and the objects within the FOV 1250 may also change. For example, the map 1210 is initially outside the user's FOV in FIG. 12. When the user looks towards the map 1210, the map 1210 may move into the user's FOV 1250, and (for example) the object 1230 may move outside the user's FOV 1250. As will be described herein, the wearable system may track the objects within the FOR 1200 and the objects within the FOV 1250.

[0107] (Examples of Interactable Objects) In FIG. 12, the user can interact with a subset of the objects within the user's FOR 1200. This subset of objects may sometimes be referred to as interactable objects. In some implementations, the interactable objects may include all of the objects (virtual and physical) within the user's environment, while in other implementations, the interactable objects may include only a portion of the objects within the user's environment.

[0108] The wearable system can identify a subgroup of interactive objects (e.g., 1242, 1244, and 1230) that are within the user's FOV1250. The subgroup of interactive objects within the FOV are sometimes also referred to as selectable objects because the user can currently perceive them and can select them (e.g., move them, activate them, obtain information about them, etc.). As discussed herein, when the user moves their body, head, or eyes, the user's FOV can change. Generally, some objects will remain within the FOV, some objects will move out of the FOV (and are no longer selectable), and other objects that were outside the FOV will move into the FOV (and become selectable). Accordingly, the wearable system can update the subgroup of interactive objects within the FOV based on the user's body, head, or eye posture.

[0109] The wearable system may identify target-interactable objects within the user's FOV. The target-interactable objects may be objects that the user desires to interact with or objects that the wearable system anticipates the user will interact with (e.g., an interactable object that the user is looking at or the nearest interactable object to the center of the user's FOV). The target-interactable objects can be identified using various rules such as the location of the object, the user's preferences, or the user's posture. For example, the wearable system can select the object closest to the center of the FOV as the target-interactable object. The wearable system can also select the leftmost or rightmost object within the user's FOV as the target-interactable object. As another example, the wearable system can use an imaging system 462 (shown in FIG. 4) facing inward, either alone or in combination with an IMU, to determine the direction of the user's line of sight. The wearable system can identify an object that collides with the direction of the user's line of sight as the target-interactable object.

[0110] In some implementations, the AR system can automatically orient the target-interactable object so that the normal of the interactable object faces the user. For example, a virtual TV screen may initially face upward towards the ceiling of the room. Once the AR system determines that the user is looking towards the virtual TV screen, the AR system can automatically rotate the virtual TV screen so that it faces the user.

[0111] (Examples of focus indicators) The wearable system may assign a focus indicator to a target-interactable object so that a user can more easily perceive the target-interactable object. The focus indicator can be displayed to the user. For example, the focus indicator can have a backlight, a color, a change in perceived size or depth (e.g., making the target object appear closer and / or larger when selected), or other visual effects that draw the user's attention. The focus indicator can also include audible or tactile effects such as vibrations, ringtones, beeps, etc.

[0112] In some embodiments, the wearable system first identifies an object as a target-interactable object based on the rules described herein and may change the target-interactable object to another object based on a change in the user's posture. As a result, the focus indicator may move from one object to another as the user changes their posture.

[0113] The wearable system may also display a cursor corresponding to the user's current position. The cursor may take various shapes such as a geometric cone, a beam of light, a crosshair, an arrow, an oval, a circle, a polygon, or other 1D, 2D, or 3D shapes. The cursor may be presented in the same form as the focus indicator. For example, the cursor may have the same visual, audio, or tactile effects as the focus indicator. As an example, the cursor may be a crosshair corresponding to the user's head position. As another example, the cursor may have the shape of an arrow corresponding to the current position associated with the user input device. As the user changes their posture or activates the user input device, the cursor can move accordingly. The cursor may indicate one or more objects or the empty space within the user's environment as the user moves. For example, referring to FIG. 16, the AR system can move the cursor on the virtual user interface 1514 from position 1620 to position 1624 or from position 1622 to position 1620, etc.

[0114] In addition to, or as an alternative to, the focus indicator, the wearable system can present a cursor. For example, in FIG. 15, the wearable system can either display a crosshair (which can correspond to the direction of the user's line of sight) or provide a thin blue backlight as the focus indicator, or both, on the virtual object 1514. In some implementations, the cursor is an embodiment of the focus indicator. For example, when the user is looking at a virtual object, the wearable system may present a backlight around the virtual object, a crosshair object, or an arrow on the object. These visual indications may represent both the target object that the user is interested in interacting with and the user's current position.

[0115] (Exemplary Interactions with Interactable Objects) The user can interact with the interactive objects within the user's FOR1200, particularly the interactive objects within the user's current FOV1250, through the wearable system. For example, the virtual object 1230 may be a graph showing the change in stock price over time. By selecting the virtual object 1230, the user can interact with the virtual object 1230, for example, obtain stock price information, purchase or sell stocks, obtain information about the company, etc. To perform these interactions, the wearable system may display a menu, toolbar, etc. associated with the virtual object that enables the user to perform various actions (such as obtaining stock price information).

[0116] The user can interact with the objects within their FOV using various techniques, such as, for example, selecting an object, moving an object, opening a menu or toolbar associated with the object, or selecting a new set of selectable objects. The user can use hand gestures to interact with the objects, such as clicking a mouse, tapping on a touchpad, swiping on a touch screen, hovering over or touching a capacitive button, pressing a key on a keyboard or game controller (e.g., a 5-way D-pad), pointing a joystick, wand, or totem towards the object, pressing a button on a remote control, or other interactions with the user input device (see, e.g., user input device 466 in FIG. 4). The user can also use head, eye, hand, foot, or other body postures, such as, for example, gazing at an object over a period of time, pointing with an arm, tapping a foot, blinking the eyes a certain number of times over a threshold time interval, etc., to interact with the interactive objects. These hand gestures on the user input device and the user's postures can cause the AR system to initiate a selection event, such as, for example, a user interface action being performed (a menu associated with the target interactive object being displayed, a game action being performed on an avatar within the game, etc.).

[0117] In response to the initiation of the selection event, the AR system can assign a focus indicator to the target interactive object within the user's FOV using the rules described herein. For example, in FIG. 12, the AR system may assign the focus indicator to object 1244 because it is the closest to the center point of the FOV.

[0118] During a selection event, the user can use various hand gestures described herein to change the target-interactable object. For example, in FIG. 12, the user can swipe left on the touch screen, which can cause the AR system to change the target-interactable object from object 1244 to object 1230. The AR system can also accordingly move the visible focus indicator from object 1244 to object 1230.

[0119] In some implementations, the hand gesture may cause the AR system to update the list of selectable objects within the user's FOV. For example, in FIG. 12, when the user swipes right, the AR system can move object 1210 into the user's FOV and move object 1230 out of the user's FOV. The AR system may also update the target-interactable object based on the new group of selectable objects. For example, after object 1210 is moved into the FOV, the system may change the target-interactable object from object 1244 to object 1242.

[0120] The user can confirm the selection of the target-interactable object using the hand gestures or postures discussed herein. The user's action to confirm the selection of the target-interactable object may be the same or different from the action used to initiate the selection event. The AR system may change the focus indicator when the user confirms the selection, for example, by changing the color, brightness, or shape of the focus indicator.

[0121] The user can select a series of user interface operations on the target interactable object. These operations can sometimes be referred to as interaction events. Interaction events can include, for example, resizing the interactable object, displaying a menu of the interactable object, browsing the menu, selecting an item on the menu, searching for an item, playing a game, watching a video, conducting a video conference, previewing the target interactable object, etc. Interaction events can occur in parallel with, or sequentially to, selection events. In some implementations, interaction events may be part of the selection event.

[0122] In some embodiments, once the selection event is initiated, the wearable system may "lock" the user's FOV so that the wearable system will stop updating the group of selectable objects within the user's FOV even if the user's FOV changes after the selection event is initiated. In some implementations, the user can still move the focus indicator between selectable objects within the user's FOV through the actuation or pose change of the user input device.

[0123] The selection event can be terminated by user input or other interaction with the wearable system. For example, the selection event can be terminated by confirming the selection of the target interactable object, initiating an interaction event, actuating the user input device, terminating the selection event, having the effect of terminating the selection event, determining a change in head or body pose, etc.

[0124] (Example of selection of a virtual object in 3D space using hand gestures) The user may target and select an interactive object by operating a user input device. FIG. 13A is an example of the selection of an interactive object using a touch gesture on the touch screen 1310 of the user input device 1300. The user input device may be an embodiment of the user input device 466 shown in FIG. 4. The touch gesture can trigger the wearable system and assign a focus indicator to a target interactive object within the user's FOV. The touch gesture can also, alone or in combination, cause the wearable system to initiate a selection event, initiate an interaction event, end a selection event, end an interaction event, confirm the selection of a target interactive object, etc.

[0125] FIG. 13B is an example of the filtering of selectable objects using a hand gesture on the user input device. The user may swipe along a path on the user input device 1300. For example, as indicated by the arrow 1314 in FIG. 13B, the user may swipe along a path towards the right on the touch screen 1310. Any type of path can be used (e.g., horizontal, vertical, diagonal, or other trajectories with respect to the input device), or any type of direction can be used (e.g., left or right, up or down, etc.).

[0126] A swipe gesture can move a visible focus indicator from one object to another in a wearable system. Referring to the embodiment shown in FIG. 12, when the user swipes right (as shown in the embodiment in FIG. 13B), the AR system can shift the focus indicator from object 1244 to object 1242. In some embodiments, the swipe gesture can cause the AR system to update a list of selectable objects within the user's FOV. For example, when the user swipes right, the AR system can move object 1210 into the user's FOV and move object 1230 out of the user's FOV. The object receiving the visible focus indicator can also be updated accordingly (e.g., from object 1244 to 1242).

[0127] The swipe gesture can be used in combination with touch gestures (described with reference to FIG. 13A) and head poses (described with reference to FIG. 14) to filter and select 3D virtual objects within the FOR or FOV. As discussed herein, the user may also use other gestures or movements to change the set of selectable objects and change the target interactable object.

[0128] (Example of selection of virtual objects in 3D space using head pose) FIG. 14 is an example of a coordinate system for head pose. The head 1410 may have multiple degrees of freedom. As the head 1410 moves in different directions, the head pose will change relative to the natural rest direction 1420. The coordinate system in FIG. 14 shows three angular degrees of freedom (e.g., yaw, pitch, and roll) that can be used to measure the head pose relative to the natural rest state 1420 of the head. As shown in FIG. 14, the head 1410 can tilt in the forward and reverse directions (e.g., pitch), turn left and right (e.g., yaw), and tilt laterally (e.g., roll). In other implementations, other techniques or angular representations for measuring head pose can also be used, e.g., any other type of Euler angle system.

[0129] The wearable system can determine the head pose of a user using various sensors described herein (see, e.g., FIGS. 2, 4, and 7). For example, the wearable system can use an IMU or an inward-facing imaging system to calculate the head pose or eye pose of the user. The wearable system can use the data acquired by these sensors to identify a target interactable object. For example, the target interactable object may be an object that collides with the direction of the user's line of sight. The wearable system can identify the target interactable object and assign a visible focus indicator to the target interactable object based on a line of sight extending in a certain direction (e.g., the focus indicator is assigned to the target object if the user looks at the object for longer than a threshold time).

[0130] The wearable system can determine and update an object to be a target interaction - capable object based on changes in the head posture, such as roll, yaw, or pitch during a selected event. For example, referring to FIG. 12, when the user turns his head 1410 to the left, the target interaction - capable object may be updated from object 1244 to its neighboring object such as object 1230. In some implementations, the wearable system can shift the focus indicator from object 1244 to object 1230 to reflect this update.

[0131] The wearable system can also update a list of selectable objects within the user's FOV based on changes in the head posture. For example, the user may turn his head 1410 to the right, which can cause the wearable system to move object 1230 out of the user's FOV and move object 1210 into the user's FOV 1250. The target interaction - capable object can also be updated accordingly using the rules described herein.

[0132] The user may use various changes in the head posture or eye posture to switch between multiple planes. The user may also use fluctuations in the head posture or eye posture to start a selection event, confirm the selection of a target interaction - capable object, start an interaction event, interact with the target interaction - capable object during the interaction event, end a selection event, or perform other interactions with the user interface.

[0133] In some implementations, the wearable system can associate a depth plane with the user's head movement so that the user can interact with virtual objects only in that depth plane. For example, the wearable system can set a cursor (e.g., a crosshair) corresponding to the user's head position in a certain depth plane. As a result, as the user moves their head, the cursor is shifted within the set depth plane, and the user can select between objects in the set depth plane even if other virtual objects are in different depth planes. In some implementations, the depth plane may be accompanied by a virtual user interface. The wearable system can set the crosshair in the depth plane of the target virtual user interface so that the user can interact with virtual objects within the target user interface plane.

[0134] (Exemplary interactions using a combination of head pose and hand gestures on a user input device) The user can also select virtual objects using a combination of the user's pose and the user's hand gestures. As an example, the user can perceive a group of virtual objects within the user's FOV. This group of virtual objects may be an embodiment of the selectable objects described herein. The group of virtual objects may be presented using the light field display described with reference to FIGS. 2, 4-6. The light field display can project virtual objects in different depth planes to the user such that some virtual objects can appear in front of other virtual objects.

[0135] The wearable system can maintain an array of virtual objects that are currently within the user's FOV. The wearable system may use the position of a virtual object within the user's environment as an array index for the virtual object. For example, the wearable system can use the y-value of a virtual object within x-y-z coordinates (e.g., the x-y-z coordinates shown in FIG. 6) as the array index for the object. In other embodiments, the wearable system can use the x-value or the z-value, alone or in combination with the y-value, to determine the array index of a virtual object.

[0136] As the user moves their head or the direction of their line of sight, the group of virtual objects that appear within the user's FOV may change. The wearable system can also update the array accordingly. In some implementations, the wearable system can use the array to maintain virtual objects within the user's FOR and identify the group of virtual objects within the FOV when the user initiates a selection event.

[0137] The user can initiate a selection event on a virtual object within the user's FOV by activating a user input device. In response to the initiation of the selection event, the wearable system can present a user interface that includes all (or a portion) of the virtual objects within the user's FOV. In some implementations, the wearable system can display "hidden virtual objects" such as a virtual user interface menu or certain information about the virtual object. The "hidden virtual objects" may become perceptible in response to the initiation of the selection event, but are hidden prior to, or after, the initiation of the selection event.

[0138] In some embodiments, in response to the start of a selection event, the wearable system may change the position of virtual objects within the user's FOV. For example, the wearable system may move a distant object closer to the user or a nearby object farther away from the user such that all virtual objects appear on substantially the same depth plane. In some implementations, the wearable system may change (increase or decrease) the size of a virtual object so that the virtual object can fit the size of the user's FOV. Additionally, or alternatively, the wearable system may show a part of a virtual object (such as showing an icon instead of the content of a virtual email application) to the user.

[0139] The wearable system may also group virtual objects within the user's FOV into multiple depth planes in response to the start of a selection event. Each depth plane may be associated with a virtual user interface.

[0140] The wearable system can use the array index of a virtual object to arrange the virtual object on one or more user interfaces. For example, the wearable system can present virtual objects whose y-axis values are within a certain range together on the same user interface. Additionally, or alternatively, the wearable system can arrange virtual objects on a depth plane based on the user's pose. For example, if there are multiple virtual objects in the direction of the user's line of sight, since they are on different depth planes within the user's environment, the wearable system can present these virtual objects inside the user's FOV while installing other virtual objects outside the user's FOV in response to the start of a selection event. The user can move virtual objects in and out of the FOV using the techniques described with reference to FIGS. 12 and 13.

[0141] The wearable system can identify a target-interactable object and present a focus indicator indicating the target-interactable object. The wearable system can rearrange virtual objects in the vicinity of the target-interactable object and present the rearranged virtual objects within the user's FOV. For example, referring to FIG. 12, the wearable system can identify a group of virtual objects in the vicinity of the target-interactable object based on the distance of the virtual objects from a reference position (such as the position of the target-interactable object). The wearable system can rearrange the positions of these virtual objects based on values within the array index x-y-z coordinates (shown in FIG. 6), or based on the distance from the target-interactable object. As shown in FIG. 12, objects 1242, 1244, and 1230 may have different initial positions within the user's FOR. For example, object 1242 may be positioned higher (e.g., closer to the ceiling of the user's room) than object 1244 and may be farther from the user than object 1244. The initial position of object 1230 may be lower (e.g., closer to the floor of the user's room) than object 1244 within the user's FOR. When the wearable system identifies object 1244 as the target-interactable object, the wearable system can "collapse" the user's 3D space into a 2D user interface based on the y-axis value, and objects with larger y-axis values are positioned on the left side of the user's FOV. Accordingly, within FOV 1250, virtual object 1242 appears to the left of virtual object 1230 and to the left of virtual object 1244. In other embodiments, different techniques for rearranging virtual objects may be used. For example, FOV 1250 may show a 2D projection of the virtual objects in 3D space. As explained in FIG. 12, the user can use hand gestures to move the focus indicator between objects within the FOV.The user can also use hand gestures to change the target-interactable object, which can cause the wearable system to present another set of virtual objects within the FOV based on the rearrangement of virtual objects near the new target-interactable object. If the wearable system is configured to present multiple user interfaces on different depth planes, the user can also use hand gestures to shift the focus indicator between the multiple depth planes and switch the user interface. In some embodiments, the wearable system may restore the position of the virtual object to its original position prior to rearrangement when the virtual object is no longer within the user's FOV. In some situations, the virtual user interface presenting the rearranged virtual objects may be generated after the user has initiated a selection event on the target-interactable object. Additional details regarding interaction with multiple user interfaces are further described with reference to FIG. 15.

[0142] (Examples of User Interaction Based on Context Information) A wearable system can automatically select or recommend a mode of user interaction (e.g., a posture or a hand gesture on a user input device) based on context information. The context information can include the type of object (e.g., physical or virtual), the layout of the objects (e.g., the density of the objects, the location and size of the objects, etc.), the characteristics of the user, or the user's current interaction, combination, or equivalent with the objects in the environment. For example, during ray casting (described with reference to FIG. 10), the wearable system can detect that the user is looking at a plurality of virtual objects located in close proximity to each other. The wearable system can calculate the density of the virtual objects within the user's FOV. When the density exceeds a certain threshold, the wearable system can recommend to the user to switch the mode of user interaction. For example, when the density exceeds a certain threshold (indicating that the objects are located very close to each other), the wearable system can switch the mode of user interaction from a head posture to a hand gesture on a user input device to enable a more precise interaction with the objects. As another example, when the density drops below a certain threshold (indicating that the objects are separated from each other), the wearable system can switch the mode of user interaction from a hand gesture on a user input device to a head posture. These implementations can be particularly advantageous because the head position can be difficult to control accurately and can fatigue the user when the user tries to position their head precisely to interact with objects clustered at a high density. On the other hand, hand gestures on a user input device can provide more refined control of the user's position but can fatigue the user when the user needs to move their hand over a long distance to select an object located at a low density.

[0143] Figures 15 and 16 provide examples of changing the mode of user interaction based on context information. Figure 15 illustrates an example of interaction with an interactive object using a head pose. Figure 16 illustrates an example of interaction with an interactive object using a hand gesture on a user input device. The user input device 1610 shown in Figure 16 may be an embodiment of the user input device 466 described in Figure 4.

[0144] In Figure 15, the user's FOR includes interactive objects such as virtual user interfaces 1512, 1514, and 1516. In some implementations, the virtual user interfaces 1512, 1514, and 1516 may be planar objects that include other virtual objects (1D, 2D, or 3D) within the virtual user interface plane. The virtual user interfaces 1512, 1514, and 1516 are large-sized and not located adjacent to each other at high density. As a result, the wearable system may determine that head movement can be the optimal mode of user interaction because the user does not need to move a long distance on the user input device to select a virtual user interface. The wearable system may use a raycasting technique to identify that the user is currently looking at the virtual user interface 1514. The user may start a selection event on the virtual user interface 1514 and interact with the objects within the user interface 1514. As shown in Figure 16, the virtual user interface 1514 may include a plurality of virtual objects, such as, for example, a virtual TV screen 1632, a virtual coffee cup 1636, a virtual wall clock 1634, a camera application 1652, a weather application 1638, and a music application.

[0145] (Examples of Recommendations for Modes of User Interaction Based on Context Information) As an example, in FIG. 16, the wearable system can determine the relative positions among the virtual TV 1632, the virtual hanging clock 1634, the virtual coffee mug 1636, and the weather application 1638. Since these four objects are close to each other, when the wearable system determines that the user's current position is at position 1620, the wearable system may present the user with an option of whether to desire to switch from the head pose to the hand control.

[0146] On the other hand, the object 1652 (camera application) does not have other objects in its vicinity. Accordingly, the wearable system may not provide an option to interact with the object 1652 using the user input device. However, the user may still interact with the object 1652 using postures such as the head pose and the eye pose.

[0147] The user can confirm the switching of the user interaction mode by using the user input device or by changing the body posture (such as nodding the head). When the user interaction mode is switched to the user input device, the user can activate the user input device 1610 and interact with the virtual object. For example, the user can swipe along a path on the user input device 1610, which moves the cursor from position 1620 to position 1624. Similarly, the user can activate the user input device 1610, which moves the cursor (which may be in the shape of an arrow) from position 1620 to 1622. In addition to these examples, the user may swipe along any type of path (e.g., horizontal, vertical, or diagonal with respect to the input device) or any type of direction (e.g., left or right, up or down, etc.) on the user input device 1610.

[0148] When the cursor is at position 1624, a part of the cursor overlaps with the virtual TV screen 1632. The user may operate the user input device 1610 (e.g., by clicking on the touch screen) to select the virtual TV screen 1632. When the wearable system receives the selection of the virtual TV screen 1632, the wearable system may display one or more virtual menus (e.g., virtual menus 1642a and 1642b) associated with the TV screen. For example, the virtual menu may include options such as adjusting the tone, selecting a video application (e.g., a movie or a TV streaming service), selecting to start a conference call, etc.

[0149] (Example of automatic switching of user interaction mode based on context information) The wearable system can also automatically switch the mode of user interaction. For example, the mode of user interaction may be set to the head pose when the user selects among the virtual user interface planes 1512, 1514, 1516. Once the user selects a virtual user interface plane, the mode of user interaction may be automatically changed to hand gestures on the user input device (as shown in FIG. 16). As another example, the mode of user interaction can be set to the body pose when the objects are sufficiently low density, or when the layout of the objects meets a certain criterion (such as when there is no occlusion between the objects). The wearable system can automatically change the mode of user interaction to hand gestures on the user input device when the objects are located at high density, or when the layout of the objects no longer meets the criterion (such as when one object occludes another object).

[0150] The wearable system can consider, in addition to or instead of, the relative positions of objects in 3D space in addition to the relative positions of objects in 2D space. For example, the user interface 1514 may be a 3D user interface instead of a 2D user interface. As shown in FIG. 16, the weather application 1638 may be located in a depth plane farther from the user than the coffee mug application 1636. When the direction of the user's line of sight is at position 1620, the wearable system can detect two virtual objects (the virtual hanging clock 1634 and the music application 1654) that intersect the direction of the user's line of sight, even if the music application 1654 may appear farther from the user than the virtual hanging clock 1634. Based on this information, the wearable system may determine that the objects are close enough to each other. Accordingly, the wearable system may automatically switch the mode of user interaction to hand gestures or prompt the user with an option to switch to hand gestures.

[0151] (Other exemplary user interface features based on context information) In some implementations, the wearable system may reposition a high-density cluster of virtual objects to one or more fixed depth planes when the user switches to hand gestures. As described with reference to FIGS. 12, 13A-13B, the user can select virtual objects within a fixed depth plane or use hand gestures to switch the depth plane. This implementation can be particularly advantageous in reducing the cumbersome operations on the user input device that occur by navigating between virtual objects located in slightly different depth planes in 3D space.

[0152] A wearable system can change a focus indicator or cursor, either alone or in combination, when the mode of user interaction changes from one way to another. For example, the wearable system may change the color of the focus indicator when the user changes the mode of user interaction from a head pose to a hand gesture on a user input device (and vice versa). In another example, the wearable system may show the option of changing the appearance of the focus indicator from a crosshair shape to an arrow shape (shown in FIG. 16) and changing the input control from a head pose to a hand gesture on a user input device.

[0153] In some implementations, the changes in the focus indicator or cursor may be used, either alone or in combination, to indicate that options for changing the mode of user interaction are available. For example, while the user is interacting with a group of objects positioned at low density using a hand gesture on a user input device, the wearable system can change the appearance of the focus indicator from an arrow to a crosshair to indicate that the option of interacting using a head pose is available. The user can confirm the change from a hand gesture to a head pose, for example, by activating the user input device (such as tapping the user input device) or changing the body pose (such as nodding the head). As another example, the wearable system can provide vibration on a user input device as a focus indicator to indicate that an alternative mode of user interaction is available.

[0154] The embodiments are described with reference to the selection of one object, but the wearable system may be configured to identify multiple target objects and select multiple target objects. The AR system may be configured to recursively perform the selection event on a subgroup of target-interactable objects. For example, the AR system may identify some target objects that collide with the virtual cone in ray casting (described in FIG. 10). The target objects may include interactable objects. The AR system can zoom in on these target objects and enable the user to select within these target objects using the postures and / or hand gestures described herein. Further, the embodiments are described with reference to changes between head postures and hand gestures on the user input device, but similar techniques can also be used to switch between other modes of user interaction. For example, the wearable system may employ similar techniques to change the mode of user interaction between body postures, hand gestures, head postures, foot postures, eye postures, etc., either alone or in combination.

[0155] (Examples of Interaction Events) After the user selects an interactive object, the user can start an interaction event on the interactive object within their FOV. In some implementations, the virtual object may correspond to a physical object. As a result, when the user performs an interaction event on the virtual object, the virtual object may communicate with the physical object, thereby enabling the user to interact with the physical object via the virtual user interface. For example, the 3D coffee mug 1636 in FIG. 16 may communicate with a coffee machine within the user's environment. The water level shown in the 3D coffee mug 1636 may represent the progress of coffee generation. As an example, the water level may initially be invisible because the coffee machine is idle. The user can select the 3D coffee mug 1636 and start an interaction event, which causes the wearable system to send a command to the coffee machine in the user's kitchen to start coffee overflow. During the overflow process, the water level in the 3D coffee mug 1636 may gradually increase as the coffee machine generates more coffee. When the coffee machine finishes overflowing, the wearable system may indicate that the 3D coffee mug 1636 is full. In some implementations, the wearable system may also provide a focus indicator (such as sound or backlight) on the 3D coffee mug to indicate that the coffee overflow has ended.

[0156] As another example, the wearable system can present virtual content associated with physical objects in the user's environment during an interaction event. In FIG. 16, the user can select the weather application 1638 and start an interaction event on the weather application 1638. FIG. 17 illustrates an exemplary interaction event with the weather application 1638. In this embodiment, the user of the wearable system may be outside their home and can perceive physical objects such as distant storm clouds 1710 and tornadoes 1720. The user can also perceive other physical objects such as road 1730, vehicle 1732, and building 1734.

[0157] The wearable system can identify the user's location by analyzing an image of the user's environment. When the user starts an interaction event on the weather application 1638 (shown in FIG. 16), the wearable system can present an AR / MR scene 1700 superimposed on physical objects within the user's FOR. The wearable system can use a geolocation sensor (e.g., a global positioning system (GPS) sensor) to determine information about the user's location and the weather in the user's vicinity (e.g., the presence of storm clouds 1710 and tornadoes 1720 that the user is viewing). In some implementations, the wearable system can also use an outward-facing imaging system to acquire an image of the user's environment. The wearable system can use sensors such as the outward-facing imaging system 464, the inward-facing imaging system 462, or the IMU (described in FIG. 4), alone or in combination, to determine that the user is looking towards the storm 1710 and tornado 1720.

[0158] The wearable system communicates with a network (wired or wireless), accesses information about a storm, and can display information such as the presence and duration of storm warning 1750, the speed of a tornado 1754, weather forecast 1762 (e.g., temperature, precipitation probability as a function of time), the direction of the storm 1756 (e.g., the position of the storm as a function of time), the expected rainfall 1766, etc. to the user as virtual content. This information can be presented to the user via text or graphics such that at least a portion of the information is perceived at or near the location of storm cloud 1710 or tornado 1720 (e.g., the virtual content can appear to be superimposed on physical content). For example, as shown in FIG. 17, arrow 1752 indicates the direction of the storm (e.g., tornado), appears as if they are 3D, and is superimposed on or around tornado 1720 from the perspective of the user of the wearable system. The wind speed 1754 can be presented in the vicinity of the direction arrow 1752. The wearable system can also present other information about the storm to the wearer, such as the temperature forecast 1762, the location where rain is currently falling or expected to fall (e.g., indicated via the dashed line 1764 below storm cloud 1710), the expected precipitation amount (as indicated using reference number 1766), the direction 1752 and speed 1754 of the wind within the storm (e.g., at different altitudes of tornado 1720), etc.

[0159] The wearable system can present realistic 3D virtual content 1750, 1752, 1754, 1756, 1762, 1764, 1766 to the user of the wearable system using a light field display (shown in FIGS. 4 - 6) such that it appears at an appropriate distance from the user (e.g., on or near storm 1710 and tornado 1720) and is appropriately sized, shaped, or scaled (e.g., as schematically shown in FIG. 17).

[0160] A wearable system can also enable two or more users to interact with interactive objects. Both users may wear their individual wearable systems (such as their head-mounted devices). FIG. 18 illustrates an exemplary user experience of multiple users interacting with a 3D virtual object. In this embodiment, the user is wearing a wearable device 1852. The user can perceive virtual content 1800 through the wearable device 1852. In this embodiment, the virtual content can include astronomical objects 1810 (e.g., stars, or in other cases, galaxies, planets, nebulae, or solar systems). Information about the virtual content 1800 can be displayed to appear on or adjacent to the virtual content. For example, the orbit 1820 of a star or planet 1830, constellations, nearby stars, etc. can be displayed in the vicinity of the astronomical object 1810. The wearable system can present a user interface to the wearer that can access a virtual menu 1832 or virtual input feature 1840 where different actions can be selected (e.g., by gesture) using a posture or user input device. For example, as shown in FIG. 18, the virtual menu 1832 may enable the wearer to edit / delete or save the profile of the displayed virtual content 1800. The profile can enable the wearer (or another authorized user) to access the virtual content at different times or locations. Another virtual menu may enable the user to interact with the virtual content and modify the displayed virtual content. For example, as shown in FIG. 18, the user input element 1840 can enable the wearer to "add a planet" (e.g., by selecting the virtual button 1840 using a gesture such as "pressing" the wearer's finger at the position of the virtual button 1840). After selection, the wearer can access functionality (e.g., via a virtual menu, virtual drop-down box, etc.) and may be able to create a planet profile using information such as the name, diameter, temperature, or distance of the planet.After selection, additional virtual content (in this embodiment, additional planets) can be displayed to the wearer.

[0161] The wearable system can enable the user to share virtual content with others, for example, by the user passing through a world map of the user's environment via a network or communicating (or updating virtual content) the virtual content between wearable systems. For example, as shown in FIG. 18, another user 1850 wearing the wearable system 200 can view the virtual content shared and manipulated by the first user. Both users can enjoy the experience of interacting with each other and with the virtual content. Although astronomical object 1810 is used in this embodiment, the virtual content 1800 can be any type of content. For example, an encyclopedia can be accessed and content regarding one or more topics can be displayed and shared (along with virtual text, virtual images, sounds, etc.). The group of users sharing the virtual content do not need to be physically present in the same location to view the virtual content, and many users (e.g., 2, 3, 4, 5, 10, 100, or more) can view the shared virtual content substantially simultaneously.

[0162] The embodiments are described with reference to virtual solar systems and weather applications, but those embodiments are not limiting. The techniques described herein can also be applied to other applications that present interactive virtual content within an AR / MR / VR environment. For example, the wearable system can be programmed to include a clothing shopping application. While the user is in a department store, the application can access a database associated with the department store and identify clothing information or racks. The application can present the accessed information on virtual content superimposed on the physical clothing within the department store.

[0163] (Exemplary Process for Selecting Virtual Objects Using a Combination of Posture and User Input Device) FIG. 19 illustrates an exemplary process for selecting virtual objects using a combination of posture and hand gestures on a user input device. Process 1900 can be performed by a wearable system (e.g., an HMD) described herein (e.g., by one or both of processing modules 260, 270). The wearable system may include a user input device (see, e.g., user input device 466 in FIG. 4) and various imaging systems such as an outward-facing imaging system (see, e.g., outward-facing imaging system 464 in FIG. 4) and an inward-facing imaging system (see, e.g., inward-facing imaging system 462 in FIG. 4).

[0164] In block 1910, the wearable system can identify interactive objects within the user's FOR. The interactive objects may be stored, for example, in a remote data repository 280 (shown in FIG. 2) using a data structure such as an array. The interactive objects within the user's FOR can be a subset of all objects within the user's FOR. The wearable system can use the outward-facing imaging system 462 and other sensors (such as IMU and GPS) to determine the user's location and use this location information to determine interactive objects within the user's environment.

[0165] In block 1920, the wearable system can measure the user's posture using the sensors described herein. The wearable system can use the inward-facing imaging system 462 to identify the user's orientation.

[0166] In block 1930, the wearable system can identify interactive objects within the user's FOV based on the direction of the user's line of sight. The FOV can also use ray casting to determine interactive objects that intersect the direction of the user's line of sight. In some implementations, interactive objects within the user's FOV can be referred to as selectable objects. The wearable system can use the positions of the selectable objects to store the selectable objects in an array. For example, the wearable system can index selectable objects (and / or interactive objects) based on x-axis values (see x-y-z coordinates in FIG. 6). The wearable system can sort objects based on the x-axis values and present the selectable objects from left to right within the user's FOV in a 1D or 2D view.

[0167] In some implementations, the wearable system may not have a separate array just for selectable objects. In these implementations, the wearable system can identify and retrieve selectable objects from the array of interactive objects using, for example, the array indices of the selectable objects (determined based on the user's FOV).

[0168] However, in some embodiments, the wearable system may maintain two arrays, where one array is for interactive objects within the user's FOR while another array is for selectable objects within the user's FOV. For example, the array for interactive objects may be maintained in a remote data repository 280 (shown in FIG. 2), while the array for selectable objects may be maintained in the local data storage of the wearable system. This implementation may be advantageous because the local storage device and data processing capacity of the wearable system may be limited, while the remote data repository may have a larger data storage device and more powerful data processing capacity. By keeping only a subset of all interactive objects in the local storage of the wearable system, the hardware requirements of the wearable system can be reduced and the possibility of data overflow within the wearable system can be decreased.

[0169] If an interactive object does not exist within the user's FOV, the wearable system can return to block 1910. The wearable system can continuously monitor the user's posture, update the user's FOV, and continuously determine the list of interactive objects within the user's FOV.

[0170] In block 1940, the wearable system can determine whether the user desires to initiate a selection event on an object within the user's FOV. The wearable system can make such a determination based on various indications, such as, for example, prolonged fixation on an object, the user's head pose such as nodding, or input from a user input device, either alone or in combination. If the wearable system receives an indication that the user desires to select an object, the wearable system may enable the user to interact with selectable objects using the various hand gestures described herein. For example, the user can swipe along a track on the user input device to browse selectable objects or click the user input device to select a selectable object.

[0171] Initiation of the selection event may cause the wearable system to identify a target-interactable object within the user's FOV (at block 1940) and assign a focus indicator to the target-interactable object (at block 1950). The target-interactable object may be an object on the left or right side of the FOV. The visible focus indicator may also be placed on an object at the center of the FOV when the object at the center is the target-interactable object. The wearable system may also use 3D eye tracking to determine the direction of the user's eye pose and place the visible focus indicator on an object that intersects the direction of the user's eye pose.

[0172] In block 1960, the user can use the various gestures described herein to move the focus indicator to another object. The target-interactable object can accordingly be updated to another object. For example, the user can move the focus indicator from its current position to a neighboring object, as a result of which the target-interactable object is updated to the neighboring object.

[0173] In some embodiments, these gestures may cause the wearable system to move virtual objects along a path within the user's FOV (instead of moving the visible focus indicator itself). For example, referring to FIG. 12, when the user swipes left on the touchpad, it may have the effect of moving object 1230 completely into the FOV while shifting the position of object 1242 to the left.

[0174] FIG. 20 illustrates an exemplary process for interacting with virtual objects using a combination of posture and hand gestures on a user input device. Process 2000 can be performed by the wearable system described herein.

[0175] In block 2010, the wearable system can determine a group of interactive objects within the user's FOR. The group of interactive objects can be a subset of the objects within the user's environment. The interactive objects can be virtual objects and / or physical objects. In some embodiments, the AR system can add virtual elements to existing physical objects. For example, the AR system may add a virtual menu to a television in a room, and the virtual menu may provide the user with the option to turn on the television using the AR system.

[0176] As described with reference to FIG. 12, the AR system may store interactive objects and information associated with the interactive objects within various data structures. The location of the object may be used as an index for storing information associated with the object.

[0177] In block 2020, the wearable system can determine the user's posture. The user's posture may be the head, eyes, feet, or other body postures, either alone or in combination. The wearable system can use various sensors shown in FIG. 4, such as an imaging system 462 facing inward, an input received on the user input device 466, or an imaging system 464 facing outward, to determine the user's posture.

[0178] In block 2030, the wearable system can determine the user's FOV based on the user's posture. The FOV can comprise a portion of the FOR perceived by the user at a given time. Based on the user's FOV, in block 2040, the AR system can determine a subgroup of interactive objects within the user's FOV. This subgroup of interactive objects is sometimes also referred to as selectable objects. As the user's FOV changes, the selectable objects within the user's FOV also change.

[0179] In block 2050, the wearable system receives a selection of a target interactive object from the subgroup of interactive objects. The wearable system can first select a target interactive object based on various rules (such as the location of the target interactive object relative to the user's FOV) as described with reference to FIG. 12. The AR system can assign a focus indicator to the target interactive object. The visible focus indicator may move from one object to another as the user's FOV changes.

[0180] In some embodiments, the wearable system can identify a target interaction-capable object after the user activates a user input device. The user can activate the user input device using various hand gestures described with reference to FIGS. 12-13. These hand gestures can trigger the wearable system and assign a focus indicator to a target interaction-capable object within the user's FOV. In some implementations, when the AR system receives a selection of a target interaction-capable object from the user, the AR system may stop updating the group of selectable objects within the user's FOV while the user's FOV is changing. The user can still browse the interaction-capable objects or shift the visible focus indicator from one object to another within their FOV.

[0181] In block 2070, the user may decide to start a selection event on a target interaction-capable object. The selection event can be started using the postures and gestures described herein. In some embodiments, starting the selection event may trigger the wearable system and assign a visible focus indicator to the target interaction-capable object. The wearable system may stop updating the group of selectable objects within the user's FOV even if the user's FOV can change with changes in the user's posture. The user may use the gestures described herein to shift the visible focus indicator from one object to another within the user's FOV. For example, the user may swipe along a track (such as left and right) on a touchpad, which can move the visible focus indicator from one object to its nearest neighbor for the wearable system.

[0182] In some implementations, the user may initiate an interaction event during or after a selection event. The interaction event may also be part of the selection event. For example, as described with reference to FIG. 12, the interaction event can include resizing an interactive object, displaying a menu of the interactive object, browsing the menu, selecting an item on the menu, searching for an item, playing a game, watching a video, conducting a teleconference, etc. The user can participate in the interaction event using various postures and gestures described herein. The user may also confirm the selection of the target interactive object using the postures and gestures discussed herein.

[0183] (Exemplary Process of Interaction with an Object Based on Context Information) FIG. 21 illustrates an exemplary process for switching input control from a head posture to a hand gesture based on context information. Process 2100 can be performed by the wearable system described herein (e.g., by one or both of processing modules 260, 270).

[0184] The wearable system can display a cursor indicating the user's current position. The cursor can be a crosshair corresponding to the user's head position. As the user moves, the cursor may be moved to a target-interactable object. The user can select a target-interactable object using a posture, a hand gesture on a user input device, either alone or in combination. In process 2100, the user may first interact with an object using a head posture. In block 2110, the wearable system can determine whether the user has selected a target-interactable object. The target-interactable object may be a 2D plane virtual user interface. If the user has not selected a target-interactable object, the process ends at block 2190. In some embodiments, the wearable system can continuously determine the user's current position as the user moves. The wearable system can also identify other target-interactable objects within the user's FOV based on the user's head posture.

[0185] As shown in block 2120, when the wearable system receives a selection of a target-interactable object, such as a 2D plane virtual user interface, the wearable system may assign a focus indicator to the target-interactable object. For example, the wearable system can display an afterglow around the target-interactable object and bring the target-interactable object closer to the user so that it appears to float within the 3D space. The wearable system can also set the normal and depth of the cursor (corresponding to the head position) to be the same as the normal and depth of the target-interactable object. As a result, the user may continue to interact with virtual objects within the target-interactable object using a head posture.

[0186] In block 2130, the wearable system can identify the context associated with the user's interaction. For example, the wearable system can determine the layout of virtual objects (or physical objects) within the user's FOV.

[0187] In block 2140, when the wearable system determines that the layout meets a certain pattern (such as one virtual object being blocked by another virtual object) or exceeds a certain density threshold, the wearable system may provide the user with an option to switch the input control mode. As an example, the wearable system may provide the user with an option to switch the interaction mode from the head pose to the user input device.

[0188] As shown in block 2142, if the user selects not to switch, the user can still use the head pose to target and select interactable objects. The process ends at block 2190.

[0189] If the user selects to switch the input control to hand gestures, in block 2150, the user can operate the user input device to interact with the virtual object. In block 2160, the wearable system can receive the user's selection of a target virtual object, such as a UI element on a 2D plane virtual user interface. For example, referring to FIG. 16, the user can select the weather application 1638, the coffee generation application 1636, etc. If the user does not select a target virtual object in block 2160, the user may continue to operate the user input device as shown in block 6150.

[0190] The wearable system can initiate on a selected virtual object for a selected event or interaction event. For example, the wearable system can provide a focus indicator on the selected virtual object. The wearable system can also present a VR / AR / MR scene associated with the selected virtual object. For example, the wearable system can present scenes 1700 (shown in FIG. 17) and 1800 (shown in FIG. 18) that the user can interact with.

[0191] In block 2180, the wearable system can determine whether the user has completed a selection event or interaction event. If the wearable system determines that the user has ended the interaction with the target virtual object, process 2100 ends in block 2190. In some embodiments, in block 2190, the wearable system can switch the mode of user interface control back from hand gestures to head pose.

[0192] FIG. 22 illustrates an exemplary process for switching the mode of user interaction based on context information. Process 2200 can be performed by the wearable system described herein (e.g., by one or both of processing modules 260, 270).

[0193] In block 2210, the wearable system can determine the current input mode that the user uses to interact with the interactive object. The current input mode may be a pose or a hand gesture on the user input device.

[0194] In block 2220, the wearable system can determine context information such as the layout of the objects in the user's FOV, the density of the objects in the user's FOV, the characteristics of the objects in the user's FOV (size, position, object type, etc.).

[0195] Based on the context information, the wearable system can present options for changing the current user input mode at block 2230. For example, while the user is using the head pose to interact with an object, if the wearable system identifies a high-density group of objects in the direction of the user's line of sight, the wearable system can present an option to change the user input mode to a user input device. As another example, if the wearable system determines that the objects are located at low density, the wearable system can provide an option to change the current user input mode from the user input device to the head pose.

[0196] As another example, the wearable system may allow the user to use the pose to interact with a user interface plane (which may include other virtual objects), while using the user input device to interact with user interface elements (such as an application within the user interface plane). As a result, if the wearable system detects that the user has selected the user interface plane, the wearable system may change the user input mode from the head pose to the user input device. On the other hand, when the user finishes interacting with the user interface element, the wearable system may change the user input mode from the user input device to the head pose.

[0197] In some implementations, the appearance of the focus indicator (including the cursor) may change based on different user input modes. For example, a wearable system may use a crosshair to indicate that the user is interacting with an object using a head pose, while using an arrow to indicate that the user is interacting with an object using a user input device. In some implementations, the wearable system can change the appearance of the focus indicator and indicate that an option for switching the user input mode is available. For example, the wearable system may first display a crosshair when the user interacts with a head pose. When the wearable system detects a high-density group of objects, the wearable system may display an arrow (instead of the crosshair) and indicate that the user can switch the user input mode to the user input device. However, when the user moves away from the high-density group of objects, the wearable system may change the focus indicator back from the arrow to the crosshair.

[0198] In block 2240, the user can select an option for a new user input mode. For example, the user can activate the user input device and confirm that they desire to change the user input mode from a head pose to a hand gesture on the user input device. The wearable system can then update the current user input mode to the newly selected mode accordingly. The wearable system may also update the focus indicator to be associated with the newly selected mode.

[0199] FIG. 23 illustrates an exemplary process of interacting with an interactive object comprising a group of virtual objects. Process 2300 can be performed by the wearable system described herein (e.g., by one or both of processing modules 260, 270). The wearable system may include various sensors such as user input device 466 (shown in FIG. 4), a light field display (described with reference to FIGS. 2 or 4 - 6), an outward-facing imaging system 464 (shown in FIG. 4), and an inward-facing imaging system 462 (shown in FIG. 4).

[0200] In block 2310, the wearable system determines the user's posture. The posture may be the head, eyes, feet, or other body postures, etc. The wearable system can use various sensors such as, for example, the inward-facing imaging system 462, the outward-facing imaging system 464 (e.g., the FOV camera described with reference to FIG. 10), an IMU, etc., to determine the user's posture.

[0201] The wearable system can determine a group of interactive objects within the user's FOR. For example, the wearable system can access a map of the user's environment that includes information about the objects in the user's environment. In block 6230, the wearable system can determine a target interactive object from the group of interactive objects. The target interactive object may be determined based on the user's posture. For example, the target interactive object may be an object that intersects the direction of the user's line of sight. As the user's posture changes, the target interactive object may also change.

[0202] In block 2330, the wearable system can receive a selection of a target-interactable object. The user can select a target-interactable object by operating the user input device alone or in combination, or by changing the posture. As shown in block 2340, after receiving the selection of the target-interactable object (as shown in block 6250), the wearable system can assign a focus indicator to the target-interactable object.

[0203] In some implementations, one or more virtual objects may further be displayed within the target-interactable object. For example, the virtual user interface may include user interface elements such as a weather application, a video streaming application, etc. In block 2350, the wearable system can determine a group of virtual objects associated with the selected target-interactable object. The user can interact with the group of virtual objects using head postures and hand gestures. For example, the user can select a virtual object within the group of virtual objects using a posture or a hand gesture. The user can also initiate interaction events such as playing a video game, viewing a VR / AR / MR scene, or other user interface interactions with the selected virtual object.

[0204] In some embodiments, the wearable system may change the mode of interaction from the pose to the user input device when the user selects a target-interactable object. For example, the user may first use the head pose to target and select an interactable object within the environment. Once the user selects the target-interactable object, the wearable system can change the input mode from the head pose to the user input device so that the user can interact with virtual objects within the target-interactable object using the user input device. Once the user finishes interacting with the virtual object, the wearable system may change the input mode back to the head pose so that the user can continue to target and interact with other interactable objects within the environment. In one implementation, the wearable system can provide an option for the user to switch the mode of input control. For example, if the user decides not to switch from the head pose to a hand gesture after selecting a target-interactable object, the user can continue to interact with the virtual objects within the target-interactable object using the head pose.

[0205] (Additional User Interface Experience) (Additional Examples of AR and MR Visual Experiences) As described above, a wearable system (such as a head-mounted display) can be configured to present 3D virtual objects superimposed on the physical world. For example, a user of a wearable device can be in a school gymnasium and not only perceive the local physical environment and physical objects (such as the gymnasium and the students sitting or standing in the gymnasium), but also perceive virtual objects superimposed on the physical world (such as the school gymnasium). The virtual object may include a jumping whale surrounded by water splashes. For example, the user can perceive the experience that the whale appears from the floor of the school gymnasium, jumps up across part of the gymnasium, lands in a large water splash on the floor, and then disappears. In this embodiment, the wearable system determines the dimensions of the area within the external world that the wearer is viewing (such as the size of the gymnasium in this embodiment) so that an image of a jumping whale is displayed as being perceived by the wearer as originating from a sub-region of the area (such as from the floor of the gymnasium), and can use a light field display as described in FIGS. 4-6 so that the jumping whale and water splashes appear realistic and alive to the user. In some implementations, the wearable system can present to the user (such as via the speaker 240 shown in FIG. 2) the sound of a jumping whale associated with the image presented to the user. The AR system can display additional virtual content such as text or graphics (in addition to, or as an alternative) on the scene viewed by the user. For example, the AR system can display information about the whale (such as the type of whale, age, habits, etc.) to the wearer before, during, or after the virtual whale appears to jump from the floor of the gymnasium.

[0206] As another example, the user of the wearable system may be in a retail market. The user can view virtual content including images of actual physical people in the environment and astronauts walking around the market. The virtual content may be superimposed within the FOV of the display of the wearable system.

[0207] A wearable system can modify an image of the physical world and provide a MR experience to a user. For example, a user may see a flock of physical birds flying in a V-formation. An imaging system facing outwards (e.g., as shown in FIG. 4) can capture this physical content, and the wearable system can process it and identify the flock of birds flying in a V-formation. The wearable system can add virtual objects (e.g., a dragon in this embodiment) flying within or near the formation to the flock, or replace one (or more than one) of the birds with it. As another example, the wearable system can add virtual objects (e.g., a virtual whale) floating on or flying over a beach to the user's view of the physical shore. The ability of a light field display to present a real-world image as appearing at different distances enables the wearable display system to present an image of the whale as being near or away from the wearer. In some implementations, the wearable system can use shadow mapping techniques so that the virtual content appears as having virtual shadows, which can also make the virtual content displayed by the wearable system more realistic.

[0208] In some implementations, a user of a wearable system can perceive a sequence of virtual images within an AR / MR environment. For example, assume the user is looking at their cupped hand. Virtual objects such as a small elephant may be displayed by the wearable system's display such that the user perceives the virtual object to be present within the user's cupped hand. The wearable system can image an area of the environment, such as an area that includes the wearer's hand (and any background behind the hand), using an outward-facing imaging system. The wearable system can determine the distance to the wearer's hand such that virtual content (e.g., the elephant) can be scaled and appear at an appropriate size and distance within a particular sub-region (e.g., the hand) of the overall region where the content is being viewed. The wearable system can make the scene appear as if the wearer is holding the elephant within the wearer's hand. The position of the elephant may change for each image such that the elephant can appear closer to the user in the temporal sequence compared to the first half of the temporal sequence. The image of the jumping elephant can be accompanied by sound (e.g., voice, music).

[0209] (Additional Examples of Interaction with Virtual Objects) As an example of user interaction with a virtual user interface, a user of a wearable system can perceive and interact with virtual objects in a physical room where people are dancing. In this example, the user may be a disc jockey (DJ), and the wearable system can display a virtual UI that can be operated by the DJ's hand movements (e.g., gestures). The virtual UI can include virtual content that enables the DJ to interact with the UI. In this example, the virtual UI can be configured to be a DJ audio control system that can control the sound played to the dancing people. The UI can include user input features such as dials (e.g., jog shuttle dials), switches, sliders, buttons, or turntables that can be adjusted by the DJ via gestures. The virtual UI can include output features such as sound level graphics or equalizers. The output features can respond in real time as the sound level or audio mix is changed by the DJ. The imaging system facing outward of the wearable system can image the DJ's hands and arms and determine the DJ's gestures (e.g., hand or finger movements). In response to the determined gestures, the wearable system can adjust the audio, for example, by increasing or decreasing the volume, fading or panning the music, mixing the music, etc.

[0210] As another example, a user of the wearable system can view the operating room together with the doctor performing the surgery on the patient. The wearable system can present virtual content to the user that displays the anatomical structure or organs (such as the heart) of the patient undergoing the surgery. The orientation and position of the virtual organ can be adjusted via gestures (e.g., by the wearer reaching the virtual image of the heart and grasping or moving it) or via a user input device. The virtual organ can represent an idealization of the heart (e.g., a textbook image) or an actual image of the patient's heart (e.g., taken during the surgery or pre-mapped prior to the surgery). The light field display capabilities of the wearable system (described in FIGS. 4-6) enable the wearer to view a 3D image of the organ. In some implementations, the user does not need to be physically present in the environment (e.g., the operating room) to interact with the objects (virtual and / or physical objects) in the environment. The user can interact with the doctor's avatar or the appearance of the operating room, for example, by communicating with the doctor (e.g., via a speaker) or interacting with the virtual image of the virtual organ.

[0211] Users can also use the wearable system to view and interact with educational virtual content. In this embodiment, the educational virtual content can include an avatar (e.g., a creation designed to be attractive to students and not intimidating) that holds a pointer and indicates graphics (e.g., numbers) displayed to the wearer as part of an educational lesson. An educational system that communicates with the wearable system can generate and distribute educational virtual content for presentation to the wearer as part of an educational lesson. The virtual content can include text, images, videos, graphics, and sounds. For example, the avatar can explain a math lesson to the student (e.g., 4×5=?). In some cases, the wearable system includes a microphone that can receive sounds in the surrounding environment, such as the voice of the student. The student can ask questions, and the wearable system (or the educational system) can use speech recognition technology to convert the questions into an electronic format, and the educational system can reply with an answer to the wearable system. For example, the avatar can respond to the student's question by indicating different parts of the virtual content that answer the question (e.g., using a wand), explaining the answer, etc.

[0212] In another AR experience, a user of a wearable system (such as an HMD) can view physical objects (e.g., other people and houses) and virtual entertainment displays. The virtual entertainment display shows the presentation of a sports event (in this example, a basketball game). The virtual entertainment display can present information to the user of the wearable system about the game being watched or other games (e.g., scores, live broadcasts, game replays, player statistics, rankings, etc.). The virtual entertainment display can appear as 3D, allowing the user to move around the display and view different sides of the virtual entertainment display (where different content, different games, different sports, or even different entertainment genres (e.g., movies) can be displayed on different sides of the display). The sports event may be presented to the user in real time as the sports event occurs. This provides the user with the ability to interact with the sports event even when the user is not physically present at the sports event. A particular user may control the virtual entertainment display by using gestures or a user input device. In some cases, a single user has control of the virtual entertainment display while other users can watch the actions on the display but cannot modify the displayed content. In other implementations, the virtual entertainment display can present content created or updated as a result of the actions of multiple users.

[0213] (Additional exemplary interactions among multiple users) When a group of people each wear a wearable device, the user can interact with another user within the AR / VR / MR environment. For example, people within the group can enjoy virtual content (which can include images, videos, sounds, texts, etc.), interact with it, share it, or operate it (e.g., via gestures) through the wearable devices they are wearing.

[0214] Such user interactions may occur within a virtual game. As players interact with each other within the game, the wearable device can present virtual content and sounds to the user of the wearable device. In this embodiment, the user can perceive a room with physical objects (e.g., a bed, a table, a window, and another player 404 of the game). The user can also perceive a fire-breathing dragon flying around another player. One or both of the players may control the position, movement, and actions (e.g., whether to breathe fire) of the dragon by gestures (e.g., hand or arm gestures), totems (e.g., a wand), or a controller or interface (physical or virtual). In some cases, the other player does not physically exist within the room and is presented to the user of the wearable device as if physically present (e.g., via telepresence). For example, the wearable device can present an avatar (in the form of a virtual child, etc.) of another player when playing the game. The child avatar (and the dragon) can be generated by the wearable system or another game system and communicated to the wearable system for display to the wearer. The child avatar may be a representation of the actual appearance of the other player or may be selected as a form such that the other player is perceived by the user of the wearable. The other player can be a human player or a machine player. In other embodiments, more than one player can appear within the wearer's environment (e.g., inside a room, outside the room, looking through a window, etc.). The ability of a light field display (e.g., the light field display described with reference to FIGS. 4-6, etc.) to present images at different distances (e.g., different depth planes) from the wearer can significantly improve the realism and playability of the game. In addition to the dragon, the wearable device can optionally display other information (e.g., text or graphics) to the user (e.g., game play statistics or status).

[0215] As described with reference to FIG. 7, the wearable device can pass a map of a physical or virtual environment and objects within the environment to another wearable device. For example, the wearable device can pass a map of the user's room and virtual objects within the room to another user's wearable device. Accordingly, the user can interact with the virtual objects as if they were in the same environment.

[0216] (Additional Embodiments) In a first aspect, a method for viewing virtual content, the method comprising: accessing region data related to a region within the user's field of view; analyzing the region data to identify a sub-region of the region where the virtual content is to be displayed; accessing or generating the virtual content, at least in part based on the region data and the sub-region data; and displaying the virtual content such that when viewed by the user, the virtual content appears to be located within or on the sub-region of the region.

[0217] In a second aspect, the identified sub-region comprises a physical object within the region, the method according to aspect 1.

[0218] In a third aspect, the region data is obtained from an analysis of an image of the region within the user's field of view, the method according to aspect 1 or aspect 2.

[0219] In a fourth aspect, the virtual content is modified, at least in part based on newly received region data or sub-region data, the method according to any one of aspects 1 to 3.

[0220] In a fifth aspect, the method according to any one of aspects 1 to 4 further comprises receiving user input; accessing or generating additional virtual content, at least in part based on the user input; and displaying the additional virtual content.

[0221] In the sixth aspect, the sub-region comprises a part of the region in the vicinity of the user, and the method according to any one of aspects 1 to 5.

[0222] In the seventh aspect, a method for interacting with virtual content, comprising the steps of accessing or generating virtual content, displaying the virtual content within the user's field of view, displaying a virtual user interface associated with the virtual content, receiving user input, associating the user input with a function associated with the user input characteristics of the virtual user interface, and performing the function.

[0223] In the eighth aspect, the method according to aspect 7, further comprising the step of operating or modifying the virtual content, at least in part, based on the received user input.

[0224] In the ninth aspect, the user input is a gesture, and the method according to any one of aspects 7 to 8.

[0225] In the tenth aspect, the user input is the movement of the user's eyes, and the method according to any one of aspects 7 to 9.

[0226] In the eleventh aspect, the virtual content comprises educational content or entertainment content, and the method according to any one of aspects 7 to 10.

[0227] In the twelfth aspect, the virtual content or the virtual user interface appears to be three-dimensional when perceived by the user, and the method according to any one of aspects 7 to 11.

[0228] In the thirteenth aspect, the user input characteristics of the virtual user interface comprise a dial, a switch, a slider, or a button, and the method according to any one of aspects 7 to 12.

[0229] In a 14th aspect, the virtual user interface comprises an output feature configured to display information related to the functionality of the virtual user interface, the method according to any one of aspects 7 to 13.

[0230] In a 15th aspect, the output feature is updated in real time, the method according to aspect 14.

[0231] In a 16th aspect, a method of displaying information within an augmented reality environment, the method comprising: determining a location of a user; identifying physical objects within the user's field of view; accessing, or generating, at least in part based on information related to the identified physical objects, the location or the identified physical objects; generating virtual content based on the information; and displaying the virtual content such that the information appears associated with the physical objects to the user.

[0232] In a 17th aspect, the virtual content comprises text, images, graphics, or video, the method according to aspect 16.

[0233] In an 18th aspect, the method according to aspect 16 or aspect 17, further comprising playing audio associated with the displayed virtual content.

[0234] In a 19th aspect, the method according to any one of aspects 16 to 18, further comprising displaying additional virtual content, at least in part based on user input.

[0235] In a 20th aspect, a hardware processor programmed with executable instructions such that, when the instructions are executed, the hardware processor performs the method according to any one of aspects 1 to 19.

[0236] In a 21st aspect, a wearable display system includes the hardware processor described in aspect 20 and a display configured to provide an image to a user's eye of the wearable display system.

[0237] In a 22nd aspect, the display of the wearable display system described in aspect 21 includes a light field display.

[0238] In a 23rd aspect, the display of the wearable display system described in aspect 21 or aspect 22 is configured to display an image on a plurality of depth planes.

[0239] In a 24th aspect, the wearable display system further includes an image capture device configured to capture an image of a region including a physical object, and is configured to display virtual content perceived by a user as associated with one or more of the physical objects. The wearable display system is the wearable display system according to any one of aspects 21 to 23.

[0240] In a 25th aspect, a method for selecting a virtual object located in a three-dimensional (3D) space, under the control of an augmented reality (AR) system comprising computer hardware, the AR system being configured to enable user interaction with interactive objects within the user's field of regard (FOR), the FOR comprising a part of the user's surrounding environment perceptible by the user via the AR system, under the control of the AR system, determining a group of interactive objects within the user's FOR, determining the user's pose, and at least partially based on the user's pose, determining the user's field of view (FOV), the FOV comprising a part of the FOR perceived by the user at a given time, updating a subgroup of interactive objects located within the user's FOV based on a change in the user's pose or FOV, receiving a selection of a target interactive object from the subgroup of interactive objects, and initiating a selection event on the target interactive object.

[0241] In a 26th aspect, the method according to aspect 25, further comprising storing the group of interactive objects in a data structure.

[0242] In a 27th aspect, each interactive object within the group of interactive objects is represented in the data structure at least partially based on the location of the interactive object within the user's FOV, the method according to aspect 26.

[0243] In a 28th aspect, the location comprises the distance from the edge of the user's FOV, the method according to aspect 27.

[0244] In the 29th aspect, the step of receiving the selection of the target-interactable object from a subgroup of interactable objects includes the step of receiving a first input from a user device and, in response to the first input, identifying the target-interactable object from the subgroup of interactable objects, as described in aspect 25.

[0245] In the 30th aspect, the method described in aspect 25 further includes the step of receiving a second input from a user device and, in response to the second input, starting an interaction event on the target-interactable object.

[0246] In the 31st aspect, the target-interactable object is an interactable object within a subgroup of interactable objects that is closest to the central point of the user's FOV, as described in aspect 30.

[0247] In the 32nd aspect, the target-interactable object is the leftmost or rightmost interactable object within a subgroup of interactable objects within the user's FOV, as described in aspect 30.

[0248] In the 33rd aspect, the step of starting an interaction event includes one or more of resizing the target-interactable object, displaying a menu of the target-interactable object, browsing the menu of the target-interactable object, selecting a first item on the menu, searching for a second item in a database, playing a video game associated with the target-interactable object, viewing a video, or conducting a teleconference, as described in any one of aspects 30 - 32.

[0249] In the 34th aspect, the step of receiving the selection of the target-interactable object is performed by determining the path of the user's line of sight based on the user's posture and selecting an object that intersects the path of the user's line of sight as the target-interactable object, as described in aspect 25.

[0250] In a 35th aspect, the method according to any one of aspects 25 - 34, further comprising the step of assigning a visible focus indicator to a target-interactable object.

[0251] In a 36th aspect, the visible focus indicator comprises a highlight, a backlight, a color change, a size change, or a change in the perceived depth of the target-interactable object, the method according to aspect 35.

[0252] In a 37th aspect, the step of initiating a selection event comprises one or more of the steps of changing the target-interactable object to a different interactable object that becomes the target-interactable object, opening a menu associated with the target-interactable object, or receiving confirmation from the user to select the target-interactable object, the method according to aspect 25.

[0253] In a 38th aspect, the group of interactable objects comprises virtual objects, the method according to any one of aspects 25 - 37.

[0254] In a 39th aspect, the user's posture comprises an eye posture, the method according to any one of aspects 25 - 38.

[0255] In a 40th aspect, the user's posture comprises a head posture, the method according to any one of aspects 25 - 39.

[0256] In a 41st aspect, an augmented reality (AR) system for selecting a virtual object located in a three-dimensional (3D) space, comprising: a display system; a network interface; and a computer processor that communicates with the network interface and the display system, determines a group of interactive objects within the user's field of regard (FOR), determines the user's pose, at least partially based on the user's pose, determines the user's field of view (FOV), where the FOV comprises a portion of the FOR perceived by the user at a given time, updates a subgroup of interactive objects located within the user's FOV based on a change in the user's pose or FOV, receives a selection of a target interactive object from the subgroup of interactive objects, and is configured to initiate a selection event on the target interactive object.

[0257] In a 42nd aspect, the system according to aspect 41, wherein the computer processor is further configured to store the group of interactive objects in a data structure.

[0258] In a 43rd aspect, one or more interactive objects within the group of interactive objects are represented in the data structure at least partially based on the location of the interactive objects within the user's FOV, the system according to aspect 42.

[0259] In a 44th aspect, the location comprises a distance from the edge of the user's FOV, the system according to aspect 43.

[0260] In a 45th aspect, the computer processor configured to receive a selection of a target interactive object from the subgroup of interactive objects receives a first input from a user device and, in response to the first input, identifies the target interactive object from the subgroup of interactive objects, the system according to any one of aspects 41 - 44.

[0261] In the 46th aspect, the target interactable object is an interactable object within a subgroup of interactable objects that is closest to the central point of the user's FOV, for the system according to any one of aspects 41 - 45.

[0262] In the 47th aspect, the target interactable object is the leftmost or rightmost interactable object within a subgroup of interactable objects within the user's FOV, for the system according to any one of aspects 41 - 45.

[0263] In the 48th aspect, a computer processor configured to receive a selection of a target interactable object from a subgroup of interactable objects determines the path of the user's line of sight based on the user's posture and selects, as the target interactable object, an object that intersects the path of the user's line of sight, for the system according to aspects 41 - 47.

[0264] In the 49th aspect, the computer processor is further configured to assign a visible focus indicator to the target interactable object, for the system according to any one of aspects 41 - 48.

[0265] In the 50th aspect, the visible focus indicator comprises a highlight, a backlight, a color change, a size change, or a change in the perceived depth of the target interactable object, for the system according to aspect 49.

[0266] In the 51st aspect, a computer processor configured to initiate a selection event includes changing the target interactable object to a different interactable object that becomes the target interactable object, opening a menu associated with the target interactable object, or receiving confirmation from the user to select the target interactable object, among one or more of these, for the system according to aspects 41 - 50.

[0267] In the 52nd aspect, the group of interactive objects is the method according to any one of aspects 41 - 51, comprising virtual objects.

[0268] In the 53rd aspect, the user's posture is the method according to any one of aspects 41 - 52, comprising an eye posture.

[0269] In the 54th aspect, the user's posture is the method according to any one of aspects 51 - 53, comprising a head posture.

[0270] In the 55th aspect, a system for interacting with virtual objects in a three - dimensional (3D) space, comprising an augmented reality display for displaying a plurality of interactive objects, a user input device, one or more sensors configured to determine the user's posture, and one or more processors, wherein the one or more processors are configured to switch between a first user input mode and a second user input mode to select a virtual object, the first user input mode is at least partially based on the user's posture, the second user input mode is at least partially based on a signal from the user input device, further, while the system is operating in the first user input mode, the user's posture is monitored and at least partially based on the monitored posture, a first focus indicator associated with the first user input mode is displayed in a direction related to the user's posture, an indication for switching to the second user input mode is received, the indication further indicates the selection of a target interactive object, while the system is operating in the second user input mode, the user input from the user input device is monitored and at least partially based on the monitored user input, one or more processors configured to display a second focus indicator associated with the second user input mode.

[0271] In the 56th aspect, the first focus indicator comprises a cursor in the first shape, and the second focus indicator comprises a cursor in the second shape, for the system described in aspect 55.

[0272] In the 57th aspect, the first shape comprises a crosshair, and the second shape comprises an arrow, for the system described in aspect 56.

[0273] In the 58th aspect, one or more processors are further configured to determine context information associated with the location of the first focus indicator or the second focus indicator and to display options for switching to different user input modes, at least in part, based on the context information, for the system described in aspect 55.

[0274] In the 59th aspect, the target interaction-capable object comprises one or more virtual objects, for the system described in any one of aspects 55-58.

[0275] In the 60th aspect, the user's posture comprises at least one of a head posture, an eye posture, or a body posture, for the system described in any one of aspects 55-59.

[0276] In the 61st aspect, the user input device is a thumb pad, a track pad, a D-pad, or a touch screen, for the system described in any one of aspects 55-60.

[0277] In a 62nd aspect, a method for selecting an interactive object located in a three-dimensional (3D) space, under the control of an augmented reality (AR) system comprising computer hardware, the AR system being configured to enable user interaction with interactive objects within the user's field of regard (FOR), the FOR comprising a part of the user's surrounding environment perceptible by the user via the AR system, the AR system further comprising a user input device, under the control of the AR system, determining a group of interactive objects associated with virtual objects within the user's FOR; determining the user's pose; at least partially based on the user's pose, determining a target interactive object from the group of interactive objects; associating a focus indicator with the selected target interactive object; and at least partially based on an input of the user input device, initiating a selection event on the selected target surface.

[0278] In a 63rd aspect, the method according to aspect 62, wherein the step of determining the target interactive object is performed by ray casting.

[0279] In a 64th aspect, the method according to aspect 63, wherein ray casting includes casting a beam of light rays.

[0280] In a 65th aspect, the method according to aspect 63, wherein ray casting includes casting a light ray with a substantial lateral width.

[0281] In a 66th aspect, the method according to aspect 62, wherein the focus indicator is visible to the user.

[0282] In a 67th aspect, the method according to aspect 66, further comprising displaying the focus indicator to the user.

[0283] In the 68th aspect, there is a step of determining a user's field of view (FOV) at least partially based on the user's posture, where the FOV comprises a part of the FOR perceived by the user at a given time, a step of accessing context information of one or more interactive objects within the field of view (FOV), and a step of providing one or more options for a user input mode at least partially based on the context information, further including the method described in aspect 62.

[0284] In the 69th aspect, there is a step of receiving a selection of an option for a user input mode, a step of determining the current user input mode, and a step of updating the current user input mode to the selected option in response to a determination that the current user input mode is different from the selected option, further including the method described in aspect 68.

[0285] In the 70th aspect, there is a step of further including updating a focus indicator to indicate that the current user input mode has been changed, further including the method described in aspect 69.

[0286] In the 71st aspect, the user input mode includes one or more of the user's posture or a user input device, further including the method described in aspect 70.

[0287] In the 72nd aspect, the group of interactive objects comprises one or more virtual user interface planes, further including the method described in any one of aspects 62 - 71.

[0288] In the 73rd aspect, the user's posture comprises at least one of an eye posture, a head posture, or a body posture, further including the method described in any one of aspects 62 - 72.

[0289] In a 74th aspect, a method for interacting with virtual objects in a three-dimensional (3D) environment, under the control of an augmented reality (AR) system comprising computer hardware, the AR system being configured to enable user interaction with interactive objects within the user's field of regard (FOR), the FOR comprising a portion of the user's surrounding environment perceptible by the user via the AR system, the AR system further comprising a user input device, the method comprising, under the control of the AR system: determining the user's pose; displaying a first focus indicator associated with a target interactive object in a direction related to the user's pose, the target interactive object comprising a plurality of virtual objects; receiving a selection of the target interactive object; displaying the plurality of virtual objects; displaying a second focus indicator associated with a target virtual object; and updating the second focus indicator at least in part based on input from the user input device.

[0290] In a 75th aspect, the method according to aspect 74, wherein the first focus indicator comprises a cursor in a first shape and the second focus indicator comprises a cursor in a second shape.

[0291] In a 76th aspect, the method according to aspect 75, wherein the first shape comprises a crosshair and the second shape comprises an arrow.

[0292] In a 77th aspect, the method according to any one of aspects 74-76, wherein the user's pose comprises at least one of a head pose, an eye pose, or a body pose.

[0293] In a 78th aspect, the method according to any one of aspects 74-77, wherein the user input device is a thumb pad, a track pad, a D-pad, or a touch screen.

[0294] In a 79th aspect, a method for interacting with a virtual object in a three-dimensional (3D) space, under the control of an augmented reality (AR) system comprising computer hardware, the AR system being configured to enable user interaction with an interactive object within a user's field of regard (FOR), the FOR comprising a part of the user's surrounding environment perceptible by the user via the AR system, the AR system further comprising a user input device, under the control of the AR system, while the AR system is in a first user input mode that operates at least in part based on the user's pose, monitoring the user's pose; displaying a first focus indicator associated with the first user input mode in a direction related to the user's pose at least in part based on the monitored pose; receiving an indication for switching to a second user input mode, the indication further indicating selection of a target interactive object; while the AR system is in a second user input mode that operates at least in part based on a signal from the user input device, monitoring user input from the user input device; and displaying a second focus indicator associated with the second user input mode at least in part based on the monitored user input.

[0295] In an 80th aspect, the method according to aspect 79, wherein the first focus indicator comprises a cursor in a first shape and the second focus indicator comprises a cursor in a second shape.

[0296] In an 81st aspect, the method according to aspect 80, wherein the first shape comprises a crosshair and the second shape comprises an arrow.

[0297] In the 82nd aspect, the method according to aspect 79 further includes the step of determining context information associated with the location of the first focus indicator or the second focus indicator, and the step of displaying an option for switching to a different user input mode based at least in part on the context information.

[0298] In the 83rd aspect, the method according to any one of aspects 79 - 82, wherein the target interactable object comprises one or more virtual objects.

[0299] In the 84th aspect, the method according to any one of aspects 79 - 83, wherein the user's posture comprises at least one of a head posture, an eye posture, or a body posture.

[0300] In the 85th aspect, the method according to any one of aspects 79 - 84, wherein the user input device is a thumb pad, a track pad, a D - pad, or a touch screen.

[0301] In the 86th aspect, an augmented reality device comprising computer hardware programmed to perform the method according to any one of aspects 62 - 85.

[0302] On the 87th aspect, a system for changing a user input mode for a wearable device, comprising a display system of the wearable device configured to present a three-dimensional (3D) view to the user, the 3D view comprising interactive objects, a user input device configured to receive user input, a sensor configured to acquire data associated with the user's pose, and a hardware processor communicating with the user input device, the hardware processor determining whether the current user input mode for interacting with the interactive objects is a first user input mode or a second user input mode, the first user input mode being at least partially based on the user's pose, the second user input mode being at least partially based on user input from the user input device, in response to determining that the current user input mode is the first user input mode, using the sensor to monitor the user's pose and presenting, via the display system, a focus indicator in a first shape associated with the first user input mode in a direction related to the user's pose based at least partially on the monitored pose, receiving a first indication, switching to the second user input mode, switching the current user input mode to the second user input mode in response to the first indication, in response to determining that the current user input mode is the second user input mode, monitoring user input from the user input device and presenting, via the display system, a focus indicator in a second shape associated with the second user input mode based at least partially on the monitored input, receiving a second indication, switching to the first user input mode, and switching the current user input mode to the first user input mode in response to the second indication.

[0303] In the 88th aspect, the user's posture comprises at least one of a head posture, an eye posture, a foot posture, or a body posture, or the sensor comprises at least one of an imaging system facing inward, an imaging system facing outward, or an inertial measurement unit, or the display system comprises a light field display configured to display one or more than one of the interactive objects in a plurality of depth planes, the system according to aspect 87.

[0304] In the 89th aspect, the processor is further configured to determine, at least in part, a user's field of view (FOV) based on the user's posture, where the FOV comprises a part of the user's environment perceived by the user at a given time, determine context information associated with the FOV, where the context information comprises at least one of a layout of interactive objects within the FOV, a size of the FOV, or a size of one or more than one of the interactive objects within the user's FOV, and be programmed to present an option to switch from a first user input mode to a second user input mode, or from the second user input mode to the first user input mode, at least in part, based on the context information, the system according to any one of aspects 87-88.

[0305] In the 90th aspect, to present the option, the processor is programmed to present a focus indicator in a second shape when the current user input mode is the first user input mode, and present a focus indicator in a first shape when the current user input mode is the second user input mode, the system according to aspect 89.

[0306] In the 91st aspect, the first indication comprises a change in the user's posture, and the second indication comprises an actuation of the user input device, the system according to any one of aspects 87-90.

[0307] On the 92nd aspect, a method for changing a user input mode for a wearable device, under the control of a wearable device comprising a computer processor, the wearable device being configured to enable user interaction with an interactive object within the user's field of regard (FOR), the FOR comprising a part of the user's surrounding environment perceptible by the user via a display system of the wearable device, the method comprising, under the control of the wearable device: determining the user's posture; displaying, via the display system, a first focus indicator associated with a target interactive object in a direction related to the user's posture, the target interactive object comprising a plurality of virtual objects; receiving a selection of the target interactive object; presenting to the user an option for switching the user input mode from posture to a hand gesture on a user input device; displaying, via the display system, the plurality of virtual objects; in response to determining that the user has switched the user input mode from posture to a hand gesture on the user input device, displaying, via the display system, a second focus indicator associated with a target virtual object among the plurality of virtual objects; and updating the second focus indicator, at least in part, based on user input from the user input device.

[0308] On the 93rd aspect, the option is presented in response to a selection of a target interactive object or a plurality of virtual objects or context information associated with the target interactive object, according to the method of aspect 92.

[0309] On the 94th aspect, the context information comprises the density of the plurality of virtual objects, and the option for switching the user input mode from posture to a hand gesture on the user input device is presented in response to determining that the density of the plurality of virtual objects exceeds a threshold density, according to the method of aspect 93.

[0310] In the 95th aspect, the method according to any one of aspects 92 - 94, wherein the target virtual object is at least partially identified based on the user's posture.

[0311] In the 96th aspect, the method according to any one of aspects 92 - 95, wherein the step of updating the second focus indicator includes the step of transitioning the second focus indicator from the target virtual object to another virtual object among a plurality of virtual objects.

[0312] In the 97th aspect, the method according to any one of aspects 92 - 96, further comprising the step of starting a selection event on the target virtual object, the selection event further including at least one of the step of opening a menu associated with the target virtual object or the step of receiving an indication of selecting the target virtual object.

[0313] In the 98th aspect, the method according to any one of aspects 92 - 97, wherein the plurality of virtual objects includes at least one of a weather application or an astronomy application, and in response to a selection of the weather application, the wearable device is programmed to display virtual weather information superimposed on an object in the user's environment, and in response to a selection of the astronomy application, the wearable device is programmed to display an interactive galaxy system with a three - dimensional virtual planet superimposed on the user's environment.

[0314] In a 99th aspect, a wearable system for selecting a virtual object located in a three-dimensional (3D) space, comprising: a display system configured to present the virtual object in the 3D space; a non-transitory data storage configured to store interactive objects in the 3D space; a sensor configured to determine a user's posture; a hardware processor that communicates with the display system, the data storage, and the sensor, determines the user's posture based at least in part on data received from the sensor, determines the user's field of view (FOV) based at least in part on the user's posture, where the FOV comprises a portion of the user's environment perceived by the user at a given time, identifies a group of interactive objects within the FOV, identifies a target interactive object within the FOV based at least in part on the user's posture, and is programmed to initiate a selection event associated with the target interactive object.

[0315] In a 100th aspect, the group of interactive objects is stored in a data structure, and an index associated with each interactive object is determined based at least in part on the position of the interactive object in the 3D space, of the wearable system according to aspect 99.

[0316] In a 101st aspect, in response to initiating a selection event on a target interactive object, the processor of the wearable system is programmed to present, within a two-dimensional (2D) interface, virtual objects within a threshold range of the target interactive object in the 3D space, of the wearable system according to any one of aspects 99-100.

[0317] In a 102nd aspect, the 2D interface is interactive via a user input device, of the wearable system according to aspect 101.

[0318] In the 103rd aspect, in order to identify target interactable objects within the FOV, the processor determines the path of the user's line of sight based on the user's posture, and selects, as target interactable objects, interactable objects that intersect the path of the line of sight, or selects, as target interactable objects, the leftmost or rightmost interactable objects within the user's FOV, where the leftmost or rightmost interactable objects are selected based at least in part on an index associated with a group of interactable objects, and is configured to perform at least one of the steps, the wearable system according to any one of aspects 99-102.

[0319] In the 104th aspect, the processor is configured to initiate a selection event in response to at least one of receiving an input from a user input device or detecting a change in the user's posture, the wearable system according to any one of aspects 99-103.

[0320] In the 105th aspect, the processor is further configured to present a focus indicator associated with the target interactable object, the wearable system according to any one of aspects 99-104.

[0321] In the 106th aspect, further comprising a geolocation sensor configured to obtain data associated with the user's location, the target interactable object is programmed to determine the user's location based on the data obtained by the geolocation sensor, communicate with a remote computing device, obtain weather data based on the user's location, generate virtual elements associated with the weather data, and superimpose the virtual elements within the user's 3D space, the wearable system according to any one of aspects 99-105 comprising a weather application.

[0322] (Conclusion) The processes, methods, and algorithms described in this specification and / or depicted in the accompanying figures are each embodied in code modules that are executed by one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, thereby being fully or partially automated. For example, a computing system can include a general-purpose computer (e.g., a server) or a dedicated computer, a dedicated circuit, etc., programmed with specific computer instructions. The code modules can be installed in dynamic link libraries that can be compiled and linked into executable programs, or can be written in a programming language that is interpreted. In some implementations, certain operations and methods can be performed by circuits specific to a given function.

[0323] Furthermore, because the functional implementations of the present disclosure are sufficiently mathematical, computer, or technically complex, a hardware specific to a particular use or one or more physical computing devices (utilizing appropriate specialized executable instructions) may be required to implement the functionality, for example, due to the amount or complexity of the calculations involved or to provide the results substantially in real time. For example, a video can include many frames, each frame can have millions of pixels, and specifically programmed computer hardware is required to process the video data to provide the desired image processing tasks or applications in a commercially reasonable amount of time.

[0324] A code module or any type of data can be stored on any type of non-transitory computer-readable medium, such as a physical computer storage device including a hard drive, solid state memory, random access memory (RAM), read only memory (ROM), optical disk, volatile or non-volatile storage device, combinations of the same, and / or equivalents. The methods and modules (or data) can also be transmitted as data signals generated on various computer-readable transmission media, including wireless-based and wired / cable-based media (e.g., as part of a carrier wave or other analog or digital propagated signal), and can take various forms (e.g., as part of a single or multiplexed analog signal or as multiple discrete digital packets or frames). The results of the disclosed process or process steps can be persistently or otherwise stored within any type of non-transitory tangible computer storage device or communicated via a computer-readable transmission medium.

[0325] Any process, block, state, step, or functionality in a flowchart described and / or depicted in the figures attached hereto should be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in a process. The various processes, blocks, states, steps, or functionality can be combined, rearranged, added, deleted, modified, or otherwise changed from the exemplary embodiments provided herein. In some embodiments, additional or different computing systems or code modules may implement some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the associated blocks, steps, or states can be performed in other suitable sequences, e.g., sequentially, in parallel, or in some other manner. Tasks or events can be added to or removed from the disclosed exemplary embodiments. Further, the separation of the various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods, and systems can generally be integrated together in a single computer product or packaged in multiple computer products. Many implementation variations are possible.

[0326] The present process, method, and system can be implemented in a network (or distributed) computing environment. The network environment can include an enterprise-wide computer network, an intranet, a local area network (LAN), a wide area network (WAN), a personal area network (PAN), a cloud computing network, a cloud source computing network, the Internet, and the World Wide Web. The network can be a wired or wireless network or any other type of communication network.

[0327] The systems and methods of the present disclosure each have several innovative aspects, none of which alone contribute to or are required for the desirable attributes disclosed herein. The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. Various modifications to the implementations described in this disclosure may be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other implementations without departing from the spirit or scope of the present disclosure. Accordingly, the claims are not intended to be limited to the implementations shown herein, but should be accorded the widest scope consistent with the present disclosure, the principles, and the novel features disclosed herein.

[0328] Certain features described herein in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, the various features described in the context of a single implementation can also be implemented separately in multiple implementations or in any suitable sub-combination. Further, features may be described above as acting in a certain combination and may further be claimed as such, but one or more features from the claimed combination can in some cases be deleted from the combination, and the claimed combination can be directed to a sub-combination or variation of a sub-combination. No single feature or group of features is necessary or essential to every embodiment.

[0329] In particular, conditional statements used herein such as "can", "could", "might", "may", "e.g.", and equivalents, generally convey that while one embodiment includes a certain feature, element, and / or step, another embodiment does not, unless specifically stated otherwise or understood otherwise in the context in which it is used. Thus, such conditional statements generally do not imply that a feature, element, and / or step is required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps should be included or implemented in any particular embodiment, regardless of the author's input or prompting. The terms "comprising", "including", "having", and equivalents are synonyms and are used inclusively in a non-limiting manner, without excluding additional elements, features, acts, operations, etc. Also, the term "or" is used in its inclusive sense (and not in its exclusive sense), and thus, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Additionally, the articles "a", "an", and "the" as used in this application and the appended claims should be construed to mean "one or more than one" or "at least one" unless otherwise defined.

[0330] As used herein, the phrase referring to "at least one of" a list of items refers to any combination of those items, including a single element. As an example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Connective phrases such as "at least one of X, Y, and Z" are generally understood in a context such that, unless otherwise specifically described, they are used to convey that an item, term, etc. can be at least one of X, Y, or Z. Thus, such connective phrases are generally not intended to suggest that an embodiment requires that at least one of X, at least one of Y, and at least one of Z each be present.

[0331] Similarly, operations may be depicted in the drawings in a particular order, but it should be recognized that such operations need not be performed in the particular order shown, or in a sequential order, to achieve the desired results, or that all illustrated operations need not be performed. Further, the drawings may schematically depict one or more exemplary processes in the form of flowcharts. However, other operations not depicted may also be incorporated into the exemplary methods and processes schematically illustrated. For example, one or more additional operations may be performed before, after, simultaneously with, or during any of the illustrated operations. Additionally, operations may be rearranged or reordered in other implementations. In some situations, multitasking and parallel processing may be advantageous. Further, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the program components and systems described may generally be integrated together in a single software product, or packaged into multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired results.

Claims

1. 1. A method for selecting a virtual object located in a three-dimensional (3D) space, the method comprising:

1. An augmented reality (AR) system comprising computer hardware configured to enable user interaction with interactable objects within a user's field of view (FOV), under control of the AR system: determining a group of interactable objects within the FOV of the user; identifying a target interactable object from the group of interactable objects based on a relative position of the target interactable object, the relative position of the target interactable object including at least one of: closest to a midpoint of the user's FOV, leftmost in the user's FOV, and / or rightmost in the user's FOV with respect to other objects in the group of interactable objects; initiating a selection event on the target interactable object; Includes doing The method, wherein identifying the target interactable object includes automatically orienting the target interactable object such that a surface of the target interactable object that is not initially facing the user is oriented to face the user.

2. The method of claim 1 , further comprising storing the group of interactable objects in a data structure.

3. The method of claim 2 , wherein a location of each interactable object in the group of interactable objects in the data structure is determined based at least in part on a location of a corresponding interactable object in the 3D space.

4. The method of claim 3 , wherein the location in the 3D space comprises a location of each interactable object in the group of interactable objects that indicates a distance from a reference position.

5. The method of claim 1 , further comprising receiving a selection of a new target interactable object from the group of interactable objects.

6. Receiving the selection of the new target interactable object from the group of interactable objects comprises: Receiving a first input from a user device; identifying the new target interactable object from the group of interactable objects in response to receiving the first input; The method of claim 5 , comprising:

7. receiving a second input from the user device; initiating an interaction event on the new target interactable object in response to receiving the second input; The method of claim 6 further comprising:

8. Initiating the interaction event includes: resizing the new target interactable object; displaying a menu of the new target interactable objects; Browsing the menu of the new target interactable object; selecting a first item on the menu; searching for a second item in the database; playing a video game associated with the new target interactable object; Watching a video, or Conducting conference calls The method of claim 7, comprising one or more of the following:

9. Receiving the selection of the new target interactable object comprises: determining a line of sight of the user based on a posture of the user; selecting an object that intersects with the path of the user's gaze as the new target interactable object; The method of claim 5 , comprising:

10. The method of claim 1 , further comprising assigning a visible focus indicator to the target interactable object.

11. The method of claim 10 , wherein the visible focus indicator comprises at least one of a highlight, a halo, a color change, a size change, or a change in the perceived depth of the target interactable object.

12. Initiating the selection event includes: selecting a different interactable object to become the new target interactable object; opening a menu associated with the target interactable object; or receiving a confirmation from the user selecting the target interactable object; The method of claim 1 , comprising one or more of:

13. The method of claim 9 , wherein the pose of the user comprises an eye pose or a head pose.

Images