Selection of virtual objects in 3D space

Abstract

To provide a suitable selection of virtual objects in a three-dimensional space. [Solution] A system and method for interacting with virtual objects in three-dimensional space using a wearable system are disclosed. The wearable system can be programmed to allow a user to interact with virtual objects using a user input device and posture. The wearable system can also automatically determine contextual information such as the layout of virtual objects in the user's environment and switch user input modes based on the contextual information.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims priority under 35 U.S.C.§119(e) to U.S. Provisional Application No. 62 / 244,115, filed October 20, 2015, with the title "USER APPLICATIONS, INTERFACES, AND EXPERIENCES WITH AUGMENTED REALITY DISPLAY DEVICES"; U.S. Provisional Application No. 62 / 301,422, filed February 29, 2016, with the title "SELECTING VIRTUAL OBJECTS IN 3D SPACE"; and U.S. Provisional Application No. 62 / 316,179, filed March 31, 2016, with the title "SELECTING VIRTUAL OBJECTS IN 3D SPACE". The entire disclosures of the above provisional applications are hereby incorporated by reference herein in their entirety.

[0002] This disclosure relates to virtual reality, augmented reality, and mixed reality imaging and visualization systems, and more particularly, to systems for interacting with virtual objects within a three - dimensional (3D) space.

Background Art

[0003] Modern computing and display technologies are driving the development of systems for so-called “virtual reality,” “augmented reality,” or “mixed reality” experiences, in which digitally reproduced images or parts thereof are presented to the user in a manner that appears, or can be perceived, as real. Virtual reality or “VR” scenarios typically involve the presentation of digital or virtual image information without transparency to other real-world visual inputs. Augmented reality or “AR” scenarios typically involve digital or virtual image information as an extension of the visualization of the real world around the user. Mixed reality or “MR” relates to the fusion of the real and virtual worlds to generate a new environment in which physical and virtual objects coexist and interact in real time. In conclusion, the human visual perception system is highly complex, making it difficult to generate VR, AR, or MR technologies that facilitate a rich and comfortable and natural presentation of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges related to VR, AR, and MR technologies. [Overview of the Initiative] [Means for solving the problem]

[0004] In some embodiments, a system for changing a user input mode for a wearable device is disclosed. The system is a display system for a wearable device configured to present a three-dimensional (3D) view to a user, the 3D view may comprise a display system comprising an interactable object, a user input device configured to receive user input, a sensor configured to acquire data associated with the user's posture, and a hardware processor communicating with the user input device. The hardware processor determines whether the current user input mode for interacting with the interactable object is a first user input mode or a second user input mode, the first user input mode may be programmed to be at least partially based on the user's posture, and the second user input mode may be programmed to be at least partially based on user input from the user input device. In response to a determination that the current user input mode is the first user input mode, the hardware processor may use sensors to monitor the user's posture and, at least partially, present a focus indicator in a first shape associated with the first user input mode via the display system, based on the monitored posture, in a direction related to the user's posture, receive a first indication, switch to the second user input mode, and in response to the first indication, switch the current user input mode to the second user input mode. In response to a determination that the current user input mode is the second user input mode, the hardware processor may monitor user input from a user input device and, at least partially, present a focus indicator in a second shape associated with the second user input mode via the display system, based on the monitored input, receive a second indication, switch to the first user input mode, and in response to the second indication, switch the current user input mode to the first user input mode.

[0005] In one embodiment, a method for changing a user input mode for a wearable device is disclosed. The method may be performed under the control of a wearable device having a computer processor. The wearable device can be configured to enable user interaction with an interactable object in the user's eye-moving field of view (FOR), the FOR comprising a portion of the user's surrounding environment that is perceptible to the user via the wearable device's display system. The method includes the steps of: determining the user's posture; displaying a first focus indicator via the display system, associated with a target interactable object, in a direction related to the user's posture, wherein the target interactable object comprises a plurality of virtual objects; receiving a selection of the target interactable object; presenting the user with an option to switch the user input mode from posture to hand gestures on a user input device; displaying the plurality of virtual objects via the display system; displaying a second focus indicator via the display system, associated with a target virtual object among the plurality of virtual objects, in response to a determination that the user has switched the user input mode from posture to hand gestures on a user input device; and updating the second focus indicator, at least in part, based on user input from the user input device.

[0006] In some embodiments, wearable systems and methods for selecting virtual objects located in three-dimensional (3D) space are disclosed. The wearable system may comprise a display system configured to present virtual objects in 3D space, non-transient data storage configured to store interactable objects in 3D space, sensors configured to determine the user's posture, and a hardware processor programmed to communicate with the display system, data storage, and sensors. The wearable system and method can, at least partially, determine the user's posture based on data received from sensors, at least partially, determine the user's field of view (FOV) based on the user's posture, the FOV comprising a portion of the user's environment as perceived by the user at a given time, identify a group of interactable objects within the FOV, at least partially, identify a target interactable object within the FOV based on the user's posture, and initiate a selection event associated with the target interactable object.

[0007] Details of one or more implementations of the subject matter described herein are shown in the accompanying drawings and the following description. Other features, aspects, and advantages will be evident from the description, drawings, and claims. Neither this abstract nor any of the following embodiments for carrying out the invention shall claim to define or limit the scope of the subject matter of the invention. The present invention provides, for example, the following: (Item 1) A system for changing the user input mode for a wearable device, A display system for a wearable device configured to present a three-dimensional (3D) view to a user, wherein the 3D view comprises an interactive object. A user input device configured to receive user input, A sensor configured to acquire data associated with the user's posture, A hardware processor that communicates with the user input device, Determining whether the current user input mode for interacting with the interactable object is a first user input mode or a second user input mode, The first user input mode described above is at least partially based on the user's posture, The second user input mode described above is at least partially based on user input from the user input device, In response to the determination that the current user input mode is the first user input mode, Using the aforementioned sensor, the user's posture is monitored, The display system, at least partially, presents a focus indicator in a first shape associated with the first user input mode in a direction related to the user's posture, based on the monitored posture. Receiving the first indication and switching to the second user input mode, In response to the first indication, the current user input mode is switched to the second user input mode, In response to the determination that the current user input mode is the second user input mode, Monitoring user input from the aforementioned user input device, The display system, at least partially, presents a focus indicator in a second shape associated with the second user input mode based on the monitored input, Upon receiving the second indication, the system switches to the first user input mode. In response to the second indication, the current user input mode is switched to the first user input mode. A hardware processor and A system equipped with these features. (Item 2) The aforementioned processor further, Determining, at least partially, the user's field of view (FOV) based on the user's posture, wherein the FOV comprises a portion of the user's environment as perceived by the user at a given time. Determining context information associated with the FOV, wherein the context information comprises at least one of the layout of the interactable objects within the FOV, the size of the FOV, and the size of one or more of the interactable objects within the user's FOV. At least partially, the system provides an option to switch from the first user input mode to the second user input mode or from the second user input mode to the first user input mode based on the contextual information. The system described in item 1, which is programmed to perform the following actions. (Item 3) The system according to item 3, wherein the processor is programmed to present the focus indicator in the second shape when the current user input mode is the first user input mode, and to present the focus indicator in the first shape when the current user input mode is the second user input mode. (Item 4) The system according to item 1, wherein the first indication comprises a change in the user's posture, and the second indication comprises the operation of the user input device. (Item 5) The system according to any one of items 1-4, wherein the user's posture comprises at least one of a head posture, an eye posture, a foot posture, or a body posture, or the sensor comprises at least one of an inward-facing imaging system, an outward-facing imaging system, or an inertial measurement unit, or the display system comprises a light field display configured to display one or more of the interactable objects in a plurality of depth planes. (Item 6) A method for changing the user input mode for a wearable device, Under the control of a wearable device comprising a computer processor, the wearable device is configured to enable user interaction with interactable objects within the user's eye-moving field of view (FOR), and the FOR comprises a portion of the user's surrounding environment that is perceptible to the user via the wearable device's display system, Steps to determine the user's posture, The steps include: displaying a first focus indicator associated with a target interactable object via the display system in a direction related to the user's posture, wherein the target interactable object comprises a plurality of virtual objects; The step of receiving the selection of the target interactable object, The steps include presenting the user with an option to switch the user input mode from posture to hand gestures on a user input device, The steps include displaying the plurality of virtual objects via the display system, In response to the determination that the user has switched the user input mode from posture to hand gestures on the user input device, the display system displays a second focus indicator associated with a target virtual object among the plurality of virtual objects. The steps include updating the second focus indicator based at least partially on user input from the user input device, and Methods that include... (Item 7) The method according to item 6, wherein the options are presented in response to the selection of the target interactable object or to the plurality of virtual objects or contextual information associated with the target interactable object. (Item 8) The method according to item 7, wherein the context information comprises the density of the plurality of virtual objects, and the option for switching the user input mode from posture to hand gestures on a user input device is presented in response to a determination that the density of the plurality of virtual objects exceeds a threshold density. (Item 9) The method according to item 6, wherein the step of updating the second focus indicator includes the step of moving the second focus indicator from the target virtual object to another virtual object among the plurality of virtual objects. (Item 10) The method according to item 6, further comprising the step of initiating a selection event on the target virtual object, wherein the selection event comprises at least one of the steps of opening a menu associated with the target virtual object or receiving an indication to select the target virtual object. (Item 11) The method according to any one of items 6-10, wherein the target virtual object is identified at least partially based on the user's posture. (Item 12) The plurality of virtual objects includes at least one of a weather application or an astronomy application, and in response to selection of the weather application, the wearable device is programmed to display virtual weather information superimposed on objects in the user's environment, and in response to selection of the astronomy application, the wearable device is programmed to display an interactive galaxy including a three-dimensional virtual planet superimposed on the user's environment, the method according to item 6. (Item 13) A wearable system for selecting a virtual object located within a three-dimensional (3D) space, A display system configured to present virtual objects within the 3D space, A non-transitory data storage configured to store interactive objects within the 3D space, A sensor configured to determine a user's pose, A hardware processor programmed to communicate with the display system, the data storage, and the sensor, Determining the user's pose, at least in part, based on data received from the sensor; Determining the user's field of view (FOV), at least in part, based on the user's pose, the FOV comprising a portion of the user's environment perceived by the user at a given time; Identifying a group of interactive objects within the FOV; Identifying a target interactive object within the FOV, at least in part, based on the user's pose; Initiating a selection event associated with the target interactive object; And a hardware processor that performs the above; A wearable system comprising the above. (Item 14) The wearable system according to item 13, wherein the group of interactable objects is stored in a data structure, and the index associated with each interactable object is determined at least in part based on the position of the interactable object in the 3D space. (Item 15) The wearable system according to item 13, wherein, in response to the initiation of a selection event on the target interactable object, the processor of the wearable system is programmed to present a virtual object within a threshold range of the target interactable object in the 3D space within a two-dimensional (2D) interface. (Item 16) The wearable system described in item 15, wherein the 2D interface is interactable via a user input device. (Item 17) To identify target interactable objects within the FOV, the processor: A step of determining the user's line of sight path based on the user's posture and selecting an interactable object that intersects the line of sight path as the target interactable object, or A step of selecting the leftmost or rightmost interactable object within the user's FOV as the target interactable object, wherein the leftmost or rightmost interactable object is selected at least partially based on an index associated with a group of interactable objects. A wearable system as described in item 13, configured to perform at least one of the following: (Item 18) The wearable system according to item 13, wherein the processor is configured to initiate the selection event in response to at least one of receiving input from a user input device or detecting a change in the user's posture. (Item 19) The wearable system according to any one of items 13-19, wherein the processor is further configured to present a focus indicator associated with the target interactable object. (Item 20) The target interactable object further comprises a geolocation sensor configured to acquire data associated with the user's location, and the target interactable object comprises a weather application, and the weather application comprises Based on the data acquired by the geolocation sensor, the user's location is determined. Communicating with a remote computing device and obtaining weather data based on the user's location, To generate a virtual element associated with the aforementioned weather data, The virtual element is superimposed within the user's 3D space. A wearable system as described in item 13, programmed to perform the following actions. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 illustrates an example of a mixed reality scenario involving a virtual reality object and a physical object visible to a person.

[0009] [Figure 2] Figure 2 schematically illustrates an example of a wearable system.

[0010] [Figure 3] Figure 3 schematically illustrates aspects of an approach to simulating a 3D image using multiple depth planes.

[0011] [Figure 4] Figure 4 schematically illustrates an example of a waveguide stack for outputting image information to the user.

[0012] [Figure 5]Figure 5 shows an exemplary output beam that can be output by a waveguide.

[0013] [Figure 6] Figure 6 is a schematic diagram showing an optical system that includes a waveguide apparatus, an optical coupler subsystem for optically coupling light to or from the waveguide apparatus, and a control subsystem used in the generation of a multifocal stereoscopic display, image, or light field.

[0014] [Figure 7] Figure 7 is a block diagram of an embodiment of the wearable system.

[0015] [Figure 8] Figure 8 is a process flow diagram of an example of how to render virtual content in relation to recognized objects.

[0016] [Figure 9] Figure 9 is a block diagram of another embodiment of the wearable system.

[0017] [Figure 10] Figure 10 is a process flow diagram of an embodiment of a method for determining user input to a wearable system.

[0018] [Figure 11] Figure 11 is a process flow diagram of an example of a method for interacting with a virtual user interface.

[0019] [Figure 12] Figure 12 illustrates embodiments of virtual objects in the field of view and virtual objects in the kinetic field of view.

[0020] [Figure 13A] Figure 13A illustrates an example of selecting an interactive object using touch gestures on a touchscreen of a user input device.

[0021] [Figure 13B] Figure 13B illustrates an example of filtering selectable objects using hand gestures from a user input device.

[0022] [Figure 14] Figure 14 shows an example of a coordinate system for head posture.

[0023] [Figure 15] Figure 15 illustrates an example of interaction with an interactable object using head posture.

[0024] [Figure 16] Figure 16 illustrates an example of interaction with an interactable object using hand gestures.

[0025] [Figure 17] Figure 17 illustrates an exemplary interaction event with a weather application.

[0026] [Figure 18] Figure 18 illustrates an exemplary user experience of interacting with a 3D virtual object.

[0027] [Figure 19] Figure 19 illustrates an exemplary process for selecting a virtual object using a combination of posture and hand gestures on a user input device.

[0028] [Figure 20] Figure 20 illustrates an exemplary process for interacting with a virtual object using a combination of posture and hand gestures on a user input device.

[0029] [Figure 21] Figure 21 illustrates an exemplary process for switching input control from head posture to hand gestures based on contextual information.

[0030] [Figure 22] Figure 22 illustrates an exemplary process for switching user interaction modes based on contextual information.

[0031] [Figure 23] Figure 23 illustrates an exemplary process of interaction with an interactable object, comprising a group of virtual objects.

[0032] Throughout the drawings, reference numbers may be reused to indicate correspondences between the referenced elements. The drawings are provided to illustrate exemplary embodiments described herein and are not intended to limit the scope of this disclosure. [Modes for carrying out the invention]

[0033] (Overview) By using AR / VR / MR devices, users may wish to target and select objects in a three-dimensional (3D) space using a virtual user interface. For example, a user may select a virtual object using physical posture, such as physically approaching, grasping, or touching an item. A user may also select a virtual object by pointing at and clicking on it using a virtual ray or beam. However, these techniques may be tiring and may make it difficult to precisely select objects, as they may require the user to maintain a static posture to achieve the selection.

[0034] This disclosure provides embodiments of a wearable system that address some or all of these problems. In one embodiment, a user may move their head to focus their eyes on a group of objects. The nearest object to the center of the user's field of view may be highlighted as a potential target object, and the user may activate a user input device (e.g., by swiping on a touchscreen) to move the highlight from one object to another. The user can confirm the selection of the target object by activating the user input device again (e.g., by touching a touchscreen). Once selected, the AR user interface may allow the user to perform additional actions on the selected target object (e.g., displaying or selecting from a menu associated with the object, performing an action associated with a game in which the target object appears, etc.). This technique may be particularly advantageous for the user to select objects of interest while reducing fatigue, because it is difficult to precisely control head posture. The wearable system may preliminarily identify objects of interest based on the user's head posture, while allowing the user to precisely select objects using hand gestures.

[0035] In some implementations, an interactable object may contain multiple virtual objects. For example, a virtual user interface plane may contain multiple virtual applications, such as a video streaming application, a virtual classroom application, a weather application, a game application, an astronomy application, etc. The wearable system may support different user input modes based on the characteristics of the interactable object. For example, when the interactable object is a user interface plane (which may be large in size), the wearable system may allow the user to interact with it using posture. On the other hand, when the interactable object is relatively small, the wearable system may instead set a user input device as the default input mode, allowing the user to interact with the virtual object precisely. These implementations may be advantageous because moving and targeting large objects may not require much precision in the user's movement, while moving and selecting small objects may require the user to target precisely.

[0036] The wearable system can also determine the user input mode based on contextual information. For example, the wearable system can determine the layout of virtual objects in the user's environment. If the wearable system detects a high-density cluster of virtual objects in the user's line of sight, it may give the user the option to switch input control from head control to hand control. In this way, the user can interact with virtual objects more precisely. In another embodiment, the AR system may detect the orientation of an object (e.g., vertical or horizontal) and provide appropriate interaction for the user (e.g., volume control of a television application appearing vertically in front of the user or typing control of a virtual keyboard appearing horizontally on the user's desk).

[0037] Wearable systems can enable users to share virtual content with others (who are also wearing wearable systems) by, for example, navigating a world map of their environment over a network, or by communicating (or updating) virtual content between wearable systems.

[0038] (Examples of 3D displays) Figure 1 illustrates an example of a mixed reality scenario involving a virtual reality object and a physical object that are visible to a person. In Figure 1, MR scene 100 is depicted, and the user of the MR technology sees a real-world park-like setting 110 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of the MR technology also perceives "seeing" a robotic figure 130 standing on the real-world platform 120 and a flying cartoonish avatar character 140 that appears to be a personification of a bumblebee, although these elements do not exist in the real world.

[0039] It may be desirable for a 3D display to generate a distance-accommodative response corresponding to the virtual depth of each point within the display's field of view, in order to produce a true sense of depth, more specifically, a simulated sense of surface depth. If the distance-accommodative response for a display point does not correspond to the virtual depth of that point as determined by the binocular depth cues for convergence and stereopsis, the human eye may experience distance-accommodative collision, which can result in unstable imaging, harmful eye strain, headaches, and, in the absence of distance-accommodative information, a near-complete loss of surface depth.

[0040] VR, AR, and MR experiences can be provided by a display system having a display that provides the viewer with images corresponding to multiple depth planes. The images may differ for each depth plane (e.g., providing slightly different presentations of scenes or objects) and can be individually focused by the viewer's eyes, thereby helping to provide the user with depth cues based on the eye's accommodation required to focus on different image features relating to scenes located on different depth planes, or based on observing different image features on different depth planes that are out of focus. As discussed elsewhere herein, such depth cues provide a reliable perception of depth.

[0041] Figure 2 illustrates an embodiment of the wearable system 200. The wearable system 200 includes a display 220 and various mechanical and electronic modules and systems to support the functions of the display 220. The display 220 may be coupled to a frame 230, which is wearable by a user, wearer, or viewer 210. The display 220 can be positioned in front of the user 210's eyes. The display 220 can present AR / VR / MR content to the user. The display 220 can present a head-mounted display (HMD) that is worn on the user's head. In some embodiments, a speaker 240 is coupled to the frame 230 and positioned adjacent to the user's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / shapeable acoustic control).

[0042] The wearable system 200 may include an outward-facing imaging system 464 (shown in Figure 4) that observes the world within the user's surrounding environment. The wearable system 200 may also include an inward-facing imaging system 462 (shown in Figure 4) that can track the user's eye movements. The inward-facing imaging system can track the movement of one eye or both eyes. The inward-facing imaging system 462 may be mounted on the frame 230 and may communicate with a processing module 260 or 270 that processes the image information acquired by the inward-facing imaging system and can determine, for example, the pupil diameter or orientation of the user's eyes, eye movements, or eye posture.

[0043] As an example, the wearable system 200 can acquire images of the user's posture using an outward-facing imaging system 464 or an inward-facing imaging system 462. The images may be still images, video frames or videos, a combination thereof, or equivalent.

[0044] The display 220 is operably coupled to a local data processing module 260 (250), which can be mounted in various configurations, such as being fixedly attached to the frame 230 by wired or wireless connections, fixed to a helmet or hat worn by the user, built into headphones, or otherwise detachably attached to the user 210 (for example, in a backpack configuration or a belt-connected configuration).

[0045] The local processing and data module 260 may include a hardware processor and digital memory such as non-volatile memory (e.g., flash memory), both of which may be used to assist in data processing, caching, and storage. The data may include a) data captured from sensors (e.g., cameras in an inward-facing imaging system and / or an outward-facing imaging system), microphones, inertial measuring units (IMUs), accelerometers, compasses, global positioning systems (GPS), wireless devices, or gyroscopes (e.g., operably coupled to frame 230 or otherwise attached to user 210), or b) data acquired or processed using the remote processing module 270 and / or remote data repository 280 for transmission to display 220 after such processing or reading. The local processing and data modules 260 may be operably coupled to the remote processing module 270 or the remote data repository 280 by communication links 262 or 264, such as via wired or wireless communication links, so that these remote modules are available as resources to the local processing and data modules 260. In addition, the remote processing module 280 and the remote data repository 280 may be operably coupled to each other.

[0046] In some embodiments, the remote processing module 270 may comprise one or more processors configured to analyze and process data and / or image information. In some embodiments, the remote data repository 280 may comprise a digital data storage facility, which may be available through the internet or other networking configurations in a “cloud” resource configuration. In some embodiments, all data is stored, and all calculations are performed in the local processing and data module, enabling fully autonomous use from the remote module.

[0047] The human visual system is complex and struggles to provide a realistic perception of depth. While not limited by theory, it is believed that an object viewer may perceive an object as three-dimensional due to a combination of vergence and accommodation. The vergence and divergence of two eyes relative to each other (i.e., rotational movement of the pupils toward or away from each other to converge the lines of sight and fix them on an object) is closely related to the focusing (or "accommodation") of the eye's lens. Under normal conditions, changing the focus of the eye's lens, or accommodating the eye to shift focus from one object to another at a different distance, will automatically produce a corresponding change in vergence and divergence at the same distance, under a relationship known as the "accommodation-vergence-divergence reflex." Similarly, a change in vergence and divergence will, under normal conditions, induce a corresponding change in accommodation. A display system that provides a better match between distance accommodation and convergence / divergence motion can create a more realistic and comfortable simulation of three-dimensional images.

[0048] Figure 3 illustrates aspects of an approach to simulating a three-dimensional image using multiple depth planes. Referring to Figure 3, objects at various distances from eyes 302 and 304 on the z-axis are accommodated by eyes 302 and 304 so that those objects are in focus. Eyes 302 and 304 take on specific accommodated states, focusing objects at different distances along the z-axis. As a result, a specific accommodated state can be said to be associated with one of the specific depth planes 306, having an associated focal length so that an object or part of an object in a particular depth plane is in focus when the eye is accommodated with respect to that depth plane. In some embodiments, the three-dimensional image may be simulated by providing a different presentation of the image for each of eyes 302 and 304, and by providing a different presentation of the image corresponding to each of the depth planes. For the sake of clarity in the illustration, although shown as separate, it should be understood that the fields of view of eyes 302 and 304 may overlap, for example, as the distance along the z-axis increases. Furthermore, although shown as flat for the sake of illustration, it should be understood that the contour of the depth plane may be curved in physical space so that all features within the depth plane are in focus with the eye in a particular distance-accommodated state. While not limited by theory, it is thought that the human eye can typically interpret a finite number of depth planes to provide depth perception. Consequently, a highly realistic simulation of perceived depth can be achieved by providing the eye with different representations of images corresponding to each of these limited number of depth planes.

[0049] (Waveguide stack assembly) Figure 4 illustrates an embodiment of a waveguide stack for outputting image information to a user. The wearable system 400 includes a waveguide stack or stacked waveguide assembly 480, which may be used to provide three-dimensional perception to the eye / brain using a plurality of waveguides 432b, 434b, 436b, 438b, 400b. In some embodiments, the wearable system 400 may correspond to the wearable system 200 of Figure 2, and Figure 4 shows some parts of the wearable system 200 in more detail. For example, in some embodiments, the waveguide assembly 480 may be integrated into the display 220 of Figure 2.

[0050] Continuing with Figure 4, the waveguide assembly 480 may also include several features 458, 456, 454, and 452 between the waveguides. In some embodiments, features 458, 456, 454, and 452 may be lenses. In other embodiments, features 458, 456, 454, and 452 may not be lenses. Rather, they may simply be spacers (e.g., cladding layers or structures for forming air gaps).

[0051] Waveguides 432b, 434b, 436b, 438b, 440b or multiple lenses 458, 456, 454, 452 may be configured to transmit image information to the eye with varying levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a specific depth plane and configured to output image information corresponding to that depth plane. Image input devices 420, 422, 424, 426, 428 may be used to input image information into waveguides 440b, 438b, 436b, 434b, 432b, each of which may be configured to disperse incident light across each individual waveguide for output toward the eye 410. Light exits from the output surfaces of image input devices 420, 422, 424, 426, and 428 and is fed into the corresponding input edges of waveguides 440b, 438b, 436b, 434b, and 432b. In some embodiments, a single beam of light (e.g., a collimated beam) may be fed into each waveguide and output an entire field of cloned collimated beams, directed toward the eye 410 at a specific angle (and divergence) corresponding to a depth plane associated with a particular waveguide.

[0052] In some embodiments, the image input devices 420, 422, 424, 426, and 428 are discrete displays that generate image information for input into their respective corresponding waveguides 440b, 438b, 436b, 434b, and 432b, respectively. In some other embodiments, the image input devices 420, 422, 424, 426, and 428 are output terminals of a single multiplexed display, which can pipe image information to each of the image input devices 420, 422, 424, 426, and 428, for example, via one or more optical conduits (such as fiber optic cables).

[0053] The controller 460 controls the operation of the stacked waveguide assembly 480 and the image input devices 420, 422, 424, 426, and 428. The controller 460 includes programming (e.g., instructions in a non-transient computer-readable medium) to coordinate the timing and delivery of image information to the waveguides 440b, 438b, 436b, 434b, and 432b. In some embodiments, the controller 460 may be a single integrated device or a distributed system connected by wired or wireless communication channels. In some embodiments, the controller 460 may be part of a processing module 260 or 270 (illustrated in Figure 2).

[0054] Waveguides 440b, 438b, 436b, 434b, and 432b may be configured to propagate light within each individual waveguide by total internal reflection (TIR). Waveguides 440b, 438b, 436b, 434b, and 432b may each be planar or have another shape (e.g., curved), with a main upper and lower surface and a rim extending between their main upper and lower surfaces. In the illustrated configuration, waveguides 440b, 438b, 436b, 434b, and 432b may each include light extraction optical elements 440a, 438a, 436a, 434a, and 432a, respectively, configured to extract light from the waveguides by redirecting the light, propagating it within each individual waveguide, and outputting image information from the waveguides to the eye 410. The extracted light may also be referred to as externally coupled light, and the light extraction optical elements may also be referred to as externally coupled optical elements. The beam of extracted light is output by the waveguide to the location where the light propagating within the waveguide strikes the light redirection element. The light extraction optical elements (440a, 438a, 436a, 434a, 432a) may be, for example, reflective or diffracting optical features. For the sake of clarity and to facilitate the explanation, they are shown positioned on the bottom main surface of waveguides 440b, 438b, 436b, 434b, 432b, but in some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be positioned on the top or bottom main surface, or directly within the volume of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, and 432a may be mounted on a transparent substrate and formed within a layer of material that forms the waveguides 440b, 438b, 436b, 434b, and 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, and 432b may be monolithic material pieces, and the light extraction optical elements 440a, 438a, 436a, 434a, and 432a may be formed on and / or inside the material piece.

[0055] Continuing with Figure 4, as discussed herein, each waveguide 440b, 438b, 436b, 434b, and 432b is configured to emit light and form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 as it is introduced into such waveguide 432b. The collimated light may represent the optical infinity focal plane. The next waveguide 434b may be configured to emit collimated light that passes through a first lens 452 (e.g., a negative lens) before reaching the eye 410. The first lens 452 may generate a slight convex wavefront curvature so that the eye / brain interprets the light emanating from the next waveguide 434b as emanating from a first focal plane that is closer inward from optical infinity toward the eye 410. Similarly, the third waveguide 436b passes its output light through both the first lens 452 and the second lens 454 before reaching the eye 410. The combined refractive power of the first and second lenses 452 and 454 may be configured to generate another increment of wavefront curvature so that the eye / brain interprets the light emanating from the third waveguide 436b as emanating from a second focal plane that is even closer inward toward the person from optical infinity than the light from the next waveguide 434b.

[0056] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured to use the highest waveguide 440b in the stack to transmit its output through all the lenses between it and the eye for aggregated focal power representing the focal plane closest to the person. When viewing / interpreting light originating from the other side world 470 of the stacked waveguide assembly 480, a compensating lens layer 430 may be positioned on top of the stack to compensate for the stack of lenses 458, 456, 454, 452, and to compensate for the aggregated power of the lower lens stacks 458, 456, 454, 452. Such a configuration provides the same number of perceived focal planes as there are available waveguide / lens pairs. Both the light-extracting optical elements of the waveguides and the focusing sides of the lenses may be static (e.g., not dynamic or electrically active). In some alternative embodiments, either or both may be dynamic using electrically active features.

[0057] Continuing with Figure 4, the light extraction optical elements 440a, 438a, 436a, 434a, and 432a may be configured to redirect light from their respective waveguides and to output the light using appropriate amounts of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides having different associated depth planes may have different configurations of light extraction optical elements that output light using different amounts of divergence corresponding to the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, and 432a may be three-dimensional or surface features that can be configured to output light at specific angles. For example, the light extraction optical elements 440a, 438a, 436a, 434a, and 432a may be volume holograms, surface holograms, and / or diffraction gratings. Optical elements for light extraction, such as diffraction gratings, are described in U.S. Patent Publication No. 2015 / 0178939, published on June 25, 2015 (which is incorporated herein by reference in its entirety).

[0058] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, and 432a are diffraction features, i.e., “diffractive optical elements” (also referred to herein as “DOEs”), that form a diffraction pattern. Preferably, the DOEs have relatively low diffraction efficiency such that only a portion of the beam light is deflected toward the eye 410 using each intersection of the DOEs, while the remainder continues to travel through the waveguide via total internal reflection. The light carrying the image information is therefore split into several associated emission beams that exit the waveguide at multiple locations, resulting in a very uniform emission pattern toward the eye 304 with respect to this particular collimated beam bouncing within the waveguide.

[0059] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer-dispersed liquid crystal in which microdroplets form a diffraction pattern in a host medium, and the refractive index of the microdroplets can be switched to a refractive index that substantially matches that of the host material (in which case the pattern does not significantly diffract incident light), or the refractive index of the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0060] In some embodiments, the number and distribution of depth planes or depth of field may vary dynamically based on the pupil size or orientation of the viewer's eye. The depth of field may change inversely with the viewer's pupil size. As a result, as the pupil size of the viewer's eye decreases, the depth of field increases so that one plane that is indistinguishable because its location is beyond the eye's depth of focus becomes discernible and appears more in focus with the decrease in pupil size and the corresponding increase in depth of field. Similarly, the number of spaced-out depth planes used to present different images to the viewer may decrease with the decreased pupil size. For example, it may not be possible for a viewer to clearly perceive the details of both the first and second depth planes at one pupil size without adjusting the eye's accommodation from one depth plane to the other. However, these two depth planes may simultaneously be sufficient to focus on the user at a different pupil size without changing accommodation.

[0061] In some embodiments, the display system may vary the number of waveguides receiving image information based on a determination of pupil size and / or orientation, or in response to the reception of an electrical signal indicating a particular pupil size and / or orientation. For example, if the user's eye is unable to distinguish between two depth planes associated with two waveguides, the controller 460 may be configured or programmed to stop providing image information to one of these waveguides. Advantageously, this can reduce the processing load on the system and thereby increase the system's responsiveness. In embodiments where the DOE for a waveguide is switchable between on and off states, the DOE may be switched off when the waveguide receives image information.

[0062] In some embodiments, it may be desirable to satisfy the condition that the emitted beam has a diameter less than the diameter of the viewer's eye. However, satisfying this condition may be difficult in light of the variability of the viewer's pupil size. In some embodiments, this condition is satisfied over a wide range of pupil sizes by varying the size of the emitted beam in response to the determination of the viewer's pupil size. For example, as the pupil size decreases, the size of the emitted beam may also decrease. In some embodiments, the size of the emitted beam may be varied using a variable aperture.

[0063] The wearable system 400 may include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the field of view (FOV), and the imaging system 464 may sometimes be referred to as an FOV camera. The entire area available for viewing or imaging by the viewer may be referred to as the eye-moving field of view (FOR). FOR may include a solid angle of 4π steradians surrounding the wearable system 400 so that the wearer moves their body, head, or eyes to perceive substantially any direction in space. In other circumstances, the wearer's movement may be more restrained, and accordingly, the wearer's FOR may tangent to a smaller solid angle. Images obtained from the outward-facing imaging system 464 can be used, for example, to track gestures made by the user (e.g., hand or finger gestures) and to detect objects in the world 470 in front of the user.

[0064] The wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera) that observes user movements such as eye and face movements. The inward-facing imaging system 466 may be used to capture an image of the eye 410 and to determine the pupil size or orientation of the eye 304. The inward-facing imaging system 466 may be used to determine the direction the user is looking (e.g., eye posture) or to obtain an image for the user's biometric identification (e.g., via iris recognition). In some embodiments, at least one camera may be used, independently for each eye, to determine the pupil size or eye posture of each eye separately, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye. In some other embodiments, the pupil diameter or orientation of only one eye 410 (e.g., using only one camera per pair of eyes) is determined and assumed to be similar with respect to both of the user's eyes. Images obtained by the inward-facing imaging system 466 may be used by the wearable system 400 to determine the user's eye posture or mood, or to determine the audio or visual content to be presented to the user. The wearable system 400 may also use sensors such as an IMU, accelerometer, and gyroscope to determine head posture (e.g., head position or head orientation).

[0065] The wearable system 400 may include a user input device 466 that allows the user to input commands to a controller 460 and interact with the wearable system 400. For example, the user input device 466 may include a trackpad, touchscreen, joystick, multi-degree-of-freedom (DOF) controller, capacitive sensing device, game controller, keyboard, mouse, directional pad (D-pad), wand, tactile device, totem (e.g., functioning as a virtual user input device), etc. In some cases, the user may use a finger (e.g., thumb) to press or swipe over a touch-sensitive input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held in the user's hand while using the wearable system 400. The user input device 466 can communicate with the wearable system 400 via wired or wireless communication.

[0066] Figure 5 shows an embodiment of an outgoing beam output by a waveguide. Although one waveguide is shown, other waveguides within the waveguide assembly 480 may function similarly, and it should be understood that the waveguide assembly 480 includes multiple waveguides. Light 520 is injected into waveguide 432b at the input edge 432c of waveguide 432b and propagates through waveguide 432b by TIR. At the point where light 520 collides with DOE 432a, a portion of the light exits the waveguide as an outgoing beam 510. The outgoing beams 510 are shown as substantially parallel, but they may also be redirected to propagate towards the eye 410 at a certain angle depending on the depth plane associated with waveguide 432b (e.g., forming a divergent outgoing beam). It should be understood that a nearly parallel emitted beam may represent a waveguide with an optical element that externally couples the light and forms an image that appears to be set on the depth plane at long distances from the eye 410 (e.g., optical infinity). Other waveguides or other sets of optical elements may output a more divergent emitted beam pattern that requires the eye 410 to adjust to a closer distance and focus on the retina, and which would be interpreted by the brain as light from a distance closer to the eye 410 than optical infinity.

[0067] Figure 6 is a schematic diagram showing an optical system that includes a waveguide apparatus, an optical coupler subsystem for optically coupling light to or from the waveguide apparatus, and a control subsystem used in the generation of a multifocal stereoscopic display, image, or light field. The optical system may include a waveguide apparatus, an optical coupler subsystem for optically coupling light to or from the waveguide apparatus, and a control subsystem. The optical system may be used to generate a multifocal stereoscopic display, image, or light field. The optical system may include one or more primary plane waveguides 632a (only one is shown in Figure 6) and one or more DOEs 632b associated with at least one of each of the primary waveguides 632a. The plane waveguides 632b may be analogous to waveguides 432b, 434b, 436b, 438b, and 440b discussed with reference to Figure 4. The optical system may employ a dispersed waveguide apparatus to relay light along a first axis (vertical or Y-axis in the diagram of Figure 6) and expand the effective exit pupil of light along the first axis (e.g., Y-axis). The dispersed waveguide apparatus may include, for example, a dispersed plane waveguide 622b and at least one DOE 622a (illustrated by a double dashed line) associated with the dispersed plane waveguide 622b. The dispersed plane waveguide 622b may be similar to or the same as a primary plane waveguide 632b having a different orientation in at least some respects. Similarly, at least one DOE 622a may be similar to or the same as a DOE 632a in at least some respects. For example, the dispersed plane waveguide 622b or DOE 622a may be made of the same material as the primary plane waveguide 632b or DOE 632a, respectively. An embodiment of the optical display system 600 shown in Figure 6 can be integrated into the wearable system 200 shown in Figure 2.

[0068] The relayed and dilated light can be optically coupled from the dispersed waveguide apparatus into one or more primary plane waveguides 632b. The primary plane waveguides 632b can relay light along a second axis (e.g., horizontal or X-axis in the diagram of Figure 6) perpendicular to the first axis. It should be noted that the second axis can be a non-orthogonal axis to the first axis. The primary plane waveguides 632b dilate the effective exit pupil of the light along their second axis (e.g., X-axis). For example, a dispersed plane waveguide 622b can relay and dilate light along the vertical or Y-axis, and can pass its light into a primary plane waveguide 632b which can relay and dilate light along the horizontal or X-axis.

[0069] The optical system may include one or more colored light sources (e.g., red, green, and blue laser light) 610 that can be optically coupled into the proximal end of a single-mode optical fiber 640. The distal end of the optical fiber 640 may be screwed or received through a hollow tube 642 made of piezoelectric material. The distal end protrudes from the tube 642 as an unfixed, flexible cantilever 644. The piezoelectric tube 642 can be associated with four quadrant electrodes (not shown). The electrodes may be plated, for example, on the outside, outer surface, outer periphery, or diameter of the tube 642. A core electrode (not shown) may also be located in the core, center, inner periphery, or inner diameter of the tube 642.

[0070] For example, a drive electronic device 650, electrically coupled via wire 660, drives a pair of opposing electrodes to independently bend the piezoelectric tube 642 along two axes. The protruding distal tip of the optical fiber 644 has a mechanical resonance mode. The resonance frequency may depend on the diameter, length, and material properties of the optical fiber 644. By vibrating the piezoelectric tube 642 near the first mechanical resonance mode of the fiber cantilever 644, the fiber cantilever 644 can be vibrated and swept through a large deflection.

[0071] By stimulating resonant vibrations in two axes, the tip of the fiber cantilever 644 is scanned in two axes within an area that fills a two-dimensional (2-D) scan. By modulating the intensity of the light source 610 in synchronization with the scanning of the fiber cantilever 644, the light emitted from the fiber cantilever 644 forms an image. A description of such a setup is provided in U.S. Patent Publication 2014 / 0003762, which is incorporated herein by reference in its entirety.

[0072] Components of the optical coupler subsystem can collimate light emitted from the scanning fiber cantilever 644. The collimated light can be reflected by the mirrored surface 648 into a narrow-dispersion planar waveguide 622b containing at least one diffractive optical element (DOE) 622a. The collimated light propagates perpendicularly along the dispersive planar waveguide 622b (with respect to the diagram in Figure 6) by TIR, thereby repeatedly intersecting with the DOE 622a. The DOE 622a preferably has a low diffraction efficiency. This causes a portion of the light (e.g., 10%) to be diffracted toward the edge of the larger primary planar waveguide 632b at each intersection with the DOE 622a, while a portion of the light continues along its original trajectory along the length of the dispersive planar waveguide 622b via TIR.

[0073] At each intersection with DOE622a, additional light can be diffracted toward the entrance of the primary waveguide 632b. By splitting the incident light into multiple external coupling sets, the exit pupil of the light can be vertically extended by DOE4 in the dispersed plane waveguide 622b. This vertically extended light, externally coupled from the dispersed plane waveguide 622b, can enter the edge of the primary plane waveguide 632b.

[0074] Light entering the primary waveguide 632b can propagate horizontally along the primary waveguide 632b (relative to the diagram in Figure 6) via TIR. As the light intersects with DOE 632a at multiple points, it propagates horizontally along at least a portion of the length of the primary waveguide 632b via TIR. DOE 632a may be advantageously designed or configured to have a phase profile which is the sum of linear and radially symmetric diffraction patterns, and to produce both deflection and focusing of light. DOE 632a may advantageously have a low diffraction efficiency (e.g., 10%) such that only a portion of the beam of light is deflected towards the viewer's eye at each intersection of DOE 632a, while the rest of the light continues to propagate through the primary waveguide 632b via TIR.

[0075] At each intersection point between the propagating light and the DOE632a, a portion of the light is diffracted toward the adjacent surface of the primary waveguide 632b, allowing the light to escape from the TIR and be emitted from the surface of the primary waveguide 632b. In some embodiments, the radially symmetric diffraction pattern of the DOE632a also imparts a certain focal level to the diffracted light, shaping the wavefronts of individual beams (e.g., imparting curvature) and steering the beams to an angle that matches the designed focal level.

[0076] Accordingly, these different paths can be used to couple light outside the primary plane waveguide 632b by resulting in different filling patterns in the DOE 632a at different angles, focal levels, and / or in the exit pupil. Different filling patterns in the exit pupil can be advantageously used to generate a light field display with multiple depth planes. Each layer in the waveguide assembly or a set of layers in a stack (e.g., three layers) may be employed to generate individual colors (e.g., red, blue, and green). For example, a first set of three adjacent layers may be employed to generate red, blue, and green light at a first depth of focus. A second set of three adjacent layers may be employed to generate red, blue, and green light at a second depth of focus, respectively. Multiple sets may be employed to generate a full 3D or 4D color image light field with various depths of focus.

[0077] (Other components of the wearable system) In many implementations, the wearable system may include other components in addition to, or as alternatives to, the components of the wearable system described above. The wearable system may include, for example, one or more tactile devices or components. The tactile devices or components may be operable to provide a sense of touch to the user. For example, the tactile devices or components may provide a sense of pressure and / or texture when touching virtual content (e.g., virtual objects, virtual tools, other virtual structures). The tactile sensation may replicate the sensation of a physical object represented by a virtual object, or the sensation of an imaginary object or character represented by virtual content (e.g., a dragon). In some implementations, the tactile devices or components may be worn by the user (e.g., user-wearable gloves). In some implementations, the tactile devices or components may be held by the user.

[0078] A wearable system may include, for example, one or more physical objects that are operable by the user and enable input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as, for example, pieces of metal or plastic, walls, or the surface of a table. In some implementations, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, a totem may simply provide a physical surface, and the wearable system may render a user interface so that it appears to the user as being on one or more of the totems. For example, a wearable system may render images of a computer keyboard and trackpad so that they appear to reside on one or more of the totems. For example, a wearable system may render a virtual computer keyboard and virtual trackpad so that they appear to be on the surface of a thin rectangular aluminum plate that acts as a totem. The rectangular plate itself does not have any physical keys, trackpads, or sensors. However, the wearable system may detect user operation or interaction or touch using the rectangular plate as a selection or input made via a virtual keyboard or virtual trackpad. The user input device 466 (shown in Figure 4) may be an embodiment of the totem, which may include a trackpad, touchpad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. The user may use the totem alone or in combination with posture to interact with the wearable system and / or other users.

[0079] Examples of wearable devices, HMDs, and display systems and usable tactile devices and totems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777 (which is incorporated herein in whole by reference).

[0080] (Examples of wearable systems, environments, and interfaces) Wearable systems may employ various mapping-related techniques to achieve high depth of field within the rendered light field. When mapping a virtual world, it is advantageous to capture all features and points in the real world and accurately depict virtual objects in relation to the real world. To achieve this objective, FOV images captured by the user of the wearable system can be added to the world model by including new images that convey information about various points and features of the real world. For example, a wearable system can collect a set of map points (2D points or 3D points, etc.), find new map points, and render a more accurate version of the world model. The world model of the first user can be communicated to a second user (e.g., via a network such as a cloud network) so that the second user can experience the world surrounding the first user.

[0081] Figure 7 is a block diagram of an embodiment of the MR environment 700. The MR environment 700 may be configured to receive inputs (e.g., visual input 702 from the user's wearable system, steady input 704 such as an indoor camera, sensory input 706 from various sensors, gestures, totems, eye tracking, user input, etc. from a user input device 504) from one or more user wearable systems (e.g., wearable system 200 or display system 220) or steady indoor systems (e.g., indoor camera, etc.). The wearable system can use various sensors (e.g., accelerometer, gyroscope, temperature sensor, motion sensor, depth sensor, GPS sensor, inward-facing imaging system, outward-facing imaging system, etc.) to determine the location of the user's environment and various other attributes. This information may be further supplemented with information from a steady camera in the room, which may provide images from different viewpoints or various cues. Image data acquired by the camera (e.g., indoor camera or camera of an outward-facing imaging system) may be reduced to a set of mapping points.

[0082] One or more object recognition devices 708 can crawl through received data (e.g., a collection of points), recognize or map the points, tag images, and link semantic information to objects using a map database 710. The map database 710 may contain various points and their corresponding objects collected over time. The various devices and the map database can be interconnected through a network (e.g., LAN, WAN, etc.) and can access the cloud.

[0083] Based on the main information and set of points in the map database, object recognition devices 708a-708n may recognize objects, supplement them with semantic information, and give them life. For example, if an object recognition device recognizes that a set of points is a door, the system may combine some semantic information (e.g., the door has a hinge and moves 90 degrees around the hinge). If an object recognition device recognizes that a set of points is a mirror, the system may combine semantic information that the mirror has a reflective surface that can reflect images of objects in the room. Over time, the map database grows as the system (which may reside locally or be accessible via a wireless network) accumulates more data from around the world. Once an object is recognized, the information may be transmitted to one or more wearable systems. For example, the MR environment 700 may contain information about the scene being generated in California. The environment 700 may be transmitted to one or more users in New York. Based on data received from the FOV camera and other inputs, the object recognition device and other software components can map points collected from various images and recognize objects, etc., so that the scene can be accurately "passed" to a second user who may be located in a different part of the world. Environment 700 may also use a topology map for location identification purposes.

[0084] Figure 8 is a process flow diagram of an embodiment of Method 800 for rendering virtual content in relation to a recognized object. Method 800 describes a way in which a virtual scene may be represented to a user of a wearable system. The user may be geographically distant from the scene. For example, the user may be in New York but may want to view a scene currently happening in California, or may want to go for a walk with a friend who is in California.

[0085] In block 810, the AR system may receive input about the user's environment from the user and other users. This may be achieved through various input devices and knowledge already held in a map database. The user's FOV camera, sensors, GPS, eye tracking, etc., transmit information to the system in block 810. In block 820, the system may determine rough points based on this information. Rough points may be used to determine posture data (e.g., head posture, eye posture, body posture, or hand gestures) which can be used to display and understand the orientation and position of various objects around the user. In block 830, object recognition devices 708a-708n may crawl through these collected points and recognize one or more objects using the map database. This information may then be transmitted to the user's individual wearable system in block 840, and a desired virtual scene may be displayed to the user in block 850, as appropriate. For example, a desired virtual scene (e.g., a user in CA) may be displayed in an appropriate orientation, position, etc., in relation to the user's various objects and other surroundings in New York.

[0086] Figure 9 is a block diagram of another embodiment of a wearable system. In this embodiment, the wearable system 900 includes a map, which may include map data about the world. The map may reside partially locally on the wearable system, or partially in a networked storage location (e.g., within a cloud system) accessible by a wired or wireless network. An attitude process 910 may run on a wearable computing architecture (e.g., a processing module 260 or a controller 460) and utilize data from the map to determine the position and orientation of the wearable computing hardware or the user. The attitude data may be calculated from data collected on the fly as the user experiences the system and operates within its world. The data may include images of objects in a real or virtual environment, data from sensors (generally including accelerometer and gyroscope components, such as an inertial measurement unit), and surface information.

[0087] A rough point representation may be the output of a simultaneous location identification and mapping (SLAM or V-SLAM, referring to configurations where the input is image / visual only) process. The system can be configured to find not only the locations of various components in the world, but also what the world is made of. A posture may be a building block that achieves many goals, including filling in a map and using data from the map.

[0088] In one embodiment, the approximate point location may not be entirely accurate in itself, and further information may be required to generate a multi-focus AR, VR, or MR experience. A dense representation, generally referring to depth map information, may be used, at least partially, to fill this gap. Such information may be calculated from a process referred to as stereoscopic viewing 940, where the depth information is determined using techniques such as triangulation or time-of-flight sensing. Image information and active patterns (such as infrared patterns generated using an active projector) may serve as input to the stereoscopic viewing process 940. A significant amount of depth map information may be fused together, some of which may be summarized using surface representations. For example, mathematically definable surfaces may be an efficient (e.g., compared to large point clouds) and applicable input to other processing devices such as a game engine. Thus, the output of the stereoscopic viewing process (e.g., depth map) 940 may be combined in the fusion process 930. The orientation may also be an input to the fusion process 930, and the output of the fusion 930 becomes an input to fill the map process 920. Subsurfaces may connect with each other in topographic mapping, etc., to form a larger surface, and the map becomes a large-scale hybrid of points and surfaces.

[0089] Various inputs may be used to resolve various aspects in the mixed reality process 960. For example, in the embodiment depicted in Figure 9, game parameters may be inputs for determining that the system's user is playing a monster battle game with one or more monsters in various locations, that the monsters are dead, that they are running away under various conditions (such as when the user shoots the monsters), and that there are walls or other objects and equivalents in various locations. A world map may include information about the locations where such objects exist relative to each other, which is another useful input to mixed reality. Attitudes to the world are also inputs and play an important role for almost any interactive system.

[0090] User control or input is another input to the wearable system 900. As described herein, user input can include visual input, gestures, totems, audio input, sensory input, etc. For example, to move around or play a game, the user may need to command the wearable system 900 about what they want to do. There are various forms of user control that can be utilized, not just moving around in space on their own. In one embodiment, an object such as a totem (e.g., a user input device) or a toy gun may be held by the user and tracked by the system. The system would preferably be configured to know that the user is holding the item and to understand the type of interaction the user is having with the item (for example, if the totem or object is a gun, the system may be configured to understand not only its location and orientation, but also whether the user is clicking a trigger or other sensing button or element, which may be equipped with sensors such as an IMU that can help determine what is happening even when such activity is not within the field of view of any camera).

[0091] Hand gesture tracking or recognition may also provide input information. The wearable system 900 may be configured to track and interpret hand gestures for button presses, left or right gestures, stop gestures, grips, holds, etc. For example, in one configuration, the user may want to flip through email or a calendar in a non-gaming environment, or to "fist bump" with another person or performer. The wearable system 900 may be configured to take advantage of a minimum amount of hand gestures, which may or may not be dynamic. For example, the gestures may be simple static gestures, such as spreading the hand to indicate stop, raising the thumb to indicate OK, lowering the thumb to indicate not OK, or flipping the hand left or right or up or down to indicate a directional command.

[0092] Eye tracking is another input (for example, tracking where the user is looking, controlling display technology, and rendering at a specific depth or range). In one embodiment, eye convergence and divergence may be determined using triangulation, and then accommodation may be determined using a convergence / divergence / accommodation model developed for that particular person.

[0093] Regarding the camera system, the exemplary wearable system 900 shown in Figure 9 may include three pairs of cameras, namely a relative wide-field-of-view (FOV) or passive SLAM pair of cameras arranged on either side of the user's face, and a different pair of cameras oriented in front of the user to handle the stereoscopic imaging process 940 and to capture hand gestures and totem / object trajectories in front of the user's face. The FOV camera and the pair of cameras for the stereoscopic process 940 may be part of an outward-facing imaging system 464 (shown in Figure 4). The wearable system 900 may also include an eye-tracking camera oriented toward the user's eye (which may be part of an inward-facing imaging system 462 (shown in Figure 4)) to triangulate eye vectors and other information. The wearable system 900 may also include one or more textured light projectors (such as infrared (IR) projectors) to bring texture into the scene.

[0094] Figure 10 is a process flow diagram of an embodiment of method 1000 for determining user input to a wearable system. In this embodiment, the user may interact with a totem. The user may have multiple totems. For example, the user may have one designated totem for a social media application, another totem for playing a game, etc. In block 1010, the wearable system may detect the movement of the totem. The movement of the totem may be perceived through an outward-facing system or detected through sensors (e.g., tactile gloves, image sensors, hand-tracking devices, eye-tracking cameras, head-position sensors, etc.).

[0095] Based at least partially on detected gestures, eye postures, head postures, or inputs through the totem, the wearable system detects the position, orientation, and / or movement of the totem (or the user's eyes or head or gestures) relative to a reference frame in block 1020. The reference frame may be a set of map points on which the wearable system translates the movement of the totem (or user) into actions or commands. In block 1030, the user's interaction with the totem is mapped. Based on the mapping of the user interaction to the reference frame 1020, the system determines the user input in block 1040.

[0096] For example, the user may move a totem or physical object back and forth, turn a virtual page, move to the next page, or move from one user interface (UI) display screen to another UI screen. In another embodiment, the user may move their head or eyes to view different real or virtual objects within the user's FOR. If the user's gaze is directed at a particular real or virtual object for longer than a threshold time, that real or virtual object may be selected as user input. In some implementations, the convergence and divergence movements of the user's eyes can be tracked, and a near-aft accommodation / convergence and divergence motion model can be used to determine the near-aft accommodation state of the user's eyes, providing information about the depth plane in which the user is focused. In some implementations, the wearable system can use raycasting techniques to determine real or virtual objects aligned with the direction of the user's head or eye posture. In various implementations, the raycasting technique may include casting a narrow beam of light with virtually no width, or casting a beam of light with substantial width (e.g., a cone or frustum of a cone).

[0097] The user interface may be projected by a display system as described herein (e.g., display 220 in Figure 2). It may also be displayed using various other techniques, such as one or more projectors. The projectors may project images onto a physical object such as a canvas or a sphere. Interaction with the user interface may be tracked using one or more cameras outside or within the system (e.g., using an inward-facing imaging system 462 or an outward-facing imaging system 464).

[0098] Figure 11 is a process flow diagram of an embodiment of Method 1100 for interacting with a virtual user interface. Method 1100 may be performed by a wearable system as described herein.

[0099] In block 1110, the wearable system may identify a specific UI. The type of UI may be given by the user. The wearable system may identify that a particular UI needs to be populated based on user input (e.g., gestures, visual data, audio data, sensory data, direct commands, etc.). In block 1120, the wearable system may generate data for a virtual UI. For example, data associated with the UI's boundaries, general structure, shape, etc. may be generated. In addition, the wearable system may determine the map coordinates of the user's physical location so that the wearable system can display the UI in relation to the user's physical location. For example, if the UI is body-centered, the wearable system may determine the coordinates of the user's physical standing position, head posture, or eye posture so that a ring UI can be displayed around the user, or a planar UI can be displayed on a wall or in front of the user. If the UI is hand-centered, the map coordinates of the user's hand may be determined. These map points may be derived through an FOV camera, data received through sensory input, or any other type of collected data.

[0100] In block 1130, the wearable system may send data from the cloud to the display, or the data may be sent from a local database to the display component. In block 1140, the UI is displayed to the user based on the transmitted data. For example, a light field display may project the virtual UI into one or both of the user's eyes. Once the virtual UI is generated, in block 1150, the wearable system may simply wait for a command from the user and generate more virtual content on the virtual UI. For example, the UI may be a body-centered ring around the user's body. The wearable system may then wait for a command (gesture, head or eye movement, input from a user input device, etc.), and if recognized (block 1160), the virtual content associated with the command may be displayed to the user (block 1170). In one embodiment, the wearable system may wait for a hand gesture from the user before mixing multiple stem tracks.

[0101] Additional embodiments of AR systems, UI, and user experience (UX) are described in U.S. Patent Publication 2015 / 0016777, which is incorporated herein in whole by reference.

[0102] (Exemplary objects within the eye-moving field of view (FOR) and field of view (FOV)) Figure 12 schematically illustrates embodiments of virtual objects in the field of view (FOV) and virtual objects in the eye-moving field of view (FOR). As discussed with reference to Figure 4, FOR comprises a portion of the user's surrounding environment that is perceptible to the user via the wearable system. In Figure 12, FOR 1200 may contain a group of objects (e.g., 1210, 1220, 1230, 1242, and 1244) that can be perceived by the user via the wearable system. Objects in the user's FOR 1200 may be virtual and / or physical objects. For example, the user's FOR 1200 may include physical objects such as a chair, sofa, and wall. Virtual objects may include, for example, operating system objects such as a trash can for deleted files, a terminal for entering commands, a file manager for accessing files or directories, icons, menus, applications for audio or video streaming, and notifications from the operating system. Virtual objects may also include objects within applications, such as avatars, virtual objects in games, graphics, or images. Some virtual objects can be both operating system objects and application objects. In some embodiments, a wearable system can add virtual elements to existing physical objects. For example, a wearable system may add a virtual menu associated with a television in a room, which may give the user options to turn on the television or change channels using the wearable system.

[0103] A virtual object may be a three-dimensional (3D), two-dimensional (2D), or one-dimensional (1D) object. For example, as schematically illustrated in Figure 16, the virtual object may be a 3D coffee mug 1636 (which may represent virtual controls for a physical coffee maker). The virtual object may also be a 2D graphical representation of a wall clock 1634 (which displays the user's current time). In some implementations, one or more virtual objects may be displayed within (or associated with) another virtual object. For example, referring to Figure 13, the virtual coffee mug 1636 is shown inside the user interface plane 1514, but the virtual coffee mug appears as 3D within this 2D planar virtual space.

[0104] Objects within the user's FOR can be part of a world map, as illustrated with reference to Figure 9. Data associated with objects (e.g., location, semantic information, properties, etc.) can be stored in various data structures, such as arrays, lists, trees, hashes, and graphs. The index of each stored object may be determined, where applicable, by the object's location. For example, the data structure may index objects by a single coordinate, such as the distance of the object from a reference point (e.g., distance to the left or right of the reference point, distance from the top or bottom of the reference point, or depth from the reference point). The reference point may be determined based on the user's position (e.g., the user's head position). The reference point may also be determined based on the positions of virtual or physical objects (e.g., target interactable objects) in the user's environment. Thus, the 3D space in the user's environment may be folded into a 2D user interface, and virtual objects are arranged according to their distance from the reference point.

[0105] Within FOR1200, the portion of the world perceived by the user at a given time is referred to as FOV1250 (for example, FOV1250 may encompass the portion of FOR that the user is currently looking at). In Figure 12, FOV1250 is schematically represented by the dashed line 1252. A user of a wearable system can perceive multiple objects within FOV1250, such as object 1242, object 1244, and parts of object 1230. FOV can depend on the size or optical properties of the wearable device's display. For example, an AR display may include optics that provide only AR functionality when the user looks through a specific portion of the display. FOV1250 may correspond to the solid angle perceptible to the user when viewed through an AR display, such as a stacked waveguide assembly 480 (Figure 4) or a planar waveguide 600 (Figure 6).

[0106] As the user's posture (e.g., head posture or eye posture) changes, the FOV 1250 also changes accordingly, and objects within the FOV 1250 may also change. For example, map 1210 is initially outside the user's FOV in Figure 12. If the user looks towards map 1210, map 1210 may move into the user's FOV 1250, and (e.g.) object 1230 may move outside the user's FOV 1250. As will be described herein, the wearable system may track objects within FOR 1200 and objects within FOV 1250.

[0107] (Example of an interactive object) In Figure 12, the user can interact with a subset of objects within their FOR1200 environment. This subset of objects may sometimes be referred to as interactable objects. In some implementations, interactable objects may encompass all objects (virtual and physical) in the user's environment, while in other implementations, interactable objects may include only a portion of the objects in the user's environment.

[0108] A wearable system can identify subgroups of interactable objects (e.g., 1242, 1244, and 1230) within the user's FOV. These subgroups of interactable objects within the FOV are sometimes referred to as selectable objects, as the user is currently perceiving them and can select them (e.g., move them, activate them, obtain information about them, etc.). As discussed herein, the user's FOV can change as the user moves their body, head, or eyes. Generally, some objects will remain within the FOV, some will move from inside to outside the FOV (no longer selectable), and other objects that were outside the FOV will move inside (become selectable). Accordingly, the wearable system can update the subgroups of interactable objects within the FOV based on the user's body, head, or eye posture.

[0109] A wearable system may identify target interactable objects within the user's field of view (FOV). Target interactable objects may be objects the user wishes to interact with or objects the wearable system anticipates the user will interact with (e.g., interactable objects the user is looking at or the nearest interactable object to the center of the user's FOV). Target interactable objects can be identified using various rules, such as the object's location, the user's preferences, or the user's orientation. For example, a wearable system might select the object closest to the center of the FOV as a target interactable object. The wearable system might also select the leftmost or rightmost object within the user's FOV as a target interactable object. In another embodiment, the wearable system may determine the user's line of sight direction using an inward-facing imaging system 462 (shown in Figure 4), either alone or in combination with an IMU. The wearable system can then identify objects that collide with the user's line of sight direction as target interactable objects.

[0110] In some implementations, the AR system can automatically orient a target interactable object so that its normal vector faces the user. For example, a virtual TV screen might initially face upwards towards the ceiling of the room. Once the AR system determines that the user is looking at the virtual TV screen, it can automatically rotate the virtual TV screen so that it faces the user.

[0111] (Example of a focus indicator) A wearable system may assign a focus indicator to a target interactable object so that the user can more easily perceive the target interactable object. The focus indicator may be displayed to the user. For example, the focus indicator may have a halo, color, a change in perceived size or depth (e.g., making the target object appear closer and / or larger when selected), or other visual effects to attract the user's attention. The focus indicator may also include audible or tactile effects such as vibration, ringtone, or beep.

[0112] In some embodiments, the wearable system may first identify an object as a target interactable object based on the rules described herein, and then change the target interactable object to another object based on a change in the user's posture. As a result, the focus indicator may move from one object to another as the user changes their posture.

[0113] The wearable system may also display a cursor corresponding to the user's current position. The cursor may take various shapes, such as a geometric cone, a beam of light, a crosshair, an arrow, an oval, a circle, a polygon, or other 1D, 2D, or 3D shapes. The cursor may be presented in the same form as the focus indicator. For example, the cursor may have the same visual, audio, or tactile effects as the focus indicator. In one embodiment, the cursor may be a crosshair corresponding to the user's head position. In another embodiment, the cursor may have the shape of an arrow corresponding to the current position associated with a user input device. As the user changes their posture or activates the user input device, the cursor may move accordingly. As the user moves, the cursor may point to one or more objects or empty space in the user's environment. For example, referring to Figure 16, the AR system can move the cursor on the virtual user interface 1514 from position 1620 to position 1624 or from position 1622 to position 1620, etc.

[0114] A wearable system may present a cursor in addition to, or as an alternative to, a focus indicator. For example, in Figure 15, the wearable system may display a crosshair (which may correspond to the direction of the user's gaze) or provide a light blue halo as a focus indicator, or both, on the virtual object 1514. In some implementations, the cursor is an embodiment of the focus indicator. For example, when a user is looking at a virtual object, the wearable system may present a halo around the virtual object, a crosshair object, or an arrow on the object. These visual indicators may represent both a target object that the user is interested in interacting with and the user's current position.

[0115] (Exemplary interaction with interactable objects) The user can interact with interactable objects within their FOR1200, particularly interactable objects within their current FOV1250, through the wearable system. For example, virtual object 1230 may be a graph showing changes in stock prices over time. By selecting virtual object 1230, the user can interact with it to perform actions such as obtaining stock price information, buying or selling stocks, or obtaining information about a company. To facilitate these interactions, the wearable system may display menus, toolbars, etc., associated with the virtual object, enabling the user to perform various actions (e.g., obtaining stock price information).

[0116] Users can interact with objects within their FOV using a variety of techniques, such as selecting an object, moving an object, opening a menu or toolbar associated with an object, or selecting a new set of selectable objects. Users may also interact with objects using hand gestures, and activate user input devices (see, for example, user input device 466 in Figure 4) by, for example, clicking a mouse, tapping on a touchpad, swiping on a touchscreen, waving at or touching a capacitive button, pressing a key on a keyboard or game controller (e.g., a 5-way D-pad), pointing a joystick, wand, or totem towards an object, pressing a button on a remote control, or other interactions with user input devices. Users may also interact with interactable objects using their head, eyes, hands, feet, or other body postures, such as, for example, gazing at an object over a period of time, pointing with their arm, tapping their foot, or blinking their eyes a certain number of times over a threshold time interval. These hand gestures and user postures on a user input device can trigger selection events in an AR system, such as user interface actions (e.g., displaying a menu associated with a target interactable object, or performing game actions on an avatar in a game).

[0117] In response to the initiation of a selection event, the AR system may assign the focus indicator to a target interactable object within the user's FOV using the rules described herein. For example, in Figure 12, the AR system may assign the focus indicator to object 1244 because it is closest to the center of the FOV.

[0118] During a selection event, the user can change the target interactable object using various hand gestures described herein. For example, in Figure 12, the user can swipe left on the touchscreen, which causes the AR system to change the target interactable object from object 1244 to object 1230. The AR system can also, accordingly, shift the visible focus indicator from object 1244 to object 1230.

[0119] In some implementations, hand gestures may cause the AR system to update the list of selectable objects within the user's field of view (FOV). For example, in Figure 12, when the user swipes to the right, the AR system may move object 1210 into the user's FOV and object 1230 out of the user's FOV. The AR system may also update the target interactable object based on the new group of selectable objects. For example, after object 1210 has been moved into the FOV, the system may change the target interactable object from object 1244 to object 1242.

[0120] The user can confirm the selection of a target interactable object using hand gestures or postures discussed herein. The user's action to confirm the selection of a target interactable object may be the same as or different from the action used to initiate the selection event. The AR system may change the focus indicator when the user confirms the selection, for example, by changing the color, brightness, or shape of the focus indicator.

[0121] A user can select a series of user interface actions on a target interactable object. These actions may sometimes be referred to as interaction events. Interaction events may include, for example, resizing the interactable object, displaying a menu on the interactable object, browsing the menu, selecting an item from the menu, searching for an item, playing a game, watching a video, conducting a conference call, or previewing the target interactable object. Interaction events may occur in parallel with or consecutively with selection events. In some implementations, interaction events may be part of selection events.

[0122] In some embodiments, once a selection event is initiated, the wearable system may "lock" the user's FOV so that the wearable system will stop updating the group of selectable objects within the user's FOV, even if the user's FOV changes after the selection event has been initiated. In some implementations, the user can still move the focus indicator between selectable objects within the user's FOV through the operation or change of posture of the user input device.

[0123] A selection event can be terminated by user input or other interaction with the wearable system. For example, a selection event can be terminated by confirming the selection of a target interactable object, initiating an interaction event, activating a user input device to terminate the selection event, or determining a change in head or body posture that has the effect of terminating the selection event.

[0124] (An example of selecting a virtual object in 3D space using hand gestures) The user may target and select an interactable object by activating a user input device. Figure 13A shows an example of selecting an interactable object using a touch gesture on the touchscreen 1310 of the user input device 1300. The user input device may be an embodiment of the user input device 466 shown in Figure 4. The touch gesture can trigger the wearable system and assign a focus indicator to a target interactable object within the user's FOV. The touch gesture can also, alone or in combination, cause the wearable system to initiate a selection event, initiate an interaction event, terminate a selection event, terminate an interaction event, confirm the selection of a target interactable object, and so on.

[0125] Figure 13B shows an example of filtering selectable objects using hand gestures on a user input device. The user may swipe along a path on the user input device 1300. For example, the user may swipe along a path to the right on the touchscreen 1310, as indicated by arrow 1314 in Figure 13B. Any type of path can be used (e.g., horizontal, vertical, diagonal, or other trajectories relative to the input device), or any type of direction can be used (e.g., left or right, up or down, etc.).

[0126] A swipe gesture can cause a wearable system to move a visible focus indicator from one object to another. Referring to the embodiment shown in Figure 12, if the user swipes to the right (as shown in the embodiment in Figure 13B), the AR system can move the focus indicator from object 1244 to object 1242. In some embodiments, a swipe gesture can cause the AR system to update the list of selectable objects within the user's FOV. For example, if the user swipes to the right, the AR system can move object 1210 into the user's FOV and object 1230 out of the user's FOV. The object receiving the visible focus indicator can also be updated accordingly (e.g., from object 1244 to 1242).

[0127] Swipe gestures may be used in combination with touch gestures (described with reference to Figure 13A) and head poses (described with reference to Figure 14) to filter and select 3D virtual objects within the FOR or FOV. As discussed herein, the user may also use other gestures or movements to change the set of selectable objects and modify the target interactable object.

[0128] (An example of selecting virtual objects in 3D space using head posture) Figure 14 shows an example of a coordinate system for head pose. The head 1410 may have multiple degrees of freedom. As the head 1410 moves in different directions, the head pose will change relative to the natural resting direction 1420. The coordinate system in Figure 14 shows three angular degrees of freedom (e.g., yaw, pitch, and roll) that can be used to measure the head pose relative to the natural resting state of the head 1420. As illustrated in Figure 14, the head 1410 can tilt forward and backward (e.g., pitch), rotate left and right (e.g., yaw), and tilt laterally (e.g., roll). In other implementations, other techniques or angular representations for measuring head pose can also be used, e.g., any other type of Euler angular system.

[0129] A wearable system can determine the user's head posture using various sensors described herein (see Figures 2, 4, and 7, etc.). For example, a wearable system can calculate the user's head posture or eye posture using an IMU or an inward-facing imaging system. Using the data acquired by these sensors, the wearable system can identify target interactable objects. For example, a target interactable object may be an object that collides with the direction of the user's line of sight. The wearable system can identify a target interactable object and assign a visible focus indicator to the target interactable object based on the line of sight extending in a certain direction (for example, the focus indicator is assigned to the target object if the user has been looking at the object for longer than a threshold time).

[0130] A wearable system can determine and update the target interactable object based on changes in head posture, such as roll, yaw, or pitch, between selected events. For example, referring to Figure 12, if the user rotates their head 1410 to the left, the target interactable object may be updated from object 1244 to a neighboring object such as object 1230. In some implementations, the wearable system can reflect this update by shifting the focus indicator from object 1244 to object 1230.

[0131] The wearable system can also update the list of selectable objects within the user's FOV based on changes in head posture. For example, the user may rotate their head 1410 to the right, which can cause the wearable system to move object 1230 out of the user's FOV and object 1210 into the user's FOV 1250. Target interactable objects can also be updated accordingly using the rules described herein.

[0132] The user may switch between multiple planes using various changes in head or eye posture. The user may also use variations in head or eye posture to initiate a selection event, confirm the selection of a target interactable object, initiate an interaction event, interact with the target interactable object during an interaction event, end a selection event, or perform other interactions with the user interface.

[0133] In some implementations, a wearable system can associate a depth plane with the user's head movement so that the user can only interact with virtual objects within that depth plane. For example, a wearable system can set a cursor (e.g., a crosshair) corresponding to the user's head position in a depth plane. As a result, as the user moves their head, the cursor moves within the set depth plane, allowing the user to select between objects in the set depth plane, even if other virtual objects reside in different depth planes. In some implementations, the depth plane may be associated with a virtual user interface. The wearable system can set the crosshair to the depth plane of a target virtual user interface so that the user can interact with virtual objects within that target user interface plane.

[0134] (Exemplary interaction using a combination of head posture and hand gestures on a user input device) The user can also select virtual objects using a combination of the user's posture and hand gestures. For example, the user may perceive a group of virtual objects within the user's field of view (FOV). This group of virtual objects may be embodiments of selectable objects as described herein. The group of virtual objects may be presented using a light field display, as described with reference to Figures 2, 4-6. The light field display can project virtual objects in different depth planes to the user such that some virtual objects may appear in front of other virtual objects.

[0135] A wearable system can maintain an array of virtual objects currently within the user's field of view (FOV). The wearable system may use the location of the virtual objects within the user's environment as an array index for the virtual objects. For example, the wearable system can use the y-value of a virtual object in xyz coordinates (e.g., xyz coordinates shown in Figure 6) as an array index for the object. In other embodiments, the wearable system can determine the array index of a virtual object using the x-value or z-value, either alone or in combination with the y-value.

[0136] As the user moves their head or the direction of their gaze, the group of virtual objects that appear within the user's field of view (FOV) may change. The wearable system can also update the array accordingly. In some implementations, when the user initiates a selection event, the wearable system can use the array to maintain virtual objects within the user's FOR and identify the group of virtual objects within the FOV.

[0137] A user can initiate a selection event on a virtual object within their field of view (FOV) by activating a user input device. Upon initiation of the selection event, the wearable system can present a user interface that includes all (or some) of the virtual objects within the user's FOV. In some implementations, the wearable system may display "hidden virtual objects," such as virtual user interface menus, or information about the virtual objects. These "hidden virtual objects" may become perceptible upon initiation of the selection event, but are hidden prior to or after the event.

[0138] In some embodiments, in response to the initiation of a selection event, the wearable system may change the position of virtual objects within the user's field of view (FOV). For example, the wearable system may move distant objects closer to the user or nearby objects further away from the user so that all virtual objects appear on substantially the same depth plane. In some implementations, the wearable system may change (increase or decrease) the size of virtual objects so that they can fit the size of the user's FOV. In addition, or alternatively, the wearable system may show the user a portion of the virtual objects (e.g., an icon instead of the content of a virtual email application).

[0139] The wearable system can also group virtual objects within the user's field of view (FOV) into multiple depth planes in response to the initiation of a selection event. Each depth plane may be associated with a virtual user interface.

[0140] A wearable system can arrange one or more virtual objects on a user interface using an array index of virtual objects. For example, a wearable system can group virtual objects whose y-axis values ​​are within a certain range on the same user interface. In addition, or alternatively, a wearable system can arrange virtual objects on a depth plane based on the user's orientation. For example, if there may be multiple virtual objects in the direction of the user's line of sight, and they are in different depth planes within the user's environment, the wearable system can, in response to the initiation of a selection event, present these virtual objects inside the user's FOV while placing other virtual objects outside the user's FOV. The user can move virtual objects inside and outside the FOV using the techniques described with reference to Figures 12 and 13.

[0141] The wearable system can identify target interactable objects and present a focus indicator indicating the target interactable object. The wearable system can reorganize virtual objects in the vicinity of the target interactable object and present the reorganized virtual objects within the user's FOV. For example, referring to Figure 12, the wearable system can identify a group of virtual objects in the vicinity of a target interactable object based on their distance from a reference position (such as the location of the target interactable object). The wearable system can reorganize the positions of these virtual objects based on values ​​in array index xyz coordinates (shown in Figure 6), or their distance from the target interactable object, etc. As shown in Figure 12, objects 1242, 1244, and 1230 may have different initial positions within the user's FOR. For example, object 1242 may be positioned higher than object 1244 (e.g., closer to the ceiling of the user's room) or further from the user than object 1244. The initial position of object 1230 may be lower than object 1244 within the user's FOR (e.g., closer to the floor of the user's room). When the wearable system identifies object 1244 as a target interactable object, it can "fold" the user's 3D space into a 2D user interface based on its y-axis value, with objects having larger y-axis values ​​positioned to the left of the user's FOV. Accordingly, within FOV 1250, virtual object 1242 appears to the left of virtual object 1230, and to the left of virtual object 1244. In other embodiments, different techniques for rearranging virtual objects may be used. For example, FOV 1250 may represent a 2D projection of virtual objects in 3D space. As illustrated in Figure 12, the user can use hand gestures to move the focus indicator between objects within the FOV.The user can also use hand gestures to change the target interactable object, which may cause the wearable system to present a different set of virtual objects in the FOV based on the reorganization of virtual objects in the vicinity of the new target interactable object. If the wearable system is configured to present multiple user interfaces in different depth planes, the user can also use hand gestures to move the focus indicator between the multiple depth planes and switch between user interfaces. In some embodiments, the wearable system may restore the position of a virtual object to its original position prior to reorganization when the virtual object is no longer in the user's FOV. In some situations, the virtual user interface presenting the reorganized virtual object may be generated after the user initiates a selection event on the target interactable object. Additional details regarding interaction with multiple user interfaces are further described with reference to Figure 15.

[0142] (Examples of user interaction based on context information) A wearable system can automatically select or recommend a mode of user interaction (e.g., posture or hand gestures on a user input device) based on contextual information. Contextual information may include the type of object (e.g., physical or virtual), the layout of objects (e.g., object density, location, and size), user characteristics, or the user's current interactions, combinations, or equivalents with objects in the environment. For example, during raycasting (as illustrated with reference to Figure 10), a wearable system may detect that the user is viewing multiple virtual objects located in close proximity to each other. The wearable system can calculate the density of virtual objects within the user's field of view (FOV). If the density exceeds a certain threshold, the wearable system can recommend that the user switch modes of user interaction. For example, if the density exceeds a certain threshold (indicating that objects are located very close to each other), the wearable system can switch the mode of user interaction from head posture to hand gestures on a user input device to allow for more precise interaction with the objects. In another embodiment, when density drops below a certain threshold (indicating that objects are moving away from each other), the wearable system can switch the mode of user interaction from hand gestures on a user input device to head posture. These implementations may be particularly advantageous because head positioning can be difficult to control precisely, and users can become fatigued when trying to precisely position their heads to interact with densely clustered objects. Hand gestures on a user input device, on the other hand, can provide more refined control of the user's position, but can become fatigued when users need to move their hands over long distances to select objects located at low density.

[0143] Figures 15 and 16 provide examples of changing user interaction modes based on contextual information. Figure 15 illustrates an example of interaction with an interactable object using head posture. Figure 16 illustrates an example of interaction with an interactable object using hand gestures on a user input device. The user input device 1610 shown in Figure 16 may be an embodiment of the user input device 466 described in Figure 4.

[0144] In Figure 15, the user's FOR includes interactable objects such as virtual user interfaces 1512, 1514, and 1516. In some implementations, virtual user interfaces 1512, 1514, and 1516 may be planar objects that contain other virtual objects (1D, 2D, or 3D) within the virtual user interface plane. Virtual user interfaces 1512, 1514, and 1516 are large in size and are not densely located adjacent to each other. As a result, the wearable system may determine that head movement may be the optimal mode of user interaction, as the user does not need to move long distances on the user input device to select a virtual user interface. The wearable system may use raycasting techniques to identify that the user is currently looking at virtual user interface 1514. The user may initiate a selection event on virtual user interface 1514 and interact with objects within user interface 1514. As shown in Figure 16, the virtual user interface 1514 can include multiple virtual objects, such as a virtual TV screen 1632, a virtual coffee cup 1636, a virtual wall clock 1634, a camera application 1652, a weather application 1638, and a music application.

[0145] (An example of recommending user interaction modes based on context information) As an example, in Figure 16, the wearable system can determine the relative positions between a virtual TV 1632, a virtual wall clock 1634, a virtual coffee mug 1636, and a weather application 1638. Since these four objects are close to each other, once the wearable system determines that the user's current position is at position 1620, the wearable system may present the user with the option of whether they wish to switch from head posture to hand control.

[0146] On the other hand, object 1652 (camera application) does not have any other objects in its vicinity. Accordingly, the wearable system does not need to provide an option to interact with object 1652 using a user input device. However, the user may still interact with object 1652 using postures such as head posture and eye posture.

[0147] The user can confirm a switch in the mode of user interaction by using the user input device or by changing their body posture (e.g., nodding their head). Once the mode of user interaction is switched to the user input device, the user can activate the user input device 1610 and interact with the virtual object. For example, the user can swipe along a path on the user input device 1610, which moves the cursor from position 1620 to position 1624. Similarly, the user can activate the user input device 1610, which moves the cursor (which may be in the shape of an arrow) from position 1620 to 1622. In addition to these embodiments, the user may swipe along any type of path on the user input device 1610 (e.g., horizontal, vertical, or diagonal relative to the input device) or any type of direction (e.g., left or right, up or down, etc.).

[0148] When the cursor is at position 1624, a portion of the cursor overlaps with the virtual TV screen 1632. The user may activate the user input device 1610 (for example, by clicking the touchscreen) to select the virtual TV screen 1632. When the wearable system receives a selection of the virtual TV screen 1632, the wearable system may display one or more virtual menus associated with the TV screen (e.g., virtual menus 1642a and 1642b). For example, the virtual menus may include options such as adjusting the tone, selecting a video application (e.g., a movie or TV streaming service), or selecting to start a conference call.

[0149] (An example of automatic switching of user interaction modes based on context information) The wearable system can also automatically switch the mode of user interaction. For example, the mode of user interaction may be set to head posture when the user selects between virtual user interface planes 1512, 1514, and 1516. Once the user has selected a virtual user interface plane, the mode of user interaction may automatically change to hand gestures on the user input device (as shown in Figure 16). In another embodiment, the mode of user interaction may be set to body posture when the density of objects is sufficiently low or when the layout of objects meets certain criteria (e.g., when there is no occlusion between objects). The wearable system can automatically change the mode of user interaction to hand gestures on the user input device when the objects are located at a high density or when the layout of objects no longer meets the criteria (e.g., when one object occludes another).

[0150] The wearable system may consider, in addition to or alternatively to, the relative positions of objects in 2D space, the relative positions of objects in 3D space. For example, user interface 1514 may be a 3D user interface instead of a 2D user interface. As shown in Figure 16, the weather application 1638 may be located in a depth plane further away from the user than the coffee mug application 1636. When the user's line of sight is at position 1620, the wearable system may detect two virtual objects (virtual wall clock 1634 and music application 1654) that intersect with the user's line of sight, even if the music application 1654 may appear further away from the virtual wall clock 1634. Based on this information, the wearable system may determine that the objects are sufficiently close to each other. Accordingly, the wearable system may automatically switch the mode of user interaction to hand gestures, or prompt the user for the option to switch to hand gestures.

[0151] (Other exemplary user interface features based on contextual information) In some implementations, a wearable system may reposition a high-density cluster of virtual objects to one or more fixed depth planes when the user switches to hand gestures. As illustrated with reference to Figures 12, 13A–13B, the user can select virtual objects within a fixed depth plane or switch depth planes using hand gestures. This implementation may be particularly advantageous because it reduces the cumbersome operation on the user input device that would result from navigating between virtual objects located in slightly different depth planes in 3D space.

[0152] A wearable system may change a focus indicator or cursor, either alone or in combination, when the mode of user interaction changes from one method to another. For example, a wearable system may change the color of the focus indicator when the user changes the mode of user interaction from head posture to hand gestures on a user input device (and vice versa). In another embodiment, the wearable system may change the appearance of the focus indicator from a crosshair shape to an arrow shape (shown in Figure 16) to indicate the option of changing input control from head posture to hand gestures on a user input device.

[0153] In some implementations, changes in the focus indicator or cursor may be used alone or in combination to indicate that an option for changing the mode of user interaction has become available. For example, while a user interacts with a group of low-density located objects using hand gestures on a user input device, the wearable system may change the appearance of the focus indicator from an arrow to a crosshair to indicate that an option for interaction using head posture is available. The user can confirm the change from hand gesture to head posture by, for example, activating the user input device (e.g., tapping the user input device) or changing their body posture (e.g., nodding their head). In another embodiment, the wearable system may provide vibration on the user input device as a focus indicator to indicate that an alternative mode of user interaction has become available.

[0154] While the embodiments are described with reference to the selection of a single object, the wearable system may be configured to identify and select multiple target objects. The AR system may be configured to perform selection events recursively on subgroups of target interactable objects. For example, the AR system may identify several target objects that collide with a virtual cone in raycasting (described in Figure 10). The target objects may include interactable objects. The AR system can zoom in on these target objects and allow the user to select within these target objects using the postures and / or hand gestures described herein. Furthermore, while the embodiments are described with reference to changes between head posture and hand gestures on a user input device, similar techniques can also be used to switch between other modes of user interaction. For example, the wearable system may employ similar techniques, alone or in combination, to change the mode of user interaction between body posture, hand gestures, head posture, foot posture, eye posture, etc.

[0155] (Examples of interaction events) After the user selects an interactable object, they can initiate an interaction event on that interactable object within their field of view (FOV). In some implementations, the virtual object may correspond to a physical object. As a result, when the user performs an interaction event on a virtual object, the virtual object may communicate with the physical object, thereby allowing the user to interact with the physical object through a virtual user interface. For example, the 3D coffee mug 1636 in Figure 16 may communicate with a coffee machine in the user's environment. The water level shown in the 3D coffee mug 1636 may represent the progress of coffee production. In one embodiment, the water level may be invisible initially because the coffee machine is idle. The user can select the 3D coffee mug 1636 and initiate an interaction event, which causes the wearable system to send a command to the coffee machine in the user's kitchen to start brewing coffee. During the brewing process, the water level in the 3D coffee mug 1636 may gradually increase as the coffee machine produces more coffee. When the coffee machine finishes brewing, the wearable system may indicate that the 3D coffee mug 1636 is full. In some implementations, the wearable system may also provide a focus indicator (such as sound or a halo) on the 3D coffee mug to indicate that coffee brewing has finished.

[0156] In another embodiment, the wearable system may present virtual content associated with physical objects in the user's environment during an interaction event. In Figure 16, the user can select a weather application 1638 and initiate an interaction event on the weather application 1638. Figure 17 illustrates an exemplary interaction event with the weather application 1638. In this embodiment, the user of the wearable system may be outside their home and can perceive physical objects such as distant storm clouds 1710 and tornadoes 1720. The user can also perceive other physical objects such as roads 1730, vehicles 1732, and buildings 1734.

[0157] The wearable system can identify the user's location by analyzing images of the user's environment. When the user initiates an interaction event on the weather application 1638 (shown in Figure 16), the wearable system can present an AR / MR scene 1700 superimposed on physical objects within the user's FOR. The wearable system can use geolocation sensors (e.g., a Global Positioning System (GPS) sensor) to determine the user's location and information about the weather near the user (e.g., the presence of storm clouds 1710 and tornadoes 1720 that the user is seeing). In some implementations, the wearable system can also use outward-facing imaging systems to acquire images of the user's environment. The wearable system can use sensors such as outward-facing imaging system 464, inward-facing imaging system 462, or IMU (described in Figure 4) alone or in combination to determine that the user is looking towards storms 1710 and tornadoes 1720.

[0158] The wearable system communicates with a network (wired or wireless) to access information about a storm and can display to the user as virtual content such as the presence and duration of a storm warning 1750, tornado speed 1754, weather forecast 1762 (e.g., temperature, probability of precipitation as a function of time), storm direction 1756 (e.g., storm location as a function of time), and expected rainfall 1766. This information can be presented to the user via text or graphics so that at least part of the information is perceived as being in or near the location of the storm cloud 1710 or tornado 1720 (e.g., virtual content may appear superimposed on physical content). For example, as shown in Figure 17, arrows 1752 indicate the direction of the storm (e.g., tornado), appear in 3D, and are superimposed on or around the tornado 1720 from the wearable system user's viewpoint. Wind speed 1754 may be presented near the direction arrows 1752. The wearable system may also provide the wearer with other information about the storm, such as temperature forecasts 1762, locations where it is currently raining or is expected to rain (e.g., indicated by a dashed line 1764 below storm clouds 1710), expected precipitation (as indicated using reference number 1766), and wind direction 1752 and speed 1754 within the storm (e.g., at different altitudes of tornadoes 1720).

[0159] The wearable system can present realistic 3D virtual content 1750, 1752, 1754, 1756, 1762, 1764, 1766 to the user of the wearable system using a light field display (shown in Figure 4-6) so that it appears at an appropriate distance from the user (e.g., on or near the storm 1710 and tornado 1720) and is appropriately sized, shaped, or scaled (e.g., as schematically shown in Figure 17).

[0160] A wearable system can also enable two or more users to interact with an interactable object. Both users may wear their own separate wearable systems (such as their head-mounted devices). Figure 18 illustrates an exemplary user experience of multiple users interacting with a 3D virtual object. In this embodiment, the user is wearing a wearable device 1852. The user can perceive virtual content 1800 through the wearable device 1852. In this embodiment, the virtual content may include an astronomical object 1810 (e.g., a star, or in other cases, a galaxy, planet, nebula, or solar system). Information about the virtual content 1800 may be displayed so as to appear on or adjacent to the virtual content. For example, the orbit 1820 of a star or planet 1830, constellations, nearby stars, etc., may be displayed in the vicinity of the astronomical object 1810. The wearable system can present a user interface to the wearer, allowing them to access a virtual menu 1832 or virtual input feature 1840, where different actions can be selected using posture or a user input device (e.g., by gesture). For example, as shown in Figure 18, the virtual menu 1832 may allow the wearer to edit / delete or save a profile of the displayed virtual content 1800. The profile may allow the wearer (or another authorized user) to access the virtual content at different times or locations. Another virtual menu may allow the user to interact with the virtual content and modify the displayed virtual content. For example, as shown in Figure 18, the user input element 1840 may allow the wearer to “add a planet” (e.g., by selecting the virtual button 1840 using a gesture such as “pressing” the wearer’s finger at the location of the virtual button 1840). After selection, the wearer may be able to access functionality (e.g., via a virtual menu, virtual dropdown box, etc.) and create a planet profile using information such as the planet’s name, diameter, temperature, or distance.After selection, additional virtual content (in this embodiment, additional planets) can be displayed to the wearer.

[0161] A wearable system can enable a user to share virtual content with others, for example, by traversing a world map of the user's environment over a network, or by communicating (or updating) virtual content between wearable systems. For example, as shown in Figure 18, another user 1850 wearing a wearable system 200 can view virtual content being shared and manipulated by the first user. Both users can enjoy an experience of interacting with each other and with the virtual content. While an astronomical object 1810 is used in this embodiment, the virtual content 1800 can be any type of content. For example, an encyclopedia may be accessed, and content on one or more subjects may be displayed and shared (along with virtual text, virtual images, sounds, etc.). A group of users sharing virtual content does not need to be physically present in the same location to view the virtual content, and many users (e.g., 2, 3, 4, 5, 10, 100, or more) can view the shared virtual content substantially simultaneously.

[0162] Examples are described with reference to virtual planetary system and weather applications, but these examples are not limiting. The techniques described herein can also be applied to other applications that present interactive virtual content within an AR / MR / VR environment. For example, a wearable system could be programmed to include a clothing shopping application. While the user is in the department store, the application can access a database associated with the department store to identify clothing information or racks. The application can then present the accessed information on virtual content superimposed on the physical clothing in the department store.

[0163] (An illustrative process for selecting virtual objects using a combination of posture and user input devices) Figure 19 illustrates an exemplary process for selecting a virtual object using a combination of posture and hand gestures on a user input device. Process 1900 can be performed by a wearable system (e.g., HMD) described herein (e.g., by one or both of processing modules 260, 270). The wearable system may include a user input device (e.g., see user input device 466 in Figure 4) and various imaging systems such as an outward-facing imaging system (e.g., see outward-facing imaging system 464 in Figure 4) and an inward-facing imaging system (e.g., see inward-facing imaging system 462 in Figure 4).

[0164] In block 1910, the wearable system can identify interactable objects within the user's FOR. These interactable objects may be stored, for example, in a remote data repository 280 (shown in Figure 2) using data structures such as arrays. The interactable objects within the user's FOR can be a subset of all objects within the user's FOR. The wearable system can determine the user's location using an outward-facing imaging system 462 and other sensors (such as an IMU and GPS), and use this location information to determine interactable objects within the user's environment.

[0165] In block 1920, the wearable system can measure the user's posture using the sensors described herein. The wearable system can identify the user's orientation using an inward-facing imaging system 462.

[0166] In block 1930, the wearable system can identify interactable objects within the user's field of view (FOV) based on the user's line of sight direction. The FOV can also use raycasting to determine interactable objects that intersect with the user's line of sight direction. In some implementations, interactable objects within the user's FOV may be referred to as selectable objects. The wearable system can use the location of the selectable objects to store them in an array. For example, the wearable system can index selectable objects (and / or interactable objects) based on x-axis values ​​(e.g., xyz coordinate reference in Figure 6). The wearable system can sort objects based on x-axis values ​​and present selectable objects from left to right within the user's FOV in a 1D or 2D view.

[0167] In some implementations, the wearable system does not need to have a separate array solely for selectable objects. In these implementations, the wearable system can identify and retrieve selectable objects from the array of interactable objects, for example, using an array index of the selectable objects (determined based on the user's FOV).

[0168] However, in some embodiments, the wearable system may maintain two arrays, one for interactable objects within the user's FOR (Foreground of Field) and the other for selectable objects within the user's FOV (Field of View). For example, the array for interactable objects may be kept in a remote data repository 280 (shown in Figure 2), while the array for selectable objects may be kept in the wearable system's local data storage. This implementation may be advantageous because the wearable system's local storage and data processing capacity may be limited, while the remote data repository may have larger data storage and more powerful data processing capacity. By keeping only a subset of all interactable objects in the wearable system's local storage, the hardware requirements of the wearable system can be reduced, and the possibility of data overflow within the wearable system can be decreased.

[0169] If no interactable objects are present within the user's field of view (FOV), the wearable system can return to block 1910. The wearable system can continuously monitor the user's posture, update the user's FOV, and continuously determine a list of interactable objects within the user's FOV.

[0170] In block 1940, the wearable system can determine whether the user wishes to initiate a selection event on an object within the user's field of view (FOV). The wearable system may make such a determination, either alone or in combination, based on various indicators, such as prolonged gazing at an object, the user's head posture (e.g., nodding), or input from a user input device. If the wearable system receives an indication that the user wishes to select an object, the wearable system may allow the user to interact with the selectable object using various hand gestures as described herein. For example, the user may swipe along a trajectory on a user input device to browse selectable objects, or click on a user input device to select a selectable object.

[0171] The selection event may initiate the wearable system to identify a target interactable object within the user's FOV (in block 1940) and assign a focus indicator to the target interactable object (in block 1950). The target interactable object may be an object on the left or right side of the FOV. The visible focus indicator may also be placed on an object at the center of the FOV when the object at the center is a target interactable object. The wearable system may also use 3D eye tracking to determine the direction of the user's eye posture and place the visible focus indicator on an object that intersects with the direction of the user's eye posture.

[0172] In block 1960, the user can move the focus indicator to another object using various gestures described herein. The target interactable object can be updated accordingly to the other object. For example, the user can move the focus indicator from its current position to a neighboring object, and as a result, the target interactable object is updated to the neighboring object.

[0173] In some embodiments, these gestures may cause the wearable system to move a virtual object along a path within the user's field of view (instead of moving the visible focus indicator itself). For example, referring to Figure 12, when the user swipes left on the touchpad, it may have the effect of moving object 1230 entirely into the field of view, while shifting the position of object 1242 to the left.

[0174] Figure 20 illustrates an exemplary process for interacting with a virtual object using a combination of posture and hand gestures on a user input device. Process 2000 can be performed by a wearable system described herein.

[0175] In block 2010, the wearable system can determine a group of interactable objects within the user's FOR. This group of interactable objects may be a subset of objects in the user's environment. Interactable objects may be virtual and / or physical objects. In some embodiments, the AR system may add virtual elements to existing physical objects. For example, the AR system may add a virtual menu to a television in a room, which may give the user the option to turn on the television using the AR system.

[0176] As illustrated with reference to Figure 12, the AR system may store interactable objects and information associated with them in various data structures. The location of an object may be used as an index for storing information associated with the object.

[0177] In Block 2020, the wearable system can determine the user's posture. The user's posture may be the posture of the head, eyes, feet, or other body parts, either individually or in combination. The wearable system can determine the user's posture using various sensors shown in Figure 4, such as an inward-facing imaging system 462, input received on a user input device 466, or an outward-facing imaging system 464.

[0178] In block 2030, the wearable system can determine the user's field of view (FOV) based on the user's posture. The FOV may comprise a portion of the FOR (Field of View) perceived by the user at a given time. Based on the user's FOV, in block 2040, the AR system can determine a subgroup of interactable objects within the user's FOV. This subgroup of interactable objects is sometimes also referred to as selectable objects. As the user's FOV changes, the selectable objects within the user's FOV also change.

[0179] In block 2050, the wearable system receives a selection of a target interactable object from a subgroup of interactable objects. The wearable system can initially select a target interactable object based on various rules (such as the location of the target interactable object relative to the user's FOV), as illustrated with reference to Figure 12. The AR system can assign a focus indicator to the target interactable object. The visible focus indicator may move from one object to another as the user's FOV changes.

[0180] In some embodiments, the wearable system can identify a target interactable object after the user activates a user input device. The user can activate the user input device using various hand gestures, as described with reference to Figures 12-13. These hand gestures can trigger the wearable system, which can assign a focus indicator to a target interactable object within the user's FOV. In some implementations, once the AR system receives a selection of a target interactable object from the user, the AR system may stop updating the group of selectable objects within the user's FOV while the user's FOV changes. The user can still browse the interactable objects or move the visible focus indicator from one object to another within their FOV.

[0181] In block 2070, the user may decide to initiate a selection event on a target interactable object. A selection event can be initiated using the postures and gestures described herein. In some embodiments, initiating a selection event may trigger a wearable system to assign a visible focus indicator to the target interactable object. The wearable system may stop updating the group of selectable objects within the user's FOV, even if the user's FOV may change with changes in the user's posture. The user may move the visible focus indicator from one object to another within the user's FOV using the gestures described herein. For example, the user may swipe along a trajectory (left and right, etc.) on a touchpad, which can cause the wearable system to move the visible focus indicator from one object to its nearest neighbor.

[0182] In some implementations, the user may initiate an interaction event during or after a selection event. The interaction event may also be part of the selection event. For example, as illustrated with reference to Figure 12, an interaction event may include resizing an interactable object, displaying a menu for an interactable object, browsing the menu, selecting an item on the menu, searching for an item, playing a game, watching a video, conducting a conference call, and so on. The user may participate in an interaction event using the various poses and gestures described herein. The user may also use the poses and gestures discussed herein to confirm the selection of a target interactable object.

[0183] (An illustrative process of interacting with an object based on contextual information) Figure 21 illustrates an exemplary process for switching input control from head posture to hand gestures based on contextual information. Process 2100 can be performed by a wearable system described herein (for example, by one or both of processing modules 260, 270).

[0184] The wearable system can display a cursor indicating the user's current location. The cursor may be a crosshair corresponding to the user's head position. As the user moves, the cursor may move to a target interactable object. The user can select a target interactable object using, alone or in combination, posture and hand gestures on a user input device. In process 2100, the user may first interact with the object using head posture. In block 2110, the wearable system can determine whether the user has selected a target interactable object. The target interactable object may be a 2D planar virtual user interface. If the user has not selected a target interactable object, the process terminates in block 2190. In some embodiments, the wearable system can continuously determine the user's current location as the user moves. Based on the user's head posture, the wearable system can also identify other target interactable objects within the user's FOV.

[0185] As shown in block 2120, when the wearable system receives a selection of a target interactable object, such as a 2D planar virtual user interface, the wearable system may assign a focus indicator to the target interactable object. For example, the wearable system may display a halo around the target interactable object and bring the target interactable object closer to the user so that it appears to float in 3D space. The wearable system may also set the normal and depth of a cursor (corresponding to the head position) to be identical to the normal and depth of the target interactable object. As a result, the user may continue to interact with the virtual object within the target interactable object using head pose.

[0186] In block 2130, the wearable system can identify the context associated with the user's interaction. For example, the wearable system can determine the layout of virtual (or physical) objects within the user's field of view (FOV).

[0187] In block 2140, if the wearable system determines that the layout satisfies a certain pattern (e.g., one virtual object is occluded by another virtual object) or that the density exceeds a certain threshold, the wearable system may provide the user with an option to switch input control modes. As an embodiment, the wearable system may provide the user with an option to switch interaction modes from head posture to user input device.

[0188] As shown in block 2142, if the user chooses not to switch, the user can still use head posture to target and select interactable objects. The process ends in block 2190.

[0189] If the user chooses to switch input control to hand gestures, in block 2150, the user can activate the user input device and interact with a virtual object. In block 2160, the wearable system can receive the user's selection of a target virtual object, such as a UI element on a 2D planar virtual user interface. For example, referring to Figure 16, the user can select a weather application 1638, a coffee brewing application 1636, etc. If the user does not select a target virtual object in block 2160, the user may continue to activate the user input device, as shown in block 6150.

[0190] A wearable system can initiate selection or interaction events on a selected virtual object. For example, a wearable system can provide a focus indicator on a selected virtual object. A wearable system can also present VR / AR / MR scenes associated with the selected virtual object. For example, a wearable system can present scenes 1700 (shown in Figure 17) and 1800 (shown in Figure 18) in which the user can interact.

[0191] In block 2180, the wearable system can determine whether the user has completed a selection event or an interaction event. If the wearable system determines that the user has finished interacting with the target virtual object, process 2100 terminates in block 2190. In some embodiments, in block 2190, the wearable system can switch the mode of user interface control from hand gestures back to head posture.

[0192] Figure 22 illustrates an exemplary process for switching user interaction modes based on contextual information. Process 2200 can be performed by a wearable system as described herein (for example, by one or both of processing modules 260, 270).

[0193] In block 2210, the wearable system can determine the current input mode that the user is using to interact with an interactable object. The current input mode may be a posture or a hand gesture on a user input device.

[0194] In block 2220, the wearable system can determine contextual information such as the layout of objects within the user's field of view (FOV), the density of objects within the user's FOV, and the characteristics of objects within the user's FOV (size, position, object type, etc.).

[0195] Based on contextual information, the wearable system may present options in block 2230 for changing the current user input mode. For example, while a user is interacting with an object using head posture, the wearable system may present an option to change the user input mode to a user input device when the wearable system identifies a high-density group of objects in the user's line of sight. In another embodiment, the wearable system may provide an option to change the current user input mode from a user input device to head posture when the wearable system determines that objects are located at a low density.

[0196] In another embodiment, the wearable system may allow the user to interact with a user interface plane (which may include other virtual objects) using posture, while also interacting with user interface elements (such as applications within the user interface plane) using a user input device. As a result, when the wearable system detects that the user has selected a user interface plane, the wearable system may change the user input mode from head posture to the user input device. Conversely, when the user has finished interacting with the user interface elements, the wearable system may change the user input mode from the user input device to head posture.

[0197] In some implementations, the appearance of the focus indicator (including the cursor) may change based on different user input modes. For example, a wearable system may use a crosshair to indicate that the user is interacting with an object using head pose, while using an arrow to indicate that the user is interacting with an object using a user input device. In some implementations, a wearable system may change the appearance of the focus indicator to indicate that an option to switch user input modes is available. For example, a wearable system may initially display a crosshair when the user is interacting with head pose. When the wearable system detects a high-density group of objects, it may display an arrow (instead of a crosshair) to indicate that the user can switch the user input mode to a user input device. However, when the user moves away from the high-density group of objects, the wearable system may change the focus indicator back from an arrow to a crosshair.

[0198] In block 2240, the user can select an option for a new user input mode. For example, the user can activate the user input device and confirm that they wish to change the user input mode from head posture to hand gestures on the user input device. The wearable system can then update the current user input mode to the newly selected mode. The wearable system may also update the focus indicator to one associated with the newly selected mode.

[0199] Figure 23 illustrates an exemplary process of interacting with an interactable object comprising a group of virtual objects. Process 2300 can be performed by a wearable system described herein (for example, by one or both of processing modules 260, 270). The wearable system may include various sensors such as a user input device 466 (shown in Figure 4), a light field display (described with reference to Figures 2 or 4-6), an outward-facing imaging system 464 (shown in Figure 4), and an inward-facing imaging system 462 (shown in Figure 4).

[0200] In block 2310, the wearable system determines the user's posture. The posture may be the head, eyes, feet, or other body position. The wearable system can determine the user's posture using various sensors, such as an inward-facing imaging system 462, an outward-facing imaging system 464 (such as an FOV camera as described with reference to Figure 10), and an IMU.

[0201] A wearable system can determine a group of interactable objects within the user's FOR. For example, the wearable system can access a map of the user's environment containing information about objects in the user's environment. In block 6230, the wearable system can determine a target interactable object from the group of interactable objects. The target interactable object may be determined based on the user's posture. For example, the target interactable object may be an object that intersects the user's line of sight. As the user's posture changes, the target interactable object may also change.

[0202] In block 2330, the wearable system can receive the selection of a target interactable object. The user can select the target interactable object by activating user input devices, changing their posture, either alone or in combination. As shown in block 2340, after receiving the selection of a target interactable object (as shown in block 6250), the wearable system can assign a focus indicator to the target interactable object.

[0203] In some implementations, one or more virtual objects may also be displayed within the target interactable object. For example, a virtual user interface may include user interface elements such as a weather application or a video streaming application. In block 2350, the wearable system can determine a group of virtual objects associated with a selected target interactable object. The user can interact with the group of virtual objects using head posture and hand gestures. For example, the user can select a virtual object within the group of virtual objects using posture or hand gestures. The user can also initiate interaction events such as playing a video game, viewing a VR / AR / MR scene, or other user interface interactions with the selected virtual object.

[0204] In some embodiments, the wearable system may change the mode of interaction from posture to user input device once the user has selected a target interactable object. For example, the user may initially target and select an interactable object in their environment using head posture. Once the user has selected a target interactable object, the wearable system may change the input mode from head posture to user input device so that the user can interact with a virtual object within the target interactable object using the user input device. Once the user has finished interacting with the virtual object, the wearable system may change the input mode back to head posture so that the user can continue to target and interact with other interactable objects in their environment. In some implementations, the wearable system may provide the user with an option to switch modes of input control. For example, if the user decides not to switch from head posture to hand gestures after selecting a target interactable object, the user can continue to interact with the virtual object within the target interactable object using head posture.

[0205] (Additional user interface experience) (Additional examples of AR and MR visual experiences) As described above, wearable systems (such as head-mounted displays) can be configured to present 3D virtual objects superimposed on the physical world. For example, a user of a wearable device may be in a school gymnasium and perceive not only the local physical environment and physical objects (e.g., the gymnasium and students sitting or standing within it), but also virtual objects superimposed on the physical world (e.g., the school gymnasium). The virtual objects may include a leaping whale surrounded by splashes of water. For example, the user may perceive the whale emerging from the floor of the school gymnasium, leaping across part of the gymnasium, landing in a large splash of water on the floor, and then disappearing. In this embodiment, the wearable system can use a light field display, as described in Figure 4-6, to display an image of a leaping whale so that it appears to the wearer as originating from a sub-region of the area (e.g., from the gymnasium floor), by determining the dimensions of an area in the external environment that the wearer is viewing (e.g., the size of the gymnasium in this embodiment), and so that the leaping whale and splashes appear to the user in a realistic and lifelike manner. In some implementations, the wearable system can present the user (e.g., via speaker 240 shown in Figure 2) with the sound of a leaping whale accompanying the image displayed to the user. The AR system can (in addition, or alternatively) display additional virtual content such as text or graphics on the scene viewed by the user. For example, the AR system may display information about the whale (whale type, age, habits, etc.) to the wearer before, during, or after the virtual whale appears to leap from the gymnasium floor.

[0206] In another embodiment, the user of the wearable system may be in a retail market. The user can view virtual content, including images of real physical people in the environment and astronauts walking around the market. The virtual content may be superimposed within the field of view of the wearable system's display.

[0207] A wearable system can modify images of the physical world to provide users with mixed reality (MR) experiences. For example, a user might see a flock of physical birds flying in a v-formation. An outward-facing imaging system (e.g., shown in Figure 4) can capture this physical content, and the wearable system can process it and identify the flock of birds flying in a v-formation. The wearable system can add a virtual object (e.g., a dragon in this embodiment) flying within or near the formation to the flock, or replace one (or more) birds with it. In another embodiment, the wearable system can add a virtual object (e.g., a virtual whale) floating on or flying over a beach to the user's view of a physical beach. The ability of a light field display to present realistic images appearing at different distances allows the wearable display system to present an image of the whale near or away from the wearer. In some implementations, wearable systems can use shading mapping techniques so that virtual content appears to have virtual shading, which can also make the virtual content displayed by the wearable system more realistic.

[0208] In some implementations, a user of a wearable system can perceive a sequence of virtual images within an AR / MR environment. For example, suppose the user is looking at their cupped hand. A virtual object, such as a small elephant, may be displayed on the wearable system's display so that the user perceives the virtual object as being present within the user's cupped hand. The wearable system can use an outward-facing imaging system to image a region of the environment, such as the area containing the wearer's hand (and any background behind the hand). The wearable system can determine the distance to the wearer's hand so that the virtual content (e.g., the elephant) may be scaled and appear at an appropriate size and distance within a specific sub-region (e.g., the hand) of the overall region in which the content is viewed. The wearable system can make the scene appear as if the wearer is holding the elephant in their hand. The elephant's position may change from image to image so that the elephant appears closer to the user in the temporal sequence compared to the first half of the temporal sequence. Images of a jumping elephant may be accompanied by sound (e.g., voice, music).

[0209] (Additional examples of interaction with virtual objects) As an embodiment of user interaction with a virtual user interface, a user of a wearable system can perceive and interact with virtual objects within a physical room where people are dancing. In this embodiment, the user may be a disc jockey (DJ), and the wearable system can display to the DJ a virtual UI that can be operated by the DJ's hand movements (e.g., gestures). The virtual UI may include virtual content that allows the DJ to interact with the UI. In this embodiment, the virtual UI may be configured to be a DJ audio control system that can control the sounds played to the person dancing. The UI may include user input features such as dials (e.g., jog shuttle dials), switches, sliders, buttons, or turntables that can be adjusted by the DJ via gestures. The virtual UI may include output features such as sound level graphics or equalizers. The output features may respond in real time as the sound levels or audio mix are changed by the DJ. An outward-facing imaging system of the wearable system can image the DJ's hands and arms and determine the DJ's gestures (e.g., hand or finger movements). In response to the detected gesture, the wearable system can adjust the audio, for example, by increasing or decreasing the volume, fading or panning the music, or mixing the music.

[0210] In another embodiment, a user of a wearable system can visualize the operating room along with the doctor performing surgery on the patient. The wearable system can present the user with virtual content displaying the anatomical structures or organs (such as the heart) of the patient undergoing surgery. The orientation and position of the virtual organs can be adjusted via gestures (e.g., the wearer reaching for a virtual image of the heart and grasping or moving it) or via a user input device. The virtual organs can represent an idealized version of the heart (e.g., a textbook image) or an actual image of the patient's heart (e.g., taken during surgery or pre-mapped prior to surgery). The wearable system's light field display capabilities (described in Figure 4-6) allow the wearer to visualize 3D images of the organs. In some implementations, the user does not need to be physically present in the environment (e.g., the operating room) to interact with objects (virtual and / or physical objects) in the environment. Users can interact with a doctor's avatar or the appearance of an operating room, for example, by communicating with a doctor (e.g., via a speaker) or by interacting with virtual images of virtual organs.

[0211] Users can also use the wearable system to view and interact with educational virtual content. In this embodiment, the educational virtual content may include an avatar (e.g., a creation designed to be engaging and not frightening to students) that holds a pointer and directs graphics (e.g., numbers) to be displayed to the wearer as part of an educational lesson. An educational system communicating with the wearable system can generate and deliver educational virtual content to be presented to the wearer as part of an educational lesson. The virtual content may include text, images, videos, graphics, and sound. For example, the avatar could explain a math lesson to a student (e.g., 4 x 5 = ?). In some cases, the wearable system may include a microphone that can receive sounds in the surrounding environment, such as the student's voice. The student may ask a question, and the wearable system (or educational system) may use speech recognition technology to convert the question into an electronic format, and the educational system may send a response back to the wearable system. For example, the avatar may respond to students' questions by answering them, pointing to different parts of the virtual content (for example, using a wand), or explaining the answer.

[0212] In another AR experience, a user of a wearable system (such as an HMD) can view physical objects (e.g., another person and a house) and a virtual entertainment display. The virtual entertainment display shows a presentation of a sporting event (in this embodiment, a basketball game). The virtual entertainment display can present the user of the wearable system with information about the game being watched or other games (e.g., scores, commentary, game replays, player statistics, standings, etc.). The virtual entertainment display can appear in 3D, allowing the user to move around the display and view different sides of the virtual entertainment display (different content, different games, different sports, or even different entertainment genres (e.g., movies) can be displayed on different sides of the display). The sporting event may be presented to the user in real time as the sporting event unfolds. This provides the user with the ability to interact with the sporting event even if the user is not physically present at the sporting event. A particular user may control the virtual entertainment display by using gestures or a user input device. In some implementations, a single user has control over the virtual entertainment display, while other users can observe the actions on the display but cannot modify the displayed content. In other implementations, the virtual entertainment display may present content created or updated as a result of actions by multiple users.

[0213] (Additional and illustrative interactions between multiple users) When individuals within a group wear wearable devices, users can interact with other users within an AR / VR / MR environment. For example, people in a group can enjoy, interact with, share, or manipulate (e.g., via gestures) virtual content (which can include images, videos, sounds, text, etc.) through the wearable devices they are wearing.

[0214] Such user interactions may occur within a virtual game. As players interact with each other in the game, the wearable device may present virtual content and sounds to the user of the wearable device. In this embodiment, the user may perceive a room with physical objects (e.g., a bed, a table, a window, and another player 404 of the game). The user may also perceive a fire-breathing dragon flying around the other player. One or both players may control the dragon's position, movement, and actions (e.g., whether it breathes fire) by gestures (e.g., hand or arm gestures), totems (e.g., wands), or controllers or interfaces (physical or virtual). In some cases, the other player may not be physically present in the room but appear to the user of the wearable device as if they were physically present (e.g., via telepresence). For example, the wearable device may present avatars (e.g., in the form of a virtual child) of other players when playing the game. The child's avatar (and dragon) may be generated by the wearable system or another gaming system and communicated to the wearable system for display to the wearer. The child's avatar may be a representation of the actual appearance of another player, or may be selected as a form chosen by the wearable user to be perceived by the wearable user. The other player may be a human player or a machine player. In other embodiments, more than one player may appear in the wearer's environment (e.g., inside the room, outside the room, or viewed through a window). The ability of a light field display (e.g., a light field display as described with reference to Figure 4-6) to present images as if they were at different distances from the wearer (e.g., different depth planes) can greatly enhance the realism and playability of the game. In addition to the dragon, the wearable device may optionally display other information (e.g., text or graphics) to the user (e.g., gameplay statistics or status).

[0215] As illustrated with reference to Figure 7, a wearable device can pass maps of a physical or virtual environment and objects within that environment to another wearable device. For example, a wearable device can pass a map of a user's room and virtual objects within that room to another user's wearable device. Accordingly, the user can interact with the virtual objects as if they were in the same environment.

[0216] (Additional embodiment) In the first aspect, a method for viewing virtual content, comprising: accessing region data relating to a region within a user's field of view; analyzing the region data to identify subregions of a region on which the virtual content will be displayed; accessing or generating virtual content based in part on the region data and subregion data; and displaying the virtual content such that, when viewed by a user, it appears to be positioned within or on a subregion of a region.

[0217] In the second aspect, the method according to aspect 1, wherein the identified sub-region comprises physical objects within the region.

[0218] In the third aspect, the region data is obtained from the analysis of images of regions within the user's field of view, as described in aspect 1 or aspect 2.

[0219] In the fourth aspect, the virtual content is modified in part based on newly received region data or sub-region data, as described in any one of aspects 1 to 3.

[0220] The fifth aspect of the method described in any one of aspects 1 to 4 further includes the steps of receiving user input, accessing or generating additional virtual content based at least partially on the user input, and displaying the additional virtual content.

[0221] The sixth aspect is the method described in any one of aspects 1 to 5, wherein the sub-region comprises a portion of the region near the user.

[0222] A seventh aspect is a method for interacting with virtual content, comprising the steps of accessing or generating virtual content; displaying the virtual content within a user's field of view; displaying a virtual user interface associated with the virtual content; receiving user input; associating the user input with a function associated with the user input features of the virtual user interface; and performing the function.

[0223] The method according to aspect 7, wherein the eighth aspect further includes the step of manipulating or modifying virtual content based in part on received user input.

[0224] In the ninth aspect, user input is a gesture, as described in any one of aspects 7 to 8.

[0225] In the tenth aspect, user input is the movement of the user's eyes, as described in any one of aspects 7 to 9.

[0226] In the eleventh aspect, the virtual content comprises educational or entertainment content, as described in any one of aspects 7 to 10.

[0227] In the twelfth aspect, the method described in any one of aspects 7 to 11, wherein the virtual content or virtual user interface appears to be three-dimensional when perceived by the user.

[0228] In the 13th aspect, the user input features of the virtual user interface are provided by any one of the methods described in aspects 7 to 12, comprising a dial, switch, slider, or button.

[0229] Aspect 14 is the method of any one of Aspects 7 to 13, wherein the virtual user interface has an output feature configured to display information related to the functionality of the virtual user interface.

[0230] In the 15th aspect, the output features are updated in real time, as described in aspect 14.

[0231] A sixteenth aspect of the method for displaying information in an augmented reality environment, the method comprising: determining the location of a user; identifying a physical object within the user's field of view; accessing or generating a location or an identified physical object based at least in part on information relating to the identified physical object; generating virtual content based on the information; and displaying the virtual content such that the information appears to the user in association with a physical object.

[0232] In the 17th aspect, the method according to aspect 16, wherein the virtual content comprises text, images, graphics, or video.

[0233] The method as described in side 16 or side 17, further comprising the step of playing audio associated with the displayed virtual content, in side 18.

[0234] Aspect 19 further includes the step of displaying additional virtual content based on user input, in any one of Aspects 16 to 18.

[0235] In the 20th aspect, a hardware processor is programmed with executable instructions such that, when an instruction is executed, the hardware processor performs the method described in any one of aspects 1 through 19.

[0236] A 21st aspect is a wearable display system comprising the hardware processor described in aspect 20 and a display configured to provide images to the eyes of a user of the wearable display system.

[0237] On the 22nd side, the display is a wearable display system as described on side 21, comprising a light field display.

[0238] A 23rd aspect is the wearable display system according to aspect 21 or aspect 22, wherein the display is configured to display images in multiple depth planes.

[0239] Aspect 24 further comprises an image capturing device configured to capture an image of an area comprising physical objects, the wearable display system configured to display virtual content perceived by a user in such a way that it is associated with one or more of the physical objects, as described in any one of aspects 21 to 23.

[0240] In a 25th aspect, a method for selecting a virtual object located within a three-dimensional (3D) space, under the control of an augmented reality (AR) system comprising computer hardware, the AR system being configured to enable user interaction with interactive objects within the user's field of regard (FOR), the FOR comprising a part of the user's surrounding environment perceptible by the user via the AR system, under the control of the AR system, determining a group of interactive objects within the user's FOR; determining the user's pose; determining the user's field of view (FOV) at least partly based on the user's pose, the FOV comprising a part of the FOR perceived by the user at a given time; updating a subgroup of interactive objects located within the user's FOV based on a change in the user's pose or FOV; receiving a selection of a target interactive object from the subgroup of interactive objects; and initiating a selection event on the target interactive object.

[0241] In a 26th aspect, the method according to aspect 25, further comprising storing the group of interactive objects in a data structure.

[0242] In a 27th aspect, each interactive object within the group of interactive objects is represented in the data structure at least partly based on the location of the interactive object within the user's FOV, the method according to aspect 26.

[0243] In a 28th aspect, the location comprises the distance from the edge of the user's FOV, the method according to aspect 27.

[0244] In aspect 29, the method according to aspect 25, wherein the step of receiving a selection of a target interactable object from a subgroup of interactable objects includes the steps of receiving a first input from a user device and identifying a target interactable object from a subgroup of interactable objects in response to the first input.

[0245] Aspect 30 of the method according to aspect 25, further comprising the steps of receiving a second input from a user device and initiating an interaction event on a target interactable object in response to the second input.

[0246] In aspect 31, the method according to aspect 30, the target interactable object is an interactable object within the subgroup of interactable objects closest to the center point of the user's FOV.

[0247] In aspect 32, the method according to aspect 30, the target interactable object is the leftmost or rightmost interactable object within a subgroup of interactable objects in the user's FOV.

[0248] In Aspect 33, the method described in any one of Aspects 30–32, wherein the step of initiating an interaction event includes resizing the target interactable object, displaying the menu of the target interactable object, browsing the menu of the target interactable object, selecting a first item on the menu, searching for a second item in a database, playing a video game associated with the target interactable object, watching a video, or conducting a conference call, or

[0249] Aspect 34 is the method according to aspect 25, wherein the step of receiving the selection of a target interactable object is performed by determining the user's line of sight path based on the user's posture and selecting an object that intersects the user's line of sight path as the target interactable object.

[0250] Aspect 35 is the method of any one of Aspects 25-34, further comprising the step of assigning a visible focus indicator to a target interactable object.

[0251] In aspect 36, the method according to aspect 35, wherein the visible focus indicator comprises a highlight, halo, color change, size change, or change in the perceived depth of the target interactable object.

[0252] Aspect 37 is the method of Aspect 25, wherein the step of initiating a selection event includes one or more of the following steps: changing a target interactable object to a different interactable object that becomes a target interactable object; opening a menu associated with the target interactable object; or receiving confirmation from the user that they select a target interactable object.

[0253] In aspect 38, the group of interactable objects comprises virtual objects, as described in any one of aspects 25-37.

[0254] In aspect 39, the user's posture is as described in any one of aspects 25-38, comprising an eye posture.

[0255] In the 40th aspect, the user's posture is the method described in any one of the sections 25-39, comprising a head posture.

[0256] In the forty-first aspect, an augmented reality (AR) system for selecting virtual objects located in a three-dimensional (3D) space, comprising: a display system; a network interface; and a computer processor configured to communicate with the network interface and the display system, determine a group of interactable objects in the user's FOR, determine the user's posture, determine the user's field of view (FOV) at least in part based on the user's posture, the FOV comprising a portion of the FOR perceived by the user at a given time, update a subgroup of interactable objects located within the user's FOV based on a change in the user's posture or FOV, receive a selection of a target interactable object from the subgroup of interactable objects, and initiate a selection event on the target interactable object.

[0257] In aspect 42, the system described in aspect 41 is further configured such that the computer processor stores a group of interactable objects in a data structure.

[0258] In Aspect 43, one or more interactable objects within a group of interactable objects are represented in a data structure, at least partially, based on the location of the interactable object within the user's FOV, as described in Aspect 42 of the system.

[0259] In the 44th aspect, the location is the system described in aspect 43, with the distance from the edge of the user's FOV.

[0260] Aspect 45, the system according to any one of aspects 41-44, wherein a computer processor configured to receive a selection of a target interactable object from a subgroup of interactable objects includes receiving a first input from a user device and, in response to the first input, identifying a target interactable object from a subgroup of interactable objects.

[0261] In the 46th aspect, the target interactable object is an interactable object within a subgroup of interactable objects that is closest to the central point of the user's FOV, for the system according to any one of aspects 41 - 45.

[0262] In the 47th aspect, the target interactable object is the leftmost or rightmost interactable object within a subgroup of interactable objects within the user's FOV, for the system according to any one of aspects 41 - 45.

[0263] In the 48th aspect, a computer processor configured to receive a selection of a target interactable object from a subgroup of interactable objects determines a path of the user's line of sight based on the user's posture and selects, as the target interactable object, an object that intersects the path of the user's line of sight, for the system according to aspects 41 - 47.

[0264] In the 49th aspect, the computer processor is further configured to assign a visible focus indicator to the target interactable object, for the system according to any one of aspects 41 - 48.

[0265] In the 50th aspect, the visible focus indicator comprises a highlight, a backlight, a color change, a size change, or a change in the perceived depth of the target interactable object, for the system according to aspect 49.

[0266] In the 51st aspect, a computer processor configured to initiate a selection event includes changing the target interactable object to a different interactable object that becomes the target interactable object, opening a menu associated with the target interactable object, or receiving confirmation from the user to select the target interactable object, including one or more than one of these, for the system according to aspects 41 - 50.

[0267] In Aspect 52, the group of interactable objects comprises virtual objects, as described in any one of Aspects 41-51.

[0268] In aspect 53, the user's posture is the method described in any one of aspects 41-52, comprising an eye posture.

[0269] In aspect 54, the user's posture is the method described in any one of aspects 51-53, comprising a head posture.

[0270] In aspect 55, a system for interacting with virtual objects in a three-dimensional (3D) space, comprising an augmented reality display for displaying a plurality of interactable objects, a user input device, one or more sensors configured to determine the user's posture, and one or more processors, the one or more processors configured to switch between a first user input mode and a second user input mode to select a virtual object, the first user input mode being at least partially based on the user's posture, and the second user input mode being at least partially based on signals from the user input device, further comprising a system for interacting with virtual objects in a three-dimensional (3D) space, comprising an augmented reality display for displaying a plurality of interactable objects, a user input device, one or more sensors configured to determine the user's posture, and one or more processors configured to switch between a first user input mode and a second user input mode to select a virtual object, the first user input mode being at least partially based on signals from the user input device, and further comprising a system for interacting with virtual objects in a three-dimensional (3D) space, A system comprising one or more processors, configured to monitor the user's posture while operating in a user input mode, and at least partially, based on the monitored posture, display a first focus indicator associated with the first user input mode in a direction related to the user's posture, receive an indication to switch to a second user input mode, the indication further indicating the selection of a target interactable object, and while the system is operating in a second user input mode, monitor user input from a user input device and at least partially, based on the monitored user input, display a second focus indicator associated with the second user input mode.

[0271] The system according to side 55, wherein in side 56, the first focus indicator comprises a cursor in a first shape, and the second focus indicator comprises a cursor in a second shape.

[0272] The system as described on side 56, wherein on side 57, the first shape comprises a crosshair and the second shape comprises an arrow.

[0273] In aspect 58, one or more processors are further configured to determine contextual information associated with the location of a first focus indicator or a second focus indicator, and to display, at least in part, an option to switch to a different user input mode, as described in aspect 55.

[0274] In Aspect 59, the target interactable object is a system as described in any one of Aspects 55–58, comprising one or more virtual objects.

[0275] In aspect 60, the system according to any one of aspects 55-59, wherein the user's posture comprises at least one of the following: head posture, eye posture, or body posture.

[0276] In Aspect 61, the user input device is a thumb pad, trackpad, D-pad, or touchscreen, as described in any one of Aspects 55-60 of the system.

[0277] In aspect 62, a method for selecting an interactable object located in a three-dimensional (3D) space, under the control of an augmented reality (AR) system comprising computer hardware, wherein the AR system is configured to enable user interaction with interactable objects in the user's eye-tracking field of view (FOR), the FOR comprising a portion of the user's surrounding environment perceptible to the user via the AR system, and the AR system further comprising a user input device, the method comprising: determining a group of interactable objects associated with a virtual object in the user's FOR, determining the user's posture, determining a target interactable object from the group of interactable objects, at least in part based on the user's posture, associating a focus indicator with the selected target interactable object, and at least in part based on input from the user input device, initiating a selection event on the selected target surface.

[0278] In aspect 63, the step of determining the target interactable object is performed by raycasting, as described in aspect 62.

[0279] In aspect 64, raycasting is the method according to aspect 63, which includes the step of projecting a beam of light.

[0280] In aspect 65, the method according to aspect 63, wherein raycasting includes the step of projecting a ray with a substantial transverse width.

[0281] In aspect 66, the focus indicator is visible to the user, as described in aspect 62.

[0282] The method according to aspect 66, further comprising the step of displaying a focus indicator to the user in aspect 67.

[0283] The method according to aspect 62, further comprising, at least in part, the step of determining the user's field of view (FOV) based on the user's posture, wherein the FOV comprises a portion of FOR perceived by the user at a given time; accessing contextual information of one or more interactable objects within the field of view (FOV); and at least in part, providing one or more options for a user input mode based on the contextual information.

[0284] The method according to aspect 68, further comprising the steps of receiving a selection of an option for a user input mode, determining the current user input mode, and updating the current user input mode to the selected option in response to a determination that the current user input mode is different from the selected option.

[0285] The method according to aspect 69, further comprising the step of updating the focus indicator to indicate that the current user input mode has changed, in aspect 70.

[0286] In aspect 71, the method according to aspect 70, wherein the user input mode includes one or more of the user's posture or user input devices.

[0287] Aspect 72 is the method of any one of Aspects 62-71, wherein the group of interactable objects comprises one or more virtual user interface planes.

[0288] Aspect 73, the method according to any one of Aspects 62-72, wherein the user's posture comprises at least one of eye posture, head posture, or body posture.

[0289] In its 74th aspect, a method for interacting with virtual objects in a three-dimensional (3D) environment, under the control of an augmented reality (AR) system comprising computer hardware, wherein the AR system is configured to enable user interaction with interactable objects in the user's eye-moving field of view (FOR), the FOR comprising a portion of the environment surrounding the user that is perceptible to the user via the AR system, and the AR system further comprising a user input device, the method comprising: determining the user's posture; displaying a first focus indicator associated with a target interactable object in a direction related to the user's posture, wherein the target interactable object comprises a plurality of virtual objects; receiving a selection of the target interactable object; displaying the plurality of virtual objects; displaying a second focus indicator associated with the target virtual object; and updating the second focus indicator at least in part based on input from the user input device.

[0290] The method according to side 74, in side 75, wherein the first focus indicator comprises a cursor in a first shape, and the second focus indicator comprises a cursor in a second shape.

[0291] The method according to side 75, wherein on side 76, the first shape comprises a crosshair and the second shape comprises an arrow.

[0292] Aspect 77, the method according to any one of Aspects 74-76, wherein the user's posture comprises at least one of the head posture, eye posture, or body posture.

[0293] In aspect 78, the user input device is a thumb pad, trackpad, D-pad, or touchscreen, as described in any one of aspects 74-77.

[0294] In aspect 79, a method for interacting with virtual objects in a three-dimensional (3D) space, under the control of an augmented reality (AR) system comprising computer hardware, wherein the AR system is configured to enable user interaction with interactable objects in the user's eye-moving field of view (FOR), the FOR comprising a portion of the user's surrounding environment perceptible to the user via the AR system, and the AR system further comprising a user input device, comprising the steps of monitoring the user's posture while the AR system is operating in a first user input mode, which is at least partially based on the user's posture, and at least partially the monitored posture A method comprising the steps of: displaying a first focus indicator associated with a first user input mode in a direction relating to the user's posture; receiving an indication to switch to a second user input mode, the indication further indicating the selection of a target interactable object; and monitoring user input from a user input device while the AR system is operating in a second user input mode, which is at least partially based on a signal from a user input device; and displaying a second focus indicator associated with the second user input mode, at least partially based on the monitored user input.

[0295] The method according to side 79, wherein in side 80, the first focus indicator comprises a cursor in a first shape, and the second focus indicator comprises a cursor in a second shape.

[0296] The method according to side 80, wherein on side 81, the first shape is provided with a crosshair and the second shape is provided with an arrow.

[0297] The method according to aspect 79, further comprising, in aspect 82, the steps of determining contextual information associated with the location of a first focus indicator or a second focus indicator, and at least in part, displaying options for switching to a different user input mode based on the contextual information.

[0298] In Aspect 83, the method described in any one of Aspects 79-82, wherein the target interactable object comprises one or more virtual objects.

[0299] In aspect 84, the method according to any one of aspects 79-83, wherein the user's posture comprises at least one of the head posture, eye posture, or body posture.

[0300] In Aspect 85, the user input device is a thumb pad, trackpad, D-pad, or touchscreen, as described in any one of Aspects 79-84.

[0301] Aspect 86 is an augmented reality device comprising computer hardware programmed to perform the actions described in any one of Aspects 62-85.

[0302] In aspect 87, a system for changing a user input mode for a wearable device, a display system for a wearable device configured to present a three-dimensional (3D) view to a user, the 3D view comprising: the display system for a wearable device comprising an interactable object; a user input device configured to receive user input; a sensor configured to acquire data associated with the user's posture; and a hardware processor communicating with the user input device, which determines whether the current user input mode for interacting with the interactable object is a first user input mode or a second user input mode, and in response to the determination that the current user input mode is a first user input mode, the first user input mode is determined at least in part based on the user's posture, and the second user input mode is determined at least in part based on user input from the user input device. A system comprising a processor programmed to use sensors to monitor the user's posture, and via a display system, at least partially, based on the monitored posture, present a focus indicator in a first shape associated with a first user input mode in a direction related to the user's posture, receive a first indication, switch to a second user input mode, switch the current user input mode to a second user input mode in response to the first indication, monitor user input from a user input device in response to a determination that the current user input mode is a second user input mode, and via a display system, at least partially, based on the monitored input, present a focus indicator in a second shape associated with a second user input mode, receive a second indication, switch to a first user input mode, and switch the current user input mode to a first user input mode in response to the second indication.

[0303] In aspect 88, the system according to aspect 87, wherein the user's posture comprises at least one of a head posture, an eye posture, a foot posture, or a body posture, or the sensors comprises at least one of an inward-facing imaging system, an outward-facing imaging system, or an inertial measurement unit, or the display system comprises a light field display configured to display one or more of the interactable objects in a plurality of depth planes.

[0304] In aspect 89, the system according to any one of aspects 87-88, wherein the processor further determines, at least in part, the user's field of view (FOV) based on the user's posture, the FOV comprising a portion of the user's environment as perceived by the user at a given time, and determines contextual information associated with the FOV, the contextual information comprising at least one of the layout of interactable objects in the FOV, the size of the FOV, and the size of one or more of the interactable objects in the user's FOV, and is programmed, at least in part, based on the contextual information, to present an option to switch from a first user input mode to a second user input mode, or from a second user input mode to a first user input mode.

[0305] In aspect 90, the system according to aspect 89, in order to present an option, the processor is programmed to present a focus indicator in a second shape when the current user input mode is a first user input mode, and to present a focus indicator in a first shape when the current user input mode is a second user input mode.

[0306] In aspect 91, the system according to any one of aspects 87-90, wherein the first indication comprises a change in the user's posture, and the second indication comprises the activation of a user input device.

[0307] In aspect 92, a method for changing a user input mode for a wearable device, under the control of a wearable device comprising a computer processor, the wearable device being configured to enable user interaction with interactable objects in the user's eye-moving field of view (FOR), the FOR comprising a portion of the user's surrounding environment perceptible to the user via the wearable device's display system, the steps of determining the user's posture and, via the display system, displaying a first focus indicator associated with a target interactable object in a direction related to the user's posture, wherein the target interactable object is a plurality A method comprising the steps of: providing a virtual object; receiving a selection of a target interactable object; presenting the user with an option to switch the user input mode from posture to hand gestures on a user input device; displaying a plurality of virtual objects via a display system; displaying a second focus indicator via the display system associated with a target virtual object among the plurality of virtual objects in response to a determination that the user has switched the user input mode from posture to hand gestures on a user input device; and updating the second focus indicator at least in part based on user input from a user input device.

[0308] Aspect 93 is the method of Aspect 92, wherein the options are presented in response to the selection of a target interactable object or to a set of virtual objects or contextual information associated with a target interactable object.

[0309] In aspect 94, the context information comprises a density of multiple virtual objects, and an option to switch the user input mode from posture to hand gestures on a user input device is presented in response to a determination that the density of multiple virtual objects exceeds a threshold density, as described in aspect 93.

[0310] In Aspect 95, the target virtual object is identified, at least in part, based on the user's posture, as described in any one of Aspects 92-94.

[0311] In Aspect 96, the method of any one of Aspects 92–95, wherein the step of updating the second focus indicator includes the step of moving the second focus indicator from the target virtual object to another virtual object among the multiple virtual objects.

[0312] The method of any one of Aspects 92–96, further comprising the step of initiating a selection event on a target virtual object, wherein the selection event further comprises at least one of the steps of opening a menu associated with the target virtual object or receiving an indication to select the target virtual object.

[0313] The method according to any one of aspects 92-97, wherein the multiple virtual objects comprise at least one of a weather application or an astronomy application, and in response to the selection of the weather application, the wearable device is programmed to display virtual weather information superimposed on objects in the user's environment, and in response to the selection of the astronomy application, the wearable device is programmed to display an interactable planetary system comprising three-dimensional virtual planets superimposed on the user's environment.

[0314] In the 99th aspect, a wearable system for selecting virtual objects located in a three-dimensional (3D) space, comprising: a display system configured to present virtual objects in the 3D space; a non-transient data storage configured to store interactable objects in the 3D space; a sensor configured to determine the user's posture; and a hardware processor that communicates with the display system, the data storage, and the sensor, and at least partially determines the user's posture based on data received from the sensor, at least partially determines the user's field of view (FOV) based on the user's posture, the FOV comprising a portion of the user's environment as perceived by the user at a given time, identifies a group of interactable objects in the FOV, at least partially identifies a target interactable object in the FOV based on the user's posture, and is programmed to initiate a selection event associated with the target interactable object.

[0315] In aspect 100, the wearable system described in aspect 99, wherein a group of interactable objects is stored in a data structure, and the index associated with each interactable object is determined, at least in part, based on the position of the interactable object in 3D space.

[0316] Aspect 101, the wearable system according to any one of Aspects 99–100, wherein, in response to the initiation of a selection event on a target interactable object, the processor of the wearable system is programmed to present a virtual object within a two-dimensional (2D) interface that is within a threshold range of the target interactable object in 3D space.

[0317] In aspect 102, the wearable system described in aspect 101 has a 2D interface that is interactive via a user input device.

[0318] In Aspect 103, the wearable system according to any one of Aspects 99-102, wherein the processor is configured to perform at least one of the following steps in order to identify a target interactable object within the FOV: determining the user's line of sight path based on the user's posture and selecting an interactable object that intersects the line of sight path as a target interactable object; or selecting the leftmost or rightmost interactable object within the user's FOV as a target interactable object, wherein the leftmost or rightmost interactable object is selected at least in part based on an index associated with a group of interactable objects.

[0319] Aspect 104, the wearable system according to any one of aspects 99-103, wherein the processor is configured to initiate a selection event in response to at least one of receiving input from a user input device or detecting a change in the user's posture.

[0320] In aspect 105, the wearable system according to any one of aspects 99-104, wherein the processor is further configured to present a focus indicator associated with a target interactable object.

[0321] Aspect 106 further comprises a geolocation sensor configured to acquire data associated with the user's location, and includes a weather application programmed to determine the user's location based on data acquired by the geolocation sensor, communicate with a remote computing device, acquire weather data based on the user's location, generate a virtual element associated with the weather data, and superimpose the virtual element onto the user's 3D space, as described in any one of aspects 99-105.

[0322] (Conclusion) The processes, methods, and algorithms described herein and / or depicted in accompanying diagrams may be embodied in code modules executed by one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to perform specific computer instructions, thereby being fully or partially automated. For example, a computing system may include a general-purpose computer (e.g., a server) or a dedicated computer, dedicated circuit, etc., programmed with specific computer instructions. The code modules may be written in a programming language that can be compiled and linked into an executable program, installed in a dynamic link library, or interpreted. In some implementations, specific operations and methods may be performed by circuits specific to a given function.

[0323] Furthermore, functional implementations of the present disclosure are sufficiently mathematical, computational, or technically complex that application-specific hardware (utilizing appropriate specialized executable instructions) or one or more physical computing devices may need to implement the functionality, for example, due to the volume or complexity of the computations involved or to provide results substantially in real time. For example, video may contain many frames, each frame may have millions of pixels, and specifically programmed computer hardware needs to process the video data to provide a desired image processing task or application in a commercially reasonable amount of time.

[0324] Code modules or any type of data may be stored on any type of non-transient computer-readable medium, such as physical computer storage devices, including hard drives, solid-state memory, random-access memory (RAM), read-only memory (ROM), optical discs, volatile or non-volatile storage devices, combinations thereof, and / or equivalents. The method and modules (or data) may also be transmitted as data signals generated on various computer-readable transmission media, including wireless-based and wired / cable-based media (e.g., as part of a carrier wave or other analog or digital propagation signal), and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed process or process steps may be stored persistently or otherwise in any type of non-transient tangible computer storage device, or communicated via computer-readable transmission media.

[0325] Any process, block, state, step, or functionality in the flowcharts described herein and / or depicted in the accompanying diagrams should be understood as potentially representing a code module, segment, or portion of code containing one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in the process. Various processes, blocks, states, steps, or functionality may be combined, rearranged, added, deleted, modified, or otherwise changed from the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules may implement some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps, or states associated therewith may be implemented in other appropriate sequences, for example, sequentially, in parallel, or in some other manner. Tasks or events may be added to or removed from the illustrative embodiments disclosed. Furthermore, the isolation of various system components in the implementations described herein is for illustrative purposes only and should not be understood as requiring such isolation in all implementations. It should be understood that the program components, methods, and systems described can generally be integrated together in a single computer product or packaged across multiple computer products. Many implementation variations are possible.

[0326] This process, method, and system may be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowdsourced computing networks, the Internet, and the World Wide Web. The network may be a wired or wireless network or any other type of communication network.

[0327] Each system and method of this disclosure has several innovative aspects, none of which alone contribute to or are required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of each other or in various combinations. All possible and secondary combinations are intended to fall within the scope of this disclosure. Various modifications of the implementations described herein may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations without departing from the spirit or scope of this disclosure. Accordingly, the claims are not intended to be limited to the implementations shown herein, but should be given the broadest scope consistent with the disclosure, principles, and novel features disclosed herein.

[0328] Some features described herein in the context of separate implementations may also be implemented in combinations within a single implementation. Conversely, various features described in the context of a single implementation may also be implemented separately in multiple implementations or in any preferred secondary combination. Furthermore, features described above as acting in a combination and further, which may be initially claimed as such, may in some cases be removed from the combination, and the claimed combination may be subject to secondary combinations or variations of secondary combinations. No single feature or group of features is required or essential in any embodiment.

[0329] In particular, conditional statements used herein, such as “can,” “could,” “might,” “may,” “e.g.,” and equivalents, are generally intended to convey that one embodiment includes certain features, elements, and / or steps, while other embodiments do not, unless otherwise specifically stated or understood in the context in which they are used. Therefore, such conditional statements are generally not intended to suggest that features, elements, and / or steps are required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps are included or should be implemented in any particular embodiment, with or without input or prompting by the author. The terms “equipment,” “includes,” “have,” and equivalents are synonyms and are used in a non-restrictive manner to encompass additional elements, features, actions, behaviors, etc. Furthermore, the term "or" is used in its inclusive sense (and not in its exclusive sense), and therefore, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. In addition, the articles "a," "an," and "the," as used in this application and the attached claims, should be interpreted as meaning "one or more" or "at least one," unless otherwise specified.

[0330] As used herein, the phrase “at least one of” a list of items refers to any combination of those items that includes a single element. In one embodiment, “at least one of A, B, or C” is intended to encompass A, B, C, A and B, A and C, B and C, and A, B, and C. Connecting phrases such as “at least one of X, Y, and Z” are generally understood differently in contexts such as those used to convey that an item, term, etc., may be at least one of X, Y, or Z, unless otherwise specifically stated. Thus, such connecting phrases are generally not intended to suggest that one embodiment requires the presence of at least one of X, at least one of Y, and at least one of Z, respectively.

[0331] Similarly, while actions may be depicted in a diagram in a specific order, it should be recognized that such actions do not necessarily need to be performed in a specific or sequential order to achieve the desired result, nor do all illustrated actions need to be performed. Furthermore, a diagram may graphically depict one or more exemplary processes in the form of a flowchart. However, other actions not depicted may also be incorporated into the graphically illustrated exemplary methods and processes. For example, one or more additional actions may be performed before, after, simultaneously with, or in between any of the illustrated actions. In addition, actions may be rearranged or rearranged in other implementations. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products. In addition, other implementations are also within the scope of the following claims. In some cases, the actions enumerated in the claims may be performed in a different order, and the desired result can still be achieved.

Claims

1. A wearable system for selecting a virtual object located in a three-dimensional (3D) space, wherein the wearable system is A display system configured to present virtual objects in a 3D space, Non-transient data storage configured to store interactable objects in the 3D space, A sensor configured to determine the user's posture, A hardware processor programmed to communicate with the display system, the data storage, and the sensor, wherein the hardware processor is Determining the user's location and orientation, Based on the user's posture, the method involves identifying physical objects within the user's field of view (FOV), wherein the FOV comprises a portion of the user's environment that is perceptible to the user at a given time. Accessing information related to the identified physical object based at least partially on the location or the identified physical object, To generate virtual content based on the aforementioned information, Displaying the virtual content such that the information appears to the user in association with the physical object. The hardware processor performs the task A wearable system equipped with [features / equipment].

2. The system according to claim 1, wherein the hardware processor is further programmed to initiate selection events associated with the virtual content.

3. The system according to claim 2, wherein the hardware processor is configured to initiate the selection event in response to the reception of input from a user input device or the detection of a change in the user's posture.

4. The system according to claim 1, wherein the hardware processor is further programmed to present the virtual content within a threshold range of the physical objects in the 3D space within a two-dimensional (2D) interface.

5. The system according to claim 1, wherein the hardware processor is further programmed to index the virtual content based on coordinates associated with the distance from the location of the physical object.

6. The system according to claim 5, wherein the coordinates represent one or more of the depth distance, vertical distance, and lateral distance of the physical object with respect to the position.

7. The system according to claim 5, wherein the hardware processor is further programmed to determine the position of the physical object based on the user's posture.

8. The system according to claim 1, wherein the virtual content comprises text, images, graphics, or video.

9. The system according to claim 1, wherein the hardware processor is further programmed to play audio associated with the displayed virtual content.

10. The system according to claim 1, wherein the hardware processor is further programmed to display additional virtual content based at least in part on user input.

11. The system according to claim 1, wherein the hardware processor is configured to determine the path of the user's line of sight based on the user's orientation and select a possible physical object that intersects the path of the line of sight as the physical object, or select a possible physical object that is the leftmost or rightmost in the user's FOV as the physical object, wherein the leftmost or rightmost physical object is selected at least in part on coordinates associated with the position of the physical object.

12. The system according to claim 1, wherein the hardware processor is further configured to present a focus indicator associated with the virtual content.

13. The system according to claim 1, further comprising a geolocation sensor configured to acquire data associated with the user's location, wherein the virtual content comprises a weather application, the weather application programmed to determine the user's location based on the data acquired by the geolocation sensor, communicate with a remote computing device to acquire weather data based on the user's location, and superimpose the virtual content onto the user's 3D space.

14. A method for selecting a virtual object located in a three-dimensional (3D) space, wherein the method is: Under the control of a hardware processor programmed to communicate with a display system, data storage, and sensors configured to determine the user's posture, Determining the user's location and orientation, and identifying physical objects within the user's field of view (FOV) based on the user's orientation, wherein the FOV comprises a portion of the user's environment as perceived by the user at a given time. Accessing information related to the identified physical object based at least partially on the location or the identified physical object, To generate virtual content based on the aforementioned information, and to display the virtual content such that the information appears to the user in association with the physical object. A method that includes performing the following.

15. The method according to claim 14, further comprising initiating a selection event associated with the virtual content.

16. The method according to claim 15, wherein initiating the selection event is in response to receiving input from a user input device or detecting a change in the user's posture.

17. The method according to claim 14, further comprising presenting the virtual content within a threshold range of the physical object in the 3D space within a two-dimensional (2D) interface.

18. The method according to claim 14, further comprising determining the position of the physical object based on the user's posture.

19. The method according to claim 14, wherein the virtual content comprises text, images, graphics, or video.

20. The method according to claim 14, further comprising playing audio associated with the displayed virtual content.

Images