Interactions with 3D virtual objects using poses and multiple-DOF controllers

Abstract

To provide interactions with 3D virtual objects using poses and multiple-DOF controllers.SOLUTION: A wearable system can comprise a display system configured to present virtual content in a three-dimensional space, a user input device configured to receive a user input, and one or more sensors configured to detect a user's pose. The wearable system can support various user interactions with objects in the user's environment based on contextual information. As an example, the wearable system can adjust the size of an aperture of a virtual cone during a cone cast (e.g., with the user's poses) based on the contextual information. As another example, the wearable system can adjust the amount of movement of virtual objects associated with an actuation of the user input device based on the contextual information.SELECTED DRAWING: Figure 12B

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority under 35 U.S.C. § 119(e) to U.S. Provisional Application No. 62 / 316,030, filed March 31, 2016, entitled "CONE CASTING WITH DYNAMICALLY UPDATED APERTURE," and U.S. Provisional Application No. 62 / 325,679, filed April 21, 2016, entitled "DYNAMIC MAPPING OF USER INPUT DEVICE," both of which are incorporated herein by reference in their entireties.

[0002] The present disclosure relates to virtual reality and augmented reality imaging and visualization systems, and more particularly to interaction with virtual objects based on contextual information. [Background technology]

[0003] Modern computing and display technologies have facilitated the development of systems for so-called “virtual reality,” “augmented reality,” or “mixed reality” experiences in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears or can be perceived as real. Virtual reality or “VR” scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual input. Augmented reality or “AR” scenarios typically involve the presentation of digital or virtual image information as an augmentation to the visualization of the real world around the user. Mixed reality or “MR” relates to the merging of real and virtual worlds to create new environments in which physical and virtual objects coexist and interact in real time. Consequently, the human visual perception system is highly complex, making it challenging to create VR, AR, or MR technologies that facilitate a comfortable, natural-feeling, and rich presentation of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies. Summary of the Invention [Means for solving the problem]

[0004] In one embodiment, a system for interacting with objects for a wearable device is disclosed. The system includes a display system of the wearable device configured to present a three-dimensional (3D) view to a user and enable user interaction with objects within the user's field of view (FOR). The FOR can include a portion of the user's surrounding environment perceivable by the user through the display system. The system can also include a sensor configured to acquire data associated with the user's posture and a hardware processor in communication with the sensor and the display system. The hardware processor is programmed to: determine the user's posture based on the data acquired by the sensor; initiate a cone projection onto a group of objects within the FOR, the cone projection including projecting a virtual cone with an opening in a direction based at least in part on the user's posture; analyze contextual information associated with the user's environment; update the opening of the virtual cone based at least in part on the contextual information; and render a visual representation of the virtual cone for the cone projection.

[0005] In another embodiment, a method for interacting with an object for a wearable device is disclosed. The method includes receiving a selection of a target virtual object to be displayed to a user at a first location in three-dimensional (3D) space, receiving an indication of movement for the target virtual object, analyzing contextual information associated with the target virtual object, calculating a multiplier to be applied to the movement of the target virtual object based at least in part on the contextual information, calculating an amount of movement for the target virtual object, where the amount of movement is based at least in part on the indication of movement and the multiplier, and displaying the target virtual object to the user at a second location, where the second location is based at least in part on the first location and the amount of movement.

[0006] In yet another embodiment, a system for interacting with objects for a wearable device is disclosed. The system includes a display system of the wearable device configured to present a three-dimensional (3D) view to a user, the 3D view comprising a target virtual object. The system may also include a hardware processor in communication with the display system. The hardware processor is programmed to: receive an indication of movement for the target virtual object; analyze contextual information associated with the target virtual object; calculate a multiplier to apply to the movement of the target virtual object based at least in part on the contextual information; calculate an amount of movement for the target virtual object, the amount of movement based at least in part on the indication of movement and the multiplier; and display, by the display system, the target virtual object at a second location, the second location based at least in part on the first location and the amount of movement.

[0007] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Neither this summary nor the following detailed description purports to define or limit the scope of the inventive subject matter. The present invention provides, for example, the following. (Item 1) 1. A system for interacting with an object for a wearable device, the system comprising: a display system of a wearable device configured to present a three-dimensional (3D) view to a user and enable user interaction with objects within the user's field of view (FOR), the FOR including a portion of the user's surrounding environment perceivable by the user through the display system; a sensor configured to obtain data associated with a posture of the user; a hardware processor in communication with the sensor and the display system, the hardware processor comprising: determining a posture of the user based on data obtained by the sensors; initiating a cone projection onto a group of objects in the FOR, the cone projection including projecting a virtual cone with an opening in a direction based at least in part on a pose of the user; analyzing contextual information associated with the user's environment; updating an opening of the virtual cone based at least in part on the context information; rendering a visual representation of the virtual cone for the cone projection; and a hardware processor programmed to perform the A system comprising: (Item 2) Item 10. The system of claim 1, wherein the context information comprises at least one of the type, layout, location, size, or density of a subgroup of objects within the user's field of view (FOV), the FOV including a portion of the FOR that is perceptible by the user at a given time via the display system. (Item 3) The density of the subgroup of objects within the user's FOV is calculating the number of objects in said subgroup of objects; calculating the percentage of the FOV that is covered by the subgroup of objects; or calculating a contour map for objects in said subgroup of objects; Item 3. The system according to item 2, wherein the calculation is performed by at least one of the following: (Item 4) 4. The system of any one of items 1-3, wherein the hardware processor is further programmed to detect a collision between the virtual cone and one or more objects in the group of objects in the FOR, and in response to detecting the collision, the hardware processor is further programmed to present a focus indicator on the one or more objects. (Item 5) Item 5. The system of item 4, wherein the hardware processor is programmed to apply an occlusion disambiguation technique to one or more objects that collide with the virtual cone to identify occluded objects. (Item 6) 6. The system of any one of items 1-5, wherein the cone comprises a central ray and the aperture intersects the central ray. (Item 7) 7. The system of any one of items 1-6, wherein the virtual cone has a proximal end that is anchored to at least one of a location between the user's eyes, a location on a portion of the user's arm, a location on a user input device, or any other location in the user's environment. (Item 8) 8. The system of any one of items 1-7, wherein the hardware processor is further programmed to receive an indication from a user input device anchoring the depth of the virtual cone to a depth plane, and a cone projection is performed onto the group of objects within the depth plane. (Item 9) 1. A method for interacting with an object for a wearable device, the method comprising: receiving a selection of a target virtual object to be displayed to a user at a first location in three-dimensional (3D) space; receiving an indication of movement for the target virtual object; analyzing contextual information associated with the target virtual object; calculating a multiplier to apply to the movement of the target virtual object based at least in part on the context information; calculating a movement amount for the target virtual object, the movement amount based at least in part on the indication of movement and the multiplier; displaying the target virtual object to the user at a second position, the second position based at least in part on the first position and the amount of movement; and A method comprising: (Item 10) Item 10. The method of item 9, wherein the context information includes a distance from the user to the target virtual object. (Item 11) Item 11. The method of item 10, wherein the multiplier increases proportionally as the distance increases. (Item 12) 12. The method of any one of items 9-11, wherein the movement comprises one or more of a change in position, a velocity, or an acceleration. (Item 13) 13. The method of any one of items 9-12, wherein the indication of movement includes at least one of actuation of a user input device associated with the wearable device or a change in the user's posture. (Item 14) Item 14. The method of item 13, wherein the pose comprises one or more of a head pose, an eye pose, or a body pose. (Item 15) 1. A system for interacting with an object for a wearable device, the system comprising: a display system of a wearable device configured to present a three-dimensional (3D) view to a user, the 3D view comprising a target virtual object; and a hardware processor in communication with the display system, the hardware processor comprising: receiving an indication of movement for the target virtual object; analyzing contextual information associated with the target virtual object; calculating a multiplier to apply to the movement of the target virtual object based at least in part on the context information; calculating a movement amount for the target virtual object, the movement amount based at least in part on the indication of movement and the multiplier; displaying, by the display system, the target virtual object at a second position, the second position based at least in part on the first position and the amount of movement; a hardware processor programmed to perform the A system comprising: (Item 16) Item 16. The system of item 15, wherein the indication of the movement of the target virtual object includes a change in posture of a user of the wearable device or input received from a user input device associated with the wearable device. (Item 17) 17. The system of any one of items 15-16, wherein the context information includes a distance from the user to the target virtual object. (Item 18) Item 18. The system of item 17, wherein the multiplier is equal to 1 when the distance is less than a threshold distance, the threshold distance being equal to the user's reach. (Item 19) 19. The system of any one of items 17-18, wherein the multiplier increases proportionally as the distance increases. (Item 20) 20. The system of any one of items 15-19, wherein the movement includes one or more of a change in position, velocity, or acceleration. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 depicts an illustration of a mixed reality scenario with a virtual reality object and a physical object viewed by a person.

[0009] [Figure 2] FIG. 2 illustrates diagrammatically an example of a wearable system.

[0010] [Figure 3] FIG. 3 diagrammatically illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes.

[0011] [Figure 4] FIG. 4 illustrates diagrammatically an embodiment of a waveguide stack for outputting image information to a user.

[0012] [Figure 5] FIG. 5 shows an exemplary output beam that may be output by a waveguide.

[0013] [Figure 6] FIG. 6 is a schematic diagram showing an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem used in generating a multifocal stereoscopic display, image, or light field.

[0014] [Figure 7] FIG. 7 is a block diagram of an embodiment of a wearable system.

[0015] [Figure 8] FIG. 8 is a process flow diagram of an embodiment of a method for rendering virtual content in relation to recognized objects.

[0016] [Figure 9] FIG. 9 is a block diagram of another embodiment of a wearable system.

[0017] [Figure 10] FIG. 10 is a process flow diagram of an example method for determining user input to a wearable system.

[0018] [Figure 11] FIG. 11 is a process flow diagram of an embodiment of a method for interacting with a virtual user interface.

[0019] [Figure 12A] FIG. 12A illustrates an example of a cone projection with a non-negligible aperture.

[0020] [Figure 12B] 12B and 12C are examples of selecting virtual objects using cone projections with different dynamically adjusted apertures. [Figure 12C] 12B and 12C are examples of selecting virtual objects using cone projections with different dynamically adjusted apertures.

[0021] [Figure 12D] 12D, 12E, 12F, and 12G illustrate an embodiment that dynamically adjusts the aperture based on object density. [Figure 12E] 12D, 12E, 12F, and 12G illustrate an embodiment that dynamically adjusts the aperture based on object density. [Figure 12F] 12D, 12E, 12F, and 12G illustrate an embodiment that dynamically adjusts the aperture based on object density. [Figure 12G] 12D, 12E, 12F, and 12G illustrate an embodiment that dynamically adjusts the aperture based on object density.

[0022] [Figure 13] 13, 14, and 15 are flowcharts of an exemplary process for selecting an interactable object using a cone projection with a dynamically adjustable aperture. [Figure 14] 13, 14, and 15 are flowcharts of an exemplary process for selecting an interactable object using a cone projection with a dynamically adjustable aperture. [Figure 15] 13, 14, and 15 are flowcharts of an exemplary process for selecting an interactable object using a cone projection with a dynamically adjustable aperture.

[0023] [Figure 16] FIG. 16 diagrammatically illustrates an example of moving a virtual object using a user input device.

[0024] [Figure 17] FIG. 17 graphically illustrates an example of the multiplier as a function of distance.

[0025] [Figure 18] FIG. 18 illustrates a flowchart of an exemplary process for moving a virtual object in response to movement of a user input device.

[0026] Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0027] overview The wearable system can be configured to display virtual content within an AR / VR / MR environment. The wearable system can allow a user to interact with physical or virtual objects within the user's environment. The user can interact with the objects, for example, by selecting and moving the object, by using posture, or by actuating a user input device. For example, the user may move the user input device over a distance, and the virtual object will follow the user input device and move the same distance amount. Similarly, the wearable system may use cone projection to allow the user to select or target virtual objects with posture. As the user moves their hand, the wearable system can target and select different virtual objects within the user's field of view accordingly.

[0028] These approaches can tire the user when the objects are relatively far apart because the user must move the user input device or increase body movement (e.g., increase arm or head movement) over a similarly long distance to move the virtual object to a desired location or reach the desired object. In addition, precise positioning for distant objects can be difficult because it can be difficult to confirm minute adjustments at long distances. On the other hand, when the objects are closer together, the user may prefer more precise positioning to accurately interact with the desired object.

[0029] To reduce user fatigue and provide dynamic user interaction with the wearable system, the wearable system can automatically adjust user interface behavior based on context information.

[0030] As an example of providing dynamic user interaction based on context information, the wearable system can automatically update the opening of the cone in the cone projection based on the context coefficient. For example, if the user turns their head toward a direction with a high density of objects, the wearable system may automatically decrease the cone opening so that there are fewer virtual selectable objects within the cone. Similarly, if the user turns their head toward a direction with a low density of objects, the wearable system may automatically increase the cone opening to either include more objects within the cone or reduce the amount of movement required to overlap the virtual objects with the cone opening.

[0031] As another example, the wearable system can provide a multiplier that can convert the amount of movement of the user input device (and / or the user's movement) into a larger amount of movement of the virtual object. As a result, the user does not need to physically move a long distance to move the virtual object to a desired location when the object is located far away. However, the multiplier may be set to 1 when the virtual object is close to the user (e.g., within the user's reach). Thus, the wearable system can provide one-to-one manipulation between user movement and virtual object movement. This may enable the user to interact with nearby virtual objects with increased precision. Examples of user interaction based on context information are described in detail below.

[0032] Example of a 3D display for a wearable system A wearable system (also referred to herein as an augmented reality (AR) system) can be configured to present 2D or 3D virtual images to a user. The images may be still images, frames of video, or videos, in combination or the like. The wearable system can include a wearable device that can present a VR, AR, or MR environment, alone or in combination, for user interaction. The wearable device can be a head-mounted device (HMD).

[0033] Figure 1 depicts an illustration of a mixed reality scenario involving certain virtual reality objects and certain physical objects viewed by a person. In Figure 1, an MR scene 100 is depicted in which a user of the MR technology sees a real-world park-like setting 110 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of the MR technology also perceives as "seeing" a robotic figure 130 standing on the real-world platform 120 and a flying, cartoon-like avatar character 140 that appears to be an anthropomorphic bumblebee, although these elements do not exist in the real world.

[0034] In order for a 3D display to produce a true depth sensation, and more specifically, a simulated sensation of surface depth, it may be desirable for the display to generate, for each point in its field of view, an accommodation response that corresponds to that point's virtual depth. If the accommodation response to a display point does not correspond to that point's virtual depth as determined by the binocular depth cues of convergence and stereopsis, the human eye may experience accommodation conflict, resulting in unstable imaging, adverse eye strain, headaches, and, in the absence of accommodative information, a near-complete lack of surface depth.

[0035] VR, AR, and MR experiences can be provided by a display system having a display that provides a viewer with images corresponding to multiple depth planes. The images may be different for each depth plane (e.g., providing slightly different presentations of a scene or object) and may be focused separately by the viewer's eyes, thereby serving to provide depth cues to the user based on the ocular accommodation required to focus on different image features of a scene located on different depth planes, or based on observing different image features on different depth planes that are out of focus. As discussed elsewhere herein, such depth cues provide a believable perception of depth.

[0036] FIG. 2 illustrates an example of a wearable system 200. The wearable system 200 includes a display 220 and various mechanical and electronic modules and systems to support the functionality of the display 220. The display 220 may be coupled to a frame 230, which is wearable by a user, wearer, or viewer 210. The display 220 can be positioned directly in front of the eyes of the user 210. The display 220 can present AR / VR / MR content to the user. The display 220 can comprise a head-mounted display (HMD) worn on the user's head. In some embodiments, a speaker 240 is coupled to the frame 230 and positioned adjacent to the user's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / shapeable sound control).

[0037] The wearable system 200 may include an outward-facing imaging system 464 (shown in FIG. 4 ) that observes the world in the user's surrounding environment. The wearable system 200 may also include an inward-facing imaging system 462 (shown in FIG. 4 ) that can track the user's eye movements. The inward-facing imaging system may track either one eye's movements or both eyes' movements. The inward-facing imaging system 462 may be mounted to the frame 230 and may be in electrical communication with a processing module 260 or 270 that may process image information acquired by the inward-facing imaging system and determine, for example, pupil diameter or orientation of the user's 210 eyes, eye movements, or eye posture.

[0038] As an example, the wearable system 200 can capture an image of the user's posture using the outward-facing imaging system 464 or the inward-facing imaging system 462. The image may be a still image, a frame of video or video, a combination thereof, or the like.

[0039] The display 220 is operably coupled (250) to a local data processing module 260, which may be mounted in a variety of configurations, such as fixedly attached to the frame 230, by wired or wireless connection, fixedly attached to a helmet or hat worn by the user, built into headphones, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, in a belt-connected configuration).

[0040] Local processing and data module 260 may comprise a hardware processor and digital memory, such as non-volatile memory (e.g., flash memory), both of which may be utilized to aid in processing, caching, and storing data. The data may include (a) data captured from sensors (e.g., that may be operatively coupled to frame 230 or otherwise attached to user 210), such as image capture devices (e.g., cameras in inward-facing and / or outward-facing imaging systems), microphones, inertial measurement units (IMUs), accelerometers, compasses, global positioning system (GPS) units, wireless devices, or gyroscopes), or (b) data acquired or processed using remote processing module 270 or remote data repository 280, possibly for transmission to display 220 after such processing or retrieval. The local processing and data module 260 may be operably coupled to a remote processing module 270 or a remote data repository 280 by a communication link 262 or 264, such as via a wired or wireless communication link, so that these remote modules are available as resources to the local processing and data module 260. In addition, the remote processing module 280 and the remote data repository 280 may be operably coupled to each other.

[0041] In some embodiments, remote processing module 270 may comprise one or more processors configured to analyze and process data and / or image information. In some embodiments, remote data repository 280 may comprise a digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module, allowing for fully autonomous use from the remote module.

[0042] The human visual system is complex and difficult to provide a realistic perception of depth. Without being limited by theory, it is believed that viewers of an object may perceive the object as three-dimensional due to a combination of vergence and accommodation. The vergence of the two eyes relative to one another (i.e., the rolling of the pupils toward or away from one another to converge and fixate the gaze on an object) is closely coupled to the focusing of the eye's lenses (or "accommodation"). Under normal conditions, changing the focus of the eye's lenses, or accommodating the eyes, from one object to another at a different distance will automatically produce a corresponding change in vergence at the same distance, a relationship known as the "accommodation-vergence reflex." Similarly, a change in vergence will induce a corresponding change in accommodation under normal conditions. Display systems that provide better matching between accommodation and convergence-divergence movements may produce more realistic and comfortable simulations of three-dimensional images.

[0043] FIG. 3 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes. With reference to FIG. 3 , objects at various distances from the eyes 302 and 304 on the z-axis are accommodated by the eyes 302 and 304 such that the objects are in focus. The eyes 302 and 304 assume particular accommodated states, focusing objects at different distances along the z-axis. As a result, a particular accommodated state may be said to be associated with a particular one of the depth planes 306 having an associated focal length such that an object or portion of an object at a particular depth plane is in focus when the eye is in an accommodated state relative to that depth plane. In some embodiments, a three-dimensional image may be simulated by providing different representations of an image for each of the eyes 302 and 304, and by providing different representations of an image corresponding to each of the depth planes. While shown as separate for clarity of illustration, it should be understood that the fields of view of the eyes 302 and 304 may overlap, for example, as the distance along the z-axis increases. Additionally, while shown as flat for ease of illustration, it should be understood that the contours of the depth planes may be curved in physical space such that all features within the depth plane are in focus with the eye in a particular state of accommodation. Without being limited by theory, it is believed that the human eye can interpret a finite number of depth planes to typically provide depth perception. As a result, a highly realistic simulation of perceived depth may be achieved by providing the eye with different presentations of images corresponding to each of these limited number of depth planes.

[0044] Waveguide Stack Assembly FIG. 4 illustrates an example of a waveguide stack for outputting image information to a user. Wearable system 400 includes a stack of waveguides or stacked waveguide assembly 480 that can be utilized to provide three-dimensional perception to the eye / brain using multiple waveguides 432b, 434b, 436b, 438b, 4400b. In some embodiments, wearable system 400 may correspond to wearable system 200 of FIG. 2, and FIG. 4 schematically illustrates several portions of wearable system 200 in more detail. For example, in some embodiments, waveguide assembly 480 may be integrated into display 220 of FIG. 2.

[0045] 4, the waveguide assembly 480 may also include multiple features 458, 456, 454, 452 between the waveguides. In some embodiments, the features 458, 456, 454, 452 may be lenses. In other embodiments, the features 458, 456, 454, 452 may not be lenses. Rather, they may simply be spacers (e.g., cladding layers or structures to form air gaps).

[0046] Waveguides 432b, 434b, 436b, 438b, 440b or multiple lenses 458, 456, 454, 452 may be configured to transmit image information to the eye using various levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a particular depth plane and configured to output image information corresponding to that depth plane. Image injection devices 420, 422, 424, 426, 428 may be utilized to inject image information into waveguides 440b, 438b, 436b, 434b, 432b, respectively, which may be configured to disperse incident light across each individual waveguide for output toward the eye 410. Light exits the output surfaces of image injection devices 420, 422, 424, 426, 428 and is injected into the corresponding input edges of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam of light (e.g., a collimated beam) may be injected into each waveguide, outputting an entire field of cloned collimated beams directed toward eye 410 at a particular angle (and divergence) corresponding to the depth plane associated with the particular waveguide.

[0047] In some embodiments, image input devices 420, 422, 424, 426, 428 are discrete displays that generate image information for input into each corresponding waveguide 440b, 438b, 436b, 434b, 432b, respectively. In some other embodiments, image input devices 420, 422, 424, 426, 428 are outputs of a single multiplexed display that may, for example, send image information to each of image input devices 420, 422, 424, 426, 428 via one or more optical conduits (such as fiber optic cables).

[0048] A controller 460 controls the operation of stacked waveguide assembly 480 and image injection devices 420, 422, 424, 426, 428. Controller 460 includes programming (e.g., instructions in a non-transitory computer-readable medium) that coordinates the timing and provision of image information to waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, controller 460 may be a single integrated device or a distributed system connected by a wired or wireless communication channel. Controller 460 may, in some embodiments, be part of processing module 260 or 270 (shown in FIG. 2).

[0049] Waveguides 440b, 438b, 436b, 434b, 432b may be configured to propagate light within each individual waveguide by total internal reflection (TIR). Waveguides 440b, 438b, 436b, 434b, 432b may each be planar or have another shape (e.g., curved) with major top and bottom surfaces and edges extending between their major top and bottom surfaces. In the illustrated configuration, waveguides 440b, 438b, 436b, 434b, 432b may each include light extraction optical elements 440a, 438a, 436a, 434a, 432a configured to extract light from the waveguides by redirecting the light to propagate within each individual waveguide and outputting image information from the waveguides to the eye 410. The extracted light may also be referred to as out-coupled light, and the light extraction optical element may also be referred to as out-coupling optical element. The extracted light beam is output by the waveguide where the light propagating within the waveguide strikes the light redirecting element. The light extraction optical element (440a, 438a, 436a, 434a, 432a) may be, for example, a reflective or diffractive optical feature. While shown disposed on the bottom major surfaces of the waveguides 440b, 438b, 436b, 434b, 432b for ease of explanation and clarity of the drawings, in some embodiments, the light extraction optical element 440a, 438a, 436a, 434a, 432a may be disposed on the top or bottom major surfaces or directly within the volume of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed in a layer of material that is attached to a transparent substrate and forms the waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, 432b may be a monolithic piece of material, and the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed on and / or within that piece of material.

[0050] Continuing with reference to FIG. 4, as discussed herein, each waveguide 440b, 438b, 436b, 434b, 432b is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 as it is launched into such waveguide 432b. The collimated light may represent an optical infinity focal plane. The next waveguide 434b may be configured to send collimated light that passes through a first lens 452 (e.g., a negative lens) before reaching the eye 410. The first lens 452 may be configured to create a slight convex wavefront curvature so that the eye / brain interprets light emerging from the next upper waveguide 434b as emerging from a first focal plane closer inward from optical infinity toward the eye 410. Similarly, the third upper waveguide 436b passes its output light through both the first lens 452 and the second lens 454 before reaching the eye 410. The combined refractive power of the first and second lenses 452 and 454 may be configured to produce another, increasing amount of wavefront curvature such that the eye / brain interprets the light emerging from the third waveguide 436b as originating from a second focal plane that is even closer inward from optical infinity towards the person than was the light from the next upper waveguide 434b.

[0051] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured, with the highest waveguide 440b in the stack sending its output through all of the lenses between it and the eye for a collective focal power representing the focal plane closest to the person. To compensate for the stack of lenses 458, 456, 454, 452 when viewing / interpreting light originating from the world 470 on the other side of the stacked waveguide assembly 480, a compensating lens layer 430 may be placed on top of the stack to compensate for the collective power of the lower lens stacks 458, 456, 454, 452. Such a configuration provides as many perceived focal planes as there are available waveguide / lens pairs. Both the light extraction optical elements of the waveguides and the focusing sides of the lenses may be static (e.g., not dynamic or electro-active). In some alternative embodiments, one or both may be dynamic using electro-active features.

[0052] Continuing with reference to FIG. 4, light extraction optical elements 440a, 438a, 436a, 434a, 432a may be configured to redirect light from their respective waveguides and output this light with the appropriate amount of divergence or collimation for the particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes may have differently configured light extraction optical elements that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, light extraction optical elements 440a, 438a, 436a, 434a, 432a may be solid or surface features that can be configured to output light at specific angles. For example, light extraction optical elements 440a, 438a, 436a, 434a, 432a may be volume holograms, surface holograms, and / or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published June 25, 2015, which is incorporated herein by reference in its entirety.

[0053] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffractive features, i.e., "diffractive optical elements" (also referred to herein as "DOEs"), that form a diffraction pattern. Preferably, the DOEs have a relatively low diffraction efficiency so that only a portion of the light in the beam is deflected toward the eye 410 with each intersection point of the DOE, while the remainder continues traveling through the waveguide via total internal reflection. The light carrying the image information is thus split into several related output beams that exit the waveguide at multiple locations, which can result in a very uniform pattern of output emission toward the eye 304 for this particular collimated beam bouncing within the waveguide.

[0054] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer-dispersed liquid crystal in which microdroplets comprise a diffractive pattern in a host medium, and the refractive index of the microdroplets can be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0055] In some embodiments, the number and distribution of depth planes or depths of field may be dynamically varied based on the size or orientation of the viewer's pupil. The depth of field may vary inversely with the viewer's pupil size. As a result, as the size of the viewer's pupil decreases, the depth of field increases so that a plane that is indistinguishable because its location exceeds the eye's depth of focus may become distinguishable and appear more focused with a corresponding decrease in pupil size and an increase in depth of field. Similarly, the number of spaced depth planes used to present different images to the viewer may be reduced with a decreased pupil size. For example, a viewer may not be able to clearly perceive details in both a first depth plane and a second depth plane at one pupil size without adjusting their eye's accommodation from one depth plane to the other. However, these two depth planes may be sufficient to simultaneously focus on the user at another pupil size without changing accommodation.

[0056] In some embodiments, the display system may vary the number of waveguides receiving image information based on a determination of pupil size and / or orientation, or in response to receiving an electrical signal indicating a particular pupil size and / or orientation. For example, if a user's eye is unable to distinguish between two depth planes associated with two waveguides, the controller 460 may be configured or programmed to stop providing image information to one of those waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing system responsiveness. In embodiments in which the DOE for a waveguide is switchable between on and off states, the DOE may be switched to the off state when the waveguide receives image information.

[0057] In some embodiments, it may be desirable to have the output beam satisfy the condition of having a diameter less than the diameter of the viewer's eye. However, meeting this condition may be difficult in light of the variability in the size of the viewer's pupil. In some embodiments, this condition is met over a wide range of pupil sizes by varying the size of the output beam in response to a determination of the size of the viewer's pupil. For example, as the pupil size decreases, the size of the output beam may also decrease. In some embodiments, the output beam size may be varied using a variable aperture.

[0058] The wearable system 400 may include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the field of view (FOV), and the imaging system 464 is sometimes referred to as an FOV camera. The entire area available for viewing or imaging by a viewer may be referred to as the field of view (FOR). The FOR may include a solid angle of 4π steradians surrounding the wearable system 400 as the wearer moves their body, head, or eyes to perceive virtually any direction in space. In other situations, the wearer's movement may be more constrained, and the wearer's FOR may correspond accordingly to a smaller solid angle. Images obtained from the outward-facing imaging system 464 can be used to track gestures (e.g., hand or finger gestures) made by the user, detect objects in the world 470 in front of the user, etc.

[0059] The wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera) that observes user movements, such as eye and facial movements. The inward-facing imaging system 466 may be used to capture images of the eyes 410 and determine the size or orientation of the pupils of the eyes 304. The inward-facing imaging system 466 can be used to obtain images for use in determining the direction the user is looking (e.g., eye pose) or for biometric identification of the user (e.g., via iris identification). In some embodiments, at least one camera may be utilized for each eye independently to separately determine the pupil size or eye pose of each eye, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye. In some other embodiments, the pupil diameter or orientation of only a single eye 410 (e.g., using only a single camera per pair of eyes) is determined and assumed to be similar for both eyes of the user. Images obtained by inward-facing imaging system 466 may be analyzed to determine the user's eye posture or mood, which may be used by wearable system 400 to determine audio or visual content to be presented to the user. Wearable system 400 may also determine head pose (e.g., head position or head orientation) using sensors such as an IMU, accelerometer, gyroscope, etc.

[0060] The wearable system 400 may include a user input device 466 through which a user may input commands into the controller 460 and interact with the wearable system 400. For example, the user input device 466 may include a trackpad, touchscreen, joystick, multi-degree-of-freedom (DOF) controller, capacitive sensing device, game controller, keyboard, mouse, directional pad (D-pad), wand, tactile device, totem (e.g., functioning as a virtual user input device), etc. A multi-DOF controller may sense user input in possible translation (e.g., left / right, forward / backward, or up / down) or rotation (e.g., yaw, pitch, or roll) of some or all of the controller. A multi-DOF controller that supports translation may be referred to as 3DOF, while a multi-DOF controller that supports translation and rotation may be referred to as 6DOF. In some cases, a user may use a finger (e.g., a thumb) to press or swipe across a touch-sensitive input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held by the user's hand during use of the wearable system 400. The user input device 466 may communicate with the wearable system 400 via wired or wireless communication.

[0061] FIG. 5 shows an example of an output beam output by a waveguide. While one waveguide is shown, it should be understood that other waveguides in waveguide assembly 480 may function similarly, and that waveguide assembly 480 includes multiple waveguides. Light 520 is launched into waveguide 432b at input edge 432c of waveguide 432b and propagates within waveguide 432b by TIR. At the point where light 520 impinges on DOE 432a, a portion of the light exits the waveguide as output beam 510. Although output beams 510 are shown as approximately parallel, they may also be redirected to propagate to eye 410 at an angle (e.g., forming a diverging output beam) depending on the depth plane associated with waveguide 432b. It should be understood that a nearly collimated exit beam may refer to a waveguide with light extraction optics that outcouples light and forms an image that appears to be set at a depth plane at a long distance (e.g., optical infinity) from the eye 410. Other waveguides or other sets of light extraction optics may output a more divergent exit beam pattern, which would require the eye 410 to accommodate to closer distances and focus on the retina, and would be interpreted by the brain as light from a distance closer to the eye 410 than optical infinity.

[0062] FIG. 6 is a schematic diagram showing an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem used in generating a multifocal volumetric display, image, or light field. The optical system can include a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem. The optical system can be used to generate a multifocal volumetric display, image, or light field. The optical system can include one or more primary planar waveguides 632a (only one is shown in FIG. 6) and one or more DOEs 632b associated with each of at least some of the primary waveguides 632a. The planar waveguides 632b can be similar to the waveguides 432b, 434b, 436b, 438b, and 440b discussed with reference to FIG. 4. The optical system may employ a dispersive waveguide device to relay light along a first axis (the vertical or Y-axis in the illustration of FIG. 6 ) and expand the effective exit pupil of the light along the first axis (e.g., the Y-axis). The dispersive waveguide device may include, for example, a dispersive planar waveguide 622 b and at least one DOE 622 a (illustrated by a double-dashed line) associated with the dispersive planar waveguide 622 b. The dispersive planar waveguide 622 b may be similar or identical in at least some respects to a primary planar waveguide 632 b having a different orientation therefrom. Similarly, the at least one DOE 622 a may be similar or identical in at least some respects to the DOE 632 a. For example, the dispersive planar waveguide 622 b or the DOE 622 a may be made of the same material as the primary planar waveguide 632 b or the DOE 632 a, respectively. The embodiment of the optical display system 600 shown in FIG. 6 can be integrated into the wearable system 200 shown in FIG.

[0063] The relayed, exit-pupil-expanded light can be optically coupled from the dispersive waveguide device into one or more primary planar waveguides 632b. The primary planar waveguides 632b can relay the light along a second axis, preferably orthogonal to the first axis (e.g., the horizontal or X-axis in the diagram of FIG. 6). Notably, the second axis can be non-orthogonal to the first axis. The primary planar waveguides 632b expand the effective exit pupil of the light along its second axis (e.g., the X-axis). For example, the dispersive planar waveguide 622b can relay and expand the light along the vertical or Y-axis and pass the light to the primary planar waveguide 632b, which can relay and expand the light along the horizontal or X-axis.

[0064] The optical system may include one or more colored light sources (e.g., red, green, and blue laser light) 610, which may be optically coupled into the proximal end of a single-mode optical fiber 640. The distal end of the optical fiber 640 may be threaded or received through a hollow tube 642 of piezoelectric material. The distal end protrudes from the tube 642 as a free-standing, flexible cantilever 644. The piezoelectric tube 642 may be associated with four quadrant electrodes (not shown). The electrodes may be plated, for example, on the outside, outer surface or outer periphery, or diameter of the tube 642. A core electrode (not shown) may also be located in the core, center, inner periphery, or inner diameter of the tube 642.

[0065] For example, drive electronics 650, electrically coupled via wires 660, drive opposing pairs of electrodes to bend piezoelectric tube 642 independently in two axes. The protruding distal tip of optical fiber 644 has a mechanical resonant mode. The frequency of the resonance may depend on the diameter, length, and material properties of optical fiber 644. By oscillating piezoelectric tube 642 near the first mechanical resonant mode of fiber cantilever 644, fiber cantilever 644 may be caused to oscillate and sweep through a large deflection.

[0066] By stimulating resonant vibrations in two axes, the tip of fiber cantilever 644 is scanned biaxially within an area filling a two-dimensional (2-D) scan. By modulating the intensity of light source 610 synchronously with the scanning of fiber cantilever 644, light emitted from fiber cantilever 644 can form an image. A description of such a setup is provided in U.S. Patent Publication No. 2014 / 0003762, which is incorporated herein by reference in its entirety.

[0067] Components of the optical coupler subsystem can collimate light emitted from the scanning fiber cantilever 644. The collimated light can be reflected by a mirrored surface 648 into a narrow dispersive planar waveguide 622b containing at least one diffractive optical element (DOE) 622a. The collimated light can propagate perpendicularly (with respect to the view of FIG. 6) along the dispersive planar waveguide 622b via TIR, and in doing so, repeatedly intersect with the DOE 622a. The DOE 622a preferably has a low diffraction efficiency. This causes a portion of the light (e.g., 10%) to diffract toward the edge of the larger primary planar waveguide 632b at each point of intersection with the DOE 622a, allowing a portion of the light to continue on its original trajectory down the length of the dispersive planar waveguide 622b via TIR.

[0068] At each point of intersection with DOE 622a, additional light can be diffracted toward the entrance of primary waveguide 632b. By splitting the incident light into multiple outcoupled sets, the exit pupil of the light can be vertically expanded by DOE 4 within dispersive planar waveguide 622b. This vertically expanded light outcoupled from dispersive planar waveguide 622b can enter the edge of primary planar waveguide 632b.

[0069] Light entering the primary waveguide 632b can propagate horizontally (with respect to the illustration of FIG. 6) along the primary waveguide 632b via TIR. The light propagates horizontally along at least a portion of the length of the primary waveguide 632b via TIR as it intersects the DOE 632a at multiple points. The DOE 632a advantageously has a phase profile that is the sum of a linear diffraction pattern and a radially symmetric diffraction pattern, and may be designed or configured to produce both deflection and focusing of the light. The DOE 632a advantageously may have a low diffraction efficiency (e.g., 10%) so that only a portion of the light in the beam is deflected toward the viewer's eye at each intersection of the DOE 632a, while the remainder of the light continues to propagate through the primary waveguide 632b via TIR.

[0070] At each point of intersection between the propagating light and the DOE 632a, a portion of the light is diffracted toward the adjacent face of the primary waveguide 632b, allowing the light to escape the TIR and emerge from the face of the primary waveguide 632b. In some embodiments, the radially symmetric diffraction pattern of the DOE 632a additionally imparts a focal level to the diffracted light, both shaping (e.g., imparting curvature) the optical wavefronts of the individual beams and steering the beams to angles consistent with the designed focal level.

[0071] Thus, these different paths can couple light out of the primary planar waveguide 632b by providing different fill patterns at the DOE 632a's multiplicity, focal level, and / or exit pupil at different angles. Different fill patterns at the exit pupil can be advantageously used to generate light field displays with multiple depth planes. Each layer in the waveguide assembly or set of layers (e.g., three layers) in the stack may be employed to generate distinct colors (e.g., red, blue, and green). Thus, for example, a first set of three adjacent layers may be employed to generate red, blue, and green light, respectively, at a first focal depth. A second set of three adjacent layers may be employed to generate red, blue, and green light, respectively, at a second focal depth. Multiple sets may be employed to generate full 3D or 4D color image light fields with various focal depths.

[0072] (Other components of the wearable system) In many implementations, the wearable system may include other components in addition to or as an alternative to the components of the wearable system described above. The wearable system may include, for example, one or more tactile devices or components. The tactile device or component may be operable to provide a haptic sensation to the user. For example, the tactile device or component may provide a sensation of pressure and / or texture upon touching virtual content (e.g., a virtual object, virtual tool, other virtual structure). The haptic sensation may replicate the sensation of a physical object represented by the virtual object, or may replicate the sensation of an imaginary object or character (e.g., a dragon) represented by the virtual content. In some implementations, the tactile device or component may be worn by the user (e.g., a user-wearable glove). In some implementations, the tactile device or component may be held by the user.

[0073] A wearable system may include, for example, one or more physical objects that can be manipulated by a user to enable input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as, for example, a piece of metal or plastic, a wall, the surface of a table, etc. In some implementations, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, the totem may simply provide a physical surface, and the wearable system may render a user interface to appear to the user on one or more surfaces of the totem. For example, the wearable system may render an image of a computer keyboard and trackpad to appear to reside on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and virtual trackpad to appear on the surface of a thin rectangular plate of aluminum that serves as the totem. The rectangular plate itself does not have any physical keys or trackpads or sensors. However, the wearable system may detect user manipulation or interaction or touch with the rectangular plate as a selection or input made via a virtual keyboard or virtual trackpad. User input device 466 (shown in FIG. 4) may be an embodiment of a totem, which may include a trackpad, touchpad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. A user may use the totem alone or in combination with posture to interact with the wearable system and / or other users.

[0074] Examples of tactile devices and totems usable with the wearable devices, HMDs, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated herein by reference in its entirety.

[0075] Exemplary Wearable Systems, Environments, and Interfaces The wearable system may employ various mapping-related techniques to achieve a high depth of field within the rendered light field. When mapping a virtual world, it is advantageous to capture all features and points in the real world and accurately depict virtual objects in relation to the real world. To achieve this goal, FOV images captured from a user of the wearable system can be added to the world model by including new photos that convey information about various points and features in the real world. For example, the wearable system can collect a set of map points (such as 2D or 3D points), find new map points, and render a more accurate version of the world model. The world model of a first user can be communicated to a second user (e.g., via a network such as a cloud network) so that the second user can experience the world surrounding the first user.

[0076] 7 is a block diagram of an example MR environment 700. The MR environment 700 may be configured to receive inputs (e.g., visual input 702 from a user's wearable system, stationary input 704 such as a room camera, sensory input 706 from various sensors, gestures, totems, eye tracking, user input, etc. from user input device 466) from one or more of the user's wearable system (e.g., wearable system 200 or display system 220) or stationary room systems (e.g., room cameras, etc.). The wearable systems can determine the location and various other attributes of the user's environment using various sensors (e.g., accelerometers, gyroscopes, temperature sensors, movement sensors, depth sensors, GPS sensors, inward-facing imaging systems, outward-facing imaging systems, etc.). This information may be further supplemented with information from stationary cameras in the room, which may provide images from different perspectives or various cues. Image data acquired by cameras (e.g., room cameras or outward-facing imaging system cameras) may be reduced to a set of mapping points.

[0077] One or more object recognizers 708 can crawl through the received data (e.g., a collection of points), recognize or map the points, tag the images, and associate semantic information with the objects using a map database 710. The map database 710 may comprise various points and their corresponding objects collected over time. The various devices and the map database may be interconnected through a network (e.g., a LAN, a WAN, etc.) and accessible to the cloud.

[0078] Based on this information and the set of points in the map database, the object recognizers 708a-708n may recognize objects in the environment. For example, the object recognizers may recognize faces, people, windows, walls, user input devices, televisions, other objects in the user's environment, etc. One or more object recognizers may be specialized for objects with certain characteristics. For example, object recognizer 708a may be used to recognize faces, while another object recognizer may be used to recognize totems.

[0079] Object recognition may be performed using various computer vision techniques. For example, the wearable system may analyze images acquired by the outward-facing imaging system 464 (shown in FIG. 4) and perform scene reconstruction, event detection, video tracking, object recognition, object pose estimation, learning, indexing, motion estimation, or image restoration, etc. One or more computer vision algorithms may be used to perform these tasks. Non-limiting examples of computer vision algorithms include Scale Invariant Feature Transform (SIFT), Speed-Up Robust Features (SURF), Orientation FAST and Rotation BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retinal Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, visual simultaneous localization and mapping (vSLAM) techniques, sequential Bayes estimators (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, adaptive thresholding (and other thresholding techniques), iterative nearest neighbor (ICP), semi-global matching (SGM), semi-global block matching (SGBM), feature point histograms, various machine learning algorithms (e.g., support vector machines, k-nearest neighbor algorithms, naive Bayes, neural networks (including convolutional or deep neural networks), or other supervised / unsupervised models, etc.), etc.

[0080] Object recognition can additionally or alternatively be performed by various machine learning algorithms. Once trained, the machine learning algorithms can be stored by the HMD. Some examples of machine learning algorithms can include supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., ordinary least squares regression, etc.), instance-based algorithms (e.g., learning vector quantization, etc.), decision tree algorithms (e.g., classification and regression trees, etc.), Bayesian algorithms (e.g., naive Bayes, etc.), clustering algorithms (e.g., k-means clustering, etc.), association rule learning algorithms (e.g., a priori algorithm, etc.), artificial neural network algorithms (e.g., Perceptron, etc.), deep learning algorithms (e.g., Deep Boltzmann Machine, i.e., deep neural networks, etc.), dimensionality reduction algorithms (e.g., principal component analysis, etc.), ensemble algorithms (e.g., stacked generalization, etc.), and / or other machine learning algorithms. In some embodiments, individual models can be customized for individual datasets. For example, the wearable device can generate or store a base model. The base model may be used as a starting point to generate additional models specific to a data type (e.g., a particular user in a telepresence session), a data set (e.g., a set of additional images acquired of a user in a telepresence session), a conditional situation, or other variations. In some embodiments, the wearable HMD can be configured to generate models for analysis of aggregated data using multiple techniques. Other techniques may include using predefined thresholds or data values.

[0081] Based on this information and the set of points in the map database, the object recognizers 708a-708n may recognize objects, complement them with semantic information, and bring them to life. For example, if the object recognizer recognizes that a set of points is a door, the system may associate some semantic information (e.g., a door has a hinge and 90-degree movement around the hinge). If the object recognizer recognizes that a set of points is a mirror, the system may associate the semantic information that a mirror has a reflective surface that can reflect images of objects in the room. Over time, the map database grows as the system (which may reside locally or be accessible through a wireless network) accumulates more data from the world. Once an object is recognized, the information may be transmitted to one or more wearable systems. For example, the MR environment 700 may contain information about a scene generating in California. The environment 700 may be transmitted to one or more users in New York. Based on the data received from the FOV camera and other inputs, the object recognizer and other software components can map points collected from various images, recognize objects, etc. so that the scene can be accurately "passed" to a second user who may be in a different part of the world. The environment 700 may also use a topology map for localization purposes.

[0082] 8 is a process flow diagram of an example method 800 for rendering virtual content in relation to recognized objects. Method 800 describes how a virtual scene can be presented to a user of a wearable system. The user may be geographically remote from the scene. For example, a user may be in New York but may want to view a scene currently occurring in California, or may want to go for a walk with a friend who is in California.

[0083] In block 810, the wearable system may receive input from the user and other users regarding the user's environment. This may be accomplished through various input devices and knowledge already held in a map database. The user's FOV camera, sensors, GPS, eye tracking, etc., communicate information to the system in block 810. The system may determine sparse points based on this information in block 820. The sparse points may be used to determine pose data (e.g., head pose, eye pose, body pose, or hand gestures) that may be used in displaying and understanding the orientation and position of various objects in the user's surroundings. The object recognizers 708a-708n may crawl through these collected points and recognize one or more objects using the map database in block 830. This information may then be communicated to the user's respective wearable system in block 840, and the desired virtual scene may be displayed to the user appropriately in block 850. For example, a desired virtual scene (eg, a user in CA) may be displayed in the proper orientation, position, etc., relative to various objects and other surroundings of the user in New York.

[0084] FIG. 9 is a block diagram of another example of a wearable system. In this example, the wearable system 900 includes a map, which may include map data about the world. The map may reside partially locally on the wearable system and partially in a networked storage location (e.g., in a cloud system) accessible by a wired or wireless network. An attitude process 910 may run on the wearable computing architecture (e.g., processing module 260 or controller 460) and utilize data from the map to determine the position and orientation of the wearable computing hardware or the user. The attitude data may be calculated from data collected on the fly as the user experiences the system and moves within its world. The data may include images of objects in the real or virtual environment, data from sensors (such as inertial measurement units, which generally include accelerometer and gyroscope components), and surface information.

[0085] The sparse point representation may be the output of a simultaneous localization and mapping (SLAM or V-SLAM, which refers to configurations where the input is only image / vision) process. The system can be configured to find what the world is made of, not just the locations of various components in the world. Poses may be building blocks that accomplish many goals, including filling in maps and using data from maps.

[0086] In one embodiment, the sparse point locations may not be entirely adequate by themselves, and additional information may be required to generate a multifocal AR, VR, or MR experience. A dense representation, generally referring to depth map information, may be utilized to fill in this gap, at least in part. Such information may be calculated from a process referred to as stereoscopic vision 940, where depth information is determined using techniques such as triangulation or time-of-flight sensing. Image information and active patterns (such as infrared patterns generated using an active projector) may serve as inputs to the stereoscopic vision process 940. A significant amount of depth map information may be fused together, and some of this may be summarized using a surface representation. For example, a mathematically definable surface may be an efficient (e.g., compared to a large-scale point cloud) and digestible input to other processing devices, such as a game engine. Thus, the outputs of the stereoscopic vision process (e.g., depth map) 940 may be combined in a fusion process 930. The pose may also be input to this fusion process 930, the output of which is the input to fill the map process 920. Sub-surfaces may connect to each other to form larger surfaces, such as in topographic mapping, and the map becomes a large-scale hybrid of points and surfaces.

[0087] Various inputs may be utilized to resolve various aspects of the mixed reality process 960. For example, in the embodiment depicted in FIG. 9, game parameters may be inputs for determining whether a user of the system is playing a monster battle game with one or more monsters in various locations, whether the monsters are dead or fleeing under various conditions (such as when the user shoots the monster), walls or other objects in various locations, and the like. A world map may contain information about where such objects are located relative to one another, which is another useful input for mixed reality. Attitude relative to the world is likewise an input and plays an important role for nearly any interactive system.

[0088] Controls or inputs from the user are another input to the wearable system 900. As described herein, user inputs can include visual inputs, gestures, totems, audio inputs, sensor inputs, etc. To move around or play games, for example, the user may need to command the wearable system 900 as to what they want to do. There are various forms of user control that can be utilized beyond just moving around in space. In one embodiment, a totem (e.g., a user input device), or an object such as a toy gun, may be held by the user and tracked by the system. The system would preferably be configured to know that the user is holding an item and understand the type of interaction the user is having with the item (e.g., if the totem or object is a gun, the system may be configured to understand not only the location and orientation, but also whether the user is clicking a trigger or other sensitive button or element, which may be equipped with sensors (e.g., an IMU) that can help determine what is happening even when such activity is not within the field of view of any of the cameras).

[0089] Hand gesture tracking or recognition may also provide input information. The wearable system 900 may be configured to track and interpret hand gestures to gesture for button presses, left or right, stop, grasp, hold, etc. For example, in one configuration, a user may wish to flip through email or calendar in a non-gaming environment, or to perform a “fist bump” with another person or performer. The wearable system 900 may be configured to utilize a minimal amount of hand gestures, which may or may not be dynamic. For example, gestures may be simple static gestures, such as extending the hand to indicate stop, thumbs up to indicate OK, thumbs down to indicate not OK, or flipping the hand left and right or up and down to indicate a directional command.

[0090] Eye tracking is another input (e.g., to track where the user is looking and control display technology to render at a specific depth or range). In one embodiment, eye convergence may be determined using triangulation, and then accommodation may be determined using a convergence / accommodation model developed for that particular person.

[0091] With respect to the camera system, the exemplary wearable system 900 shown in FIG. 9 may include three pairs of cameras: a pair of relatively wide-FOV or passive SLAM cameras arranged on either side of the user's face, and a different pair of cameras oriented in front of the user to handle the stereoscopic imaging process 940 and capture hand gestures and totem / object trajectories in front of the user's face. The FOV cameras and pair of cameras for the stereo process 940 may be part of the outward-facing imaging system 464 (shown in FIG. 4). The wearable system 900 may include an eye-tracking camera (which may be part of the inward-facing imaging system 462 shown in FIG. 4) oriented toward the user's eyes to triangulate eye vectors and other information. The wearable system 900 may also include one or more textured light projectors (such as infrared (IR) projectors) to inject texture into the scene.

[0092] 10 is a process flow diagram of an example embodiment of a method 1000 for determining user input to a wearable system. In this example, a user may interact with a totem. A user may have multiple totems. For example, a user may have one totem designated for social media applications, another totem for playing games, etc. In block 1010, the wearable system may detect movement of the totem. The movement of the totem may be recognized through an outward-facing system or may be detected through sensors (e.g., tactile gloves, image sensors, hand tracking devices, eye tracking cameras, head pose sensors, etc.).

[0093] Based at least in part on the detected gestures, eye poses, head poses, or input through the totem, the wearable system detects the position, orientation, and / or movement of the totem (or the user's eyes or head or gestures) relative to a frame of reference in block 1020. The frame of reference may be a set of map points based on which the wearable system translates the totem's (or the user's) movements into actions or commands. In block 1030, the user's interactions with the totem are mapped. Based on the mapping of the user interactions to the frame of reference 1020, the system determines the user input in block 1040.

[0094] For example, a user may move a totem or physical object back and forth, turn a virtual page, move to the next page, or move from one user interface (UI) display screen to another. As another example, a user may move their head or eyes to view different real or virtual objects within the user's FOR. If the user's gaze at a particular real or virtual object is longer than a threshold time, that real or virtual object may be selected as user input. In some implementations, the user's eye vergence-divergence can be tracked, and an accommodation / vergence-divergence model can be used to determine the user's eye accommodation state, which provides information about the depth plane the user is focusing on. In some implementations, the wearable system can use ray-casting techniques to determine real or virtual objects that are aligned with the user's head or eye pose. In various implementations, ray casting techniques can include casting a thin bundle of rays with substantially little lateral width, or casting rays with substantial lateral width (e.g., a cone or truncated cone).

[0095] The user interface may be projected by a display system as described herein (such as display 220 in FIG. 2 ). It may also be displayed using various other techniques, such as one or more projectors. A projector may project an image onto a physical object, such as a canvas or a sphere. Interactions with the user interface may be tracked using one or more cameras outside or part of the system (e.g., using inward-facing imaging system 462 or outward-facing imaging system 464).

[0096] 11 is a process flow diagram of an example method 1100 for interacting with a virtual user interface. Method 1100 may be performed by a wearable system described herein.

[0097] In block 1110, the wearable system may identify a particular UI. The type of UI may be provided by the user. The wearable system may identify that a particular UI needs to be populated based on user input (e.g., gestures, visual data, audio data, sensory data, direct commands, etc.). In block 1120, the wearable system may generate data for the virtual UI. For example, data associated with the UI's boundaries, general structure, shape, etc. may be generated. Additionally, the wearable system may determine map coordinates of the user's physical location so that the wearable system may display the UI in relation to the user's physical location. For example, if the UI is body-centered, the wearable system may determine coordinates of the user's physical position, head pose, or eye pose so that a ring UI may be displayed around the user or a planar UI may be displayed on a wall or in front of the user. If the UI is hand-centered, map coordinates of the user's hand may be determined. These map points may be derived through data received through an FOV camera, sensor input, or any other type of collected data.

[0098] In block 1130, the wearable system may send data from the cloud to the display, or the data may be sent from a local database to the display component. In block 1140, a UI is displayed to the user based on the sent data. For example, a light field display can project the virtual UI into one or both of the user's eyes. Once the virtual UI is generated, the wearable system may simply wait for commands from the user and generate more virtual content on the virtual UI in block 1150. For example, the UI may be a body-centered ring around the user's body. The wearable system may then wait for a command (such as a gesture, head or eye movement, input from a user input device, etc.) and, if recognized (block 1160), the virtual content associated with the command may be displayed to the user (block 1170). As an example, the wearable system may wait for a user's hand gesture before mixing multiple stem tracks.

[0099] Additional examples of wearable systems, UIs, and user experiences (UX) are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated herein by reference in its entirety.

[0100] (Overview of user interactions based on context information) The wearable system can support various user interactions with objects in the FOR based on the context information. For example, the wearable system can adjust the size of the cone projection opening as the user interacts with the object using the cone projection. As another example, the wearable system can adjust the amount of movement of a virtual object associated with actuation of a user input device based on the context information. Detailed examples of these interactions are provided below.

[0101] (Example Object) The user's FOR 1200 may contain a group of objects that can be perceived by the user through the wearable system. The objects in the user's FOR 1200 may be virtual and / or physical objects. Virtual objects may include operating system objects, such as a trash can for deleted files, a terminal for entering commands, a file manager for accessing files or directories, icons, menus, applications for audio or video streaming, notifications from the operating system, etc. Virtual objects may also include objects within an application, such as avatars, virtual objects in a game, graphics, or images. Some virtual objects can be both operating system objects and objects within an application. In some embodiments, the wearable system can add virtual elements to existing physical objects. For example, the wearable system may add a virtual menu associated with a television in a room, and the virtual menu may give the user options to turn on the television or change the channel using the wearable system.

[0102] Objects in the user's FOR can be part of a world map such as that described with reference to FIG. 9. Data associated with objects (e.g., location, semantic information, properties, etc.) can be stored in various data structures, such as arrays, lists, trees, hashes, graphs, etc. The index of each stored object may be determined, if applicable, by the object's location. For example, the data structure may index objects by a single coordinate, such as the object's distance from a reference position (e.g., distance to the left (or right) of the reference position, distance from the top (or bottom) of the reference position, or depth from the reference position). In some implementations, the wearable system includes a light field display capable of displaying virtual objects to the user at different depth planes. Interactive objects can be organized into multiple arrays located at different fixed depth planes.

[0103] A user can interact with a subset of the objects in the user's FOR. This subset of objects may sometimes be referred to as interactable objects. A user can interact with the objects using various techniques, such as by selecting the objects, moving the objects, opening a menu or toolbar associated with the objects, or selecting a new set of interactable objects. A user may interact with the interactable objects by actuating a user input device (e.g., see user input device 466 in FIG. 4 ) using hand gestures, such as clicking a mouse, tapping a touchpad, swiping a touchscreen, hovering over or touching a capacitive button, pressing a key on a keyboard or game controller (e.g., a five-way d-pad), pointing a joystick, wand, or totem toward the object, pressing a button on a remote control, or other interaction with the user input device. A user may also interact with the interactable objects using head, eye, or body posture, such as gazing at or pointing at an object for a period of time. These hand gestures and user postures can cause the wearable system to initiate a selection event, for example, performing a user interface action (displaying a menu associated with the target interactable object, performing a game action on an avatar in a game, etc.).

[0104] (Example of cone projection) As described herein, a user can use posture to interact with objects in their environment. For example, a user may look around a room and see a table, a chair, a wall, and a virtual television display on one of the walls. To determine the object the user is looking at, the wearable system may use a cone projection technique, which generally describes projecting an invisible cone in the direction the user is looking and identifying any objects that intersect with the cone. Cone projection may involve casting a single ray of light, with no lateral thickness, from the HMD (of the wearable system) toward a physical or virtual object. Cone projection with a single ray of light may also be referred to as ray projection.

[0105] Ray casting can use a collision detection agent to trace along the ray and identify whether and where any objects intersect with the ray. The wearable system can track the user's posture (e.g., body, head, or eye direction) using an inertial measurement unit (e.g., accelerometer), eye-tracking cameras, etc. to determine the direction the user is looking. The wearable system can use the user's posture to determine the direction in which to cast the ray. Ray casting techniques can also be used in conjunction with a user input device 466, such as a handheld multi-degree-of-freedom (DOF) input device. For example, a user can actuate the multi-DOF input device to anchor the size and / or length of the ray while the user moves around. As another example, rather than casting a ray from an HMD, the wearable system can cast the ray from the user input device.

[0106] In one embodiment, rather than projecting a ray with negligible thickness, the wearable system can project a cone with a non-negligible aperture (intersecting the central ray 1224). FIG. 12A illustrates an example of a cone projection with a non-negligible aperture. The cone projection can project a cone-shaped (or other shaped) volume 1220 with an adjustable aperture. The cone 1220 can be a geometric cone, having a proximal end 1228 a and a distal end 1228 b. The size of the aperture can correspond to the size of the distal end 1228 b of the cone. For example, a large aperture may correspond to a large surface area at the distal end 1228 b of the cone (e.g., the end away from the HMD, the user, or the user input device). As another example, the large aperture can correspond to a large diameter 1226 on the distal end 1228b of the cone 1220, while the small aperture can correspond to a small diameter 1226 on the distal end 1228b of the cone 1220. As will be described further with reference to FIG. 12A , the proximal end 1228a of the cone 1220 can have its origin at various locations, such as the center of the user's ARD (e.g., between the user's eyes), a point on one of the user's extremities (e.g., the hand, such as the fingers of a hand), a user input device or totem (e.g., a toy weapon) held or manipulated by the user.

[0107] The central ray 1224 can represent the direction of the cone. The direction of the cone can correspond to the user's body posture (head posture, hand gesture, etc.) or the user's gaze direction (also referred to as eye posture). Example 1206 in FIG. 12A illustrates cone projection with posture, and the wearable system can use the user's head posture or eye posture to determine the cone direction 1224. This example also illustrates a coordinate system for head posture. The head 1250 may have multiple degrees of freedom. As the head 1250 moves toward different directions, the head posture will change relative to the natural rest direction 1260. The coordinate system in FIG. 12A shows three angular degrees of freedom (e.g., yaw, pitch, and roll) that can be used to measure head posture relative to the head's natural rest state 1260. As illustrated in FIG. 12A, the head 1250 can tilt forward and backward (e.g., pitch), steer left and right (e.g., yaw), and tilt sideways (e.g., roll). In other implementations, other techniques or angle representations for measuring head pose can also be used, such as any other type of Euler angle system. The wearable system may use an IMU to determine the user's head pose. An inward-facing imaging system 462 (shown in FIG. 4) can be used to determine the user's eye pose.

[0108] Example 1204 shows another example of cone projection with pose, where the wearable system can determine the direction 1224 of the cone based on a user's hand gestures. In this example, the proximal end 1228a of the cone 1220 is at the tip of the user's finger 1214. As the user points their finger in a different location, the position of the cone 1220 (and central ray 1224) can be moved accordingly.

[0109] The direction of the cone can also correspond to the position or orientation of the user input device or the actuation of the user input device. For example, the direction of the cone can be based on a user drawing trajectory on the touch surface of the user input device. The user can move their finger forward on the touch surface to indicate that the direction of the cone is forward. Example 1202 illustrates another cone projection using a user input device. In this example, the proximal end 1228a is located at the tip of the weapon-shaped user input device 1212. As the user input device 1212 is moved, the cone 1220 and central ray 1224 can also move with the user input device 1212.

[0110] The direction of the cone can further be based on the position or orientation of the HMD. For example, the cone may be projected in a first direction when the HMD is tilted and in a second direction when the HMD is not tilted.

[0111] (Start of cone projection) The wearable system can initiate a cone projection when the user 1210 activates the user input device 466, for example, by clicking a mouse, tapping a touchpad, swiping a touchscreen, hovering over or touching a capacitive button, pressing a key on a keyboard or game controller (e.g., a 5-way d-pad), pointing a joystick, wand, or totem towards an object, pressing a button on a remote control, or other interaction with the user input device.

[0112] The wearable system may also initiate cone projection based on the user's posture 1210, such as a prolonged gaze in one direction or a hand gesture (e.g., waving a hand in front of the outward-facing imaging system 464). In some implementations, the wearable system can automatically initiate a cone projection event based on context information. For example, the wearable system may automatically initiate cone projection when the user is on the main page of the AR display. In another example, the wearable system can determine the relative position of objects in the user's gaze direction. If the wearable system determines that objects are located relatively far from each other, the wearable system may automatically initiate cone projection, so that the user does not need to move with precision to select an object in a group of sparsely located objects.

[0113] (Example properties of cones) The cone 1220 may have various properties, such as, for example, size, shape, or color. These properties may be displayed to the user so that the cone is perceptible to the user. In some cases, a portion of the cone 1220 may be displayed (e.g., the end of the cone, the surface of the cone, the central ray of the cone, etc.). In other embodiments, the cone 1220 may be a rectangular prism, a polyhedron, a pyramid, a truncated cone, etc. The distal end 1228b of the cone can have any cross-section, for example, circular, oval, polygonal, or irregular.

[0114] 12A, 12B, and 12C, the cone 1220 can have a proximal end 1228a and a distal end 1228b. The proximal end 1228a (also referred to as the zero point of the central ray 1224) can be associated with the location where the cone projection occurs. The proximal end 1228a may be anchored to a location in 3D space where a virtual cone appears to emanate from that location. The location may be a position on the user's head (such as between the user's eyes), a user input device functioning as a pointer (e.g., a 6DOF handheld controller or a 3DOF handheld controller), a fingertip (which can be detected by gesture recognition), or the like. For a handheld controller, the location where the proximal end 1228a is anchored may depend on the form factor of the device. For example, for a weapon-shaped controller 1212 (for use in a shooting game), the proximal end 1228a may be at the tip of the muzzle of the controller 1212. In this example, the proximal end 1228a of the cone originates from the center of the barrel, and the cone 1220, or (the central ray 1224 of the cone 1220), can be projected forward such that the center of the cone projection will be concentric with the barrel of the weapon-shaped controller 1212. The proximal end 1228a of the cone can, in various embodiments, be anchored anywhere within the user's environment.

[0115] Once the proximal end 1228a of the cone 1220 is anchored to a location, the orientation and movement of the cone 1220 may be based on the movement of an object associated with that location. For example, as described with reference to example 1206, once the cone is anchored to a user's head, the cone 1220 can move based on the user's head pose. As another example, in example 1202, once the cone 1220 is anchored to a user input device, the cone 1220 can be moved based on actuation of the user input device, for example, based on changes in the position or orientation of the user input device.

[0116] The distal end 1228b of the cone can extend until it reaches a termination threshold. The termination threshold may involve a collision between the cone and a virtual or physical object (e.g., a wall) in the environment. The termination threshold may also be based on a threshold distance. For example, the distal end 1228b can continue to extend away from the proximal end 1228a until the cone collides with the object or until the distance between the distal end 1228b and the proximal end 1228a reaches a threshold distance (e.g., 20 centimeters, 1 meter, 2 meters, 10 meters, etc.). In some embodiments, the cone can extend beyond the object even if a collision could occur between the cone and the object. For example, the distal end 1228b can extend through a real-world object (a table, chair, wall, etc.) and terminate when it reaches the termination threshold. Assuming the termination threshold is a wall of a virtual room located outside the user's current room, the wearable system can extend the cone beyond the current room until it reaches the surface of the virtual room. In some embodiments, a world mesh can be used to define the extent of one or more rooms. The wearable system can detect the presence of a termination threshold by determining whether a virtual cone intersects with a portion of the world mesh. Advantageously, in some embodiments, when the cone extends through a real-world object, the user can easily target the virtual object. As an example, the HMD can present a virtual hole in a physical wall through which the user can remotely interact with virtual content in other rooms, even if the user is not physically present in the other rooms. The HMD can determine objects in other rooms based on the world map illustrated in FIG. 9.

[0117] The cone 1220 may have a depth. The depth of the cone 1220 may be represented by the distance between the proximal end 1228a and the distal end 1228b of the cone 1220. The depth of the cone may be adjusted automatically by the wearable system, the user, or a combination thereof. For example, if the wearable system determines that an object is located far away from the user, the wearable system may increase the depth of the cone. In some implementations, the depth of the cone may be anchored to a depth plane. For example, a user may select the depth of the cone to be anchored to a depth plane within one meter of the user. As a result, during cone projection, the wearable system will not capture objects outside the one-meter boundary. In one embodiment, if the depth of the cone is anchored to a depth plane, the cone projection will only capture objects in that depth plane. Thus, the cone projection will not capture objects closer or farther from the user than the anchored depth plane. In addition to, or as an alternative to, setting the depth of the cone 1220, the wearable system can set the distal end 1228b at the depth plane so that the cone projection may enable user interaction with objects at or below the depth plane.

[0118] The wearable system can anchor the depth, proximal end 1228a, or distal end 1228b of the cone in response to detection of a hand gesture, body posture, gaze direction, actuation of a user input device, voice command, or other technique. In addition to or as an alternative to the examples described herein, the anchoring location or anchored depth of proximal end 1228a, distal end 1228b can be based on contextual information, such as the type of user interaction or the function of the object to which the cone is anchored. For example, proximal end 1228a can be anchored to the center of the user's head due to user usability and operability. As another example, when a user points to an object using a hand gesture or a user input device, proximal end 1228a can be anchored to the tip of the user's finger or the tip of the user input device to increase the accuracy of the user's pointing direction.

[0119] The cone 1220 may have a color. The color of the cone 1220 may depend on the user's preferences, the user's environment (virtual or physical), etc. For example, when the user is in a virtual jungle, which is a collection of trees with green leaves, the wearable system may provide a dark gray cone to increase the contrast between the cone and objects in the user's environment so that the user may have better visibility of the cone's location.

[0120] The wearable system can generate a visual representation of at least a portion of the cone for display to the user. The properties of the cone 1220 may be reflected in the visual representation of the cone 1220. The visual representation of the cone 1220 can correspond to at least a portion of the cone, such as the cone's aperture, the cone's surface, or a central ray. For example, if the virtual cone is a geometric cone, the visual representation of the virtual cone may include a gray geometric cone extending from a position between the user's eyes. As another example, the visual representation may include a portion of the cone that interacts with real or virtual content. Assuming the virtual cone is a geometric cone, the visual representation may include a circular pattern representing the base of the geometric cone, as the base of the geometric cone may be used to target and select virtual objects. In one embodiment, the visual representation is triggered based on a user interface action. As an example, the visual representation may be associated with the state of the object. The wearable system can present a visual representation when the object changes from a resting state or a hovering state (where the object can be moved or selected). The wearable system may further hide the visual representation when the object changes from the hovering state to the selected state. In some implementations, when the object is in the hovering state, the wearable system may receive input from the user input device (in addition to or as an alternative to the cone projection) and may allow the user to select the virtual object using the user input device when the object is in the hovering state.

[0121] In some embodiments, the cone 1220 may be invisible to the user. The wearable system may assign a focus indicator to one or more objects to indicate the direction and / or location of the cone. For example, the wearable system may assign a focus indicator to an object that is in front of the user and intersects the user's direction of gaze. The focus indicator may comprise a halo, color, a perceived size or depth change (e.g., causing the target object to appear closer and / or larger when selected), a shape change in a cursor sprite graphic (e.g., the cursor changes from a circle to an arrow), or other audible, tactile, or visual effect that draws the user's attention.

[0122] The cone 1220 can have an aperture that intersects the central ray 1224. In some embodiments, the central ray 1224 is not visible to the user 1210. The size of the aperture can correspond to the size of the distal end 1228b of the cone. For example, a large aperture can correspond to a large diameter 1226 on the distal end 1228b of the cone 1220, while a small aperture can correspond to a small diameter 1226 on the distal end 1228b of the cone 1220.

[0123] 12B and 12C, the aperture can be adjusted by the user, the wearable system, or a combination thereof. For example, the user may adjust the aperture through a user interface action, such as selecting an aperture option shown on the AR display. The user may also adjust the aperture by actuating a user input device, for example, by scrolling the user input device or by pressing a button to anchor the aperture size. In addition to, or as an alternative to, input from the user, the wearable system can update the aperture size based on one or more context factors described below.

[0124] (Example of a cone projection with dynamically updated aperture) Cone projection can be used to increase accuracy when interacting with objects in the user's environment, especially when those objects are located at a distance where small amounts of movement from the user can translate into large movements of light rays. Cone projection can also be used to reduce the amount of movement required from the user to overlap the cone with one or more virtual objects. In some implementations, the user can manually update the cone opening to improve the speed and accuracy of selecting a target object, for example, by using a narrower cone when many objects are present and a wider cone when fewer objects are present. In other implementations, the wearable system can determine context coefficients associated with objects in the user's environment in addition to or as an alternative to manual updates and enable automatic cone updates, which can advantageously make it easier for the user to interact with objects in the environment because less user input is required.

[0125] 12B and 12C provide an example of a cone projection onto a group 1230 of objects (e.g., 1230a, 1230b, 1230c, 1230d, 1230e) in a user's FOR 1200 (at least some of these objects are within the user's FOV). The objects may be virtual and / or physical objects. During cone projection, the wearable system projects a cone (visible or invisible to the user) 1220 in a direction and can identify any objects that intersect with the cone 1220. For example, in FIG. 12B, object 1230a (shown in bold) intersects with the cone 1220. In FIG. 12C, objects 1230d and 1230e (shown in bold) intersect with the cone 1220. Objects 1230b, 1230c (shown in gray) lie outside the cone 1220 and do not intersect with it.

[0126] The wearable system can automatically update the aperture based on the context information. The context information may include information related to the user's environment (e.g., lighting conditions in the user's virtual or physical environment), user preferences, the user's physical conditions (e.g., whether the user is nearsighted), information associated with objects in the user's environment, such as the type of objects in the user's environment (e.g., physical or virtual), or the layout of objects (e.g., object density, object location and size, etc.), characteristics of objects with which the user interacts (e.g., object function, type of user interface action supported by the object, etc.), combinations thereof, or the like. Density can be measured in various ways, such as, for example, the number of objects per projected area, the number of objects per solid angle, etc. Density may also be expressed in other ways, such as, for example, the spacing between neighboring objects (smaller spacing reflects increased density). The wearable system can use the object location information to determine the layout and density of objects in an area. As shown in FIG. 12B, the wearable system may determine that a group 1230 of objects is densely packed. The wearable system may therefore use a cone 1220 with a smaller aperture. In Figure 12C, objects 1230d and 1230c are located relatively far from one another, so the wearable system may use a cone 1220 with a larger aperture (compared to the cone in Figure 12B). Additional details regarding calculating object density and adjusting aperture size based on density are further described in Figures 12D-12G.

[0127] The wearable system can dynamically update the aperture (e.g., size or shape) based on the user's posture. For example, the user may initially be looking at the group of objects 1230 in FIG. 12B, but as the user turns their head, the user may end up seeing the group of objects in FIG. 12C (where the objects are sparsely positioned relative to each other). As a result, the wearable system may increase the size of the aperture (e.g., as shown by the change in the cone's aperture between FIG. 12B and FIG. 12C). Similarly, if the user turns their head back and sees the group of objects 1230 in FIG. 12B, the wearable system may decrease the size of the aperture.

[0128] Additionally or alternatively, the wearable system can update the aperture size based on the user's preferences. For example, if the user prefers to select large groups of items simultaneously, the wearable system may increase the size of the aperture.

[0129] As another example of dynamically updating the aperture based on context information, if the user is in a dark environment or is nearsighted, the wearable system may increase the size of the aperture to make it easier for the user to capture an object. In one implementation, a first cone projection can capture multiple objects. The wearable system can perform a second cone projection to further select a target object among the captured objects. The wearable system can also allow the user to select a target object from the captured objects using a body posture or a user input device. The object selection process can be a recursive process, and one, two, three, or more cone projections may be performed to select a target object.

[0130] (Example of dynamic update of aperture based on object density) As described with reference to Figures 12B and 12C, the aperture of the cone can be dynamically updated during cone projection based on the density of objects in the user's FOR. Figures 12D, 12E, 12F, and 12G illustrate examples of dynamically adjusting the aperture based on the density of objects. Figure 12D illustrates a contour map associated with the density of objects in the user's FOR 1208. Virtual objects 1271 are represented by small textured dots. The density of virtual objects is reflected by the amount of contour lines in a given region. For example, contour lines are closer to each other in region 1272, representing a higher density of objects in region 1272. As another example, contour lines in region 1278 are relatively sparse. Therefore, the density of objects in region 1278 is lower.

[0131] The visual representation of aperture 1270 is illustrated in FIG. 12D as a shaded circle. The visual representation in this example may correspond to distal end 1228b of virtual cone 1220. The aperture size may vary based on the density of objects within a given region. For example, the aperture size may depend on the density of objects where the center of the circle is located. As illustrated in FIG. 12D , when the aperture is within region 1272, the size of aperture 1270 may decrease (as indicated by the relatively small size of the aperture circle). However, when the user starts from region 1276 within FOR 1208, the size of aperture 1270 is slightly larger in region 1272. When the user further changes their head pose and looks at region 1274, the size of the aperture becomes larger than the size in region 1276 because the density of objects in region 1274 is lower than that in region 1276. As yet another example, in region 1278, the size of aperture 1270 would increase due to the absence of any objects within region 1278 of FOR 1208. Density is illustrated using a contour map in these examples, but density can also be determined using a heat map, surface plot, or other graphical or numerical representation. In general, the term "contour map" also includes these other types of density representations (in 1D, 2D, or 3D). Furthermore, contour maps are generally not presented to the user, but may be calculated and used by the ARD processor to dynamically determine the nature of the cone. Contour maps may be dynamically updated as physical or virtual objects move within the user's FOV or FOR.

[0132] Various techniques can be employed to calculate the density of objects. As one example, density can be calculated by counting all of the virtual objects within the user's FOV. The number of virtual objects may be used as input to a function that defines the size of the aperture based on the number of virtual objects in the FOV. Image 1282a in FIG. 12E shows an FOV with three virtual objects represented by a circle, an oval, and a triangle, and a virtual representation of aperture 1280 illustrated using a textured circle. However, if the number of virtual objects decreases from three (in image 1282a) to two (in image 1282b), the size of aperture 1280 can increase accordingly. The wearable system can calculate the amount of increase using function 1288 in FIG. 12F. In this figure, the size of the aperture is represented by the y-axis 1286b, while the number (or density) of virtual objects in the FOV is represented by the x-axis 1286a. As shown, as the number of virtual objects increases (e.g., density increases), the size of the aperture decreases according to function 1288. In one embodiment, the minimum aperture size is zero, which reduces the cone to a single ray. While function 1288 is illustrated as a linear function, any other type of function, such as one or more power-law functions, may also be used. In some embodiments, function 1288 may include one or more threshold conditions. For example, when the density of objects reaches a certain low threshold, the size of aperture 1280 will no longer increase, even if the density of objects may further decrease. On the other hand, when the density of objects reaches a certain high threshold, the size of aperture 1280 will no longer decrease, even if the density of objects may further increase. However, when the density is between the low and high thresholds, the aperture size may decrease, for example, according to an exponential function.

[0133] 12G illustrates another exemplary technique for calculating density. For example, in addition to or as an alternative to calculating the number of virtual objects in the FOV, the wearable system can calculate the percentage of the FOV covered by virtual objects. Images 1292a and 1292b illustrate adjustment of aperture size based on the number of objects in the FOV. As illustrated in this example, the percentage of the FOV covered by virtual objects differs between images 1292a and 1292b (objects in image 1292a are more sparsely positioned), but the size of aperture 1280 does not change in these two images because the number of objects (e.g., three virtual objects) is the same across images 1292a and 1292b. In contrast, images 1294a and 1294b illustrate adjustment of aperture size based on the percentage of the FOV covered by virtual objects. As shown in image 1294a, aperture 1280 will increase in size as a lower percentage of the FOV is covered by the virtual object (as opposed to the remaining identical one in image 1292a).

[0134] (Example of a collision) The wearable system can determine whether one or more objects collide with the cone during the cone projection. The wearable system may detect collisions using a collision detection agent. For example, the collision detection agent can identify objects that intersect with the surface of the cone and / or identify objects that are inside the cone. The wearable system can make such identification based on the volume and location of the cone and the object's location information (as stored in the world map described with reference to FIG. 9 ). Objects in the user's environment may be associated with a mesh (also referred to as a world mesh). The collision detection agent can determine whether a portion of the cone overlaps with the object's mesh to detect a collision. In some implementations, the wearable system may be configured to only detect collisions between the cone and objects on a certain depth plane.

[0135] The wearable system may provide a focus indicator for objects that collide with the cone. For example, in FIGS. 12B and 12C, the focus indicator may be a red highlight around all or part of the object. Thus, in FIG. 12B, when the wearable system determines that object 1230a intersects with cone 1220, the wearable system may display a red highlight around object 1230a to user 1210. Similarly, in FIG. 12C, the wearable system identifies objects 1230e and 1230d as objects that intersect with cone 1220. The wearable system may provide a red highlight around objects 1230d and 1230e.

[0136] When a collision involves multiple objects, the wearable system may present user interface elements to select one or more objects from among the multiple objects. For example, the wearable system may provide a focus indicator that indicates the target object with which the user is currently interacting. The user may use hand gestures to actuate the user input device to move the focus indicator to another target object.

[0137] In some embodiments, an object may be behind another object in the user's 3D environment (e.g., a nearby object at least partially occludes a more distant object). Advantageously, the wearable system may apply disambiguation techniques during cone projection (e.g., to determine occluded objects, determine depth order or position between occluded objects, etc.) to capture both objects in front and objects in back. For example, a paper shredder may be behind a computer in the user's room. The user may not be able to see the shredder (because it is occluded by the computer), but the wearable system can project a cone in the direction of the computer and detect a collision involving both the shredder and the computer (because both are in the wearable system's world map). The wearable system may display a pop-up menu and offer the user the option of selecting either the shredder or the computer, or the wearable system may use context information to determine which object to select (e.g., if the user is attempting to delete a document, the system may select the paper shredder). In some implementations, the wearable system may be configured to capture objects only in front of it. In this example, the wearable system would only detect collisions between the cone and the shredder.

[0138] In response to detecting a collision, the wearable system may enable the user to interact with the interactable object in various ways, such as, for example, selecting the object, moving the object, opening a menu or toolbar associated with the object, or performing a game action on an avatar in the game. The user may interact with the interactable object through posture (e.g., head, body posture), hand gestures, input from a user input device, combinations thereof, or the like. For example, when a cone collides with multiple interactable objects, the user may actuate a user input device to select from among the multiple interactable objects.

[0139] (Example Process for Dynamically Updating Apertures) 13 is a flowchart of an exemplary process for selecting an object using a cone projection with a dynamically adjustable aperture. This process 1300 can be implemented by a wearable system (shown in FIGS. 2 and 4).

[0140] In block 1310, the wearable system can initiate a cone projection. The cone projection can be triggered by the user's posture or a hand gesture on the user input device. For example, the cone projection may be triggered by a click on the user input device and / or by the user looking in a certain direction for an extended period of time. As shown in block 1320, the wearable system can analyze salient features of the user's environment, such as the type of object, the layout of the objects (physical or virtual), the location of the objects, the size of the objects, the density of the objects, the distance between the objects and the user, etc. For example, the wearable system can calculate the density of objects in the user's direction of gaze by determining the number of objects in front of the user and the size of the objects. The salient features of the environment may be part of the context information described herein.

[0141] In block 1330, the wearable system can adjust the size of the aperture based on the contextual information. As discussed with reference to FIGS. 12B and 12C, the wearable system can increase the aperture size when objects are sparsely located and / or when no obstacles are present. A large aperture size can correspond to a larger diameter 1226 on the distal end 1228b of the cone 1220. As the user moves around and / or changes the environment, the wearable system may update the size of the aperture based on the contextual information. The contextual information can be combined with other information, such as user preferences, user posture, and cone characteristics (e.g., depth, color, location, etc.), to determine and update the aperture.

[0142] The wearable system may render a cone projection visualization at block 1340. The cone projection visualization may include a cone with a non-negligible aperture. As described with reference to Figures 12A, 12B, and 12C, the cone may have various sizes, shapes, or colors.

[0143] In block 1350, the wearable system can transform the cone projection and scan for collisions. For example, the wearable system can transform the movement of the cone using techniques described with reference to FIGS. 16-18. The wearable system can also determine whether the cone is colliding with one or more objects by calculating the position of the cone relative to the positions of objects in the user's environment. As discussed with reference to FIGS. 12A, 12B, and 12C, one or more objects can intersect the surface of the cone or be within the cone.

[0144] If the wearable system does not detect a collision, then in block 1360, the wearable system repeats block 1320, where the wearable system analyzes the user's environment and may update the aperture based on the user's environment (as shown in block 1330). If the wearable system detects a collision, the wearable system may indicate the collision, for example, by placing a focus indicator on the hit object. If the cone collides with multiple interactable objects, the wearable system may use disambiguation techniques to capture one or more occluded objects.

[0145] In block 1380, the user may optionally interact with the collided object in various ways, as described with reference to Figures 12A, 12B, and 12C. For example, the user may select the object, open a menu associated with the object, move the object, etc.

[0146] 14 is another flowchart of an exemplary process for selecting an object using a cone projection with a dynamically adjustable aperture. This process 1400 can be performed by a wearable system (shown in FIGS. 2 and 4). In block 1410, the wearable system determines a group of objects in the user's FOR.

[0147] In block 1420, the wearable system may initiate a cone projection onto a group of objects in the user's FOR. The wearable system may initiate the cone projection based on input from a user input device (e.g., a swing of a wand) or posture (e.g., a hand gesture). The wearable system may also automatically trigger the cone projection based on certain conditions. For example, the wearable system may automatically initiate the cone projection when the user is on the main display of the wearable system. The cone projection may use a virtual cone, which may have a central ray and an aperture that intersects the central ray. The central ray may be based on the user's gaze direction.

[0148] In block 1430, the wearable system can determine a user's posture. The user's posture may be head, eye, or body posture, alone or in combination. The wearable system can determine the user's FOV based on the user's posture. The FOV can include the portion of the FOR perceived by the user at a given time.

[0149] Based on the user's FOV, in block 1440, the wearable system may determine a subgroup of objects within the user's FOV. As the user's FOV changes, the objects within the user's FOV may also change. The wearable system may be configured to analyze context information of the objects within the user's FOV. For example, the wearable system may determine the density of objects based on the size and location of the objects within the FOV.

[0150] In block 1450, the wearable system may determine the size of an aperture for the cone projection event. The size of the aperture may be determined based on contextual information. For example, if the wearable system determines that the density of objects is high, the wearable system may use a cone with a small aperture to increase the accuracy of user interaction. In some embodiments, the wearable system may also adjust the depth of the cone. For example, if the wearable system determines that all of the objects are located far from the user, the wearable system may extend the cone to a depth plane that has these objects. Similarly, if the wearable system determines that the objects are located close to the user, the wearable system may contract the depth of the cone.

[0151] The wearable system may generate a visual representation of the cone projection in block 1460. The visual representation of the cone projection may incorporate properties of a cone, as described with reference to FIGS. 12B and 12C. For example, the wearable system may display a virtual cone with color, shape, and depth. The location of the virtual cone may be associated with the user's head pose, body posture, or gaze direction. The cone may be a geometric cone, a cuboid, a polyhedron, a pyramid, a truncated cone, or other three-dimensional shape that may or may not be a regular shape.

[0152] As the user moves around, the cone can also move with the user. As the user moves around, the amount of movement of the cone corresponding to the user's movement can also be calculated based on context information, as described further with reference to Figures 15-18. For example, if the density of objects in the FOV is low, a small movement of the user can result in a large movement of the cone. On the other hand, if the density is high, that same movement can result in a smaller movement of the cone, thereby allowing for more refined interaction with the objects.

[0153] FIG. 15 is an example process 1500 for cone projection with a dynamically adjustable aperture. Process 1500 in FIG. 15 can be implemented by a wearable system (shown in FIGS. 2 and 4). In block 1510, the wearable system can determine contextual information within the user's environment. The contextual information may include information of the user's environment and / or information associated with the objects, such as the layout of objects, the density of objects, the distance between the objects and the user, etc.

[0154] In block 1520, the wearable system can project a cone with a dynamically adjustable aperture based on context information. For example, when the density of objects is low, the aperture may be large.

[0155] At block 1530, the wearable system may detect a collision between the object and the cone. In some embodiments, the wearable system may detect a collision based on the location of the object and the location of the cone. A collision is detected if at least a portion of the object overlaps the cone. In some embodiments, the cone may collide with multiple objects. The wearable system may apply a disambiguation technique to capture one or more occluded objects. As a result, the wearable system may detect a collision between the cone and the occluded object.

[0156] In response to detecting a collision, the wearable system may assign a focus indicator to the object colliding with the cone. The wearable system may also provide user interface options, such as selecting an object from the collided objects. In block 1540, the wearable system may be configured to receive user interaction with the collided object. For example, the user may move the object, open a menu associated with the object, select the object, etc.

[0157] (Overview of Transforming Movement Based on Contextual Information) In addition to, or as an alternative to, adjusting the opening of the cone during cone projection, contextual information can also be used to translate movements associated with a user input device or part of the user's body (e.g., a change in the user's posture) into user interface actions, such as moving a virtual object.

[0158] A user can move a virtual object or transport a focus indicator by actuating a user input device and / or using postures such as head, eye, or body posture. As is evident in the AR / VR / MR world, virtual object movement does not refer to actual physical movement of the virtual object because the virtual object is a computer-generated image and not a physical object. Virtual object movement refers to the apparent movement of the virtual object as displayed to the user by the AR or VR system.

[0159] FIG. 16 diagrammatically illustrates an example of moving a virtual object using a user input device. For example, a user may hold and move a virtual object by selecting it using the user input device and move the virtual object by physically moving the user input device 466. The user input device 466 may initially be at a first position 1610a. The user 1210 may select a target virtual object 1640 located at a first position 1610b by actuating the user input device 466 (e.g., by actuating a touch-sensitive pad on the device). The target virtual object 1640 can be any type of virtual object that can be displayed and moved by the wearable system. For example, the virtual object may be an avatar, a user interface element (e.g., a virtual display), or any type of graphical element displayed by the wearable system (e.g., a focus indicator, etc.). The user 1210 can move the target virtual object from a first position 1610b to a second position 1620b by moving the user input device 466 along the trajectory 1650b. However, because the target virtual object may be far from the user, the user may need to move the user input device a long distance before the target virtual object reaches its desired location, which may cause the user to use large hand and arm movements and ultimately lead to user fatigue.

[0160] Embodiments of a wearable system may provide techniques for rapidly and efficiently moving far-distance virtual objects by moving the virtual object by an amount based on controller movement and a multiplier that tends to increase with distance to the virtual object. Such embodiments may advantageously allow a user to move far-distance virtual objects using shorter hand and arm movements, thereby reducing user fatigue.

[0161] The wearable system can calculate a multiplier for mapping movement of the user input device to movement of the target virtual object. The movement of the target virtual object may be based on movement of the input controller and the multiplier. For example, the amount of movement of the target virtual object may be equal to the amount of movement of the input controller multiplied by the multiplier. This may reduce the amount the user needs to move before the target virtual object reaches a desired location. For example, as shown in FIG. 16 , the wearable system may determine a multiplier that allows the user to move the user input device along trajectory 1650a (shorter than trajectory 1650b) to move the virtual object from position 1620b to position 1610b.

[0162] Additionally or alternatively, the user 1210 can use head pose to move virtual objects. For example, as shown in FIG. 16 , the head may have multiple degrees of freedom. As the head moves toward different directions, the head pose will change relative to the natural resting direction 1260. The example coordinate system in FIG. 16 shows three angular degrees of freedom (e.g., yaw, pitch, and roll) that can be used to measure head pose relative to the head's natural resting state 1260. As illustrated in FIG. 16 , the head can tilt forward and backward (e.g., pitch), turn left and right (e.g., yaw), and tilt sideways (e.g., roll). In other implementations, other techniques or angular representations for measuring head pose, e.g., any other type of Euler angle system, can also be used. A wearable system (e.g., see wearable system 200 in FIG. 2 and wearable system 400 in FIG. 4 ) may be used to determine the user's head pose using, for example, an accelerometer, an inertial measurement unit, etc., as discussed herein. The wearable system may also move virtual objects based on eye pose (e.g., as measured by an eye-tracking camera) and head pose. For example, a user may select a virtual object by gazing at the object for an extended period of time and use head pose to move the selected object. The techniques for mapping user input device movement described herein can also be applied to changes in the user's head, eye, and / or body pose; i.e., the amount of movement of the virtual object is a multiplier times the amount of physical movement of the user's body (e.g., eyes, head, hands, etc.).

[0163] (Example of distance-based multiplier) As described above, the wearable system can calculate a multiplier for mapping movement of the user input device to movement of the target virtual object. The multiplier may be calculated based on context information, such as the distance between the user and the target virtual object. For example, as shown in FIG. 16, the multiplier may be calculated using the distance between the position of the head of the user 1210 and the position of the virtual object 1640.

[0164] FIG. 17 graphically illustrates an example of a multiplier as a function of distance. As shown in FIG. 17, axis 1704 indicates the magnitude of the multiplier. Axis 1702 illustrates various distances (e.g., in feet or meters) between two endpoints. The endpoints may be determined in various ways. For example, one endpoint may be the user's position (e.g., measured from the user's ARD) or the location of a user input device. The other endpoint may be the position of a target virtual object.

[0165] The distance between the user and the virtual object may change as the endpoints for calculating the distance change. For example, the user and / or the virtual object may move around. The user 1210 may actuate a user input device to pull the virtual object closer. During this process, the multiplier may change based on various coefficients described herein. For example, the multiplier may decrease as the virtual object moves closer to the user or increase as the virtual object moves farther away from the user.

[0166] Curves 1710, 1720, and 1730 illustrate examples of the relationship between the multiplier and distance. As shown by curve 1710, the multiplier may be equal to 1 when the distance is less than threshold 1752. Curve 1710 shows a linear relationship between distance and the multiplier between thresholds 1752 and 1754. As described with reference to FIG. 16 , this proportional linear relationship may allow the wearable system to map small changes in the position of the user input device to large changes in position (up to threshold 1754) for objects located more distantly. Curve 1710 reaches its maximum value at threshold 1754, and therefore, any further increases in distance will not change the magnitude of the multiplier. This may prevent very far-away virtual objects from moving very large distances in response to small movements of the user input device.

[0167] The thresholding of the multiplier in curve 1710 is optional (at either or both thresholds 1752, 1754). The wearable system may also generate the multiplier without using multiple thresholds.

[0168] To enable more precise one-to-one manipulation, one exemplary threshold may be the user's reach. The user's reach may be an adjustable parameter that can be set by the user or the HMD (to account for users with different reach ranges). In various implementations, the user's reach may be within a range of approximately 10 cm to approximately 1.5 m. Referring to FIG. 16 , for example, if the target virtual object is within reach, as the user 1210 moves the user input device 466 along a trajectory 1650 a from position 1610 a to position 1620 a, the target virtual object may also move along the trajectory 1650 a. If the target virtual object 1640 moves beyond reach, the multiplier may increase. For example, in FIG. 16, if the target virtual object 1640 is initially at position 1610b, as the user input device 466 moves from position 1610a to position 1620a, the target virtual object 1640 moves from position 1610b to position 1620b, thereby moving a distance amount greater than that of the user input device 466.

[0169] The relationship between distance and multiplier is not limited to a linear relationship. Rather, it may be determined based on various algorithms and / or coefficients. For example, as shown in FIG. 17, curve 1720 may be generated using one or more power-law functions between distance and multiplier, e.g., the multiplier is proportional to a certain power of the distance. The power may be 0.5, 1.5, or 2. Similarly, curve 1730 may be generated based on user preferences, with the multiplier equal to 1 when the object is within a user-adjustable threshold distance.

[0170] As an example, the movement (e.g., angular movement) of the virtual object may be represented by the variable delta_object, and the movement of the user input device may be represented by the variable delta_input. The deltas are related by multipliers. [ka]

[0171] A sensor in the user input device or an outward-facing camera of the ARD may be used to measure delta_input. The multiplier as a function of distance d can be determined from a lookup table, a functional form (e.g., a power law), or a curve (see, e.g., an example in FIG. 17). In some implementations, the distance may be normalized by the distance from the user to the input device. For example, the distance d may be determined as follows: [ka] In equation (2), the normalized distance is dimensionless and equals 1 when the object is at the distance of the input device. As mentioned above, the multiplier may be set to 1 for objects within reach (e.g., within the distance from the camera to the input device). Thus, equation (2) allows the wearable system to dynamically adjust the hand length distance based on where the user holds the input device. An example power law multiplier may be: [ka] where the power p is, for example, 1 (linear), 2 (quadratic), or any other integer or real number.

[0172] (Other example multipliers) The multiplier can also be calculated using other factors, such as contextual information about the user's physical and / or virtual environment. For example, if a virtual object is located within a dense cluster of objects, the wearable system may use a smaller multiplier to increase the accuracy of placing the object. Contextual information may also include the nature of the virtual object. For example, in a driving game, the wearable system may provide a larger multiplier for superior cars and a smaller multiplier for mediocre cars.

[0173] The multiplier may depend on the direction of movement. For example, in the xyz coordinate system shown in FIG. 6, the multiplier for the x-axis may be different from the multiplier for the z-axis. Referring to FIG. 16, instead of moving virtual object 1640 from 1610b to 1620b, user 1210 may want to pull virtual object 1640 closer to him / her. In this situation, the wearable system may use a multiplier that is smaller than the multiplier that moves virtual object 1640 from 1610b to 1620b. In this way, virtual object 1640 may not suddenly appear very close to the user.

[0174] The wearable system may allow a user to configure the multiplier. For example, the wearable system may provide the user with several options for selecting a multiplier. A user who prefers slow movement may select a multiplier with a small magnitude. The user may also provide certain coefficients and / or coefficient importances that the wearable system will use to automatically determine the multiplier. For example, the user may set a weighting for distance higher than a weighting associated with the nature of the virtual object. Thus, distance will have a greater influence on the magnitude of the multiplier than the nature of the virtual object. Furthermore, as described with reference to FIG. 17 , the multiplier may have one or more thresholds. One or more of the thresholds may be calculated based on the values ​​of a set of coefficients (e.g., coefficients determined from context information). In an embodiment, one threshold may be calculated based on one set of coefficients, while another threshold may be calculated based on another set of coefficients (which may not overlap with the first set of coefficients).

[0175] (Example Application of Multipliers) 16 and 17, the wearable system can apply a multiplier to map movement of the user input device to movement of the virtual object. The movement may include a speed, acceleration, or position change (rotation, movement from one location to another, etc.). For example, the wearable system may be configured to cause the virtual object to move faster when the virtual object is located further away.

[0176] As another example, a multiplier may also be used to determine the acceleration of a virtual object. When a virtual object is far from a user, the virtual object may have a large initial acceleration when the user activates a user input device to move the virtual object. In some embodiments, the multiplier for acceleration may peak or decrease after a certain threshold. For example, to avoid moving the object too quickly, the wearable system may decrease the multiplier for acceleration when the virtual object reaches the midpoint of its trajectory or when the velocity of the virtual object reaches a threshold.

[0177] In some implementations, the wearable system may use a focus indicator to indicate the current position of the user input device and / or the user's posture (e.g., head, body, eye posture). A multiplier may be applied to indicate position changes of the focus indicator. For example, the wearable system may display a virtual cone during cone projection (see the description of cone projection in Figures 12-15). When the depth of the cone is set to a far-field location, the wearable system may apply a large multiplier. Thus, as the user moves around, the virtual cone may move a long distance.

[0178] Additionally or alternatively, the wearable system can map movement of the user input device to movement of multiple virtual objects. For example, in a virtual game, a player can move a group of virtual soldiers together by actuating a user input device. The wearable system can translate movement of the user input device into movement of the group of virtual soldiers by applying a multiplier to the group of virtual soldiers together and / or by applying a multiplier to each virtual soldier in the group.

[0179] (Example Process for Moving a Virtual Object) 18 illustrates a flowchart of an example process for moving a virtual object in response to movement of a user input device. Process 1800 can be implemented by the wearable systems shown in FIGS. 2 and 4.

[0180] In block 1810, the wearable system receives a selection of a target virtual object. The virtual object may be displayed by the wearable system at a first position in 3D space. The user may select the target virtual object by actuating a user input device. Additionally, or alternatively, the wearable system may be configured to support the user in moving the target virtual object using various body, head, or eye postures. For example, the user may select the target virtual object by pointing their finger toward it and move the target virtual object by moving their arm.

[0181] In block 1820, the wearable system may receive an indication of movement relative to the target virtual object. The wearable system may receive such an indication from a user input device. The wearable system may also receive such an indication from a sensor (e.g., an outward-facing imaging system 464, etc.), which can determine a change in the user's posture. The indication can be a change in the trajectory of movement or the position of a portion of the user's body or the user input device.

[0182] In block 1830, the wearable system determines the value of a multiplier to be applied based on the context information described herein. For example, the wearable system may calculate the multiplier based on the distance between the object and the user input device, and the multiplier may increase with increasing distance of the target virtual object (at least over a range of distances from the user input device; see, e.g., the example in equation (3)). In some embodiments, the multiplier is a non-decreasing function of the distance between the object and the user input device.

[0183] As shown in block 1840, this multiplier may be used to calculate the amount of movement for the target virtual object. For example, if the multiplier is calculated using the distance between the object and the user input device, the multiplier may be larger for a more distant target virtual object. The wearable system may use equation (3) to relate the amount of movement of the input device and the multiplier to determine the amount of movement of the target virtual object. The trajectory of the movement of the target virtual object may be calculated using other coefficients along with the multiplier. For example, the wearable system may calculate the trajectory based on the user's environment. When another object is present along the path of the target virtual object, the wearable system may be configured to move the target virtual object to avoid a collision with the other object.

[0184] In block 1850, the wearable system may display movement of the target virtual object based on the calculated trajectory or multiplier. For example, the wearable system may calculate a second position in 3D space based on the amount of movement calculated in block 1840. The wearable system may thus display the target virtual object at the second position. As discussed with reference to FIG. 16 , the wearable system may also be configured to display movement of a visual focus indicator using a multiplier.

[0185] (Additional Embodiments) In a first aspect, a method for selecting a virtual object located in three-dimensional (3D) space, the method being performed under control of an augmented reality (AR) system comprising computer hardware, the AR system configured to enable user interaction with objects within the user's field of view (FOR), the FOR comprising a portion of the user's surrounding environment perceivable by the user via the AR system, the method comprising: determining a group of objects within the user's FOR; determining a pose of the user; and initiating a cone projection onto the group of objects, the cone projection comprising projecting a virtual cone with an aperture in a direction based at least in part on the user's pose; analyzing contextual information associated with a subgroup of objects within the group of objects; updating the aperture for the cone projection event based at least in part on the contextual information; and rendering a visual representation of the cone projection.

[0186] In a second aspect, the method of aspect 1, wherein the subgroup of objects is within a user's field of view (FOV), the FOV comprising a portion of the FOR that is perceivable by the user at a given time via the AR system.

[0187] In a third aspect, the method of aspect 1 or 2, wherein the contextual information includes one or more of the type, layout, location, size, or density of one or more objects in the subgroup of objects.

[0188] In a fourth aspect, the method of aspect 3, wherein the context information further comprises user preferences.

[0189] In a fifth aspect, the method of any one of aspects 1-4 further comprises detecting collisions between the cone and one or more objects.

[0190] In a sixth aspect, the method of aspect 5, wherein the one or more objects include an interactable object.

[0191] In a seventh aspect, the method of aspect 6, wherein in response to detecting a collision with the interactable object, the method further includes performing an action on the interactable object.

[0192] In an eighth aspect, the method of aspect 7, wherein the action includes one or more of selecting an interactable object, moving an interactable object, or opening a menu associated with the interactable object.

[0193] In a ninth aspect, the method of aspect 5 or 6 further comprises applying an occlusion disambiguation technique to one or more objects collided with the cone.

[0194] In a tenth aspect, the method of any one of aspects 1-9 further comprises updating the opening of the cone based at least in part on changes in the user's posture.

[0195] In an eleventh aspect, the method of any one of aspects 1-10, wherein the cone has a certain shape.

[0196] In a twelfth aspect, the method of aspect 11, wherein the shape comprises one or more of a geometric cone projection, a rectangular prism, a polyhedron, a pyramid, or a truncated cone.

[0197] In a thirteenth aspect, the method of any one of aspects 1-12, wherein the cone has a central ray.

[0198] In a fourteenth aspect, the method of aspect 13, wherein the central ray is determined at least in part based on the user's posture.

[0199] In a fifteenth aspect, the method of any one of aspects 13 or 14, wherein the aperture intersects the central ray.

[0200] In a sixteenth aspect, the method of any one of aspects 1-15 further comprises disambiguating objects that collide with the cone.

[0201] In a seventeenth aspect, an augmented reality system configured to implement a method according to any one of aspects 1-16.

[0202] In an eighteenth aspect, a method for transforming a virtual object located in three-dimensional (3D) space, the method being performed under control of an augmented reality (AR) system comprising computer hardware and a user input device, the AR system being configured to enable user interaction with a virtual object within a user's field of view (FOR), the FOR including a portion of the user's surrounding environment perceivable by the user via the AR system, the virtual object being presented via the AR system for display to the user, the method including: determining a group of virtual objects within the user's FOR; receiving a selection of a target virtual object within the group of virtual objects in the user's FOR; calculating a distance to the target virtual object; determining a multiplier based at least in part on the distance to the target virtual object; receiving a first movement of the user input device; calculating a second movement of the target virtual object, the second movement based at least in part on the first movement and the multiplier; and moving the target virtual object by an amount based at least in part on the second movement.

[0203] In a nineteenth aspect, the method of aspect 18, wherein calculating the distance to the virtual object includes calculating the distance between the virtual object and a user input device, the distance between the virtual object and a sensor on the AR system, or the distance between the user input device and a sensor on the AR system.

[0204] In a twentieth aspect, the method of aspect 18, wherein the second move is equal to the first move multiplied by the multiplier.

[0205] In a twenty-first aspect, the method of aspect 18, wherein the multiplier increases with increasing distance over the first distance range.

[0206] In a twenty-second aspect, the method of aspect 21, wherein the multiplier increases linearly with increasing distance over the first range.

[0207] In a twenty-third aspect, the method of aspect 21, wherein the multiplier increases as a power of the distance over the first range.

[0208] In a twenty-fourth aspect, the method of aspect 18, wherein the multiplier is equal to the first threshold when the distance is less than the first distance.

[0209] In a twenty-fifth aspect, the method of aspect 24, wherein the first distance is equal to the reach of the user.

[0210] In a twenty-sixth aspect, the method of aspect 24, wherein the first threshold is equal to 1.

[0211] In a 27th aspect, the method of any one of aspects 18-26, wherein the first movement or the second movement includes a first speed or a second speed, respectively.

[0212] In a twenty-eighth aspect, the method of any one of aspects 18-26, wherein the first movement and the second movement include a first acceleration and a second acceleration, respectively.

[0213] In a 29th aspect, the method of any one of aspects 18-28, wherein the AR system comprises a head-mounted display.

[0214] In a 30th aspect, the method of any one of aspects 18-29, wherein the target virtual object is interactive.

[0215] In a thirty-first aspect, a method for moving a virtual object located in three-dimensional (3D) space, the method being performed under control of an augmented reality (AR) system comprising computer hardware and a user input device, the AR system being configured to present the virtual object in the 3D space for display to a user, the method including: receiving a selection of a target virtual object to be displayed to the user at a first position in the 3D space; receiving an indication of movement for the target virtual object; determining a multiplier to apply to the movement of the target virtual object; calculating an amount of movement for the target virtual object, the amount of movement being based at least in part on the indication of movement and the multiplier; and displaying the target virtual object to the user at a second position, the second position being based at least in part on the first position and the amount of movement.

[0216] In a thirty-second aspect, the method of aspect 31, wherein determining a multiplier to apply to the movement of the target virtual object includes calculating a distance to the target virtual object.

[0217] In a thirty-third aspect, the method described in aspect 32, wherein the distance is between the target virtual object and a user input device, between the target virtual object and a sensor on the AR system, or between the user input device and a sensor on the AR system.

[0218] In a thirty-fourth aspect, the method of aspect 32, wherein the multiplier increases as the distance increases.

[0219] In a thirty-fifth aspect, the method of any one of aspects 31-34, wherein the multiplier is based at least in part on user preferences.

[0220] In a 36th aspect, the method of any one of aspects 31-35, wherein the movement includes one or more of a change in position, velocity, or acceleration.

[0221] In a 37th aspect, the method of any one of aspects 31-36, wherein the target virtual object includes a group of virtual objects.

[0222] In a thirty-eighth aspect, the method of any one of aspects 31-37, wherein the target virtual object is interactive.

[0223] In a thirty-ninth aspect, the method of any one of aspects 31-38, wherein receiving an indication of movement includes receiving an indication of movement from a user input device.

[0224] In a fortieth aspect, the method of any one of aspects 31-38, wherein receiving an indication of movement includes receiving an indication of a change in the user's posture.

[0225] In a forty-first aspect, the method of aspect 40, wherein the user's posture includes one or more of a head posture, an eye posture, or a body posture.

[0226] In a forty-second aspect, an augmented reality (AR) system for transforming virtual objects located in three-dimensional (3D) space comprises a system display system, a user input device, and a computer processor, the computer processor in communication with the display system and the user input device and configured to: determine a group of virtual objects in a user's FOR; receive a selection of a target virtual object in the group of virtual objects in the user's FOR; calculate a distance to the target virtual object; determine a multiplier based at least in part on the distance to the target virtual object; receive a first movement of the user input device; calculate a second movement of the target virtual object, the second movement based at least in part on the first movement and the multiplier; and move the target virtual object by an amount based at least in part on the second movement.

[0227] In a forty-third aspect, the system described in aspect 42, wherein calculating the distance to the target virtual object includes calculating the distance between the target virtual object and a user input device, the distance between the virtual object and a sensor on the AR system, or the distance between the user input device and a sensor on the AR system.

[0228] In a forty-fourth aspect, the system of aspect 42, wherein the second move is equal to the first move multiplied by a multiplier.

[0229] In a forty-fifth aspect, the system described in aspect 42, wherein the multiplier increases with increasing distance over the first distance range.

[0230] In a forty-sixth aspect, the system of aspect 45, wherein the multiplier increases linearly with increasing distance over the first range.

[0231] In a 47th aspect, the system described in aspect 45, wherein the multiplier increases as a power of the distance over the first range.

[0232] In aspect 48, the system of aspect 42, wherein the multiplier is equal to the first threshold when the distance is less than the first distance.

[0233] In a forty-ninth aspect, the system described in aspect 48, wherein the first distance is equal to the reach of the user's hand.

[0234] In a 50th aspect, the system described in aspect 48, wherein the first threshold is equal to 1.

[0235] In a 51st aspect, the system described in any one of aspects 42-50, wherein the first movement or the second movement includes a first speed or a second speed, respectively.

[0236] In a 52nd aspect, the system described in any one of aspects 42-50, wherein the first movement and the second movement include a first acceleration and a second acceleration, respectively.

[0237] In a 53rd aspect, the AR system is a system described in any one of aspects 42-52, wherein the AR system includes a head-mounted display.

[0238] In a 54th aspect, the system described in any one of aspects 42-53, wherein the target virtual object is interactive.

[0239] In a fifty-fifth aspect, an augmented reality system (AR) for moving a virtual object located in three-dimensional (3D) space includes a system display system, a user input device, and a computer processor configured to: communicate with the display system and the user input device and receive a selection of a target virtual object to be displayed to a user at a first location in the 3D space; receive an indication of movement for the target virtual object; determine a multiplier to apply to the movement of the target virtual object; calculate an amount of movement for the target virtual object, the amount of movement based at least in part on the indication of movement and the multiplier; and display the target virtual object to the user at a second location, the second location based at least in part on the first location and the amount of movement.

[0240] In a fifty-sixth aspect, the system described in aspect 55, wherein determining a multiplier to apply to the movement of the target virtual object includes calculating a distance to the target virtual object.

[0241] In aspect 57, the system described in aspect 56, wherein the distance is between the virtual object and a user input device, between the virtual object and a sensor on the AR system, or between the user input device and a sensor on the AR system.

[0242] In a fifty-eighth aspect, the system of aspect 56, wherein the multiplier increases as the distance increases.

[0243] In a fifty-ninth aspect, the system of any one of aspects 55-58, wherein the multiplier is based at least in part on user preferences.

[0244] In a 60th aspect, the system of any one of aspects 55-59, wherein the movement includes one or more of a change in position, velocity, or acceleration.

[0245] In a 61st aspect, the system described in any one of aspects 55-60, wherein the target virtual object includes a group of virtual objects.

[0246] In aspect 62, the system described in any one of aspects 55-61, wherein the target virtual object is interactive.

[0247] In a 63rd aspect, the system described in any one of aspects 55-62, wherein receiving an indication of movement includes receiving an indication of movement from a user input device.

[0248] In a 64th aspect, the system described in any one of aspects 55-63, wherein receiving an indication of movement includes receiving an indication of a change in the user's posture.

[0249] In aspect 65, the system described in aspect 64, wherein the user's posture includes one or more of a head posture, an eye posture, or a body posture.

[0250] In a sixty-sixth aspect, a system for interacting with objects for a wearable device is provided, the system being configured to present a three-dimensional (3D) view to a user and enable user interaction with objects within the user's field of view (FOR), the FOR including a portion of the user's surrounding environment perceivable by the user through the display system; the system comprising: a display system of the wearable device; a sensor configured to acquire data associated with the user's posture; and a hardware processor in communication with the sensor and the display system, the hardware processor programmed to: determine the user's posture based on the data acquired by the sensor; initiate a cone projection onto a group of objects within the FOR, the cone projection including projecting a virtual cone with an opening in a direction based at least in part on the user's posture; analyze contextual information associated with the user's environment; update the opening of the virtual cone based at least in part on the contextual information; and render a visual representation of the virtual cone for the cone projection.

[0251] In a 67th aspect, the system described in aspect 66, wherein the context information includes at least one of the type, layout, location, size, or density of a subgroup of objects within a user's field of view (FOV), the FOV including a portion of the FOR that is perceptible by the user via the display system at a given time.

[0252] In aspect 68, the system described in aspect 67, wherein the density of a subgroup of objects within a user's FOV is calculated by at least one of calculating the number of objects in the subgroup of objects, calculating the percentage of the FOV covered by the subgroup of objects, or calculating a contour map for the objects in the subgroup of objects.

[0253] In a 69th aspect, the hardware processor is further programmed to detect a collision between the virtual cone and one or more objects in the group of objects in the FOR, and in response to detecting the collision, the hardware processor is further programmed to present a focus indicator on the one or more objects.

[0254] In a seventieth aspect, the hardware processor is programmed to apply an occlusion ambiguity resolution technique to one or more objects that collide with the virtual cone to identify the occluded objects, a system as described in aspect 69.

[0255] In aspect 71, the cone has a central ray, and the aperture intersects the central ray, in the system described in any one of aspects 66-70.

[0256] In aspect 72, the system described in any one of aspects 66-71, wherein the virtual cone has a proximal end that is anchored to at least one of a location between the user's eyes, a location on a portion of the user's arm, a location on a user input device, or any other location in the user's environment.

[0257] In aspect 73, the hardware processor is further programmed to receive an indication from a user input device anchoring the depth of the virtual cone to the depth plane, and the cone projection is performed onto a group of objects within the depth plane, a system described in any one of aspects 66-72.

[0258] In aspect 74, the system described in any one of aspects 66-73, wherein the sensor comprises at least one of an inertial measurement unit or an outward-facing imaging system.

[0259] In aspect 75, the system described in any one of aspects 66-74, wherein the virtual cone comprises at least one of a geometric cone projection, a rectangular prism, a polyhedron, a pyramid, or a truncated cone.

[0260] In a 76th aspect, a method for interacting with an object for a wearable device includes receiving a selection of a target virtual object to be displayed to a user at a first position in three-dimensional (3D) space; receiving an indication of movement for the target virtual object; analyzing contextual information associated with the target virtual object; calculating a multiplier to be applied to the movement of the target virtual object based at least in part on the contextual information; calculating an amount of movement for the target virtual object, wherein the amount of movement is based at least in part on the indication of movement and the multiplier; and displaying the target virtual object to the user at a second position, wherein the second position is based at least in part on the first position and the amount of movement.

[0261] In a seventy-seventh aspect, the method described in aspect 76, wherein the context information includes a distance from the user to the target virtual object.

[0262] In a seventy-eighth aspect, the method of aspect 77, wherein the multiplier increases proportionally with increasing distance.

[0263] In a 79th aspect, the method of any one of aspects 76-78, wherein the movement includes one or more of a change in position, velocity, or acceleration.

[0264] In an 80th aspect, the method of any one of aspects 76-79, wherein the indication of movement includes at least one of actuation of a user input device associated with the wearable device or a change in the user's posture.

[0265] In an 81st aspect, the method of aspect 80, wherein the posture includes one or more of a head posture, an eye posture, or a body posture.

[0266] In an 82nd aspect, a system for interacting with objects for a wearable device includes: a display system of the wearable device configured to present a three-dimensional (3D) view to a user, the 3D view comprising a target virtual object; and a hardware processor in communication with the display system, the hardware processor programmed to: receive an indication of movement for the target virtual object; analyze contextual information associated with the target virtual object; calculate a multiplier to be applied to the movement of the target virtual object based at least in part on the contextual information; calculate an amount of movement for the target virtual object, the amount of movement based at least in part on the indication of movement and the multiplier; and display the target virtual object at a second position by the display system, the second position based at least in part on the first position and the amount of movement.

[0267] In aspect 83, the system described in aspect 82, wherein the indication of movement of the target virtual object includes a change in the posture of a user of the wearable device or input received from a user input device associated with the wearable device.

[0268] In aspect 84, the system described in any one of aspects 82-83, wherein the context information includes a distance from the user to the target virtual object.

[0269] In aspect 85, the system described in aspect 84, wherein the multiplier is equal to 1 when the distance is less than a threshold distance, the threshold distance being equal to the user's reach.

[0270] In an 86th aspect, the system of any one of aspects 84-85, wherein the multiplier increases proportionally with increasing distance.

[0271] In an 87th aspect, the system described in any one of aspects 82-86, wherein the movement includes one or more of a change in position, velocity, or acceleration.

[0272] (Conclusion) Each of the processes, methods, and algorithms described herein and / or depicted in the accompanying figures may be embodied in code modules executed by one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, and thereby may be fully or partially automated. For example, a computing system may include a general-purpose computer (e.g., a server) or a special-purpose computer programmed with specific computer instructions, special-purpose circuitry, etc. Code modules may be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language. In some implementations, particular operations and methods may be performed by circuitry specific to a given function.

[0273] Furthermore, certain implementations of the functionality of the present disclosure may be sufficiently mathematically, computationally, or technically complex that special-purpose hardware (utilizing appropriate specialized executable instructions) or one or more physical computing devices may be required to perform the functionality, e.g., due to the amount or complexity of the calculations involved or to provide results in substantially real time. For example, a video may contain many frames, each frame may have millions of pixels, and specifically programmed computer hardware may be required to process the video data to provide the desired image processing task or application in a commercially reasonable amount of time.

[0274] Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as physical computer storage devices, including hard drives, solid-state memory, random-access memory (RAM), read-only memory (ROM), optical disks, volatile or non-volatile storage devices, combinations of the same, and / or the like. The methods and modules (or data) may also be transmitted as data signals (e.g., as part of a carrier wave or other analog or digital propagated signal) generated over various computer-readable transmission media, including wireless-based and wired / cable-based media, and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored, persistently or otherwise, in any type of non-transitory tangible computer storage device or communicated via a computer-readable transmission medium.

[0275] Any process, block, state, step, or functionality in the flow diagrams described herein and / or depicted in the accompanying figures should be understood as potentially representing a code module, segment, or portion of code, comprising one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in the process. Various processes, blocks, states, steps, or functionality can be combined, rearranged, added, deleted, modified, or otherwise changed from the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules may perform some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps, or states associated therewith can be performed in other suitable sequences, e.g., serially, in parallel, or in some other manner. Tasks or events may be added to or removed from the disclosed exemplary embodiments. Furthermore, the separation of various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods, and systems may generally be integrated together in a single computer product or packaged in multiple computer products. Many implementation variations are possible.

[0276] The processes, methods, and systems can be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowdsourced computing networks, the Internet, and the World Wide Web. The network can be a wired or wireless network or any other type of communication network.

[0277] The systems and methods of the present disclosure each have several innovative aspects, none of which is solely responsible for or required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of one another or combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of the present disclosure. Various modifications of the implementations described in the present disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations without departing from the spirit or scope of the present disclosure. Therefore, the claims are not intended to be limited to the implementations shown herein, but are to be accorded the widest scope consistent with the present disclosure, the principles, and novel features disclosed herein.

[0278] Certain features described herein in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation can also be implemented separately in multiple implementations or in any suitable subcombination. Furthermore, while features may be described above as acting in a combination and may even be initially claimed as such, one or more features from the claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination. No single feature or group of features is required or essential to every embodiment.

[0279] Conditional statements used herein, such as "can," "could," "might," "may," "e.g.," and the like, among others, are intended to generally convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not, unless specifically stated otherwise or understood otherwise within the context as used. Thus, such conditional statements are generally not intended to imply that features, elements, and / or steps are in any way required for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are to be included or performed in any particular embodiment, with or without authorial input or prompting. The terms "comprise," "include," "have," and the like are synonymous and used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. Also, the term "or" is used in its inclusive sense (and not its exclusive sense), so, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Additionally, the articles "a," "an," and "the," as used in this application and the appended claims, unless otherwise specified, should be interpreted to mean "one or more" or "at least one."

[0280] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single elements. As an example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Transitional phrases such as "at least one of X, Y, and Z," unless specifically stated otherwise, are generally understood differently in the context in which they are used to convey that an item, term, etc. may be at least one of X, Y, or Z. Thus, such transitional phrases generally are not intended to suggest that an embodiment requires that at least one of X, at least one of Y, and at least one of Z, respectively, be present.

[0281] Similarly, while operations may be depicted in the figures in a particular order, it should be recognized that such operations need not be performed in the particular order shown, or in sequential order, or that all of the depicted operations need not be performed to achieve desirable results. Furthermore, the figures may diagrammatically depict one or more example processes in the form of a flowchart. However, other operations not depicted may be incorporated within the diagrammatically depicted example methods and processes. For example, one or more additional operations may be performed before, after, simultaneously with, or between any of the depicted operations. Additionally, operations may be rearranged or reordered in other implementations. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results.

Claims

1. A system, comprising: a display system for a wearable device configured to provide a three-dimensional (3D) view to a user and enable user interaction with objects within a field of view (FOV) of the user, the FOV including a portion of the environment surrounding the user perceivable by the user through the display system; and a hardware processor in communication with the display system; Equipped with The hardware processor includes: Initiating a cone projection of a virtual cone; determining context information within the FOV of the user; anchoring a proximal end of the virtual cone to an anchoring location associated with a physical object designated by the user via user input, wherein the user is enabled to anchor the proximal end to a pointing device, and movement of the anchoring location associated with the physical object designated by the user causes a corresponding movement of the virtual cone, whereby the proximal end of the virtual cone remains at the anchoring location associated with the physical object, the virtual cone further including a distal end or a depth, and the hardware processor is programmed to anchor the distal end and / or the depth of the virtual cone to the physical object based on at least one of user input, body gesture, body posture, gaze direction, or voice command; determining one or more objects within the virtual cone that are anchored to the physical object; performing an action on one or more of the objects within the virtual cone anchored to the physical object; A system that is programmed to:

2. The system of claim 1, wherein the hardware processor is further programmed to determine contextual information about the user's environment, the contextual information including at least one of type, layout, location, size, distance, or density of objects within the user's FOV.

3. To determine the density of the objects within the FOV of the user, the hardware processor: Calculating the number of objects within the FOV; Calculating the portion of the FOV that is covered by the object; or calculating a contour map for said object; 3. The system of claim 2, wherein the system is programmed to:

4. The system described in claim 2, wherein the context information includes at least one of the user's preferences, physical conditions associated with the user, or information associated with the environment.

5. The system described in claim 1, wherein the opening is dynamically adjustable, and to update the dynamically adjustable opening, the hardware processor is programmed to select the opening between a minimum size and a maximum size.

6. The system described in claim 5, wherein the minimum size is zero.

7. The system of claim 1, wherein the action includes rendering a focus indicator associated with the one or more objects.

8. The hardware processor is further programmed to detect collisions between the virtual cone and one or more objects in the environment; 6. The system of claim 5, wherein to detect the collision between the virtual cone and the one or more objects, the hardware processor is programmed to determine that the one or more objects intersect a virtual surface of the virtual cone or determine that the one or more objects are within the opening of the virtual cone.

9. The system of claim 1, wherein the one or more objects include a plurality of collided objects, and the hardware processor is programmed to identify occluded objects or determine a depth order or position between occluded objects by applying an occlusion ambiguity resolution technique to the plurality of collided objects.

10. The system of claim 1, wherein the one or more objects include a plurality of collided objects, and the hardware processor is programmed to select one or more of the plurality of collided objects by presenting a user interface element.

11. The action with the one or more objects: selecting said one or more objects; movement of said one or more objects; displaying a menu or toolbar associated with said one or more objects; or Game actions for virtual avatars in the game The system of claim 1 , comprising one or more of:

12. The system described in claim 1, wherein the virtual cone includes a central ray, the opening of the virtual cone intersects the central ray, and the direction of the central ray is based on the user's posture.

13. The hardware processor, determining a density of objects within the FOV of the user; updating the opening of the virtual cone based on the density of the objects within the FOV of the user; The system of claim 1 configured to:

14. The system of claim 1, wherein the hardware processor is further programmed to detect a collision between the virtual cone and one or more objects in the environment, the virtual cone including a distal end, and to detect the collision, the hardware processor is programmed to scan for a collision with the one or more objects within the distal end of the virtual cone.

15. The system of claim 1, wherein the virtual cone further includes a distal end or a depth, and the hardware processor is programmed to anchor the distal end or the depth of the virtual cone based in part on the context information.

16. The system further comprising a posture sensor configured to obtain posture data associated with a posture of the user; The system of claim 1 , wherein the hardware processor is programmed to translate the virtual cone based on the pose data associated with the pose of the user.

17. The system of claim 1, wherein the hardware processor is further programmed to resize the opening of the virtual cone based at least in part on the contextual information within the FOV of the user.

18. The system of claim 1, wherein to initiate the cone projection, the hardware processor is programmed to extend the distal end of the virtual cone until the distal end reaches a termination threshold.

19. The system described in claim 18, wherein the termination threshold includes a threshold distance or a boundary of the environment.

20. The system of claim 1, wherein the hardware processor is further programmed to render a visual representation of at least a portion of the virtual cone to the user via the display system.

Images