Occlusion cursor for virtual content in mixed reality displays

Abstract

To provide a system and a method for displaying a cursor and a focus indicator that are associated with real or virtual objects in a virtual, augmented, or mixed reality environment by a wearable device.SOLUTION: A system can determine a spatial relationship between a cursor that can be moved by a user and a target object within an environment. The system may render a focus indicator (e.g., a halo, shading, or highlighting) around or adjacent objects that are near the cursor. When the cursor overlaps with the target object, the system can render the object in front of the cursor (or not render the cursor at all), so the object is not occluded by the cursor. The object can be rendered closer to the user than the cursor.SELECTED DRAWING: None

Description

[Technical Field]

[0001] (Copyright Notice) A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to anyone copying this patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. (CROSS-REFERENCE TO RELATED APPLICATIONS)

[0002] This application was filed on March 14, 2018, and is related to the disclosure of "ECLIPSE CURSOR FOR This application is a continuation-in-part of U.S. patent application Ser. No. 15 / 920,830, filed Jan. 30, 2018, entitled "ECLIPSE CURSOR FOR MIXED REALITY DISPLAYS," each of which is incorporated herein by reference in its entirety. (Field)

[0003] The present disclosure relates to virtual, augmented, or mixed reality imaging and visualization systems, and more particularly to assigning focus indicators to one or more real or virtual objects within a user's field of view. [Background technology]

[0004] Modern computing and display technologies have facilitated the development of systems for so-called “virtual reality,” “augmented reality,” or “mixed reality” experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears or can be perceived as real. Virtual reality or “VR” scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual input. Augmented reality or “AR” scenarios typically involve the presentation of digital or virtual image information as an augmentation to the visualization of the real world around the user. Mixed reality or “MR” relates to the merging of real and virtual worlds to create new environments in which physical and virtual objects coexist and interact in real time. Consequently, the human visual perception system is highly complex, making it challenging to create VR, AR, or MR technologies that facilitate comfortable, natural-feeling, and rich presentation of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies. Summary of the Invention [Means for solving the problem]

[0005] Techniques are described for displaying cursors and focus indicators associated with real or virtual objects in a virtual, augmented, or mixed reality environment via a wearable display device. For example, rendering a cursor in front of an object in the environment tends to occlude the object and place emphasis on the cursor itself in the visual hierarchy rather than on the target object being interacted with. Thus, embodiments of the wearable system can utilize an occlusion cursor that moves behind the target object (e.g., so that the cursor is “occluded” by the target object), which tends to preserve emphasis on the target object rather than the cursor in the user's visual hierarchy. To assist a user in navigating the cursor among objects in the environment, the system can render focus indicators around objects near the cursor. The focus indicator may include a halo, a shadow, or a highlight around at least a portion of the object near the cursor. The focus indicator may be emphasized (e.g., brighter, with a different color or shadow, or larger size) closer to the cursor (and de-emphasized further away from the cursor), which provides a visual cue to the user in navigating the cursor among objects in the environment and selecting a target object. The occlusion cursor and focus indicator can provide a more natural and immersive user experience for the user.

[0006] In various embodiments, the wearable display system can determine the spatial relationship between a cursor movable by a user and one or more target objects in the environment. For example, the user may move the cursor by actuating a handheld user input device (e.g., a totem). The wearable system may render a focus indicator (e.g., a halo, a shadow, or a highlight) around or adjacent to an object near the cursor. The focus indicator may be emphasized in a direction closer to the cursor and de-emphasized in a direction farther from the cursor. When the cursor overlaps a target object, the system may render the cursor behind the object (or not render the cursor at all), so that the object is not occluded by the cursor (e.g., the object obscures the cursor). The cursor and focus indicator can provide positional feedback to the user and help the user navigate among objects in the environment.

[0007] In various aspects, a wearable display system can include a user interface that presents a user with multiple interactable virtual items arranged in a grid (regular or irregular) of thumbnails located at one or more depths. The thumbnails can comprise miniature representations of the virtual items (e.g., document pages or images) that can be used to identify the virtual items by their content. In some implementations, selecting (e.g., clicking or double-clicking) a thumbnail opens the content of the virtual item (e.g., by executing an application configured to launch, play, view, or edit the virtual content). The thumbnails can be rendered to appear at one depth (e.g., as 2D thumbnails) or at multiple depths (e.g., to appear in 3D). In response to a cursor moving behind one of the thumbnails in the grid, the thumbnail for that item may be rendered with one or more of the following effects: expanding in size, including a focus indicator (e.g., a halo surrounding at least a portion of the thumbnail), moving to a different depth (e.g., a depth that appears closer to the user), or having different virtual content (e.g., a higher resolution image, a caption, sound, a graphical video or animated playback, etc.). The thumbnails may be ordered according to one or more grouping criteria (e.g., alphabetically by item name, content type, date, etc.). The grid of thumbnails may be scrollable by the user (e.g., using user input from head, eye, or body gestures, or a totem). Scrolling the edge of the grid (e.g., in the direction of the scroll) may dynamically display (e.g., as a semi-transparent thumbnail, optionally at a different depth than the grid edge) an indication of the virtual content to be displayed next (e.g., upcoming content) while scrolling.

[0008] In various aspects, the present disclosure provides decorative designs for a display screen or portion thereof with icons or transitional (e.g., animated) graphical user interfaces. An augmented, mixed, or virtual reality display device can comprise the display screen or portion thereof.

[0009] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Neither this summary nor the following detailed description purports to define or limit the scope of the inventive subject matter. The present invention provides, for example, the following. (Item 1) 1. A wearable display system, comprising: a display configured to be positioned in front of a user's eyes, the display configured to project virtual content towards the user's eyes; a user input device configured to receive user input data associated with movement of a virtual cursor; a hardware processor in communication with the display and the user input device; Equipped with The hardware processor includes: identifying a location of the virtual cursor in the user's environment; determining a spatial relationship between the virtual cursor and objects in the user's environment; directing the display to render a focus indicator associated with the object based at least in part on the spatial relationship; and a wearable display system programmed to: (Item 2) Item 1. The wearable display system of item 1, wherein the user input device comprises one or more of: a handheld totem having a touch-sensitive surface; an outward-facing imaging system configured to detect user gestures; an inward-facing imaging system configured to detect the user's eye posture; or an inertial measurement unit configured to detect the user's head posture. (Item 3) To determine the spatial relationship between the virtual cursor and the object, the hardware processor: determining a distance between a location of the virtual cursor and a portion of the object or determining a relative orientation between the virtual cursor and the object; directing the display to render the focus indicator based at least in part on the determined distance or the determined orientation; and Item 1. The wearable display system of item 1, programmed to: (Item 4) Item 1. The wearable display system of item 1, wherein the focus indicator comprises one or more of a glow or halo at least partially surrounding the object, a size or depth change of the object, or a graphical highlight. (Item 5) To instruct the display to render a focus indicator, the hardware processor: performing a first rendering step to render a cursor glow into a first buffer, the location of the cursor glow being based at least in part on the location of the virtual cursor; performing a second rendering step to render a shape mask representation of the object into a second buffer; performing a third rendering step to render an iris mask associated with the object into a third buffer; performing a fourth rendering step; It is programmed to Item 1. The wearable display system of item 1, wherein the fourth rendering step is configured to combine at least the first buffer, the second buffer, and the third buffer for presentation to a user. (Item 6) Item 6. The wearable display system of item 5, wherein, to perform the fourth rendering step, the hardware processor is programmed to combine at least the first buffer, the second buffer, and the third buffer with a fourth buffer having virtual scene content. (Item 7) The hardware processor includes: determining a distance between the location of the virtual cursor and the object; If the distance is less than a threshold distance, updating the location of the virtual cursor to be a location representing the object. Item 14. The wearable display system of item 1, further programmed to: (Item 8) The hardware processor includes: determining an orientation between the location of the virtual cursor and the object; instructing the display to render the focus indicator preferentially toward the orientation of the virtual cursor; Item 1. The wearable display system of item 1, programmed to: (Item 9) The hardware processor includes: Item 14. The wearable display system of item 1, programmed to update the location of the virtual cursor based at least in part on a previous movement path of the virtual cursor. (Item 10) Item 10. The wearable display system of item 9, wherein the hardware processor is programmed to update the location of the virtual cursor based at least in part on the previous movement path in response to cessation of user input from the user input device. (Item 11) The hardware processor includes: determining a distance between a location of the virtual cursor and a portion of the object; and in response to determining that the distance is less than a threshold, rendering the object in front of the virtual cursor or ceasing rendering of the virtual cursor. Item 1. The wearable display system of item 1, programmed to: (Item 12) 1. A method of rendering a cursor relative to an object in a mixed reality environment, the method comprising: under control of a mixed reality display device comprising a display and a hardware processor; determining a location of the cursor in the mixed reality environment; determining a location associated with the object; determining whether an overlap occurs between the object and the cursor; responsive to determining that the overlap does not occur, rendering the cursor; and Responsive to determining that the overlap occurs, rendering the object in front of the cursor or ceasing rendering of the cursor. A method comprising: (Item 13) Item 13. The method of item 12, wherein determining whether an overlap occurs between the object and the cursor includes determining whether a distance between a location of the cursor and a location of the object is less than a distance threshold. (Item 14) Item 13. The method of item 12, wherein in response to determining that the overlap occurs, the method further comprises rendering a focus indicator associated with the object. (Item 15) Item 15. The method of item 14, wherein the focus indicator comprises one or more of a glow or halo at least partially surrounding the object, a size or depth change of the object, or a graphical highlight. (Item 16) determining a spatial relationship between the cursor and the object; rendering a focus indicator associated with the object based at least in part on the spatial relationship; and Item 13. The method of item 12, further comprising: (Item 17) 1. An augmented reality display system, comprising: a user input device configured to receive user input related to a position of a cursor in a user's environment; a display through which a user can perceive virtual objects in the user's environment; and a hardware processor in communication with the user input device and the display; Equipped with The hardware processor includes: causing a virtual object to be rendered via said display; tracking a position of the cursor in the environment based at least in part on the user input; identifying a location of the virtual object; determining whether a first distance between the virtual object and a position of the cursor satisfies a distance threshold; in response to determining that the first distance satisfies the distance threshold, causing a focus indicator to be rendered proximate to the object via the display; and an augmented reality display system that is programmed to: (Item 18) 1. An augmented reality display system, comprising: a user input device configured to receive user input related to a position of a cursor in a user's environment; a display through which a user can perceive virtual objects in the user's environment; a hardware processor in communication with the user input device and the display; Equipped with The hardware processor includes: causing a plurality of virtual objects to be rendered via the display; tracking the position of the cursor within a field of view of the display; identifying locations of the plurality of virtual objects; comparing the position of the cursor with the positions of the plurality of virtual objects to determine the object that is closest to the position of the cursor; causing a focus indicator to be rendered proximate to the nearest object via said display; an augmented reality display system that is programmed to: (Item 19) 1. A method of rendering a graphical user interface for a mixed reality display device, the method comprising: under control of a mixed reality display device comprising a display and a hardware processor; determining a location of a cursor within a field of view of the display; rendering a focus indicator associated with an object within a field of view of the display; tracking movement of the cursor relative to the object; adjusting rendering of the focus indicator based at least in part on movement of the cursor relative to the object; A method comprising: (Item 20) 1. A wearable display system, comprising: a display configured to be positioned in front of a user's eyes, the display configured to project virtual content towards the user's eyes; a user input device configured to receive user input data associated with movement of a virtual cursor in the virtual environment; a hardware processor in communication with the display and the user input device; Equipped with The hardware processor includes: directing the display to render a virtual icon at a first depth in a virtual environment; directing the display to render a virtual cursor at a second depth within the virtual environment; tracking movement of the virtual cursor in the virtual environment; determining whether an overlap occurs between the virtual cursor and the virtual icon; instructing the display to render the virtual icon at a third depth closer to the user than the first depth or the second depth in response to determining the overlap; a wearable display system programmed to: (Item 21) 1. A wearable display system, comprising: a display configured to be positioned in front of a user's eyes, the display configured to project virtual content towards the user's eyes; a user input device configured to receive user input data associated with movement of a virtual cursor in the virtual environment; a hardware processor in communication with the display and the user input device; Equipped with The hardware processor includes: instructing the display to render a virtual layout of a plurality of virtual icons, at least some of the plurality of virtual icons in the virtual layout being rendered at a first depth in the virtual environment; receiving a user instruction to scroll the virtual layout; instructing the display to render a virtual content panel in response to receiving the user instruction to scroll the virtual layout; It is programmed to A wearable display system, wherein the virtual content panel includes representations of virtual icons that were not rendered prior to the scrolling, and the virtual content panel is rendered at a second depth in the virtual environment. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 depicts an illustration of a mixed reality scenario with a virtual reality object and a physical object viewed by a person.

[0011] [Figure 2] FIG. 2 illustrates diagrammatically an example of a wearable system.

[0012] [Figure 3] FIG. 3 diagrammatically illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes.

[0013] [Figure 4]FIG. 4 diagrammatically illustrates an example of a waveguide stack of a wearable device for outputting image information to a user.

[0014] [Figure 5] FIG. 5 shows an exemplary output beam that may be output by a waveguide.

[0015] [Figure 6] FIG. 6 is a schematic diagram illustrating an optical system including a waveguide device used in generating a multifocal volumetric display, image, or bright field, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem.

[0016] [Figure 7] FIG. 7 is a block diagram of an example of a wearable system.

[0017] [Figure 8] FIG. 8 is a process flow diagram of an example method for rendering virtual content in relation to recognized objects.

[0018] [Figure 9] FIG. 9 is a block diagram of another example of a wearable system.

[0019] [Figure 10] FIG. 10 is a process flow diagram of an example method for determining user input to a wearable system.

[0020] [Figure 11] FIG. 11 is a process flow diagram of an example method for interacting with a virtual user interface.

[0021] [Figure 12] 12A-12C illustrate various examples of objects and cursors that may be perceived by a user via a wearable system.

[0022] [Figure 13] 13A and 13B illustrate non-limiting embodiments of a focus indicator and cursor.

[0023] [Figure 14] 14A-14C illustrate examples of a cursor and objects in a 3D environment perceivable by a user via a wearable display system.

[0024] [Figure 15A] 15A and 15B illustrate an example implementation of multiple focus indicators with different intensities, positions, or spatial extents based on the proximity of the cursor to the location of objects in the environment. [Figure 15B] 15A and 15B illustrate an example implementation of multiple focus indicators with different intensities, positions, or spatial extents based on the proximity of the cursor to the location of objects in the environment.

[0025] [Figure 16A] 16A-16D illustrate an example of a process for rendering a focus indicator. [Figure 16B] 16A-16D illustrate an example of a process for rendering a focus indicator. [Figure 16C] 16A-16D illustrate an example of a process for rendering a focus indicator. [Figure 16D] 16A-16D illustrate an example of a process for rendering a focus indicator.

[0026] [Figure 17] FIG. 17 shows an example of a grid and user input on a totem (with a touch-sensitive surface).

[0027] [Figure 18] 18A-18C illustrate examples of moving a cursor towards an object with a focus indicator.

[0028] [Figure 19] 19-22 illustrate various examples of focus indicators that may be rendered by the system. [Figure 20] 19-22 illustrate various examples of focus indicators that may be rendered by the system. [Figure 21] 19-22 illustrate various examples of focus indicators that may be rendered by the system. [Figure 22] 19-22 illustrate various examples of focus indicators that may be rendered by the system.

[0029] [Figure 23] FIG. 23 is a flowchart illustrating an exemplary method for rendering a focus indicator within a 3D scene.

[0030] [Figure 24] 24-28 are front views of embodiments of a display screen or portion thereof with icons. [Figure 25] 24-28 are front views of embodiments of a display screen or portion thereof with icons. [Figure 26] 24-28 are front views of embodiments of a display screen or portion thereof with icons. [Figure 27] 24-28 are front views of embodiments of a display screen or portion thereof with icons. [Figure 28] 24-28 are front views of embodiments of a display screen or portion thereof with icons.

[0031] [Figure 29A] 29A-29F are front views of an embodiment of a transition sequence for a graphical user interface (GUI) on a display screen or portion thereof. [Figure 29B]29A-29F are front views of an embodiment of a transition sequence for a graphical user interface (GUI) on a display screen or portion thereof. [Figure 29C] 29A-29F are front views of an embodiment of a transition sequence for a graphical user interface (GUI) on a display screen or portion thereof. [Figure 29D] 29A-29F are front views of an embodiment of a transition sequence for a graphical user interface (GUI) on a display screen or portion thereof. [Figure 29E] 29A-29F are front views of an embodiment of a transition sequence for a graphical user interface (GUI) on a display screen or portion thereof. [Figure 29F] 29A-29F are front views of an embodiment of a transition sequence for a graphical user interface (GUI) on a display screen or portion thereof.

[0032] [Figure 30A] 30A-30F illustrate an embodiment of a transition sequence for a GUI on a display screen or portion thereof. [Figure 30B] 30A-30F illustrate an embodiment of a transition sequence for a GUI on a display screen or portion thereof. [Figure 30C] 30A-30F illustrate an embodiment of a transition sequence for a GUI on a display screen or portion thereof. [Figure 30D] 30A-30F illustrate an embodiment of a transition sequence for a GUI on a display screen or portion thereof. [Figure 30E] 30A-30F illustrate an embodiment of a transition sequence for a GUI on a display screen or portion thereof. [Figure 30F] 30A-30F illustrate an embodiment of a transition sequence for a GUI on a display screen or portion thereof.

[0033] [Figure 31A] 31A-31C illustrate an embodiment of a transition sequence for a GUI on a display screen or portion thereof. [Figure 31B] 31A-31C illustrate an embodiment of a transition sequence for a GUI on a display screen or portion thereof. [Figure 31C] 31A-31C illustrate an embodiment of a transition sequence for a GUI on a display screen or portion thereof. DETAILED DESCRIPTION OF THE INVENTION

[0034] Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure. Additionally, the figures within this disclosure are for illustrative purposes and are not drawn to scale. (overview)

[0035] The wearable device may include a display for presenting an interactive VR / AR / MR environment. The VR / AR / MR environment may include data elements that can be interacted with by a user through various postures, such as head pose, eye gaze, or body posture, or user input through a user input device. To provide the user with an accurate sense of their interaction with real or virtual objects in the VR / AR / MR environment, the system may render on-screen visual aids to help the user navigate among, select, or interact with objects in the environment.

[0036] In some cases, the on-screen visual aids may include a virtual cursor (sometimes also referred to herein as a reticle), which responds to user interaction (e.g., user input via a handheld totem) and identifies the location of a movable indicator (relative to the user) that can be used to select or interact with an object within the VR / AR / MR environment. For example, a user may move their thumb over a touch-sensitive portion of the totem to move the cursor within the 3D VR / AR / MR environment. When the cursor is sufficiently close to or hovering over an object, the user may be able to select or interact with the object (e.g., by pressing a touch-sensitive portion of the totem), which may initiate further context-dependent functionality by the wearable device. For example, a user may move the cursor near a virtual video display showing a movie, select that display, invoke a menu of other movie options, volume control, etc. In some cases, the cursor is displayed to the user so that the user can easily find the cursor within the environment. This may occur in a relatively sparse environment where relatively few objects are present. In other cases, the cursor is not displayed to the user, and a focus indicator described herein (e.g., a glow around the object) is used to provide the user with a visual cue as to the location of the cursor (e.g., the cursor is positioned near the object with the brightest glow). This may occur in a relatively dense environment where there are relatively many objects and the display of the cursor itself may not be needed or may be distracting.

[0037] However, traditional cursors are rendered without consideration of scene content. In other words, when a cursor is moved in a VR / AR / MR environment, the cursor moves over (e.g., is rendered in front of) objects in the environment. Continuing with the example above, a traditional cursor may appear in front of a virtual video display, which not only occludes the virtual display but also distracts the user from the content being shown (e.g., users tend to focus more on the cursor itself than on the movie, which can be distracting).

[0038] As a result, when a cursor is hovered over an object or used to select an object, the cursor effectively occludes or covers at least a portion of the object. This obstructed view of the object can significantly affect a user's experience within the environment. For example, the object may include content such as text, images, etc., and a user may find it difficult to select the object while also viewing the object's content.

[0039] While these problems exist in 2D environments, they can be exacerbated in 3D environments. For example, in 2D environments, objects and cursors do not have depth. Therefore, rendering a cursor in front of an object consists of rendering the cursor and the object on the same plane. In contrast, in a 3D environment, cursors and objects have depth relative to the user. Thus, at a given time, a cursor in a 3D environment is not necessarily at the same depth as an object in that environment. For example, the cursor may be closer or farther from the user relative to the object. Due to this difference in depth, when a user focuses on one of the object or the cursor, the other may appear blurry to the user due to accommodation disparity between their relative depths. Furthermore, even in instances where the cursor and object have the same or similar depth relative to the user in a 3D environment, for the cursor to "overlap" an object in 3D space, the system must change the depth of the cursor to avoid the appearance of the cursor moving through the object. 12A and 12B, the system may move the cursor closer to the user so that it appears as if it is located between the object and the user. By rendering the cursor closer to the user, the system effectively (and potentially undesirably) emphasizes the cursor relative to the object because a person's eyes are typically attracted to objects that are closer to the viewer, and the user is more likely to focus on the cursor because it appears closer to the user than the object.

[0040] To address these and other issues, embodiments of the system can render content-aware on-screen visual aids. For example, when a cursor and an object overlap, the system can render the cursor behind the object (rather than in front of it), or not render the cursor at all (because the cursor is behind the object and not visible to the user). Thus, the cursor does not block the object from the user's view, and the system does not inadvertently emphasize the cursor by rendering it closer to the user. Such cursor (or reticle) embodiments are sometimes referred to as occlusion cursors or occlusion reticles, because the target object "occludes" the cursor.

[0041] When the cursor is obscured by an object, it may be difficult for the user to get an accurate sense of the direction the user is facing in the scene or where the cursor is currently located because the cursor is at least partially blocked by the object. Thus, to continue to give the user an accurate sense of the cursor's location in the environment, the system may render another (or alternative) on-screen visual aid (e.g., a focus indicator) to highlight the object when the cursor moves behind that object (or within a distance threshold thereof).

[0042] A focus indicator may include a halo, a change in color, a change in perceived size or depth (e.g., making an object appear closer or larger when selected), a shadow, a virtual ray of light, or other graphical highlight likely to attract a user's attention emanating from or associated with the object. For example, a focus indicator may include a glow that appears to radiate outward from the object, as if a shining light source were mounted behind the object (such that the object "obscures" the light source). The intensity of the glow may be more intense nearer the outer edges of the object and less intense at greater distances from the outer edges of the object. Because a focus indicator does not occlude an object (because focus indicators are typically rendered to at least partially surround the object), the focus indicator instead highlights the object, advantageously providing a user-friendly, non-distracting alternative to the cursor and indicating to the user which object is currently being interacted with.

[0043] In some cases, the cursor may appear to have an attractive effect on the object, such that the proximity of the cursor to the object affects the intensity or positioning of the focus indicator or cursor. The attractive effect may tend to act as if the cursor and object (or focus indicator) are magnetically or gravitationally attracted to one another. For example, in some cases, each object may have a focus indicator (e.g., an outer glow), and the intensity, size, or location of the focus indicator may vary based on the location of the cursor relative to the object (or focus indicator). For example, as the cursor moves closer to an object, the focus indicator for that object may become brighter, stronger, or move in the direction of the cursor (e.g., as if being drawn toward it). As the cursor is moved closer to an object, the system may render the cursor as if being drawn behind the object, while simultaneously increasing the intensity of the focus indicator. This behavior may allow a user to select objects more naturally and easily, as the cursor is drawn toward (or snapped onto) the nearest object as the cursor approaches the desired target object, without the user having to make small adjustments to position the cursor on the target object. The cursor may therefore behave as if it has mass or inertia (such that the cursor tends to continue moving in the direction initially applied) and is drawn toward nearby objects by an attractive effect. As the location of the cursor in the environment changes, the intensity of the focus indicator associated with objects in the cursor's immediate vicinity may also change.

[0044] In some cases, the system can assign a focus indicator to more than one object, or the focus indicator can have a varying intensity or glow, for example, fading in or out based on the object's proximity to the cursor's location in the environment. Thus, one or more focus indicators can provide positional feedback to the user, for example, by highlighting one or more objects at varying intensities. The varying intensity or glow can shift position as the user input shifts, providing continuous input feedback and a precise sense of cursor position.

[0045] In various aspects, the system may include a user interface that presents a user with multiple interactable virtual items located at one or more depths and arranged in a grid of thumbnails (regular or irregular). In response to a cursor moving behind one of the thumbnails in the grid, the thumbnail for that item may be rendered with one or more of the following effects: expanding in size, including a focus indicator (e.g., a halo surrounding at least a portion of the thumbnail), moving to a different depth (e.g., a depth that appears closer to the user), or having different virtual content (e.g., a higher resolution image, captions, sound, graphical video or animation playback, etc.). The thumbnails may be ordered according to one or more grouping criteria (e.g., alphabetically by item name, content type, date, etc.). The grid of thumbnails may be scrollable by the user (e.g., using user input from head, eye, or body gestures, or a totem). Scrolling the edge of the grid (e.g., in the direction of the scroll) may dynamically display (e.g., as a semi-transparent thumbnail, optionally at a different depth than the grid edge) an indication of the virtual content to be displayed next (e.g., upcoming content) during the scroll. (Example of a 3D display in a wearable system)

[0046] A wearable system (also referred to herein as an augmented reality (AR) system) can be configured to present two-dimensional (2D) or three-dimensional (3D) virtual images to a user. The images can be still images, frames of video, or videos, such as in combination. A wearable system can include a wearable device that can present a VR, AR, or MR environment, alone or in combination, for user interaction. The wearable device can be a head-mounted device (HMD), which is used synonymously with AR device (ARD).

[0047] Figure 1 depicts an illustration of a mixed reality scenario involving a virtual reality object and a physical object viewed by a person. In Figure 1, an MR scene 100 is depicted in which a user of the MR technology is viewing a real-world park-like setting 110 featuring people, trees, a building in the background, and a concrete platform 120. In addition to these items, the user of the MR technology perceives as "seeing" a robotic figure 130 standing on the real-world platform 120 and a flying, cartoon-like avatar character 140 that appears to be an anthropomorphic bumblebee, although these elements do not exist in the real world.

[0048] In order for a 3D display to produce a true depth sensation, or more specifically, a simulated sensation of surface depth, it may be desirable to generate, for each point in the display's field of view, an accommodation response that corresponds to that point's virtual depth. If the accommodation response to a display point does not correspond to that point's virtual depth as determined by convergence and stereoscopic binocular depth cues, the human eye may experience accommodation conflict, resulting in unstable imaging, adverse eye strain, headaches, and, in the absence of accommodative information, a near-complete lack of surface depth.

[0049] VR, AR, and MR experiences can be provided by a display system having a display in which images corresponding to multiple depth planes are provided to a viewer. The images may be different for each depth plane (e.g., providing a slightly different presentation of a scene or object) and may be focused separately by the viewer's eyes, thereby serving to provide depth cues to the user based on the ocular accommodation required to focus on different image features of a scene located on different depth planes, or based on observing different image features on different depth planes that are out of focus. As discussed elsewhere herein, such depth cues provide a believable perception of depth.

[0050] FIG. 2 illustrates an example of a wearable system 200. The wearable system 200 includes a display 220 and various mechanical and electronic modules and systems to support the functionality of the display 220. The display 220 may be coupled to a frame 230 that is wearable by a user, wearer, or viewer 210. The display 220 may be positioned in front of the eyes of the user 210. The display 220 may present AR / VR / MR content to the user. For example, the display 220 may embody (e.g., render and present to the user) the occultation cursor icon and focus indicator described below. Examples of decorative designs for the occultation cursor icon and focus indicator are shown in FIGS. 24-29F. The display 220 may include a head-mounted display (HMD) worn on the user's head. In some embodiments, a speaker 240 is coupled to the frame 230 and positioned adjacent to the user's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / shapeable sound control).

[0051] The wearable system 200 may include an outward-facing imaging system 464 (shown in FIG. 4) that observes the world in the user's surrounding environment. The wearable system 200 may also include an inward-facing imaging system 462 (shown in FIG. 4) that can track the user's eye movements. The inward-facing imaging system may track the movements of either one eye or both eyes. The inward-facing imaging system 462 may be mounted to the frame 230 and may be in electrical communication with a processing module 260 or 270 that may process image information obtained by the inward-facing imaging system and determine, for example, the pupil diameter or orientation of the user's 210 eyes, eye movement, or eye posture.

[0052] By way of example, the wearable system 200 may obtain an image of the user's posture using the outward-facing imaging system 464 or the inward-facing imaging system 462. The image may be a still image, a frame of video or video, a combination thereof, etc.

[0053] The display 220 can be operably coupled (250) to a local data processing module 260, which can be mounted in a variety of configurations, such as fixedly attached to the frame 230, fixedly attached to a helmet or hat worn by the user, built into headphones, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, in a belt-coupled configuration), such as by wired or wireless connection.

[0054] Local processing and data module 260 may comprise a hardware processor and digital memory, such as non-volatile memory (e.g., flash memory), both of which may be utilized to aid in processing, caching, and storing data. The data may include (a) data captured from sensors (e.g., that may be operatively coupled to frame 230 or otherwise attached to user 210), such as image capture devices (e.g., cameras in inward-facing and / or outward-facing imaging systems), microphones, inertial measurement units (IMUs) (e.g., accelerometers, gravimeters, magnetometers, etc.), compasses, global positioning system (GPS) units, wireless devices, or gyroscopes, or (b) data obtained or processed using remote processing module 270 and / or remote data repository 280, perhaps for passage to display 220 after such processing or retrieval. The local processing and data module 260 may be operably coupled to a remote processing module 270 or a remote data repository 280 over a communication link 262 or 264, such as via a wired or wireless communication link, such that these remote modules are available as resources to the local processing and data module 260. In addition, the remote processing module 280 and the remote data repository 280 may be operably coupled to each other.

[0055] In some embodiments, remote processing module 270 may comprise one or more processors configured to analyze and process data or image information. In some embodiments, remote data repository 280 may comprise a digital data storage facility, which may be available through the Internet or other networking configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module, allowing for fully autonomous use from the remote module.

[0056] The human visual system is complex, making it difficult to provide a realistic perception of depth. Without being limited by theory, it is believed that viewers of an object perceive it as three-dimensional due to a combination of vergence and accommodation. Vergence movement of the two eyes relative to each other (i.e., the rotation of the pupils toward or away from each other to converge the lines of sight of the eyes and fixate on an object) is closely linked to the focusing (or "accommodation") of the eye's lenses. Under normal conditions, a change in the focus of the eye's lenses or accommodation of the eye to change focus from one object to another at a different distance will automatically result in a matching change in vergence to the same distance, a relationship known as the "accommodation-vergence reflex." Similarly, a change in vergence will induce a matching change in accommodation under normal conditions. A display system that provides better matching between accommodation and vergence may produce a more realistic and comfortable simulation of three-dimensional images.

[0057] FIG. 3 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes. Referring to FIG. 3 , objects at various distances from the eyes 302 and 304 on the z-axis are accommodated by the eyes 302 and 304 so that the objects are in focus. The eyes 302 and 304 assume particular accommodated states and focus on objects at different distances along the z-axis. As a result, a particular accommodated state may be said to be associated with a particular one of the depth planes 306 having an associated focal length, and an object or portion of an object at a particular depth plane is in focus when the eye is in an accommodated state relative to that depth plane. In some embodiments, a three-dimensional image may be simulated by providing a different presentation of an image for each of the eyes 302 and 304, and may also be simulated by providing a different presentation of an image corresponding to each of the depth planes. While shown as separate for clarity of illustration, it should be understood that the fields of view of the eyes 302 and 304 may overlap, for example, as the distance along the z-axis increases. Additionally, while shown as flat for ease of illustration, it should be understood that the contours of the depth plane can be curved in physical space so that all features within the depth plane are in focus with the eye in a particular accommodation state. The depth plane 306 need not be set at a fixed distance from the display (or the user's eyes 302, 304) but can be dynamically updated. For example, if a user views virtual content at a close distance to the user (e.g., within about 1-2 m), the array of depth planes shown in FIG. 3 can be adjusted to be closer to the user, which increases depth resolution at close distances. Similarly, if a user views virtual content at a medium distance (e.g., 2 m-5 m) or a far distance (e.g., 5 m-infinity), the depth plane can be adjusted to be primarily within those distances. The depth plane can be adjusted, for example, by adjusting the waveguide stacks described with reference to FIGS. 4-6. Without being limited by theory, it is believed that the human eye can typically interpret a finite number of depth planes to provide depth perception.As a result, a highly realistic simulation of perceived depth can be achieved by providing the eye with a different representation of the image corresponding to each of these limited number of depth planes. (Waveguide stack assembly)

[0058] FIG. 4 illustrates an example of a waveguide stack for outputting image information to a user. Wearable system 400 includes a stack of waveguides or stacked waveguide assembly 480 that can be utilized to provide three-dimensional perception to the eye / brain using multiple waveguides 432b, 434b, 436b, 438b, 4400b. In some embodiments, wearable system 400 can correspond to wearable system 200 of FIG. 2, and FIG. 4 diagrammatically illustrates several portions of wearable system 200 in more detail. For example, in some embodiments, waveguide assembly 480 can be integrated into display 220 of FIG. 2.

[0059] 4, the waveguide assembly 480 may also include multiple features 458, 456, 454, 452 between the waveguides. In some embodiments, the features 458, 456, 454, 452 may be lenses. In other embodiments, the features 458, 456, 454, 452 may not be lenses. Rather, they may simply be spacers (e.g., cladding layers or structures to form air gaps).

[0060] Waveguides 432b, 434b, 436b, 438b, 440b or multiple lenses 458, 456, 454, 452 may be configured to transmit image information to the eye using various levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a particular depth plane and configured to output image information corresponding to that depth plane. Image injection devices 420, 422, 424, 426, 428 may be utilized to inject image information into waveguides 440b, 438b, 436b, 434b, 432b, each of which may be configured to distribute incident light across each respective waveguide for output toward the eye 410. Light exits the output surfaces of image injection devices 420, 422, 424, 426, 428 and is injected into the corresponding input edges of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam of light (e.g., a collimated beam) can be injected into each waveguide to output an entire field of cloned collimated beams that are directed toward eye 410 at a particular angle (and divergence) corresponding to the depth plane associated with the particular waveguide.

[0061] In some embodiments, image input devices 420, 422, 424, 426, 428 are separate displays that each generate image information for input into a corresponding waveguide 440b, 438b, 436b, 434b, 432b, respectively. In some other embodiments, image input devices 420, 422, 424, 426, 428 are outputs of a single multiplexed display that may, for example, send image information to each of image input devices 420, 422, 424, 426, 428 via one or more optical conduits (such as fiber optic cables).

[0062] A controller 460 controls the operation of stacked waveguide assembly 480 and image injection devices 420, 422, 424, 426, 428. Controller 460 includes programming (e.g., instructions in a non-transitory computer-readable medium) that coordinates the timing and provision of image information to waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, controller 460 may be a single, integrated device or a distributed system connected by a wired or wireless communication channel. In some embodiments, controller 460 may be part of processing module 260 or 270 (shown in FIG. 2).

[0063] Waveguides 440b, 438b, 436b, 434b, 432b may be configured to propagate light within each respective waveguide by total internal reflection (TIR). Each of waveguides 440b, 438b, 436b, 434b, 432b may be planar with major top and bottom surfaces and edges extending between the major top and bottom surfaces, or may have another shape (e.g., curved). In the illustrated configuration, each of waveguides 440b, 438b, 436b, 434b, 432b may include light extraction optical elements 440a, 438a, 436a, 434a, 432a configured to extract light from the waveguide by redirecting the light to propagate within each respective waveguide and outputting image information from the waveguide to eye 410. The extracted light may also be referred to as out-coupled light, and the light extraction optical element may also be referred to as an out-coupling optical element. The extracted light beam is output by the waveguide where the light propagating within the waveguide strikes the light redirecting element. The light extraction optical elements (440a, 438a, 436a, 434a, 432a) may be, for example, reflective or diffractive optical features. While shown disposed on the bottom major surfaces of the waveguides 440b, 438b, 436b, 434b, 432b for ease of explanation and clarity of drawing, in some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be disposed on the top or bottom major surfaces or directly within the volume of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be attached to a transparent substrate and formed within a layer of material that forms the waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, 432b may be a monolithic piece of material, and the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed on and / or within that piece of material.

[0064] Continuing with reference to FIG. 4, as discussed herein, each waveguide 440b, 438b, 436b, 434b, 432b is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 when launched into such waveguide 432b. The collimated light may represent an optical infinity focal plane. The next upper waveguide 434b may be configured to send collimated light, which passes through a first lens 452 (e.g., a negative lens) before reaching the eye 410. The first lens 452 may be configured to generate a slight convex wavefront curvature so that the eye / brain interprets light emerging from the next waveguide 434b as emerging from a first focal plane closer inward from optical infinity toward the eye 410. Similarly, the third upper waveguide 436b passes its output light through both a first lens 452 and a second lens 454 before reaching the eye 410. The combined refractive power of the first and second lenses 452, 454 may be configured to produce another, increasing amount of wavefront curvature such that the eye / brain interprets light emerging from the third waveguide 436b as emerging from a second focal plane that is closer inward from optical infinity towards the person than light from the next upper waveguide 434b.

[0065] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured, with waveguide 440b, the highest in the stack, sending its output through all of the lenses between it and the eye due to the aggregate focal power representing the focal plane closest to the person. To compensate for the stack of lenses 458, 456, 454, 452 when viewing / interpreting light originating from the world 470 on the other side of stacked waveguide assembly 480, a compensating lens layer 430 may be placed on top of the stack to compensate for the aggregate power of the lower lens stacks 458, 456, 454, 452. Such a configuration provides as many perceived focal planes as there are available waveguide / lens pairs. Both the light extraction optical elements of the waveguides and the focusing sides of the lenses may be static (e.g., not dynamic or electro-active). In some alternative embodiments, one or both may be dynamic using electro-active features.

[0066] Continuing with reference to FIG. 4, light extraction optical elements 440a, 438a, 436a, 434a, 432a can be configured to both redirect light out of their respective waveguides and output this light with an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes can have differently configured light extraction optical elements that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, light extraction optical elements 440a, 438a, 436a, 434a, 432a can be solid or surface features that can be configured to output light at a particular angle. For example, light extraction optical elements 440a, 438a, 436a, 434a, 432a can be volume holograms, surface holograms, or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published June 25, 2015, which is incorporated herein by reference in its entirety.

[0067] In some embodiments, light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffractive features or "diffractive optical elements" (also referred to herein as "DOEs") that form a diffraction pattern. Preferably, the DOEs have a relatively low diffraction efficiency so that only a portion of the light in the beam is deflected toward the eye 410 at each intersection of the DOE, while the remainder continues traveling through the waveguide via total internal reflection. The light carrying the image information is thus split into several related output beams that exit the waveguide at multiple locations, and the result is a very uniform pattern of output emission toward the eye 304 for this particular collimated beam bouncing within the waveguide.

[0068] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not diffract much. For example, a switchable DOE may comprise a layer of polymer-dispersed liquid crystal in which microdroplets comprise a diffractive pattern in a host medium, and the refractive index of the microdroplets can be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0069] In some embodiments, the number and distribution of depth planes or depths of field can be dynamically varied based on the size or orientation of the viewer's pupil. The depth of field can change inversely with the viewer's pupil size. As a result, as the size of the viewer's pupil decreases, the depth of field increases so that a plane that is indistinguishable because its location exceeds the eye's depth of focus becomes distinguishable and appears more focused with a corresponding decrease in pupil size and an increase in depth of field. Similarly, the number of spaced depth planes used to present different images to the viewer can be reduced with a decreased pupil size. For example, a viewer may not be able to clearly perceive details in both a first depth plane and a second depth plane at one pupil size without adjusting their eye's accommodation from one depth plane to the other. However, these two depth planes may be sufficiently focused to the user simultaneously at another pupil size without changing accommodation.

[0070] In some embodiments, the display system may vary the number of waveguides receiving image information based on a determination of pupil size and / or orientation, or in response to receiving an electrical signal indicating a particular pupil size and / or orientation. For example, if a user's eye is unable to distinguish between two depth planes associated with two waveguides, controller 460 may be configured or programmed to stop providing image information to one of those waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing system responsiveness. In embodiments in which the DOE for a waveguide is switchable between on and off states, the DOE may be switched to the off state when the waveguide is receiving image information.

[0071] In some embodiments, it may be desirable for the exit beam to satisfy the condition of having a diameter less than the diameter of the viewer's eye. However, meeting this condition may be difficult in light of the variability in the size of the viewer's pupil. In some embodiments, this condition is met over a wide range of pupil sizes by varying the size of the exit beam in response to a determination of the size of the viewer's pupil. For example, as the pupil size decreases, the size of the exit beam may also decrease. In some embodiments, the exit beam size may be varied using a variable aperture.

[0072] The wearable system 400 may include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the world camera's field of view (FOV), and the imaging system 464 is sometimes referred to as an FOV camera. The entire area available for viewing or imaging by a viewer may be referred to as the ocular field of view (FOR). The FOR may include a solid angle of 4π steradians encompassing the wearable system 400 as the wearer moves their body, head, or eyes to perceive virtually any direction in space. In other situations, the wearer's movement may be more constrained, and the wearer's FOR may correspondingly cover a smaller solid angle. Images obtained from the outward-facing imaging system 464 can be used to track gestures (e.g., hand or finger gestures) made by the user, detect objects in the world 470 in front of the user, etc.

[0073] The wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera) that observes user movements, such as eye and facial movements. The inward-facing imaging system 466 may be used to capture images of the eyes 410 and determine the size or orientation of the pupils of the eyes 304. The inward-facing imaging system 466 may be used to determine the direction the user is looking (e.g., eye pose) or to obtain images for biometric identification of the user (e.g., via iris identification). In some embodiments, at least one camera may be utilized for each eye independently to separately determine the pupil size or eye pose of each eye, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye. In some other embodiments, the pupil diameter or orientation of only a single eye 410 (e.g., using only a single camera per pair of eyes) is determined and assumed to be similar for both eyes of the user. Images obtained by inward-facing imaging system 466 may be analyzed to determine the user's eye posture or mood, which may be used by wearable system 400 to determine audio or visual content to be presented to the user. Wearable system 400 may also determine head pose (e.g., head position or head orientation) using sensors such as an IMU, accelerometer, gyroscope, etc.

[0074] The wearable system 400 may include a user input device 466 through which a user may input commands into the controller 460 and interact with the wearable system 400. For example, the user input device 466 may include a trackpad, touchscreen, joystick, multi-degree-of-freedom (DOF) controller, capacitive sensing device, game controller, keyboard, mouse, directional pad (D-pad), wand, tactile device, totem (e.g., functioning as a virtual user input device), etc. A multi-DOF controller may sense user input in any possible translation (e.g., left / right, forward / backward, or up / down) or rotation (e.g., yaw, pitch, or roll) of some or all of the controller. A multi-DOF controller that supports translation may be referred to as 3DOF, while a multi-DOF controller that supports translation and rotation may be referred to as 6DOF. In some cases, a user may use a finger (e.g., a thumb) to press or swipe on the touch-sensitive input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held by the user's hand while the wearable system 400 is in use. The user input device 466 may communicate with the wearable system 400 via wired or wireless communication.

[0075] 5 shows an example of an output beam output by a waveguide. While one waveguide is shown, it should be understood that other waveguides in waveguide assembly 480 may function similarly, and that waveguide assembly 480 includes multiple waveguides. Light 520 is launched into waveguide 432b at input edge 432c of waveguide 432b and propagates within waveguide 432b by TIR. At the point where light 520 impinges on DOE 432a, a portion of the light exits the waveguide as output beam 510. Although output beams 510 are shown as substantially parallel, they may be redirected (e.g., forming a diverging output beam) to propagate to eye 410 at an angle depending on the depth plane associated with waveguide 432b. It should be understood that a substantially collimated exit beam may refer to a waveguide with light extraction optics that outcouples light and forms an image that appears to be set on a depth plane at a long distance (e.g., optical infinity) from the eye 410. Other waveguides or other sets of light extraction optics may output a more divergent exit beam pattern, which would require the eye 410 to accommodate to a closer distance and focus it on the retina, and would be interpreted by the brain as light from a distance closer to the eye 410 than optical infinity.

[0076] FIG. 6 is a schematic diagram illustrating an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem used in generating a multifocal stereoscopic display, image, or bright field. The optical system can include a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem. The optical system can be used to generate a multifocal stereoscopic image or bright field. The optical system can include one or more primary planar waveguides 632a (only one is shown in FIG. 6) and one or more DOEs 632b associated with each of at least some of the primary waveguides 632a. The planar waveguides 632b can be similar to the waveguides 432b, 434b, 436b, 438b, and 440b discussed with reference to FIG. 4. The optical system may employ a distribution waveguide device to relay light along a first axis (vertical or Y-axis in the illustration of FIG. 6 ) and expand the effective exit pupil of the light along the first axis (e.g., Y-axis). The distribution waveguide device may include, for example, a distribution planar waveguide 622 b and at least one DOE 622 a (illustrated by a double-dashed line) associated with the distribution planar waveguide 622 b. The distribution planar waveguide 622 b may be similar or identical in at least some respects to a primary planar waveguide 632 b having a different orientation therefrom. Similarly, the at least one DOE 622 a may be similar or identical in at least some respects to the DOE 632 a. For example, the distribution planar waveguide 622 b or the DOE 622 a may be made of the same material as the primary planar waveguide 632 b or the DOE 632 a, respectively. The embodiment of the optical display system 600 shown in FIG. 6 can be integrated into the wearable system 200 shown in FIG.

[0077] The relayed, exit-pupil-expanded light can be optically coupled from the distribution waveguide device into one or more primary planar waveguides 632b. The primary planar waveguides 632b can relay the light along a second axis, preferably orthogonal to the first axis (e.g., the horizontal or X-axis in the diagram of FIG. 6). Notably, the second axis can be non-orthogonal to the first axis. The primary planar waveguides 632b expand the effective exit pupil of the light along the second axis (e.g., the X-axis). For example, the distribution planar waveguide 622b can relay and expand the light along the vertical or Y-axis and pass the light to a primary planar waveguide 632b that can relay and expand the light along the horizontal or X-axis.

[0078] The optical system may include one or more colored light sources (e.g., red, green, and blue laser light) 610, which may be optically coupled into the proximal end of a single-mode optical fiber 640. The distal end of the optical fiber 640 may be threaded into or received through a hollow tube 642 of piezoelectric material. The distal end protrudes from the tube 642 as a free-standing, flexible cantilever 644. The piezoelectric tube 642 may be associated with four quadrant electrodes (not shown). The electrodes may be plated, for example, on the outside, outer surface or circumference, or diameter of the tube 642. A core electrode (not shown) may also be located in the core, center, inner circumference, or inner diameter of the tube 642.

[0079] For example, drive electronics 650, electrically coupled via wires 660, drive opposing pairs of electrodes to bend piezoelectric tube 642 independently in two axes. The protruding distal tip of optical fiber 644 has a mechanical resonant mode. The frequency of the resonance may depend on the diameter, length, and material properties of optical fiber 644. By oscillating piezoelectric tube 642 near the first mechanical resonant mode of fiber cantilever 644, fiber cantilever 644 may be caused to oscillate and sweep through a large deflection.

[0080] By stimulating resonant vibrations in two axes, the tip of fiber cantilever 644 is scanned biaxially over an area filling a two-dimensional (2D) scan. By modulating the intensity of light source 610 synchronously with the scanning of fiber cantilever 644, light emitted from fiber cantilever 644 can form an image. A description of such a setup is provided in U.S. Patent Publication No. 2014 / 0003762, which is incorporated herein by reference in its entirety.

[0081] Components of the optical coupler subsystem can collimate light emitted from the scanning fiber cantilever 644. The collimated light can be reflected by a mirrored surface 648 into a narrow distribution planar waveguide 622b, which includes at least one diffractive optical element (DOE) 622a. The collimated light can propagate perpendicularly (with respect to the view of FIG. 6) along the distribution planar waveguide 622b via TIR, thereby repeatedly intersecting with the DOE 622a. The DOE 622a preferably has a low diffraction efficiency. This causes a portion of the light (e.g., 10%) to diffract toward the edge of the larger primary planar waveguide 632b at each point of intersection with the DOE 622a, allowing a portion of the light to continue on its original trajectory down the length of the distribution planar waveguide 622b via TIR.

[0082] At each point of intersection with DOE 622a, additional light can be diffracted toward the entrance of primary waveguide 632b. By splitting the incident light into multiple outcoupled sets, the exit pupil of the light can be vertically expanded by DOE 4 within distribution planar waveguide 622b. This vertically expanded light outcoupled from distribution planar waveguide 622b can enter the edge of primary planar waveguide 632b.

[0083] Light entering the primary waveguide 632b can propagate horizontally (with respect to the illustration of FIG. 6) along the primary waveguide 632b via TIR. The light intersects the DOE 632a at multiple points as it propagates horizontally along at least a portion of the length of the primary waveguide 632b via TIR. The DOE 632a advantageously has a phase profile that is the sum of a linear diffraction pattern and a radially symmetric diffraction pattern, and may be designed or configured to produce both deflection and focusing of the light. The DOE 632a advantageously may have a low diffraction efficiency (e.g., 10%) so that only a portion of the light in the beam is deflected toward the viewer's eye at each intersection of the DOE 632a, while the remainder of the light continues to propagate through the primary waveguide 632b via TIR.

[0084] At each point of intersection between the propagating light and the DOE 632a, a portion of the light is diffracted toward the adjacent face of the primary waveguide 632b, allowing the light to escape TIR and emerge from the face of the primary waveguide 632b. In some embodiments, the radially symmetric diffraction pattern of the DOE 632a additionally imparts a focal level to the diffracted light, both shaping the optical wavefronts of the individual beams (e.g., imparting curvature) and steering the beams to angles that match the designed focal level.

[0085] These different paths can therefore allow light to be coupled out of the primary planar waveguide 632b by the multiple DOEs 632a at different angles, focal levels, or result in different fill patterns at the exit pupil. Different fill patterns at the exit pupil can be advantageously used to generate bright-field displays with multiple depth planes. Each layer or set of layers (e.g., three layers) in the waveguide assembly or stack can be employed to generate a respective color (e.g., red, blue, green). Thus, for example, a first set of three adjacent layers can be employed to generate red, blue, and green light, respectively, at a first focal depth. A second set of three adjacent layers can be employed to generate red, blue, and green light, respectively, at a second focal depth. Multiple sets can be employed to generate full 3D or 4D color image bright-fields with various focal depths.

[0086] Some embodiments of the wearable system may render virtual objects on different depth planes (e.g., as described with reference to FIG. 3 ), although this is illustrative and not intended to be limiting. Other optical techniques can also be used to render virtual objects so that they appear to the user at different depths. For example, a variable focus element (VFE) can be used, for example, as described in U.S. Patent Publication No. 2015 / 0346495 (incorporated herein by reference in its entirety). In other embodiments of the wearable system, different virtual objects may be rendered on the same depth plane but nevertheless appear to the user as if they are at different depths. For example, the apparent depth of virtual content rendered on a depth plane can be varied by varying the rendering location of pixels associated with the virtual content so that the virtual content has different vergence locations (and different perceived depths). Thus, two virtual objects can be rendered on the same depth plane, but a second virtual object (relative to the first virtual object) can be perceived as being closer to the user, at the same depth from the user, or farther from the user by modifying the pixel rendering location and creating a different vergence location for the second virtual object (relative to the first virtual object). Thus, the perceived depth of different virtual content can be achieved by rendering the different virtual content on the same depth plane but adjusting for vergence. (Other components of the wearable system)

[0087] In many implementations, the wearable system may include other components in addition to or as an alternative to the components of the wearable system described above. The wearable system may include, for example, one or more tactile devices or components. The tactile device or component may be operable to provide a tactile sensation to the user. For example, the tactile device or component may provide a tactile sensation of pressure and / or texture upon touching virtual content (e.g., a virtual object, virtual tool, other virtual structure). The tactile sensation may replicate the sensation of a physical object represented by the virtual object, or may replicate the sensation of an imaginary object or character (e.g., a dragon) represented by the virtual content. In some implementations, the tactile device or component may be worn by the user (e.g., a user-wearable glove). In some implementations, the tactile device or component may be held by the user.

[0088] A wearable system may include, for example, one or more physical objects that can be manipulated by a user and enable input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as, for example, a piece of metal or plastic, a wall, the surface of a table, etc. In some implementations, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, the totem may simply provide a physical surface, and the wearable system may render a user interface to appear to the user as being on one or more surfaces of the totem. For example, the wearable system may render images of a computer keyboard and trackpad to appear to reside on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and virtual trackpad to appear on the surface of a thin rectangular plate of aluminum that serves as the totem. The rectangular plate itself does not have any physical keys, trackpads, or sensors. However, the wearable system may detect user manipulation or interaction or touch with the rectangular plate as a selection or input made via a virtual keyboard or virtual trackpad. User input device 466 (shown in FIG. 4) may be an embodiment of a totem, which may include a trackpad, touchpad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. A user may use the totem alone or in combination with posture to interact with the wearable system and / or other users.

[0089] Examples of tactile devices and totems usable with the wearable devices, HMDs, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated herein by reference in its entirety. Exemplary Wearable Systems, Environments, and Interfaces

[0090] The wearable system may employ various mapping-related techniques to achieve a high depth of field within the rendered light field. In mapping a virtual world, it is advantageous to capture all features and points in the real world and accurately depict virtual objects in relation to the real world. To achieve this goal, FOV images captured from a user of the wearable system can be added to the world model by including new photos that convey information about various points and features in the real world. For example, the wearable system can collect a set of map points (such as 2D or 3D points), find new map points, and render a more accurate version of the world model. The world model of a first user can be communicated to a second user (e.g., via a network such as a cloud network) so that the second user can experience the world surrounding the first user.

[0091] 7 is a block diagram of an example MR environment 700. The MR environment 700 may be configured to receive inputs (e.g., visual input 702 from a user's wearable system, stationary input 704 such as a room camera, sensory input 706 from various sensors, gestures, totems, eye tracking, user input, etc. from user input device 466) from one or more user-wearable systems (e.g., wearable system 200 or display system 220) or stationary room systems (e.g., room cameras, etc.). The wearable systems can use various sensors (e.g., accelerometers, gyroscopes, temperature sensors, movement sensors, depth sensors, GPS sensors, inward-facing imaging systems, outward-facing imaging systems, etc.) to determine the location and various other attributes of the user's environment. This information may be further supplemented with information from stationary cameras in the room, which may provide images from different perspectives or various cues. Image data acquired by cameras (e.g., room cameras or outward-facing imaging system cameras) may be compiled into a set of mapping points.

[0092] One or more object recognizers 708 can crawl through the received data (e.g., a collection of points), recognize or map the points, tag the images, and attach semantic information to the objects using a map database 710. The map database 710 can comprise various points and their corresponding objects collected over time. The various devices and the map database can be connected to each other through a network (e.g., a LAN, a WAN, etc.) and can be accessed from the cloud.

[0093] Based on this information and the collection of points in the map database, the object recognizers 708a-708n can recognize objects in the environment. For example, the object recognizers can recognize faces, people, windows, walls, user input devices, televisions, other objects in the user's environment, etc. One or more object recognizers may be specialized for objects with certain characteristics. For example, object recognizer 708a may be used to recognize faces, while another object recognizer may be used to recognize totems.

[0094] Object recognition can be performed using various computer vision techniques. For example, the wearable system can analyze images acquired by the outward-facing imaging system 464 (shown in FIG. 4) and perform scene reconstruction, event detection, video tracking, object recognition, object pose estimation, learning, indexing, motion estimation, or image restoration, etc. One or more computer vision algorithms can be used to perform these tasks. Non-limiting examples of computer vision algorithms include Scale Invariant Feature Transform (SIFT), Speed-Up Robust Features (SURF), Orientation Fast and Rotation Brief (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retinal Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, visual simultaneous localization and mapping (vSLAM) techniques, sequential Bayes estimators (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, adaptive thresholding (and other thresholding techniques), iterative nearest neighbor (ICP), semi-global matching (SGM), semi-global block matching (SGBM), feature point histograms, various machine learning algorithms (e.g., support vector machines, k-nearest neighbor algorithms, naive Bayes, neural networks (including convolutional or deep neural networks), or other supervised / unsupervised models, etc.), etc.

[0095] Object recognition can additionally or alternatively be performed by various machine learning algorithms. Once trained, the machine learning algorithms can be stored by the HMD. Some examples of machine learning algorithms can include supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., ordinary least squares regression, etc.), instance-based algorithms (e.g., learning vector quantization, etc.), decision tree algorithms (e.g., classification and regression trees, etc.), Bayesian algorithms (e.g., naive Bayes, etc.), clustering algorithms (e.g., k-means clustering, etc.), association rule learning algorithms (e.g., a priori algorithm, etc.), artificial neural network algorithms (e.g., Perceptron, etc.), deep learning algorithms (e.g., Deep Boltzmann Machine, i.e., deep neural networks, etc.), dimensionality reduction algorithms (e.g., principal component analysis, etc.), ensemble algorithms (e.g., stacked generalization, etc.), or other machine learning algorithms. In some embodiments, individual models can be customized for individual datasets. For example, the wearable device can generate or store a base model. The base model may be used as a starting point to generate additional models specific to a data type (e.g., a particular user in a telepresence session), a data set (e.g., an additional set of images acquired of a user in a telepresence session), a conditional situation, or other variations. In some embodiments, the wearable HMD can be configured to generate models for analysis of aggregated data using multiple techniques. Other techniques may include using predefined thresholds or data values.

[0096] Based on this information and the set of points in the map database, the object recognizers 708a-708n can recognize objects, complement them with semantic information, and bring them to life. For example, if the object recognizer recognizes that a set of points is a door, the system can attach some semantic information (e.g., a door has a hinge and 90-degree movement around the hinge). If the object recognizer recognizes that a set of points is a mirror, the system can attach semantic information that a mirror has a reflective surface that can reflect images of objects in the room. Over time, the map database expands as the system (which may reside locally or be accessible over a wireless network) accumulates more data from the world. Once an object is recognized, the information can be transmitted to one or more wearable systems. For example, the MR environment 700 may include information about a scene generating in California. The environment 700 may be transmitted to one or more users in New York. Based on the data received from the FOV camera and other inputs, the object recognizer and other software components can map points collected from various images, recognize objects, etc. so that the scene can be accurately "passed" to a second user who may be in a different part of the world. The environment 700 may also use a topology map for localization purposes.

[0097] The object recognizer may identify objects within a 3D environment, and from the system's knowledge of the current location of a cursor used to select or interact with the object, this information may be used to implement the occlusion cursor techniques described herein. For example, if the cursor location is near a target object identified by the object recognizer, a focus indicator may be provided or highlighted around the target object. The object recognizer may determine the object's location (e.g., center) or the object's edge or boundary, and the location of the cursor (e.g., a ray of light from the user toward the cursor location) relative to the object's center or edge or boundary may be used to determine whether and how to render a focus indicator, whether to accelerate the cursor toward the object (e.g., attraction effects described herein), etc.

[0098] 8 is a process flow diagram of an example method 800 for rendering virtual content in relation to recognized objects. Method 800 describes how a virtual scene can be presented to a user of a wearable system. The user may be geographically remote from the scene. For example, a user may be in New York but may want to view a scene currently occurring in California, or may want to go for a walk with a friend who is in California.

[0099] In block 810, the wearable system may receive input from the user and other users regarding the user's environment. This may be accomplished through various input devices and knowledge already held in a map database. The user's FOV camera, sensors, GPS, eye tracking, etc., communicate information to the system in block 810. The system may determine sparse points based on this information in block 820. The sparse points may be used to determine pose data (e.g., head pose, eye pose, body pose, or hand gestures) that may be used in displaying and understanding the orientation and position of various objects in the user's surroundings. The object recognizer 708a, 708n may crawl through these collected points and recognize one or more objects using the map database in block 830. This information may then be communicated to the user's respective wearable system in block 840, and the desired virtual scene may be displayed to the user appropriately in block 850. For example, a desired virtual scene (eg, a user in CA) may be displayed in the appropriate orientation, position, etc., relative to various objects and other surroundings of the user in New York.

[0100] FIG. 9 is a block diagram of another example of a wearable system. In this example, the wearable system 900 includes a map that may include map data about the world. The map may reside partially locally on the wearable system and partially in a networked storage location (e.g., in a cloud system) accessible by a wired or wireless network. An attitude process 910 runs on the wearable computing architecture (e.g., processing module 260 or controller 460) and may utilize data from the map to determine the position and orientation of the wearable computing hardware or the user. The attitude data may be calculated from data collected in real time as the user experiences the system and moves within the world. The data may include images, data from sensors (such as an inertial measurement unit (IMU) that may include accelerometers, gyroscopes, magnetometers, or a combination of such components), and surface information about objects in the real or virtual environment.

[0101] A sparse point representation can be the output of a simultaneous localization and mapping (SLAM or V-SLAM: refers to configurations where the input is only image / vision) process. The system can be configured to find not only the location of various components in the world, but also what the world is made up of. Poses can be building blocks that accomplish many goals, including populating and using data from maps.

[0102] In one embodiment, sparse point locations may not be entirely adequate by themselves, and additional information may be required to generate a multifocal AR, VR, or MR experience. A dense representation, generally referring to depth map information, may be utilized to at least partially fill this gap. Such information may be calculated from a process referred to as stereoscopic vision 940, with depth information determined using techniques such as triangulation or time-of-flight sensing. Image information and active patterns (such as infrared patterns generated using an active projector) may serve as inputs to the stereoscopic vision process 940. A significant amount of depth map information may be fused together, some of which may be summarized using a surface representation. For example, mathematically definable surfaces may be an efficient (e.g., compared to large-scale point clouds) and applicable input to other processing devices, such as game engines. Thus, the outputs of the stereoscopic vision process (e.g., depth map) 940 may be combined in a fusion process 930. Pose may also be an input to this fusion process 930, the output of which is the input for populating the map process 920. Sub-surfaces can connect to each other to form larger surfaces, such as in topographic mapping, where the map becomes a large-scale hybrid of points and surfaces.

[0103] Various inputs may be utilized to resolve various aspects of the mixed reality process 960. For example, in the embodiment depicted in FIG. 9, game parameters may be inputs for determining whether a user of the system is playing a monster battle game with one or more monsters in various locations, whether a monster is dead or fleeing under various conditions (such as when the user shoots the monster), walls or other objects in various locations, etc. A world map may contain information about where such objects reside relative to one another, which is another useful input for mixed reality. Orientation relative to the world is also an input and plays an important role for nearly any interactive system. Parameters and inputs such as these may be used to provide occlusion cursor functionality within the mixed reality process 960.

[0104] Controls or inputs from the user are another input to the wearable system 900. As described herein, user inputs can include visual inputs, gestures, totems, audio inputs, sensory inputs, etc. To move around or play games, for example, the user may need to command the wearable system 900 as to what they want to do. There are various forms of user control that can be utilized beyond moving around in space. In one embodiment, a totem (e.g., a user input device), or an object such as a toy gun, may be held by the user and tracked by the system. The system would preferably be configured to know that the user is holding an item and understand the type of interaction the user is having with the item (e.g., if the totem or object is a gun, the system may be configured to understand not only the location and orientation, but also whether the user is clicking a trigger or other sensitive button or element, which may be equipped with sensors such as an IMU, which can help determine what is happening even when such activity is not within the field of view of any of the cameras).

[0105] Hand gesture tracking or recognition may also provide input information. The wearable system 900 may be configured to track and interpret hand gestures with respect to button presses, gesturing left or right, stop, grasp, hold, etc. For example, in one configuration, a user may want to flip through email or calendar in a non-gaming environment, or perform a “fist bump” with another person or player. The wearable system 900 may be configured to utilize a minimal amount of hand gestures, which may or may not be dynamic. For example, gestures may be simple static gestures, such as extending the hand to indicate stop, giving a thumbs up to indicate OK, giving a thumbs down to indicate not OK, or flipping the hand left and right or up and down to indicate a directional command.

[0106] Eye tracking is another input (e.g., to track where the user is looking, control display technology, render at a particular depth or range). In one embodiment, eye vergence can be determined using triangulation, and then accommodation can be determined using a vergence / accommodation model developed for that particular person.

[0107] With regard to the camera system, the exemplary wearable system 900 shown in FIG. 9 may include three pairs of cameras: a relatively wide-FOV or passive SLAM pair of cameras arranged on either side of the user's face, a different pair of cameras pointed in front of the user for handling the stereoscopic imaging process 940, and capturing hand gestures and totem / object trajectories in front of the user's face. The FOV cameras and pair of cameras for the stereo process 940 may be part of the outward-facing imaging system 464 (shown in FIG. 4). The wearable system 900 may include eye-tracking cameras (which may be part of the inward-facing imaging system 462 shown in FIG. 4) pointed at the user's eyes to triangulate eye vectors and other information. The wearable system 900 may also include one or more textured light projectors (such as infrared (IR) projectors) to inject texture into the scene.

[0108] 10 is a process flow diagram of an example method 1000 for determining user input to a wearable system. In this example, a user may interact with a totem. A user may have multiple totems. For example, a user may have one totem designated for social media applications, another totem for playing games, etc. In block 1010, the wearable system may detect movement of the totem. The movement of the totem may be recognized through an outward-facing system or may be detected through sensors (e.g., tactile gloves, image sensors, hand tracking devices, eye tracking cameras, head pose sensors, etc.).

[0109] Based at least in part on the detected gestures, eye postures, head postures, or inputs through the totem, the wearable system detects the position, orientation, or movement of the totem (or the user's eyes, head, or gestures) relative to a reference frame in block 1020. The reference frame may be a set of map points based on which the wearable system translates the totem's (or the user's) movements into actions or commands. In block 1030, the user's interactions with the totem are mapped. Based on the mapping of the user interactions to the reference frame 1020, the system determines the user input in block 1040.

[0110] For example, a user may move a totem or physical object back and forth to indicate turning a virtual page, moving to the next page, or moving from one user interface (UI) display screen to another. As another example, a user may move their head or eyes to view different real or virtual objects in the user's FOR. If the user's gaze at a particular real or virtual object is longer than a threshold time, that real or virtual object may be selected as user input. In some implementations, the user's vergence movements can be tracked, and an accommodation / vergence model can be used to determine the user's eye accommodation state, which provides information about the depth plane on which the user is focusing. In some implementations, the wearable system can use ray casting techniques to determine real or virtual objects that are aligned with the user's head or eye pose. In various implementations, ray casting techniques can include casting a thin bundle of rays with substantially little lateral width, or casting rays with substantial lateral width (e.g., a cone or truncated cone).

[0111] The user interface may be projected by a display system (such as display 220 in FIG. 2) as described herein. It may also be displayed using various other techniques, such as one or more projectors. A projector may project an image onto a physical object, such as a canvas or a sphere. Interactions with the user interface may be tracked using one or more cameras outside or part of the system (e.g., using inward-facing imaging system 462 or outward-facing imaging system 464).

[0112] 11 is a process flow diagram of an example method 1100 of interacting with a virtual user interface. Method 1100 may be performed by the wearable systems described herein.

[0113] In block 1110, the wearable system may identify a particular UI. The type of UI may be predetermined by the user. The wearable system may identify that a particular UI needs to be populated with data based on user input (e.g., gestures, visual data, audio data, sensory data, direct commands, etc.). In block 1120, the wearable system may generate data for a virtual UI. For example, data associated with the UI's boundaries, general structure, shape, etc. may be generated. Additionally, the wearable system may determine map coordinates of the user's physical location so that the wearable system may display the UI in relation to the user's physical location. For example, if the UI is body-centered, the wearable system may determine coordinates of the user's physical position, head pose, or eye pose so that a ring UI may be displayed around the user or a planar UI may be displayed on a wall or in front of the user. If the UI is hand-centered, map coordinates of the user's hand may be determined. These map points may be derived through an FOV camera, data received through sensory input, or any other type of collected data.

[0114] In block 1130, the wearable system may transmit data from the cloud to the display, or data may be transmitted from a local database to the display component. In block 1140, a UI is displayed to the user based on the transmitted data. For example, a bright field display can project the virtual UI into one or both of the user's eyes. Once the virtual UI is generated, the wearable system may simply wait for a command from the user to generate more virtual content on the virtual UI in block 1150. For example, the UI may be a body-centered ring around the user's body. The wearable system may then wait for a command (gesture, head or eye movement, input from a user input device, etc.), and if recognized (block 1160), virtual content associated with the command may be displayed to the user (block 1170). By way of example, the virtual content may include a virtual cursor (or reticle) and focus indicator associated with an object in the environment. The virtual cursor and focus indicator may comprise aspects of the occlusion cursor technology described with reference to FIGS. 12A-24 .

[0115] Additional examples of wearable systems, UIs, and user experiences (UX) are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated herein by reference in its entirety. (Example objects in the field of view (FOV))

[0116] 12A-12C illustrate various examples of objects 1204 and cursors 1202 that may be perceived by a user via a wearable system. FIG. 12A shows an example of a 2D environment, and FIGS. 12B-12C show an example of a 3D environment. In various embodiments, objects within a user's field of view (FOV) may be virtual or physical. For example, one or more objects may include physical objects such as a chair, a tree, a sofa, a wall, etc., while virtual objects may include operating system objects such as a trash can for deleted files, a terminal for entering commands, a file manager for accessing files or directories, icons, menus, applications for audio or video streaming, notifications from the operating system, etc. Virtual objects may also include objects within an application, such as an avatar, a virtual object in a game, a graphic, or an image. Some virtual objects can be both operating system objects and objects within an application. In some embodiments, the wearable system can add virtual elements to existing physical objects. For example, the wearable system may add a virtual menu associated with a television in a room, and the virtual menu may give the user options to turn on the television or change its channel using the wearable system.

[0117] A virtual object can be a three-dimensional (3D), two-dimensional (2D), or one-dimensional (1D) object. For example, a virtual object can be a 3D coffee mug (which may represent virtual controls for a physical coffee maker). A virtual object can also be a 2D graphical representation of a clock (which displays the current time to the user). In some implementations, one or more virtual objects can be displayed within (or associated with) another virtual object. A virtual coffee mug can be shown inside the user interface plane, but the virtual coffee mug appears to be 3D within this 2D planar virtual space. (Using cursors)

[0118] Continuing with reference to FIGS. 12A-12C , the wearable system displays cursor 1202, which may be a movable indicator that a user can utilize to select or interact with an object in the environment. The cursor may be displayed within a limited region of the environment (e.g., a location within the FOV). In some cases, the cursor represents a location where user interaction with a real or virtual object may occur. For example, a user may utilize cursor 1202 to select, view, or point to an object, such as object 1204. By changing the location of cursor 1202, a user can alter the selection or view or change where cursor 1202 is pointing. In various implementations, a user can change the location of the cursor by, for example, translating or rotating a handheld totem, moving a finger (e.g., a thumb) across a touch-sensitive portion of the totem or other user input device, translating or rotating a body part (e.g., a finger, hand, or arm), or moving their head or eyes.

[0119] The appearance of cursor 1202 can take any of a variety of different colors, outlines, shapes, symbols, sizes, images, graphics, combinations, etc. For example, cursor 1202 can take on a variety of shapes, such as a cursor, a geometric cone, a beam of light, an arrow, an oval, a circle, a polygon, or other 1D, 2D, or 3D shape.

[0120] The cursor 1202 may be used to select, view, or point to an object, such as object 1204, by moving the cursor 1202 so that it hovers over, hovers behind, or otherwise points to the target object 1204. Once the cursor 1202 and the target object 1204 are sufficiently aligned, the user may select or interact with the target object 1204 that the cursor 1204 is hovering or pointing at, for example, by making a hand gesture, activating a touch-sensitive portion of a totem, etc.

[0121] The user can move their body, head, or eyes to move cursor 1202. For example, changes in the user's posture (e.g., head posture, body posture, or eye gaze) can alter the location of cursor 1202 within the FOV. Similarly, cursor 1202 can be controlled through a user input device, such as user input device 466 of FIG. 4. For example, the user input device can include a trackpad, touchscreen, joystick, multi-degree-of-freedom (DOF) controller, capacitive sensing device, game controller, keyboard, mouse, directional pad (D-pad), wand, haptic device, totem (e.g., functioning as a virtual user input device), etc. For example, as the user moves their hand over the user input device, cursor 1202 can move from a first position to a second position. (content obfuscation)

[0122] Some systems render the cursor without considering the scene content. In other words, as the user moves the cursor around the scene, the cursor is rendered in front of objects in the scene. When the cursor is used to target or select an object, it can occlude or obscure the object as it hovers over the object. This can affect the user's experience in the environment. For example, the user may want to see the object, but the cursor is rendered in front of the object, thereby blocking the user's view of the object. These problems can be exacerbated when the object contains text, images, or other content that the user wants to view. Furthermore, when the cursor is rendered in front of the target object, the cursor is higher in the user's visual hierarchy, which can be distracting. Because the user is attempting to interact with real or virtual objects in the environment, not the cursor, the cursor should preferably function as a tool, not the highest or highest object in the visual hierarchy.

[0123] 12A illustrates some examples of problems associated with a cursor in a 2D environment. As shown, FIG. 12A illustrates various locations of a cursor 1202 as it moves around the 2D environment, and specifically, as the cursor 1202 moves from position 1212 (e.g., cursor 1202 is above object 1204), to position 1214 (e.g., cursor 1202 is in front of object 1204), to position 1216 (e.g., cursor 1202 is below object 1204).

[0124] 12A , cursor 1202 and object 1204 do not have depth. In other words, cursor 1202 is rendered at the same depth as object 1204, and when cursor 1202 and object 1204 overlap, cursor 1202 is shown in place of object 1204. For example, as cursor 1202 moves from position 1212 to position 1214, cursor 1202 appears to be "rolling over" object 1204 such that cursor 1202 is shown and a portion of object 1204 behind the cursor is obstructed from the user's view.

[0125] 12B illustrates an example of how cursor occlusion may be exacerbated within a 3D environment. As shown, FIG. 12B illustrates various locations of cursor 1202 as it moves about in the 3D environment, specifically as cursor 1202 moves from position 1222 (e.g., cursor 1202 is above and centered on object 1204), to position 1224 along path 1250a (e.g., cursor 1202 is in front of object 1204), to position 1226 (e.g., cursor 1202 is centered below and centered on object 1204).

[0126] In the 3D environment of Figure 12B, cursor 1202 and object 1204 have depth. In other words, in order for cursor 1202 to "roll over" or "roll around" an object in 3D space, cursor 1202 must move closer or farther from the user so that cursor 1202 and object 1204 do not overlap. For example, if cursor 1202 and the object were kept at the same depth during the "roll," cursor 1202 might appear to pass through object 1204, which may be undesirable because it disrupts realism and may obscure part of cursor 1202 or part of object 1204. In Figure 12B, when moved along path 1250a, the cursor is moved closer to the user so that it is in front of object 1204 and between object 1204 and the user (see, e.g., position 1224). For example, object 1204 in FIG. 12B is a person with their arms outstretched in front of their body. Initially, at position 1222, cursor 1202 is approximately the same distance from the user as person 1204. However, as cursor 1202 moves along path 1250a from position 1222 to position 1224, the system must bring cursor 1202 closer to the user so that cursor 1202 is in front of person 1204's outstretched arm. By bringing cursor 1202 closer to the user, the system effectively emphasizes cursor 1202 relative to person 1204, because users are more likely to focus on objects that appear closer to them. To reduce the emphasis on the cursor, the system may dynamically adjust the cursor's size to maintain a consistent appearance of the cursor. However, because the cursor's perceived dimensions will change based on its distance from the user, this type of perceptual change may confuse the user or, at a minimum, provide misleading information to the user. Thus, although it would be desirable for a user to highlight the person 1204 they are attempting to interact with by utilizing a cursor 1202 passing in front of the person, the system undesirably highlights the cursor 1202 over the person.

[0127] To reduce the likelihood of highlighting the cursor 1202 when the cursor 1202 and the object 1204 overlap, the cursor may move along a path 1250b that effectively passes behind the object 1204 (so that the object 1204 “obscures” the cursor). The cursor 1202 is thereby de-highlighted relative to the foreground object 1204. When the cursor 1202 is behind the object 1204 (such as the person in FIG. 12B ), the wearable system may stop rendering the cursor because the cursor is invisible to the user (e.g., the object is opaque and “obscures” the cursor). In some embodiments, when behind the object 1204, the cursor 1202 continues to be rendered by the wearable system but remains de-highlighted relative to the object, or is rendered at a reduced brightness level such that the object 1204 is rendered in front of the cursor 1202 (e.g., the object occludes the cursor), which may effectively reduce the perceivability of the cursor relative to the object. Additionally or alternatively, when the cursor is behind an object, the system may render a focus indicator (e.g., a glow or halo) around the object 1204 to provide the user with a visual indication of the cursor's location, as described further below.

[0128] 12C illustrates an example of how an object may change in size or shape or perceived depth as the cursor moves behind the object in a 3D environment. As shown, FIG. 12C illustrates various locations of cursor 1202 as it moves around in the 3D environment, and specifically as cursor 1202 moves from position 1222 (e.g., cursor 1202 is above and centered on object 1204) to position 1226 (e.g., cursor 1202 is centered on and below object 1204).

[0129] 12C , rather than moving cursor 1202 closer to the user (e.g., path 1250a in FIG. 12B ) or moving cursor 1202 further away from the user (e.g., path 1250b in FIG. 12B ), the system may move cursor 1202 along path 1250c. In response, object 1204 may become larger, or object 1204 may move closer to the user, or the object's foreground may expand (or any combination thereof), causing icon 1204 to shift closer or appear larger or closer to the user. In other words, rather than altering the depth of cursor 1202 relative to the user or object 1204, the system may keep the cursor at the same depth and adjust the relative depth of object 1204 so that object 1204 appears closer to the user than the cursor. In some embodiments, object 1204 may shift to a new depth (e.g., a depth closer to the user). In some embodiments, object 1204 may remain at the same depth as the cursor, but may be rendered with closer vergence display input so that it appears closer to the user. By bringing object 1204 closer to the user, the system advantageously emphasizes object 1204 because the user is more likely to focus on an object that appears closer to them. Furthermore, similar to path 1250b in FIG. 12b, as the object is shifted toward the user, cursor 1202 may move along path 1250c, which effectively passes behind object 1204 (so that object 1204 “obscures” the cursor). While behind object 1204, cursor 1202 may not be rendered or may be rendered with reduced brightness or increased transparency. Cursor 1202 is thereby de-emphasized relative to foreground object 1204.Once the cursor 1202 is no longer behind the object 1204, the wearable system may shift the object 1204 back to its original position and adjust the object to its original size, thereby de-highlighting the object 1204 and rendering the cursor 1202 (no longer occluded) so that the user can once again see the cursor and object. As described above, although the cursor 1202 is occluded by the object 1204, the system may render a focus indicator around or adjacent to at least a portion of the object 1204, which further emphasizes the foreground object relative to the cursor.

[0130] Additional examples of occlusion cursor behavior are described below with reference to Figures 18A-18C and 30A-30F. (Using focus indicators)

[0131] In some cases, when cursor 1202 is positioned behind object 1204, the user may have difficulty getting an accurate sense of the cursor's location within the scene. For example, cursor 1202 may be (at least partially) occluded by object 1204, making it difficult for the user to visually regain the cursor in the future or to remember which object was selected. Thus, to give the user an accurate sense of the cursor's location within the environment, in some cases the system may highlight object 1204 when cursor 1202 moves behind that object 1204. For example, the system may assign a focus indicator (e.g., some form of visual highlight) to object 1204. Thus, when cursor 1202 moves near or behind object 1204, the focus indicator is activated and object 1204 is highlighted, while the user still gets an accurate sense of the cursor's location within the scene and which object was selected.

[0132] 13A and 13B illustrate non-limiting embodiments of the focus indicator 1302 and cursor 1202. The appearance of the focus indicator 1302 can take any of a variety of different colors, outlines, shapes, symbols, sizes, images, graphics, combinations thereof, and the like. For example, the cursor 1202 can take various shapes, such as a cursor, a geometric cone, a beam of light, an arrow, a crosshair, an oval, a circle, a polygon, or other 1D, 2D, or 3D shape. The focus indicator 1302 can include, but is not limited to, a halo, a color, a perceived size or depth change (e.g., causing an object to appear closer or larger when selected), a virtual ray, line, or arc, or other graphical highlight (emanating from, surrounding, or associated with at least a portion of the object or its periphery) to draw the user's attention to the object. The focus indicator 1302 can include a glow that radiates from behind the object, and the intensity of the glow can correspond to the spatial relationship between the location of the cursor and the location of the object. For example, the intensity of the glow may be greatest near the edge of the object and decrease with distance away from the object. The intensity of the glow may be greatest on the side of the object closest to the cursor (e.g., along a line between the object and the cursor), and the intensity may be lower (or zero) on the side of the object farther from the cursor. As illustrated in Figure 13B, the cursor or focus indicator may have a rainbow-colored appearance. The focus indicator 1302 (or cursor 1202) may also include audible or tactile effects, such as vibrations, ring tones, beeps, etc.

[0133] When cursor 1202 is positioned behind an object, cursor 1202 may be large enough, or the object may be small enough, so that an outer portion of the cursor gives the appearance of a focus indicator surrounding the object. In some cases, as illustrated in Figures 13A and 13B, focus indicator 1302 may appear as a larger version of cursor 1202. For example, the system may render cursor 1202, and in response to the cursor being below a threshold distance between it and the object, cursor 1202 may be obscured (e.g., not rendered) in favor of focus indicator 1302 appearing around the object. Furthermore, in response to the cursor being above the threshold distance between it and the object, the cursor may again be rendered by the system and the focus indicator may be de-emphasized. (Example of cursor and focus indicator)

[0134] 14A and 14B illustrate an example of an environment including a cursor 1202 and an object 1204. In this example, the object 1204 is a cube resting on a table. When the cursor 1202 transitions from a position 1412 in FIG. 14A to a position 1414 in FIG. 14B, the cursor 1202 moves onto (or in front of) the object 1204. Thus, when the cursor 1202 reaches position 1414, the cursor 1202 blocks a portion of the object 1204 from the user's view, which, as discussed above, can be a nuisance to the user.

[0135] Figure 14C illustrates an example environment that graphically illustrates features of the occlusion cursor functionality. In contrast to Figure 14B, cursor 1414 moves behind object 1204 and is not rendered to the user (indicated by a dashed line in Figure 14C). As cursor 1414 approaches and passes behind object 1204, focus indicator 1302 is rendered to the user. In this example, focus indicator 1302 is a circular halo surrounding a cube, with the intensity of the halo being highest near the edge of the cube and decreasing in magnitude with increasing distance from the cube. (Focus Indicator Threshold)

[0136] As described herein, focus indicator 1302 can advantageously provide the user with an accurate sense of the cursor's location in the environment when the cursor is obscured by an object. For example, as can be seen in Figure 14C, cursor 1414 is invisible to the user, but focus indicator 1302 surrounding object 1204 identifies to the user that the object is being selected (or otherwise interacted with) and further indicates that the cursor is behind object 1204.

[0137] The system can assign a focus indicator 1302 to an object based, at least in part, on a determination that the location of the cursor in the environment has exceeded a distance threshold for the object. In other words, the focus indicator 1302 can be assigned to an object based on a determined spatial relationship between the location of the cursor in the environment and the object's location, size, shape, orientation, etc. The cursor location can be determined via ray casting or cone casting, and the distance to the object can be determined as the perpendicular distance between the ray (or cone) and the object.

[0138] The focus indicator 1302 can give the user an accurate sense of the cursor's location within the environment. For example, as the user changes the cursor's location within the environment, the system can assign, unassign, or modify focus indicators associated with one or more objects within the environment. The system can adjust the intensity, size, shape, color, or other characteristics of the focus indicator 1302 to indicate the relative distance between the cursor and the object. For example, as the cursor's location within the environment moves closer to an object, the focus indicator assigned to that object may be shown stronger or larger (at least in the direction toward the cursor). As the cursor's location within the environment moves away from an object, the focus indicator assigned to the object may be less strong or smaller, or the system may stop rendering the focus indicator.

[0139] 14A illustrates examples of various distance threshold considerations of the system when determining how (or whether) to render a focus indicator. The system may monitor the location 1412 of the cursor 1202, and based at least in part on that location, the system may determine whether to assign a focus indicator 1302 and the characteristics of the focus indicator. For example, the system may assign or modify the focus indicator 1302 to the object 1204 if the location 1412 of the cursor 1202 is below a distance threshold corresponding to the object 1204. The system may not assign or modify the focus indicator 1302 based at least in part on a determination that the location 1412 of the cursor 1202 is above a distance threshold corresponding to the object 1204.

[0140] The distance threshold can vary across embodiments and can be based on various factors, including, but not limited to, the size of the object, the number or density of objects in the environment, the proximity of the object to another object, etc. For example, in a busy environment, the distance threshold at which a focus indicator is activated (or deactivated) may be smaller than in a less crowded environment to avoid visual clutter caused by overlapping focus indicators or having a focus indicator on nearly every object near the cursor. In various embodiments, the distance threshold can be a percentage of the object's size or a percentage of the average distance between objects in the user's field of view (e.g., 10%, 20%, 50%, 100%, etc.). The distance threshold can be dynamic. For example, if an object (associated with a first distance threshold) moves into a busier area of ​​the environment, the object's distance threshold may decrease due to the presence of a closer object. Conversely, if the object moves into a less crowded environment, the object's distance threshold may increase.

[0141] 14A , in some cases, the distance threshold may correspond to the distance between a location of the cursor in the environment (e.g., location 1412) and a portion of the object 1204. For example, the distance threshold may correspond to the distance 1424 between the location of the cursor 1412 and the center of the object 1420. Similarly, the distance threshold may correspond to the distance 1426 between the location of the cursor 1412 and the nearest portion 1422 of the object 1204 (e.g., an edge or boundary of the object).

[0142] 14A illustrates a cursor 1202. However, because the distance between the cursor 1202 and the object 1204 is not below the distance threshold, a focus indicator is not being rendered for the object. For example, the system has determined that the cursor location 1412 is not close enough to the center 1420 of the object 1204, or that the cursor location 1412 is not close enough to the nearest portion 1422 of the object 1204. Thus, the system is rendering the cursor 1202 but not the focus indicator 1302.

[0143] In contrast, as another non-limiting example, Figure 14C illustrates an example situation where the cursor location is below the distance threshold (and, in this illustration, behind the object). Cursor 1414 is no longer rendered, and focus indicator 1302 is rendered. If the user were to move the cursor such that its distance to the object again exceeds the distance threshold, the user's view may return to the illustration in Figure 14A, with cursor 1202 displayed but the focus indicator not displayed. (Attraction effect between cursor and object)

[0144] In some cases, an object may act as if it has an adhesive, gravitational, or magnetizing effect on the cursor, such that the cursor appears to "snap" to the object (e.g., when the cursor is sufficiently close to the object). For example, the system may determine the location of the cursor within the user's field of view, and similarly, the location of one or more objects within the user's ocular field of view. Based on the spatial relationship between the cursor location and the one or more objects, the system may determine the object or objects to which to assign a focus indicator (e.g., the focus indicator may be displayed (or displayed more prominently) on an object closer to the cursor). The system may persistently assign at least one focus indicator to at least one object within the user's ocular field of view. For example, the system may assign a focus indicator to an object determined to be closest to the cursor location. As the cursor location changes, the object to which a focus indicator is assigned may also change.

[0145] To assist a user in moving a cursor over a desired object (e.g., to select that object for further interaction), the system may simulate the effect of an attractive force between the object and the cursor. The attractive force may mimic gravity, magnetism, spring-like, or other attractive forces between objects. For example, the attractive force may decrease as the distance between the object and the cursor increases. Thus, as a user moves the cursor closer to the desired object, the attractive force may increase, tending to draw the cursor toward the desired object. The attractive effect may make it easier for a user to select an object, since the user need only move the cursor close enough to the desired object and the system will attract or snap the cursor onto the desired object.

[0146] If the system (or user) mistakenly moves the cursor over an object (or the user changes their mind), the user can remove the cursor from the object by applying sufficient user input to move the cursor away from the object.

[0147] The amount of attractive force or range of attractive force can vary for different objects or types of objects. For example, objects that a user may want to interact with (e.g., controls for a virtual display, objects or characters in a virtual game) may be more attractive than objects that play a more passive role in the user's environment (e.g., a desk, a graphic on a wall). The attractive force can have a strength that can be modeled as an inverse function of the distance between the cursor and the object (e.g., an inverse square law similar to gravity or an inverse cube law similar to a magnetic dipole). The strength or range can be user-selectable, as some users may prefer a very strong attractive effect that causes the system to more aggressively pull the cursor onto the object, while other users may prefer little (or no) attractive effect.

[0148] Thus, embodiments of the wearable system can simulate the effect of attraction between the cursor and an object, as the cursor will be attracted to (and "snap") to the nearest object. This "attraction" provides persistent input feedback and can, in some cases, give the user an accurate sense of position without requiring the system to display the cursor (because a focus indicator tells the user where the cursor is attracted). This can be particularly advantageous when the oculomotor field includes many objects (e.g., objects in a dense grid layout) or when the objects are relatively close to each other.

[0149] As an example of an attractive force, when a user releases the touchpad of a user input device, the cursor may slide (e.g., as if attracted by gravity) to the nearest object (e.g., a button, icon, etc.) or to a position within it. In various embodiments, this sliding of the cursor may occur always, never, or if the nearest object is within a certain distance tolerance to the cursor. The system may provide settings including whether the cursor will move to the nearest position on the nearest object or to a position aligned with either / both of the object's X and Y axes (e.g., if there is a long row of adjacent objects, it may be desirable to snap to the center Y or center X of a vertical stack of objects). Settings may also include whether the use of attractive forces is desired within the entire user environment, or whether attractive forces are applied within a display panel that includes a list or grid of selectable buttons, icons, etc.

[0150] In some cases, a cursor that is "attached" to an object may not immediately become "detached" from the object unless the user moves the input device sufficiently to indicate to the system that they want to detach the cursor from the previously selected object. This also mimics the effects of adhesion, gravity, or magnetic forces between the cursor and the object, so that the system acts as if it is holding the selected object over the cursor until the user moves the cursor sufficiently away from the object.

[0151] Additionally, to aid user accuracy, when targeting an obscured object, the system can implement an attraction effect that will tend to attract the cursor toward the nearest object after active user input ceases. Thus, the cursor may act as if it has inertia and continue to move toward the object even if the user stops actuating the totem or other input device. The attraction of the attraction force moves the cursor onto the desired object in a natural way, with relatively minimal user action. This can advantageously make it easier for the user to select an object and reduce or minimize user fatigue.

[0152] As an example of cursor inertia, when a user is providing touchpad input and cursor movement is being determined or rendered, the system can also associate a cursor movement that can mimic a degree of inertia. For example, this movement can be applied from the moment active user input ceases on the touchpad (e.g., the user lifts their finger and stops movement). This can cause the cursor to continue along its motion path until damping forces reduce the inertia back to zero. Controls can limit the amount of inertia that can accumulate and allow an inertia increase (e.g., corresponding to a configurable threshold) to be applied when the user releases the touchpad at the end of a fast swipe action. The inertia increase can support fast swiping through long itemized lists (e.g., allowing one large swipe to carry the cursor from top to bottom when the user makes a selection).

[0153] The cursor can have a magnetizing effect on the focus indicator associated with an object, such that the proximity of the cursor affects the intensity or positioning of the focus indicator. For example, in some cases, each object may have a focus indicator (e.g., an outer glow), and the intensity, size, and location of the focus indicator may vary based on the location of the cursor. For example, as the cursor moves closer to an object, the focus indicator for that object may become brighter, stronger, or move in the direction of the cursor (e.g., be attracted toward it). When the cursor selects an object, the system moves the cursor behind the object while simultaneously increasing the intensity of the focus indicator. For example, when an object is selected, the focus indicator may give the appearance of a halo or halo around the object. (Focus indicator fluctuation intensity)

[0154] In some cases, the system can assign focus indicators to two or more objects based, for example, on the proximity of the objects to the cursor's location in the environment. Each object can have one or more corresponding distance thresholds (e.g., a close distance threshold, a medium distance threshold, etc.) or a dynamic distance threshold based at least in part on environmental factors (e.g., the density of objects in the user's FOV). If the cursor's location in the environment exceeds a distance threshold, the system can render a focus indicator for the corresponding object. To provide the user with additional positional feedback regarding where the cursor is located in the environment, the focus indicators assigned to various objects can have different attributes (e.g., intensity, color, size, etc.). For example, focus indicators for objects near the cursor can be rendered brighter than focus indicators for objects farther away from the cursor. Focus indicators on the side of objects closer to the cursor can be more emphasized than focus indicators on the side of objects farther away from the cursor. Thus, when the cursor is positioned between an upper object and a lower object, the focus indicators for the bottom portion of the upper object and the top portion of the lower object may be rendered more prominently than the focus indicators for the more distant top portion of the upper object and the bottom portion of the lower object (if focus indicators are used at all for these portions). These focus indicators thus provide the user with a strong visual cue that the cursor is located between the upper object and the lower object. By visually sweeping the FOV, the user can easily identify where the cursor is located by observing the pattern of focus indicators associated with objects in the FOV.

[0155] Thus, the intensity or glow (or size or shape) of the focus indicator can fade in or out depending on the spatial relationship (e.g., distance) between the location of the cursor in the environment and the location, size, or shape of nearby objects.

[0156] 15A and 15B illustrate an example implementation of multiple focus indicators with varying intensities based on the object's proximity to the location of a cursor 1516 in the field of view. As shown, a user's field of view 1520 includes multiple objects: a cup 1502, a wall clock 1504, a phone 1506, a basketball 1508, and a camera 1510. Additionally, FIGS. 15A and 15B show two distance thresholds for the wall clock 1504 (thresholds 1526, 1528) and the basketball (thresholds 1522, 1524). The system can provide different functionality when different distance thresholds are exceeded. For example, when the cursor comes within a large distance threshold (1524, 1528), a focus indicator for the associated object may be displayed to the user. When the cursor comes within a smaller distance threshold (1522, 1526), ​​the system may implement different functionality, such as further enhancing the appearance of the focus indicator or activating an attraction effect (described below) in which the cursor is drawn toward and obscured by the associated object. Distance thresholds 1522, 1524, 1526, 1528 are shown in Figures 15A and 15B for illustrative purposes only and need not be rendered by the system. Additionally, although Figures 15A and 15B illustrate only two thresholds for the objects, any number of thresholds are contemplated (e.g., 1, 2, 3, 4, or more).

[0157] As described herein, the wearable system 400 can track, monitor, or otherwise determine the location of the cursor 1516 within the field of view 1520. Here, the system has determined that the location of the cursor within the environment is between the wall clock 1504 and the basketball 1508 (and is closer to the wall clock 1504 than the basketball 1508). Further, the system has determined that the location of the cursor within the environment exceeds a smaller distance threshold 1526 corresponding to the wall clock 1504, and that the location of the cursor within the environment exceeds a larger distance threshold 1524 corresponding to the basketball 1508 (but does not exceed the basketball's smaller threshold 1522). Thus, because the location of the cursor within the environment exceeds the thresholds of both the wall clock 1504 and the basketball 1508, the system renders a focus indicator on each of the wall clock 1504 and the basketball 1508. However, to provide the user with an understanding that the location of the cursor in the environment is closer to the wall clock 1504 than to the basketball 1508, the system may render the focus indicator 1512 assigned to the wall clock 1504 differently than the system renders the focus indicator 1514 assigned to the basketball 1508. For example, the system may assign a larger or brighter focus indicator 1512 to the wall clock 1504 (or the portion of the wall clock closest to the cursor 1516) and a smaller or less intense focus indicator 1514 to the basketball 1508.

[0158] The cursor 1516 is farther from the cup 1502, the phone 1506, and the camera 1510 than their respective distance thresholds, and therefore the system does not render focus indicators around these objects in this example (or, in other examples, may render focus indicators that are less prominent than those for the wall clock and basketball).

[0159] In some cases, one or more distance thresholds are predetermined, while in other cases, one or more distance thresholds are dynamic and adjusted by the system in response to environmental factors. For example, the system may determine a relatively large distance threshold based, at least in part, on a determination that objects in the user's field of view 1520 are relatively far from one another. In contrast, the system may determine a relatively small distance threshold based, at least in part, on a determination that objects in the user's field of view 1520 are relatively close to one another or that there is a relatively large number of objects in the field of view 1520. This may advantageously allow the user to confidently select objects despite many objects being grouped or closely positioned together.

[0160] The intensity (or brightness) of the focus indicator can also include the presence of a glow within a particular area around or adjacent to the object. For example, as the cursor moves closer to an object, the focus indicator for that object can begin to fill (or be within) a larger area around the object. For example, with reference to FIG. 15A , when the cursor 1516 is within the distance threshold 1526, the focus indicator 1512 can surround the object 1504, and the focus indicator 1512 can be larger, brighter, or more intense than the focus indicators corresponding to the other objects 1504.

[0161] 16A-16D, 18A-18C, and 29A-29F, the focus indicator can be more visually perceptible on the side of the object closer or nearest to the cursor and less perceptible on the opposite side of the object, which provides the user with a clear perception that the cursor is closer to that particular side of the object (and farther from the opposite side of the object). For example, as illustrated in FIG. 15B, as cursor 1516 approaches object 1504, focus indicator 1512 can begin to move out from behind object 1504 and meet cursor 1516 (see also FIGS. 29A-29F). This gives the appearance that focus indicator 1512 is being attracted toward or pulled away from the object by cursor 1516. Object 1508 is farther from cursor 1516 than object 1504 (e.g., relative to thresholds 1522, 1524), and in Figure 15B, focus indicator 1514 around the basketball has not moved outward toward cursor 1516 as much as focus indicator 1512 around the wall clock. Thus, some or all of the focus indicator's relative brightness, position, spatial extent (e.g., circumferential extent around the object), color, graphical decoration, etc., can provide a user with a strong visual indication of where the cursor is located, such as objects near the cursor (and their proximity or distance), an object selected by the user, etc. (Implementation of occlusion cursor and focus indicator)

[0162] A focus indicator represents a way to highlight or emphasize a user selection within an AR, MR, or VR environment associated with a wearable system described herein. Rather than the traditional approach of showing a cursor moving over and at least partially occluding interactable content, the system can render the cursor as moving behind and obscured by a real or virtual object. The use of a focus indicator provides positional feedback to the user, for example, via the relative appearance of a halo or halo shining out from behind an object in the environment. Furthermore, by continuing to track or determine user input (e.g., head pose, eye pose, body pose, input from a user input device, etc.) even after assigning a focus indicator to an object, the system can modify the focus indicator of the environmental object, thereby providing the user with persistent input feedback, an immersive user experience, and an accurate sense of cursor position.

[0163] 16A-16D illustrate an example process for rendering a focus indicator. Implementation of the focus indicator and occlusion cursor can be performed in various ways. For example, implementation of the focus indicator can include utilizing a graphics processor (GPU) or a computing device with at least moderate CPU power. In some cases, the GPU is configured to (i) launch a fragment shader program, (ii) render an off-screen rendering buffer, or (iii) perform several rounds of full-screen processing work.

[0164] To determine the proximity of objects within the user's FOV, the system can determine the location of each of the objects relative to the cursor's location in the environment. For example, many of the objects in the environment can be represented by 2D shapes placed on a 3D world selection plane. The system can cast a ray to the 3D world selection plane to determine the proximity of the cursor's location in the environment to any given object. The system can also determine one or more characteristics of the object, such as the object's shape or orientation. In some cases, based at least in part on the object's shape, silhouette, orientation, or proximity to the cursor's location in the environment, the system can determine a spatial relationship between the cursor's location in the environment and a portion of the object. For example, the system can determine when the cursor's location in the environment overlaps with the object or exceeds a threshold distance to a portion of the object (e.g., the nearest portion of the object, the center of the object, etc.). In some cases, for example, when the environment includes multiple objects, the system can determine the object that is closest to the cursor's location in the environment. In some cases, the displayed characteristics of the focus indicator can be based at least in part on the object's proximity to the cursor's location in the environment.

[0165] 16A-16D diagrammatically illustrate an example process by which a wearable system may render an occlusion cursor and focus indicator. This process may be performed by local processing and data module 260 of wearable display system 200 described with reference to FIG. 2. The example rendering process described with reference to FIGS. 16A-16D may be performed twice to represent each eye in a stereoscopically rendered AR / MR / VR environment.

[0166] FIG. 16A illustrates a first off-screen rendering step for rendering a focus indicator. Based at least in part on the determined cursor location within the environment, the system renders a cursor glow 1610 to an off-screen buffer 1600A (sometimes referred to as a "cursor source buffer"). The cursor glow 1610 can be positioned within the environment as if it were a traditional cursor. For example, the center 1630 of the cursor glow 1610 may correspond to the location of the cursor within the environment. The cursor glow 1610 will ultimately serve as a mask defining the maximum display space area within which the focus indicator will be visible. The size or shape of the cursor glow 1610 can be based on various criteria, such as the desired focus indicator size, the proximity of the object to other objects, the size or shape of the objects, the number or density of objects, etc.

[0167] FIG. 16B illustrates a second off-screen rendering process for rendering a focus indicator. The second off-screen rendering process may be rendered to another off-screen buffer 1600B (sometimes referred to as a “shape mask buffer”) and may include a mask representation of one or more of the objects 1204. The mask representation may be based, at least in part, on the location, orientation, or size of the object in 3D space. For example, as illustrated in FIG. 16B, the system may create only masks for objects that are within the outer boundary or silhouette 1620 of the cursor glow 1610 from FIG. 16A. In some cases, the system may create only masks for objects that will be assigned a focus indicator. Thus, in this example, the system renders masks 1602, 1604, and 1606, but does not render a mask for object 1608 that is beyond the outer boundary 1620 of the cursor glow.

[0168] The system can determine the shape representation (e.g., mask) of the object 1204 in various ways. For example, the system can render a mask that reflects the 3D camera transformation of those shapes. In some cases, the object 1204 can be represented by a 2D sphere, rectangle, or capsule-shaped shape that corresponds to the object's actual shape. Similarly, an object (such as a real-world object) can be represented by a silhouette of the object. In some cases, to draw each shape mask, the system utilizes a shader program that algorithmically renders a given shape from a mathematical formula. In some cases, such as for 3D objects, the system can render a flat color projection of the object. In some cases, the system uses a camera-space approximation of the object's shape.

[0169] Figure 16C illustrates a third off-screen rendering process for rendering a focus indicator. The third off-screen rendering process may be rendered to another off-screen buffer 1600C (sometimes referred to as an "iris mask buffer") and may include masks of objects (e.g., masks 1602, 1604, 1606) from shape mask buffer 1600B, but may also include iris masks for one or more of the objects (e.g., iris masks 1622, 1624, 1626, described below). For example, similar to the masks of Figure 16B, the system may render iris masks for objects that are within the outer boundary or silhouette 1620 of cursor iris 1630 from Figure 16A.

[0170] The system may render a glow mask (e.g., a rim glow, halo, shadow, or other visual indicator) that at least partially illuminates the perimeter of the masks 1602, 1604, 1606. For example, similar to the masks of FIG. 16B, the system may utilize a shader program (or multi-tap blur of the object mask) that can paint the object shape to blur the borders of the object shape by an adjustable border thickness amount. The shader program may use the cursor location 1630 and vary the border thickness, glow brightness, color, or intensity of the glow mask to reflect its proximity to the cursor location. For example, a stronger or brighter glow mask may be associated with closer objects, while a less strong or darker glow mask may be associated with more distant objects. 16C, glow mask 1622 is brighter than glow masks 1624 or 1626 because object 1632 associated with glow mask 1622 is closer to cursor location 1630 than objects 1636, 1624. In this example, a glow mask is not generated for object 1608 because its distance from center 1630 of cursor glow 1610 exceeds the size of the cursor glow. Varying the intensity of the glow mask can advantageously provide precise position feedback even when the system is not rendering an on-screen visual aid (e.g., a cursor) to indicate the cursor's location.

[0171] To modify the edge thickness, glow brightness, color, intensity, or other properties of the glow mask, the shader program can consider the x and y display space distance from each rendered pixel to the cursor and expand or contract the boundary feather parameters (such as the edge thickness or glow intensity of the glow mask) accordingly.

[0172] 16D illustrates a fourth rendering step for rendering a focus indicator. The fourth rendering step may be rendered on-screen and may represent a later (or final) stage 1600D of the rendering work for the entire scene. For this rendering step, the system has access to a cursor source buffer 1600A, a shape mask buffer 1600B, and an iris mask 1600C buffer, and (optionally) one or more on-screen or off-screen buffers (sometimes referred to as "scene buffers") that contain the 3D scene content rendered by the rendering application.

[0173] The system can use various techniques to combine the cursor source buffer 1600A, the shape mask buffer 1600B, and the glow mask 1600C buffer, and the scene buffer. For example, a shader program can combine the various buffers together to generate the scene 1600D. For example, the shader program can subtract each non-zero shape mask 1600B pixel from the scene buffer color. In addition, the shader program can add to the scene buffer color the combination of the glow mask 1600C buffer minus the shape mask buffer 1600B, multiplied by the cursor source buffer.

[0174] As shown in scene 1600D of FIG. 16D , objects 1632, 1634, and 1636 are assigned focus indicators 1202a, 1202b, and 1202c, respectively. Focus indicators 1202a-1202c illuminate at least partially around each of the respective objects 1632, 1634, and 1636. In this example, the portion illuminated by focus indicator 1202 corresponds to the portion of the object that is within the outer boundary or silhouette 1620 of the cursor glow. Portions of the object outside boundary 1620 are not rendered with a focus indicator in this example. However, in some cases, if at least a portion of the object is within the silhouette 1620 of the cursor glow, the entire object is assigned a focus indicator, not just a portion of the object. As shown, because no portion of object 1638 is within the silhouette 1620 of cursor glow 1610 (with respect to the illustrated location 1630 of the cursor), object 1638 is not assigned a focus indicator.

[0175] Thus, a user viewing the rendered scene in FIG. 16D will be provided with a strong visual cue that the cursor is behind object 1632 due to the stronger focus indicator 1202a (compared to indicators 1202b and 1202c) and the fact that focus indicator 1202a extends almost entirely around object 1632, while focus indicators 1202b, 1202c extend only partially around their associated objects 1636, 1634.

[0176] In areas where there are sufficient objects or they are relatively densely occluded, the occlusion cursor and focus indicator can be effective in indicating the cursor location. However, in areas where there are few or no objects, the system can render a graphical element (e.g., a small glow sprite) to indicate the cursor position to the user.

[0177] The system may provide adjustable parameters for each selectable object that allow for controlling the intensity with which the edges of the object may glow when selected or interacted with, or for increasing or decreasing the extent to which the object may glow as the cursor approaches it. In some cases, when rendering shape or glow masks, the system may use mathematical shape representations to incorporate anti-aliasing into the source rendering process. (Real World Objects)

[0178] 16A-16D shows a focus indicator associated with virtual content, similar techniques are also applicable in assigning focus indicator highlights to real-world objects in an augmented or mixed reality environment. For example, the system could use a camera-space approximation of the object's shape in conjunction with a multi-tap blur used to generate the focus indicator. (Occlusion cursor in Flat Layout)

[0179] For environments with many selectable objects, e.g., organized grids or lists, the system may display a cursor that behaves more like a focus indicator. Figure 17 shows an example of user input on a grid 1700 and a totem 1702 (with a touch-sensitive surface 1704). A user's touch on a trajectory 1706 on the touch-sensitive surface 1704 moves the cursor along substantially the same trajectory 1710 across the grid layout 1700 of selectable objects. Objects may have attraction effects as described herein, so the cursor need not be displayed between objects, as it is attracted to the nearest object. For example, one of the objects in the grid 1700 may have focus at all times (e.g., highlighted by a focus indicator). The visual focus indicator and (optionally) a tactile event on the totem 1702 may involve hovering over (or selecting) an object.

[0180] For flexible navigation of more complex layouts that include areas with fine-grained selection, such as a browser or document with a lot of selectable text, the cursor can be visible at all times because the layout is not full of occluding objects that would occlude the cursor. Visual focus indicators and optional tactile events can still accompany hovering over selectable objects. (Example of adding an occlusion cursor and focus indicator)

[0181] 18A-18C illustrate an example of moving cursor 1202 toward object 1204 having focus indicator 1302. Object 1204 may be a selectable object, such as an icon, that may be selected to initiate the execution of an application. In this example, for illustrative purposes, the icon is shown as an Earth and Moon within a star-spangled background, and the executable application is labeled Space Explorer. The dashed lines around icon 1202, focus indicator 1302, and object 1204 indicate that each of these graphical elements may be rendered by the display system in different buffers, which may be combined as described with reference to FIGS. 16A-16D.

[0182] 18A illustrates an example when the cursor is remote from the object 1204. In this case, the cursor 1202 is relatively bright to aid visibility to the user, and the focus indicator 1302 is relatively small and less intense. The system may render the focus indicator 1302 at approximately the same size as the object 1202 (as indicated by the dashed line in FIG. 18A ), render the object 1202 on top of the focus indicator 1302, or render the object 1202 behind the focus indicator 1302. Thus, when the cursor 1202 is remote from the object 1204, the focus indicator 1302 is said to remain “hidden” behind the object 1204 and may be visually imperceptible or nearly imperceptible to the user.

[0183] FIG. 18B illustrates an example of the interaction between the cursor, object, and focus indicator as the cursor approaches the object. As shown, the focus indicator 1302 in FIG. 13B is larger, brighter, or more intense than the focus indicator in FIG. 18A . Furthermore, as the cursor 1202 approaches the object 1204, the focus indicator 1302 begins to move out from behind the object and meets the cursor 1202. This gives the appearance of the focus indicator being drawn toward the cursor 1202. Additionally, FIGS. 18A and 18B illustrate how the intensity of the focus indicator may fade in or out depending on the object's proximity to the cursor and its position relative to the cursor. For example, in FIG. 18B , the focus indicator 1302 is more visually perceptible on the side of the object 1204 closest to the cursor 1202 and less perceptible on the opposite side of the object, which provides the user with a clear perception that the cursor 1202 is in proximity to that particular side of the object 1204.

[0184] 18C illustrates an example of the interaction between the cursor, object, and focus indicator as the cursor moves (or hovers) behind the object. As cursor 1202 moves behind object 1204, object 1204 becomes larger (in this example) and the foreground of the object expands so that object 1204 appears closer or larger to the user. Additionally, focus indicator 1302 may become brighter and substantially surround the object, indicating that object 1204 is selected. In this example, the icon (represented in 2D in FIGS. 18A and 18B) is expanded outward so that the Earth (and Moon) appear in front of a star-studded background (e.g., at a depth closer to the user than that of the star-studded background). (Illustrative Focus Indicator)

[0185] Figures 19-22 illustrate various examples of focus indicators that may be rendered by the system. The focus indicator 1302 can be circular (e.g., as shown in Figures 18, 19, and 21), rectangular (e.g., as shown in Figure 20), or other shapes. In some cases, the focus indicator can include a label positioned adjacent to (e.g., above, below, or to the side of) the object 1302. For example, Figure 19 shows the label "Search" to the side of the object 1302, and Figure 20 shows the label "Collect Pages" above the object 1302. The label can be highlighted (e.g., made brighter) when the object is selected to provide the user with a visual cue about the object. Although these examples show text labels, the label can be any graphical element. Figure 22 shows an example in which a user has selected an application selection icon that allows the user to select between applications such as a browser, a social network, etc. In Figure 22, the user has selected the browser application, and the system renders the focus indicator 1302 as a halo around the browser icon. (Example process of implementing an occlusion cursor)

[0186] 23 illustrates a flowchart for an exemplary method for rendering a focus indicator within a 3D scene. Process 2300 may be performed, alone or in combination, by one or more components of wearable system 200, such as remote processing module 270, local processing and data module 260, a graphics processor (GPU), or another processor. Display 220 of wearable system 200 can present the scene to a user, and the system can obtain user input data, such as eye pose data from an inward-facing imaging system 462, or head pose data from an IMU, accelerometer, or gyroscope, or user input data for moving a cursor or selecting an object from a user input device 466, such as a handheld totem 1702.

[0187] In block 2302, the wearable system can determine the location of a cursor within the user's environment. The system can obtain user input data, such as eye pose data from inward-facing imaging system 462, head pose data from an IMU, accelerometer, or gyroscope, or data from a user input device, such as user input device 466 of FIG. 4 or totem 1702 of FIG. 17. Based at least in part on the user input data, the system can determine the location of the cursor within the environment. In some cases, in addition to determining the location of the cursor within the environment, the system can also render cursor 1202 or other on-screen visual aids corresponding to the location of the cursor within the environment.

[0188] In block 2304, the system may determine a spatial relationship between the location of the cursor in the environment and one or more objects in the user's field of view (or oculomotor field). In some cases, the system may determine one or more characteristics of the object, such as the location, shape, orientation, or size of the one or more objects. Based at least in part on the one or more object characteristics and the location of the cursor in the environment determined in block 2302, the system may determine a spatial relationship between the location of the cursor in the environment and any portion of the object. The spatial relationship may include relative location information, such as the distance of the portion of the object from the location of the cursor in the environment, or the relative orientation between the cursor and the portion of the object (e.g., whether the cursor is above, below, left, or right of the object). The system may determine whether the location of the cursor in the environment overlaps with or is behind the object, or may determine the distance between the location of the cursor in the environment and a portion of the object (e.g., the nearest portion of the object, the center of the object, etc.). In some cases, the system may determine the object closest to the location of the cursor in the environment.

[0189] In some implementations, virtual objects in the environment can be represented by 2D shapes placed on a 3D world selection plane. The system can cast rays to the 3D world selection plane to determine the proximity of the cursor's location in the environment to any given object. Spatial relationships can include the distance between the cursor and the object (or part of the object) and the relative orientation of the cursor and the object.

[0190] In block 2306, the system may assign a focus indicator to at least a portion of one or more objects based at least in part on the determined spatial relationship. For example, the system may render the focus indicator using techniques described with reference to Figures 16A-16D. As described with reference to Figures 12B and 18A-18C, if the determined spatial relationship provides for the cursor to overlap or be behind an object, the system may also render the object in front of the cursor so that the cursor does not occlude the object.

[0191] Process 2300 is illustrative and not intended to be limiting. The various blocks described herein can be implemented in various orders, and the wearable system can implement one or more of the blocks in parallel or change the order as desired. Fewer, more, or different blocks can be used as part of process 2300. For example, process 2300 can include a block for displaying a cursor or a block for performing other user interface actions. (Example of part of a display with icons or a graphical user interface)

[0192] Figures 24-28 are front views of examples of portions of a display screen with icons. In these examples, the icons comprise stylized representations of a head primarily within a circle. Figures 24-28 show examples of focus indicators that at least partially surround the icon. The focus indicators are represented as short lines that appear to radiate outward from the icon. In these examples of icons, the focus indicators are generally substantially surrounding (with a greater extent below the icon in Figure 25), below (Figure 24), to the right (Figure 26), to the left (Figure 27), and above and below (Figure 28).

[0193] 29A-29F are front views of an embodiment of a graphical user interface for a display screen or portion thereof. The appearance of the graphical user interface sequentially transitions between the images shown in FIGS. 29A-29F. No decorative aspects are associated with the process or period in which one image transitions to another. In these illustrative figures, the virtual object (e.g., an icon) is represented by a dashed circle; in other embodiments, it may be a rectangle, polygon, or other shape. In FIG. 29A, the focus indicator is shown in grayscale as a circular ring surrounding the icon. The cursor is shown in dashed lines. In the transition image sequence continuing in FIGS. 29B-29F, as the cursor moves away from the icon, the focus indicator is drawn outward, away from the icon and toward the cursor until it separates from the icon in FIG. 29E, then transitions to a circular shape in FIG. 29F. In FIG. 29F, the focus indicator is represented as a grayscale circle, and the cursor is not rendered by the display.

[0194] The designs shown in Figures 24-29F can be embodied in an augmented reality or mixed reality display, such as a head-mounted display. For example, the display can constitute the display 220 of the wearable system 200 described with reference to Figure 2, or the display of the wearable system 400 described with reference to Figure 4, or the display of the optical display system 600 described with reference to Figure 6. The display, or portions thereof, are represented in Figures 24-29F by outer dashed rectangular lines. Neither the display nor the icons (or other graphical elements of the animated graphical user interface) are limited to the scale shown in Figures 24-29F. The dashed lines representing the display do not form part of the design.

[0195] Thus, in various aspects, the present disclosure provides decorative designs for a display screen or portion thereof with icons or transitioning (or animated) graphical user interfaces, as shown and described. (Example of part of a display with icons or a graphical user interface)

[0196] 30A-30F illustrate an embodiment of a transition sequence for a GUI on a display screen or portion thereof. The GUI can be rendered by any of the wearable displays described herein, such as, for example, wearable display systems 200, 400, 600 described with reference to FIGS. 2, 4, and 6. FIGS. 30A-30F show a GUI with a cursor 1202 transitioning continuously from point A to point F along an illustrative path 3001 shown by a dashed line. As shown, the GUI includes multiple icons 3002 presented in a grid layout. The GUI uses cursor 1202 or focus indicator 1302 as described herein to indicate the interaction between the cursor, icons, or focus indicator as the cursor moves behind (or hovers over) an icon. The grid layout, icon shapes (e.g., rectangles in these figures), and cursor path are illustrative and not intended to be limiting. The icons in the grid layout can be rendered at a single depth (e.g., to appear 2D) or multiple depths (e.g., to appear 3D). The icons 3002 can be thumbnails. The grid layout need not be flat, but can also be rendered as a curved surface (e.g., some parts of the layout are at closer depths than others).

[0197] 30A illustrates an example of an interaction between an icon 3002 and the cursor 1202 when the cursor 1202 is positioned at point A. The icon 3002 may correspond to any type of interactable object within the user's virtual environment of the wearable system. The interactable objects include, but are not limited to, applications (e.g., apps), content folders, and digital media such as, but not limited to, still images, videos, audio, music, albums, documents, etc. As shown, at point A, the cursor 1202 is between icons 3010, 3012 (B3 and D3) in the grid layout. In other words, at point A, the cursor 1202 is not selecting any of the icons 3002.

[0198] FIG. 30B illustrates an example of the interaction between icon 3010 (B3) and focus indicator 1302 when cursor 1202 moves (or hovers) behind icon 3010 at point B. As shown in this example, when cursor 1202 moves behind icon 3010, the GUI assigns focus indicator 1302 to surround icon 3010, indicating that icon 3010 is being hovered under by the cursor. Focus indicator 1302 is shown in grayscale in this example as a curved shape that substantially surrounds the icon. Cursor 1202 is occluded by icon 3010 in FIG. 30B. Cursor 1202 is shown with a dashed line, which simply indicates the location where cursor 1202 would be if rendered.

[0199] In some cases, the visual appearance of the selected icon 3010 may change to indicate that the icon has been selected by the user. For example, the user may select the icon 3010 by hovering the cursor 1202 under the icon 3010 for a period of time (e.g., several seconds or more), by user input from a totem (e.g., activating a touch-sensitive surface, such as a click or double-click), by eye, head, or body gesture, etc. For example, the wearable system may detect user selection of an icon based at least in part on eye gaze, e.g., an eye-tracking camera detecting that the user has fixated the icon 3010 for longer than a threshold time (e.g., one second or more).

[0200] The visual appearance of other icons in the layout (e.g., icons that do not obscure the cursor) may change to indicate that icon 3010 is hovering under it or that it has been selected. Icon 3010 or other icons may change size or shape when cursor 1202 moves behind icon 3010. For example, icon 3010 may become larger, or the foreground of the icon may expand, so that icon 3010 appears closer or larger to the user (e.g., at a depth closer to the user than that of the background). Similarly, unselected icons may become smaller, or the foreground of unselected icons may shrink, so that the icons appear farther or smaller to the user. Additional or alternative changes in the size, shape, or visual appearance of icons may also be used. For example, when icon 3010 is hovered under or selected, selected icon 3010 may become smaller, or other icons 3002 may become larger.

[0201] The selected icon 3010 or other icons may change clarity (including transparency), resolution, etc. as the cursor 1202 moves behind the icon 3010. For example, when no icons are selected (e.g., as shown in FIG. 30A ), each of the icons 3002 may be presented with a first clarity or a first resolution. When the cursor 1202 moves behind the icon 3010, the clarity or resolution of the icon hovered under or selected may change, for example, to a second clarity or a second resolution. In some cases, the second clarity is clearer than the first clarity, and in some cases, the second resolution is a higher resolution than the first resolution. Thus, when the icon 3010 is selected, the icon 3010 may be in better focus, higher resolution, or higher quality than the previously selected icon.

[0202] Additionally or alternatively, when cursor 1202 moves behind icon 3010, the clarity or resolution of other icons (e.g., non-selected icons) may change, for example, to a third clarity or third resolution. In some cases, the third clarity may be less clear than the first clarity, or the third resolution may be a lower resolution than the first resolution. Thus, when icon 3010 is selected, the other icons may appear blurred, out of focus, and of lower resolution or quality.

[0203] However, in some cases, the clarity or resolution of a selected icon may decrease when it is selected. Similarly, the clarity or resolution of a non-selected icon may increase when it is selected. Additional or alternative changes in clarity, resolution, etc. may also be implemented.

[0204] In some cases, when the cursor 1202 moves behind an icon 3010, additional details may be shown for the selected icon 3010. For example, the additional details may include a caption 3014, which may include a title for the app or media. Similarly, the additional details may include size (e.g., in bytes), created date, modified date, location, file type, resolution, video details (e.g., video length, producer, actors, etc.), or other characteristics corresponding to the selected icon 3010.

[0205] In some cases, one or more features of the selected icon 3010 may become active when the cursor 1202 moves behind the icon 3010. For example, if the icon 3010 corresponds to a video, selecting the icon 3010 may cause the video to begin playing. Similarly, selecting the icon 3010 may cause the GUI to cycle through images, play an album, play a GIF, etc.

[0206] FIG. 30C illustrates an example of the interaction between the icon 3010 and the focus indicator 1302 as the cursor 1202 transitions behind the icon 3010 to point C near the center of the icon 3010. In the illustrated embodiment, as the cursor 1202 moves toward the center of the icon 3010, the icon 3010 continues to grow larger (e.g., compared to FIG. 30B ) so that the icon 3010 appears even closer or larger (e.g., at a closer depth) to the user. For example, the icon 3010 may expand in size such that at least a portion of the icon 3010 overlaps one or more other icons. In examples such as these, the overlapped icons may become more transparent or blurred, or they may be partially covered by the selected icon 3010.

[0207] Additionally, or alternatively, as the cursor 1202 transitions to a more central location behind the icon 3010, the intensity of the focus indicator 1302 can change. For example, the focus indicator 1302 can become brighter or larger. Furthermore, the clarity, resolution, etc. of the selected icon 3010 or other icons can continue to increase or decrease. By continuing to track the cursor (even after assigning a focus indicator) and modifying the intensity of the focus indicator or the characteristics of the icon, the system can provide the user with persistent input feedback and an accurate sense of cursor position.

[0208] 30D illustrates an example of the interaction between icons 3010 and 3012 and cursor 1202 as cursor 1202 moves along the path from point C (below icon 3010) to point D (between icons 3010 and 3012). As shown, once icon 3010 is no longer selected (or hovered under), the icon can return to its original size, shape, resolution, focus, clarity, etc., as illustrated in FIG.

[0209] 30E illustrates an example of the interaction between the icon 3012 and the focus indicator 1302 when the cursor 1202 moves (or hovers) behind the icon 3012 to point E. As described herein with respect to FIG. 30B, when the cursor 1202 moves behind the icon 3012, the GUI may assign a focus indicator 1302 that surrounds the icon 3012 to indicate that the icon 3010 is selected. The focus indicator may include a caption 3016.

[0210] 30F illustrates an example of the interaction between icon 3010 and focus indicator 1302 as cursor 1202 transitions to point F behind icon 3012. As described herein with respect to FIG. 30C, as cursor 1202 moves toward the center of icon 3012, icon 3012 continues to become larger (e.g., compared to FIG. 30E) so that icon 3012 appears even closer or larger to the user. For example, icon 3010 may expand in size such that at least a portion of icon 3012 overlaps one or more other icons.

[0211] Similarly, as described with reference to Figures 30A-30F, the GUI can continue to dynamically update the icon, cursor, or focus indicator as the cursor continues to move along an extension of path 3001 or along different paths between icons in the grid layout. (Example of scrolling data in a graphical user interface)

[0212] 31A-31C illustrate an embodiment of a scrolling sequence of a GUI on a display screen or portion thereof. The GUI can be rendered by any of the wearable displays described herein, such as, for example, wearable display systems 200, 400, 600 described with reference to FIGS. 2, 4, and 6. Scrolling text, graphics, or other content in the GUI advantageously allows a user to move large distances and navigate the content. In the illustrated example, the GUI includes multiple icons 3102 arranged in a grid layout, further illustrating scrolling of the icons 3102 in the grid. The grid layout and icon shapes (e.g., generally rectangular or triangular in these figures) are illustrative and not intended to be limiting. The icons in the grid layout can be rendered at a single depth or multiple depths. The scrolling sequence depicted in FIGS. 31A-31C and the occlusion cursor feature depicted in FIGS. 30A-30F can be used separately or together. For example, once scrolling stops, the user can move the cursor to one of the icons in the grid layout, and the GUI can illustrate this cursor movement and hovering over or selecting the icon, as described with reference to Figures 30A-30F.

[0213] FIG. 31A illustrates multiple arranged icons 3012 in a GUI. As described with respect to FIGS. 30A-30F, the icons 3102 may correspond to virtual content such as apps or digital media. Although the icons 3102 are arranged in a grid, the icons 3012 can be presented on the GUI in a variety of ways. For example, the positioning of the icons 3012 can be selected or controlled by the user, or the icons 3012 can be automatically arranged according to one or more grouping criteria (e.g., alphabetically by item name, by content type, by date, by frequency of use, etc.). The icons in a grid layout can be rendered at a single depth (e.g., to appear 2D) or multiple depths (e.g., to appear 3D). The icons 3002 can be thumbnails. The grid layout need not be flat but can also be rendered as a curved surface (e.g., some parts of the layout are at closer depths than others).

[0214] FIG. 31B illustrates an example GUI after a scrolling sequence has been initiated by the user (e.g., by activating a totem, swiping across the layout, hovering the cursor near the edge of a grid or display, etc.). User initiation can provide a scrolling direction or a scrolling speed. The scrolling sequence can mimic a momentum of the scrolling content (from stationary) that increases the scrolling speed so that the scrolling content is blurred, unreadable, etc. during scrolling. The scrolling sequence can simulate a drag force so that the scrolling speed slows down and comes to a stop. The wearable system can receive additional user input to pause the scrolling (e.g., a further activation of the totem or a stop gesture with the user's hand).

[0215] During scrolling, the icon 3102 can move to a greater depth, change its size (e.g., become smaller), or appear less distinct (e.g., with more transparency) or with lower resolution. For example, FIG. 3B graphically depicts the icon 3102 as appearing less distinct (compared to FIG. 3A or 3C). As the icon 3102 scrolls, to aid the user in seeing the next icon, the GUI can display a content panel 3104 corresponding to the scrolling icon. In this example, the icon 3102 scrolls horizontally to the right (as indicated by dashed arrow 3200, which may be visible to the user, but need not be)), causing a new icon to appear from the left (and disappear from view to the right). Thus, the content panel 3104 is displayed in the general location where a new icon would appear (e.g., on the left side of the display in this example). In other examples, the icon 3102 can scroll in any direction (e.g., left to right, right to left, bottom to top, top to bottom, diagonally, etc.). Because the wearable system can display content at multiple depths, content can scroll from foreground (e.g., closer depth) to background (e.g., further away depth), or from background to foreground. Any combination of these scrolling techniques can be used.

[0216] The content panel 3104 may contain information about the scrolling content. For example, the icon 3102 may be part of a library, and the content panel 3104 may contain favorite icons, recently used icons, or most used icons in the library. In some cases, the library may be grouped or sorted by grouping criteria, such as by date created, date modified, name, icon type (e.g., image, video, GIF, album, app, document, etc.), size, etc. When content is scrolling, the content panel 3104 may correspond to a particular group or class that corresponds to the content scrolling behind the content panel 3104. As the icon 3102 continues to scroll, the content panel 3104 may be periodically updated with new information representing the passing content. FIG. 31B shows a moment when the content panel 3104 is equipped with icons B1-B4.

[0217] As a non-limiting example, icons 3102 may be sorted by date. As the content scrolls, content panel 3104 is updated with new information periodically representing the passing dates. For example, if scrolling icon 3102 includes dates for October, the content panel may include information about October. For example, message 3114 may include the abbreviation "OCT," and content panel 3104 may include favorite icons from October, recently used icons from October, most used icons from October, etc. As the content continues to scroll to the next month (e.g., November), content panel 3104 may be updated to include information representing November (e.g., the abbreviation may change to "NOV," and the panel may show favorite icons from November, recently used icons from November, most used icons from November, etc.). Content panel 3104 may continue to update as additional dates pass.

[0218] The content panel 3104 can be anchored in place on the GUI while the content scrolls off-screen (and the same content can return when the user scrolls back). In the illustrated embodiment, the content panel 3104 is anchored to the left side of the GUI. However, the content panel 3104 can be located anywhere in the GUI, such as the center, bottom, top, or right side. In some cases, the location of the content panel 3104 is configurable by the user.

[0219] The content panel 3104 can be presented in a variety of ways. For example, the size of the content bar can vary based, at least in part, on the scroll speed. A faster scroll speed can cause the content bar to appear at a first size, while a slower scroll speed can cause the content bar to appear at a second size (e.g., smaller than the first size). Furthermore, the shape of the content panel 3104 can also vary. In the illustrated embodiment, the content panel 3104 includes a vertical list. However, the list can be vertical, horizontal, diagonal, square, etc. Additionally, or alternatively, the content panel 3104 may not include a list and instead include a single object, icon, image, text, etc. The content panel 3104 can be displayed at a different depth or depths from the grid layout. For example, it can be displayed in front of the grid layout (e.g., as shown in FIG. 31B ), behind the layout, etc. In some cases, the characteristics of the content panel 3104 are configurable by the user.

[0220] The content panel 3104 may include details (e.g., message 3114) that may correspond to the content presented in the content panel. For example, the details may include a caption, title, or other characteristic that corresponds to the scrolling content. For example, referring back to the example where the icons were sorted by date, the message 3114 may include a date abbreviation (e.g., "OCT").

[0221] When the scrolling sequence ends, the icon 3102 can become stationary and the content panel 3104 can disappear. For example, FIG. 31C illustrates an example of the icon 3102 after the scrolling sequence has ended. In contrast to FIG. 31B, the icon 3102 is shown as being in focus and slightly shifted compared to FIG. 31A. Similar techniques can also be used for other types of scrolling, such as vertical or diagonal scrolling.

[0222] In some implementations, the GUI can utilize edge scrolling, in which scrolling begins when the user hovers the cursor near the edge of the grid (or display). The GUI can maintain user behavior history data so that the next time the user opens or accesses the grid layout, the GUI displays the cursor over the most recent icon added to the layout (e.g., the most recent music album or video added by the user) or the most recently accessed icon. (Example Software Code)

[0223] Appendix A includes example code in the C# programming language that may be used to implement embodiments of the occultation cursor technology described herein. An embodiment of process 2300 may be implemented, at least in part, by the example code in Appendix A. Appendix A also includes a description of the software code. The disclosure of Appendix A is intended to illustrate example implementations of various features of the occultation cursor technology and is not intended to limit the scope of the technology. Appendix A is incorporated herein by reference in its entirety as if it formed a part of this specification. (Additional Aspects)

[0224] In a first aspect, a wearable display system includes a display configured to be positioned in front of a user's eye, the display configured to project virtual content toward the user's eye; a user input device configured to receive user input data associated with movement of a virtual cursor; and a hardware processor in communication with the display and the user input device, the hardware processor being programmed to identify a location of the virtual cursor within the user's environment, determine a spatial relationship between the virtual cursor and an object within the user's environment, and direct the display to render a focus indicator associated with the object based at least in part on the spatial relationship.

[0225] In a second aspect, the wearable display system of aspect 1, wherein the user input device comprises one or more of a handheld totem having a touch-sensitive surface, an outward-facing imaging system configured to detect user gestures, an inward-facing imaging system configured to detect the user's eye posture, or an inertial measurement unit configured to detect the user's head posture.

[0226] In a third aspect, the wearable display system of aspect 1 or aspect 2 is programmed to determine the spatial relationship between the virtual cursor and the object, wherein the hardware processor is programmed to determine a distance between a location of the virtual cursor and a portion of the object or determine a relative orientation between the virtual cursor and the object, and instruct the display to render a focus indicator based at least in part on the determined distance or the determined orientation.

[0227] In a fourth aspect, the wearable display system of any one of aspects 1-3, wherein the focus indicator comprises one or more of a glow or halo that at least partially surrounds the object, a size or depth change of the object, or a graphical highlight.

[0228] In a fifth aspect, the wearable display system of any one of aspects 1-4 is programmed to: instruct the display to render a focus indicator; the hardware processor to perform a first rendering step to render a cursor glow in a first buffer, the location of the cursor glow being based at least in part on the location of the virtual cursor; a second rendering step to render a shape mask representation of the object in a second buffer; a third rendering step to render a glow mask associated with the object in a third buffer; and a fourth rendering step configured to combine at least the first buffer, the second buffer, and the third buffer for presentation to the user.

[0229] In a sixth aspect, the wearable display system of aspect 5 is configured such that, to perform the fourth rendering step, the hardware processor is programmed to combine at least the first buffer, the second buffer, and the third buffer with a fourth buffer having virtual scene content.

[0230] In a seventh aspect, the wearable display system of any one of aspects 1-6 is further programmed to determine a distance between the location of the virtual cursor and the object, and if the distance is less than a threshold distance, update the location of the virtual cursor to be a location representing the object.

[0231] In an eighth aspect, a wearable display system described in any one of aspects 1-7, wherein the hardware processor is programmed to determine an orientation between the location of the virtual cursor and the object and to instruct the display to preferentially render the focus indicator toward the orientation of the virtual cursor.

[0232] In a ninth aspect, the wearable display system of any one of aspects 1-8, wherein the hardware processor is programmed to update the location of the virtual cursor based at least in part on the previous movement path of the virtual cursor.

[0233] In a tenth aspect, the wearable display system of aspect 9, wherein the hardware processor is programmed to update the location of the virtual cursor in response to cessation of user input from the user input device, based at least in part on the previous movement path.

[0234] In an eleventh aspect, a wearable display system described in any one of aspects 1-10, wherein the hardware processor is programmed to determine a distance between the location of the virtual cursor and a portion of the object, and in response to determining that the distance is less than a threshold, render the object in front of the virtual cursor or cease rendering the virtual cursor.

[0235] In a twelfth aspect, a method of rendering a cursor relative to an object in a mixed reality environment, the method including: determining, under control of a mixed reality display device having a display and a hardware processor, a location of the cursor in the mixed reality environment; determining a location associated with the object; determining whether an overlap occurs between the object and the cursor; rendering the cursor in response to determining that no overlap occurs; and rendering the object in front of the cursor or ceasing rendering of the cursor in response to determining that overlap occurs.

[0236] In a thirteenth aspect, the method of aspect 12, wherein determining whether an overlap occurs between the object and the cursor includes determining whether the distance between the location of the cursor and the location of the object is less than a distance threshold.

[0237] In a fourteenth aspect, the method of aspect 12 or aspect 13, wherein in response to determining that an overlap occurs, the method further comprises rendering a focus indicator associated with the object.

[0238] In a fifteenth aspect, the method of aspect 14, wherein the focus indicator comprises one or more of a glow or halo at least partially surrounding the object, a size or depth change of the object, or a graphical highlight.

[0239] In a sixteenth aspect, the method of any one of aspects 12-15 further includes determining a spatial relationship between the cursor and the object, and rendering a focus indicator associated with the object based at least in part on the spatial relationship.

[0240] In a seventeenth aspect, an augmented reality display system includes a user input device configured to receive user input related to a position of a cursor in a user's environment, a display through which a user can perceive virtual objects in the user's environment, and a hardware processor in communication with the user input device and the display, the hardware processor being programmed to: cause the virtual object to be rendered via the display; track a position of the cursor in the environment based at least in part on the user input; identify a position of the virtual object; determine whether a first distance between the virtual object and the position of the cursor satisfies a distance threshold; and, in response to determining that the first distance satisfies the distance threshold, cause a focus indicator to be rendered in proximity to the object via the display.

[0241] In an eighteenth aspect, the system described in aspect 17, wherein in response to determining that the first distance does not satisfy the distance threshold, the hardware processor is further programmed to cause a reticle indicating the position of the cursor to be rendered via the display.

[0242] In a nineteenth aspect, the system of aspect 17 or aspect 18, wherein the virtual object comprises a selectable object, whereby in response to user input selecting the object, the hardware processor performs an action associated with the selectable object.

[0243] In a twentieth aspect, the system described in any one of aspects 17-19, wherein the first distance satisfies the distance threshold when the cursor position overlaps at least a portion of the virtual object.

[0244] In a 21st aspect, the system described in any one of aspects 17-20, wherein the first distance satisfies a distance threshold if the cursor position and at least a portion of the virtual object would be mapped to the same pixel on the display.

[0245] In aspect 22, the system described in any one of aspects 17-21, wherein the first distance satisfies the distance threshold if the cursor position and at least a portion of the virtual object would be co-located when rendered on the display.

[0246] In a 23rd aspect, the system of any one of aspects 17-22, wherein the virtual object is a first object, the focus indicator is a first focus indicator, and the hardware processor is further programmed to: cause a second virtual object to be rendered via the display; identify a position of the second object; determine whether a distance between the second object and the position of the cursor satisfies a second distance threshold; and, in response to determining that the distance between the second object and the position of the cursor satisfies the second distance threshold, cause a second focus indicator to be rendered via the display in proximity to the second virtual object.

[0247] In a 24th aspect, the system described in aspect 23, wherein the hardware processor is programmed to, in response to determining that the distance between the first virtual object and the cursor is less than the distance between the second virtual object and the cursor, cause the first focus indicator to be rendered more prominently than the second focus indicator.

[0248] In a 25th aspect, the system described in aspect 24, wherein, to cause the first focus indicator to be rendered more prominently than the second focus indicator, the hardware processor is programmed to perform one or more of: rendering the first focus indicator brighter than the second focus indicator; rendering the first focus indicator larger than the second focus indicator; or rendering the first focus indicator more prominently in a direction towards the cursor than in a direction away from the cursor.

[0249] In a 26th aspect, the system described in any one of aspects 17-25, wherein the hardware processor is programmed to identify a shape of a virtual object, determine a cursor orientation relative to the virtual object, and cause a focus indicator to be rendered based at least in part on the shape of the virtual object and the cursor orientation relative to the virtual object.

[0250] In a 27th aspect, the system described in aspect 26, wherein the hardware processor is programmed to cause the focus indicator to be rendered more prominently along the line between the cursor and the virtual object.

[0251] In a twenty-eighth aspect, an augmented reality display system includes a user input device configured to receive user input related to a position of a cursor in a user's environment, a display through which a user can perceive virtual objects in the user's environment, and a hardware processor in communication with the user input device and the display, the hardware processor being programmed to: cause a plurality of virtual objects to be rendered via the display; track a position of the cursor within a field of view of the display; identify positions of the plurality of virtual objects; compare the position of the cursor with the positions of the plurality of virtual objects to determine an object closest to the position of the cursor; and cause a focus indicator to be rendered via the display in proximity to the closest object.

[0252] In a 29th aspect, the hardware processor is programmed to cause the second focus indicator to be rendered in proximity to at least one other virtual object that is not the closest object, and the first focus indicator is rendered more prominently than the second focus indicator.

[0253] In a 30th aspect, the hardware processor is programmed to determine whether the cursor position overlaps with the nearest object, and in response to a determination that no overlap occurs, render the cursor, and in response to a determination that overlap occurs, render the nearest object in front of the cursor or cease rendering the cursor, in a system described in aspect 28 or aspect 29.

[0254] In a thirty-first aspect, the system described in any one of aspects 28-30, wherein the hardware processor is programmed to accelerate the position of the cursor to the position of the nearest object.

[0255] In a thirty-second aspect, a method of rendering a graphical user interface for a mixed reality display device, the method including, under control of a mixed reality display device having a display and a hardware processor, determining a location of a cursor within a field of view of the display, rendering a focus indicator associated with an object within the field of view of the display, tracking movement of the cursor relative to the object, and adjusting the rendering of the focus indicator based at least in part on the movement of the cursor relative to the object.

[0256] In a thirty-third aspect, the method of aspect 32, wherein the focus indicator comprises one or more of a glow or halo at least partially surrounding the object, a size or depth change of the object, or a graphical highlight.

[0257] In a thirty-fourth aspect, the method of aspect 32 or aspect 33 further comprises rendering a cursor.

[0258] In a thirty-fifth aspect, the method of aspect 34 further comprises rendering the cursor less prominently or not at all when the cursor is located behind the object.

[0259] In a 36th aspect, the method of any one of aspects 32-35, wherein rendering the focus indicator includes rendering at least a portion of the focus indicator more prominently in a direction toward the cursor than in a direction away from the cursor.

[0260] In a thirty-seventh aspect, the method of aspect 36, wherein rendering at least a portion of the focus indicator more prominently includes one or more of rendering a portion brighter, rendering a portion larger, rendering a portion in a different color, or rendering a portion in a different graphical style.

[0261] In a thirty-eighth aspect, the method of any one of aspects thirty-two to thirty-seven, wherein adjusting the rendering of the focus indicator includes simulating the visual appearance of attraction between the focus indicator and the cursor.

[0262] In a 39th aspect, the method of any one of aspects 32-38, wherein adjusting the rendering of the focus indicator includes emphasizing the visual appearance of a first portion of the focus indicator that is closer to the cursor compared to the visual appearance of a second portion of the focus indicator that is farther from the cursor.

[0263] In a fortieth aspect, the method of any one of aspects 32-39, wherein adjusting the rendering of the focus indicator includes simulating the visual appearance of a cursor that pulls a portion of the focus indicator from the object toward the cursor.

[0264] In a forty-first aspect, a wearable display system includes: a display configured to be positioned in front of a user's eyes, the display configured to project virtual content toward the user's eyes; a user input device configured to receive user input data associated with movement of a virtual cursor in a virtual environment; and a hardware processor in communication with the display and the user input device, the hardware processor being programmed to: instruct the display to render a virtual icon at a first depth in the virtual environment, instruct the display to render a virtual cursor at a second depth in the virtual environment, track movement of the virtual cursor in the virtual environment, determine whether an overlap occurs between the virtual cursor and the virtual icon, and in response to a determination of an overlap, instruct the display to render the virtual icon at a third depth closer to the user than the first depth or the second depth.

[0265] In aspect 42, the wearable display system of aspect 41, wherein the user input device comprises one or more of a handheld totem having a touch-sensitive surface, an outward-facing imaging system configured to detect user gestures, an inward-facing imaging system configured to detect the user's eye posture, or an inertial measurement unit configured to detect the user's head posture.

[0266] In aspect 43, the wearable display system of aspect 41 or aspect 42, wherein the first depth is the same as the second depth.

[0267] In aspect 44, a wearable display system described in any one of aspects 41-43, further wherein in response to determining the overlap, the hardware processor is programmed to enhance the visual appearance of the virtual icon compared to the visual appearance of the virtual cursor.

[0268] In aspect 45, the wearable display system of aspect 44 is programmed to instruct the display to: render the virtual icon larger than when rendered at the first depth, or render a focus indicator around at least a portion of the virtual icon, or render a caption associated with the virtual icon, or render the virtual icon with additional virtual content than when rendered at the first depth, or render the virtual icon at a higher resolution or higher brightness than when rendered at the first depth, or cease rendering the virtual cursor, or render the virtual cursor at a reduced brightness or increased transparency, in order to enhance the visual appearance of the virtual icon.

[0269] In a forty-sixth aspect, the wearable display system of any one of aspects 41-45, wherein the hardware processor is further programmed to receive user input for selecting a virtual icon.

[0270] In aspect 47, the wearable display system of any one of aspects 41-46 is further programmed to track further movement of the virtual cursor within the virtual environment, determine whether overlap between the virtual cursor and the virtual icon no longer occurs, and in response to determining that overlap no longer occurs, instruct the display to render the virtual icon at the first depth.

[0271] In a forty-eighth aspect, the wearable display system of aspect 47 is programmed to render the virtual icon with a visual appearance that is the same as the visual appearance of the virtual icon before overlapping, or to render the virtual cursor with a visual appearance that is the same as the visual appearance of the virtual icon before overlapping.

[0272] In a forty-ninth aspect, the hardware processor is programmed to render a virtual layout comprising a plurality of virtual icons, the plurality of virtual icons comprising a virtual icon, a wearable display system described in any one of aspects 41-48.

[0273] In a 50th aspect, the wearable display system of aspect 49, wherein the hardware processor is programmed to receive user instructions for scrolling the virtual layout.

[0274] In a fifty-first aspect, the wearable display system of aspect 50, wherein the user instruction includes a scrolling direction or a scrolling speed.

[0275] In aspect 52, the wearable display system of aspect 50 or aspect 51, wherein the user instruction includes hovering a virtual cursor over the edge of the virtual layout.

[0276] In aspect 53, a wearable display system described in any one of aspects 50-52, wherein the hardware processor is programmed to scroll the virtual layout and instruct the display to render a virtual content panel including representations of virtual icons that had not been rendered prior to the scrolling.

[0277] In a fifty-fourth aspect, the wearable display system of aspect 53, wherein the virtual content panel is rendered at a depth different from the depth at which at least a portion of the virtual layout is rendered.

[0278] In a fifty-fifth aspect, the wearable display system of any one of aspects 49-54, wherein the virtual layout comprises a regular grid or an irregular grid.

[0279] In a 56th aspect, the wearable display system of any one of aspects 41-55, wherein the virtual icon includes a thumbnail of an application or media file.

[0280] In a fifty-seventh aspect, the wearable display system of aspect 56, wherein the media file includes a video file, an audio file, or a document file.

[0281] In a fifty-eighth aspect, a wearable display system includes: a display configured to be positioned in front of a user's eyes, the display configured to project virtual content toward the user's eyes; a user input device configured to receive user input data associated with movement of a virtual cursor in a virtual environment; and a hardware processor in communication with the display and the user input device, wherein the hardware processor is programmed to: instruct the display to render a virtual layout of a plurality of virtual icons, at least some of the plurality of virtual icons in the virtual layout being rendered at a first depth in the virtual environment; receive user instructions to scroll the virtual layout; and in response to receiving the user instructions to scroll the virtual layout, instruct the display to render a virtual content panel including representations of the virtual icons that were not rendered prior to the scrolling, the virtual content panel being rendered at a second depth in the virtual environment.

[0282] In a fifty-ninth aspect, the wearable display system of aspect 58, wherein the first depth is different from the second depth.

[0283] In a 60th aspect, the wearable display system of aspect 58 or aspect 59, wherein the user instruction includes a scrolling direction or a scrolling speed.

[0284] In aspect 61, the wearable display system of any one of aspects 58-60, wherein the user instruction includes hovering a virtual cursor over the edge of the virtual layout.

[0285] In aspect 62, the wearable display system of any one of aspects 58-61, wherein the virtual content panel includes a message indicating the content of the multiple icons being scrolled.

[0286] In a 63rd aspect, the wearable display system of any one of aspects 58-62, wherein the representation of the virtual icon includes a thumbnail.

[0287] In aspect 64, the wearable display system of any one of aspects 58-63, wherein scrolling of the virtual layout is performed using a driving force or a drag force.

[0288] In aspect 65, a wearable display system described in any one of aspects 58-64, wherein the hardware processor is further programmed to receive a user instruction to pause scrolling of the virtual layout and, in response to receiving the user instruction to pause scrolling of the virtual layout, instruct the display to stop scrolling of the virtual layout and stop rendering of the virtual content panel. (Additional Considerations)

[0289] Each of the processes, methods, and algorithms described herein or depicted in the accompanying figures may be embodied in code modules executed by one or more physical computing systems, hardware computer processors, application-specific circuits, or electronic hardware configured to execute specific computer instructions, and thereby may be fully or partially automated. For example, a computing system may include a general-purpose computer (e.g., a server) or a special-purpose computer, special-purpose circuitry, etc., programmed with specific computer instructions. Code modules may be installed in a dynamic link library that is compiled and linked into an executable program, or may be written in an interpreted programming language. In some implementations, specific operations and methods may be performed by circuitry that is specific to a given function.

[0290] Furthermore, certain implementations of the functionality of the present disclosure may be sufficiently mathematically, computationally, or technically complex that special-purpose hardware (utilizing appropriate specialized executable instructions) or one or more physical computing devices may be required to implement the functionality, e.g., due to the amount or complexity of the calculations involved, or to provide results in substantially real time. For example, a video may contain many frames, each frame may have millions of pixels, and specifically programmed computer hardware may be required to process the video data to provide the desired image processing task or application in a commercially reasonable amount of time.

[0291] Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as physical computer storage, including hard drives, solid-state memory, random-access memory (RAM), read-only memory (ROM), optical disks, volatile or non-volatile storage, combinations thereof, etc. The methods and modules (or data) may be transmitted as data signals (e.g., as part of a carrier wave or other analog or digital propagated signal) generated over a variety of computer-readable transmission media, including wireless-based and wired / cable-based media, and may also take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple separate digital packets or frames). The results of the disclosed processes or process steps may be stored, persistently or otherwise, in any type of non-transitory tangible computer storage or communicated via a computer-readable transmission medium.

[0292] Any process, block, state, step, or functionality in the flow diagrams described herein or depicted in the accompanying figures should be understood as potentially representing a code module, segment, or portion of code, comprising one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in the process. Various processes, blocks, states, steps, or functionality can be combined, rearranged, added, deleted, modified, or otherwise changed from the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules may perform some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps, or states associated therewith can be performed in other suitable sequences, e.g., serially, in parallel, or in some other manner. Tasks or events may be added to or removed from the disclosed exemplary embodiments. Furthermore, the separation of various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods, and systems may generally be integrated together in a single computer product or packaged in multiple computer products. Many implementation variations are possible.

[0293] The processes, methods, and systems may be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowdsourced computing networks, the Internet, and the World Wide Web. The network may be a wired or wireless network or any other type of communication network.

[0294] Each of the systems and methods of the present disclosure has several innovative aspects, none of which is solely responsible for or required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of one another or combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of the present disclosure. Various modifications of the implementations described in this disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations without departing from the spirit or scope of the present disclosure. Therefore, the claims are not intended to be limited to the implementations shown herein, but are to be accorded the widest scope consistent with the present disclosure, the principles, and novel features disclosed herein.

[0295] Certain features described herein in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination. Furthermore, while features may be described above as functioning in a combination and may even be initially claimed as such, one or more features from the claimed combination can, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination. No single feature or group of features is required or essential to every embodiment.

[0296] In particular, conditional statements used herein, such as "can," "could," "might," "may," "e.g.," and the like, are generally intended to convey that certain embodiments include certain features, elements, or steps, while other embodiments do not, unless specifically stated otherwise or understood otherwise within the context in which they are used. Thus, such conditional statements are generally not intended to imply that features, elements, or steps are in any way required for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, or steps should be included or performed in any particular embodiment, with or without authorial input or prompting. The terms "comprising," "including," "having," and the like, are synonymous and used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. The term "or" is also used in its inclusive sense (and not its exclusive sense), so, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Additionally, the articles "a," "an," and "the," as used in this application and the appended claims, should be interpreted to mean "one or more" or "at least one," unless otherwise specified.

[0297] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single elements. As an example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Transitional phrases such as "at least one of X, Y, and Z," unless specifically stated otherwise, are generally understood differently in the context in which they are used to convey that an item, term, etc. may be at least one of X, Y, or Z. Thus, such transitional phrases generally are not intended to suggest that an embodiment requires that at least one of X, at least one of Y, and at least one of Z, respectively, be present.

[0298] Similarly, while operations may be depicted in the figures in a particular order, it should be recognized that such operations need not be performed in the particular order shown, or in sequential order, or that all of the depicted operations need not be performed, to achieve desirable results. Furthermore, the figures may diagrammatically depict one or more example processes in the form of a flowchart. However, other operations not depicted may be incorporated within the diagrammatically depicted example methods and processes. For example, one or more additional operations may be performed before, after, simultaneously with, or between any of the depicted operations. Additionally, operations may be rearranged or reordered in other implementations. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. (Appendix A) A portion of the disclosure of this Appendix contains material that is subject to copyright protection. The copyright owner has no objection to anyone copying this patent document or this patent disclosure (including this Appendix A) as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyrights whatsoever. The following computer code and explanations are intended to illustrate various embodiments of the occlusion cursor technique, but are not intended to limit the scope of the occlusion cursor technique. I. Occultation Cursor Overview The occultation cursor represents a way to highlight user selections when using a pointing device. Rather than the traditional approach of showing a small pointer "sprite" that moves over or on top of selectable content, the occultation cursor moves behind that content, providing the user with positional feedback through the movement of a glow that shines out from behind the selected item. By continuing to precisely track user input while highlighting the selected item, the occultation glow will shift position, giving the user persistent input feedback and a precise sense of cursor position. In the example described in this appendix, the user uses the occultation cursor to target planar user interface (UI) elements by moving a touch-controlled cursor or focus indicator. Relative cursor logic and data are based on touchpad information, which is provided by GripTotem script. This document presents an example of a relative occultation cursor input algorithm. II. Occluded Cursor Features The cursor can have inertia. The cursor position can be clipped to the panel. The panel can have rounding settings so that the input area can be round, capsule-shaped, or rectangular with some degree of corner rounding. The cursor can have the functionality to snap onto elements when the user's finger is released from the touchpad. III. Relative Dynamics The class CursorRelativeInput can implement a relative cursor. It can respond to user input (e.g., totem touchpad feedback) to update the position of a cursor placed within a limited area of ​​a 3D plane. The term "relative" can be used to describe the core input / motion response of the cursor. That is, when the user presses the totem's touchpad, the system will update the cursor so that it appears to move along the equivalent motion progression in the control plane. Each motion step can be relative to the previous position. (A. Cursor and Panel Interaction) Instances of CursorRelativeInput are spawned by EclipsePanel (as one of several cursor control options). Panels can provide a notion of activityscope for the cursor; that is, a cursor instance can be updated when the panel has Focus. They can also define cursor movement bounds and with them the primary set of elements the cursor can interact with (e.g., buttons that are children of that same panel instance). An example of an EclipsePanel is shown in Figure 22. The areas below "Social", "Application", and "Store" are EclipsePanels. The cursor bounds can be defined as a rectangular region that maps exactly to the dimensions defined for the EclipsePanel, or it can be a quadratic set of custom bounds provided by the panel (e.g., if the control area simply represents a subset of the space occupied by the panel). Panels can have a "rounded" attribute that matches other occlusion elements. This can mean that the panel (and therefore the cursor boundary) can be a perfect circle or capsule, a sharp square or rectangle, or any rounded corner shape in between. Relative cursors can respect the rounded state of the panel as boundaries may be applied. In some cases, there can be an arbitrary number of panels active. Therefore, there can be multiple relative cursor instances. In some cases, only one panel can have an InputFocus. This can be the only panel that will actively update its cursor. This can be achieved via a call from a central EclipseUI class, and the result of this update can be a cursor ray that will cast from the user-centered position (e.g., head pose or totem position) through the position on the panel's control plane. In addition to, or as an alternative to, detecting buttons belonging to its host panel, the cursor may detect buttons belonging to other panels that are allowed to share focus with the active input panel. (B. Cursor Collision) Using the cursor ray provided by the input panel's cursor updates, Eclipse UI can perform ray casting on the active interactable element (e.g., a button belonging to the panel that currently has Focus). The tests performed here can use mathematics-based ray casting, which offers several advantages over the use of colliders. For example, advantages include, but are not limited to: Tests can reflect the round shape of buttons (e.g., using the same math used to render buttons, for optimal consistency). The system can determine both whether a button is being hovered and how close the button may be to the cursor. This can serve at least two important functions. First, the system can determine which buttons are close to the cursor, and based on this determination, the system can begin to show an occultation glow as it approaches those buttons. Second, in some cases it is desirable to find the closest button to the current cursor position (e.g., for gravity well support). To avoid the need for colliders, scenes can appear clearer and the system can avoid the complexities inherent in correct filtering of collisions or accidental occlusion of buttons by either other buttons or other scene-based colliders. Based at least in part on ray casting, the system can determine that a particular button has been "collided." In some cases, in response to this determination, the system can employ a secondary collider-based test. This test can cast the same cursor ray as the button location. However, here, the system tests the collider that owns the "UI" collision layer. In some cases, this provides a mechanism that allows a given EclipsePanel to be constructed with a solid "backplane." In other words, cursor-ray casting can be prevented from passing through gaps in the panel and colliding with interactable objects that may be behind it. One example is a keyboard placed in front of active search results. It may not be desirable for those results to be interactive through the keyboard. Instead, it may be desirable for results to be interactive only when adjacent to the keyboard. (Collision implementation:) A collision can be aligned against one interactable object per frame (e.g., the first frame to be bumped), however the system can continue to test against others to update its proximity glow state. Some interactables can be given priority testing. For example, each frame that contained a hovered interactable from the previous update can be given priority in testing that frame against that same interactable. This can help ensure a stable collision response. In some cases, if the frames do not collide, the system will again test at the position it would have occupied if it had not been hovered. This is a means to deal with cases of hysteresis that might otherwise occur if the button were to advance in response to being hovered. In that case, the cursor position would remain unchanged, and the next cursor ray could lead to the button being missed, causing it to retract and loop again. The user may be able to modify the collision size scalar for actively hovered buttons. This may be useful, for example, for small buttons to make it more difficult to accidentally move them past or away from them while attempting to use a trackpad press to click the button. For example, the size of a button may be increased slightly while it is hovered, and returned to a one-to-one scale when it is unhovered. When an interactive element is hovered over by the cursor, further interactions (e.g., button presses) may be handled by specific classes. This is handled in the standard Eclipse manner via OnHover / DeHover, OnClick, OnClickStart / End style events. This can occur via the Event mechanism. (C. Cursor Rendering) Another aspect of cursor handling can be cursor rendering. In most cases, only one cursor is visible (e.g., the one from the Input Focus panel). This cursor can be shown through an occlusion rendering process. Whether the cursor position is indicated in the form of an obscuring "backglow," a more traditional position dot, etc., is based at least in part on a variety of factors. For example, factors may include whether an obscuring element is currently being hovered over, and if not, whether a "dot" cursor is enabled to be visible for the active panel. In some cases, the cursor may be entirely hidden. IV. Relative Cursor Update Implementation For any cursor mode, the system has a settings structure (configurable per panel, if any) that allows the user or the system to customize the behavior. For example, for relative cursors, the settings structure can include, but is not limited to: Movement scalar This allows for control of cursor speed, e.g., the size of movement steps based on touchpad input. Select X or Y input / orthogonalize swipe These options can allow you to bias input handling to favor one axis over another, or to prioritize a base motion direction. For example, an option can allow a choice between the larger of the X or Y component of the current touch input. The bias factor can add additional control. Orthogonalizing swipes can mean that when input on one axis exceeds a threshold, movement can be zero on other axes. This can be useful, for example, for a keyboard with many buttons positioned in a grid, so the system knows that the user's intent will often be to move the cursor smoothly along a column of adjacent letters. Gravity well support o When the user releases the touchpad, these options allow the cursor to slide (e.g., as if pulled by gravity) to a position within the nearest button. In some cases, this sliding of the cursor can occur always, never, or only if the nearest button is within a certain distance tolerance. Settings can include whether the cursor will move to the nearest position on the nearest button, or to a position that aligns with one or both of the button's X and Y axes (e.g., if there is a long row of adjacent buttons, it may be desirable to snap to center Y, or perhaps center X, for a vertical stack of smaller circular buttons). Settings can also include whether use of gravity wells is desired only within the host panel, or whether elements present on other "in focus" panels can also be considered. Edge push Using touchpad controls, the user can switch the input focus from one panel to another. When the cursor encounters the edge boundary of a panel, the system can send an event to the user, which the user can use to initiate a panel transition according to the push direction. In some cases, the system may select a spring-loaded edge, which can cause visual feedback (e.g., movement of the host panel) and help communicate that an edge push is occurring. In this case, if a certain push range is not exceeded, the panel can rebound to its original position where no edge push event was sent. In some cases, settings include timing options (e.g., push against the edge for a certain amount of time) and double-tap options (e.g., hit against the edge, release the input, then swipe against the same edge again). Inertia Control o Each time the user provides touchpad input and cursor movement is determined or rendered, the system can also associate a cursor movement, which can mimic a degree of "inertia." For example, this movement can be applied from the moment active input ceases, causing the cursor to continue along its motion path until a damping force reduces the "inertia" to zero. Controls can limit the amount of inertia accumulated, and allow inertia boost to be applied on the event (e.g., corresponding to a configurable threshold) of the user releasing the touchpad at the end of a fast "swipe" action. Inertia boost is intended to support fast swiping through long itemized lists (e.g., to allow one large swipe to carry the cursor from top to bottom when the user makes a selection). Scrolling support o If a panel has content that exceeds its available screen area, it can be scrolled. Relative cursors can have built-in push-scroll support. A configurable parameter can control the distance from the panel edge at which the push-scroll movement step will be applied. Depending on how the relative cursor is configured (eg, via a configurable setting), the relative cursor update may include one or more of the following steps. Check touchpad swipes For example, checking whether a fast finger movement across the touchpad has just ended. o Potentially apply inertia multiplication. Check for regular touch input o Apply the movement to the cursor position o Accumulate directional inertia. When there is no touchpad input Apply inertia-based motion to the cursor position o If touchpad input has just ended, find the nearest gravity well o Implement potential gravity well processing and apply it to cursor movement. Note: This can be layered on top of inertia so that the two can work together. For example, gravity snapping can kick in once inertia has subsided sufficiently. Handling push-scroll for scroll panel types o Push cursor position - clip against scroll bounds and apply "overflow" to scroll offset Clip cursor position to panel borders o This can be implemented to ensure that the cursor can slide smoothly around any curved corner. Check edge push Track time-based / double tap / spring-loaded edge pushes or send events to the user. Dampens inertia V. Input Language Eclipse UI Eclipse UI is a set of Unity classes to support button and cursor rendering. (EclipsePanel) EclipsePanel is a class within EclipseUI. Panels support per-panel cursor rendering and setting. CursorRelativeInput (and other optional cursor type classes) are distributed and updated by the EclipsePanel instance. Panels have the concept of a "Focus" (the button on which they can be clicked) and an "Input Focus" (the cursor will be refreshed and rendered as the active system cursor). A panel with a "Focus" can be set when a head pose targets it. There can be multiple "Focus" panels, but only one with an "Input Focus". (CursorRelativeInput) The CursorRelativeInput class is implemented by the ICursor interface. This cursor has inertia, button snapping, and edge pushing characteristics. Totem (TouchPad) The Touch Pad is a circular surface (device) for pointing (controlling input position) on the totem. (A. GripTotem) GripTotem is a class that reads raw data from a serial port. VI. ECLIPSE PANEL (A. [Parameters]) [Table 1] (B. [Function]) [Table 2] VII. ICURSOR [Table 3] VIII. Relative Cursor (A. [Serializable Class]) [Table 4] (B. [Parameters]) [Table 5] (C. [Function]) [Table 6-1] [Table 6-2] [Table 6-3] IX. Clip cursor position to panel [Table 7-1] Table 7-2

Claims

1. 1. A method of rendering a graphical user interface for a mixed reality display device, the method comprising, under control of a mixed reality display device comprising a display and a hardware processor: determining a location of a cursor within a field of view of the display; Rendering a focus indicator associated with an object within the field of view of the display, the focus indicator being on, near, or at least partially surrounding the object, or a visual modification of the object involving a size or depth change or a graphical highlighting of the object; tracking movement of the cursor relative to the object; adjusting the rendering of the focus indicator based on the movement of the cursor relative to the object, wherein adjusting the rendering of the focus indicator includes simulating a visual appearance of attraction between the focus indicator and the cursor; A method comprising:

2. 2. The method of claim 1, wherein rendering the focus indicator comprises rendering at least a portion of the focus indicator more prominently in a direction toward the cursor than in a direction away from the cursor, and wherein rendering at least a portion of the focus indicator more prominently comprises one or more of rendering the portion brighter, rendering the portion larger, rendering the portion in a different color, or rendering the portion in a different graphical style.

3. The method of claim 1 , wherein rendering the focus indicator comprises rendering at least a portion of the focus indicator more prominently in a direction toward the cursor than in a direction away from the cursor.

4. The method of claim 1 , wherein the focus indicator comprises one or more of a glow or halo at least partially surrounding the object, a size or depth change of the object, or a graphical highlight.

5. The method of claim 1 , further comprising rendering the cursor less prominently or not at all when the cursor is located behind the object.

6. 2. The method of claim 1, wherein adjusting the rendering of the focus indicator comprises enhancing the visual appearance of a first portion of the focus indicator that is closer to the cursor compared to the visual appearance of a second portion of the focus indicator that is farther from the cursor.

7. The method of claim 1 , further comprising rendering the cursor.

8. 1. A method of rendering a graphical user interface for a mixed reality display device, the method comprising, under control of a mixed reality display device comprising a display and a hardware processor: determining a location of a cursor within a field of view of the display; Rendering a focus indicator associated with an object within the field of view of the display, the focus indicator being on, near, or at least partially surrounding the object, or a visual modification of the object involving a size or depth change or a graphical highlighting of the object; tracking movement of the cursor relative to the object; adjusting the rendering of the focus indicator based on the movement of the cursor relative to the object, where adjusting the rendering of the focus indicator includes simulating the visual appearance of the cursor pulling a portion of the focus indicator from the object toward the cursor; A method comprising:

9. 1. An augmented reality display system, comprising: a user input device configured to receive user input related to a cursor position within the user's environment; a display through which a user can perceive virtual objects in the user's environment; and a hardware processor in communication with the user input device and the display; wherein the hardware processor comprises: determining a location of a cursor within a field of view of the display; Rendering a focus indicator associated with an object within the field of view of the display, the focus indicator being on, near, or at least partially surrounding the object, or a visual modification of the object involving a size or depth change or a graphical highlighting of the object; tracking movement of the cursor relative to the object; adjusting the rendering of the focus indicator based on the movement of the cursor relative to the object, wherein adjusting the rendering of the focus indicator includes simulating a visual appearance of attraction between the focus indicator and the cursor; an augmented reality display system that is programmed to:

10. 10. The augmented reality display system of claim 9, wherein rendering the focus indicator comprises rendering at least a portion of the focus indicator more prominently in a direction toward the cursor than in a direction away from the cursor, and wherein rendering at least a portion of the focus indicator more prominently comprises one or more of rendering the portion brighter, rendering the portion larger, rendering the portion in a different color, or rendering the portion in a different graphical style.

11. 10. The augmented reality display system of claim 9, wherein rendering the focus indicator comprises rendering at least a portion of the focus indicator more prominently in a direction toward the cursor than in a direction away from the cursor.

12. 10. The augmented reality display system of claim 9, wherein the focus indicator comprises one or more of a glow or halo at least partially surrounding the object, a size or depth change of the object, or a graphical highlight.

13. 10. The augmented reality display system of claim 9, wherein the augmented reality display system is configured to render the cursor less prominently or not at all when the cursor is located behind the object.

14. 10. The augmented reality display system of claim 9, wherein adjusting the rendering of the focus indicator includes emphasizing the visual appearance of a first portion of the focus indicator that is closer to the cursor compared to the visual appearance of a second portion of the focus indicator that is farther from the cursor.

15. The augmented reality display system of claim 9 , wherein the augmented reality display system is configured to render the cursor.

16. 1. An augmented reality display system, comprising: a user input device configured to receive user input related to a cursor position within the user's environment; a display through which a user can perceive virtual objects in the user's environment; and a hardware processor in communication with the user input device and the display; wherein the hardware processor comprises: determining a location of a cursor within a field of view of the display; Rendering a focus indicator associated with an object within the field of view of the display, the focus indicator being on, near, or at least partially surrounding the object, or a visual modification of the object involving a size or depth change or a graphical highlighting of the object; tracking movement of the cursor relative to the object; adjusting the rendering of the focus indicator based on the movement of the cursor relative to the object, where adjusting the rendering of the focus indicator includes simulating the visual appearance of the cursor pulling a portion of the focus indicator from the object toward the cursor; an augmented reality display system that is programmed to:

Images