Visual halo around the field of view

Abstract

To provide a visual aura around a field of view.SOLUTION: A wearable device can have a field of view through which a user can perceive real or virtual objects. The device can display a visual aura representing contextual information associated with an object that is outside the user's field of view. The visual aura can be displayed near an edge of the field of view, and can dynamically change as the contextual information associated with the object changes, e.g., as the relative positions of the object and the user (or the user's field of view) change.SELECTED DRAWING: Figure 13

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of priority under 35 U.S.C. § 119(e) to U.S. Provisional Application No. 62 / 325,685, filed on April 21, 2016, entitled "RENDERING AURAS FOR OBJECTS IN AN AUGMENTED OR VIRTUAL REALITY ENVIRONMENT", which is hereby incorporated by reference in its entirety.

[0002] The present disclosure relates to virtual reality and augmented reality imaging and visualization systems, and more particularly to informing a user of objects in an environment.

Background Art

[0003] Modern computing and display technologies have facilitated the development of systems for so - called "virtual reality" or "augmented reality" experiences, where digitally reproduced images or portions thereof are presented to a user in a manner that appears or is perceived to be real. A virtual reality or "VR" scenario typically involves the presentation of digital or virtual image information without transparency to other actual real - world visual inputs. An augmented reality or "AR" scenario typically involves the presentation of digital or virtual image information as an augmentation to the visualization of the actual world around the user. Mixed reality or "MR" relates to the fusion of the real world and the virtual world to create a new environment where physical and virtual objects co - exist and interact in real - time. In conclusion, the human visual perception system is very complex, and the generation of VR, AR, or MR technologies that facilitate a comfortable, natural - like, and rich presentation of virtual image elements among other virtual or real - world image elements is difficult. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies.

Summary of the Invention

[0004] In one embodiment, a system is disclosed for providing indications of interactable objects in a user's three-dimensional (3D) environment. The system may comprise a display system for a wearable device configured to present a three-dimensional view to the user and enable user interaction with objects in the user's eye-moving field of view (FOR). FOR may include a portion of the user's surrounding environment that is perceptible to the user via the display system. The system may also comprise sensors configured to obtain data associated with the user's posture, and a hardware processor communicating with the sensors and the display system. A hardware processor can be programmed to determine the user's pose based on data obtained by sensors, determine the user's field of view (FOV) at least partially based on the user's pose, the FOV including a portion of the FOR perceptible to the user at a given time via a display system, identify interactable objects located outside the user's FOV, access contextual information associated with the interactable objects, determine a visual representation of the aura based on the contextual information, and render the visual representation of the aura such that at least a portion of the visual aura perceptible to the user lies at the edge of the user's FOV.

[0005] In another embodiment, a method for providing indication of interactable objects in a user's three-dimensional (3D) environment is disclosed. The method may be performed under the control of a wearable device having a display system configured to present a three-dimensional (3D) view to a user and enable user interaction with objects in the user's eye-moving field of view (FOR), wherein the FOR may include a portion of the user's surrounding environment that is perceptible to the user via the display system; a sensor configured to obtain data associated with the user's posture; and a hardware processor communicating with the sensor and the display system. The method may at least partially include determining the user's field of view (FOV) based on the user's posture, wherein the FOV includes a portion of the FOR that is perceptible to the user via the display system at a given time; identifying interactable objects located outside the user's FOV; accessing contextual information associated with the interactable objects; determining a visual representation of the backlight based on the contextual information; and rendering the visual representation of the backlight such that at least a portion of the visual backlight perceptible to the user is at the edge of the user's FOV.

[0006] Details of one or more implementations of the subject matter described herein are shown in the accompanying drawings and the following description. Other features, aspects, and advantages will be evident from the description, drawings, and claims. Neither this abstract nor any of the following embodiments for carrying out the invention shall claim to define or limit the scope of the subject matter of the invention. For example, this application provides the following items. (Item 1) A system for providing indication of interactable objects in a user's three-dimensional (3D) environment, wherein the method is: A display system for a wearable device, configured to present a three-dimensional view to a user and enable user interaction with objects within the user's eye-moving field of view (FOR), wherein the FOR includes a portion of the environment surrounding the user that is perceptible to the user through the display system. A sensor configured to obtain data associated with the user's posture, A hardware processor that communicates with the sensor and the display system, wherein the hardware processor is Based on the data obtained by the aforementioned sensor, the user's posture is determined, Determining the user's field of view (FOV) at least partially based on the user's posture, wherein the FOV includes a portion of the FOR that is perceptible to the user at a given time via the display system, Identifying interactable objects located outside the user's FOV, Accessing context information associated with the aforementioned interactable object, Based on the aforementioned contextual information, the visual representation of the backlight is determined, Rendering the visual representation of the backlight such that at least a portion of the visual backlight perceptible to the user lies at the edge of the user's FOV. A hardware processor and A system equipped with these features. (Item 2) The display system comprises a first light field display for the user's first eye and a second light field display for the user's second eye, and to render the visual representation of the backlight, the hardware processor, The first visual representation of the backlight is rendered by the first light field display onto the first edge of the first FOV associated with the first eye, The second visual representation of the backlight is rendered by the second light field display onto the second edge of the second FOV associated with the second eye. The system described in item 1, which is programmed to perform the following actions. (Item 3) The system described in item 2, wherein the first representation of the backlight and the second representation of the backlight are rendered separately for each eye to match the user's peripheral vision. (Item 4) The system described in item 1, wherein the context information includes information associated with the characteristics of the user, the 3D environment, or the interactable object. (Item 5) The system according to item 4, wherein the information associated with the 3D environment includes the lighting conditions of the user's environment, and the placement of the backlight is determined by simulating the optical effects of the interactable object under the lighting conditions. (Item 6) The aforementioned hardware processor further, To detect changes in the user's posture, Based on the change in the user's posture, the updated location of the interactable object in the 3D environment is determined. The visual representation of the backlight is updated based on the updated location of the interactable object. A system described in any one of items 1-5, which is programmed to perform the following actions. (Item 7) The system according to item 6, in response to the determination that the updated location is within the user's FOV, the hardware processor is programmed to determine the placement of the backlight by collinearly aligning the inner edge of the backlight with at least one of the user's eyes. (Item 8) The visual representation of the backlight is a system according to any one of items 1-5, comprising at least one of the following: position, shape, color, size, or brightness. (Item 9) The system according to item 8, wherein the size of the backlight indicates at least one of the proximity or urgency of the interactable object. (Item 10) The system according to any one of items 1-5, wherein the user's posture includes at least one of the following: head posture or direction of gaze. (Item 11) A method for providing indication of an interactable object in a user's three-dimensional (3D) environment, the method being: A wearable device comprising a display system configured to present a three-dimensional (3D) view to a user and enable user interaction with objects within the user's eye-moving field of view (FOR), wherein the FOR includes a portion of the environment surrounding the user that is perceptible to the user via the display system; a sensor configured to obtain data associated with the user's posture; and a hardware processor communicating with the sensor and the display system, under the control of the wearable device, Determining, at least in part, the user's field of view (FOV) based on the user's posture, wherein the FOV includes a portion of the FOR that is perceptible to the user at a given time via the display system. Identifying interactable objects located outside the user's FOV, Accessing context information associated with the aforementioned interactable object, Based on the aforementioned contextual information, the visual representation of the backlight is determined, Rendering the visual representation of the backlight such that at least a portion of the visual backlight perceptible to the user lies at the edge of the user's FOV. Methods that include... (Item 12) Rendering the first visual representation of the backlight onto the first edge of the first FOV associated with the user's first eye, Rendering the second visual representation of the backlight to a second edge of a second FOV associated with the second eye of the user The method according to item 11, further comprising. (Item 13) The method according to item 12, wherein the first representation of the backlight and the second representation of the backlight are rendered separately for each eye to match the peripheral vision of the user. (Item 14) The method according to item 11, wherein the context information includes information associated with the user, the 3D environment, or the characteristics of the interactive object. (Item 15) The method according to item 14, wherein the information associated with the 3D environment includes the light conditions of the user's environment, and the visual representation of the backlight is determined by simulating the optical effects of the interactive object under the light conditions. (Item 16) Detecting a change in the user's posture Determining an updated location of the interactive object in the 3D environment based on the change in the user's posture Updating the visual representation of the backlight based on the updated location of the interactive object The method according to item 15, further comprising. (Item 17) Aligning the inner edge of the backlight and at least one eye of the user collinearly in response to determining that the updated location is within the FOV of the user, the method according to item 16. (Item 18) The method according to any one of items 16-17, wherein the visual representation of the backlight includes at least one of position, shape, color, size, or brightness. (Item 19) The method according to item 18, wherein the size of the backlight indicates at least one of the proximity or urgency of the interactive object. (Item 20) The user's posture is as described in any one of items 11-17, including at least one of the following: head posture or direction of gaze. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 illustrates an example of a mixed reality scenario involving a virtual reality object and a physical object visible to a person. [Figure 2] Figure 2 schematically illustrates an example of a wearable system. [Figure 3] Figure 3 schematically illustrates aspects of an approach to simulating a 3D image using multiple depth planes. [Figure 4] Figure 4 schematically illustrates an example of a waveguide stack for outputting image information to the user. [Figure 5] Figure 5 shows an exemplary output beam that can be produced by a waveguide. [Figure 6] Figure 6 is a schematic diagram showing an optical system that includes a waveguide apparatus, an optical coupler subsystem for optically coupling light to or from the waveguide apparatus, and a control subsystem used in the generation of a multifocal stereoscopic display, image, or light field. [Figure 7] Figure 7 is a block diagram of an embodiment of the wearable system. [Figure 8] Figure 8 is a process flow diagram of an example of how to render virtual content in relation to recognized objects. [Figure 9] Figure 9 is a block diagram of another embodiment of the wearable system. [Figure 10] Figure 10 is a process flow diagram of an embodiment of a method for determining user input to a wearable system. [Figure 11] Figure 11 is a process flow diagram of an example of a method for interacting with a virtual user interface. [Figure 12]Figure 12 schematically illustrates examples of virtual objects in the field of view (FOV) and virtual objects in the eye-moving field of view (FOR). [Figure 13] Figure 13 schematically illustrates an example of how the user is notified of objects within the FOR directory. [Figure 14A] Figure 14A schematically illustrates a perspective view of the visual backlight above the edge of the field of view (FOV). [Figure 14B] Figure 14B schematically illustrates an example of making the corresponding backlight for an object within the user's FOV invisible. [Figure 15A] Figures 15A and 15B schematically illustrate examples of user experiences with visual backlighting. [Figure 15B] Figures 15A and 15B schematically illustrate examples of user experiences with visual backlighting. [Figure 16] Figure 16 illustrates an exemplary process for rendering a visual representation of visual halo. [Figure 17] Figure 17 illustrates an exemplary process for determining the visual representation of visual halo based on contextual information. [Modes for carrying out the invention]

[0008] Throughout the drawings, reference numbers may be reused to indicate correspondences between referenced elements. The drawings are provided to illustrate exemplary embodiments described herein and are not intended to limit the scope of the disclosure. In addition, the figures in this disclosure are for illustrative purposes only and are not to exact scale. The figures show the FOV as a rectangle, but the present presentation of FOV is not intended to limit it. The two-dimensional representation of FOV can be any shape, e.g., circular, oval, triangular, polygonal, rounded square, combination, or equivalent.

[0009] overview Wearable systems can be configured to display augmented or virtual reality content. Therefore, the user's visual computing experience can extend to the 3D environment surrounding the user. However, the user's field of view (FOV) perceived through the wearable system (also referred to as the user's FOV) may be smaller than the natural FOV of the human eye or smaller than the entire environment surrounding the user. Thus, within the user's environment, there may be physical or virtual objects that are initially outside the user's FOV but can subsequently move into the user's FOV, or become perceptible if the user's posture changes (which would change the user's FOV). For example, in a game situation, a user might try to find a robot avatar. If the robot is slightly outside the user's current FOV, the user will not receive a cue from the wearable system indicating the robot's proximity. If the user slightly moves their head, the robot may suddenly enter the user's FOV, which could surprise the user. Furthermore, if the user's field of view (FOV) through the wearable system is relatively small, it can be proven that it is difficult for the user to find the robot unless they directly turn their head or gaze towards the robot.

[0010] To enhance the user's visual experience, a wearable system may inform the user about objects outside the user's field of view (FOV). For example, a wearable system may render a visual representation of a visual halo for corresponding objects outside the user's current FOV. The visual representation of the halo can be used to indicate an object, the user's environment, or contextual information associated with the user. For example, a brighter or larger halo may indicate that an object is closer to the FOV, while a darker or smaller halo may indicate that an object is further away from the FOV. Similarly, the color of the halo may be used to indicate the type of object. For example, an enemy avatar (in a virtual game) may be associated with a red halo, while an ally avatar (in a virtual game) may be associated with a green halo, and system notifications may be associated with a blue halo. Part of the visual halo may be positioned on the edge of the user's FOV. The size, shape, or position of the halo may change as the user's FOV changes or as the object moves. Therefore, the visual representation of backlight, or changes in the visual representation of backlight, can thereby provide the user with a useful cue about nearby objects that are currently outside the user's FOV. Examples of 3D displays for wearable systems

[0011] A wearable system (also referred to herein as an augmented reality (AR) system) can be configured to present a user with 2D or 3D virtual images. The images may be still images, video frames, or videos in combination or equivalent. A wearable system may include wearable devices that can present a VR, AR, or MR environment, either alone or in combination, for user interaction. A wearable device may be a head-mounted device (HMD), which is used synonymously with an AR device (ARD).

[0012] Figure 1 illustrates an example of a mixed reality scenario involving a virtual reality object and a physical object that are visible to a person. In Figure 1, MR scene 100 is depicted, and the user of the MR technology sees a real-world park-like setting 110 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of the MR technology also perceives "seeing" a robotic figure 130 standing on the real-world platform 120 and a flying cartoonish avatar character 140 that appears to be a personification of a bumblebee, although these elements do not exist in the real world.

[0013] It may be desirable for a 3D display to generate a distance-accommodative response corresponding to the virtual depth of each point within the display's field of view, in order to produce a true sense of depth, more specifically, a simulated sense of surface depth. If the distance-accommodative response for a display point does not correspond to the virtual depth of that point, as determined by the binocular depth cues for convergence and stereopsis, the human eye may experience distance-accommodative collision, which can result in unstable imaging, harmful eye strain, headaches, and, in the absence of distance-accommodative information, a near-complete lack of surface depth.

[0014] VR, AR, and MR experiences can be provided by a display system having a display that provides the viewer with images corresponding to multiple depth planes. The images may differ for each depth plane (e.g., providing slightly different presentations of scenes or objects) and may be individually focused by the viewer's eyes, thereby helping to provide the user with depth cues based on the eye's accommodation required to focus on different image features relating to scenes located on different depth planes, or based on observing different image features on different depth planes that are out of focus. As discussed elsewhere herein, such depth cues provide a reliable perception of depth.

[0015] Figure 2 illustrates an embodiment of the wearable system 200. The wearable system 200 includes a display 220 and various mechanical and electronic modules and systems to support the functions of the display 220. The display 220 may be coupled to a frame 230, which is wearable by a user, wearer, or viewer 210. The display 220 can be positioned in front of the user 210's eyes. The display 220 can present AR / VR / MR content to the user. The display 220 can present a head-mounted display (HMD) that is worn on the user's head. In some embodiments, a speaker 240 is coupled to the frame 230 and positioned adjacent to the user's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / shapeable acoustic control).

[0016] The wearable system 200 may include an outward-facing imaging system 464 (shown in Figure 4) that observes the world within the user's surrounding environment. The wearable system 200 may also include an inward-facing imaging system 462 (shown in Figure 4) that can track the user's eye movements. The inward-facing imaging system can track the movement of one eye or both eyes. The inward-facing imaging system 462 may be mounted on the frame 230 and may communicate with a processing module 260 or 270 that processes the image information acquired by the inward-facing imaging system and can determine, for example, the pupil diameter or orientation of the user 210's eyes, eye movements, or eye posture.

[0017] As an example, the wearable system 200 can acquire images of the user's posture using an outward-facing imaging system 464 or an inward-facing imaging system 462. The images may be still images, video frames or videos, a combination thereof, or equivalent.

[0018] The display 220 is operably coupled to a local data processing module 260 (250), which can be mounted in various configurations, such as being fixedly attached to the frame 230 by wired or wireless connections, fixed to a helmet or hat worn by the user, built into headphones, or otherwise detachably attached to the user 210 (for example, in a backpack configuration or a belt-connected configuration).

[0019] The local processing and data module 260 may include a hardware processor and digital memory such as non-volatile memory (e.g., flash memory), both of which may be used to assist in data processing, caching, and storage. The data may include (a) data captured from sensors (e.g., cameras in an inward-facing imaging system and / or an outward-facing imaging system), microphones, inertial measuring units (IMUs), accelerometers, compasses, global positioning system (GPS) units, wireless devices, or gyroscopes (e.g., operably coupled to frame 230 or otherwise attached to user 210), or (b) data acquired or processed using the remote processing module 270 or remote data repository 280 for transmission to the display 220 after such processing or reading. The local processing and data modules 260 may be operably coupled to the remote processing module 270 or the remote data repository 280 by communication links 262 or 264, such as via wired or wireless communication links, so that these remote modules are available as resources to the local processing and data modules 260. In addition, the remote processing module 280 and the remote data repository 280 may be operably coupled to each other.

[0020] In some embodiments, the remote processing module 270 may comprise one or more processors configured to analyze and process data and / or image information. In some embodiments, the remote data repository 280 may comprise a digital data storage facility, which may be available through the internet or other networking configurations in a “cloud” resource configuration. In some embodiments, all data is stored, and all calculations are performed in the local processing and data modules, enabling fully autonomous use from the remote modules.

[0021] The human visual system is complex and struggles to provide a realistic perception of depth. While not limited by theory, it is believed that an object viewer may perceive an object as three-dimensional due to a combination of vergence and accommodation. The vergence and divergence of two eyes relative to each other (i.e., the rolling of the pupils toward or away from each other to converge and fix the gaze on an object) is closely related to the focusing (or "accommodation") of the eye's lens. Under normal conditions, changing the focus of the eye's lens, or accommodation, to shift focus from one object to another at a different distance, will automatically produce a corresponding change in vergence and divergence at the same distance, under a relationship known as the "accommodation-vergence-divergence reflex." Similarly, a change in vergence and divergence will, under normal conditions, induce a corresponding change in accommodation. A display system that provides a better match between distance accommodation and convergence / divergence motion can create a more realistic and comfortable simulation of three-dimensional images.

[0022] Figure 3 illustrates aspects of an approach to simulating a three-dimensional image using multiple depth planes. Referring to Figure 3, objects at various distances from eyes 302 and 304 on the z-axis are accommodated by eyes 302 and 304 so that those objects are in focus. Eyes 302 and 304 take on specific accommodated states, focusing objects at different distances along the z-axis. As a result, a specific accommodated state can be said to be associated with one of the specific depth planes 306, having an associated focal length so that an object or part of an object in a particular depth plane is in focus when the eye is accommodated with respect to that depth plane. In some embodiments, the three-dimensional image may be simulated by providing different presentations of the image to each of eyes 302 and 304, and by providing different presentations of the image corresponding to each of the depth planes. For the sake of clarity in the illustration, although shown as separate, it should be understood that the fields of view of eyes 302 and 304 may overlap, for example, as the distance along the z-axis increases. Furthermore, although shown as flat for the sake of illustration, it should be understood that the appearance of the depth plane may be curved in physical space so that all features within the depth plane are in focus with the eye in a particular distance-accommodated state. While not limited by theory, it is thought that the human eye can typically interpret a finite number of depth planes to provide depth perception. Consequently, a highly realistic simulation of perceived depth can be achieved by providing the eye with different representations of images corresponding to each of these limited number of depth planes. Waveguide stack assembly

[0023] Figure 4 illustrates an embodiment of a waveguide stack for outputting image information to a user. The wearable system 400 includes a waveguide stack or stacked waveguide assembly 480, which may be used to provide three-dimensional perception to the eye / brain using a plurality of waveguides 432b, 434b, 436b, 438b, 400b. In some embodiments, the wearable system 400 may correspond to the wearable system 200 of Figure 2, and Figure 4 shows some parts of the wearable system 200 in more detail. For example, in some embodiments, the waveguide assembly 480 may be integrated into the display 220 of Figure 2.

[0024] Continuing with Figure 4, the waveguide assembly 480 may also include several features 458, 456, 454, and 452 between the waveguides. In some embodiments, features 458, 456, 454, and 452 may be lenses. In other embodiments, features 458, 456, 454, and 452 may not be lenses. Rather, they may simply be spacers (e.g., cladding layers or structures for forming air gaps).

[0025] Waveguides 432b, 434b, 436b, 438b, 440b or multiple lenses 458, 456, 454, 452 may be configured to transmit image information to the eye with varying levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a specific depth plane and configured to output image information corresponding to that depth plane. Image input devices 420, 422, 424, 426, 428 may be used to input image information into waveguides 440b, 438b, 436b, 434b, 432b, each of which may be configured to disperse incident light across each individual waveguide for output toward the eye 410. Light exits from the output surfaces of image input devices 420, 422, 424, 426, and 428 and is fed into the corresponding input edges of waveguides 440b, 438b, 436b, 434b, and 432b. In some embodiments, a single beam of light (e.g., a collimated beam) may be fed into each waveguide and output an entire field of cloned collimated beams, directed toward the eye 410 at a specific angle (and divergence) corresponding to a depth plane associated with a particular waveguide.

[0026] In some embodiments, the image input devices 420, 422, 424, 426, and 428 are discrete displays that each generate image information for input into their respective waveguides 440b, 438b, 436b, 434b, and 432b. In some other embodiments, the image input devices 420, 422, 424, 426, and 428 are output terminals of a single multiplexed display that can pipe image information to each of the image input devices 420, 422, 424, 426, and 428 via, for example, one or more optical conduits (such as optical fiber cables).

[0027] The controller 460 controls the operation of the stacked waveguide assembly 480 and the image input devices 420, 422, 424, 426, and 428. The controller 460 includes programming (e.g., instructions in a non-transient computer-readable medium) to coordinate the timing and delivery of image information to the waveguides 440b, 438b, 436b, 434b, and 432b. In some embodiments, the controller 460 may be a single, integrated device or a distributed system connected by wired or wireless communication channels. In some embodiments, the controller 460 may be part of a processing module 260 or 270 (illustrated in Figure 2).

[0028] Waveguides 440b, 438b, 436b, 434b, and 432b may be configured to propagate light within each individual waveguide by total internal reflection (TIR). Waveguides 440b, 438b, 436b, 434b, and 432b may each be planar or have another shape (e.g., curved) with a main upper surface and a main lower surface and a rim extending between the main upper surface and the main lower surface. In the illustrated configuration, waveguides 440b, 438b, 436b, 434b, and 432b may each include light extraction optical elements 440a, 438a, 436a, 434a, and 432a, respectively, configured to extract light from the waveguides by redirecting the light, propagating it within each individual waveguide, and outputting image information from the waveguides to the eye 410. The extracted light may also be referred to as externally coupled light, and the light extraction optical elements may also be referred to as externally coupled optical elements. The beam of extracted light is output by the waveguide to the location where the light propagating within the waveguide strikes the light redirection element. The light extraction optical elements (440a, 438a, 436a, 434a, 432a) may be, for example, reflective or diffracting optical features. For the sake of clarity and to facilitate the explanation, they are shown positioned on the bottom main surface of waveguides 440b, 438b, 436b, 434b, 432b, but in some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be positioned on the top main surface or the bottom main surface, or they may be positioned directly within the volume of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, and 432a may be mounted on a transparent substrate and formed within a layer of material that forms the waveguides 440b, 438b, 436b, 434b, and 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, and 432b may be monolithic material pieces, and the light extraction optical elements 440a, 438a, 436a, 434a, and 432a may be formed on and / or inside the material piece.

[0029] Continuing with Figure 4, as discussed herein, each waveguide 440b, 438b, 436b, 434b, 432b is configured to emit light and form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 as it is introduced into such waveguide 432b. The collimated light may represent the optical infinity focal plane. The next waveguide 434b may be configured to emit collimated light that passes through a first lens 452 (e.g., a negative lens) before reaching the eye 410. The first lens 452 may generate a slight convex wavefront curvature so that the eye / brain interprets the light emanating from the next waveguide 434b as emanating from a first focal plane closer inward from optical infinity toward the eye 410. Similarly, the third upper waveguide 436b passes its output light through both the first lens 452 and the second lens 454 before reaching the eye 410. The combined refractive power of the first and second lenses 452 and 454 may be configured to generate another increment of wavefront curvature so that the eye / brain interprets the light originating from the third upper waveguide 436b as originating from a second focal plane that is even closer inward toward the person from optical infinity than the light from the next waveguide 434b.

[0030] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured to use the highest waveguide 440b in the stack to transmit its output through all the lenses between it and the eye for concentrated focal power representing the focal plane closest to the person. When viewing / interpreting light originating from the other side world 470 of the stacked waveguide assembly 480, a compensating lens layer 430 may be positioned on top of the stack to compensate for the stack of lenses 458, 456, 454, 452 and to compensate for the concentrated force of the lower lens stacks 458, 456, 454, 452. Such a configuration provides the same number of perceived focal planes as there are available waveguide / lens pairs. Both the light-extracting optical elements of the waveguides and the focusing sides of the lenses may be static (e.g., not dynamic or electroactive). In some alternative embodiments, either or both may be dynamic using electroactive features.

[0031] Continuing with Figure 4, the light extraction optical elements 440a, 438a, 436a, 434a, and 432a may be configured to redirect light from their respective waveguides and output the light using appropriate amounts of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides having different associated depth planes may have different configurations of light extraction optical elements that output light using different amounts of divergence corresponding to the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, and 432a may be stereoscopic or surface features that can be configured to output light at a particular angle. For example, the light extraction optical elements 440a, 438a, 436a, 434a, and 432a may be volume holograms, surface holograms, and / or diffraction gratings. Optical elements for light extraction, such as diffraction gratings, are described in U.S. Patent Publication No. 2015 / 0178939, published on June 25, 2015 (which is incorporated herein by reference as a whole).

[0032] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, and 432a are diffraction features, i.e., “diffractive optical elements” (also referred to herein as “DOEs”), that form a diffraction pattern. Preferably, the DOEs have relatively low diffraction efficiency such that only a portion of the beam light is deflected toward the eye 410 at each intersection of the DOEs, while the remainder continues to travel through the waveguide via total internal reflection. The light carrying the image information is thus split into several associated emitted beams that exit the waveguide at multiple locations, the result of which can be a very uniform pattern of emitted beams toward the eye 304 with respect to this particular collimated beam bouncing within the waveguide.

[0033] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer-dispersed liquid crystal in which microdroplets contain a diffraction pattern in a host medium, and the refractive index of the microdroplets can be switched to a refractive index that substantially matches that of the host material (in which case the pattern does not significantly diffract the incident light), or the refractive index of the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts the incident light).

[0034] In some embodiments, the number and distribution of depth planes or depth of field may vary dynamically based on the pupil size or orientation of the viewer's eye. The depth of field may change inversely with the viewer's pupil size. As a result, as the pupil size of the viewer's eye decreases, the depth of field increases so that one plane that is indistinguishable because its location is beyond the eye's depth of focus becomes discernible and appears more in focus with the decrease in pupil size and the corresponding increase in depth of field. Similarly, the number of spaced-out depth planes used to present different images to the viewer may decrease with the decreased pupil size. For example, it may not be possible for a viewer to clearly perceive the details of both the first and second depth planes at one pupil size without adjusting the eye's accommodation from one depth plane to the other. However, these two depth planes may simultaneously be sufficient to focus on the user at a different pupil size without changing accommodation.

[0035] In some embodiments, the display system may vary the number of waveguides receiving image information based on a determination of pupil size or orientation, or in response to the reception of an electrical signal indicating a particular pupil size or orientation. For example, if the user's eye is unable to distinguish between two depth planes associated with two waveguides, the controller 460 may be configured or programmed to stop providing image information to one of these waveguides. Advantageously, this can reduce the processing load on the system and thereby increase the system's responsiveness. In embodiments where the DOE for a waveguide is switchable between an on and off state, the DOE may be switched off when the waveguide receives image information.

[0036] In some embodiments, it may be desirable to satisfy the condition that the emitted beam has a diameter less than the diameter of the viewer's eye. However, satisfying this condition may be difficult in light of the variability of the viewer's pupil size. In some embodiments, this condition is satisfied over a wide range of pupil sizes by varying the size of the emitted beam in response to the determination of the viewer's pupil size. For example, as the pupil size decreases, the size of the emitted beam may also decrease. In some embodiments, the size of the emitted beam may be varied using a variable aperture.

[0037] The wearable system 400 may include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the field of view (FOV) of the world camera, and the imaging system 464 may sometimes be referred to as the FOV camera. The entire area available for viewing or imaging by the viewer may be referred to as the eye-moving field of view (FOR). FOR may include a solid angle of 4π steradians surrounding the wearable system 400 so that the wearer moves their body, head, or eyes to perceive substantially any direction in space. In other circumstances, the wearer's movement may be more restrained, and accordingly, the wearer's FOR may tangent to a smaller solid angle. Images obtained from the outward-facing imaging system 464 can be used, for example, to track gestures made by the user (e.g., hand or finger gestures) and to detect objects in the world 470 in front of the user.

[0038] The wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera) that observes user movements such as eye and face movements. The inward-facing imaging system 466 may be used to capture images of the eyes 410 and to determine the size or orientation of the pupils of the eyes 304. The inward-facing imaging system 466 may be used to determine the direction the user is looking (e.g., eye posture) or to obtain images for the user's biometric identification (e.g., via iris recognition). In some embodiments, at least one camera may be used for each eye, independently, to determine the pupil size or eye posture of each eye separately, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye. In some other embodiments, the pupil diameter or orientation of only one eye 410 (e.g., using only one camera per pair of eyes) is determined and assumed to be similar with respect to both of the user's eyes. Images obtained by the inward-facing imaging system 466 may be used by the wearable system 400 to determine the user's eye posture or mood, or to determine the audio or visual content to be presented to the user. The wearable system 400 may also use sensors such as an IMU, accelerometer, and gyroscope to determine the head posture (e.g., head position or head orientation).

[0039] The wearable system 400 may include a user input device 466 that allows the user to input commands into a controller 460 and interact with the wearable system 400. For example, the user input device 466 may include a trackpad, touchscreen, joystick, multi-degree-of-freedom (DOF) controller, capacitive sensing device, game controller, keyboard, mouse, directional pad (D-pad), wand, tactile device, totem (e.g., functioning as a virtual user input device), etc. A multi-DOF controller may sense user input in translation (e.g., left / right, forward / backward, or up / down) or rotation (e.g., yaw, pitch, or roll), which can be some or all of the controller's possible movements. A multi-DOF controller that supports translation may be referred to as 3DOF, while a multi-DOF controller that supports both translation and rotation may be referred to as 6DOF. In some cases, the user may use a finger (e.g., thumb) to press or swipe over a touch sensor input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held in the user's hand while using the wearable system 400. The user input device 466 can be connected to the wearable system 400 via wired or wireless communication.

[0040] Figure 5 shows an embodiment of an outgoing beam output by a waveguide. Although one waveguide is shown, other waveguides within the waveguide assembly 480 may function similarly, and it should be understood that the waveguide assembly 480 includes multiple waveguides. Light 520 is injected into waveguide 432b at the input edge 432c of waveguide 432b and propagates through waveguide 432b by TIR. At the point where light 520 collides with DOE 432a, a portion of the light exits the waveguide as an outgoing beam 510. The outgoing beams 510 are shown as substantially parallel, but they may also be redirected to propagate towards the eye 410 at a certain angle depending on the depth plane associated with waveguide 432b (e.g., forming a divergent outgoing beam). It should be understood that a nearly parallel emitted beam may represent a waveguide with an optical element that externally couples the light and forms an image that appears to be set on the depth plane at long distances from the eye 410 (e.g., optical infinity). Other waveguides or other sets of optical elements may output a more divergent emitted beam pattern that requires the eye 410 to adjust to a closer distance and focus on the retina, and which would be interpreted by the brain as light from a distance closer to the eye 410 than optical infinity.

[0041] Figure 6 is a schematic diagram showing an optical system that includes a waveguide apparatus, an optical coupler subsystem for optically coupling light to or from the waveguide apparatus, and a control subsystem used in the generation of a multifocal stereoscopic display, image, or light field. The optical system may include a waveguide apparatus, an optical coupler subsystem for optically coupling light to or from the waveguide apparatus, and a control subsystem. The optical system may be used to generate a multifocal stereoscopic display, image, or light field. The optical system may include one or more primary plane waveguides 632a (only one is shown in Figure 6) and one or more DOEs 632b associated with at least one of the primary waveguides 632a. The plane waveguides 632b may be analogous to the waveguides 432b, 434b, 436b, 438b, and 440b discussed with reference to Figure 4. The optical system may employ a dispersed waveguide apparatus to relay light along a first axis (vertical or Y-axis in the diagram of Figure 6) and expand the effective exit pupil of light along the first axis (e.g., Y-axis). The dispersed waveguide apparatus may include, for example, a dispersed plane waveguide 622b and at least one DOE 622a (illustrated by a double dashed line) associated with the dispersed plane waveguide 622b. The dispersed plane waveguide 622b may be similar to or the same as a primary plane waveguide 632b having a different orientation in at least some respects. Similarly, at least one DOE 622a may be similar to or the same as a DOE 632a in at least some respects. For example, the dispersed plane waveguide 622b or DOE 622a may be made of the same material as the primary plane waveguide 632b or DOE 632a, respectively. An embodiment of the optical display system 600 shown in Figure 6 can be integrated into the wearable system 200 shown in Figure 2.

[0042] The relayed and dilated light can be optically coupled from the dispersed waveguide apparatus into one or more primary plane waveguides 632b. The primary plane waveguides 632b can relay light along a second axis (e.g., horizontal or X-axis in the diagram of Figure 6) perpendicular to the first axis. It should be noted that the second axis can be a non-orthogonal axis to the first axis. The primary plane waveguides 632b dilate the effective exit pupil of the light along their second axis (e.g., X-axis). For example, a dispersed plane waveguide 622b can relay and dilate light along the vertical or Y-axis, and can pass its light through a primary plane waveguide 632b that can relay and dilate light along the horizontal or X-axis.

[0043] The optical system may include one or more colored light sources (e.g., red, green, and blue laser light) 610 that can be optically coupled into the proximal end of a single-mode optical fiber 640. The distal end of the optical fiber 640 may be screwed or received through a hollow tube 642 made of piezoelectric material. The distal end protrudes from the tube 642 as an unfixed, flexible cantilever 644. The piezoelectric tube 642 can be associated with four quadrant electrodes (not shown). The electrodes may be plated, for example, on the outside, outer surface, outer periphery, or diameter of the tube 642. A core electrode (not shown) may also be located in the core, center, inner periphery, or inner diameter of the tube 642.

[0044] For example, a drive electronic device 650, electrically coupled via wire 660, drives a pair of opposing electrodes to independently bend the piezoelectric tube 642 along two axes. The protruding distal tip of the optical fiber 644 has a mechanical resonance mode. The resonance frequency may depend on the diameter, length, and material properties of the optical fiber 644. By vibrating the piezoelectric tube 642 near the first mechanical resonance mode of the fiber cantilever 644, the fiber cantilever 644 can be vibrated and swept through a large deflection.

[0045] By stimulating resonant vibrations in two axes, the tip of the fiber cantilever 644 is scanned in two axes within an area that fills a two-dimensional (2-D) scan. By modulating the intensity of the light source 610 in synchronization with the scanning of the fiber cantilever 644, the light emitted from the fiber cantilever 644 can form an image. A description of such a setup is provided in U.S. Patent Publication No. 2014 / 0003762, which is incorporated herein by reference in its entirety.

[0046] Components of the optical coupler subsystem can collimate light emitted from the scanning fiber cantilever 644. The collimated light can be reflected by the mirrored surface 648 into a narrow-dispersion planar waveguide 622b containing at least one diffractive optical element (DOE) 622a. The collimated light propagates perpendicularly along the dispersive planar waveguide 622b (with respect to the diagram in Figure 6) by TIR, thereby repeatedly intersecting with the DOE 622a. The DOE 622a preferably has a low diffraction efficiency. This allows a portion of the light (e.g., 10%) to be diffracted toward the edge of the larger primary planar waveguide 632b at each intersection with the DOE 622a, while a portion of the light can be continued along its original trajectory along the length of the dispersive planar waveguide 622b via TIR.

[0047] At each intersection with DOE622a, additional light can be diffracted toward the entrance of the primary waveguide 632b. By splitting the incident light into multiple external coupling sets, the exit pupil of the light can be vertically extended by DOE4 in the dispersed plane waveguide 622b. This vertically extended light, externally coupled from the dispersed plane waveguide 622b, can enter the edge of the primary plane waveguide 632b.

[0048] Light entering the primary waveguide 632b can propagate horizontally along the primary waveguide 632b (relative to the diagram in Figure 6) via TIR. As the light intersects with DOE 632a at multiple points, it propagates horizontally along at least a portion of the length of the primary waveguide 632b via TIR. DOE 632a may be advantageously designed or configured to have a phase profile which is the sum of linear and radially symmetric diffraction patterns, and to produce both deflection and focusing of light. DOE 632a may advantageously have a low diffraction efficiency (e.g., 10%) such that only a portion of the beam of light is deflected towards the viewer's eye at each intersection of DOE 632a, while the rest of the light continues to propagate through the primary waveguide 632b via TIR.

[0049] At each intersection point between the propagating light and the DOE632a, a portion of the light is diffracted toward the adjacent surface of the primary waveguide 632b, allowing the light to escape from the TIR and be emitted from the surface of the primary waveguide 632b. In some embodiments, the radially symmetric diffraction pattern of the DOE632a also imparts a certain focal level to the diffracted light, shaping (e.g., imparting curvature) the wavefronts of the individual beams and steering the beams to an angle that matches the designed focal level.

[0050] Therefore, these different paths can be used to couple light outside the primary plane waveguide 632b by resulting in different filling patterns in the DOE 632a at different angles, focal levels, and / or in the exit pupil. Different filling patterns in the exit pupil can be advantageously used to generate a light field display with multiple depth planes. Each layer in the waveguide assembly or a set of layers in a stack (e.g., three layers) may be employed to generate individual colors (e.g., red, blue, and green). Thus, for example, a first set of three adjacent layers may be employed to generate red, blue, and green light at a first depth of focus, respectively. A second set of three adjacent layers may be employed to generate red, blue, and green light at a second depth of focus, respectively. Multiple sets may be employed to generate a full 3D or 4D color image light field with various depths of focus. Other components of a wearable system

[0051] In many implementations, the wearable system may include other components in addition to, or as alternatives to, the components of the wearable system described above. The wearable system may include, for example, one or more tactile devices or components. The tactile devices or components may be operable to provide a sense of touch to the user. For example, the tactile devices or components may provide a sense of pressure and / or texture when touching virtual content (e.g., virtual objects, virtual tools, other virtual structures). The tactile sensation may replicate the feeling of a physical object represented by the virtual object, or the feeling of an imaginary object or character represented by the virtual content (e.g., a dragon). In some implementations, the tactile devices or components may be worn by the user (e.g., user-wearable gloves). In some implementations, the tactile devices or components may be held by the user.

[0052] A wearable system may include, for example, one or more physical objects that are operable by the user and enable input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as pieces of metal or plastic, walls, or the surface of a table. In some implementations, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, a totem may simply provide a physical surface, and the wearable system may render a user interface so that it appears to the user as being on one or more surfaces of the totem. For example, the wearable system may render images of a computer keyboard and trackpad so that they appear to reside on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and virtual trackpad so that they appear to be on the surface of a thin rectangular aluminum plate that acts as a totem. The rectangular plate itself has no physical keys or trackpads or sensors. However, the wearable system may detect user operation or interaction or touch using the rectangular plate as a selection or input made via a virtual keyboard or virtual trackpad. The user input device 466 (shown in Figure 4) may be an embodiment of the totem, which may include a trackpad, touchpad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. The user may use the totem alone or in combination with posture to interact with the wearable system and / or other users.

[0053] Examples of wearable devices, HMDs, and display systems and usable tactile devices and totems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777 (which is incorporated herein in whole by reference). Exemplary wearable systems, environments, and interfaces

[0054] Wearable systems may employ various mapping-related techniques to achieve high depth of field within the rendered light field. When mapping a virtual world, it is advantageous to capture all features and points in the real world and accurately depict virtual objects in relation to the real world. To achieve this objective, FOV images captured by the user of the wearable system can be added to the world model by including new images that convey information about various points and features of the real world. For example, a wearable system can collect a set of map points (2D points or 3D points, etc.), find new map points, and render a more accurate version of the world model. The world model of the first user can be communicated to a second user (e.g., via a network such as a cloud network) so that the second user can experience the world surrounding the first user.

[0055] Figure 7 is a block diagram of an embodiment of the MR environment 700. The MR environment 700 may be configured to receive inputs (e.g., visual input 702 from the user's wearable system, steady input 704 such as an indoor camera, sensory input 706 from various sensors, gestures, totems, eye tracking, user input, etc. from a user input device 466) from one or more user wearable systems (e.g., wearable system 200 or display system 220) or steady indoor systems (e.g., indoor camera, etc.). The wearable system can use various sensors (e.g., accelerometer, gyroscope, temperature sensor, motion sensor, depth sensor, GPS sensor, inward-facing imaging system, outward-facing imaging system, etc.) to determine the location of the user's environment and various other attributes. This information may be further supplemented with information from a steady camera in the room, which may provide images from different viewpoints or various cues. Image data acquired by cameras (indoor camera and / or outward-facing imaging system camera, etc.) may be reduced to a set of mapping points.

[0056] One or more object recognition devices 708 can crawl through received data (e.g., a collection of points), recognize or map the points, tag images, and link semantic information to objects using a map database 710. The map database 710 may contain various points and their corresponding objects collected over time. The various devices and the map database can be interconnected via a network (e.g., LAN, WAN, etc.) and can access the cloud.

[0057] Based on this information and the set of points in the map database, object recognition devices 708a-708n may recognize objects in the environment. For example, an object recognition device can recognize faces, people, windows, walls, user input devices, televisions, other objects in the user's environment, etc. One or more object recognition devices may be specialized for objects with certain characteristics. For example, object recognition device 708a may be used to recognize faces, while another object recognition device may be used to recognize totems.

[0058] Object recognition may be performed using various computer vision techniques. For example, a wearable system can analyze images acquired by an outward-facing imaging system 464 (shown in Figure 4) to perform scene reconstruction, event detection, video tracking, object recognition, object pose estimation, learning, indexing, motion estimation, or image restoration. One or more computer vision algorithms may be used to perform these tasks. Non-restrictive examples of computer vision algorithms include scale-invariant feature transformation (SIFT), speed-up robust features (SURF), orientation FAST and rotation BRIEF (ORB), binary robust invariant scalable keypoint (BRISK), fast retinal keypoint (FREAK), Viola-Jones algorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, visual simultaneous localization and mapping (vSLAM) techniques, sequential Bayesian estimators (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, adaptive thresholding (and other thresholding techniques), iterative nearest neighbor (ICP), semi-global matching (SGM), semi-global block matching (SGBM), feature point histograms, and various machine learning algorithms (e.g., support vector machines, k-nearest neighbor algorithm, Naive Bayes, neural networks (including convolutional or deep neural networks), or other supervised / unsupervised models, etc.).

[0059] Object recognition can be performed, in addition to or alternatively, by various machine learning algorithms. Once trained, machine learning algorithms can be stored by the HMD. Some embodiments of machine learning algorithms may include supervised or unsupervised machine learning algorithms, and include regression algorithms (e.g., ordinary least-squares regression), instance-based algorithms (e.g., learning vector quantization), decision tree algorithms (e.g., classification and regression trees), Bayesian algorithms (e.g., Naive Bayes), clustering algorithms (e.g., k-means clustering), association rule learning algorithms (e.g., a priori algorithms), artificial neural network algorithms (e.g., Perceptron), deep learning algorithms (e.g., Deep Boltzmann Machine, i.e., deep neural networks), dimensionality reduction algorithms (e.g., principal component analysis), ensemble algorithms (e.g., Stacked Generalization), and / or other machine learning algorithms. In some embodiments, individual models can be customized for individual datasets. For example, a wearable device can generate or store a base model. The base model is used as a starting point and may generate additional models specific to data types (e.g., a specific user in a telepresence session), datasets (e.g., a set of additional images acquired from a user in a telepresence session), conditional situations, or other modifications. In some embodiments, the wearable HMD can be configured to utilize multiple techniques to generate a model for analyzing aggregated data. Other techniques may include using predefined thresholds or data values.

[0060] Based on the main information and set of points in the map database, object recognition devices 708a-708n may recognize objects, supplement them with semantic information, and give them life. For example, if an object recognition device recognizes that a set of points is a door, the system may combine some semantic information (e.g., the door has a hinge and moves 90 degrees around the hinge). If an object recognition device recognizes that a set of points is a mirror, the system may combine semantic information that the mirror has a reflective surface that can reflect images of objects in the room. Over time, the map database grows as the system (which may reside locally or be accessible via a wireless network) accumulates more data from around the world. Once an object is recognized, the information may be transmitted to one or more wearable systems. For example, the MR environment 700 may contain information about a scene happening in California. The environment 700 may be transmitted to one or more users in New York. Based on data received from the FOV camera and other inputs, the object recognition device and other software components can map points collected from various images and recognize objects, etc., so that the scene can be accurately "passed" to a second user who may be located in a different part of the world. Environment 700 may also use a topology map for location identification purposes.

[0061] Figure 8 is a process flow diagram of an embodiment of Method 800 for rendering virtual content in relation to a recognized object. Method 800 describes a way in which a virtual scene may be represented to a user of a wearable system. The user may be geographically distant from the scene. For example, the user may be in New York but may want to view a scene currently happening in California, or may want to go for a walk with a friend who is in California.

[0062] In block 810, the wearable system may receive input about the user's environment from the user and other users. This may be achieved through various input devices and knowledge already held in a map database. The user's FOV camera, sensors, GPS, eye tracking, etc., transmit information to the system in block 810. In block 820, the system may determine rough points based on this information. These rough points may be used to determine posture data (e.g., head posture, eye posture, body posture, or hand gestures) which can be used to display and understand the orientation and position of various objects around the user. In block 830, object recognition devices 708a-708n may crawl through these collected points and recognize one or more objects using the map database. This information may then be transmitted to the user's individual wearable systems in block 840, and a desired virtual scene may be displayed to the user in block 850 as appropriate. For example, a desired virtual scene (e.g., a user in CA) may be displayed in an appropriate orientation, position, etc., in relation to the user's various objects and other surroundings in New York.

[0063] Figure 9 is a block diagram of another embodiment of a wearable system. In this embodiment, the wearable system 900 includes a map, which may include map data about the world. The map may reside in part locally on the wearable system, or in part in a networked storage location (e.g., within a cloud system) accessible by a wired or wireless network. An attitude process 910 may run on a wearable computing architecture (e.g., a processing module 260 or a controller 460) and utilize data from the map to determine the position and orientation of the wearable computing hardware or the user. The attitude data may be calculated from data collected on the fly as the user experiences the system and operates within its world. The data may include images of objects in a real or virtual environment, data from sensors (generally, such as an inertial measurement unit with accelerometer and gyroscope components), and surface information.

[0064] A rough point representation may be the output of a simultaneous location identification and mapping (SLAM or V-SLAM, referring to configurations where the input is image / visual only) process. The system can be configured to find not only the locations of various components in the world, but also what the world is made of. A posture may be a building block that achieves many goals, including filling in a map and using data from the map.

[0065] In one embodiment, the approximate point location may not be entirely accurate in itself, and further information may be required to generate a multi-focus AR, VR, or MR experience. A dense representation, generally referring to depth map information, may be used, at least partially, to fill this gap. Such information may be calculated from a process referred to as stereoscopic viewing 940, where the depth information is determined using techniques such as triangulation or time-of-flight sensing. Image information and active patterns (such as infrared patterns generated using an active projector) may serve as input to the stereoscopic viewing process 940. A significant amount of depth map information may be fused together, some of which may be summarized using surface representations. For example, mathematically definable surfaces may be an efficient (e.g., compared to large point clouds) and applicable input to other processing devices such as a game engine. Thus, the output of the stereoscopic viewing process (e.g., depth map) 940 may be combined in the fusion process 930. The orientation may also be an input to the fusion process 930, and the output of the fusion 930 becomes an input to fill the map process 920. Subsurfaces may connect with each other in topographic mapping, etc., to form a larger surface, and the map becomes a large-scale hybrid of points and surfaces.

[0066] Various inputs may be used to resolve various aspects in the mixed reality process 960. For example, in the embodiment depicted in Figure 9, game parameters may be inputs for determining that the system's user is playing a monster battle game with one or more monsters in various locations, that the monsters are dead or running away under various conditions (such as when the user shoots the monsters), and for determining walls or other objects and equivalents in various locations. A world map may include information about the locations where such objects exist relative to each other, which is another useful input to mixed reality. Attitudes to the world are also inputs and play an important role for almost any interactive system.

[0067] User control or input is another input to the wearable system 900. As described herein, user input can include visual input, gestures, totems, audio input, sensory input, etc. For example, to move around or play a game, the user may need to command the wearable system 900 about what they want to do. There are various forms of user control that can be utilized, not just moving around in space on one's own. In one embodiment, an object such as a totem (e.g., a user input device) or a toy gun may be held by the user and tracked by the system. The system would preferably be configured to know that the user is holding the item and to understand the type of interaction the user is having with the item (for example, if the totem or object is a gun, the system may be configured to understand not only its location and orientation but also whether the user is clicking a trigger or other sensing button or element, which may be equipped with sensors such as an IMU that can help determine what is happening even when such activity is not within the field of view of any camera).

[0068] Hand gesture tracking or recognition may also provide input information. The wearable system 900 may be configured to track and interpret hand gestures for button presses, left or right, stop, grasp, hold, etc. For example, in one configuration, the user may want to flip through email or a calendar in a non-gaming environment, or "fist bump" with another person or performer. The wearable system 900 may be configured to take advantage of a minimum amount of hand gestures, which may or may not be dynamic. For example, the gestures may be simple static gestures, such as spreading the hand to indicate stop, raising the thumb to indicate OK, lowering the thumb to indicate not OK, or flipping the hand left or right or up or down to indicate a directional command.

[0069] Eye tracking is another input (for example, tracking where the user is looking, controlling display technology, and rendering at a specific depth or range). In one embodiment, the convergence and divergence of the eyes may be determined using triangulation, and then accommodation may be determined using a convergence and divergence / accommodation model developed for that particular person.

[0070] With respect to the camera system, the exemplary wearable system 900 shown in Figure 9 may include three pairs of cameras, namely a relative wide-field-of-view (FOV) or passive SLAM pair of cameras arranged on either side of the user's face, and a different pair of cameras oriented in front of the user to handle the stereoscopic imaging process 940 and to capture hand gestures and totem / object trajectories in front of the user's face. The FOV camera and the pair of cameras for the stereoscopic process 940 may be part of an outward-facing imaging system 464 (shown in Figure 4). The wearable system 900 may also include an eye-tracking camera oriented toward the user's eye (which may be part of an inward-facing imaging system 462 (shown in Figure 4)) to triangulate eye vectors and other information. The wearable system 900 may also include one or more textured light projectors (such as infrared (IR) projectors) to bring textures into the scene.

[0071] Figure 10 is a process flow diagram of an embodiment of Method 1000 for determining user input to a wearable system. In this embodiment, the user may interact with a totem. The user may have multiple totems. For example, the user may have one designated totem for a social media application, another totem for playing a game, etc. In block 1010, the wearable system may detect the movement of the totem. The movement of the totem may be perceived through an outward-facing system or detected through sensors (e.g., tactile gloves, image sensors, hand tracking devices, eye tracking cameras, head posture sensors, etc.).

[0072] Based at least partially on detected gestures, eye postures, head postures, or inputs through the totem, the wearable system detects the position, orientation, and / or movement of the totem (or the user's eyes or head or gestures) relative to a reference frame in block 1020. The reference frame may be a set of map points on which the wearable system translates the movement of the totem (or user) into actions or commands. In block 1030, the user's interaction with the totem is mapped. Based on the mapping of the user interaction to the reference frame 1020, the system determines the user input in block 1040.

[0073] For example, the user may move a totem or physical object back and forth, turn a virtual page, move to the next page, or move from one user interface (UI) display screen to another UI screen. In another embodiment, the user may move their head or eyes to view different real or virtual objects within the user's FOR. If the user gazes at a particular real or virtual object for longer than a threshold time, that real or virtual object may be selected as user input. In some implementations, the convergence and divergence movements of the user's eyes can be tracked, and a near-to-near In various implementations, raycasting techniques may include projecting a narrow beam of light with virtually no width, or projecting a beam of light with substantial width (e.g., a cone or frustum).

[0074] The user interface may be projected by a display system as described herein (e.g., display 220 in Figure 2). Alternatively, it may be displayed using various other techniques, such as one or more projectors. The projectors may project images onto a canvas or a physical object such as a sphere. Interaction with the user interface may be tracked using one or more cameras, either external to or part of the system (e.g., using an inward-facing imaging system 462 or an outward-facing imaging system 464).

[0075] Figure 11 is a process flow diagram of an embodiment of Method 1100 for interacting with a virtual user interface. Method 1100 may be performed by a wearable system as described herein.

[0076] In block 1110, the wearable system may identify a specific UI. The type of UI may be given by the user. The wearable system may identify that a particular UI needs to be captured based on user input (e.g., gestures, visual data, audio data, sensory data, direct commands, etc.). In block 1120, the wearable system may generate data for a virtual UI. For example, data associated with the UI's boundaries, general structure, shape, etc., may be generated. In addition, the wearable system may determine map coordinates of the user's physical location so that the wearable system can display the UI in relation to the user's physical location. For example, if the UI is body-centered, the wearable system may determine the coordinates of the user's physical standing position, head posture, or eye posture so that a ring UI can be displayed around the user, or a planar UI can be displayed on a wall or in front of the user. If the UI is hand-centered, map coordinates of the user's hand may be determined. These map points may be derived through an FOV camera, data received through sensory input, or any other type of collected data.

[0077] In block 1130, the wearable system may send data from the cloud to the display, or the data may be sent from a local database to the display component. In block 1140, the UI is displayed to the user based on the transmitted data. For example, a light field display may project the virtual UI into one or both of the user's eyes. Once the virtual UI is generated, in block 1150, the wearable system may simply wait for a command from the user and generate more virtual content on the virtual UI. For example, the UI may be a body-centered ring around the user's body. The wearable system may then wait for a command (gesture, head or eye movement, input from a user input device, etc.), and if recognized (block 1160), the virtual content associated with the command may be displayed to the user (block 1170). In one embodiment, the wearable system may wait for a hand gesture from the user before mixing multiple stem tracks.

[0078] Additional embodiments of wearable systems, UI, and user experience (UX) are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated herein in whole by reference. Exemplary objects within the eye-moving field of view (FOR) and field of view (FOV)

[0079] Figure 12 schematically illustrates embodiments of virtual objects in the field of view (FOV) and virtual objects in the eye-moving field of view (FOR). As discussed with reference to Figure 4, FOR includes a portion of the environment around the user that is perceptible to the user via the wearable system. With respect to head-mounted augmented reality devices (ARDs), FOR can include substantially all of the 4π steradian solid angle surrounding the wearer, as the wearer can move their body, head, or eyes and perceive substantially any direction in space. In other situations, the user's movement may be narrower, and therefore the user's FOR can be obtained for smaller solid angles.

[0080] In Figure 12, FOR1200 may include a group of objects (e.g., 1210, 1220, 1230, 1242, and 1244) that can be perceived by the user through the wearable system. The objects in the user's FOR1200 may be virtual or physical objects. For example, the user's FOR1200 may include physical objects such as a chair, sofa, and wall. Virtual objects may include operating system objects such as a trash can for deleted files, a terminal for entering commands, a file manager for accessing files or directories, icons, menus, applications for audio or video streaming, and notifications from the operating system. Virtual objects may also include objects within applications, such as avatars, virtual objects in games, graphics, or images. Some virtual objects can be both operating system objects and objects within applications. In some embodiments, the wearable system may add virtual elements to existing physical objects. For example, the wearable system may add a virtual menu associated with the television in the room, which may give the user options to turn on the television or change channels using the wearable system.

[0081] Virtual objects may be three-dimensional (3D), two-dimensional (2D), or one-dimensional (1D) objects. Objects within the user's FOR may be part of a world map, as illustrated with reference to Figure 9. Data associated with objects (e.g., location, semantic information, properties, etc.) may be stored in various data structures, such as arrays, lists, trees, hashes, or graphs. The index of each stored object may be determined, where applicable, by the object's location. For example, the data structure may index objects by a single coordinate, such as the distance of the object from a reference position (e.g., distance to the left (or right) of the reference position, distance from the top (or bottom) of the reference position, or depth from the reference position). The reference position may be determined based on the user's position (e.g., the user's head position). The reference position may also be determined based on the positions of virtual or physical objects (e.g., target interactable objects) within the user's environment. Thus, the 3D space within the user's environment may be folded into a 2D user interface where virtual objects are arranged according to their distance from a reference position.

[0082] Within FOR1200, the portion of the world perceived by the user at a given time is referred to as FOV1250 (for example, FOV1250 may encompass the portion of FOR that the user is currently looking at). In Figure 12, FOV1250 is schematically illustrated by the dashed line 1252. A user of a wearable system may perceive multiple objects within FOV1250, such as object 1242, object 1244, and parts of object 1230. FOV may depend on the size or optical properties of the wearable device's display. For example, an AR display (e.g., display 220 in Figure 2) may include optics (e.g., stacked waveguide assembly 480 in Figure 4 or plane waveguide 600 in Figure 6) that provide AR / MR / VR functionality when the user looks through a specific portion of the display. FOV1250 may correspond to the solid angle perceptible to the user when looking through the AR display.

[0083] As the user's posture (e.g., head posture or eye posture) changes, the FOV 1250 also changes accordingly, and objects within the FOV 1250 may also change. For example, in Figure 12, map 1210 is initially outside the user's FOV. When the user looks towards map 1210, map 1210 may move into the user's FOV 1250, while object 1230 may move outside the user's FOV 1250. As will be described herein, the wearable system may maintain tracking of objects within FOR 1200 and objects within FOV 1250.

[0084] The user can interact with interactable objects within their FOR1200, particularly interactable objects within their current FOV1250 via the wearable system. Interactive objects may be physical or virtual objects. For example, object 1230 may be a virtual graph showing changes in stock prices over time. By selecting virtual object 1230, the user can interact with it to, for example, retrieve stock price information, buy or sell stocks, or retrieve information about a company. To facilitate these interactions, the wearable system may display menus, toolbars, etc., associated with the virtual object, which can enable the user to perform various actions (e.g., retrieving stock price information).

[0085] Users can interact with objects within the FOV using various techniques, such as selecting an object, moving an object, opening a menu or toolbar associated with an object, or selecting a new set of selectable objects. Users may also interact with interactable objects by using hand gestures to activate user input devices (see, for example, user input device 466 in Figure 4), such as clicking a mouse, tapping a touchpad, swiping a touchscreen, placing a hand over or touching a capacitive button, pressing a key on a keyboard or game controller (e.g., a 5-way d-pad), pointing a joystick, wand, or totem towards an object, pressing a button on a remote control, or other interactions with user input devices. Users may also interact with interactable objects using their head, eyes, hands, feet, or other body postures, such as gazing at an object or pointing an arm at it for a period of time, tapping a foot, or blinking their eyes a certain number of times over a threshold time interval. These hand gestures and user postures on the user input device can trigger selection events in the wearable system, for example, when user interface actions are performed (e.g., a menu associated with a target interactable object is displayed, or game actions are performed on an in-game avatar).

[0086] In some implementations, the HMD features a light field display capable of displaying virtual objects to the user in different depth planes. Virtual objects can be grouped and displayed in different fixed depth planes. The user's FOV can include multiple depth planes. Therefore, the virtual objects depicted in Figure 12 can be at different apparent distances from the user, but are not required. An example of notifying the user of objects within FOR.

[0087] Figure 13 schematically illustrates an embodiment in which the user is informed of objects within the user's FOR. User 1310's FOR 1200 has multiple objects, such as object 1302a, object 1304a, and object 1352. Object 1352 is within the user's FOV 1250, while objects 1302a and 1304a are outside the user's FOV but are within the user's FOR 1200. Any of objects 1302a, 1304a, or 1352 can be a virtual object or a physical object.

[0088] In one implementation, objects within FOR may be hidden so that the user cannot directly perceive them through the display 220. However, the display 220 may present the hidden objects when the user activates a user input device or when the user adopts a certain posture (e.g., head, body, or eye posture). For example, as illustrated in Figure 13, object 1352 may initially be hidden from the user's view. However, the display may reveal object 1352 when the user clicks the user input device 466.

[0089] A wearable system can provide indication of objects outside the user's field of view (FOV) by placing a visual backlight near the edge of the user's FOV. In Figure 13, the wearable system can place one backlight 1304b for object 1304a and another backlight 1302b for object 1302a on the edge of the user's FOV 1250.

[0090] A wearable system can calculate a visual representation of a visual halo based on contextual information associated with a corresponding object, environment, or user (e.g., user's posture or preferences). The visual representation of a visual halo may include its shape, color, brightness, position, orientation, size, or other visual effects or characteristics.

[0091] To determine the visual representation of the visual backlight based on the contextual information of the corresponding object, the wearable system can use various characteristics associated with the corresponding object, such as the object's location (including its proximity to the user), the object's urgency, the object's type (e.g., interactable vs. non-interactive, physical vs. virtual, operating system object vs. game object, etc.), the object's properties (e.g., enemy avatar vs. friendly avatar), and the amount of information (e.g., the number of notifications, etc.). In one embodiment, the visual representation of the visual backlight is calculated based on the location of the corresponding object. The backlight 1304b may appear fainter than the backlight 1302b because object 1304a is further from the user's FOV than object 1302a. In another embodiment, the backlight 1302b may have a larger and brighter appearance because object 1302a is closer to the user's FOV and / or more imminent. The visual representation of the visual backlight may change over time based on changes in the object associated with the visual backlight. For example, the backlight 1304b may become larger (or brighter) as object 1304a moves closer to the user's FOV, or smaller (or darker) as object 1304a moves further away from the user's FOV.

[0092] In some implementations, the visual representation of the backlight can be determined based on the existing amount of information of the corresponding object. For example, object 1302a may be a messaging application configured to receive and send messages for user 1310. As object 1302 receives more messages, the backlight 1302b may become darker to indicate the accumulation of messages.

[0093] A wearable system may assign a color to the backlight based on the characteristics of the associated object. For example, a wearable system may assign red to the backlight 1304b because object 1304a is associated with red. Similarly, an AR system may assign blue to object 1302b because it is an operating system object and the AR system assigns blue to all operating system objects. The assigned color is not limited to a single color and may include multiple colors, shading, contrast, saturation, etc. The visual representation of the backlight may also include visual effects such as, for example, motion (e.g., translating or rotating the backlight), fade-in, or fade-out. The visual representation of the backlight may also be accompanied by sound, touch, etc.

[0094] The wearable system can also determine the visual representation of the visual backlight based on the user's posture. For example, as user 1310 turns to the left, object 1304a may move closer to the user's FOV, while object 1302a may move further away from the user's FOV. As a result, backlight 1304b may become brighter (or larger), while backlight 1302b may become darker (or smaller). The position of the backlight may change as the object and / or user moves. For example, if object 1304a moves to the right of user 1310, the AR system may show backlight 1304b to the right of FOV 1250.

[0095] In addition to determining the visual representation of visual backlight based on the characteristics of the object or the user's posture, or alternatively, the wearable system may determine the visual representation of visual backlight using contextual information associated with the user's environment. Contextual information associated with the user's environment may include the lighting conditions of the environment. Lighting conditions may be based on the environment as perceived by the user through the display 220. The environment as perceived by the user could be the user's physical surroundings, such as the user's room or a virtual environment (e.g., a simulated jungle in a game). Referring to Figure 13, backlight 1304b may initially be invisible to the user 1310. This may be because the user perceives a dark environment through the display 220. Since dark objects are typically imperceptible, the wearable system cannot display backlight for objects in a dark environment. However, the wearable system may display backlight 1304b above the edge of the user's FOV when light illuminates object 1304a. Light can originate from one or more sources, such as screen objects or light sources (real or virtual).

[0096] In one embodiment, the visual representation of the backlight can correspond to the optical effect of an object under ambient lighting conditions. For example, the shape of the backlight (e.g., oval) may be a two-dimensional projection of the object associated with the backlight (e.g., a football). In this embodiment, the visual backlight can appear as if the football were projected onto the edge of the user's field of view (FOV).

[0097] When the backlight corresponds to a virtual object, the wearable system can simulate the optical effects of the virtual object as if it were a physical object. For example, in Figure 13, object 1304a may be a virtual object, which, if it were a physical object, could reflect red light. Therefore, the display 220 of the wearable system 220 may display the backlight 1304b such that it has red light (illustrated in an oblique parallel pattern) when light shines on object 1304a.

[0098] Optical effects can also incorporate spatial information about objects. For example, when an object is behind the user, the reflected shape of the object will inevitably be longer and thinner. The user can use this shape to distinguish this object from another object located immediately to the right or left of the user's FOV. For example, in Figure 13, object 1304a is further away from the user and behind it (compared to object 1302a, for example). Therefore, the backlight 1304b may have a thin shape with lower brightness. However, the backlight 1302b is thicker and shorter because object 1302a is immediately to the right of the user's FOV.

[0099] The wearable system can render a visual representation of a backlight using the display 220. The wearable system can render at least a portion of the visual representation of the backlight in the vicinity of the user's FOV 1250. In some embodiments, the wearable system can be configured not to determine a corresponding backlight when an object is within the FOV 1250. In other embodiments, the wearable system can still determine a visual representation of a backlight even though an object is within the FOV 1250. Rather, the wearable system can be configured not to render a visual representation of a backlight for an object within the FOV because the object is perceptible to the user. As shown in Figure 13, although object 1352 is within the user's FOV 1250, the wearable system can be configured not to project a backlight onto the edge of the FOV with respect to object 1352.

[0100] In one implementation, an object within the FOV may still have its corresponding backlight, but the wearable system can be configured to conceal the visual representation of the backlight through optical effects associated with the display 220 (e.g., so that the user does not perceive the backlight). For example, if the visual representation of the backlight is determined according to lighting conditions, the internal surface edge of the backlight (corresponding to the object within the FOV) can be configured to be collinearly aligned with the wearable system's imaging system (e.g., a simulated light source and / or the camera shown in Figure 2) or the user's eye. In such an embodiment, the user would not perceive the reflection from the internal surface edge caused by the light source within the FOV, and therefore would not perceive the backlight corresponding to the object within the FOV.

[0101] In addition to visual backlighting, wearable systems can also use tactile or audio effects to inform the user about objects outside the user's field of view (FOV). For example, in an augmented reality game, a wearable system may notify the user of an approaching enemy through vibrations on a user input device. The wearable system may provide a strong vibration when the enemy is close to the user, and a weaker vibration when the enemy is relatively far away. In another embodiment, a wearable system can use audible sound to provide location information for virtual objects. The wearable system may use a high-pitched sound to alarm the user of a nearby virtual enemy. The wearable system can also simulate a sound field that reflects the spatial location of virtual objects. Tactile, audio, or visual backlighting can be used in combination or as alternatives to inform the user about surrounding objects. Exemplary visual presentation of halo

[0102] Figure 14A schematically illustrates a perspective view of the visual backlight on the edge of the FOV. In this embodiment, the edge 1410 is the area at the outer boundary of the FOV 1250. The edge 1410 of the FOV 1250 may comprise an outer frame 1412 and an inner frame 1414. The edge may have a width 1416. The wearable system can place a portion of the backlight on the edge 1410 of the FOV 1250. For example, a portion of the backlight 1402b is on the edge 1410 of the FOV 1250. In another embodiment, the wearable system can place the entire backlight 1404b between the outer frame 1412 and the inner frame 1414. The backlight may have various sizes. Some backlights may have a diameter smaller than or equal to the width of the edge 1410 of the FOV 1250 (1416), while others may have a diameter larger than the width of the edge 1410 of the FOV 1250 (1416).

[0103] A wearable system can determine and render the visual representation of backlight separately for each eye to match human peripheral vision. For example, because object 1402a is not within the left eye's FOV, the wearable system may render the visual representation of backlight 1402b only to the right eye, instead of both eyes. This technique can also be used to simulate the human eye experience of observing the movement of an object. For example, the wearable system can determine the visual representation of backlight and simulate the effect of an object fading out from the left edge of the right eye's field of view and the right edge of the left eye's field of view.

[0104] By presenting the backlight separately to each eye, the wearable system can reduce the likelihood that the human eye will resolve the backlight into depth, as the backlights for each eye do not stereoscopically match. Advantageously, in some embodiments, this reduces visual collision between the backlight and other virtual content perceived through the wearable system.

[0105] Figure 14B schematically illustrates an embodiment that makes the corresponding backlight for objects within the user's FOV invisible. As shown in Figure 14B, a distance may exist between the human eye 1488 and the edge 1410 of the FOV as perceived by the user through the wearable system. When the user uses the HMD, an FOV angle 1420 may exist. The degree of the FOV angle 1420 may be determined based on the distance between the human eye and the edge 1410. The degree of the FOV angle 1420 may also be based on the physical and chemical properties of the display 220. As the degree of the FOV angle 1420 changes, the visual representation of the backlight may also change.

[0106] As illustrated with reference to Figure 13, a wearable system may determine the visual representation of backlight by simulating the lighting conditions in the user's environment. Such lighting conditions can be applied to both objects within and outside the FOV. The wearable system can position backlights for objects within the user's FOV such that the user will not see backlights associated with objects within the user's FOV. For example, in Figure 14B, backlight 1452 can be associated with object 1352 (shown in Figure 14A). The wearable system can be configured to position the inner surface edge of backlight 1452 so as to be collinearly aligned with the wearable system's imaging system or the user's eye. In this configuration, the user will not see the inner edge of backlight 1452. Exemplary user experience with visual backlighting

[0107] Figures 15A and 15B schematically illustrate an example of a user experience with visual backlighting. In Figure 15A, user 1210 can stand in a room. When user 1210 puts on the HMD, user 1210 can see FOR 1250 and FOV 1250. Object 1552 is within the user's FOV. Object 1552 may be an object in a certain application. For example, in Figure 15A, object 1552 may be a virtual creation that user 1210 perceives through the display 220.

[0108] As shown in Figure 15A, object 1502a is within the user's FOR 1200 but outside the user's FOV 1250. Object 1502a may be an object associated with the operating system. For example, as shown in Figure 15A, object 1502a may be a home object, which can direct the user to the main page of the user's HMD. The wearable system may display a portion of the visual backlight 1502b of object 1502a on the edge 1410 of the FOV 1250. The wearable system may determine the visual representation of the backlight and, as appropriate, render a visual representation of the visual backlight 1502b using various contextual factors, as described with reference to Figures 13, 14A, and 14B.

[0109] As the user changes their posture (e.g., by tilting their head), object 1502a can move within the user's FOV 1250, as shown in Figure 15B. In some embodiments, the wearable system's display 220 will not show a backlight for object 1502a when it is within the FOV 1250. In other embodiments, the wearable system may be configured to collinearly align the inner surface edge of the backlight 1502a with the wearable system's imaging system (e.g., a simulated light source and / or a camera as shown in Figure 2) or the user's eye. Therefore, the user may not perceive the backlight when object 1502a is within the FOV 1250. Exemplary process for determining the visual representation of the backlight

[0110] Figure 16 illustrates an exemplary process for rendering a visual representation of a visual backlight. Process 1600 can be carried out by a wearable system as described herein (see, for example, the wearable system described with reference to Figures 2 and 4).

[0111] In block 1610, the wearable system determines a group of objects within the virtual user's FOR. The group of virtual objects may be a subset of objects in the user's environment. In one embodiment, the virtual objects may be hidden from the user's view.

[0112] In block 1620, the wearable system can determine the user's posture. The user's posture may be a single or combined head, eye, or body posture. The wearable system can determine the user's posture based on data obtained from various sensors, such as an inward-facing imaging system (see, for example, inward-facing imaging system 462 in Figure 4), a user input device (see, for example, user input device 466 in Figure 4), an FOV camera, and / or inputs received on sensors (see the description referring to Figure 10).

[0113] In block 1630, the wearable system can determine the user's field of view (FOV) based on the user's posture. The FOV may include a portion of the FOR perceived by the user at a given time.

[0114] Based on the user's FOV, in block 1640, the wearable system can identify virtual objects that are within the user's FOR but outside the FOV. In some implementations, the wearable system may be configured to display a halo around certain objects within the user's FOR. The wearable system can identify virtual objects from which a visual halo is rendered based on contextual information as described herein. For example, the wearable system may be configured to render a visual halo around a virtual object based on its type, and the wearable system may render a visual halo around a virtual object if it is interactive. In another embodiment, the wearable system would render a visual halo around a virtual object if it is within a threshold distance from the user. In yet another embodiment, the wearable system may be configured to sort virtual objects (outside the FOV) based on contextual information, and the wearable system may render a halo around only the most imminent object. In some implementations, the wearable system can display a halo around all virtual objects within the FOR.

[0115] In situations where a virtual object is a hidden object, a visual halo can provide cues regarding the orientation and location of the hidden treasure. For example, in a treasure hunt game, a visual halo can provide indication of the location of the hidden treasure. In some embodiments, one visual halo can correspond to more than one object. For example, a wearable system may render a visual halo indicating that a set of office tools is available to the right of the user's current location.

[0116] In block 1650, the wearable system can determine a visual representation of a visual backlight associated with a virtual object. The wearable system can determine the visual representation of the backlight using various contextual factors. For example, the wearable system may calculate the location of the virtual object relative to the user's FOV. In one embodiment, the wearable system can determine the visual representation using the local processing and data module 260, either alone or in combination with the teleprocessing module 270 (shown in Figure 2). For example, the teleprocessing module 260 can determine the color of the visual backlight, while the local processing and data module 260 can determine the location and size of the visual backlight. In another embodiment, the teleprocessing module 270 can determine a default visual representation, while the local processing and data module 260 can track the user's movement (or the virtual object corresponding to the backlight) and adjust the default visual representation as appropriate. For example, the local processing and data module 260 can access the default visual representation of the backlight from a remote data repository and adjust the size and brightness of the default visual representation of the backlight based on the location of the virtual object relative to the user's FOV.

[0117] In block 1660, the wearable system can render a visual representation of a visual halo. The visual representation can be rendered based on the decisions made in block 1650. Part of the visual halo may be positioned on the edge of the user's FOV. In some situations where multiple visual halos are displayed, the wearable system may position parts of the visual halos to overlap with parts of other visual halos. As described herein, the wearable system may render the halo representation separately for each eye. Separate presentations may match human peripheral vision and simulate the human eye experience of observing the movement of objects. In one implementation, the visual halo of a virtual object would be rendered on the edge of the FOV for one eye but not for the other eye.

[0118] The exemplary process 1600 is described with reference to rendering a visual representation of a backlight associated with a virtual object, but in some embodiments, the backlight may also be associated with a physical object (e.g., a television, a coffee maker, etc.).

[0119] Figure 17 illustrates an exemplary process for determining the visual representation of visual halo based on contextual information. Process 1700 can be carried out by a wearable system as described herein (see, for example, the wearable system described with reference to Figures 2 and 4).

[0120] In block 1710, the wearable system can identify objects in the user's environment. The objects may be physical or virtual objects, and the environment may be the user's physical or virtual environment. The wearable system can identify objects using techniques described with reference to process 1600 in Figure 16. In one embodiment, the object identified in the user's environment may be a concealed object. The concealed object may be hidden from the user's view (e.g., blocked by another object), or it may become perceptible in response to user interface activity, such as the activation of the user input device 466 or a change in the user's posture.

[0121] As illustrated with reference to Figure 16, a wearable system can identify objects based on the user's field of view (FOV). The user's FOV may be determined based on the optical properties of the wearable system's display and the user's posture (e.g., body posture or head posture), as described herein. In some embodiments, identified objects are outside the user's FOV but within the FOV. The wearable system can use a world map 920 (described in Figure 9) of the user's environment to determine whether an object is outside the user's FOV. For example, the wearable system can determine the user's current location and position the user within the world map. The wearable system can optionally calculate objects within the FOV based on the user's position within the world map.

[0122] In block 1720, the wearable system can access contextual information associated with an object, user, or environment. This contextual information may be used in block 1730 to determine (or access) a visual representation of the backlight corresponding to an object. For example, a more pressing object may be associated with a larger and brighter backlight, while a less pressing object may be associated with a smaller or dimmer backlight. In another embodiment, the backlight may represent a 2D projection of the object under the lighting conditions of the user's environment (on the edge of the FOV).

[0123] Optionally, in block 1740, the wearable system may render a visual representation of a backlight based on the decision in block 1730. An embodiment of the visual representation of a backlight is illustrated with reference to Figure 13-15B. In some cases, the wearable system may not render a visual representation of a backlight. For example, if an object is within the FOV, the system may assume that the user can see the object and a backlight is not necessary. If the system desires to draw the user's attention to an object within the FOV, the system may render a backlight associated with the object (e.g., at least partially surrounding the object) to alert the user to the object. In another embodiment, if an object is behind the user (e.g., within the user's FOR but not within the user's FOV), the wearable system may not render a visual representation because it would prevent the user from projecting the object onto the edge of the FOV. In another embodiment, as illustrated with reference to Figure 14B, the object may be projected onto the edge of the FOV such that the position of the inner surface edge of the backlight 1452 is collinearly aligned with the imaging system of the AR system or the user's eye. In this configuration, the user cannot see the inner edge of the backlight 1452.

[0124] If an object is outside the FOV but within the FOR, the visual representation of the backlight can be rendered as described with reference to block 1660 in Figure 16.

[0125] The embodiments described herein provide backlighting for objects within the FOV or for objects within the FOR but outside the FOV, but in various embodiments, backlighting can also be provided for objects both outside the FOV and FOR. For example, in a video game, a virtual enemy may hide behind a wall or approach the user from a different room outside the user's FOR. A wearable system can provide visual backlighting as a cue of the virtual enemy's location on the edge of the user's FOV. In some implementations, the user can decide whether or not to turn off visual backlighting for certain objects. For example, once the user has progressed to a certain level in the game, the user may turn off the visual backlighting feature so that the wearable system will no longer provide visual backlighting for approaching enemies. Additional Embodiments

[0126] In the first aspect, a method for providing an indication of the presence of a virtual object in an augmented reality environment surrounding a user, the method is: An augmented reality (AR) system comprising computer hardware, wherein the AR system is configured to enable user interaction with virtual objects within the user's eye-moving field of view (FOR), and FOR comprises a portion of the user's surrounding environment that is perceptible to the user via the AR system, under the control of the AR system. A method comprising: determining a group of virtual objects within the user's FOR; determining the user's posture; determining the user's field of view (FOV) at least partially based on the user's posture, wherein the FOV includes a portion of the FOR perceptible to the user at a given time via an AR system; identifying a subgroup of the group of virtual objects that are within the user's FOR but located outside the user's FOV; determining the location of at least some of the virtual objects within the subgroup of virtual objects relative to the user's FOV; determining, at least partially based on location, the placement of a visual halo associated with the virtual objects relative to the user's FOV; and displaying the visual halo such that at least a portion of the visual halo is perceptible to the user on the edge of the user's FOV. In some embodiments, the placement of the visual halo is also referred to as a visual representation of the visual halo.

[0127] In the second aspect, the user's posture is the method described in aspect 1, including the user's eye posture.

[0128] In the third aspect, the visual backlight is the method described in any one of the descriptions in aspects 1-2, including a shape having color.

[0129] In the fourth side, the shape is the method described in side 3, including a rounded square.

[0130] In the fifth aspect, the color of the visual backlight indicates the type of virtual object, as described in any one of the sections of aspects 3-4.

[0131] On the sixth side, the installation of the backlight is as described in any one of the following ways: brightness, position, or size.

[0132] In the seventh aspect, the brightness is based at least partially on the proximity of the virtual object to the edge of the FOV, as described in aspect 6.

[0133] In the eighth aspect, the size is as described in any one of the sections of aspects 6-7, indicating the proximity and / or urgency of the virtual object.

[0134] In the ninth aspect, the user's FOV is determined at least in part based on the area of ​​the AR display in the AR system, as described in any one of aspects 1-8.

[0135] In the tenth aspect, the virtual object is the method described in any one of aspects 1-9, which includes one or more operating system virtual objects or application virtual objects.

[0136] In the eleventh aspect, the method according to any one of aspects 1-10, wherein the AR system comprises a first AR display for the user's first eye and a second AR display for the user's second eye, and the method includes displaying a first representation of the visual backlight on the edge of the user's FOV by displaying a first representation of the visual backlight by the first AR display.

[0137] The twelfth aspect further includes displaying a second representation of the visual backlight by a second AR display, wherein the second representation is different from the first representation, as described in aspect 11.

[0138] In the 13th aspect, the method of aspect 12, wherein the first and second representations of the visual halo are rendered separately for each eye and conform to peripheral vision.

[0139] In the 14th aspect, the method of any one of aspects 12-13, wherein the first and second representations of the visual halo are rendered separately for each eye to reduce or avoid the depth perception of the visual halo.

[0140] The method described in any one of Aspects 1-14, further comprising, at least in part, updating a subgroup of virtual objects based on a change in the user's posture, and updating a first or second representation of a visual backlight based on the updated subgroup of virtual objects.

[0141] The method described in any one of Aspects 1-15, further comprising, in Aspect 16, determining that a virtual object within a subgroup of virtual objects has moved into the user's FOV and stopping the display of the visual backlight associated with that virtual object.

[0142] Aspect 17 further includes determining that a virtual object within the user's FOV has moved outside the user's FOV and displaying a visual backlight associated with that virtual object, the method described in any one of Aspects 1-16.

[0143] In aspect 18, a method for providing indication of an object in a user's environment, the method is An augmented reality (AR) system comprising an imaging system, wherein the AR system is configured to enable user interaction with virtual objects within the user's eye-moving field of view (FOR), and the FOR includes, under the control of the AR system, a portion of the environment surrounding the user that is perceptible to the user via the AR system. A method comprising: determining a group of objects within the user's FOR; determining the user's field of view (FOV), wherein the FOV includes a portion of the FOR that is perceptible to the user via an AR system at a given time; identifying a subgroup of the group of objects that is within the user's FOR but located outside the user's FOV; and displaying the visual backlight of one or more objects within the subgroup of objects such that at least a portion of the visual backlight is perceptible within the user's FOV.

[0144] In the 19th aspect, the method according to aspect 18, wherein the FOV is determined at least in part based on the user's posture.

[0145] The 20th aspect is the method according to aspect 19, wherein the user's posture includes at least one of the following: head posture, eye posture, or body posture.

[0146] The method according to any one of the aspects 18-20, wherein the 21st aspect further includes determining the lighting conditions of the user's environment, simulating the optical effect of the lighting conditions on one or more virtual objects within a group of objects, and determining, at least in part, the placement of visual backlighting associated with one or more virtual objects based on the simulated optical effect.

[0147] In the 22nd aspect, the method according to aspect 21, wherein the environment is one or more of a virtual environment or a physical environment.

[0148] On the 23rd side, the installation of the backlight is the method described in any one of the sections of sides 21-22, including one or more of the brightness, position, shape, or size.

[0149] The method according to any one of aspects 21-23, wherein determining the placement of the visual backlight further includes identifying a virtual object in a subgroup of objects that is within the user's FOR but outside the user's FOV, and collinearly aligning the inner edge of the visual backlight associated with the imaging system and at least one of the user's eyes with the virtual object.

[0150] In aspect 25, the AR system comprises a first AR display for the user's first eye and a second AR display for the user's second eye, and the method according to any one of aspects 18-24, wherein displaying a visual backlight on the edge of the user's FOV is represented by displaying a first representation of the visual backlight by the first AR display.

[0151] The method according to the 26th aspect, further comprising displaying a second representation of the visual backlight by a second AR display, wherein the second representation is different from the first representation, as described in aspect 25.

[0152] In the 27th aspect, the method according to aspect 26, wherein the first and second representations of the visual halo are rendered separately for each eye and conform to peripheral vision.

[0153] Aspect 28, the method of any one of aspects 26-27, wherein the first and second representations of the visual halo are rendered separately for each eye to reduce or avoid the perception of depth of the visual halo.

[0154] The method described in any one of Aspects 18-28, further comprising, at least in part, updating a subgroup of an object based on a change in the user's posture, and updating a first or second representation of a visual backlight based on the updated subgroup of the object.

[0155] Aspect 30 is an augmented reality (AR) system comprising computer hardware programmed to implement the method described in any one of Aspects 1-29.

[0156] The 31st aspect is a system for providing indication of interactable objects in a user's three-dimensional (3D) environment, the system being a display system for a wearable device, the display system being configured to provide the user with a three-dimensional view and to enable user interaction with objects in the user's eye-moving field of view (FOR), the FOR including a portion of the environment around the user that is perceptible to the user via the display system, sensors configured to acquire data associated with the user's posture, and a hardware processor communicating with the sensors and the display system, the hardware processor being acquired by the sensors A system comprising a hardware processor programmed to: determine the user's posture based on collected data; determine the user's field of view (FOV) at least partially based on the user's posture, wherein the FOV includes a portion of the FOR that is perceptible to the user at a given time via a display system; identify interactable objects located outside the user's FOV; access contextual information associated with the interactable objects; determine a visual representation of the backlight based on the contextual information; and render the visual representation of the backlight such that at least a portion of the visual backlight perceptible to the user lies at the edge of the user's FOV.

[0157] In the 32nd aspect, the display system comprises a first light field display for the user's first eye and a second light field display for the user's second eye, and in order to render a visual representation of the backlight, a hardware processor is programmed to render the first visual representation of the backlight by the first light field display onto a first edge of a first FOV associated with the first eye, and render the second visual representation of the backlight by the second light field display onto a second edge of a second FOV associated with the second eye, the system according to aspect 1.

[0158] In the 33rd aspect, the system described in aspect 32, wherein the first and second representations of the backlight are rendered separately for each eye to match the user's peripheral vision.

[0159] In Aspect 34, contextual information is a system described in any one of Aspects 31-33, including information associated with the characteristics of the user, the 3D environment, or the interactable object.

[0160] In aspect 35, the information associated with the 3D environment includes the lighting conditions of the user's environment, and the placement of backlighting is determined by simulating the optical effects of interactable objects under the lighting conditions, as described in aspect 34 of the system.

[0161] In aspect 36, the system according to any one of aspects 31-35, wherein the hardware processor is further programmed to detect changes in the user's posture, determine updated locations of interactable objects in the 3D environment based on the changes in the user's posture, and update the visual representation of the backlight based on the updated locations of the interactable objects.

[0162] In aspect 37, the system as described in aspect 36, in response to a determination that the updated location is within the user's FOV, the hardware processor is programmed to determine the placement of the backlight by collinearly aligning the inner edge of the backlight with at least one of the user's eyes.

[0163] In Aspect 38, the visual representation of the backlight is a system as described in any one of Aspects 31-37, comprising at least one of the following: position, shape, color, size, or brightness.

[0164] In the 39th aspect, the size of the backlight indicates at least one of the proximity or urgency of the interactable object, as described in aspect 8 of the system.

[0165] In the 40th aspect, the system according to any one of aspects 31-39, wherein the user's posture includes at least one of the following: head posture or direction of gaze.

[0166] In the 41st aspect, a method for providing indication of an interactable object in a user's three-dimensional (3D) environment, the method is: A wearable device, the wearable device having a display system configured to present a three-dimensional (3D) view to a user and enable user interaction with objects in the user's eye-moving field of view (FOR), wherein the FOR includes a portion of the environment around the user that is perceptible to the user via the display system, a sensor configured to obtain data associated with the user's posture, and a hardware processor that communicates with the sensor and the display system, under the control of the wearable device, A method comprising: determining the user's field of view (FOV) based at least in part on the user's posture, wherein the FOV includes a portion of the FOR that is perceptible to the user at a given time via a display system; identifying an interactable object located outside the user's FOV; accessing contextual information associated with the interactable object; determining a visual representation of the backlight based on the contextual information; and rendering the visual representation of the backlight such that at least a portion of the visual backlight perceptible to the user lies at the edge of the user's FOV.

[0167] The method according to aspect 41, further comprising rendering a first visual representation of the backlight onto a first edge of a first FOV associated with the user's first eye, and rendering a second visual representation of the backlight onto a second edge of a second FOV associated with the user's second eye.

[0168] The method according to side 42, in side 43, wherein the first and second representations of the backlight are rendered separately for each eye to match the user's peripheral vision.

[0169] In Aspect 44, contextual information includes information associated with the characteristics of the user, the 3D environment, or the interactable object, as described in any one of Aspects 41-43.

[0170] In aspect 45, the information associated within the 3D environment includes the lighting conditions of the user's environment, and the visual representation of the backlight is determined by simulating the optical effects of interactable objects under the lighting conditions, as described in aspect 44.

[0171] The method according to aspect 45, further comprising, in aspect 46, detecting a change in the user's posture, determining the updated location of an interactable object in a 3D environment based on the change in the user's posture, and updating the visual representation of the backlight based on the updated location of the interactable object.

[0172] The method according to aspect 46, wherein, in aspect 47, in response to the determination that the updated location is within the user's FOV, the inner edge of the backlight is collinearly aligned with at least one of the user's eyes.

[0173] In the 48th aspect, the visual representation of the backlight is the method described in any one of the sections of aspects 46-47, including at least one of the position, shape, color, size, or brightness.

[0174] In the 49th aspect, the size of the backlight indicates at least one of the proximity or urgency of the interactable object, as described in the method of aspect 48.

[0175] In aspect 50, the user's posture is as described in any one of aspects 41-49, including at least one of the following: head posture or direction of gaze. conclusion

[0176] The processes, methods, and algorithms described herein and / or depicted in accompanying diagrams may be embodied in code modules executed by one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to perform specific computer instructions, thereby being fully or partially automated. For example, a computing system may include a general-purpose computer (e.g., a server) or a dedicated computer, dedicated circuit, etc., programmed with specific computer instructions. The code modules may be written in a programming language that can be installed in a dynamic link library, or interpreted, and compiled and linked into an executable program. In some implementations, specific operations and methods may be performed by circuits specific to a given function.

[0177] Furthermore, functional implementations of the present disclosure are sufficiently mathematical, computational, or technically complex that application-specific hardware (utilizing appropriate specialized executable instructions) or one or more physical computing devices may need to implement the functionality, for example, due to the volume or complexity of the computations involved, or to provide results substantially in real time. For example, video may contain many frames, each frame may have millions of pixels, and certain programmed computer hardware may need to process the video data to provide a desired image processing task or application in a commercially reasonable amount of time.

[0178] Code modules or any type of data may be stored on any type of non-transient computer-readable medium, such as physical computer storage devices, including hard drives, solid-state memory, random-access memory (RAM), read-only memory (ROM), optical discs, volatile or non-volatile storage devices, combinations thereof, and / or equivalents. The method and modules (or data) may also be transmitted as data signals generated on various computer-readable transmission media, including wireless-based and wired / cable-based media (e.g., as part of a carrier wave or other analog or digital propagation signal), and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed process or process steps may be stored persistently or otherwise in any type of non-transient tangible computer storage device, or communicated via computer-readable transmission media.

[0179] Any process, block, state, step, or functionality in the flow diagrams described herein and / or depicted in the accompanying figures should be understood as potentially representing a code module, segment, or portion of code containing one or more executable instructions for implementing a particular function (e.g., logical or arithmetic) or step in the process. Various processes, blocks, states, steps, or functionality may be combined, rearranged, added, deleted, modified, or otherwise changed from the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules may implement some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps, or states associated therewith may be implemented in other appropriate sequences, for example, sequentially, in parallel, or in some other manner. Tasks or events may be added to or removed from the illustrative embodiments disclosed. Furthermore, the isolation of various system components in the implementations described herein is for illustrative purposes only and should not be understood as requiring such isolation in all implementations. It should be understood that the program components, methods, and systems described can generally be integrated together in a single computer product or packaged across multiple computer products. Many implementation variations are possible.

[0180] This process, method, and system may be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowdsourced computing networks, the Internet, and the World Wide Web. The network may be a wired or wireless network or any other type of communication network.

[0181] Each system and method of this disclosure has several innovative aspects, none of which alone contribute to or are required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of each other or in various combinations. All possible and secondary combinations are intended to fall within the scope of this disclosure. Various modifications of the implementations described herein may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations without departing from the spirit or scope of this disclosure. Accordingly, the claims are not intended to be limited to the implementations shown herein, but should be given the broadest scope consistent with the disclosure, principles, and novel features disclosed herein.

[0182] Some features described herein in the context of separate implementations may also be implemented in combinations within a single implementation. Conversely, various features described in the context of a single implementation may also be implemented separately in multiple implementations or in any preferred secondary combination. Furthermore, features described above as acting in a combination and further claimed as such, however one or more features from the claimed combination may, in some cases, be removed from the combination, and the claimed combination may be subject to secondary combinations or variations of secondary combinations. No single feature or group of features is required or essential in any embodiment.

[0183] In particular, conditional statements used herein, such as “can,” “could,” “might,” “may,” “e.g.,” and equivalents, are generally intended to convey that one embodiment includes certain features, elements, and / or steps, while other embodiments do not, unless otherwise specifically stated or understood in the context in which they are used. Therefore, such conditional statements are generally not intended to suggest that features, elements, and / or steps are required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps are included or should be implemented in any particular embodiment, with or without input or prompting by the author. The terms “equipment,” “includes,” “have,” and equivalents are synonyms and are used in a non-restrictive manner to encompass additional elements, features, actions, behaviors, etc. Furthermore, the term "or" is used in its inclusive sense (and not in its exclusive sense), and therefore, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. In addition, the articles "a," "an," and "the," as used in this application and the attached claims, should be interpreted as meaning "one or more" or "at least one" unless otherwise specified.

[0184] As used herein, the phrase “at least one of” a list of items refers to any combination of those items that includes a single element. In one embodiment, “at least one of A, B, or C” is intended to encompass A, B, C, A and B, A and C, B and C, and A, B, and C. Connecting phrases such as “at least one of X, Y, and Z” are generally understood differently in contexts such as those used to convey that an item, term, etc., may be at least one of X, Y, or Z, unless otherwise specifically stated. Thus, such connecting phrases are generally not intended to suggest that one embodiment requires the presence of at least one of X, at least one of Y, and at least one of Z, respectively.

[0185] Similarly, while actions may be depicted in a diagram in a specific order, it should be recognized that this does not mean that such actions must be performed in a specific order, or in a sequential order, or that all illustrated actions must be performed in order to achieve the desired result. Furthermore, a diagram may graphically depict one or more exemplary processes in the form of a flowchart. However, other actions not depicted may also be incorporated into the graphically illustrated exemplary methods and processes. For example, one or more additional actions may be performed before, after, simultaneously with, or in between any of the illustrated actions. In addition, actions may be rearranged or rearranged in other implementations. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products. In addition, other implementations are also within the scope of the following claims. In some cases, the actions enumerated in the claims may be performed in a different order and still achieve the desired result.

Claims

1. A method for providing an indication of the presence of virtual objects in an augmented reality environment surrounding a user, wherein the method is: An augmented reality (AR) system comprising computer hardware, configured to enable user interaction with a plurality of virtual objects within the user's eye-tracking field of view (FOR), wherein the FOR comprises a portion of the augmented reality environment around the user that is perceptible to the user via the AR system, under the control of the AR system. Determining a group of multiple virtual objects within the FOR for the user, Determining the user's posture, Determining the user's field of view (FOV) based at least partially on the user's posture, wherein the FOV includes a portion of the FOR that is perceptible to the user at a given time via the AR system. Identifying subgroups of the group of multiple virtual objects that are located within the user's FOR but outside the user's FOV, wherein each subgroup includes at least one interactable virtual object. For one or more of the multiple virtual objects within the subgroup of the multiple virtual objects, Determining the location of the virtual object relative to the user's FOV, Based at least partially on the aforementioned location, determine the placement of the visual backlight associated with the virtual object relative to the user's FOV, Displaying the visual backlight such that at least a portion of it is perceptible to the user on the edge of the user's field of view (FOV). To do Includes, The visual background represents contextual information associated with the at least one interactable virtual object within the subgroup, The aforementioned method, Accessing the context information associated with the at least one interactable virtual object, When the aforementioned context information changes, the visual backlight is to be dynamically changed. It further includes, The at least one interactable virtual object is a messaging application for receiving messages, A method wherein the context information includes the messages received by the messaging application, a change in the context information includes an increase in the number of messages received by the messaging application, and dynamically changing the visual backlight includes making the visual backlight darker.

2. The method according to claim 1, wherein the user's posture includes the user's eye posture.

3. The method according to claim 1, wherein the visual backlight includes a shape having color.

4. The method according to claim 3, wherein the shape includes a rounded square.

5. The method according to claim 3, wherein the color of the visual backlight indicates the type of the virtual object.

6. The method according to claim 1, wherein the arrangement of the visual backlight includes one or more of the following: brightness, position, or size.

7. The method according to claim 6, wherein the visual backlight includes brightness, and the brightness is at least partially based on the proximity of the virtual object to the edge of the FOV.

8. The method according to claim 1, wherein the user's FOV is determined at least in part on the area of ​​the AR display in the AR system.

9. The method according to claim 1, wherein the plurality of virtual objects include one or more operating system virtual objects or application virtual objects.

10. The aforementioned method, Updating the subgroups of the group of the plurality of virtual objects based at least partially on the change in the user's posture, The visual backlight representation is updated based on the updated subgroups of the group of the plurality of virtual objects. The method according to claim 1, further comprising:

11. The aforementioned method, Determining that a virtual object within the subgroup of the plurality of virtual objects has moved into the user's FOV, To stop displaying the visual backlight associated with that virtual object The method according to claim 1, further comprising:

12. The aforementioned method, Determining that a virtual object within the user's FOV has moved outside the user's FOV, To display the visual backlight associated with that virtual object. The method according to claim 1, further comprising:

13. A method for providing an indication of the presence of virtual objects in an augmented reality environment surrounding a user, wherein the method is: An augmented reality (AR) system comprising computer hardware, configured to enable user interaction with a plurality of virtual objects within the user's eye-tracking field of view (FOR), wherein the FOR comprises a portion of the augmented reality environment around the user that is perceptible to the user via the AR system, under the control of the AR system. Determining a group of multiple virtual objects within the FOR for the user, Determining the user's posture, Determining the user's field of view (FOV) based at least partially on the user's posture, wherein the FOV includes a portion of the FOR that is perceptible to the user at a given time via the AR system. Identifying subgroups of the group of multiple virtual objects that are located within the user's FOR but outside the user's FOV, wherein each subgroup includes at least one interactable virtual object. For one or more of the multiple virtual objects within the subgroup of the multiple virtual objects, Determining the location of the virtual object relative to the user's FOV, Based at least partially on the aforementioned location, determine the placement of the visual backlight associated with the virtual object relative to the user's FOV, Displaying the visual backlight such that at least a portion of it is perceptible to the user on the edge of the user's field of view (FOV). To do Includes, The visual background represents contextual information associated with the at least one interactable virtual object within the subgroup, The aforementioned method, Accessing the context information associated with the at least one interactable virtual object, When the aforementioned context information changes, the visual backlight is to be dynamically changed. It further includes, The at least one interactable virtual object is a messaging application for receiving messages, The context information includes the messages received by the messaging application, a change in the context information includes an increase in the number of messages received by the messaging application, and dynamically changing the visual backlight includes the visual backlight becoming darker. The method further comprises identifying a physical group of multiple physical objects located within the user's FOR and within the user's FOV, wherein the physical group includes at least one interactable physical object.

14. A method for providing an indication of the presence of virtual objects in an augmented reality environment surrounding a user, wherein the method is: An augmented reality (AR) system comprising computer hardware, configured to enable user interaction with a plurality of virtual objects within the user's eye-tracking field of view (FOR), wherein the FOR comprises a portion of the augmented reality environment around the user that is perceptible to the user via the AR system, under the control of the AR system. Determining a group of multiple virtual objects within the FOR for the user, Determining the user's posture, Determining the user's field of view (FOV) based at least partially on the user's posture, wherein the FOV includes a portion of the FOR that is perceptible to the user at a given time via the AR system. Identifying subgroups of the group of multiple virtual objects that are located within the user's FOR but outside the user's FOV, wherein each subgroup includes at least one interactable virtual object. For one or more of the multiple virtual objects within the subgroup of the multiple virtual objects, Determining the location of the virtual object relative to the user's FOV, Based at least partially on the aforementioned location, determine the placement of the visual backlight associated with the virtual object relative to the user's FOV, Displaying the visual backlight such that at least a portion of it is perceptible to the user on the edge of the user's field of view (FOV). To do Includes, The visual background represents contextual information associated with the at least one interactable virtual object within the subgroup, The aforementioned method, Accessing the context information associated with the at least one interactable virtual object, When the aforementioned context information changes, the visual backlight is to be dynamically changed. It further includes, The at least one interactable virtual object is a messaging application for receiving messages, The context information includes the messages received by the messaging application, a change in the context information includes an increase in the number of messages received by the messaging application, and dynamically changing the visual backlight includes the visual backlight becoming darker. The aforementioned method, Displaying a first AR display for the user's first eye, Displaying a different second AR display for the user's second eye Includes, Displaying the visual backlight on the edge of the user's FOV includes the first AR display displaying a first representation of the visual backlight. The aforementioned method, Identifying at least one physical object within the FOR, Adding virtual elements to the physical object Methods that further include the above.

15. The method according to claim 14, further comprising the second AR display displaying a second representation of the visual backlight, wherein the second representation is different from the first representation.

16. The method according to claim 15, wherein the first representation and the second representation of the visual backlight are rendered separately for each eye to reduce or avoid the perception of depth of the visual backlight.

17. A method for providing indication of virtual objects in a user's augmented reality environment, wherein the method is: An augmented reality (AR) system, wherein the AR system comprises an imaging system. The AR system is configured to enable user interaction with multiple virtual objects within the user's eye-moving field of view (FOR), and the FOR comprises a portion of the augmented reality environment around the user that is perceptible to the user via the AR system, under the control of the AR system. Determining a group of multiple virtual objects within the FOR for the user, Determining the user's field of view (FOV), wherein the FOV includes a portion of the FOR that is perceptible to the user at a given time via the AR system. Identifying subgroups of the group of multiple virtual objects that are located within the user's FOR but outside the user's FOV, wherein each subgroup includes at least one interactable virtual object. Displaying a visual backlight for one or more of the multiple virtual objects within a subgroup of the group of the multiple virtual objects such that at least a portion of the visual backlight is perceptible within the user's FOV, wherein the visual backlight represents contextual information associated with the at least one interactable virtual object within the subgroup. Accessing the context information associated with the at least one interactable virtual object, When the aforementioned context information changes, the visual backlight is to be dynamically changed. Includes, The at least one interactable virtual object is a messaging application for receiving messages, The context information includes the messages received by the messaging application, a change in the context information includes an increase in the number of messages received by the messaging application, and dynamically changing the visual backlight includes the visual backlight becoming darker. The method further comprises using a tactile effect to inform the user of at least one virtual object within the subgroup of the plurality of virtual objects.

18. The method according to claim 17, wherein the FOV is determined at least in part on the user's posture, and the user's posture includes at least one of a head posture, an eye posture, and a body posture.

19. The aforementioned method, Determining the lighting conditions of the user's augmented reality environment, To simulate the optical effect of the light conditions on one or more virtual objects within the group of the plurality of virtual objects, To determine the arrangement of the visual backlight associated with one or more virtual objects, at least in part, based on the simulated optical effect. The method according to claim 17, further comprising:

20. The method according to claim 19, wherein the arrangement of the visual backlight includes one or more of brightness, position, shape, or size, and the size indicates the urgency of the virtual object.

21. The aforementioned method, Updating the subgroups of the group of the plurality of virtual objects based at least partially on the change in the user's posture, The visual backlight representation is updated based on the updated subgroups of the group of the plurality of virtual objects. The method according to claim 17, further comprising:

22. A method for providing indication of virtual objects in a user's augmented reality environment, wherein the method is: An augmented reality (AR) system comprising an imaging system, configured to enable user interaction with a plurality of virtual objects within the user's eye-moving field of view (FOR), wherein the FOR comprises a portion of the augmented reality environment around the user that is perceptible to the user via the AR system, under the control of the AR system. Determining a group of multiple virtual objects within the FOR for the user, Determining the user's field of view (FOV), wherein the FOV includes a portion of the FOR that is perceptible to the user at a given time via the AR system. Identifying subgroups of the group of multiple virtual objects that are located within the user's FOR but outside the user's FOV, wherein each subgroup includes at least one interactable virtual object. To display the visual backlight to one or more virtual objects within the subgroup of the group of the plurality of virtual objects such that at least a portion of the visual backlight is perceptible within the user's FOV. Includes, The visual background represents contextual information associated with the at least one interactable virtual object within the subgroup, The aforementioned method, Accessing the context information associated with the at least one interactable virtual object, When the aforementioned context information changes, the visual backlight is to be dynamically changed. It further includes, The at least one interactable virtual object is a messaging application for receiving messages, The context information includes the messages received by the messaging application, a change in the context information includes an increase in the number of messages received by the messaging application, and dynamically changing the visual backlight includes the visual backlight becoming darker. The AR system includes displaying a first AR display for the user's first eye, a different second AR display for the user's second eye, and displaying the visual backlight, which includes the first AR display displaying a first representation of the visual backlight. Displaying the aforementioned visual backlight includes displaying the aforementioned visual backlight on a virtual user interface (UI), The method further comprises generating virtual content on the virtual UI by interacting with the virtual UI using physical objects.

23. The method according to claim 22, further comprising the second AR display displaying a second representation of the visual backlight, wherein the second representation is different from the first representation.

24. The method according to claim 23, wherein the first representation and the second representation of the visual backlight are rendered separately for each eye, thereby avoiding depth perception of the visual backlight.

Images