Content movement and interaction using a single controller

Abstract

An example of a system and method for interacting with content using a single controller and updating the position and orientation of the content. The system can allow a user to use the same controller to move content around a room and interact with the content by tracking the range of motion of the controller.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 965,708, filed on January 24, 2020, entitled “CONTENT MOVEMENT AND INTERACTION USING A SINGLE CONTROLLER,” pursuant to 35 U.S.C. §119(e), the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to systems and methods for facilitating an interactive virtual or augmented reality environment for one or more users. Background Art

[0004] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality," "augmented reality," or "mixed reality" experiences, in which digitally reproduced images, or portions thereof, are presented to a user in such a way that they appear to be real, or they may be perceived to be real. Virtual reality or "VR" scenes typically involve the presentation of digital or virtual image information that is opaque to other actual real-world visual input; augmented reality or "AR" scenes typically involve the presentation of digital or virtual image information as an enhancement to the visualization of the actual world around the user; mixed reality, or "MR," is related to merging the real world and the virtual world to produce a new environment in which physical and virtual objects coexist and interact in real time. It turns out that the human visual perception system is very complex, and it is a challenge to produce a VR, AR, or MR technology that facilitates comfortable, natural, and rich presentation of virtual image elements and other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies. Summary of the invention

[0005] Embodiments of the present disclosure relate to devices, systems, and methods for facilitating virtual or augmented reality interaction for one or more users.

[0006] Further details of features, objects, and advantages of the present disclosure are described below in the detailed description, drawings, and claims.The foregoing general description and the following detailed description are exemplary and explanatory and are not intended to limit the scope of the present disclosure.

[0007] In some configurations, an augmented reality (AR) system may include: an AR display configured to present virtual content to a user of the AR system; an outward-facing camera configured to capture one or more images of the user's environment; a handheld controller that defines a pointing vector that indicates a pointing direction of the handheld controller; a hardware processor in communication with the AR display, the outward-facing camera, and the handheld controller, the hardware processor being programmed to: display an interactive content object via the AR display; in a first interaction mode, when the pointing vector remains within the interactive content object, indicate movement of the handheld controller within the interactive content object and allow interaction with the interactive content object via the handheld controller; monitor changes in the pointing vector with reference to the interactive content object; in response to detecting movement of the pointing vector outside the interactive content object, update the system to a second interaction mode in which the handheld controller causes movement of the interactive content object such that the interactive content object follows movement of the handheld controller in the virtual environment; and in response to detecting movement of the pointing vector of the handheld controller within the interactive content object, update the system to the first interaction mode.

[0008] In some configurations, an augmented reality (AR) system may include: an AR display configured to present virtual content to a user of the AR system; a handheld controller having at least six degrees of freedom; and a hardware processor in communication with the AR display, an outward-facing camera, and the handheld controller, the hardware processor being programmed to: display interactive content at a first content location; determine a first pointing vector comprising a direction indicated by the controller; determine whether the first pointing vector intersects a bounded volume associated with the interactive content; in response to determining that the first pointing vector does not intersect the bounded volume, move the interactive content to a second content location associated with a point along the direction of the first pointing vector; and in response to determining that the first pointing vector intersects the bounded volume, receive an indication to interact with the interactive content at the first content location.

[0009] A method for displaying virtual content may include: displaying interactive content at a first content location; determining a first pointing vector including a direction indicated by a controller; determining whether the first pointing vector intersects a bounded volume associated with the interactive content; in response to determining that the first pointing vector does not intersect the bounded volume, moving the interactive content to a second content location associated with a point along the direction of the first pointing vector; and in response to determining that the first pointing vector intersects the bounded volume, receiving an indication to interact with the interactive content at the first content location. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Throughout the drawings, reference numerals are repeatedly used to indicate corresponding relationships between referenced elements.The following drawings and associated descriptions are provided to illustrate embodiments of the present disclosure and not to limit the scope of the claims.

[0011] The accompanying drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the drawings are not drawn to scale and that elements of similar structure or function are represented by similar reference numerals throughout the drawings. In order to better understand how to obtain the above and other advantages and objects of the various embodiments of the present disclosure, the present disclosure briefly described above will be described in more detail with reference to specific embodiments of the present disclosure, which are illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the present disclosure and therefore should not be considered limiting of its scope, the present disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0012] Figure 1 An augmented reality (AR) scene view of a user through an AR device is shown.

[0013] Figure 2 A conventional display system for simulating a three-dimensional image for a user is shown.

[0014] Figures 3A-3C The relationship between the radius of curvature and the focal radius is shown.

[0015] Figure 4A A representation of the accommodation-convergence response of the human visual system is shown.

[0016] Figure 4B Examples of different accommodation and convergence states of a user's eyes are shown.

[0017] Figure 4C An example of a representation of a top plan view of a user viewing content via a display system is shown.

[0018] Figure 4D Another example of a representation of a top plan view of a user viewing content via a display system is shown.

[0019] Figure 5 Aspects of a method for simulating three-dimensional images by modifying wavefront divergence are shown.

[0020] Figure 6 An example of a waveguide stack for outputting image information to a user is shown.

[0021] Figure 7 An example of an outgoing light beam output by a waveguide is shown.

[0022] Figure 8An example of a stacked waveguide assembly is shown where each depth plane includes an image formed using multiple different component colors.

[0023] Fig.9A A cross-sectional side view of an example of a set of stacked waveguides, each stacked waveguide including an in-coupling optical element, is shown.

[0024] Fig. 9B Shows Fig.9A A perspective of an example of multiple stacked waveguides.

[0025] Fig. 9C Shows Fig.9A and Fig. 9B A top plan view of an example of multiple stacked waveguides.

[0026] Fig.9D An example of a wearable display system is shown.

[0027] Fig. 10A Examples of user input received through controller buttons are shown.

[0028] Fig. 10B An example of user input received via a controller touch pad is shown.

[0029] Fig. 10C Examples of user input received through physical movement of a controller or head mounted device (HMD) are shown.

[0030] Fig. 10D An example of how user inputs can have different durations is shown.

[0031] Fig.11A Additional examples of user input received through controller buttons are shown.

[0032] Fig. 11B Additional examples of user input received via a controller touch pad are shown.

[0033] Fig. 12A An example content control environment 1200 is shown according to one example of the following and interaction methods disclosed herein.

[0034] Fig. 12B A flow diagram of an example process that may be used to control the position of or interact with content in a user's 3D space is shown.

[0035] Fig. 12C Example content within the bounding volume is shown.

[0036] Fig.12D A top view of a controller is shown, oriented so that its pointing vector is generally toward the content within the bounding volume.

[0037] Fig.13A , Fig. 13B and Fig. 13C Example aspects of an example content movement environment are shown.

[0038] FIG. 14A to FIG. 14C Aspects of an example content targeting environment are shown.

[0039] Fig.14D A flow diagram of an example content targeting process is shown.

[0040] Fig.15 An example content access mechanism is shown.

[0041] Fig.16 An example content drop mechanism is shown.

[0042] Fig.17 An example application of the content follow system is depicted in which two users of respective wearable systems are conducting a telepresence session. DETAILED DESCRIPTION

[0043] AR and / or VR systems can display virtual content to a user or viewer. For example, the content can be displayed on a head-mounted display, such as a part of glasses, projecting image information onto the user's eyes. In addition, in the case where the system is an AR system, the display can also transmit light from the surrounding environment to the user's eyes to allow viewing of the surrounding environment. As used herein, it will be understood that a "head-mounted" or "head-mountable" display is a display that can be mounted on the head of a viewer or user. Such a display can be understood to form part of a display system.

[0044] In various augmented reality and virtual reality display systems, an outcoupling element, such as a diffractive or reflective outcoupling element, outcouples light from a waveguide to the user's eye. However, a typical outcoupling element may not be directional and may be configured to output light in various directions at all areas within the outcoupling optical element. Thus, while a portion of the light outcoupled by the outcoupling element may be usefully directed toward the pupil of the user's eye, where it will enter the eye to form an image, other light outcoupled by the outcoupling element may not be incident at or near the pupil of the eye and therefore may not contribute to the image formed by the display system.

[0045] Therefore, it may be desirable to design a display system that can improve or optimize efficiency by aiming the outcoupling of light toward the pupil of the user's eye and by reducing the amount of light that is outcoupled in other directions. By aiming the outcoupling toward the pupil, such a system can reduce the amount of light energy that must be generated by the light projection system or other display light source to produce a given brightness image in the wearer's eye. Various embodiments of the present technology provide systems including outcoupling elements, incoupling elements, and / or light projection systems that are configured to selectively direct image light toward the pupil of the wearer's eye. Such systems can therefore advantageously improve the efficiency of the display devices disclosed herein because they can increase the proportion of a given amount of light generated by the light projection system used to form an image perceived by the user, and can reduce the proportion of light that falls on other parts of the user's eye or face, or light that otherwise does not contribute to the image perceived by the user.

[0046] Reference will now be made to the drawings, wherein like reference numerals refer to like parts throughout. Unless otherwise indicated, the drawings are schematic and are not necessarily drawn to scale.

[0047] A. Terminology

[0048] To facilitate understanding of the systems and methods discussed herein, a number of terms are described below. The terms described below, as well as other terms used herein, should be interpreted to include the descriptions provided, the ordinary and customary meanings of these terms, and / or any other implied meanings of the corresponding terms, where such interpretations are consistent with the context of the terms. Therefore, the following description does not limit the meanings of these terms, but only provides exemplary descriptions.

[0049] Head pose (or "head pose"): The position and / or orientation of a wearable headset, and by proxy, the position and / or orientation of the user's head in the real world. Head pose may be determined using sensors such as an inertial measurement unit (IMU), accelerometers, gyroscopes, etc. Head pose rays extending in the direction of the head pose may be used to interact with virtual objects. For example, when a user points at or looks at a prism or object, the object or prism intersects the user's head pose ray.

[0050] Controller: A handheld controller, such as a totem. The controller may provide multiple degrees of freedom, such as 6DoF (six degrees of freedom).

[0051] Controller pose: The position and / or orientation of a controller. The controller pose can be used to determine the area, volume, or point of a mixed reality environment that the controller is pointing at.

[0052] Prism: A container, area, or volume associated with mixed reality content or a mixed reality space. For example, a prism may contain one or more virtual content items that can be selected by a user. A prism may be generated when an application is launched, and then sibling or child prisms may be generated to create a flexible layout. The application within the prism can be configured to control where these layered prisms will appear, which is usually near the first prism and easy for the user to find. The prism can provide feedback to the user. In some embodiments, the feedback can be a title that is only displayed to the user when the prism is targeted by a head gesture. In some embodiments, the feedback can be a glow around the prism. Prism glow (and / or other prism feedback) can also be used for sharing to provide feedback to the user about which prisms are being shared.

[0053] Focus: A feature of an object (e.g. a prism) that allows selection of the interactive object.

[0054] Input focus: A characteristic of an object (prism or application) that causes the object's cursor to be refreshed and rendered as the active system cursor. In some implementations, there can be multiple focus objects, but only one has input focus.

[0055] 3D Content: Virtual objects that can be displayed in a user's 3D environment. 3D content can be static, animated, manipulable, or otherwise interactive. 3D content can include web-based content generated by a user interacting with a web domain or web page using, for example, a browser block.

[0056] A. Example Augmented Reality Scenarios

[0057] Figure 1 An illustration of an augmented reality scene with certain virtual reality objects and certain actual reality objects viewed by a person is depicted. Figure 1 An augmented reality scene 100 is depicted in which a user of AR technology sees people, trees, a real-world park-like setting 110 featuring buildings in the background, and a concrete platform 120. In addition to these items, the user of AR technology also perceives that he "sees" a robotic statue 130 standing on the real-world platform 120, as well as a flying cartoon-like character 140 (e.g., a bumblebee) that appears to be a personification of a bumblebee, even though these elements do not exist in the real world.

[0058] In order for a three-dimensional (3-D) display to produce a realistic sense of depth, and more specifically, a simulated sense of surface depth, it is desirable for each point in the display's field of view to produce an accommodation response that corresponds to its virtual depth. If the accommodation response to a displayed point does not correspond to the virtual depth of that point, as determined by the binocular depth cues of convergence and stereoscopic vision, the human eye may experience an accommodation conflict, resulting in imaging instability, detrimental eye fatigue, headaches, and, in the absence of accommodation information, an almost complete lack of surface depth.

[0059] VR, AR, and MR experiences may be provided by a display system having a display in which images corresponding to multiple depth planes are provided to a viewer. The images at each depth plane may be different (e.g., providing a slightly different scene or object presentation) and may be focused on separately by the viewer's eyes, thereby facilitating providing depth cues to the user based on eye accommodation required to focus on different image features of a scene located at different depth planes, and / or based on observing different image features at different depth planes out of focus. As discussed elsewhere herein, such depth cues provide credible depth perception.

[0060] B. Example Display System

[0061] Figure 2 A conventional display system for simulating a three-dimensional image for a user is shown. It should be understood that the user's eyes are spaced apart, and when viewing a real object in space, each eye will have a slightly different view of the object, and an image of the object may be formed at a different location on the retina of each eye. This may be referred to as binocular parallax, and may be used by the human visual system to provide a perception of depth. Conventional display systems simulate binocular parallax by presenting two different images 190, 200 (one for each eye 210, 220) having slightly different views of the same virtual object, corresponding to the view of the virtual object that each eye would see if the virtual object were a real object of the desired depth. These images provide binocular cues that the user's visual system may interpret as obtaining a perception of depth.

[0062] Continue to refer Figure 2, the images 190, 200 are spaced a distance 230 from the eyes 210, 220 on the z-axis. The z-axis is parallel to the optical axis of the viewer, whose eyes are fixed on an object at optical infinity directly in front of the viewer. The images 190, 200 are flat and at a fixed distance from the eyes 210, 220. Based on slightly different views of the virtual objects in the images presented to the eyes 210, 220, respectively, the eyes can naturally rotate so that the image of the object falls on corresponding points on the retina of each eye to maintain a single binocular vision. The rotation can cause the line of sight of each of the eyes 210, 220 to converge to a point in space at which the virtual object is perceived to exist. As a result, providing a three-dimensional image typically involves providing binocular cues that can manipulate the convergence of the user's eyes 210, 220, and the human visual system interprets it as providing a perception of depth.

[0063] However, generating a realistic and comfortable perception of depth is challenging.It will be appreciated that light from objects at different distances from the eye have wavefronts with different amounts of divergence. Figures 3A-3C The relationship between distance and divergence of light is shown. The distance between the object and the eye 210 is represented by R1, R2 and R3 in order of decreasing distance. Figures 3A-3C As shown in , as the distance from the object decreases, the light becomes more divergent. Conversely, as the distance increases, the light becomes more collimated. In other words, it can be said that the light field generated by a point (object or part of an object) has a spherical wavefront curvature that is a function of how far the point is from the user's eye. The curvature increases as the distance between the object and the eye 210 decreases. Although for clarity in Figures 3A-3C Only a single eye 210 is shown in FIG. 2 and other figures herein, but the discussion regarding eye 210 may apply to both eyes 210 and 220 of a viewer.

[0064] Continue to refer Figures 3A-3C, the light emitted by the object on which the viewer's eye is focused may have different degrees of wavefront divergence. Due to different amounts of wavefront divergence, the lens of the eye may focus the light differently, which in turn may require the lens to take different shapes to form a focused image on the retina of the eye. In the case where a focused image is not formed on the retina, the resulting retinal blur may serve as a cue for adaptation, which causes a change in the shape of the lens of the eye until a focused image is formed on the retina. For example, the cue of adaptation may trigger the ciliary muscles around the lens of the eye to relax or contract, thereby adjusting the force applied to the suspensory ligaments that hold the lens, thereby causing the shape of the lens of the eye to change until the retinal blur of the focused object is eliminated or minimized, thereby forming a focused image of the focused object on the retina of the eye (e.g., the fovea). The process by which the lens of the eye changes shape may be referred to as adaptation, and the shape of the lens of the eye required to form a focused image of the focused object on the retina of the eye (e.g., the fovea) may be referred to as an adapted state.

[0065] Reference now Figure 4A , shows a representation of the accommodation-convergence response of the human visual system. Movement of the eyes in fixation on an object causes the eyes to receive light from the object, which forms an image on each retina of the eyes. The presence of retinal blur in the image formed on the retina can provide a cue for accommodation, and the relative position of the images on the retina can provide a cue for convergence. The cue of accommodation causes accommodation to occur, resulting in the lenses of the eyes each assuming a specific state of accommodation, which forms a focused image of the object on the retina of the eye (e.g., the fovea). On the other hand, the cue of convergence causes a vergence movement (rotation of the eyes) to occur, so that the images formed on each retina of each eye are located at corresponding retinal points that maintain a single binocular vision. In these positions, the eyes can be said to have assumed a specific state of convergence. Continue with reference to Figure 4A , accommodation can be understood as the process by which the eye reaches a specific state of accommodation, and vergence can be understood as the process by which the eye reaches a specific state of vergence. Figure 4A As shown in , if the user looks at another object, the accommodation and convergence states of the eyes may change. For example, if the user looks at a new object at a different depth on the z-axis, the accommodation state may change.

[0066] Without being limited by theory, it is believed that a viewer of an object may perceive the object as "three-dimensional" due to a combination of convergence and accommodation. As described above, the convergence movement of the two eyes relative to each other (e.g., rotation of the eyes so that the pupils move toward or away from each other to converge the eyes' line of sight to fixate on an object) is closely related to the accommodation of the lens of the eye. Under normal circumstances, changing the shape of the lens of the eye to change the focus from one object to another object at a different distance will automatically result in a matching change in vergence from the same distance under a relationship called the "accommodation-convergence reflex." Likewise, under normal circumstances, a change in vergence will trigger a matching change in the shape of the lens.

[0067] Reference now Figure 4B , examples of different accommodation and convergence states of eyes are shown. A pair of eyes 222a is fixated on an object at optical infinity, while a pair of eyes 222b is fixated on an object 221 at less than optical infinity. Notably, the convergence state of each pair of eyes is different, with the pair of eyes 222a pointing straight ahead, while the pair of eyes 222 converge on the object 221. The accommodation states of the eyes forming each pair of eyes 222a and 222b are also different, as represented by the different shapes of the lenses 210a, 220a.

[0068] Unexpectedly, many users of conventional "3D" display systems find such conventional systems uncomfortable or unable to perceive a sense of depth at all due to the mismatch between the accommodation state and the convergence state in these displays. As described above, many stereoscopic or "3D" display systems display a scene by providing a slightly different image to each eye. Such systems are uncomfortable for many viewers because they, among other things, provide different presentations of the scene and cause changes in the convergence state of the eyes, but do not change the accommodation state of those eyes accordingly. Instead, the image is shown at a fixed distance from the eye by the display so that the eye views all image information in a single accommodation state. Such an arrangement counteracts the "accommodation-convergence reflex" by causing changes in the convergence state without a matching change in the accommodation state. It is believed that this mismatch can cause discomfort to the viewer. A display system that provides a better match between accommodation and convergence may form a more realistic and more comfortable three-dimensional image simulation.

[0069] Without being limited by theory, it is believed that the human eye can generally interpret a limited number of depth planes to provide depth perception. Therefore, by providing the eye with different presentations of images corresponding to each of these limited number of depth planes, a highly credible simulation of perceived depth can be achieved. In some embodiments, the different presentations can provide both cues for convergence and matching cues for accommodation, thereby providing a physiologically correct accommodation-vergence match.

[0070] Continue to refer Figure 4B, two depth planes 240 are shown, corresponding to different distances in space from the eyes 210, 220. For a given depth plane 240, a vergence cue may be provided by displaying an image of appropriately different perspectives for each eye 210, 220. In addition, for a given depth plane 240, the light forming the image provided to each eye 210, 220 may have a wavefront divergence corresponding to the light field produced by a point at the distance of the depth plane 240.

[0071] In the illustrated embodiment, the distance along the z-axis of the depth plane 240 containing point 221 is 1 m. As used herein, the distance or depth along the z-axis can be measured using a zero point located at the pupil of the user's eye. Therefore, the depth plane 240 located at a depth of 1 m corresponds to a distance of 1 m from the pupil of the user's eye on the optical axis of these eyes, with the eyes directed toward optical infinity. As an approximation, the depth or distance along the z-axis can be measured from the display in front of the user's eye (e.g., from the surface of the waveguide), plus the distance value between the device and the pupil of the user's eye. This value can be called the interocular distance, and corresponds to the distance between the pupil of the user's eye and the display worn by the user in front of the eye. In practice, the value of the interocular distance can be a normalized value commonly used for all viewers. For example, it can be assumed that the interocular distance is 20 mm, and the distance of the depth plane at a depth of 1 m in front of the display can be 980 mm.

[0072] Reference now Figure 4C and 4D , respectively show examples of matched adaptation-convergence distance and mismatched adaptation-convergence distance. Figure 4C As shown in , the display system can provide an image of a virtual object to each eye 210, 220. The image can cause the eyes 210, 220 to present a state of convergence, in which the eyes converge on a point 15 on a depth plane 240. In addition, the image can be formed by light having a wavefront curvature corresponding to a real object at the depth plane 240. As a result, the eyes 210, 220 present an accommodation state, in which the image is focused on the retinas of those eyes. Therefore, the user can perceive the virtual object as being at a point 15 on the depth plane 240.

[0073] It should be understood that each of the accommodation state and the convergence state of the eyes 210, 220 is associated with a specific distance on the z-axis. For example, an object at a specific distance from the eyes 210, 220 causes those eyes to exhibit a specific accommodation state based on the distance of the object. The distance associated with a specific accommodation state may be referred to as the accommodation distance A. d Similarly, in a particular state of convergence or position relative to each other, there is a particular vergence distance V associated with the eyes. dIn the case where the adaptation distance and the vergence distance match, the relationship between adaptation and vergence can be said to be physiologically correct. This is considered the most comfortable scenario for the viewer.

[0074] However, in stereoscopic displays, the accommodation distance and the vergence distance may not always match. Figure 4D As shown in , the image displayed to the eyes 210, 220 can be displayed with a wavefront divergence corresponding to the depth plane 240, and the eyes 210, 220 can present a specific accommodation state in which points 15a, 15b on the depth plane are in focus. However, the image displayed to the eyes 210, 220 may provide a convergence cue that causes the eyes 210, 220 to converge on a point 15 that is not located on the depth plane 240. As a result, in some embodiments, the accommodation distance corresponds to the distance from the pupil of the eyes 210, 220 to the depth plane 240, while the convergence distance corresponds to the larger distance from the pupil of the eyes 210, 220 to the point 15. The accommodation distance is different from the convergence distance. Therefore, there is an accommodation-convergence mismatch. Such a mismatch is considered undesirable and may cause discomfort to the user. It should be understood that the mismatch corresponds to the distance (e.g., V d -A d ), and can be characterized using diopters.

[0075] In some embodiments, it should be understood that reference points other than the pupils of eyes 210, 220 may be used to determine the distance used to determine the accommodation-vergence mismatch, as long as the same reference point is used for the accommodation distance and the vergence distance. For example, the distance from the cornea to the depth plane, from the retina to the depth plane, from the eyepiece (e.g., a waveguide of a display device) to the depth plane, etc. may be measured.

[0076] Without being limited by theory, it is believed that users may still perceive accommodation-convergence mismatches of up to about 0.25 diopters, up to about 0.33 diopters, and up to about 0.5 diopters as physiologically correct, without the mismatch itself causing significant discomfort. In some embodiments, the display systems disclosed herein (e.g., Figure 6 In some embodiments, the display system 250 presents an image to a viewer with an accommodation-convergence mismatch of about 0.5 diopters or less. In some other embodiments, the accommodation-convergence mismatch of the image provided by the display system is about 0.33 diopters or less. In other embodiments, the accommodation-convergence mismatch of the image provided by the display system is about 0.25 diopters or less, including about 0.1 diopters or less.

[0077] Figure 5Aspects of a method for simulating a three-dimensional image by modifying wavefront divergence are shown. The display system includes a waveguide 270 configured to receive light 770 encoded with image information and output the light to an eye 210 of a user. The waveguide 270 can output light 650 having a defined amount of wavefront divergence that corresponds to the wavefront divergence of a light field generated by a point on a desired depth plane 240. In some embodiments, the same amount of wavefront divergence is provided for all objects presented on the depth plane. Additionally, it will be described that image information from a similar waveguide can be provided to the user's other eye.

[0078] In some embodiments, a single waveguide can be configured to output light having a set amount of wavefront divergence corresponding to a single or limited number of depth planes, and / or a waveguide can be configured to output light of a limited wavelength range. Thus, in some embodiments, multiple waveguides or stacked waveguides can be utilized to provide different amounts of wavefront divergence for different depth planes and / or output light of different wavelength ranges. As used herein, it should be understood that at a depth plane can be a flat surface or can follow the contour of a curved surface.

[0079] Figure 6 An example of a waveguide stack for outputting image information to a user is shown. The display system 250 includes a stack of waveguides, or a stacked waveguide assembly 260, which can be used to provide a three-dimensional perception to the eye / brain using multiple waveguides 270, 280, 290, 300, 310. It will be understood that in some embodiments, the display system 250 can be considered a light field display. Additionally, the waveguide assembly 260 can also be referred to as an eyepiece.

[0080] In some embodiments, the display system 250 can be configured to provide a substantially continuous cue of convergence and multiple discrete cues of accommodation. The cues of convergence can be provided by displaying a different image to each eye of the user, and the cues of accommodation can be provided by outputting image-forming light with selectable discrete amounts of wavefront divergence. In other words, the display system 250 can be configured to output light with variable levels of wavefront divergence. In some embodiments, each discrete level of wavefront divergence corresponds to a specific depth plane and can be provided by a specific waveguide of the waveguides 270, 280, 290, 300, 310.

[0081] Continue to refer Figure 6, the waveguide assembly 260 may also include a plurality of features 320, 330, 340, 350 between the waveguides. In some embodiments, the features 320, 330, 340, 350 may be one or more lenses. The waveguides 270, 280, 290, 300, 310 and / or the plurality of lenses 320, 330, 340, 350 may be configured to send image information to the eye at various levels of wavefront curvature or light divergence. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane. The image injection device 360, 370, 380, 390, 400 may serve as a light source for the waveguides and may be used to inject image information into the waveguides 270, 280, 290, 300, 310, as described herein, which may be configured to distribute incident light over each respective waveguide for output toward the eye 210. Light exits the output surfaces 410, 420, 430, 440, 450 of the image injection devices 360, 370, 380, 390, 400 and is injected into the corresponding input surfaces 460, 470, 480, 490, 500 of the waveguides 270, 280, 290, 300, 310. In some embodiments, each of the input surfaces 460, 470, 480, 490, 500 may be an edge of the corresponding waveguide, or may be a portion of a major surface of the corresponding waveguide (that is, one of the waveguide surfaces that directly faces the world 510 or the viewer's eye 210). In some embodiments, a single beam (e.g., a collimated beam) may be injected into each waveguide to output an entire field of cloned collimated beams directed toward the eye 210 at a particular angle (and divergence) corresponding to the depth plane associated with the particular waveguide. In some embodiments, a single one of the image injection devices 360, 370, 380, 390, 400 may be associated with multiple (eg, three) of the waveguides 270, 280, 290, 300, 310 and inject light therein.

[0082] In some embodiments, the image injection devices 360, 370, 380, 390, 400 are discrete displays that each generate image information for injection into the corresponding waveguides 270, 280, 290, 300, 310, respectively. In some other embodiments, the image injection devices 360, 370, 380, 390, 400 are outputs of a single multiplexed display that can deliver image information to each of the image injection devices 360, 370, 380, 390, 400, for example, via one or more light conduits (such as fiber optic cables). It should be understood that the image information provided by the image injection devices 360, 370, 380, 390, 400 can include light of different wavelengths or colors (e.g., different component colors as described herein).

[0083] In some embodiments, the light injected into the waveguides 270, 280, 290, 300, 310 is encoded with image information and provided by the light projection system 1010, as further discussed herein. In some embodiments, the light projection system 1010 may include one or more emissive pixel arrays. It should be understood that the emissive pixel arrays may each include a plurality of light-emitting pixels that may be configured to emit light of different intensities and colors. It should be understood that the image injection devices 360, 370, 380, 390, 400 are schematically shown, and in some embodiments, these image injection devices may present different light paths and positions in a common projection system configured to output light to the associated waveguides in the waveguides 270, 280, 290, 300, 310. In some embodiments, the waveguide of the waveguide assembly 260 may be used as an ideal lens while relaying the light injected into the waveguide to the user's eyes. In this concept, the object may be an array of pixels of the light projection system 1010, and the image may be an image on a depth plane.

[0084] The controller 560 controls the operation of one or more stacked waveguide assemblies 260, including the operation of the image injection devices 360, 370, 380, 390, 400, and the light projection system 1010. In some embodiments, the controller 560 is part of the local data processing module 140. The controller 560 includes programming (e.g., instructions in a non-transitory medium) that regulates the timing and provision of image information to the waveguides 270, 280, 290, 300, 310 according to, for example, any of the various schemes disclosed herein. In some embodiments, the controller can be a single integral device, or a distributed system connected by wired or wireless communication channels. In some embodiments, the controller 560 can be a processing module 140 or 150 ( Fig.9D ) part.

[0085] Continue to refer Figure 6, the waveguides 270, 280, 290, 300, 310 can be configured to propagate light within each respective waveguide by total internal reflection (TIR). The waveguides 270, 280, 290, 300, 310 can each be planar or have another shape (e.g., curved) with major top and bottom surfaces and edges extending between those major top and bottom surfaces. In the illustrated configuration, the waveguides 270, 280, 290, 300, 310 can each include an outcoupling optical element 570, 580, 590, 600, 610 configured to extract light from the waveguide by redirecting light propagating within each respective waveguide out of the waveguide to output image information to the eye 210. The extracted light can also be referred to as outcoupled light, and the outcoupling optical element can also be referred to as a light extraction optical element. The extracted light beam can be output by the waveguide at the location where the light propagating in the waveguide hits the light extraction optical element. As further discussed herein, the out-coupling optical element 570, 580, 590, 600, 610 may be, for example, a grating including a diffractive optical feature. Although shown as being disposed on the bottom major surface of the waveguide 270, 280, 290, 300, 310, for ease of description and clarity of drawing, in some embodiments, as further discussed herein, the out-coupling optical element 570, 580, 590, 600, 610 may be disposed on the top and / or bottom major surface, and / or may be disposed directly in the volume of the waveguide 270, 280, 290, 300, 310. In some embodiments, the out-coupling optical element 570, 580, 590, 600, 610 may be formed as a material layer that is attached to a transparent substrate to form the waveguide 270, 280, 290, 300, 310. In some other embodiments, the waveguides 270, 280, 290, 300, 310 may be a single piece of material and the outcoupling optical elements 570, 580, 590, 600, 610 may be formed on a surface and / or in the interior of the piece of material.

[0086] Continue to refer Figure 6As described herein, each waveguide 270, 280, 290, 300, 310 is configured to output light to form an image corresponding to a particular depth plane. For example, the waveguide 270 closest to the eye can be configured to deliver collimated light (which is injected into such waveguide 270) to the eye 210. The collimated light can represent an optical infinity focal plane. The next waveguide up 280 can be configured to send the collimated light through a first lens 350 (e.g., a negative lens) before the collimated light can reach the eye 210; such first lens 350 can be configured to produce a slightly convex wavefront curvature so that the eye / brain interprets the light from the next waveguide up 280 as coming from a first focal plane closer inward toward the eye 210 from optical infinity. Similarly, the third upward waveguide 290 has its output light pass through both the first lens 350 and the second lens 340 before reaching the eye 210; the combined optical power of the first 350 and second 340 lenses can be configured to produce another incremental wavefront curvature so that the eye / brain interprets the light from the third waveguide 290 as coming from a second focal plane that is closer inward toward the person from optical infinity than the light from the next waveguide upward 280.

[0087] The other waveguide layers 300, 310 and lenses 330, 320 are similarly configured, with the highest waveguide 310 in the stack sending its output through all the lenses between it and the eye to represent the aggregate focal power closest to the person's focal plane. To compensate for the stack of lenses 320, 330, 340, 350 when viewing / interpreting light from the world 510 on the other side of the stacked waveguide assembly 260, a compensating lens layer 620 can be provided on top of the stack to compensate for the aggregate focal power of the underlying lens stack 320, 330, 340, 350. Such a configuration provides as many perceived focal planes as there are waveguide / lens pairs available. Both the outcoupling optics of the waveguide and the focusing aspects of the lens can be static (i.e., not dynamic or electrically activated). In some alternative embodiments, one or both of the above can be dynamic using electrically activated features.

[0088] In some embodiments, two or more of the waveguides 270, 280, 290, 300, 310 may have the same associated depth plane. For example, a plurality of the waveguides 270, 280, 290, 300, 310 may be configured to output images set to the same depth plane, or a plurality of subsets of the waveguides 270, 280, 290, 300, 310 may be configured to output images set to the same plurality of depth planes, with one set for each depth plane. This may provide advantages for forming tiled images to provide an expanded field of view at those depth planes.

[0089] Continue to refer Figure 6 , the out-coupling optical elements 570, 580, 590, 600, 610 can be configured to both redirect light out of its corresponding waveguide and output the light with an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes can have different configurations of out-coupling optical elements 570, 580, 590, 600, 610 that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, the light extraction optical elements 570, 580, 590, 600, 610 can be volume or surface features that can be configured to output light at specific angles. For example, the light extraction optical elements 570, 580, 590, 600, 610 can be volume holograms, surface holograms, and / or diffraction gratings. In some embodiments, the features 320, 330, 340, 350 may not be lenses; instead, they may simply be spacers (e.g., cladding and / or structures used to form air gaps).

[0090] In some embodiments, the outcoupling optical elements 570, 580, 590, 600, 610 are diffractive features that form a diffraction pattern, or "diffractive optical elements" (also referred to herein as "DOEs"). Preferably, the DOE has a sufficiently low diffraction efficiency that only a portion of the beam is deflected out toward the eye 210 at each intersection of the DOE, while the remainder continues to move through the waveguide via TIR. The light carrying the image information is thus split into multiple related exit beams that exit the waveguide at multiple locations, and the result is a fairly uniform pattern of exit toward the eye 210 for that particular collimated beam that bounces within the waveguide.

[0091] In some embodiments, one or more DOEs can be switched between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE can include a polymer dispersed liquid crystal layer in which a droplet includes a diffraction pattern in a host medium, and the refractive index of the droplet can be switched to substantially match that of the host material (in which case the pattern does not significantly diffract incident light), or the droplet can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0092] In some embodiments, a camera assembly 630 (e.g., a digital camera including visible light and infrared light cameras) can be provided to capture images of the eye 210 and / or tissue surrounding the eye 210, for example, to detect user input and / or monitor the physiological state of the user. As used herein, a camera can be any image capture device. In some embodiments, the camera assembly 630 can include an image capture device and a light source to project light (e.g., infrared light) into the eye, which can then be reflected by the eye and detected by the image capture device. In some embodiments, the camera assembly 630 can be attached to the frame 80 ( Fig.9D ), and may be in electrical communication with processing modules 140 and / or 150, which may process image information from camera assembly 630. In some embodiments, one camera assembly 630 may be used for each eye to monitor each eye separately.

[0093] Reference now Figure 7 , shows an example of an outgoing beam output by a waveguide. One waveguide is shown, but it should be understood that the waveguide assembly 260 ( Figure 6 ) can function similarly, where the waveguide assembly 260 includes multiple waveguides. Light 640 is injected into the waveguide 270 at the input surface 460 of the waveguide 270 and propagates within the waveguide 270 by TIR. At the point where the light 640 is incident on the DOE 570, a portion of the light exits the waveguide as an exit beam 650. The exit beams 650 are shown as being substantially parallel, but, as described herein, they may also be redirected to propagate to the eye 210 at a certain angle depending on the depth plane associated with the waveguide 270 (e.g., forming diverging exit beams). It should be understood that substantially parallel exit beams may be indicative of a waveguide having an out-coupling optical element that out-couples light to form an image that appears to be disposed at a depth plane at a large distance (e.g., optical infinity) from the eye 210. Other waveguides or other outcoupling optics may output a more divergent exit beam pattern that would require the eye 210 to adapt to a closer distance to focus it on the retina and would be interpreted by the brain as light coming from a distance closer to the eye 210 than optical infinity.

[0094] In some embodiments, a full color image may be formed at each depth plane by overlaying an image in each component color, such as three or more component colors. Figure 8An example of a stacked waveguide assembly is shown, wherein each depth plane includes an image formed using multiple different component colors. The illustrated embodiment shows depth planes 240a-240f, but more or fewer depths may also be considered. Each depth plane may have three or more component color images associated with it, including: a first image of a first color G; a second image of a second color R; and a third image of a third color B. In the figure, different depth planes are represented by different numbers of diopters (dpt) after the letters G, R, and B. As an example, the numbers after each of these letters indicate the diopters (1 / m) or the inverse of the distance of the depth plane from the viewer, and each box in the figure represents a separate component color image. In some embodiments, in order to account for differences in the eye's focus on light of different wavelengths, the exact location of the depth planes for different component colors may be different. For example, different component color images for a given depth plane may be placed on depth planes corresponding to different distances from the user. Such an arrangement can increase visual acuity and user comfort and / or can reduce chromatic aberration.

[0095] In some embodiments, light of each component color may be output by a single dedicated waveguide, and therefore, each depth plane may have multiple waveguides associated with it. In such embodiments, each box in the figure including the letters G, R, or B may be understood to represent a separate waveguide, and three waveguides may be provided per depth plane, with three component color images provided per depth plane. Although the waveguides associated with each depth plane are shown as being adjacent to each other in the figure for ease of description, it should be understood that in a physical device, the waveguides may all be arranged in a stack with one waveguide per level. In some other embodiments, the same waveguide may output multiple component colors, such that, for example, only a single waveguide may be provided per depth plane.

[0096] Continue to refer Figure 8 , in some embodiments, G is green, R is red, and B is blue. In some other embodiments, other colors associated with other wavelengths of light may be used in addition to or in place of one or more of red, green, or blue, including magenta and cyan.

[0097] It should be understood that throughout this disclosure, references to light of a given color will be understood to encompass light of one or more wavelengths within a range of wavelengths perceived by a viewer as light of the given color. For example, red light may include light of one or more wavelengths within a range of about 620-780 nm, green light may include light of one or more wavelengths within a range of about 492-577 nm, and blue light may include light of one or more wavelengths within a range of about 435-493 nm.

[0098] In some embodiments, the light projection system 1010 ( Figure 6 ) can be configured to emit one or more wavelengths of light outside the visual range of the viewer, for example, infrared and / or ultraviolet wavelengths. Additionally, the in-coupling, out-coupling, and other light redirection structures of the waveguides of the display 250 can be configured to direct this light toward the user's eye 210 and emit it out of the display, for example, for imaging and / or user stimulation applications.

[0099] Reference now Fig.9A In some embodiments, it may be desirable to redirect light impinging on a waveguide to in-couple the light into the waveguide. In-coupling optical elements may be used to redirect and in-couple light into its corresponding waveguide. Fig.9A A cross-sectional side view of an example of a plurality of stacked waveguides or stacked waveguide groups 660 is shown, each stacked waveguide including an in-coupling optical element. The waveguides can each be configured to output light of one or more different wavelengths or one or more different wavelength ranges. It should be understood that stack 660 can correspond to stack 260 ( ) except that light from one or more of the image injection devices 360, 370, 380, 390, 400 is injected into the waveguide from a location where light needs to be redirected for in-coupling. Figure 6 ), and the waveguide shown in stack 660 can correspond to a portion of multiple waveguides 270, 280, 290, 300, 310.

[0100] The stacked waveguide group 660 shown includes waveguides 670, 680, and 690. Each waveguide includes an associated incoupling optical element (which may also be referred to as a light input region on the waveguide), for example, an incoupling optical element 700 disposed on a major surface (e.g., upper major surface) of the waveguide 670, an incoupling optical element 710 disposed on a major surface (e.g., upper major surface) of the waveguide 680, and an incoupling optical element 720 disposed on a major surface (e.g., upper major surface) of the waveguide 690. In some embodiments, one or more of the incoupling optical elements 700, 710, 720 may be disposed on the bottom major surface of the corresponding waveguide 670, 680, 690 (particularly where one or more of the incoupling optical elements are reflective deflecting optical elements). As shown, the incoupling optical elements 700, 710, 720 may be disposed on the upper major surface of their corresponding waveguides 670, 680, 690 (or the top of the next lower waveguide), particularly where those incoupling optical elements are transmissive deflecting optical elements. In some embodiments, the in-coupling optical elements 700, 710, 720 may be disposed in the body of the respective waveguides 670, 680, 690. In some embodiments, as discussed herein, the in-coupling optical elements 700, 710, 720 are wavelength selective such that they selectively redirect light of one or more wavelengths while transmitting light of other wavelengths. Although shown on one side or corner of their respective waveguides 670, 680, 690, it should be understood that in some embodiments, the in-coupling optical elements 700, 710, 720 may be disposed in other areas of their respective waveguides 670, 680, 690.

[0101] As shown, the in-coupling optical elements 700, 710, 720 can be offset from each other in the lateral direction. In some embodiments, each in-coupling optical element can be offset so that it receives light without the light passing through another in-coupling optical element. Figure 6 As shown in , each in-coupling optical element 700, 710, 720 can be configured to receive light from a different image injection device 360, 370, 380, 390 and 400, and can be separated from the other in-coupling optical elements 700, 710, 720 (e.g., laterally spaced apart) so that it does not substantially receive light from the other in-coupling optical elements 700, 710, 720.

[0102] Each waveguide further includes an associated light distribution element, such as a light distribution element 730 disposed on a major surface (e.g., top major surface) of the waveguide 670, a light distribution element 740 disposed on a major surface (e.g., top major surface) of the waveguide 680, and a light distribution element 750 disposed on a major surface (e.g., top major surface) of the waveguide 690. In some other embodiments, the light distribution elements 730, 740, 750 may be disposed on the bottom major surface of the associated waveguides 670, 680, 690, respectively. In some other embodiments, the light distribution elements 730, 740, 750 may be disposed on both the top major surface and the bottom major surface of the associated waveguides 670, 680, 690, respectively; or the light distribution elements 730, 740, 750 may be disposed on different major surfaces of the top major surface and the bottom major surface in different associated waveguides 670, 680, 690, respectively.

[0103] The waveguides 670, 680, 690 may be spaced apart and separated, for example, by layers of gas, liquid, and / or solid material. For example, as shown, layer 760a may separate waveguides 670 and 680; and layer 760b may separate waveguides 680 and 690. In some embodiments, layers 760a and 760b are formed of a low refractive index material (that is, a material having a lower refractive index than the material forming the waveguides 670, 680, 690 that is immediately adjacent to the waveguide). Preferably, the refractive index of the material forming layers 760a, 760b is 0.05 or greater, or 0.10 or less, of the refractive index of the material forming waveguides 670, 680, 690. Advantageously, the lower refractive index layers 760a, 760b may serve as cladding that facilitates total internal reflection (TIR) ​​of light passing through the waveguides 670, 680, 690 (e.g., TIR between the top and bottom major surfaces of each waveguide). In some embodiments, the layers 760a, 760b are formed of air. Although not shown, it is understood that the top and bottom of the illustrated waveguide set 660 may include an adjacent cladding layer.

[0104] Preferably, for ease of manufacturing and other considerations, the materials forming waveguides 670, 680, 690 are similar or the same, and the materials forming layers 760a, 760b are similar or the same. In some embodiments, the materials forming waveguides 670, 680, 690 may be different, and / or the materials forming layers 760a, 760b may be different between one or more waveguides while still maintaining the various refractive index relationships described above.

[0105] Continue to refer Fig.9A , light rays 770, 780, 790 are incident on the waveguide group 660. It should be understood that the image can be injected by one or more image injection devices 360, 370, 380, 390, 400 ( Figure 6)Inject light 770, 780, 790 into waveguides 670, 680, 690.

[0106] In some embodiments, the light rays 770, 780, 790 are intended for different waveguides (e.g., waveguides configured to output light with different amounts of wavefront divergence and / or configured to output light with different characteristics (e.g., different wavelengths or colors). Thus, in some embodiments, the light rays 770, 780, 790 can have different characteristics, such as different wavelengths or different ranges of wavelengths, which can correspond to different colors. The in-coupling optical elements 700, 710, 720 each deflect incident light so that the light propagates through a corresponding waveguide of the waveguides 670, 680, 690 by TIR. In some embodiments, the in-coupling optical elements 700, 710, 720 each selectively deflect light of one or more specific wavelengths while transmitting other wavelengths to the underlying waveguide and associated in-coupling optical element.

[0107] For example, the incoupling optical element 700 can be configured to deflect light 770 having a first wavelength or wavelength range while transmitting light 780 and 790 having different second and third wavelengths or wavelength ranges, respectively. The transmitted light 780 impinges upon and is deflected by the incoupling optical element 710, which is configured to deflect light of the second wavelength or wavelength range. The light 790 is deflected by the incoupling optical element 720, which is configured to selectively deflect light of the third wavelength or wavelength range.

[0108] Continue to refer Fig.9A , the deflected light rays 770, 780, 790 are deflected so that they propagate through the corresponding waveguides 670, 680, 690; that is, the incoupling optical element 700, 710, 720 of each waveguide deflects light into the corresponding waveguide 670, 680, 690 to incouple the light into the corresponding waveguide. The light rays 770, 780, 790 are deflected at an angle to propagate through the corresponding waveguide 670, 680, 690 by TIR. The light rays 770, 780, 790 propagate through the corresponding waveguide 670, 680, 690 by TIR until they impinge on the corresponding light distribution element 730, 740, 750 of the waveguide.

[0109] Reference now Fig. 9B , showing Fig.9A6. As described above, the in-coupled light rays 770, 780, 790 are deflected by the in-coupling optical elements 700, 710, 720, respectively, and then propagate through TIR within the waveguides 670, 680, 690, respectively. The light rays 770, 780, 790 then impinge on the light distribution elements 730, 740, 750, respectively. The light distribution elements 730, 740, 750 deflect the light rays 770, 780, 790 so that they propagate toward the out-coupling optical elements 800, 810, 820, respectively.

[0110] In some embodiments, the light distribution elements 730, 740, 750 are orthogonal pupil expanders (OPEs). In some embodiments, the OPEs deflect or distribute light to the out-coupling optical elements 800, 810, 820, and in some embodiments, may also increase the beam or spot size of the light as it propagates to the out-coupling optical elements. In some embodiments, the light distribution elements 730, 740, 750 may be omitted, and the in-coupling optical elements 700, 710, 720 may be configured to deflect light directly to the out-coupling optical elements 800, 810, 820. For example, referring to Fig.9A , the light distribution elements 730, 740, 750 can be replaced by outcoupling optical elements 800, 810, 820, respectively. In some embodiments, the outcoupling optical elements 800, 810, 820 are exit pupils (EP) or exit pupil expanders (EPE) that direct light into the viewer's eye 210 ( Figure 7 ). It should be understood that the OPE may be configured to increase the size of the eyebox in at least one axis, while the EPE may be configured to increase the eyebox in an axis that intersects (e.g., is orthogonal) to the axis of the OPE. For example, each OPE may be configured to redirect a portion of the light that strikes the OPE to the EPE of the same waveguide, while allowing the remainder of the light to continue to propagate down the waveguide. Upon striking the OPE again, another portion of the remaining light is redirected to the EPE, and the remainder of that portion continues to propagate further down the waveguide, and so on. Similarly, upon striking the EPE, a portion of the light that strikes is directed outward from the waveguide toward the user, and the remainder of that light continues to propagate through the waveguide until it strikes the EP again, at which time another portion of the light that strikes is directed out of the waveguide, and so on. Therefore, each time a portion of the light is redirected by an OPE or EPE, a single in-coupled beam can be "copied," thereby forming a field of cloned beams, such as Figure 6 As shown in . In some embodiments, the OPE and / or EPE can be configured to modify the size of the light beam.

[0111] Accordingly, refer to Fig.9A and Fig. 9B, in some embodiments, the waveguide set 660 includes waveguides 670, 680, 690; in-coupling optical elements 700, 710, 720; light distribution elements (e.g., OPE) 730, 740, 750; and out-coupling optical elements (e.g., EP) 800, 810, 820 for each component color. The waveguides 670, 680, 690 can be stacked with an air gap / cladding between each waveguide. The in-coupling optical elements 700, 710, 720 redirect or deflect the incident light (with different in-coupling optical elements receiving light of different wavelengths) into their waveguides. The light then propagates at an angle that will cause TIR within the corresponding waveguide 670, 680, 690. In the example shown, light ray 770 (e.g., blue light) is deflected by the first in-coupling optical element 700 and then continues to bounce down the waveguide in the manner previously described, interacting with the light distribution element (e.g., OPE) 730 and then the out-coupling optical element (e.g., EP) 800. Light rays 780 and 790 (e.g., green and red, respectively) will pass through the waveguide 670, with light ray 780 impinging on and being deflected by the in-coupling optical element 710. Light ray 780 then bounces down the waveguide 680 via TIR, continues to its light distribution element (e.g., OPE) 740, and then to the out-coupling optical element (e.g., EP) 810. Finally, light ray 790 (e.g., red light) passes through the waveguide 690 and impinges on the light in-coupling optical element 720 of the waveguide 690. The light in-coupling optical element 720 deflects the light ray 790 so that it propagates by TIR to the light distribution element (e.g., OPE) 750 and then propagates by TIR to the out-coupling optical element (e.g., EP) 820. The out-coupling optical element 820 then ultimately out-couples the light ray 790 to the viewer, which also receives out-coupled light from the other waveguides 670, 680.

[0112] Fig. 9C Shows Fig.9A and Fig. 9BA top plan view of an example of a plurality of stacked waveguides. As shown, the waveguides 670, 680, 690 and the associated light distribution elements 730, 740, 750 and the associated outcoupling optical elements 800, 810, 820 of each waveguide can be vertically aligned. However, as described herein, the incoupling optical elements 700, 710, 720 are not vertically aligned; instead, the incoupling optical elements are preferably non-overlapping (e.g., laterally spaced apart as seen in the top plan view). As further discussed herein, this non-overlapping spatial arrangement facilitates one-to-one injection of light from different resources into different waveguides, thereby allowing a specific light source to be uniquely coupled to a specific waveguide. In some embodiments, arrangements including non-overlapping spatially separated incoupling optical elements can be referred to as shifted pupil systems, and the incoupling optical elements within these arrangements can correspond to sub-pupils.

[0113] Fig.9D An example of a wearable display system 60 is shown into which the various waveguides and related systems disclosed herein may be integrated. In some embodiments, the display system 60 is Figure 6 System 250, Figure 6 Some parts of the system 60 are schematically shown in more detail. For example, Figure 6 The waveguide assembly 260 can be part of the display 70.

[0114] Continue to refer Fig.9D, the display system 60 includes a display 70, and various mechanical and electronic modules and systems that support the functions of the display 70. The display 70 can be coupled to a frame 80, which can be worn by a display system user or viewer 90, and is configured to position the display 70 in front of the eyes of the user 90. In some embodiments, the display 70 can be considered as glasses. In some embodiments, a speaker 100 is coupled to the frame 80 and is configured to be positioned near the ear canal of the user 90 (in some embodiments, another speaker not shown can be optionally positioned near the other ear canal of the user to provide stereo / shaped sound control). The display system 60 may also include one or more microphones 110 or other devices that detect sound. In some embodiments, the microphone is configured to allow the user to provide input or commands to the system 60 (e.g., selection of voice menu commands, natural language questions, etc.), and / or may allow audio communication with other people (e.g., other users of similar display systems). The microphone can be further configured as a peripheral sensor to collect audio data (e.g., sounds from the user and / or the environment). In some embodiments, the display system may further include a peripheral sensor 120a, which may be separated from the frame 80 and attached to the body of the user 90 (e.g., the head, torso, limbs, etc. of the user 90). In some embodiments, the peripheral sensor 120a may be configured to obtain data representing the physiological state of the user 90. For example, the sensor 120a may be an electrode.

[0115] Continue to refer Fig.9D, the display 70 is operably coupled to a local data processing module 140 via a communication link 130 (such as via a wired conductor or a wireless connection), which can be mounted in various configurations, such as fixedly attached to the frame 80, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user 90 (e.g., in a backpack configuration, in a belt-coupled configuration). Similarly, the sensor 120a can be operably coupled to the local processor and data module 140 via a communication link 120b (e.g., a wired conductor or a wireless connection). The local processing and data module 140 can include a hardware processor and digital memory, such as non-volatile memory (e.g., flash memory or a hard drive), both of which can be used to assist in the processing, caching, and storage of data. Optionally, the local processor and data module 140 can include one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, etc. The data may include data that is: a) captured from a sensor (e.g., that may be operably coupled to the frame 80 or otherwise attached to the user 90), such as an image capture device (such as a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, a gyroscope, and / or other sensors disclosed herein; and / or b) obtained and / or processed using the remote processing module 150 and / or the remote data repository 160 (including data related to virtual content), possibly communicated to the display 70 after such processing or retrieval. The local processing and data module 140 may be operably coupled to the remote processing module 150 and the remote data repository 160 by communication links 170, 180, such as via a wired or wireless communication link, so that these remote modules 150, 160 are operably coupled to each other and can be used as a resource for the local processing and data module 140. In some embodiments, the local processing and data module 140 may include one or more image capture devices, microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes. In some other embodiments, one or more of these sensors may be attached to the frame 80, or may be a stand-alone structure that communicates with the local processing and data module 140 via a wired or wireless communication path.

[0116] Continue to refer Fig.9DIn some embodiments, the remote processing module 150 may include one or more processors configured to analyze and process data and / or image information, such as one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, etc. In some embodiments, the remote data repository 160 may include a digital data storage facility that may be available via the Internet or other network configurations in a "cloud" resource configuration. In some embodiments, the remote data repository 160 may include one or more remote servers that provide information to the local processing and data module 140 and / or the remote processing module 150, such as information for generating augmented reality content. In some embodiments, all data is stored in the local processing and data module, and all calculations are performed in the local processing and data module, thereby allowing full autonomous use from the remote module. Optionally, an external system (e.g., a system of one or more processors, one or more computers) including a CPU, GPU, etc. may perform at least a portion of the processing (e.g., generating image information, processing data), and provide information to the modules 140, 150, 160, and receive information from the modules 140, 150, 160, such as via a wireless or wired connection.

[0117] C. User Input Example

[0118] Fig. 10A and 10B Examples of user input received via controller buttons or input areas on a user input device are shown. In particular, Fig. 10A and Fig. 10B Controller 3900 is shown, which may be part of a wearable system disclosed herein and may include a home button 3902, a trigger 3904, a bumper 3906, and a touch pad 3908. In various embodiments of the wearable system, a user input device or totem may be used as controller 3900.

[0119] Potential user input that may be received by the controller 3900 includes, but is not limited to, pressing and releasing the main button 3902; half and full presses (and other partial presses) of the trigger 3904; releasing the trigger 3904; pressing and releasing the stop rubber 3906; touching, moving while touching, releasing touch, increasing or decreasing touch pressure, touching a particular portion (e.g., the edge of the touch pad 3908), or making a gesture on the touch pad 3908 (e.g., by drawing a shape with a thumb).

[0120] Fig. 10A and Fig. 10BVarious examples of user input that can be received and recognized by the system are shown. User input can be received through one or more user input modes (alone or in combination, as shown). User input can include input through controller buttons (e.g., main button 3902, trigger 3904, stop rubber 3906, and touch pad 3908); physical movement of controller 3900 or HMD 3910; eye gaze direction; head posture direction; gestures; voice input; etc.

[0121] like Fig. 10A As shown, a short press and release of the main button 3902 can indicate a main tapping action, while a long press of the main button 3902 can indicate a main press & hold action. Similarly, a short press and release of the trigger 3904 or the stop rubber 3906 can indicate a trigger tapping action or a stop rubber tapping action, respectively; while a long press of the trigger 3904 or the stop rubber 3906 can indicate a trigger press action or a stop rubber press action, respectively.

[0122] like Fig. 10B As shown, the touch of the touch pad 3908 moving on the touch pad can indicate a touch drag action. A short touch and release of the touch pad 3908, wherein the touch is substantially not moved, can indicate a tapping action. If this short touch and release of the touch pad 3908 is completed with a force exceeding a certain threshold level (which can be a predetermined threshold, a dynamically determined threshold, a learned threshold, or some combination thereof), the input can indicate a forceful tapping input. The touch of the touch pad 3908 with a force greater than the threshold level can indicate a forceful pressing action, and a long touch with this force can indicate a forceful pressing input. The touch near the edge of the touch pad 3908 can indicate an edge pressing action. In some embodiments, the edge pressing action can also involve an edge touch exceeding a threshold level pressure. Fig. 10B It is also shown that a touch moving in an arc on the touch pad 3908 can indicate a touch circle action.

[0123] Fig. 10C Examples of user input received through physical movement of a controller or head mounted device (HMD) are shown. Fig. 10C As shown, physical movement of the controller 3900 and the head mounted display 3910 (HMD) can form the user's input to the system. The HMD 3910 may include Fig.9D, 110. In some embodiments, the controller 3900 provides three degrees of freedom (3DOF) input by identifying rotation of the controller 3900 in any direction. In other embodiments, the controller 3900 also provides six degrees of freedom (6DOF) input by identifying translation of the controller in any direction. In other embodiments, the controller 3900 can provide inputs less than 6DOF or less than 3DOF. Similarly, the head mounted display 3910 can recognize and receive 3DOF, 6DOF, less than 6DOF, or less than 3DOF inputs.

[0124] Fig. 10D shows an example of how user inputs can have different durations. Fig. 10D As shown, some user inputs may have a shorter duration (e.g., a duration less than a fraction of a second, such as 0.25 seconds) or may have a longer duration (e.g., a duration greater than a fraction of a second, such as more than 0.25 seconds). In at least some embodiments, the duration of the input itself can be recognized by the system and used as input. The wearable system can process short duration and long duration inputs differently. For example, a short duration input can represent the selection of an object, while a long duration input can represent the activation of an object (e.g., causing the execution of an application associated with the object).

[0125] Fig.11A and Fig. 11B Various examples of user input that may be received and recognized by the system are shown. User input may be received through one or more user input modes (alone or in combination, as shown). User input may include input through controller buttons (e.g., home button 3902, trigger 3904, stop rubber 3906, and touch pad 3908); physical movement of controller 3900 or HMD 3910; eye gaze direction; head posture direction; gestures; voice input; etc.

[0126] like Fig.11A As shown, a short press and release of the main button 3902 can indicate a main tapping action, while a long press of the main button 3902 can indicate a main pressing action. Similarly, a short press and release of the trigger 3904 or the stop rubber 3906 can indicate a trigger tapping action or a stop rubber tapping action, respectively; while a long press of the trigger 3904 or the stop rubber 3906 can indicate a trigger pressing action or a stop rubber pressing action, respectively.

[0127] like Fig. 11BAs shown, the touch of the touch pad 3908 moving on the touch pad can indicate a touch drag action. A short touch and release of the touch pad 3908, wherein the touch is substantially not moved, can indicate a tapping action. If this short touch and release of the touch pad 3908 is completed with a force exceeding a certain threshold level (which can be a predetermined threshold, a dynamically determined threshold, a learned threshold, or some combination thereof), the input can indicate a hard tap input. The touch of the touch pad 3908 with a force greater than the threshold level can indicate a hard press action, and a long touch with this force can indicate a hard hold input. The touch near the edge of the touch pad 3908 can indicate an edge press action. In some embodiments, the edge press action can also involve an edge touch exceeding a threshold level pressure. Fig. 11B It is also shown that a touch moving in an arc on the touch pad 3908 can indicate a touch circle action.

[0128] Other example user inputs may include an input device (e.g., a user's hand) or mechanism that may have six degrees of freedom. Advantageously, using the user's hand as input may improve the user experience by allowing the user to make intuitive pointing or other input gestures to provide information to the AR system. In the case of object interaction and control, using the user's hand or other pointing device may help the user more intuitively adapt to moving objects and aiming objects based on the position or orientation of the user's hand (or other pointing device). Therefore, discrete button activation may not be required. However, in some examples, a combination of discrete button activation and pointing using a six-degree-of-freedom input device may be used.

[0129] D. Single controller content movement and interaction

[0130] Some applications implemented by augmented reality (AR) or virtual reality (VR) systems may include interactive and / or movable virtual content that accepts user input through, for example, user head gestures, body gestures, eye gaze, controller input, etc., or a combination thereof. For example, an application may have a virtual control menu in which a user can select, highlight, or otherwise interact with a menu or information associated with a menu. In another example, an application may have a web browser in which a user can enter information within a browser window or control information using interactive features such as refresh, home, or control buttons. The application may also allow interactive virtual content to move in the user's environment as the user moves in their environment or through the user's active input. However, it may be difficult and awkward for the user to interact with content and move content in their environment using the sometimes limited controls provided by the VR or AR system. For example, if the user wants to interact with the interactive content, they can provide a set of inputs, and if the user wants to move the interactive content, they can provide a second set of inputs. Combining these two sets of inputs may be awkward and uncomfortable for the user. Systems and methods are described herein for simplifying movement of and / or simplifying interaction with interactable virtual content in order to reduce the burden of interacting with the interactable virtual content.

[0131] One way to facilitate user interaction and content movement within a user environment is to separate the action of interaction (e.g., within a user interface) and the movement (e.g., of the user interface). For example, an AR system may place content at a fixed location within the user environment and then accept input from the user to interact with the placed content based on user gestures, controllers, user gaze, etc., or some combination thereof. However, placing an interaction space (e.g., a prism containing a bounded volume of interactive content) at a set location within the user environment is not always ideal in a virtual or augmented reality environment. For example, in a virtual or augmented reality experience, the user may be moving around in their 3D environment. If the interaction space was fixed at a set location, then the user would have to return to the location of the interaction space in order to input information.

[0132] Another way to facilitate user interaction and movement of content is to utilize separate controllers for interaction and movement. For example, an AR system may equip each hand of a user with a controller. A first controller in the user's first hand may move the interaction space within the user's 3D environment, while a second controller in the user's second hand may select content within the interaction space. However, using two controllers in this way can result in an awkward and uncomfortable user experience, in part because it requires dexterous movements that users are typically not accustomed to.

[0133] Disclosed herein are systems and methods for interactive and mobile mechanisms that allow content placement and interaction using a single controller. The interactive and mobile mechanisms can allow the controller to move at different amplitudes to affect different aspects of the system input. For example, small movements within an interactive content object (e.g., a six-degree-of-freedom pointing device) can be used to aim or select content within the interactive content object. Larger movements outside the interactive content object can cause the position of the interactive content object to follow (or update / respond to) the control device. Advantageously, such a system can allow users to carry content with them as they move in their 3D environment and selectively interact with the content using the same controller.

[0134] In general, the systems and methods described herein allow a user to move a cursor or pointer (e.g., in response to movement of a handheld controller) within an interactive content object (e.g., within a prism) to interact with the interactive content object. Once the user moves the cursor or pointer outside of the interactive content object (or outside of some additional border area around the interactive content object), the functionality of the controller movement is updated to move the position of the interactive content object to correspond to the movement of the controller, rather than attempting to interact with the content of the interactive content object. Thus, the functionality of the controller can be alternating by the user providing movement of the controller. Depending on the embodiment, the threshold movement of the controller can be defined by the position of the controller gesture relative to the plane of the interactive content object and / or the angle between the controller gesture and the plane of the interactive content object.

[0135] Fig. 12A An example content control environment 1200 is shown according to one example of the following and interaction process disclosed herein. For example, a user 1202 may use a wearable device 1204 (e.g., the wearable device 1204 described above) that may allow the user to perceive virtual content in their 3D environment. Fig.9D The wearable device 1204 can display virtual content 1206 at a location in front of the user. The location in front of the user can be controlled by a controller 1208 of the wearable device (e.g., the user's hand or reference FIG. 10A to FIG. 11B1202 . As described in further detail below, if the user moves controller 1208 so that the pointing vector of the controller does not intersect the bounded area or volume 1212 associated with content 1206, the wearable device 1204 can update the position of content 1206 toward the new focus of the controller. If the user moves controller 1208 so that the pointing vector of the controller intersects the bounded area or volume 1212, the wearable device 1204 can allow the user to select and enter information associated with content 1206 without moving the content 1206. Advantageously, when content 1206 moves along with the head posture, the orientation of content 1206 can be adjusted to match the current head posture of user 1202. Therefore, no matter where content 1206 is placed in the user's 3D environment, the content is angled to facilitate viewing under the user's current head posture. For example, in some embodiments, content 1206 can be oriented perpendicular to head posture direction 1210.

[0136] Advantageously, the following and interaction processes disclosed herein can allow content to move and redirect with the user from one location to another as the user moves through their environment. Fig. 12A As shown, the user can have a first orientation and position 1201a. The user can move to a new position 1201b or 1201c and have a new orientation. As the user moves and reorients himself, by tracking the user's head pose and the pointing vector associated with the controller 1208, the AR system can move the position and orientation of the content 1206 and effectively follow the user to the new position 1201b or 1201c. However, in the event that the user makes minor adjustments to the position and orientation of the controller 1206, the AR system can maintain the position of the content 1206 and allow the user to provide input or interact with the content via those minor movements of the controller 1206.

[0137] Fig. 12B A flow chart of an example process 1303 is shown that can be used to control whether the movement of the controller is used to update the position of the content in the user's 3D space or to allow the user to interact with the content. The process 1303 may include a direction determination box 1350, a content boundary determination box 1352, a boundary comparison box 1354, a content movement box 1356, and an input identification box 1358, or some combination of fewer or more boxes.

[0138] In direction determination block 1350, the AR system can determine a pointing vector for a controller, device, gesture, or other input mechanism capable of indicating a direction. The pointing vector can include a direction within the user's 3D environment indicated by one or more input devices, whether the input device is electromechanical (e.g., a totem or handheld controller), mechanical, or an object (e.g., a user's hand, finger, or pen). In some examples, the pointing vector can include a direction indicated by one or more user gestures. In some examples, the pointing vector can include an indicated direction of a handheld controller, such as described above with reference to FIG. FIG. 10A to FIG. 11B In some examples, the pointing vector can be determined by more than one input, such as controller orientation and user eye gaze or other combination of inputs.

[0139] At content boundary determination block 1352, the AR system can determine a boundary of the virtual content. The boundary can be a region of some space including a boundary around the virtual content, some subset of the volume of space (e.g., if the virtual content is very large), or one or more edges of the volume of space associated with the virtual content. In some examples, a content boundary can be associated with more than one piece of virtual content.

[0140] The content boundary can be any shape. For example, the content boundary can be a rectangular prism, a sphere, a truncated cone, or other shapes. In some embodiments, the content boundary may or may not have a shape similar to the shape of the virtual content. For example, if the content is a rectangular menu, the content boundary can be a rectangle. If the content is circular, the content boundary can be a circle or a rectangle. In some examples, the content boundary can be a boundary that is the same or similar to the virtual content. For example, the virtual content can be a prism surrounding a rectangular interactive menu. The prism can have a rectangular prism with a length and height that is equal to or greater than the interactive menu. In some embodiments, the content boundary can be the edge of a rectangular prism.

[0141] Additionally or alternatively, the content border may be the same or may be different in the horizontal and vertical directions. For example, the content border may be further away from the edge of the virtual content in the vertical direction than in the horizontal direction. Thus, the controller may move the content more in the vertical direction than in the horizontal direction, with less change in direction.

[0142] The content boundary may be smaller or larger than the boundary of the virtual content or the prism containing the virtual content. For example, the content boundary may coincide with the boundary of the interactive portion of the virtual content (e.g., a menu tile), and the interactive portion of the virtual content may be smaller than the full size of the virtual content. In some examples, the size or shape of the content boundary may vary based on one or more aspects of the content and / or AR system. For example, the content boundary may be different for different types of virtual content or applications associated with virtual content. In some embodiments, the content boundary may be extended to include a portion or percentage of the user's field of view. In some examples, the content boundary may be a cuboid with 1 / 3 meter, 1 / 2 meter, one meter, or other values. In other examples, the boundary may be 10%, 25%, or other amounts of the user's field of view. In some examples, the virtual content may be resized or adjusted to fit within a set threshold or boundary.

[0143] Fig. 12C shows example content 1206 within bounded volume 1212, Fig.12D A top view of a controller 1208 is shown, with the controller 1208 oriented so that its pointing vector is generally toward content 1206 within a bounded volume 1212. In this example, the bounded volume 1212 is larger than an area 1230 of the content 1206. Advantageously, this additional area allows for padding, where a user can change the direction of the pointing vector of the controller 1208 outside of the immediately adjacent area 1230 of the content 1206 without triggering movement of the content (e.g., without causing the content to follow the movement of the controller). The additional area can be defined by an angular distance 1232 from an edge of the content area 1230. The angular distance 1232 can be any number of degrees, such as 5 degrees, 10 degrees, or 20 degrees. The angular distance 1232 can be different at different edges of the area 1230 of the content 1206. For example, the angular distance 1232 can be greater at the left or right edge of the content area 1230 than at the top or bottom edge of the content area 1230. In another example, angular distance 1232 can be different based on content 1206. For example, content 1206 can be a menu with more interactive content on the left side than on the right side. Since a user interacting with the content is more likely to move the pointing vector toward the side with more interactive content, the AR system can provide greater padding on the left side to help avoid accidentally moving content 1206 to a new location and make it easier for its users to interact with content 1206.

[0144] Continue to refer Fig. 12BAt boundary comparison block 1354, the AR system can determine whether the pointing vector of the controller intersects the content boundary. For example, the pointing vector of the controller can pass inside the content boundary or outside the content boundary. If the pointing vector intersects the content boundary, the AR system can move to block 1358. If the pointing vector does not intersect the content boundary, the AR system can move to block 1356.

[0145] In content move block 1356, the AR system can move the virtual content to a new location. For example, the AR system can move the content toward the pointing vector of the controller. The point can be determined based on one or more factors associated with the AR system or application, as described in further detail below. The AR system can move the content so that the pointing vector just intersects the content boundary, or can move the content so that the pointing vector intersects a specific point within the content, such as the center of the content.

[0146] The AR system can move content at a constant or variable speed. For example, the AR system can move content at a speed calculated based on the distance between the current content location and the desired content location. In some examples, the speed may be faster for greater distances and slower for smaller distances. In some examples, the speed may be variable. For example, the content may move slowly at first, then speed up, and then slow down as it approaches the destination.

[0147] In input identification box 1358, the AR system can receive user input associated with the content. For example, the user can indicate interaction with the interactive content (rather than continued movement of the interactive content associated with the movement of the controller) with a controller (and / or gestures, voice commands, etc.). The AR system can receive the indication and perform an action based on the indication. For example, the content may include a virtual menu with selectable buttons. The AR system can receive an indication from the user to select one or more selectable buttons and perform one or more actions based on the selection.

[0148] E. Example content following action

[0149] As described above, AR systems can control the movement of virtual content within a 3D environment through manipulation of a controller, causing the content to effectively follow the user as the user moves around their space. Fig.13A , Fig. 13B and Fig. 13C Example aspects of an example content movement environment are shown, which may include moving virtual content at content location 1304 using controller 1302 .

[0150] Controller 1302 may be any multi-degree-of-freedom input device. For example, controller 1302 may include a user's hand or a portion thereof, a controller of a wearable device, such as the one described above with reference to FIG. 10A to FIG. 11BThe controller 1302 may include a six-degree-of-freedom device, a three-degree-of-freedom touch input device, or the like, or some combination thereof, that can provide a pointing direction. The controller 1302 may have some combination of sensors, controls, or output components that allow the controller 1302 to indicate a direction. In some examples, the indicated direction may coincide with a direction parallel to an axis of the controller 1302. In some examples, the controller 1302 may indicate a direction using an orientation or position of the controller 1302. The indicated direction may then define the direction of the pointing vector of the controller 1302.

[0151] Virtual content may include one or more prisms, which generally describe a three-dimensional container, area, or volume associated with mixed reality content, which may contain multiple virtual content items, such as representations of 3D objects. Fig. 13B As shown, the prism may include a bounded volume 1318, which may have a width w and a height h and another dimension. The prism in which the content is restricted may be controlled or placed, and the content restricted in the prism may be controlled or placed in the user's environment. As used herein, a virtual object may be or include a prism. Various features, uses, and embodiments of prisms are described in U.S. Patent Publication No. 2019 / 0197785, published on June 27, 2019, which is incorporated herein by reference in its entirety. In one example, the content within the prism may include a menu associated with an application. For example, the virtual content may include a control menu that can accept user input through input at one or more locations within the menu.

[0152] The virtual content can be centered at a location 1304 within the user's 3D environment. Location 1304 can be based on one or more factors associated with the user, the wearable device, or the user's environment. For example, location 1304 can be relative to the location of controller 1302. Controller 1302 can define a coordinate system with an origin at a point on controller 1302. Where location 1304 is relative to controller 1302 or other reference point, location 1304 can fall within a distance range from the origin associated with controller 1302 or other reference point along pointing vector 1312. For example, vector 1322 (e.g., Fig. 13B ) can be defined as from the origin of the controller 1302 to the content location (or a reference point associated with the content), and the position 1304 can be along the vector 1322.

[0153] refer to Fig.13A, when determining the content position 1304, the AR system may calculate or determine the position 1306 of the content along the pointing vector 1312 of the controller 1302. For example, the AR system may define a coordinate system having an origin at the controller 1302. The coordinate system may have a z-axis parallel to the pointing vector 1312 of the controller 1302. The content position 1304 may be defined within the coordinate system of the controller 1302. Thus, the content position 1304 may have a z component (e.g., Fig.13A 13). The z component may correspond to a content position 1306 along a pointing vector 1312 of the controller 1302. If the z component falls below a minimum distance 1307, the AR system may move the content from the position 1304 along the pointing vector 1312 until the z component is greater than or equal to the minimum distance. If the z component falls above a maximum distance 1308, the AR system may move the content from the position 1304 along the pointing vector 1312 until the z component is less than or equal to the maximum distance.

[0154] The content location 1304 may additionally have a reference point with respect to the width w and height h of the bounding volume 1318. The pointing vector 1312 of the controller 1302 may intersect the bounding volume 1318 having the height h and the width w at the intersection point 1314. If the location of the point 1314 falls outside the width w of the bounding volume 1318, the AR system may move the content horizontally from the location 1304 until the distance is within the bounds. If the vertical distance exceeds the bounding height h of the bounding volume 1318, the AR system may move the content vertically from the location 1304 until the distance is within the bounds. Thus, if the controller 1302 is rotated or moved horizontally or vertically within the bounding volume 1318, the location 1304 of the bounding volume 1318 may remain relatively fixed in space. However, if the controller 1302 is moved horizontally or vertically outside the bounding volume 1318, the AR system may move the virtual content and / or the bounding volume 1318 to a new location in the space 1314.

[0155] Fig. 13C Another example content movement environment 1301 is shown. In this example content movement environment 1301, the AR system can maintain content 1362 between a minimum and maximum distance from the controller 1302. For example, if the distance between the content 1362 and the controller 1302 is less than a minimum value, the AR system can set the distance to the minimum value. Additionally or alternatively, if the distance between the content 1262 and the controller 1302 is greater than a maximum value, the AR system can set the distance to the maximum distance. Otherwise, the AR system can maintain the current distance between the content 1362 and the controller 1302 without making any adjustments.

[0156] Additionally or alternatively, if content 1362 is held in front of controller 1302, e.g., so that the controller is pointed at a point within the boundaries of content 1362, the user's ability to easily locate and interact with content 1362 may be improved. To help achieve a maintained distance and allow the user to point to content 1362 or a point within the prism, the AR system may move the content (e.g., a prism or other 3D content item) on the surface of invisible sphere 1364.

[0157] refer to Fig. 13C , invisible sphere 1364 can have center 1360 at point 1360 on controller 1302 (or some other point on the controller). The AR system can convert the location of content 1362 into spherical coordinates with an origin at point 1360. Spherical coordinates can include distance, azimuth, and (eg, rotation in a horizontal plane), and polar angle θ (eg, similar to pitch). In other embodiments, another coordinate system may be used.

[0158] As discussed above, content 1362 (e.g., a prism or other three-dimensional content) can have a set of boundaries. The set of boundaries can also be associated with spherical coordinates referenced to a specified center 1360. In this example, the AR system can determine the outer surface of sphere 1364 with reference to the boundaries of content 1362 (e.g., so that the boundaries of the content remain near the surface of sphere 1364). In one specific example, spherical coordinates are determined, where C represents the controller position and P represents the content (e.g., prism) position, so that: The vector CP from the controller 1302 to the content 1362 is defined as:

[0159] CP=PC

[0160] Wherein the distance between the controller 1302 and the content 1362 is the length of the CP, the azimuth of the vector is the inverse tangent of CP.z / CP.x, the horizontal distance of the CP is the distance of the vector projected onto the XZ plane, which can be calculated as hypot(CP,x,CP.y), and the height of the content position is the inverse tangent of the height / horizontal distance or arctan(CP.y / horizontal distance). In some embodiments, the position of the content 1362 can be an anchor associated with the prism, such as a point at the center of the prism volume or a point at the center of one side of the prism (or other position), for example, it can indicate the position at which the prism and therefore the content can be attached to other real or virtual objects (such as walls, tables, chairs, etc.).

[0161] Advantageously, in some embodiments, as the controller moves, the position of the content 1362 is adjusted to remain on the outer surface of the sphere 1364. For example, if the current azimuth and polar angle of the controller pointing vector falls within the bounded area of ​​the sphere, the AR system can maintain the position of the content 1362. If the current azimuth and / or polar angle of the pointing vector falls outside the bounded area of ​​the sphere, the AR system can update the position of the content 1362. For example, the AR system can determine the azimuth difference between the nearest point on the bounded area and the current azimuth of the CP. The AR system can then move the content 1362 to reduce the difference. Additionally or alternatively, the AR system can adjust the height of the content to reduce the height difference between the content position and the CP.

[0162] In one particular example, the position of the content 1360 can be adjusted with respect to distance (e.g., as noted above with respect to the minimum and maximum distances) as well as with respect to azimuth and altitude. To determine whether the azimuth of the content needs to be adjusted, the horizontal boundary can be converted to an angle, such as half the horizontal boundary divided by the distance and then taking the inverse tangent: arctan((horizontal boundary (horiz_bound) / 2) / distance). If the difference in azimuth is greater than this angle, the AR system can reduce the difference in azimuth to this angle. To determine whether the altitude of the content needs to be adjusted, the vertical boundary can be converted to an angle, such as half the vertical boundary divided by the distance and then taking the inverse tangent: arctan((vertical boundary (vert_bound) / 2) / distance). If the height difference is greater than this angle, the AR system can reduce the height difference to this angle.

[0163] In some examples, the AR system can convert the coordinates of content 1360 to Cartesian coordinates. Previously, the spherical coordinates of the content were referenced to the controller. Advantageously, using Cartesian coordinates can allow the AR system to position the content in the user's environment without reference to controller 1302. To convert from spherical coordinates to Cartesian (rectangular) coordinates, the AR system can apply the following formula:

[0164] X = distance * cos (azimuth) * cos (altitude)

[0165] Y = distance * sin (height)

[0166] Z = distance * sin (azimuth) * cos (height)

[0167] Where X corresponds to the x-coordinate, Y corresponds to the y-coordinate in a Cartesian coordinate system, and Z corresponds to the z-coordinate in a Cartesian coordinate system.

[0168] F. Sample content orientation

[0169] An AR system may be able to control the orientation of virtual content within the user's 3D environment. FIG. 14A to FIG. 14C Aspects of an example content orientation environment 1400 for controlling the orientation of virtual content 1404 are shown, including a controller 1302 and a head mounted device 1402 .

[0170] refer to Fig.14A , content orientation environment 1400 can include user 1406 wearing head mounted display 1402 and / or using controller 1302 to manipulate and / or view virtual content 1404. The user can view virtual content 1404 at location 1412 in the 3D environment of user 1406. The content can be displayed at an angle 1410 relative to a gaze vector 1405 associated with the user's head pose. In some examples, angle 1410 can be optimized so that the user can more easily perceive the surface of content 1404. In some examples, angle 1410 can make the surface of content 1404 perpendicular to the user's gaze direction.

[0171] Fig. 14B A top view of an example content orientation environment 1401 is shown. Fig. 14B As shown, content 1420 can be rotated so that it faces controller 1302 (e.g., remains perpendicular to the controller gesture). Therefore, the surface of content 1420 facing head-mounted display 1402 can be rotated by angle 1422 relative to the x-axis of a coordinate system centered on the user's head-mounted display. Angle 1422 can be updated as the controller moves so that the surface 1420 of the content remains locked at an angle perpendicular to the pointing vector of controller 1302. In another example, angle 1422 can be updated based on user or application input.

[0172] Fig. 14C A side view of an example content targeting environment 1401 is shown. Fig. 14C As shown, content 1420 can be rotated so that it faces the user's head or head-mounted display 1402. Thus, the surface of content 1420 that faces the head-mounted display 1402 or the user's head can be rotated by angle 1432 relative to the y-axis of a coordinate system centered on the user's head-mounted display. As the user moves their head, eye gaze, and / or the head-mounted display, angle 1432 can be updated so that the surface of content 1420 remains locked at an angle perpendicular to gaze vector 1405 associated with the user's head pose. In another example, angle 1432 can be updated based on user or application input. In some examples, angle 1432 can be based on the height of content 1420 so that the surface of content 1420 can be oriented to face an origin associated with the user, such as a point on the user's head. In another example, angle 1432 and / or height of content 1420 can be fixed.

[0173] Fig.14DA flow diagram of an example content orientation process 1403 is shown. For example, the content orientation process 1403 may include a content position determination box 1442, a head pose determination box 1444, a content orientation box, or more or fewer boxes.

[0174] At content location determination block 1442, the AR system may determine the location of the virtual content within the user's 3D environment. For example, the virtual content may be located at a point in front of the user (e.g., at a point defined by a reference FIG. 12A to FIG. 13B The AR system may determine the content location based on input from or data associated with the virtual content, application, and / or AR system.

[0175] At head pose determination block 1444, the AR system can determine the user's head pose. For example, the AR system can detect one or more parameters associated with the user's head pose using one or more sensors associated with the AR system, such as one or more outward-facing cameras associated with a head-mounted display worn by the user, an inertial measurement unit, some combination thereof, or other sensors. The user's head pose can be used to help determine the user's gaze direction.

[0176] In content orientation box 1446, the AR system can use the content position and the user's gaze direction to reorient the virtual content. For example, the AR system can orient the surface of the content to be perpendicular to the user's gaze direction at the determined content position. In some examples, the AR system can additionally move the position of the content to achieve a comfortable viewing experience when the user views the content in the updated orientation.

[0177] G. Example Content Control Mechanism

[0178] Fig.15 An example content access mechanism is shown, and Fig.16 An example content placement mechanism is shown that may be used in conjunction with the interaction and movement mechanisms described above.

[0179] 1. Menu opening and closing

[0180] In some examples, the content may include a control menu. Fig.15 An example menu access process 1500 for accessing a control menu is shown. Fig.15 , the menu access process 1500 may include an indication box 1502, a direction determination box 1504, an open position determination box 1506, an animation box 1508, a menu display box 1510, fewer or more boxes. Although the reference control menu is discussed Fig.15 , but the process can be used with other content, such as a prism including one or more interactive virtual objects.

[0181] In indication box 1502, the AR system may receive an indication to open or access a menu or other content. The indication may include input, gesture, or posture. For example, the input may include pointing, pressing a button, or other input components of a controller associated with the AR system. In another example, the input may include a gesture associated with accessing a menu or other content. In the absence of input involving direct button presses or other direct inputs to the controller, the AR system may use one or more sensors associated with the AR system (e.g., an outward-facing imaging system of a head-mounted display) to sense the input. For example, the input may include six degrees of freedom pointing by a user's hand or pointing device. In another example, the input may include input to a multi-degree-of-freedom touchpad. The AR system may then determine whether to give an indication based on a gesture, posture, or other indirect or direct input detected by one or more sensors. In some examples, different inputs may bring up different types of menus or different content. In some examples, different inputs may be used to call up the same menu or content.

[0182] In direction determination block 1504, the AR system can determine a pointing vector for other input mechanisms or controllers, devices, or gestures that can indicate a direction. The pointing vector can include a direction within the user's 3D environment indicated by one or more input devices. In some examples, the pointing vector can include a direction indicated by a six-degree-of-freedom input device or a user's hand by one or more user gestures. In another example, the pointing vector can include a direction indicated by a three-degree-of-freedom touch input. In some examples, the pointing vector can include an indicated direction of a handheld controller, such as described above with reference to FIG. FIG. 10A to FIG. 11B In some examples, the pointing vector can be determined by more than one input, such as controller orientation and user eye gaze or other combination of inputs.

[0183] At the open position determination block 1506, the AR system can determine or identify a location to open the indicated menu or otherwise display content within the user's 3D environment. The location can be based on a determined focus direction or other input from the user, application, or AR system. For example, a user can point a controller in their environment. The AR system can display a menu or other virtual content at a point along the pointing vector. The point along the pointing vector can be determined based on any number of factors, including but not limited to one or more rules associated with content location, such as reference FIG. 12A to FIG. 13B In some examples, the open position can be at a set distance away from the controller along the pointing vector. In another example, the open position can be located at the center of the user's field of view.

[0184] In animation box 1508, the AR system can generate animation, sound, feedback, or other effects to indicate to the user the opening position of the menu or other content. For example, the opening position can be at a point along the direction of the focusing light. The direction of the focusing light can point to an area in the user's 3D environment that is not currently perceived by the user through his head-mounted display. In order to draw the user's attention to the opening position of the menu or other content, the AR system can display an animation, generate tactile feedback, or play a sound indicating that the content will be opened at that position. For example, the AR system can display sparks or other content emitted from a controller or other reference point along the focus direction toward the opening position. Therefore, the user's gaze or attention may be more likely to be drawn to the menu or content location.

[0185] In menu display frame 1510, the AR system can display the menu in an open position in an orientation that is comfortable for the user to view the menu or other interactive virtual content. For example, the orientation and / or position of the menu or other content can be based on the above reference FIG. 12A to FIG. 14D Describes one or more processes to display and / or update.

[0186] Additionally or alternatively, the AR system may close or stop displaying content or menus based on one or more close indications. The close indication may include some combination of input to a controller, gestures, commands, other inputs, or inputs to stop displaying a menu. Upon receiving the close indication, the AR system may stop displaying a menu or other content.

[0187] 2. Content Drop

[0188] Fig.16 An example menu access process 1600 for stopping and starting content following movement is shown, such as that described above with reference to Fig. 12A and Fig. 12B Described. Reference Fig.16 , the content pitching process 1600 may include a position determination box 1602 , a follow decision box 1604 , a display box 1606 , a follow decision box 1608 , and a follow box 1610 .

[0189] In position determination block 1602, the AR system can determine the current position of the virtual content or menu being manipulated. For example, the virtual content or menu can be moved to a specified position associated with the focus direction, such as the one described above with reference to FIG. FIG. 12A to FIG. 13B As described. The AR system can determine an updated position or other position associated with the user or the user's 3D environment to identify where the content is or should be located. In some examples, the position can be updated as the user walks around the room because the position of the controller and the orientation of the focus direction change. In some embodiments, the content can be placed at a fixed distance in front of the user.

[0190] At follow decision block 1604, the AR system may determine whether the AR system should continue updating the location of the content. For example, the AR system may identify a stop condition to continue updating the location of the content (or a following process or mechanism). A stop condition may include an indication from a user or other source to stop updating the location of the content. For example, a user may make a gesture, issue a command, or press a button or other input on a controller to stop the content from following the user. In another example, the AR system may identify that the user has exited a bounded volume or exceeded a threshold following condition. In some examples, a combination of stop conditions may be used. If a stop condition is detected, the AR system may move to block 1606. If a stop condition is not detected, the AR system may move to block 1610 to continue updating the location of the content (in other words, continue the following process or mechanism), such as described above with reference to FIG. FIG. 12A to FIG. 14B as described.

[0191] At display block 1606, the AR system can display the content at the current or specified location of the user's 3D environment. For example, if the AR system receives an instruction to stop following at block 1604, the AR system can throw a menu or other content at the current location of the menu or content so that the menu or content stays at the last updated location. In some examples, the AR system can continue to update the orientation of the menu or content, such as in the above reference FIG. 14A to FIG. 14B as described. In some examples, the AR system can also freeze the orientation of a menu or content to a current orientation. In some examples, the AR system can freeze the orientation of a menu or content as a result of detecting a freeze condition, such as a gesture, a command, pressing a button, or otherwise providing input to the AR system to instruct the AR system to freeze or stop updating the orientation of the content. In some examples, when the AR system receives a stop condition in box 1610, the AR system can update the orientation to a fixed or preset orientation and / or height within the user's 3D environment. For example, the AR system can fix the content at eye height and fix the orientation of the content so that the surface of the content is perpendicular to the floor of the user's 3D environment, so that the user can easily view the content.

[0192] At follow decision block 1608, the AR system may determine whether the AR system should continue updating the location of the content. For example, the AR system may identify a start condition for updating the location of the content (or a following process or mechanism). The start condition may include an indication from a user or other source to begin updating the location of the content. For example, a user may make a gesture, issue a command, press a button, or other input to begin having the content follow the user. In another example, the AR system may recognize that the user has entered a bounding volume or passed a threshold following condition. In some examples, a combination of start conditions may be used. If a start condition is detected, the AR system may move to block 1610 to continue updating the location of the content (in other words, continue the following process or mechanism), such as described above with reference to FIG. FIG. 12A to FIG. 14B As described.

[0193] In some examples, when the AR system receives an indication to start updating the position and / or orientation of the content, the AR system can summon the content to the user's current position. For example, the AR system can move the current position of the content to a new position that is closer to the user. In some examples, the new position can be based on the above reference FIG. 12A to FIG. 14B One or more rules describing the location of content. In some examples, the AR system may not update the location and / or orientation of the content to summon the content to the user's current location until a summon indication is detected. The summon indication may include a gesture, command, button press, or other input.

[0194] H. Example Application of Content Following System

[0195] Fig.17 An example application 1700 for a content following system is depicted in which two users of respective wearable systems are conducting a telepresence session. Two users (named Alice 912 and Bob 914 in this example) are shown in this figure. The two users are wearing their respective wearable devices 902 and 904, which may include reference Fig.9D The HMD described herein (e.g., display device 70 of system 60) is used to represent another user's avatar in a telepresence session. Two users can use the wearable device to conduct a telepresence session. Note that Fig.17 The vertical line separating the two users is intended to illustrate that Alice and Bob may (but need not) be in two different locations when communicating via telepresence (e.g., Alice may be in her office in Atlanta, while Bob is outdoors in Boston).

[0196] Wearable devices 902 and 904 can communicate with each other or with other user devices and computer systems. For example, Alice's wearable device 902 can communicate with Bob's wearable device 904, for example, via network 990. Wearable devices 902 and 904 can track the user's environment and movement in the environment (e.g., through respective outward-facing imaging systems 464, or one or more position sensors) and voice (e.g., through corresponding audio sensors 232). Wearable devices 902 and 904 can also track the user's eye movements or gaze based on data acquired by the inward-facing imaging system 462. In some cases, the wearable device can also capture or track the user's facial expressions or other body movements (e.g., arm or leg movements), where the user is close to a reflective surface, and the outward-facing imaging system 464 can obtain a reflected image of the user to observe the user's facial expressions or other body movements.

[0197] The wearable device may use information obtained from the first user and the environment to animate a virtual avatar to be rendered by the wearable device of the second user to create a sense of presence of the first user in the environment of the second user. For example, wearable devices 902 and 904, remote computing system 920 may process an image or motion of Alice for presentation by Bob's wearable device 904, or may process an image or motion of Bob for presentation by Alice's wearable device 902, alone or in combination. As further described herein, the avatar may be rendered based on contextual information, such as the user's intent, the user's environment, or the environment in which the avatar is rendered, or other biological characteristics of the person.

[0198] Although the example involves only two users, the technology described herein should not be limited to two users. Multiple users (e.g., two, three, four, five, six or more) using wearable devices (or other telepresence devices) can participate in a telepresence session. The wearable device of a particular user can present the avatars of other users to the particular user during the telepresence session. In addition, although the example in the figure shows that the user is standing in the environment, the user does not need to stand. During the telepresence session, any user can stand, sit, kneel, lie down, walk or run, or be in any position or motion. The user can also be in a physical environment other than that described in the examples of this article. When conducting a telepresence session, the user may be in different environments, or may be in the same environment. Not all users need to wear their respective HMDs in a telepresence session. For example, Alice can use other image acquisition and display devices, such as a webcam and a computer screen, while Bob wears a wearable device 904.

[0199] Bob may provide instructions, such as by button presses on controller 1704, to display a chat menu or other content 1702 during the avatar chat session. Bob's AR system may provide instructions, such as by button presses on controller 1704, to display a chat menu or other content 1702 during the avatar chat session. Fig.15 The access process 1500 described above is used to display content 1702. Content 1702 can appear at a location along pointing vector 1706 in Bob's environment. Fig.14D 1702. In some examples, Bob can place a chat menu or other content 1702 in a desired location in Bob's environment. Additionally or alternatively, Bob can use smaller movements of the controller 1704 to select or interact with portions of the content 1702. Bob can close the chat menu or other content 1702 by one or more indications to Bob's AR system.

[0200] I. Other Examples

[0201] Additional examples of AR systems are disclosed herein. Any of the disclosed examples may be combined.

[0202] Example 1: An augmented reality (AR) system, comprising:

[0203] an AR display configured to present virtual content to a user of the AR system;

[0204] an outward-facing camera configured to capture one or more images of the user's environment;

[0205] a handheld controller defining a pointing vector indicating a pointing direction of the handheld controller;

[0206] a hardware processor in communication with the AR display, the outward-facing camera, and the handheld controller, the hardware processor being programmed to:

[0207] displaying an interactive content object via the AR display;

[0208] in a first interaction mode, directing movement of the handheld controller within the interactive content object while the pointing vector remains within the interactive content object and allowing interaction with the interactive content object via the handheld controller;

[0209] monitoring changes in the pointing vector with reference to the interactive content object;

[0210] In response to detecting movement of the pointing vector outside of the interactive content object, updating the system to a second interaction mode in which the handheld controller causes movement of the interactive content object such that the interactive content object follows movement of the handheld controller in the virtual environment; and

[0211] In response to detecting movement of the pointing vector of the handheld controller within the interactive content object, the system is updated to the first interaction mode.

[0212] Example 2: The system of Example 1, wherein the interactive content object comprises a prism containing a virtual object.

[0213] Example 3: A system according to any of Examples 1 or 2, wherein at least one edge of the interactive content object is 10 degrees further away from a center of the virtual object than a corresponding edge of the virtual object.

[0214] Example 4: The system of any one of Examples 1 to 3, wherein the hardware processor is configured to receive an indication to display the interactive content at a first content location.

[0215] Example 5: The system of Example 4, wherein the indication comprises pressing and releasing a button on the controller.

[0216] Example 6: The system of any one of Examples 4 to 5, wherein the hardware processor is configured to: alert the user of display of the interactive content at the first content location.

[0217] Example 7: The system of Example 6, wherein the reminder comprises at least one of: a graphic, tactile feedback, or a sound.

[0218] Example 8: The system of any one of Examples 1 to 7, wherein the hardware processor is configured to receive an instruction to stop displaying the interactive content in the environment of the user.

[0219] Example 9: The system of Example 8, wherein the instruction comprises pressing and releasing a button on the controller.

[0220] Example 10: The system of any one of Examples 1 to 9, wherein the hardware processor is configured to:

[0221] determining a first user head pose;

[0222] Based on the first user head posture, identifying a first user gaze vector; and

[0223] The interactive content is oriented so that a surface of the interactive content object is perpendicular to the first user gaze vector.

[0224] Example 11: The system of Example 10, wherein the hardware processor is configured to:

[0225] determining a second user head pose;

[0226] Based on the second user head posture, identifying a second user gaze vector; and

[0227] The interactive content object is oriented so that a surface of the interactive content object is perpendicular to the second user gaze vector.

[0228] Example 12: A system according to any of Examples 1 to 11, wherein the spacing of the interactive content objects is fixed relative to the height of the user's head.

[0229] Example 13: The system of any one of Examples 1 to 12, wherein the hardware processor is configured to:

[0230] receiving an instruction to stop moving the interactive content; and

[0231] In response to receiving the indication to stop moving, display of the interactive content is maintained at a current content location within the environment of the user.

[0232] Example 14: The system of Example 13, wherein the hardware processor is configured to:

[0233] receiving an instruction to start moving the interactive content; and

[0234] In response to receiving the indication to initiate movement, the interactive content is enabled to be displayed at a new location within the environment of the user.

[0235] Example 15: An augmented reality (AR) system, comprising:

[0236] an AR display configured to present virtual content to a user of the AR system;

[0237] a handheld controller having at least six degrees of freedom; and

[0238] a hardware processor in communication with the AR display, the outward-facing camera, and the handheld controller, the hardware processor being programmed to:

[0239] displaying interactive content at a first content location;

[0240] determining a first pointing vector comprising a direction indicated by the controller;

[0241] determining whether the first pointing vector intersects a bounding volume associated with the interactive content;

[0242] In response to determining that the first pointing vector does not intersect the bounding volume, moving the interactive content to a second content location associated with a point along the direction of the first pointing vector; and

[0243] In response to determining that the first pointing vector intersects the bounding volume, an indication is received to interact with the interactive content at the first content location.

[0244] Example 16: The system of Example 15, wherein the interactive content comprises a prism containing a virtual object.

[0245] Example 17: A system according to any of Examples 15 or 16, wherein at least one edge of the interactive content is 10 degrees farther from the center of the virtual object than a corresponding edge of the virtual object.

[0246] Example 18: The system of Example 15, wherein the hardware processor is configured to receive an indication to display the interactive content at the first content location.

[0247] Example 19: The system of Example 18, wherein the indication comprises pressing and releasing a button on the controller.

[0248] Example 20: The system of any one of Examples 18 to 19, wherein the hardware processor is configured to: alert the user to display of the interactive content at the first content location.

[0249] Example 21: The system of Example 20, wherein the reminder comprises at least one of: a graphic, tactile feedback, or a sound.

[0250] Example 22: The system of any one of Examples 15 to 21, wherein the hardware processor is configured to receive an instruction to stop displaying the interactive content in the environment of the user.

[0251] Example 23: The system of Example 22, wherein the indication comprises pressing and releasing a button on the controller.

[0252] Example 24: The system of any one of Examples 15 to 24, wherein the hardware processor is configured to:

[0253] determining a first user head pose;

[0254] Based on the first user head posture, identifying a first user gaze vector; and

[0255] The interactive content is oriented such that a surface of the interactive content is perpendicular to the first user gaze vector.

[0256] Example 25: The system of Example 24, wherein the hardware processor is configured to:

[0257] determining a second user head pose;

[0258] Based on the second user head posture, identifying a second user gaze vector; and

[0259] The interactive content is oriented such that a surface of the interactive content is perpendicular to the second user gaze vector.

[0260] Example 26: The system of any one of Examples 15 to 25, wherein the first content location and the second content location are at the same height from the ground.

[0261] Example 27: The system of Example 26, wherein the spacing of the interactive content is fixed relative to the height of the user's head.

[0262] Example 28: The system of any one of Examples 15 to 27, wherein the hardware processor is configured to:

[0263] receiving an instruction to stop moving the interactive content; and

[0264] In response to receiving the indication to stop moving, display of the interactive content is maintained at a current content location within the environment of the user.

[0265] Example 29: The system of Example 28, wherein the hardware processor is configured to:

[0266] receiving an instruction to start moving the interactive content; and

[0267] In response to receiving the indication to initiate movement, the interactive content is enabled to be displayed at a new location within the environment of the user.

[0268] Example 30: A method for displaying virtual content, the method comprising:

[0269] displaying interactive content at a first content location;

[0270] determining a first pointing vector comprising a direction indicated by a controller;

[0271] determining whether the first pointing vector intersects a bounding volume associated with the interactive content;

[0272] In response to determining that the first pointing vector does not intersect the bounding volume, moving the interactive content to a second content location associated with a point along the direction of the first pointing vector; and

[0273] In response to determining that the first pointing vector intersects the bounding volume, an indication is received to interact with the interactive content at the first content location.

[0274] Example 31: The method of Example 30, comprising: receiving an instruction to display the interactive content at the first content location.

[0275] Example 32: The method of Example 31, wherein the indication comprises pressing and releasing a button on the controller.

[0276] Example 33: The method of any one of Examples 31 to 32, comprising: communicating to the user a reminder associated with display of the interactive content at the first content location.

[0277] Example 34: The method of Example 33, wherein the reminder comprises at least one of: a graphic, tactile feedback, or a sound.

[0278] Example 35: A method according to any one of Examples 30 to 34, comprising: receiving an instruction to stop displaying the interactive content in the environment of the user.

[0279] Example 36: The method of Example 35, wherein the indication comprises pressing and releasing a button on the controller.

[0280] Example 37: The method according to any one of Examples 30-36, comprising:

[0281] determining a first user head pose;

[0282] Based on the first user head posture, identifying a first user gaze vector; and

[0283] The interactive content is oriented such that a surface of the interactive content is perpendicular to the first user gaze vector.

[0284] Example 38: The method according to Example 37, comprising:

[0285] determining a second user head pose;

[0286] Based on the second user head posture, identifying a second user gaze vector; and

[0287] The interactive content is oriented such that a surface of the interactive content is perpendicular to the second user gaze vector.

[0288] Example 39: The method of any one of Examples 30 to 38, wherein the first content position and the second content position are at the same height from the ground.

[0289] Example 40: The method of Example 39, wherein the spacing of the interactive content is fixed relative to the height of the user's head.

[0290] Example 41: The method according to any one of Examples 30 to 40, comprising:

[0291] receiving an instruction to stop moving the interactive content; and

[0292] In response to receiving the indication to stop moving, display of the interactive content is maintained at a current content location within the environment of the user.

[0293] Example 42: The method according to Example 41, comprising:

[0294] receiving an instruction to start moving the interactive content; and

[0295] In response to receiving the indication to initiate movement, the interactive content is enabled to be displayed at a new location within the environment of the user.

[0296] Example 43: According to any of the above examples, the interactive content object includes a bounded volume containing the virtual object.

[0297] J. Other Considerations

[0298] Each of the processes, methods, and algorithms described herein and / or depicted in the accompanying drawings can be embodied in a code module executed by one or more physical computing systems, hardware computer processors, dedicated circuits, and / or electronic hardware configured to execute specific and particular computer instructions, and fully or partially automated by the code module. For example, a computing system may include a general-purpose computer (e.g., a server) programmed with specific computer instructions or a dedicated computer, dedicated circuit, etc. The code module can be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language. In some embodiments, specific operations and methods can be performed by circuits specific to a given function.

[0299] Furthermore, certain implementations of the functionality of the present disclosure may be mathematically, computationally or technically complex enough that application-specific hardware or one or more physical computing devices (with appropriate dedicated executable instructions) may be necessary to perform the functionality, e.g., due to the amount or complexity of the calculations involved or to provide results in substantially real time. For example, an animation or video may include many frames, each with millions of pixels, and require specially programmed computer hardware to process the video data to provide the desired image processing task or application in a commercially reasonable amount of time.

[0300] Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as a physical computer storage device including a hard drive, solid-state memory, random access memory (RAM), read-only memory (ROM), optical disk, volatile or non-volatile storage devices, combinations thereof, and the like. Methods and modules (or data) may also be sent as a generated data signal (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media (including wireless-based and wired / cable-based media), and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps or actions may be stored persistently or otherwise in any type of non-transitory tangible computer storage device, or may be transmitted via a computer-readable transmission medium.

[0301] Any process, frame, state, step or function in the flowchart described herein and / or depicted in the accompanying drawings should be understood to represent a code module, code segment or code portion, which includes one or more executable instructions for implementing a specific function (e.g., logic or arithmetic) or step in a process. Various processes, blocks, states, steps or functions can be combined, rearranged, added, deleted, modified or otherwise changed with the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules can perform some or all of the functions described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps or states associated therewith can be executed in other appropriate sequences, for example, in serial, parallel or in some other manner. Tasks or events can be added to or deleted from the disclosed example embodiments. In addition, the separation of various system components in the embodiments described herein is for illustrative purposes and should not be understood as requiring such separation in all embodiments. It should be understood that the described program components, methods and systems can generally be integrated together in a single computer product or packaged into multiple computer products. Many implementation variations are possible.

[0302] The processes, methods and systems can be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowd-sourced computing networks, the Internet, and the World Wide Web. The network can be a wired or wireless network or any other type of communication network.

[0303] The systems and methods of the present disclosure each have several innovative aspects, none of which is solely responsible for or requires the desired attributes disclosed herein. The various features and processes described above can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications to the embodiments described in the present disclosure will be apparent, and the general principles defined herein can be applied to other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the claims are not intended to be limited to the embodiments shown herein, but are consistent with the widest range consistent with the present disclosure, the principles disclosed herein, and the novel features.

[0304] Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. In addition, although the above features may be described as functioning in certain combinations and even initially claimed as such, in some cases one or more features from the claimed combination may be excised from the combination, and the claimed combination may be directed to a subcombination or variations of the subcombination. It is not necessary or essential for each and all embodiments to require a single feature or group of features.

[0305] Unless otherwise specified, or understood in the context of use in other ways, conditional language used herein, such as "can", "may", "possibly", "may", "for example", etc., is generally intended to convey that certain embodiments include certain features, elements and / or steps, while other embodiments do not include certain features, elements and / or steps. Therefore, such conditional language is generally not intended to imply that one or more embodiments require features, elements and / or steps in any way, or that one or more embodiments must include logic for determining whether these features, elements and / or steps are included or to be performed in any particular embodiment (regardless of whether there is author input or prompt). The terms "comprise", "include", "have", etc. are synonymous and are used inclusively in an open-ended manner, and do not exclude additional elements, features, actions, operations, etc. In addition, the term "or" is used in its inclusive sense (rather than in its exclusive sense), so when used, for example, to connect a list of elements, the term "or" means one, some or all of the elements in the list. In addition, unless otherwise specified, the articles "a", "an" and "the" used in this application and the appended claims should be understood to mean "one or more" or "at least one".

[0306] As used herein, phrases referring to "at least one" of a list of items refer to any combination of those items, including single members. For example, "at least one of A, B, or C" is intended to cover: A, B, C, A and B, A and C, B and C, and A, B and C. Unless otherwise specifically stated, conjunctive language such as the phrase "at least one of X, Y, and Z" is otherwise understood by context as being generally used to convey that an item, term, etc. may be at least one of X, Y, or Z. Thus, such conjunctive language is generally not intended to imply that certain embodiments require the presence of at least one of X, at least one of Y, and at least one of Z, respectively.

[0307] Similarly, although operations may be depicted in the drawings in a particular order, it should be appreciated that it is not necessary to perform these operations in the particular order shown or in sequence, or to perform all the operations shown, to achieve the desired result. In addition, the drawings may schematically depict one or more example processes in the form of a flow chart. However, other operations not shown may be included in the schematically illustrated example methods and processes. For example, one or more additional operations may be performed before, after, simultaneously, or between any of the illustrated operations. In addition, operations may be rearranged or reordered in other embodiments. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products. In addition, other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in different orders and still achieve the desired results.

Claims

1. An augmented reality (AR) system, comprising: an AR display configured to present virtual content to a user of the AR system; an outward-facing camera configured to capture one or more images of the user's environment; a handheld controller defining a pointing vector indicating a pointing direction of the handheld controller; as well as a hardware processor in communication with the AR display, the outward-facing camera, and the handheld controller, the hardware processor being programmed to: displaying an interactive content object via the AR display; in a first interaction mode, directing movement of the handheld controller within the interactive content object while the pointing vector remains within the interactive content object and allowing interaction with the interactive content object via the handheld controller; monitoring changes in the pointing vector with reference to the interactive content object; in response to detecting movement of the pointing vector outside of the interactive content object, updating the system to a second interaction mode in which the handheld controller causes movement of the interactive content object such that the interactive content object follows movement of the handheld controller in the virtual environment; as well as In response to detecting movement of the pointing vector of the handheld controller within the interactive content object, the system is updated to the first interaction mode.

2. The system according to claim 1, wherein: The interactive content object includes a bounded volume containing a virtual object.

3. The system according to claim 2, wherein: At least one edge of the interactive content object is 10 degrees further away from a center of the virtual object than a corresponding edge of the virtual object.

4. The system according to any one of claims 1 to 3, wherein: The hardware processor is configured to receive an indication to display the interactive content object at a first content location.

5. The system according to claim 4, wherein: The instructions include pressing and releasing a button on the handheld controller.

6. The system according to claim 5, wherein: The hardware processor is configured to alert the user of display of the interactive content object at the first content location.

7. The system according to claim 6, wherein: The reminder includes at least one of the following: a graphic, a tactile feedback, or a sound.

8. The system according to claim 7, wherein: The hardware processor is configured to receive an indication to cease displaying the interactive content object in the environment of the user.

9. The system according to claim 8, wherein: The instructions include pressing and releasing a button on the handheld controller.

10. The system according to claim 9, wherein: The hardware processor is configured to: determining a first user head pose; Based on the first user head posture, identifying a first user gaze vector; and The interactive content object is oriented such that a surface of the interactive content object is perpendicular to the first user gaze vector.

11. The system according to claim 10, wherein: The hardware processor is configured to: determining a second user's head pose; Based on the second user head posture, identifying a second user gaze vector; and The interactive content object is oriented such that a surface of the interactive content object is perpendicular to the second user gaze vector.

12. The system according to claim 11, wherein: The spacing of the interactive content objects is fixed relative to the height of the user's head.

13. The system according to claim 12, wherein: The hardware processor is configured to: receiving an instruction to stop moving the interactive content object; and In response to receiving the indication to cease movement, display of the interactive content object is maintained at a current content location within the environment of the user.

14. The system according to claim 13, wherein: The hardware processor is configured to: receiving an instruction to start moving the interactive content object; and In response to receiving the indication to initiate movement, the interactive content object is allowed to be displayed at a new location within the environment of the user.

15. An augmented reality (AR) system, comprising: an AR display configured to present virtual content to a user of the AR system; a handheld controller having at least six degrees of freedom; as well as a hardware processor in communication with the AR display, the outward-facing camera, and the handheld controller, the hardware processor being programmed to: displaying interactive content at a first content location; determining a first pointing vector comprising a direction indicated by the handheld controller; determining whether the first pointing vector intersects a bounding volume associated with the interactive content; In response to determining that the first pointing vector does not intersect the bounding volume, moving the interactive content to a second content location associated with a point along the direction of the first pointing vector; as well as In response to determining that the first pointing vector intersects the bounding volume, an indication is received to interact with the interactive content at the first content location.

16. The system of claim 15, wherein: The interactive content includes a bounded volume containing a virtual object.

17. A system according to any one of claims 15 or 16, wherein: At least one edge of the interactive content is 10 degrees further away from a center of the virtual object than a corresponding edge of the virtual object.

18. The system of claim 15, wherein: The hardware processor is configured to receive an indication to display the interactive content at the first content location.

19. The system of claim 18, wherein: The instructions include pressing and releasing a button on the handheld controller.

20. The system according to any one of claims 18 to 19, wherein: The hardware processor is configured to alert the user of display of the interactive content at the first content location.

Images