Virtual Reticle for Augmented Reality Systems

Abstract

To provide a system and a method for displaying a virtual reticle in an augmented or virtual reality environment by a wearable device.SOLUTION: An environment, for example, can include real or virtual objects which may be interacted with by a user through a variety of poses, such as, e.g., head pose, eye pose or gaze, or body pose. The user may select an object by pointing the virtual reticle toward a target object by changing pose or gaze. A wearable device can recognize that an orientation of a user's head or eyes is outside of a range of acceptable or comfortable head or eye poses and accelerate the movement of the reticle away from a default position and toward a position in the direction of the user's head or eye movement, which can reduce the amount of movement by the user to align the reticle and target.SELECTED DRAWING: None

Description

[Technical field]

[0001] (Copyright Notice) A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to anyone copying this patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves any and all copyrights whatsoever. (CROSS REFERENCE TO RELATED APPLICATIONS)

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 578,094, filed October 27, 2017, and entitled "VIRTUAL RETICLE FOR AUGUMENTED REALITY SYSTEMS," which is incorporated by reference in its entirety herein. (Field)

[0003] The present disclosure relates to virtual reality and augmented reality imaging and visualization systems, and more particularly to displaying a virtual reticle based on head pose. [Background technology]

[0004] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality", "augmented reality", or "mixed reality" experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner that appears or may be perceived as real. Virtual reality or "VR" scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs. Augmented reality or "AR" scenarios typically involve the presentation of digital or virtual image information as an extension to the visualization of the real world around the user. Mixed reality or "MR" refers to the merging of real and virtual worlds to generate new environments in which physical and virtual objects coexist and interact in real time. In conclusion, the human visual perception system is highly complex, making it difficult to create VR, AR, or MR technologies that promote comfortable, natural-feeling, and rich presentations of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies. Summary of the Invention [Means for solving the problem]

[0005] Various embodiments of systems and methods for displaying a virtual reticle are disclosed. The environment can include real or virtual objects that can be interacted with by a user using the virtual reticle. The position or orientation of the user's head, eyes, shoulders, chest, arms, or other body parts can dictate the position or movement speed of the virtual reticle in the environment, and the user can select or point to an object by pointing the virtual reticle at or focusing the reticle on a target object. The wearable device can recognize that the user's head, eyes, shoulders, chest, arms, etc. are positioned or oriented in an uncomfortable or otherwise undesirable manner. In response to the recognition of an uncomfortable or undesirable posture, the system can adjust the position or movement speed of the virtual reticle to assist in positioning the virtual reticle in a desirable manner, thereby reducing the likelihood that the user's posture will remain uncomfortable or undesirable.

[0006] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Neither this summary nor the following detailed description purport to define or limit the scope of the inventive subject matter. The present invention provides, for example, the following: (Item 1) 1. A system comprising: a head pose sensor configured to acquire head pose data of a user of the system; a non-transitory memory configured to store the head pose data; a display configured to be positioned in front of an eye of a user, the display configured to project a virtual reticle towards the eye of the user; a hardware processor in communication with the head pose sensor, the display, and the non-transitory memory; Equipped with The hardware processor includes: acquiring the head pose data of the user; identifying a head pose of the user based at least in part on the head pose data; determining an adjustment for a position of the virtual reticle based at least in part on a comparison between the user's head pose and a reference head pose; altering the virtual reticle to a home position based at least in part on the determined adjustment; and A system that is programmed to: (Item 2) 2. The system of claim 1, wherein the virtual reticle comprises a movable indicator that identifies the user's position within the user's field of view. (Item 3) 2. The system of claim 1, wherein the head pose data corresponds to at least one of an indication of yaw, pitch, or roll of the user's head. (Item 4) 4. The system of claim 3, wherein the instruction of yaw, pitch, or roll is with respect to the reference head pose. (Item 5) 5. The system of claim 4, wherein the reference head pose corresponds to a horizontal head pose of the user's head. (Item 6) Item 6. The system of item 5, wherein the horizontal head pose comprises a head pose in which a coronal plane of the user's head, a sagittal plane of the user's head, and an axial plane of the user's head are each mutually orthogonal. (Item 7) Item 2. The system of item 1, wherein the reference head pose comprises a head pose corresponding to a neutrally positioned state of the user's head. (Item 8) 8. The system of claim 7, wherein the neutral position of the user's head corresponds to -5 to 5 degrees yaw, -5 to 5 degrees roll, and -15 to -25 degrees pitch relative to the horizontal head posture. (Item 9) Item 9. The system of item 8, wherein the reference head pose corresponds to at least one of 0 degrees yaw, 0 degrees roll, or -20 degrees pitch relative to the horizontal head pose. (Item 10) 2. The system of claim 1, wherein the hardware processor is further programmed to identify a head pose vector corresponding to the head pose of the user and to identify a reference head pose vector corresponding to the reference head pose. (Item 11) Item 11. The system of item 10, wherein the hardware processor is further programmed to determine an angular difference between the head pose vector and the reference head pose vector based at least in part on a comparison between the head pose of the user and the reference head pose, the angular difference corresponding to at least one of a difference in yaw, pitch, or roll of the head pose of the user relative to the reference head pose. (Item 12) Item 12. The system of item 11, wherein the hardware processor is programmed to compare the determined angular difference with one or more head pose thresholds to determine the adjustment for the position of the virtual reticle. (Item 13) Item 13. The system of item 12, wherein the one or more head pose thresholds comprise at least one of a maximum head pose threshold or a minimum head pose threshold. (Item 14) Item 14. The system of item 13, wherein the maximum head pose threshold corresponds to at least one of a maximum head yaw threshold, a maximum head roll threshold, or a maximum head pitch threshold. (Item 15) Item 15. The system of item 14, wherein the maximum head yaw threshold is 50 degrees, the maximum head roll threshold is 20 degrees, or the maximum head pitch threshold is 5 degrees, relative to the reference head pose. (Item 16) Item 14. The system of item 13, wherein the minimum head pose threshold corresponds to at least one of a minimum head yaw threshold, a minimum head roll threshold, or a minimum head pitch threshold. (Item 17) Item 17. The system of item 16, wherein the minimum head yaw threshold is -50 degrees, the minimum head roll threshold is -20 degrees, or the minimum head pitch threshold is -45 degrees, relative to the reference head pose. (Item 18) The hardware processor includes: Item 14. The system of item 13, further programmed to determine the adjustment for the position of the virtual reticle based at least in part on a first adjustment value in response to determining that the angular difference does not meet the minimum head pose threshold. (Item 19) 20. The system of claim 18, wherein the first adjustment value is approximately -12 degrees. (Item 20) Item 14. The system of item 13, wherein the hardware processor is further programmed to determine the adjustment for the position of the virtual reticle based at least in part on a second adjustment value in response to determining that the angular difference satisfies the maximum head pose threshold. (Item 21) 21. The system of claim 20, wherein the second adjustment value is approximately +5 degrees. (Item 22) The hardware processor includes: Item 14. The system of item 13, further programmed to determine the adjustment for the position of the virtual reticle based at least in part on a third adjustment value in response to determining that the angular difference meets the minimum head pose threshold and does not meet the maximum head pose threshold. (Item 23) Item 23. The system of item 22, wherein the third adjustment value corresponds to an easing function. (Item 24) 23. The system of claim 22, wherein the third adjustment value is approximately 0 degrees. (Item 25) 2. The system of claim 1, wherein altering the virtual reticle to a fixed position includes altering the virtual reticle to a new position from a default reticle position in the user's field of view. (Item 26) Item 26. The system of item 25, wherein the default reticle position comprises a center of the user's field of view. (Item 27) Item 14. The system of item 1, wherein the head posture sensor comprises an inertial measurement unit (IMU), an accelerometer, a gyroscope, or a magnetometer. (Item 28) Item 1, wherein the wearable system comprises a head-mounted wearable system. (Item 29) 1. A method for adjusting a position of a virtual reticle that identifies a user's position within a field of view corresponding to a display of a display system, the method comprising: acquiring head pose data of a user of a display system from a head pose sensor configured to track a head pose of the user; identifying a head pose vector corresponding to the head pose of the user based at least in part on the head pose data; identifying a reference head pose vector corresponding to a reference head pose; determining an angular difference between the head pose vector and the reference head pose vector based at least in part on a comparison between the head pose of the user and the reference head pose, the angular difference corresponding to at least one of a difference in yaw, pitch, or roll of the head pose of the user relative to the reference head pose; comparing the determined angular difference to one or more head pose thresholds, the one or more head pose thresholds comprising at least one of a maximum head pose threshold or a minimum head pose threshold; determining, in response to determining that the angular difference does not meet the minimum head pose threshold, an adjustment for a position of a virtual reticle based at least in part on a first adjustment value, the position of the virtual reticle corresponding to a position of the movable indicator identifying a position of the user within a field of view of the user; determining the adjustment for a position of the virtual reticle based at least in part on a second adjustment value in response to determining that the angular difference satisfies the maximum head pose threshold; and determining the adjustment for the position of the virtual reticle based at least in part on a third adjustment value in response to determining that the angular difference meets the minimum head pose threshold and does not meet the maximum head pose threshold; adjusting a position of the virtual reticle from a default reticle position in the user's field of view based on the determined adjustment; A method comprising: (Item 30) 1. A method for adjusting a position of a virtual reticle that identifies a user's position within a field of view corresponding to a display of a display system, the method comprising: acquiring head pose data of the user of the display system; identifying a head pose based at least in part on the head pose data; Identifying a reference head pose; determining an adjustment for a position of a virtual reticle projected on a head mounted display based at least in part on a comparison between the head pose and the reference head pose; A method comprising: (Item 31) 1. A method of adjusting a position of a movable indicator identifying a position of a user within a field of view of the user relative to a display of a display system, the method comprising: identifying at least one of a maximum head pitch threshold, a minimum head pitch threshold, or a neutral head pitch threshold; Identifying a head pose vector corresponding to a head pose of the user; Identifying a reference head pose vector; calculating an angular difference between the head pose vector and the reference head pose vector; calculating a reticle adjustment based at least in part on the angular difference and at least one of the maximum head pitch threshold, the minimum head pitch threshold, or the neutral head pitch threshold; determining an adjusted reticle position based at least in part on the calculated reticle adjustment; and A method comprising: (Item 32) 1. A method for adjusting a position of a virtual reticle that identifies a user's position within a field of view corresponding to a display of a display system, the method comprising: calculating an angular difference between a head pose vector and a reference head pose vector, the head pose vector corresponding to a head pose of a user of a display system and the reference head pose vector corresponding to a reference head pose; determining that the angular difference does not meet a minimum head pitch threshold; determining an adjusted reticle position based at least in part on an angular adjustment associated with the minimum head pitch threshold; A method comprising: (Item 33) 1. A method for adjusting a position of a virtual reticle that identifies a user's position within a field of view corresponding to a display of a display system, the method comprising: calculating an angular difference between a head pose vector and a reference head pose vector, the head pose vector corresponding to a head pose of a user of a display system and the reference head pose vector corresponding to a reference head pose; determining that the angular difference satisfies a minimum head pitch threshold; determining that the angular difference does not satisfy a maximum head pitch threshold; determining an adjusted reticle position based at least in part on an angular adjustment associated with the maximum head pitch threshold; A method comprising: (Item 34) 1. A method for adjusting a position of a virtual reticle that identifies a user's position within a field of view corresponding to a display of a display system, the method comprising: calculating an angular difference between a head pose vector and a reference head pose vector, the head pose vector corresponding to a head pose of a user of a display system and the reference head pose vector corresponding to a reference head pose; determining that the angular difference satisfies a minimum head pitch threshold; determining that the angular difference does not satisfy a neutral head pitch threshold; determining an adjusted reticle position based at least in part on the easing function; A method comprising: (Item 35) 1. A method for adjusting a position of a virtual reticle that identifies a user's position within a field of view corresponding to a display of a display system, the method comprising: calculating an angular difference between a head pose vector and a reference head pose vector, the head pose vector corresponding to a head pose of a user of a display system and the reference head pose vector corresponding to a reference head pose; determining that the angular difference satisfies a neutral head pitch threshold; determining that the angular difference does not satisfy a maximum head pitch threshold; determining an adjusted reticle position based at least in part on the easing function; A method comprising: (Item 36) 1. A system comprising: a head pose sensor configured to measure head pose data of a user of the system; a non-transitory memory configured to store the head pose data corresponding to at least one of a yaw, pitch, or roll indication of the user's head; a display configured to be positioned in front of a user's eye, the display configured to project a virtual reticle toward the user's eye, the virtual reticle comprising a movable indicator that identifies the user's position within a field of view; a hardware processor in communication with the head pose sensor, the display, and the non-transitory memory; Equipped with The hardware processor includes: acquiring the head pose data of the user; identifying a head pose vector corresponding to the head pose of the user based at least in part on the head pose data; identifying a reference head pose vector corresponding to a reference head pose; determining an angular difference between the head pose vector and the reference head pose vector based at least in part on a comparison between the head pose of the user and the reference head pose, the angular difference corresponding to at least one of a difference in yaw, pitch, or roll of the head pose of the user relative to the reference head pose; comparing the determined angular difference to one or more head pose thresholds, the one or more head pose thresholds comprising at least one of a maximum head pose threshold or a minimum head pose threshold; determining an adjustment for a position of the virtual reticle based at least in part on a first adjustment value in response to determining that the angular difference does not meet the minimum head pose threshold; and determining the adjustment for a position of the virtual reticle based at least in part on a second adjustment value in response to determining that the angular difference satisfies the maximum head pose threshold; and determining the adjustment for the position of the virtual reticle based at least in part on a third adjustment value in response to determining that the angular difference meets the minimum head pose threshold and does not meet the maximum head pose threshold; adjusting a position of the virtual reticle from a default reticle position in the user's field of view based on the determined adjustment; A system that is programmed to: [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 depicts an illustration of a mixed reality scenario involving a virtual reality object and a physical object viewed by a person.

[0008] [Diagram 2] FIG. 2 illustrates diagrammatically an example of a wearable system.

[0009] [Diagram 3] FIG. 3 diagrammatically illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes.

[0010] [Figure 4] FIG. 4 diagrammatically illustrates an example of a waveguide stack for outputting image information to a user.

[0011] [Diagram 5] FIG. 5 shows an exemplary output beam that may be output by a waveguide.

[0012] [Figure 6] FIG. 6 is a schematic diagram illustrating an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem for use in generating a multifocal stereoscopic display, image, or bright field.

[0013] [Figure 7] FIG. 7 is a block diagram of an example of a wearable system.

[0014] [Figure 8]FIG. 8 is a process flow diagram of an example method for rendering virtual content in relation to recognized objects.

[0015] [Figure 9] FIG. 9 is a block diagram of another example of a wearable system.

[0016] [Figure 10] FIG. 10 is a process flow diagram of an example method for determining user input to a wearable system.

[0017] [Figure 11] FIG. 11 is a process flow diagram of an example method for interacting with a virtual user interface.

[0018] [Figure 12A] FIG. 12A diagrammatically illustrates examples of the field of view (FOR), the world camera's field of view (FOV), the user's field of view, and the user's fixation field.

[0019] [Figure 12B] FIG. 12B diagrammatically illustrates an example of a virtual object in a user's field of view and in the kinetic field of view.

[0020] [Figure 13A] 13A and 13B demonstrate an example of accelerating reticle movement in response to changes in head pose in the vertical (FIG. 13A) and horizontal (FIG. 13B). [Figure 13B] 13A and 13B demonstrate an example of accelerating reticle movement in response to changes in head pose in the vertical (FIG. 13A) and horizontal (FIG. 13B).

[0021] [Figure 14] FIG. 14 illustrates an example of adjusting the position of a reticle based on the user's head pose.

[0022] [Figure 15]FIG. 15 demonstrates an example relationship between a user's head pose and adjustments to the position of the reticle.

[0023] [Figure 16] FIG. 16 illustrates a flow chart for an exemplary reticle alignment process.

[0024] [Figure 17] FIG. 17 illustrates a flow chart for an exemplary reticle adjustment process.

[0025] [Figure 18] FIG. 18 illustrates a flow chart for another exemplary reticle adjustment process.

[0026] Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure. Additionally, the figures in this disclosure are for illustration purposes and are not to scale. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] (overview) The wearable device may include a display for presenting an interactive VR / AR / MR environment. The VR / AR / MR environment may include data elements that may be interacted with by the user through various postures, such as, for example, head posture, eye gaze, or body posture. The VR / AR / MR environment may include a reticle that corresponds to the user's current position relative to the user's field of view (e.g., the extent of the VR / AR / MR environment seen at any given moment). For example, the reticle may represent the user's gaze direction. As the user moves around (e.g., by moving the eyes, head, body, or any combination thereof), the reticle may also move with the user. The reticle may point to one or more objects, and the user may select the target object that the reticle is pointing to. For example, the user may move their head, point the reticle at the object to be selected, and then click a handheld user input totem to select the object. The wearable display device can perform an appropriate action on the selected object (e.g., moving the object, highlighting or enlarging the object, calling up information about the object, displaying a menu of actions that can be performed associated with the object, etc.).

[0028] Occasionally, a user may wish to redirect their head (as a non-limiting example, to select an object that is high in the environment (e.g., the ceiling), low to the ground (e.g., the floor or the user's feet), far to the right, far to the left, etc.), which may require the user to bend, twist, or extend their neck so that the reticle is positioned on the desired object. Bending, twisting, and / or extension of the user's neck may result in, among other things, neck strain or discomfort during use of the wearable device. Thus, the wearable system may recognize that the orientation of the user's head is outside of a range of acceptable (e.g., comfortable, not causing strain, etc.) head postures. As a result, to assist the user in moving or adjusting the reticle, the wearable system may modify or accelerate the movement of the reticle (e.g., modify the angle) in a direction corresponding to the direction the user's head is moving. By modifying or accelerating the movement of the reticle, the wearable system advantageously reduces the degree to which a user must bend, twist, or stretch their neck to align the reticle with the target object, thereby reducing the potential for neck strain or discomfort.

[0029] As an example, if the user looks up toward an object near the ceiling, the wearable display device may move the reticle from a default reticle position (by way of non-limiting example, near the center of the display's field of view (FOV)) to another position, such as toward the top of the user's FOV. Similarly, if the user looks down, the wearable display device may move the reticle from a default position in the FOV to another position, such as near the bottom of the user's FOV. The wearable display device may also redirect the reticle if the user looks rightward or leftward (e.g., moving the reticle to the right or left side of the FOV, respectively), if the user looks diagonally up to the right or left, if the user looks diagonally down to the right or left, etc. The default reticle position is not limited to the center of the FOV. Rather, any location within the FOV can be utilized as the default reticle position.

[0030] A user may wish to use their eye to adjust the reticle. The wearable device may collect eye data, such as eye images (e.g., via an eye camera in an inward-facing imaging system of the wearable device). The wearable system may calculate the user's eye gaze direction based on a mapping matrix that provides an association between the user's eye gaze and a gaze vector (which may indicate the user's gaze direction). When the user's eye is moving, the wearable system may determine the user's eye gaze direction, and the user's field of view or reticle may move in response to the change in eye gaze direction. If the user wishes to use changes in eye gaze to view or target high, low, far left, far right objects, etc., rather than or in addition to changes to head pose, the user's eye movement and / or eye fixation at a particular location may result in eye fatigue, discomfort, or headaches, among other things. Thus, the wearable system can recognize that the orientation of the user's eyes is outside of a range of acceptable (e.g., comfortable, not causing strain, etc.) eye postures. As a result, to assist the user in moving or adjusting the reticle, the wearable system may accelerate the movement of the reticle (e.g., modify the angle) in a direction corresponding to the direction the user's eyes are moving. By accelerating the movement of the reticle, the wearable system advantageously reduces the possibility of eye strain, headaches, eye discomfort, etc.

[0031] By providing a process by which a user may align a virtual reticle with a target using a combination of head posture or eye gaze, the wearable system can provide an intuitive process by which a user can more easily effect alignment while reducing or minimizing neck strain or eye strain. (Example of a 3D display for a wearable system)

[0032] A wearable system (also referred to herein as an augmented reality (AR) system) can be configured to present two-dimensional (2D) or three-dimensional (3D) virtual images to a user. The images can be still images, frames of video, or videos, in combination or the like. A wearable system can include a wearable device that can present a VR, AR, or MR environment, alone or in combination, for user interaction. The wearable device can be a head-mounted device (HMD), which is used synonymously with AR device (ARD).

[0033] Figure 1 depicts an illustration of a mixed reality scenario with certain virtual reality objects and certain physical objects viewed by a person. In Figure 1, an MR scene 100 is depicted in which a user of the MR technology sees a real-world park-like setting 110 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of the MR technology also perceives that they "see" a robotic figure 130 standing on the real-world platform 120, and a flying cartoon-like avatar character 140 that appears to be an anthropomorphic bumblebee, although these elements do not exist in the real world.

[0034] For a 3D display to produce a true depth sensation, or more specifically, a simulated sensation of surface depth, it may be desirable to generate, for each point in the field of view of the display, an accommodation response that corresponds to that point's virtual depth. If the accommodation response to a display point does not correspond to that point's virtual depth as determined by the binocular depth cues of convergence and stereopsis, the human eye will experience accommodation conflicts, which can result in unstable imaging, harmful eye strain, headaches, and, in the absence of accommodation information, a near-complete lack of surface depth.

[0035] VR, AR, and MR experiences can be provided by a display system having a display in which images corresponding to multiple depth planes are provided to a viewer. The images may be different for each depth plane (e.g., providing a slightly different presentation of a scene or object), and the images may be focused separately by the viewer's eyes, thereby serving to provide depth cues to the user based on ocular accommodation required to focus on different image features of a scene located on different depth planes, or based on observing different image features on different depth planes that are out of focus. As discussed elsewhere herein, such depth cues provide a believable perception of depth.

[0036] FIG. 2 illustrates an example of a wearable system 200. The wearable system 200 includes a display 220 and various mechanical and electronic modules and systems to support the functionality of the display 220. The display 220 may be coupled to a frame 230 that is wearable by a user, wearer, or viewer 210. The display 220 may be positioned in front of the eyes of the user 210. The display 220 may present AR / VR / MR content to the user. The display 220 may include a head mounted display (HMD) that is worn on the user's head. In some embodiments, a speaker 240 is coupled to the frame 230 and positioned adjacent to the user's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / shapeable sound control).

[0037] The wearable system 200 may include an outwardly facing imaging system 464 (shown in FIG. 4) that observes the world in the user's surrounding environment. The wearable system 200 may also include an inwardly facing imaging system 462 (shown in FIG. 4) that may track the user's eye movements. The inwardly facing imaging system may track either one eye's movements or both eyes' movements. The inwardly facing imaging system 462 may be mounted to the frame 230 and in electrical communication with the processing module 260 or 270, which may process the image information acquired by the inwardly facing imaging system to determine, for example, the pupil diameter or orientation of the user's 210 eyes, eye movements, or eye pose.

[0038] By way of example, the wearable system 200 can capture images of the user's posture using the outward facing imaging system 464 or the inward facing imaging system 462. The images can be still images, frames of video or videos, combinations thereof, etc.

[0039] The display 220 is operably coupled (250), such as by a wired or wireless connection, to a local data processing module 260, which may be mounted in a variety of configurations, such as fixedly attached to the frame 230, fixedly attached to a helmet or hat worn by the user, integrated into headphones, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, in a belt-connected configuration).

[0040] The local processing and data module 260 may comprise a hardware processor and digital memory such as non-volatile memory (e.g., flash memory), both of which may be utilized to aid in processing, caching, and storing data. The data may include (a) data captured from sensors (e.g., that may be operatively coupled to the frame 230 or otherwise attached to the user 210), such as image capture devices (e.g., cameras in the inwardly facing and / or outwardly facing imaging systems), microphones, inertial measurement units (IMUs) (e.g., accelerometers, gravimeters, magnetometers, etc.), compasses, global positioning systems (GPS), wireless devices, or gyroscopes, or (b) data acquired or processed using the remote processing module 270 and / or remote data repository 280, possibly for transmission to the display 220 after such processing or readout. The local processing and data module 260 may be operably coupled to a remote processing module 270 or a remote data repository 280 over a communication link 262 or 264, such as via a wired or wireless communication link, such that these remote modules are available as resources to the local processing and data module 260. In addition, the remote processing module 280 and the remote data repository 280 may be operably coupled to each other.

[0041] In some embodiments, remote processing module 270 may comprise one or more processors configured to analyze and process data and / or image information. In some embodiments, remote data repository 280 may comprise a digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module, allowing for fully autonomous use from the remote module.

[0042] The human visual system is complex and providing a realistic perception of depth is difficult. Without being limited by theory, it is believed that a viewer of an object may perceive the object as three-dimensional due to a combination of vergence and accommodation. Vergence of the two eyes relative to each other (i.e., the rolling of the pupils toward or away from each other to converge the gaze of the eyes and fixate on the object) is closely linked to the focusing (or "accommodation") of the eye's lenses. Under normal conditions, changing the focus of the eye's lenses or accommodating the eyes and changing focus from one object to another object at a different distance will automatically produce a matching change in vergence for the same distance, under a relationship known as the "accommodation-vergence reflex." Similarly, a change in vergence will induce a matching change in accommodation under normal conditions. A display system that provides a better match between accommodation and vergence may create a more realistic and comfortable simulation of a three-dimensional image.

[0043] FIG. 3 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes. With reference to FIG. 3, objects at various distances from the eyes 302 and 304 on the z-axis are accommodated by the eyes 302 and 304 such that the objects are in focus. The eyes 302 and 304 assume a particular accommodated state and focus on objects at different distances along the z-axis. As a result, a particular accommodated state may be said to be associated with a particular one of the depth planes 306 having an associated focal length, whereby an object or part of an object at a particular depth plane is in focus when the eye is in an accommodated state relative to that depth plane. In some embodiments, a three-dimensional image may be simulated by providing a different presentation of an image for each of the eyes 302 and 304, and may also be simulated by providing a different presentation of an image corresponding to each of the depth planes. Although shown as separate for clarity of illustration, it should be understood that the fields of view of the eyes 302 and 304 may overlap, for example, as the distance along the z-axis increases. In addition, while shown to be flat for ease of illustration, it should be understood that the contours of the depth plane may be curved in physical space such that all features within the depth plane are in focus with the eye in a particular accommodation state. Without being limited by theory, it is believed that the human eye may typically interpret a finite number of depth planes to provide depth perception. As a result, a highly realistic simulation of perceived depth may be achieved by providing the eye with a different representation of an image corresponding to each of these limited number of depth planes. (Waveguide Stack Assembly)

[0044] FIG. 4 illustrates an example of a waveguide stack for outputting image information to a user. The wearable system 400 includes a stack of waveguides or a stacked waveguide assembly 480 that can be utilized to provide a three-dimensional perception to the eye / brain using multiple waveguides 432b, 434b, 436b, 438b, 400b. In some embodiments, the wearable system 400 can correspond to the wearable system 200 of FIG. 2, and FIG. 4 diagrammatically illustrates some portions of the wearable system 200 in more detail. For example, in some embodiments, the waveguide assembly 480 can be integrated into the display 220 of FIG. 2.

[0045] 4, the waveguide assembly 480 may include multiple features 458, 456, 454, 452 between the waveguides. In some embodiments, the features 458, 456, 454, 452 may be lenses. In other embodiments, the features 458, 456, 454, 452 may not be lenses. Rather, they may simply be spacers (e.g., cladding layers or structures to form air gaps).

[0046] The waveguides 432b, 434b, 436b, 438b, 440b or multiple lenses 458, 456, 454, 452 may be configured to transmit image information to the eye with various levels of wavefront curvature or light beam divergence. Each waveguide level may be associated with a particular depth plane and configured to output image information corresponding to that depth plane. Image injection devices 420, 422, 424, 426, 428 may be utilized to inject image information into the waveguides 440b, 438b, 436b, 434b, 432b, each of which may be configured to distribute incident light across each respective waveguide for output toward the eye 410. Light exits output surfaces of image injection devices 420, 422, 424, 426, 428 and is injected into corresponding input edges of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam of light (e.g., a collimated beam) is injected into each waveguide to output an entire field of cloned collimated beams, which may be directed towards eye 410 at a particular angle (and divergence) corresponding to the depth plane associated with the particular waveguide.

[0047] In some embodiments, the image input devices 420, 422, 424, 426, 428 are separate displays that each generate image information for input into a corresponding waveguide 440b, 438b, 436b, 434b, 432b. In some other embodiments, the image input devices 420, 422, 424, 426, 428 are the output of a single multiplexed display that may send image information to each of the image input devices 420, 422, 424, 426, 428 via, for example, one or more optical conduits (such as fiber optic cables).

[0048] A controller 460 controls the operation of the stacked waveguide assembly 480 and the image input devices 420, 422, 424, 426, 428. The controller 460 includes programming (e.g., instructions in a non-transitory computer-readable medium) that coordinates the timing and provision of image information to the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the controller 460 may be a single integrated device or a distributed system connected by a wired or wireless communication channel. The controller 460 may be part of the processing module 260 or 270 (illustrated in FIG. 2) in some embodiments.

[0049] The waveguides 440b, 438b, 436b, 434b, 432b may be configured to propagate light within each respective waveguide by total internal reflection (TIR). Each of the waveguides 440b, 438b, 436b, 434b, 432b may be planar with major top and bottom surfaces and edges extending between the major top and bottom surfaces, or may have another shape (e.g., curved). In the illustrated configuration, each of the waveguides 440b, 438b, 436b, 434b, 432b may include light extraction optical elements 440a, 438a, 436a, 434a, 432a that are configured to extract light from each respective waveguide by redirecting light propagating within the waveguide out of the waveguide to output image information to the eye 410. The extracted light may also be referred to as out-coupled light, and the light extraction optical element may also be referred to as out-coupling optical element. The extracted light beam is output by the waveguide where the light propagating in the waveguide strikes the light redirecting element. The light extraction optical element (440a, 438a, 436a, 434a, 432a) may be, for example, a reflective or diffractive optical feature. Although shown disposed on the bottom major surface of the waveguides 440b, 438b, 436b, 434b, 432b for ease of explanation and clarity of drawing, in some embodiments the light extraction optical element 440a, 438a, 436a, 434a, 432a may be disposed on the top or bottom major surface, or directly within the volume of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be attached to a transparent substrate and formed within a layer of material that forms the waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, 432b may be a monolithic piece of material and the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed on and / or within that piece of material.

[0050] Continuing with reference to FIG. 4, as discussed herein, each waveguide 440b, 438b, 436b, 434b, 432b is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 when launched into such waveguide 432b. The collimated light may represent an optical infinity focal plane. The next upper waveguide 434b may be configured to send collimated light, which passes through a first lens 452 (e.g., a negative lens) before it can reach the eye 410. The first lens 452 may be configured to generate a slight convex wavefront curvature so that the eye / brain interprets the light emerging from the next waveguide 434b as emerging from a first focal plane closer inward from optical infinity toward the eye 410. Similarly, the third upper waveguide 436b passes its output light through both a first lens 452 and a second lens 454 before reaching the eye 410. The combined refractive power of the first and second lenses 452 and 454 may be configured to produce another incremental amount of wavefront curvature such that the eye / brain interprets the light emerging from the third upper waveguide 436b as emerging from a second focal plane that is closer inward toward the person from optical infinity, which was the light from the next waveguide 434b.

[0051] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured, with the highest waveguide 440b in the stack sending its output through all of the lenses between it and the eye for an aggregate focal power that represents the focal plane closest to the person. To compensate the stack of lenses 458, 456, 454, 452 when viewing / interpreting light originating from the world 470 on the other side of the stacked waveguide assembly 480, a compensating lens layer 430 may be placed on top of the stack to compensate for the aggregate power of the lens stacks 458, 456, 454, 452 below. Such a configuration provides as many perceived focal planes as there are waveguide / lens pairs available. Both the light extraction optical elements of the waveguides and the focusing sides of the lenses may be static (e.g., not dynamic or electroactive). In some embodiments, one or both may be dynamic using electroactive features.

[0052] Continuing with reference to FIG. 4, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be configured to redirect light from their respective waveguides and output this light with the appropriate amount of divergence or collimation for the particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes can have different configurations of light extraction optical elements that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be solid or surface features that can be configured to output light at a particular angle. For example, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be volume holograms, surface holograms, and / or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published June 25, 2015, which is incorporated by reference in its entirety.

[0053] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffractive features, or "diffractive optical elements" (also referred to herein as "DOEs"), that form a diffraction pattern. Preferably, the DOEs have a relatively low diffraction efficiency, such that only a portion of the light of the beam is deflected toward the eye 410 with each intersection of the DOE, while the remainder continues traveling through the waveguide via total internal reflection. The light carrying the image information can thus be split into several related exit beams that exit the waveguide at multiple locations, resulting in a very uniform pattern of exit emission toward the eye 304 for this particular collimated beam bouncing within the waveguide.

[0054] In some embodiments, one or more DOEs may be switchable between an actively diffracting "on" state and a less diffracting "off" state. For example, a switchable DOE may comprise a layer of polymer dispersed liquid crystal, where the microdroplets comprise a diffractive pattern in a host medium, and the refractive index of the microdroplets can be switched to substantially match the refractive index of the host material (in which case the pattern does not diffract incident light significantly), or the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0055] In some embodiments, the number and distribution of depth planes or depth of field can be dynamically varied based on the size or orientation of the pupil of the viewer's eye. The depth of field can vary inversely with the size of the viewer's pupil. As a result, as the size of the viewer's pupil decreases, the depth of field increases, so that one plane that is indistinguishable because its location exceeds the focal depth of the eye becomes distinguishable and appears more focused with the reduction of the pupil size and the corresponding increase of the depth of field. Similarly, the number of spaced depth planes used to present different images to the viewer can be reduced with the reduced pupil size. For example, the viewer may not be able to clearly perceive the details of both the first and second depth planes at one pupil size without the eye accommodating from one depth plane to the other. However, these two depth planes may be sufficient to be simultaneously in focus for the user at different pupil sizes without changing accommodation.

[0056] In some embodiments, the display system may vary the number of waveguides that receive image information based on a determination of pupil size and / or orientation, or upon receiving an electrical signal indicative of a particular pupil size and / or orientation. For example, if the user's eye is unable to distinguish between two depth planes associated with two waveguides, the controller 460 may be configured or programmed to stop providing image information to one of these waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments in which the DOE for a waveguide is switchable between on and off states, the DOE may be switched to the off state when the waveguide receives image information.

[0057] In some embodiments, it may be desirable to have the exit beam meet a condition of having a diameter less than the diameter of the viewer's eye. However, meeting this condition may be difficult in light of the variability in the size of the viewer's pupil. In some embodiments, this condition is met over a wide range of pupil sizes by varying the size of the exit beam in response to a determination of the size of the viewer's pupil. For example, as the pupil size decreases, the size of the exit beam may also decrease. In some embodiments, the exit beam size may be varied using a variable aperture.

[0058] The wearable system 400 may include an outwardly facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the field of view (FOV) of the world camera, and the imaging system 464 is sometimes also referred to as the FOV camera. The entire area available for viewing or imaging by a viewer may be referred to as the ocular field of view (FOR). The FOR may include a solid angle of 4π steradians encompassing the wearable system 400, since the wearer can move their body, head, or eyes and perceive virtually any direction in space. In other situations, the wearer's movements may be more constrained, and thus the wearer's FOR may be for a smaller solid angle. Images obtained from the outwardly facing imaging system 464 may be used to track gestures (e.g., hand or finger gestures) made by the user, detect objects in the world 470 in front of the user, etc.

[0059] The wearable system 400 may also include an inwardly facing imaging system 466 (e.g., a digital camera) that observes user movements such as eye and face movements. The inwardly facing imaging system 466 may be used to capture images of the eye 410 and determine the size or orientation of the pupil of the eye 304. The inwardly facing imaging system 466 may be used to obtain images for use in determining the direction the user is looking (e.g., eye pose) or for biometric identification of the user (e.g., via iris identification). In some embodiments, at least one camera may be utilized for each eye to determine the pupil size or eye pose of each eye independently, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye. In some other embodiments, the pupil diameter or orientation of only a single eye 410 (e.g., using only a single camera per pair of eyes) is determined and assumed to be similar for both eyes of the user. Images obtained by the inward-facing imaging system 466 may be analyzed to determine the user's eye posture or mood, which may be used by the wearable system 400 to determine audio or visual content to be presented to the user. The wearable system 400 may include one or more head posture sensors (accelerometers, gyroscopes, etc.) , or magnetometer) or the like may be used to determine head pose (e.g., head position or head orientation).

[0060] The wearable system 400 may include a user input device 466 through which a user may input commands into the controller 460 and interact with the wearable system 400. For example, the user input device 466 may include a trackpad, a touch screen, a joystick, a multi-degree-of-freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a wand, a tactile device, a totem (e.g., serving as a virtual user input device), and the like. A multi-DOF controller may sense user input in any possible translation (e.g., left / right, forward / backward, or up / down) or rotation (e.g., yaw, pitch, or roll) of some or all of the controller. A multi-DOF controller that supports translation may be referred to as 3DOF, while a multi-DOF controller that supports translation and rotation may be referred to as 6DOF. In some cases, a user may use a finger (e.g., thumb) to press or swipe on the touch-sensitive input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held by the user's hand during use of the wearable system 400. The user input device 466 may communicate with the wearable system 400 in wired or wireless communication.

[0061] FIG. 5 shows an example of an exit beam output by a waveguide. Although one waveguide is shown, it should be understood that other waveguides in the waveguide assembly 480 may function similarly and that the waveguide assembly 480 includes multiple waveguides. Light 520 is launched into the waveguide 432b at the input edge 432c of the waveguide 432b and propagates within the waveguide 432b by TIR. At the point where the light 520 impinges on the DOE 432a, a portion of the light exits the waveguide as an exit beam 510. Although the exit beams 510 are shown as substantially parallel, they may also be redirected to propagate to the eye 410 at an angle (e.g., forming a diverging exit beam) depending on the depth plane associated with the waveguide 432b. It should be understood that a substantially parallel exit beam may indicate a waveguide with light extraction optics that outcouples light and forms an image that appears to be set on the depth plane at a long distance (e.g., optical infinity) from the eye 410. Other waveguides or other sets of light extraction optics may output a more divergent exit beam pattern, which would require the eye 410 to accommodate to a closer distance and for the eye 410 to focus on the retina, and would be interpreted by the brain as light from a distance closer to the eye 410 than optical infinity.

[0062] FIG. 6 is a schematic diagram illustrating an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem used in generating a multifocal stereoscopic display, an image, or a bright field. The optical system can include a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem. The optical system can be used to generate a multifocal stereoscopic display, an image, or a bright field. The optical system can include one or more primary planar waveguides 632a (only one is shown in FIG. 6) and one or more DOEs 632b associated with each of at least some of the primary waveguides 632a. The planar waveguides 632b can be similar to the waveguides 432b, 434b, 436b, 438b, 440b discussed with reference to FIG. 4. The optical system may employ a distribution waveguide device to relay light along a first axis (vertical or Y-axis in the illustration of FIG. 6) and expand the effective exit pupil of the light along the first axis (e.g., Y-axis). The distribution waveguide device may include, for example, a distribution planar waveguide 622b and at least one DOE 622a (illustrated by a double dashed line) associated with the distribution planar waveguide 622b. The distribution planar waveguide 622b may be similar or the same in at least some respects as the primary planar waveguide 632b, which has a different orientation therefrom. Similarly, the at least one DOE 622a may be similar or the same in at least some respects as the DOE 632a. For example, the distribution planar waveguide 622b or the DOE 622a may be made of the same material as the primary planar waveguide 632b or the DOE 632a, respectively. The embodiment of the optical display system 600 shown in FIG. 6 can be integrated into the wearable system 200 shown in FIG.

[0063] The relayed and exit pupil expanded light may be optically coupled from the distribution waveguide arrangement into one or more primary planar waveguides 632b. The primary planar waveguides 632b may relay the light along a second axis, preferably orthogonal to the first axis (e.g., the horizontal or X-axis in the diagram of FIG. 6). It should be noted that the second axis may be a non-orthogonal axis to the first axis. The primary planar waveguides 632b may expand the effective exit pupil of the light along the second axis (e.g., the X-axis). For example, the distribution planar waveguide 622b may relay and expand the light along the vertical or Y-axis and pass the light to a primary planar waveguide 632b that may relay and expand the light along the horizontal or X-axis.

[0064] The optical system may include one or more colored light sources (e.g., red, green, and blue laser light) 610 that may be optically coupled into a proximal end of a single mode optical fiber 640. The distal end of the optical fiber 640 may be threaded into or received through a hollow tube 642 of piezoelectric material. The distal end protrudes from the tube 642 as an unfixed flexible cantilever 644. The piezoelectric tube 642 may be associated with four quadrant electrodes (not shown). The electrodes may be plated, for example, on the outside, outer surface or outer periphery, or diameter of the tube 642. A core electrode (not shown) may also be located in the core, center, inner periphery, or inner diameter of the tube 642.

[0065] For example, drive electronics 650, electrically coupled via wires 660, drive opposing pairs of electrodes to bend the piezoelectric tube 642 in two axes independently. The protruding distal tip of the optical fiber 644 has a mechanical resonant mode. The frequency of the resonance may depend on the diameter, length, and material properties of the optical fiber 644. By oscillating the piezoelectric tube 642 near the first mechanical resonant mode of the fiber cantilever 644, the fiber cantilever 644 may be caused to oscillate and sweep through a large deflection.

[0066] By stimulating resonant vibrations in two axes, the tip of the fiber cantilever 644 is scanned in two axial directions over an area that fills a two-dimensional (2-D) scan. By modulating the intensity of the light source 610 in synchronism with the scanning of the fiber cantilever 644, the light emitted from the fiber cantilever 644 can form an image. A description of such a setup is provided in U.S. Patent Publication No. 2014 / 0003762, which is incorporated herein by reference in its entirety.

[0067] The optical coupler subsystem components can collimate the light emitted from the scanning fiber cantilever 644. The collimated light can be reflected by the mirrored surface 648 into the narrow distribution planar waveguide 622b, which includes at least one diffractive optical element (DOE) 622a. The collimated light can propagate perpendicularly (with respect to the view of FIG. 6) along the distribution planar waveguide 622b by TIR, and in doing so, repeatedly intersect with the DOE 622a. The DOE 622a preferably has a low diffraction efficiency. This can cause a portion of the light (e.g., 10%) to be diffracted toward the edge of the larger primary planar waveguide 632b at each point of intersection with the DOE 622a, and a portion of the light to continue on its original trajectory down the length of the distribution planar waveguide 622b via TIR.

[0068] At each point of intersection with the DOE 622a, additional light can be diffracted toward the entrance of the primary planar waveguide 632b. By splitting the incoming light into multiple out-coupled sets, the exit pupil of the light can be vertically expanded by the DOE 4 in the distribution planar waveguide 622b. This vertically expanded light out-coupled from the distribution planar waveguide 622b can enter the edge of the primary planar waveguide 632b.

[0069] Light entering the primary waveguide 632b can propagate horizontally (with respect to the diagram of FIG. 6) along the primary waveguide 632b via TIR. As the light propagates horizontally along at least a portion of the length of the primary waveguide 632b via TIR, it intersects the DOE 632a at multiple points. The DOE 632a advantageously has a phase profile that is the sum of a linear diffraction pattern and a radially symmetric diffraction pattern, and may be designed or configured to produce both deflection and focusing of the light. The DOE 632a advantageously may have a low diffraction efficiency (e.g., 10%) such that only a portion of the light in the beam is deflected toward the viewer's eye with each intersection of the DOE 632a, while the remainder of the light continues to propagate through the primary waveguide 632b via TIR.

[0070] At each point of intersection between the propagating light and the DOE 632a, a portion of the light is diffracted toward an adjacent face of the primary waveguide 632b, allowing the light to escape TIR and emerge from the face of the primary waveguide 632b. In some embodiments, the radially symmetric diffraction pattern of the DOE 632a additionally imparts a focal level to the diffracted light, both shaping the optical wavefronts of the individual beams (e.g., imparting a curvature) and steering the beams to angles that match the designed focal level.

[0071] Thus, these different paths can cause light to be coupled out of the primary planar waveguide 632b by resulting in different fill patterns at the exit pupil, focus levels, and / or multiplicities of the DOE 632a at different angles. Different fill patterns at the exit pupil can be advantageously used to generate brightfield displays with multiple depth planes. Each layer or set of layers in a stack in a waveguide assembly (e.g., three layers) can be employed to generate a respective color (e.g., red, blue, green). Thus, for example, a first set of three adjacent layers can be employed to generate red, blue, and green light at a first focal depth, respectively. A second set of three adjacent layers can be employed to generate red, blue, and green light at a second focal depth, respectively. Multiple sets can be employed to generate full 3D or 4D color image brightfields with various focal depths. (Other components of the wearable system)

[0072] In many implementations, the wearable system may include other components in addition to or as an alternative to the components of the wearable system described above. The wearable system may include, for example, one or more tactile devices or components. The tactile devices or components may be operable to provide a sense of touch to the user. For example, the tactile device or component may provide a sense of pressure and / or texture upon touching the virtual content (e.g., a virtual object, virtual tool, other virtual structure). The haptics may replicate the sensation of a physical object that the virtual object represents, or may replicate the sensation of an imaginary object or character (e.g., a dragon) that the virtual content represents. In some implementations, the tactile device or component may be worn by the user (e.g., a user-wearable glove). In some implementations, the tactile device or component may be held by the user.

[0073] A wearable system may include one or more physical objects that can be manipulated by a user, for example, to enable input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as, for example, pieces of metal or plastic, walls, table surfaces, etc. In some implementations, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, a totem may simply provide a physical surface, and the wearable system may render a user interface to appear to the user as being on one or more surfaces of the totem. For example, the wearable system may render images of a computer keyboard and trackpad to appear to reside on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and virtual trackpad to appear on the surface of a thin rectangular plate of aluminum that serves as the totem. The rectangular plate itself does not have any physical keys or trackpads or sensors. However, the wearable system may detect user manipulation or interaction or touch with the rectangular plate as a selection or input made via a virtual keyboard or virtual trackpad. User input device 466 (shown in FIG. 4) may be an embodiment of the totem, which may include a trackpad, touchpad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. A user may use the totem alone or in combination with posture to interact with the wearable system and / or other users.

[0074] Examples of tactile devices and totems usable with the wearable devices, HMDs, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated by reference in its entirety. Exemplary Wearable Systems, Environments, and Interfaces

[0075] The wearable system may employ various mapping-related techniques to achieve a high depth of field in the rendered light field. In mapping a virtual world, it is advantageous to capture all features and points in the real world and accurately depict the virtual objects in relation to the real world. To this end, FOV images captured from a user of the wearable system can be added to the world model by including new photos that convey information about various points and features of the real world. For example, the wearable system can collect a set of map points (such as 2D or 3D points), find new map points, and render a more accurate version of the world model. The world model of the first user can be communicated (e.g., via a network such as a cloud network) to a second user so that the second user can experience the world surrounding the first user.

[0076] 7 is a block diagram of an example of a MR environment 700. The MR environment 700 may be configured to receive inputs (e.g., visual inputs 702 from a user's wearable system, stationary inputs 704 such as room cameras, sensory inputs 706 from various sensors, gestures, totems, eye tracking, user inputs, etc. from user input devices 466) from one or more user wearable systems (e.g., wearable system 200 or display system 220) or stationary room systems (e.g., room cameras, etc.). The wearable systems may use various sensors (e.g., accelerometers, gyroscopes, temperature sensors, movement sensors, depth sensors, GPS sensors, inward-facing imaging systems, outward-facing imaging systems, etc.) to determine the location and various other attributes of the user's environment. This information may be further supplemented with information from stationary cameras in the room that may provide images from different perspectives or various cues. Image data acquired by the cameras (e.g., the room cameras and / or the outward-facing imaging system cameras) may be compiled into a set of mapping points.

[0077] One or more object recognizers 708 can crawl through the received data (e.g., a collection of points), recognize or map the points, tag the images, and leverage a map database 710 to associate semantic information with the objects. The map database 710 can comprise various points and their corresponding objects collected over time. The various devices and the map database can be connected to each other through a network (e.g., a LAN, a WAN, etc.) and can be accessed to the cloud.

[0078] Based on this information and the collection of points in the map database, the object recognizers 708a-708n may recognize objects in the environment. For example, the object recognizers may recognize faces, people, windows, walls, user input devices, televisions, other objects in the user's environment, etc. One or more object recognizers may be specialized for objects with certain characteristics. For example, the object recognizer 708a may be used to recognize faces, while another object recognizer may be used to recognize totems.

[0079] Object recognition can be performed using a variety of computer vision techniques. For example, the wearable system can analyze images acquired by the outward-facing imaging system 464 (shown in FIG. 4) and perform scene reconstruction, event detection, video tracking, object recognition, object pose estimation, learning, indexing, motion estimation, image restoration, etc. One or more computer vision algorithms can be used to perform these tasks. Non-limiting examples of computer vision algorithms include Scale Invariant Feature Transform (SIFT), Speed ​​Up Robust Features (SURF), Orientation FAST and Rotation BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retinal Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, visual simultaneous localization and mapping (vSLAM) techniques, sequential Bayes estimators (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, adaptive thresholding (and other thresholding techniques), iterative nearest neighbor (ICP), semi-global matching (SGM), semi-global block matching (SGBM), feature point histograms, various machine learning algorithms (e.g., support vector machines, k-nearest neighbor algorithms, naive Bayes, neural networks (including convolutional or deep neural networks), or other supervised / unsupervised models, etc.), and the like.

[0080] In some embodiments, object recognition can be performed by various machine learning algorithms. Once trained, the machine learning algorithms can be stored by the HMD. Some examples of machine learning algorithms can include supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., ordinary least squares regression, etc.), instance-based algorithms (e.g., learning vector quantization, etc.), decision tree algorithms (e.g., classification and regression trees, etc.), Bayesian algorithms (e.g., naive Bayes, etc.), clustering algorithms (e.g., k-means clustering, etc.), association rule learning algorithms (e.g., a priori algorithm, etc.), artificial neural network algorithms (e.g., Perceptron, etc.), deep learning algorithms (e.g., Deep Boltzmann Machine, i.e., deep neural networks, etc.), dimensionality reduction algorithms (e.g., principal component analysis, etc.), ensemble algorithms (e.g., stacked generalization, etc.), and / or other machine learning algorithms. In some embodiments, individual models can be customized for individual datasets. For example, the wearable device can generate or store a base model. The base model may be used as a starting point to generate additional models specific to a data type (e.g., a particular user in a telepresence session), a data set (e.g., a set of additional images acquired of a user in a telepresence session), a conditional situation, or other variations. In some embodiments, the wearable HMD may be configured to utilize multiple techniques to generate models for analysis of the aggregated data. Other techniques may include using predefined thresholds or data values.

[0081] Based on this information and the set of points in the map database, the object recognizers 708a-708n can recognize objects, complement the objects with semantic information, and bring them to life. For example, if the object recognizer recognizes that the set of points is a door, the system can attach some semantic information (e.g., a door has a hinge and has 90 degree movement around the hinge). If the object recognizer recognizes that the set of points is a mirror, the system can attach the semantic information that a mirror has a reflective surface that can reflect an image of an object in a room. Over time, the map database grows as the system (which may reside locally or be accessible through a wireless network) accumulates more data from the world. Once an object is recognized, the information can be transmitted to one or more wearable systems. For example, the MR environment 700 can include information about a scene generating in California. The environment 700 can be transmitted to one or more users in New York. Based on the data received from the FOV camera and other inputs, an object recognizer and other software components can map points collected from various images, recognize objects, etc., so that the scene can be accurately "passed" to a second user who may be in a different part of the world. The environment 700 may also use a topology map for localization purposes.

[0082] 8 is a process flow diagram of an example method 800 for rendering virtual content in relation to recognized objects. Method 800 describes how a virtual scene may be presented to a user of a wearable system. The user may be geographically remote from the scene. For example, a user may be in New York but may wish to view a scene currently taking place in California, or may wish to go for a walk with a friend who is in California.

[0083] In block 810, the wearable system may receive input from the user and other users regarding the user's environment. This may be accomplished through various input devices and knowledge already held in the map database. The user's FOV camera, sensors, GPS, eye tracking, etc. communicate information to the system in block 810. The system may determine sparse points based on this information in block 820. The sparse points may be used in determining pose data (e.g., head pose, eye pose, body pose, or hand gestures) that may be used in displaying and understanding the orientation and position of various objects in the user's surroundings. The object recognizers 708a, 708n may crawl through these collected points in block 830 and recognize one or more objects using the map database. This information may then be communicated to the user's individual wearable system in block 840, and the desired virtual scene may be displayed to the user in block 850 as appropriate. For example, a desired virtual scene (eg, user in CA) may be displayed in the proper orientation, position, etc., relative to various objects and other surroundings of the user in New York.

[0084] FIG. 9 is a block diagram of another example of a wearable system. In this example, the wearable system 900 includes a map that may include map data about the world. The map may reside partially locally on the wearable system and partially in a networked storage location (e.g., in a cloud system) accessible by a wired or wireless network. An attitude process 910 may run on the wearable computing architecture (e.g., processing module 260 or controller 460) and utilize data from the map to determine the position and orientation of the wearable computing hardware or the user. The attitude data may be calculated from data collected in real time as the user experiences the system and operates within the world. The data may include images, data from sensors (such as an inertial measurement unit (IMU) that may comprise accelerometers, gyroscopes, magnetometers, or a combination of such components), and surface information about objects in the real or virtual environment.

[0085] The sparse representation can be the output of a simultaneous localization and mapping (SLAM or V-SLAM, which refers to configurations where the input is image / vision only) process. The system can be configured to find the things that make up the world, as well as the locations of various components in the world. Poses can be building blocks to accomplish many goals, including filling in maps and using data from maps.

[0086] In one embodiment, the sparse point locations may not be entirely adequate by themselves, and further information may be required to generate a multifocal AR, VR, or MR experience. A dense representation, generally referring to depth map information, may be utilized to at least partially fill this gap. Such information may be calculated from a process referred to as stereoscopic vision 940, where depth information is determined using techniques such as triangulation or time-of-flight sensing. Image information and active patterns (such as infrared patterns generated using active projectors) may serve as inputs to the stereoscopic vision process 940. A significant amount of depth map information may be fused together, some of which may be summarized using a surface representation. For example, a mathematically definable surface may be an efficient (e.g., compared to a large-scale point cloud) and easy-to-use input to other processing devices such as a game engine. Thus, the output of the stereoscopic vision process (e.g., depth map) 940 may be combined in a fusion process 930. Pose may be an input to this fusion process 930 as well, and the output of fusion 930 becomes an input to fill the map process 920. Sub-surfaces can connect to each other to form larger surfaces, such as in topographic mapping, making the map a large-scale hybrid of points and surfaces.

[0087] Various inputs may be utilized to resolve various aspects in the mixed reality process 960. For example, in the embodiment depicted in FIG. 9, game parameters may be inputs to determine that a user of the system is playing a monster battle game with one or more monsters in various locations, that a monster is dead, that it is fleeing under various conditions (such as when the user shoots the monster), walls or other objects in various locations, etc. A world map may contain information about where such objects are relative to each other, which is another useful input for mixed reality. Attitude relative to the world is also an input and plays an important role for almost any interactive system.

[0088] Controls or inputs from the user are another input to the wearable system 900. As described herein, user inputs can include visual inputs, gestures, totems, audio inputs, sensory inputs, and the like. To move around or play a game, for example, the user may need to command the wearable system 900 as to what they want to do. There are various forms of user control that can be utilized beyond just moving around in space themselves. In one embodiment, a totem (e.g., a user input device), or an object such as a toy gun, may be held by the user and tracked by the system. The system would preferably be configured to know that the user is holding an item and understand the type of interaction the user is having with the item (e.g., if the totem or object is a gun, the system may be configured to understand not only the location and orientation, but also whether the user is clicking a trigger or other sensitive button or element that may be equipped with a sensor such as an IMU, which may aid in determining what is happening even when such activity is not within the field of view of any of the cameras).

[0089] Hand gesture tracking or recognition may also provide input information. The wearable system 900 may be configured to track and interpret hand gestures to press a button, gesture left or right, stop, grab, hold, etc. For example, in one configuration, a user may want to flip through email or calendar in a non-gaming environment, or to "fist bump" with another person or player. The wearable system 900 may be configured to utilize a minimal amount of hand gestures that may or may not be dynamic. For example, gestures may be simple static gestures, such as spreading the hand to indicate stop, thumbs up to indicate OK, thumbs down to indicate not OK, or flipping the hand left and right or up and down to indicate a directional command.

[0090] Eye tracking is another input (e.g., to track where the user is looking, control display technology, render at a particular depth or range). In one embodiment, eye vergence can be determined using triangulation, and then accommodation can be determined using a vergence / accommodation model developed for that particular person.

[0091] With regard to the camera system, the exemplary wearable system 900 shown in FIG. 9 may include three pairs of cameras: a pair of relatively wide FOV or passive SLAM cameras arranged on either side of the user's face, a different pair of cameras pointed in front of the user to handle the stereoscopic imaging process 940 and also capture hand gestures and totem / object trajectories in front of the user's face. The FOV cameras and pair of cameras for the stereo process 940 may be part of the outward facing imaging system 464 (shown in FIG. 4). The wearable system 900 may include eye tracking cameras pointed at the user's eyes (which may be part of the inward facing imaging system 462 shown in FIG. 4) to triangulate eye vectors and other information. The wearable system 900 may also include one or more textured light projectors (such as infrared (IR) projectors) to inject texture into the scene.

[0092] FIG. 10 is a process flow diagram of an example method 1000 for determining user input to a wearable system. In this example, a user may interact with a totem. A user may have multiple totems. For example, a user may have one totem designated for social media applications, another totem for playing games, etc. In block 1010, the wearable system may detect the movement of the totem. The movement of the totem may be recognized through an outward facing system or may be detected through sensors (e.g., tactile gloves, image sensors, hand tracking devices, eye tracking cameras, head pose sensors, etc.).

[0093] Based at least in part on the detected gestures, eye poses, head poses, or inputs through the totem, the wearable system detects the position, orientation, and / or movement of the totem (or the user's eyes or head or gestures) relative to a reference frame in block 1020. The reference frame may be a set of map points based on which the wearable system translates the totem's (or user's) movements into actions or commands. In block 1030, the user's interactions with the totem are mapped. Based on the mapping of the user interactions to the reference frame 1020, the system determines the user input in block 1040.

[0094] For example, a user may move a totem or physical object back and forth to indicate turning a virtual page and moving to the next page, or moving from one user interface (UI) display screen to another UI screen. As another example, a user may move their head or eyes to look at different real or virtual objects in the user's FOR. If the user's gaze at a particular real or virtual object is longer than a threshold time, that real or virtual object may be selected as the user input. In some implementations, the vergence of the user's eyes can be tracked and an accommodation / vergence model can be used to determine the accommodation state of the user's eyes, which provides information about the depth plane on which the user is focusing. In some implementations, the wearable system can use ray casting techniques to determine real or virtual objects that are in line with the direction of the user's head or eye pose. In various implementations, the ray casting techniques can include casting a thin bundle of rays with substantially little lateral width, or casting a ray (e.g., a cone or a truncated cone) with substantial lateral width.

[0095] The user interface may be projected by a display system as described herein (such as display 220 in FIG. 2). It may also be displayed using a variety of other techniques, such as one or more projectors. A projector may project an image onto a physical object, such as a canvas or a sphere. Interactions with the user interface may be tracked using one or more cameras outside or part of the system (e.g., using inward-facing imaging system 462 or outward-facing imaging system 464).

[0096] 11 is a process flow diagram of an example of a method 1100 for interacting with a virtual user interface. The method 1100 may be performed by a wearable system described herein.

[0097] At block 1110, the wearable system may identify a particular UI. The type of UI may be predetermined by the user. The wearable system may identify that a particular UI needs to be populated based on user input (e.g., gestures, visual data, audio data, sensory data, direct commands, etc.). At block 1120, the wearable system may generate data for a virtual UI. For example, data associated with the UI's boundaries, general structure, shape, etc. may be generated. In addition, the wearable system may determine map coordinates of the user's physical location so that the wearable system may display the UI in relation to the user's physical location. For example, if the UI is body-centered, the wearable system may determine coordinates of the user's physical standing position, head pose, or eye pose, so that a ring UI may be displayed around the user, or a planar UI may be displayed on a wall or in front of the user. If the UI is hand-centered, map coordinates of the user's hand may be determined. These map points may be derived through data received through the FOV camera, sensory input, or any other type of collected data.

[0098] In block 1130, the wearable system may send data from the cloud to the display, or data may be sent from a local database to the display component. In block 1140, a UI is displayed to the user based on the sent data. For example, a bright field display may project the virtual UI into one or both of the user's eyes. Once the virtual UI is generated, the wearable system may simply wait for a command from the user in block 1150 and generate more virtual content on the virtual UI. For example, the UI may be a body-centered ring around the user's body. The wearable system may then wait for a command (gesture, head or eye movement, input from a user input device, etc.) and, if recognized (block 1160), the virtual content associated with the command may be displayed to the user (block 1170). As an example, the wearable system may wait for a user's hand gesture before mixing multiple steam tracks.

[0099] Additional examples of wearable systems, UIs, and user experiences (UX) are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated by reference in its entirety. (Example objects in the FOR and FOV)

[0100] FIG. 12A diagrammatically illustrates an example of a kinetic field of view (FOR) 1200, a world camera field of view (FOV) 1270, a user's field of view 1250, and a user's fixation field of view 1290. As described with reference to FIG. 4, the FOR 1200 includes a portion of the user's surrounding environment perceivable by the user via the wearable system. The FOR may include substantially all of the 4π steradian solid angle encompassing the wearable system 400, since the wearer can move their body, head, or eyes and perceive substantially any direction in space. In other situations, the user's movement may be more constricted, and thus the user's FOR may be for a smaller solid angle.

[0101] The world camera field of view 1270 may include a portion of the user's FOR that is currently being observed by the outwardly facing imaging system 464. With reference to FIG. 4, the world camera field of view 1270 may include the world 470 that is observed by the wearable system 400 at a given time. The size of the world camera FOV 1270 may depend on the optical properties of the outwardly facing imaging system 464. For example, the outwardly facing imaging system 464 may include a wide-angle camera that may image a 190 degree space around the user. In some implementations, the world camera FOV 1270 may be larger than or equal to the natural FOV of the user's eyes.

[0102] The user's FOV 1250 may include a portion of the FOR 1200 that the user is perceiving at a given time. The FOV may depend on the size or optical properties of the wearable device's display. For example, the AR display may include optics that provide AR functionality only when the user looks through a particular portion of the display. The FOV 1250 may correspond to a solid angle that is perceivable by the user when looking through an AR display, such as, for example, a stacked waveguide assembly 480 (FIG. 4) or a planar waveguide 600 (FIG. 6). In an embodiment, the user's FOV 1250 may be smaller than the natural FOV of the user's eyes.

[0103] The wearable system may also determine the user's field of fixation (FOF) 1290. The FOF 1290 may include a portion of the FOV 1250 on which the user's eyes may fixate (e.g., maintain visual gaze at). The FOF 1290 may be smaller than the user's FOV 1250, e.g., the FOF may range from a few degrees to about 5 degrees. As a result, the user may perceive some virtual objects within the FOV 1250 that are not within the field of fixation 1290 but are within the user's peripheral FOV.

[0104] FIG. 12B diagrammatically illustrates an example of virtual objects in a user's FOV 1250 and in a FOR 1200. In FIG. 12B, the FOR 1200 can include a group of objects (e.g., 1210, 1220, 1230, 1242, and 1244) that can be perceived by the user through the wearable system. The objects in the FOR 1200 can be virtual or physical objects. For example, the FOR 1200 can include physical objects such as chairs, sofas, walls, etc. The virtual objects can include operating system objects such as, for example, a trash can for deleted files, a terminal for entering commands, a file manager for accessing files or directories, icons, menus, applications for audio or video streaming, notifications from the operating system, etc. The virtual objects can also include objects in applications such as, for example, avatars, virtual objects in games, graphics or images, etc. Some virtual objects can be both operating system objects and objects in applications. In some embodiments, the wearable system can add virtual elements to existing physical objects. For example, the wearable system may add a virtual menu associated with a television in a room, and the virtual menu may give the user options to turn on the television or change the channel using the wearable system.

[0105] A virtual object may be a three-dimensional (3D), two-dimensional (2D), or one-dimensional (1D) object. For example, a virtual object may be a 3D coffee mug (which may represent a virtual control for a physical coffee maker). A virtual object may also be a 2D graphical representation of a clock (displaying the current time to the user). In some implementations, one or more virtual objects may be displayed within (or associated with) another virtual object. A virtual coffee mug may be shown inside the user interface plane, but the virtual coffee mug appears to be 3D within this 2D planar virtual space.

[0106] The objects in the user's FOR can be part of a world map, as described with reference to FIG. 9. Data associated with the objects (e.g., location, semantic information, properties, etc.) can be stored in various data structures, such as, for example, arrays, lists, trees, hashes, graphs, etc. An index for each stored object can be determined, for example, by the object's location, if applicable. For example, the data structure can index the objects by a single coordinate, such as the object's distance from a base position (e.g., distance to the left or right of the base position, distance from the top or bottom of the base position, or distance per depth from the base position). The base position can be determined based on the user's position (such as the user's head or eye position). The base position can also be determined based on the position of a virtual or physical object (such as a target object) in the user's environment. In this way, the 3D space in the user's environment can be folded into a 2D user interface, and the virtual objects are arranged according to the object's distance from the base position. (Use of reticle)

[0107] 12B , the VR / AR / MR system can display a virtual reticle 1256, which may include a movable indicator that identifies the user's position within the FOV 1250. For example, the reticle 1256 may represent the user's gaze direction, such as a fixation field, a point that will be affected by input from the user, etc. The appearance of the reticle 1256 can take any of a variety of different colors, contours, shapes, symbols, sizes, images, graphics, combinations, etc. For example, the reticle 1256 can take a variety of shapes, such as a cursor, a geometric cone, a thin beam, an arrow, an oval, a circle, a hit point, a polygon, or other 1D, 2D, or 3D shapes. The reticle 1256 may be a virtual object that may be capable of being fixed not only within the rig space (e.g., a coordinate system associated with the wearable device, such as a Cartesian xyz coordinate system), but also within the user's 3D environment. The reticle 1256 may be represented by a virtual object that a user can drag and drop (e.g., from a location in the rig space) to a particular location in the user's 3D space.

[0108] A user can move their body, head, or eyes to move the reticle 1256. For example, a change in the user's posture (e.g., head posture, body posture, or eye gaze) can alter the location of the reticle 1256 within the FOV 1250 and / or alter what is shown or observable within the FOV 1250. Similarly, the reticle 1256 can be controlled through a user input device, such as a track pad, touch screen, joystick, multiple degree of freedom (DOF) controller, capacitive sensing device, game controller, keyboard, mouse, directional pad (D-pad), wand, haptic device, totem (e.g., functioning as a virtual user input device), etc. For example, as the user moves their hand over the user input device, the reticle 1256 can move from a first position to a second position.

[0109] The reticle 1256 may be used to select, view, or point to an object, such as one of the objects 1210, 1242, 1244, 1230, 1220, by moving the reticle 1256 to hover over or point at the target object. For example, to effectively align the target object with the reticle 1256, the user may tilt, turn, or redirect their head to a position corresponding to the location of the target object. Once the reticle 1256 and the target object are aligned, the user may select the target object over which the reticle 1256 is hovering or pointing. In an embodiment, the target object may also receive a focus indicator (e.g., a virtual light beam or other graphical highlight emanating from the reticle 1256 or the selected object). (Acceleration of reticle position)

[0110] 13A-13B demonstrate an example of accelerating reticle movement in response to a change in head pose. The environment 1300A, 1300B includes a user 1304 wearing a wearable system 1320, such as the system 200 of FIG. 2, and further includes a representation of the ocular field of view (FOR) 1310 of the wearable system 1320. As described with reference to FIG. 4, the FOR 1310 includes a portion of the environment around the user 1304 that can be perceived by the user via the wearable system 1320.

[0111] The user 1304 can move its body, head, or eyes to perceive virtually any direction within the FOR 1310. For example, the user's head may have multiple degrees of freedom. As the user's head moves (e.g., tilts or turns), the user's head pose changes. The change in head pose can be quantified by determining the change in angle relative to a reference head pose vector. The reference head pose can be any head pose of the user. For example, given coordinate values ​​in the xyz coordinate system shown in FIGS. 13A and 13B, the reference head pose vector may correspond to the position of the user's head when each of the following planes are orthogonal to each other: a coronal plane of the user's head (e.g., a vertical plane that divides the body into abdominal and dorsal parts), a sagittal plane of the user's head (e.g., an anatomical plane that divides the body into right and left parts), and an axial plane of the user's head (e.g., a plane that divides the body into upper and lower parts that is approximately perpendicular to the spine).

[0112] In some applications, the user may look more frequently in a particular direction, and the system may adjust the reference head pose to reduce or improve neck strain. For example, both a game related to shooting flying birds and a stargazing application that teaches the user about constellations may involve the user periodically or constantly looking up. Similarly, a game of chess may involve the user periodically or constantly looking down. Other games or applications may involve the user periodically or constantly looking left, right, diagonally, etc. Thus, in some cases, the reference head pose may be updated to hedge against the head pose associated with this more frequent direction. For example, the reference head pose may be the average or most common user head pose over a particular period of time, or the average or most common user head pose while the user is using a particular application or playing a particular game. By configuring the reference head pose to hedge against or match a head pose associated with a user's usual or average head pose, the system may reduce or improve neck strain.

[0113] The coordinate system in Figures 13A and 13B shows three angular degrees of freedom (e.g., yaw, pitch, and roll) that can be used to measure head pose relative to a reference head pose vector. For example, a user's head can tilt forward or backward (e.g., pitch), turn left or right (e.g., yaw), or tilt sideways (e.g., roll), and the angular difference between the new head pose and the reference head pose vector can be determined to quantify the change. In other implementations, other techniques or angle representations for measuring head pose can be used, such as quaternion angles or any type of Euler angle methodology.

[0114] As the user's posture (e.g., body, head or eye posture) changes, the user's FOV may change correspondingly, and any objects within the FOV may change. FIG. 13A illustrates three FOVs 1350a, 1350b, 1350c, which are represented diagrammatically by rectangles and may correspond to portions of the user's FOR 1310 viewed at different time periods. In each of the three illustrated scenes 1350a, 1350b, 1350c, the user can perceive a reticle 1356 through the display 1320. In addition, as illustrated in the FOVs 1350b, 1350c, the user can perceive a target object, such as a cylinder 1382 or a box 1380, which may represent a virtual or physical object at a given location in the user's environment. As the user adjusts their head (e.g., pitch, yaw, or roll), the system may adjust the user's FOV so that the user perceives that they are tilting or turning their head within the VR / AR / MR environment.

[0115] As a non-limiting example, the user 1304 may wish to view or interact with object 1380 and / or object 1382. For simplicity, the initial pose 1348 of the user's head corresponds to a reference head pose vector, as described herein. In addition, in the initial head pose 1348, the user perceives FOV 1350a. Since no objects are perceivable within FOV 1350a, the user may begin scanning the FOR 1310 and search for objects 1380, 1382. As the user tilts his / her head downward toward the floor, the user's FOV correspondingly adjusts such that the user may eventually perceive object 1382 in FOV 1350b. The user may continue to scan the FOR 1310 and search for more objects, for example, by tilting his / her head upward toward the ceiling. At some later point in time, the user may adjust his / her head pose such that the user perceives FOV 1350c. As shown, an object 1380 is perceivable within the FOV 1350c. In some instances, in addition to simply perceiving the object, the user may select the object using the reticle 1356.

[0116] The reticle 1356 can be positioned on the target object using a variety of methods. As a first example, the user's FOV can be temporarily or permanently fixed within the FOR. Thus, changes in the user's pose (e.g., head pose, body pose, eye pose) can cause the reticle to move within the user's FOV (e.g., relative to a default reticle position that may be near the center of the FOV) but not change the user's FOV. For example, if the user looks up or tilts their head back, the movement can cause the reticle to move upward within the FOV, such as moving closer to the top of the FOV. Thus, in examples such as these, to select an object using the reticle, the object may need to be within the FOV and the user may need to position their head so that the reticle points to the target object.

[0117] As a second example, the position of the reticle within the FOV may be fixed, temporarily or permanently. Thus, changes in the user's posture may alter the user's FOV but do not change the location of the reticle within the FOV. For example, the reticle may be fixed or locked at a location (e.g., center) of the user's FOV as illustrated in FOV 1350a. As the user's posture changes, the FOV may change, but the reticle remains in a default location (e.g., center) of the dynamic FOV. Thus, to select an object using the reticle, the user may need to position their head so that the target object is in the center of the FOV (e.g., the location of the reticle).

[0118] As a third example, the reticle can be positioned on the target object using a combination of the first two examples. The reticle can be temporarily fixed to a position within the FOV. For example, the reticle can be fixed when the user's head pose is between a minimum head pose threshold and a maximum head pose threshold (e.g., meets the minimum head pose threshold and does not meet the maximum head pose threshold). As the user's pose changes, the user's FOV can change, but the position of the reticle within the FOV remains the same. However, as the user's head moves toward an uncomfortable or undesirable pose (e.g., the user's head pose does not meet the minimum head pose threshold or meets the maximum head pose threshold), the reticle can become unfixed and move freely within the user's FOV. For example, the wearable system can accelerate the movement of the reticle in a direction corresponding to the direction the user's head is moving. In some instances, this acceleration may reduce the possibility of neck strain or discomfort as the reticle moves toward the position the user would move their head to, thereby reducing or minimizing the amount of head movement required to position the reticle with the target object. Although the reticle is accelerated, in some embodiments, the reticle is not accelerated past a threshold position (e.g., a position within the FOV). For example, in some cases, the reticle is not accelerated out of the user's FOV. This advantageously aids in reducing the possibility of neck strain or discomfort while also preserving the user's ability to interact with objects within the FOV via the reticle.

[0119] From time to time, when the user desires to redirect their head (as a non-limiting example, to select an object that is high in the air (e.g., the ceiling), low on the ground (e.g., the floor), far to the right, far to the left, etc.), the user may need to bend, twist, or straighten their neck so that the reticle 1356 is positioned at the desired object. Bending, twisting, and / or straightening the user's neck may result in neck strain or discomfort, among other things. Thus, the wearable system may recognize that the orientation of the user's head is outside of a range of acceptable (e.g., comfortable, not causing strain, etc.) head poses (e.g., below a minimum head pose threshold, above a maximum head pose threshold). As a result, to assist the user in moving the reticle 1356, the wearable system may accelerate the movement of the reticle 1356 in a direction corresponding to the direction the user's head is moving (e.g., adjust the angle relative to a reference head pose vector or adjust the position within the FOV). By accelerating the movement of the reticle 1356, the wearable system advantageously reduces the extent to which the user must bend, twist, or extend their neck to align the reticle 1356 with the target object, thereby reducing the potential for neck strain or discomfort.

[0120] 13A , a reticle 1356 in the FOV 1350a is at a fixed location at the center 1360 of the FOV 1350a. As the user tilts their head toward the ceiling or floor, the wearable system may make a real-time determination of the user's head pose. If the user's head pose begins to correspond to any of the ranges of undesirable, uncomfortable, or strain-inducing head poses, the wearable system may begin to accelerate the reticle 1356.

[0121] For example, as illustrated in FIG. 13A, the user's real-time head pose may be compared to one or more threshold head poses (e.g., maximum head pitch threshold 1394, minimum head pitch threshold 1396) to determine whether the user's head pose corresponds to or falls within one or more of a range of desired head poses or one or more of a range of undesirable head poses. The wearable system 1320 may determine the user's head pose by calculating one or more angles corresponding to the degree to which the user bends, turns, tilts, or rotates their neck. For example, an angle difference between a vector corresponding to the user's current head pose and a reference head pose vector may be calculated. If the one or more angles correspond to angles associated with undesirable head poses, the wearable system may accelerate the movement of the reticle 1356, for example, by adding or removing an offset to the angle associated with the reticle and adjusting the reticle position within the FOV.

[0122] Examples of adding offsets are illustrated in FIG. 13A and FIG. 13B. As a first example, continuing with the example where the user's initial head pose corresponds to FOV 1350a, the user must tilt his head downward toward the ground to perceive FOV 1350b. When the user's head pose is at an angle such that the user perceives FOV 1350b, the system can determine an angle 1386 between the user's new head pose and a reference head pose (e.g., the user's initial head pose 1348). As illustrated in FIG. 13A, the user's new head pose, corresponding to the center 1360 of FOV 1350b, falls below (or does not meet) a minimum head pitch threshold 1396. Based on the user's new head pose, the system can determine that the user is looking down and / or that the user's head or neck is in or directed toward an uncomfortable or undesirable position. Thus, the system can accelerate the reticle 1356 in a downward direction such that the position is no longer at the default reticle position (e.g., the center) of the FOV. To make this adjustment to the reticle position, the system may add an offset 1364 to the angle 1386 at which the reticle is positioned relative to the reference head pose. The offset (e.g., about 12 degrees) can be added to the angle 1386 using various methods known in the art. For example, the offset 1364 can be added to or subtracted from the angle 1386 to make the new angle more negative. Similarly, the offset can increase or decrease the absolute value of the new angle.

[0123] As another example, continuing with the assumption that the user's initial head pose corresponds to FOV 1350a, the user must tilt their head upward toward the ceiling to perceive FOV 1350c. When the user's head pose is at an angle such that the user perceives FOV 1350c, the system can determine an angle 1390 between the user's new head pose and a reference head pose (e.g., the user's initial head pose 1348). As illustrated in FIG. 13A, the user's new head pose, which corresponds to the center 1360 of FOV 1350c, exceeds (or meets) a maximum head pitch threshold 1394. Based on the user's new head pose, the system can determine that the user is looking up and / or that the user's head or neck is in or directed toward an uncomfortable or undesirable position. Thus, the system can accelerate the reticle 1356 in an upward direction such that the position is no longer at a default reticle position for the FOV (e.g., the center of the FOV). To make this adjustment to the reticle position, the system may add an offset 1368 to the angle 1390 at which the reticle is positioned relative to the reference head pose. The offset (e.g., about 8 degrees) can be added to the angle 1390 using various methods known in the art. For example, the offset 1368 can be added to or subtracted from the angle 1390 to make the new angle more positive. Similarly, the offset can increase or decrease the absolute value of the new angle.

[0124] As another example, similar to FIG. 13A, FIG. 13B illustrates three FOVs 1350d, 1350e, 1350f, which may correspond to portions of the user's FOR 1310, represented diagrammatically by rectangles, observed at different time periods. In each of the three illustrated scenes 1350d, 1350e, 1350f, the user may perceive a reticle 1356 via the display 1320. Initially, the reticle 1356 is at a fixed location in the center 1360 of the FOV 1350d. As the user turns his head to the right, the wearable system 1320 may make a real-time determination of the user's head pose. As the user's head pose begins to move toward any of the ranges of undesirable, uncomfortable, or strain-inducing head poses, the wearable system may begin to accelerate the reticle 1356, thereby moving the reticle from its fixed location within the FOV.

[0125] As a non-limiting example, a user 1304 may wish to view or interact with an object 1384. For simplicity, the initial pose of the user's head 1348 corresponds to a reference head pose as described herein, and the initial FOV perceived by the user is FOV 1350d. As the user's head pose changes, the head pose can be compared to one or more threshold head poses (e.g., a maximum head yaw threshold 1374, an easing threshold 1372, a minimum head yaw threshold, etc.) to determine if the head pose reaches a threshold and the reticle position should be accelerated.

[0126] Returning to the example, the user may begin scanning the FOR 1310 and search for object 1384. As the user turns their head to the right, the user's FOV correspondingly adjusts so that the user may eventually perceive FOV 1350e. While perceiving FOV 1350e, the user's head pose (which may correspond to center 1360 of FOV 1350e) meets easing threshold 1372. Thus, the system may accelerate the position of reticle 1356 so that it is positioned slightly to the right of center 1360 of FOV 1350e. For example, the system may add offset 1376 (e.g., corresponding to an easing function) to angle 1392 corresponding to reticle 1356 within FOV 1350e.

[0127] The user may continue to scan the FOR 1310, for example, by turning their head further to the right, searching for object 1384. At some later point in time, the user may adjust their head pose so that the user perceives FOV 1350f. As shown, object 1384 is perceivable within FOV 1350f, and it may be advantageous for the system to, for example, accelerate the reticle toward object 1384 to make it easier for the user to select the object. Thus, the system may determine that the user's head pose (which may correspond to center 1360 of FOV 1350f) meets maximum head yaw threshold 1374. Thus, the system may accelerate the position of reticle 1356 such that it is positioned further to the right of the center of FOV 1350e than a reticle positioned within FOV 1350e. For example, the system may add a larger offset 1378 to angle 1388 corresponding to a reticle within FOV 1350f.

[0128] In some instances, to determine the amount of cursor acceleration in the horizontal direction, the system determines the pose of the user's head relative to the user's body, torso, shoulders, etc. For example, more neck strain may be more likely if the head pose vector is more offset from the body pose vector. Thus, the system may accelerate the reticle at a relatively faster or slower rate depending on the alignment of the user's head, neck, shoulders, body, etc.

[0129] In some cases, an additional inertial measurement unit (IMU) configured to track the posture of the user's body, torso, or shoulders can be added, and the system can determine the angle or position of the user's head relative to the user's body. The additional IMU can include ancillary devices such as pins, necklaces, backpacks, belt packs, totems, etc. on the patient's shoulders, chest, or waist. In some embodiments, the system can include external or user-facing sensors or cameras that can calculate body vectors, for example, using computer vision processing.

[0130] The example illustrated in FIGS. 13A and 13B corresponds to a user tilting their head up and down and turning their head to the right, but is not limiting; diagonal head movements (e.g., a combination of vertical and horizontal movements) can also be measured and the reticle accelerated to a corner of the FOV if the diagonal threshold (or a combination of vertical and horizontal thresholds) is exceeded. Additionally, the user can move their body, head, or eyes to perceive virtually any direction in space. In some embodiments, similar to adjusting the user's FOV based on changes in head pose, the system can adjust the position or location of the reticle based on changes in eye pose (e.g., eye gaze). Eye pose can be determined by an inward-facing imaging system 504 (shown in FIG. 4). The system can use one or more thresholds to determine whether the user's eyes may be strained and can modify the position of the reticle based on that determination. (Exemplary Head Posture Angle Adjustment)

[0131] FIG. 14 illustrates an example of adjusting the position of a reticle based on a user's head pose. A wearable system, such as the wearable system 200 of FIG. 2, can determine a user's head pose, for example, by determining a head pose vector that corresponds to the user's head pose. Based on the angular difference between the head pose vector and a reference vector, the system can adjust the position of the reticle relative to the user's FOV. As described herein, the wearable system can include one or more head pose sensors, such as, for example, an IMU that can be used to determine head pose, or an eye-tracking camera that can be used to determine eye gaze. The wearable system can use data from such sensors to determine the poses and angles described herein.

[0132] The head pose vector can provide an indication of the user's head pose. For example, the head pose vector can illustrate when the user is gazing, how the user's head is pointed, how the user's neck is bent, etc. In some embodiments, the head pose vector can be defined as a vector extending orthogonally from a coronal plane of the user's head (e.g., a frontal plane that divides the user's head into a ventral portion and a dorsal portion). For example, the head pose plane can be a plane that is parallel (or substantially parallel) to the user's forehead. In some embodiments, the head pose plane can be parallel to the coronal plane and can include a line connecting the user's eyes or other facial features. In some embodiments, the head pose vector can extend orthogonally from a coronal plane (or a plane parallel to the coronal plane) from a central point on the user's head, such as the center of the user's forehead, the center of the user's eyes, the user's nose, etc. In some embodiments, the head pose vector can extend from any other point that corresponds to the user's head or neck. Thus, as the user's head pose changes, the head pose vector also changes. As some examples, FIG. 14 illustrates various example head pose vectors 1408, 1412, 1416, 1420, 1424, 1428.

[0133] The reference vector may be part of a potential head pose vector and may be used as a reference for determining the angle at which the user's head is tilted or rotated. The reference vector may be a vector corresponding to any head pose. For example, considering coordinate values ​​in the xyz coordinate system shown in FIG. 14, the reference vector may correspond to a head pose vector having a zero vertical or y-component (sometimes referred to as a horizontal head pose vector 1416). In some cases, the reference vector is determined by identifying a vector in a horizontal plane perpendicular to the plane of the display. In some embodiments, the reference vector may be a head pose vector when the user's head is neutrally positioned (e.g., as a neutral head pitch vector 1420). In some embodiments, the horizontal head pose vector corresponds to the position of the user's head when each of the coronal plane of the user's head (e.g., a vertical plane that divides the body into abdominal and dorsal portions), the sagittal plane of the user's head (e.g., an anatomical plane that divides the body into right and left portions), and the axial plane of the user's head (e.g., a plane that divides the body into upper and lower portions that are approximately perpendicular to the spine) are orthogonal to each other. As illustrated in FIG. 14, in some instances, the angular difference between the neutral head pitch vector 1420 and the horizontal head pose vector 1416 is about -20 degrees (e.g., the neutral head pitch vector 1420 is about 20 degrees below the horizontal head pose vector 1416). In some applications, the user may look in a particular direction more frequently, and the system may adjust the baseline head pose to reduce or improve neck strain. For example, both a game related to shooting birds in flight and a star gazing application that teaches the user about constellations may involve the user periodically or constantly looking up. Similarly, a game of chess may involve the user periodically or constantly looking down. Other games or applications may involve the user periodically or constantly looking left, right, diagonally, etc. Thus, in some cases, the baseline head pose can be updated to hedge against the head pose associated with this more frequent direction.For example, the reference head pose can be the average or most common user's head pose over a particular period of time, or while the user uses a particular application or plays a particular game. By configuring the reference head pose to hedge against or match a head pose associated with the user's typical head pose, the system can reduce or ameliorate neck strain.

[0134] As described herein, the wearable system can identify a range of head positions or a range of head orientations, which together serve to define one or more ranges of head postures that may correspond to uncomfortable or strain-inducing head postures. The boundaries of such ranges may be considered to correspond to thresholds. For example, if a user tilts or turns their head too far in one direction, the user's head posture may not meet (e.g., be below) a minimum head pitch, head yaw, or head roll threshold. Similarly, if a user tilts or turns their head too far in another direction, the user's head posture may meet (e.g., be above) a maximum head pitch, head yaw, or head roll threshold. In these and other instances, head postures below (or not meeting) the minimum head posture threshold and head postures above (or meeting) the maximum head posture threshold may correspond to one or more ranges of head tilt or rotation that may cause neck strain or discomfort.

[0135] 14 illustrates three examples of adjusting the reticle to a position or angle relative to a reference head pose. In a first example, the user's head pose corresponds to a head pitch vector 1420, which in this example corresponds to a neutral head pose threshold. The neutral head pose threshold may correspond to a neutrally resting condition of the head as described herein and may represent a more comfortable head pose than, for example, the user's horizontal head pose 1416. Thus, in some instances, the neutral head pose vector may be used as the reference head pose vector instead of or in addition to the horizontal head pose vector.

[0136] Returning to the first example, an angular difference of -20 degrees exists between the head pitch vector 1420 and the reference head pose vector 1416, which in this case is also the horizontal head pose vector 1416. Based on the determined angular difference of -20 degrees, the system determines that the user's head pose is above (e.g., meets) a minimum pitch threshold of -45 degrees. In addition, the system determines that the angular difference of -20 degrees is below (e.g., does not meet) a maximum pitch threshold of 5 degrees. Based on these determinations, the system may decide not to adjust the position of the reticle. Thus, as seen in scene 1436, the reticle remains in a default location (e.g., center) within the scene 1436.

[0137] In a second example, the user's head pose corresponds to head pose vector 1412. As shown, an angular difference of +5 degrees exists between head pose vector 1412 and reference head pose vector 1416. Based on the determined angular difference of +5 degrees, the system determines that the user's pose equals (e.g., meets) a maximum pitch threshold of +5 degrees. Thus, the system adjusts the angle of the reticle by an offset amount, in this case 8 degrees. This adjustment can be seen in scene 1432, where the position of the reticle appears towards the top of scene 1432 rather than at a default position (e.g., the center) of scene 1432.

[0138] In a third example, the user's head pose corresponds to head pose vector 1424. As shown, an angular difference of -45 degrees exists between head pose vector 1424 and reference head pose vector 1416. Therefore, based on the determined angular difference of -45 degrees, the system determines that the user's pose is equal to (e.g., does not meet) the minimum head pitch threshold of -45 degrees. Therefore, the system adjusts the angle of the reticle by an offset amount, in this case -12 degrees. This adjustment can be seen in scene 1440, where the position of the reticle appears towards the bottom of scene 1440 rather than in its default position (e.g., the center) of scene 1440.

[0139] It should be noted that the example shown in FIG. 14 is merely illustrative and should not be construed as limiting. Thus, in some embodiments, a reference vector other than the horizontal head pose vector 1416 is used to determine the angle difference. Fewer, more, or different thresholds can be used by the system to determine if the head position reaches a threshold and adjust the position of the reticle. Additionally, the values ​​corresponding to the maximum head pitch threshold 1412, the minimum head pitch threshold 1424, the neutral head pitch vector 1420, the offset amount corresponding to the maximum head pitch threshold 1412, and / or the offset amount corresponding to the minimum head pitch threshold 1424 are for illustrative purposes. In other implementations, the maximum head pitch threshold 1412 can be in the range of 10 degrees below the horizontal head pose vector 1416 to 25 degrees above the horizontal head pose vector 1416. The minimum head pitch threshold 1424 can be in the range of 60 degrees below the horizontal head pose vector 1416 to 30 degrees below the horizontal head pose vector 1416. The neutral head pitch vector 1420 can range from 30 degrees below the horizontal head pose vector 1416 to a value that is level with the horizontal head pose vector 1416. The offset amount corresponding to the maximum pitch threshold 1412 can be in the range of +1 degree to +25 degrees. The offset amount corresponding to the minimum pitch threshold 1424 can be in the range of -1 degree to -25 degrees. Additionally, one or more of the thresholds may be met in a variety of ways. For example, in some cases, a value corresponding to a threshold will meet the threshold while in other cases, a value corresponding to a threshold will not meet the threshold. (Example Head Pose Angle Adjustment to Reticle Position)

[0140] 15 demonstrates an example relationship between a user's head pose and adjustments to the position of a reticle. Graph 1500 illustrates an example reticle angle adjustment versus angle difference. As described herein, the angle difference (e.g., x-axis) can be defined as the difference in angle between a head pose vector and a reference vector (e.g., horizontal head pose vector, neutral head pose vector, etc.). Additionally, adjustments to the reticle angle (e.g., y-axis) can correspond to how the system changes the position of the reticle based on the determined head pose.

[0141] Graph 1500 illustrates several different possible relationships between adjustments to the reticle angle and the angular difference (e.g., lines 1560, 1564, 1568). For example, lines 1560, 1568 illustrate incremental adjustments to the reticle angle as the angular difference changes, while line 1564 illustrates no adjustment to the reticle angle within a defined range (e.g., from -30 degrees to -10 degrees), and then incremental adjustments to the reticle angle as the angular difference falls outside the defined range (e.g., below the minimum head pose threshold and above the maximum head pose threshold).

[0142] As illustrated by each of relationships 1564, 1560, and 1568, if the angle difference is 5 degrees or higher (e.g., the user is looking up at about 5 degrees or more at the horizontal head pose vector), the system determines that the user's head pose is greater than or equal to (e.g., meets) the maximum head pitch threshold (e.g., +5 degrees). Thus, the system adjusts the angle of the reticle by a maximum offset amount of 8 degrees. Additionally, if the angle difference is less than or equal to -45 degrees (e.g., the user is looking down at about 45 degrees or more at the horizontal head pose vector), the system determines that the user's head pose is less than or equal to (e.g., does not meet) the minimum head pitch threshold (e.g., -45 degrees). Thus, the system adjusts the angle of the reticle by a minimum offset amount of -12 degrees.

[0143] Furthermore, if the angular difference (e.g., between the minimum head pose threshold and the maximum head pose threshold) is between about -45 degrees and about 5 degrees, the system determines that the user's head pose is above (e.g., meets) the minimum head pitch threshold and below (e.g., does not meet) the maximum head pitch threshold. Thus, the system adjusts the reticle angle by an offset amount that may be determined, for example, by various linear, exponential, piecewise, or easing functions. For example, the easing function may provide a gradually increasing or decreasing reticle angle adjustment. (Example process for positioning a reticle) (Reticle positioning or adjustment based on head pose)

[0144] 16 illustrates a flow chart for an example reticle position adjustment process 1600. The example process 1600 may be performed, alone or in combination, by one or more components of the wearable system 200, such as, for example, the remote processing module 270 or the local processing and data module 260. The display 220 of the wearable system 200 may present a reticle to the user, the inward-facing imaging system 462 may acquire eye images for eye gaze determination, and an IMU, accelerometer, or gyroscope may determine head pose.

[0145] In block 1610, the wearable system may receive data indicating the user's current head pose. The head pose may describe the position and orientation of the user's head. The data may include the current position and orientation of the user's head or the movement of the user's head in 3D space. The position may be represented by translation coordinate values ​​(e.g., coordinate values ​​in the xyz coordinate system shown in FIG. 6). For example, as the user's head tilts or turns, the wearable system may track and record the user's head movement. The orientation may be represented by vector or angle values ​​relative to a neutrally placed state of the head. For example, the vector or angle values ​​may represent head tilt in forward and backward (e.g., pitch), rotation to the left and right (e.g., yaw), and tilt in the lateral direction (e.g., roll).

[0146] The wearable system may identify a range of head positions or ranges of head orientations, which together serve to define one or more ranges of head postures that may correspond to uncomfortable or straining head postures. The boundaries of such ranges may be considered to correspond to thresholds. For example, a maximum or minimum head pitch, head yaw, or head roll threshold may correspond to one or more ranges of head tilt or rotation to the left that may cause neck strain or discomfort. Similarly, a neutral or reference head posture threshold may correspond to a neutrally placed state of the head.

[0147] Head poses that fall into these ranges may correspond to head poses for which an adjustment to the position of the reticle may be desired. For example, the wearable system may render a reticle in 3D space for the user. The reticle may be rendered in rig space (which may be represented by a coordinate system relative to the HMD). The reticle may be represented in a variety of graphical forms, which may include 1D, 2D, and 3D images. The reticle may correspond to the user's current position relative to the user's field of view, and may represent, for example, the user's direction of gaze. As the user moves around, the reticle may also move with the user. As the user's head pose changes, for example, to fall into one of the aforementioned ranges, the position of the reticle may be adjusted to prevent, relieve, or reduce neck strain or discomfort.

[0148] A wearable system can track head pose using one or more sensors either inside the HMD (e.g., to track a reflected image of the user's head), such as an IMU or an outward-facing imaging system, or outside the HMD (e.g., a camera mounted on a wall inside the user's room).

[0149] In block 1620, the wearable system may identify or determine a real-time head pose of the user based at least in part on the head pose data obtained from block 1610. For example, the wearable system may identify or determine a head pose vector corresponding to the user's head pose as described herein. In some cases, the head pose vector is determined using an AR software development kit (SDK).

[0150] In block 1630, the wearable system may identify, access, or determine a reference head pose for the user. For example, the reference head pose may correspond to a neutral head pose and may be based at least in part on the head pose data obtained from block 1610. The wearable system may determine a reference head pose vector as described herein, which may correspond to a neutral head pose, such as a head pose for a head neutrally placed state. In some cases, the reference head pose vector is determined using the AR SDK. In some implementations, the user's reference or neutral head pose is set to a default value (e.g., angle -20 degrees), and the wearable system determines the reference head pose by accessing the default value (e.g., by querying the AR SDK).

[0151] In block 1640, the wearable system may determine an adjustment for a position of a virtual reticle projected on the head mounted display based at least in part on a comparison between the current head pose and the reference head pose. For example, the system may determine a value indicative of a difference between the head pose determined from block 1620 and the reference head pose determined from block 1630. For example, an angular difference between the head pose and the reference head pose may be calculated relative to a coordinate value (e.g., a coordinate value in the xyz coordinate system shown in FIG. 6 ). In some embodiments, the wearable system may determine an angular difference between a head pose vector and a reference head pose vector. For example, as shown in FIG. 14 , the angular difference may be used to determine an angle corresponding to a degree of tilt or rotation of the user's head relative to the reference head pose. Based at least in part on the angular difference, the wearable system may determine whether the user is looking up or down, or tilting or turning their head otherwise. In some cases, the angular difference is determined using the AR SDK.

[0152] The wearable system can determine the adjusted reticle position based at least in part on the user's head pose. For example, the wearable system can determine whether the user has a head pose that falls within one or more of the identified ranges of head poses. The wearable system can determine whether the user's head pose is in a position or orientation that may result in an adjustment of the position of the reticle. As an example, the wearable system can determine whether the user's head pose falls within an identified range of head positions and an identified range of head orientations. The wearable system can make such a determination by comparing the user's head pose to thresholds (e.g., translation coordinate values) that define the boundaries of the identified ranges of head positions, or by comparing a head orientation (e.g., an angular difference) associated with the user's head pose to thresholds (e.g., angle values) that define the boundaries of the identified ranges of head orientations.

[0153] 14, based on determining that the user is in a head pose corresponding to head pose vector 1412, the wearable system can determine that the angular difference (e.g., difference between the reference head pose vector) is 5 degrees. Thus, by comparing the angular difference to a threshold (e.g., a neutral head pose threshold corresponding to vector 1416), the wearable system can determine whether an adjustment of the position of the reticle is desired. For example, returning to the example of FIG. 14, the wearable system may determine that the 5 degree angular difference corresponding to head pose vector 1412 satisfies the maximum head pose threshold. The wearable system may then adjust the position of the reticle such that the head pose vector 1408 pointing to the newly adjusted reticle has an angular difference of 8 degrees. Thus, the wearable system can accelerate or decelerate reticle movement based at least in part on the head pose or a change in head pose.

[0154] The various blocks described herein can be implemented in various orders, and the wearable system can implement one or more of the blocks in parallel and / or vary the order, as desired. Fewer, more, or different blocks can be used as part of process 1600. For example, process 1600 can include blocks for determining a position of a reticle, providing an indication that a position of a reticle has been updated, etc.

[0155] Additionally, although the process 1600 is logically associated with preventing or reducing the possibility of neck strain, similar techniques can also be utilized to prevent or reduce the possibility of eye fatigue. For example, the system can obtain eye gaze data and determine an eye gaze vector and a neutral eye gaze vector from the eye gaze data. The system can further determine an angle difference between the eye gaze vector and the neutral eye gaze vector and determine an adjustment for at least one of the position of the virtual reticle or the 3D view of the display based on the angle difference and one or more thresholds. (Reticle positioning or adjustment based on angle difference)

[0156] 17 illustrates a flow chart for an example reticle adjustment process 1700. The example process 1700 may be performed, alone or in combination, by one or more components of the wearable system 200, such as, for example, the remote processing module 270 or the local processing and data module 260. The display 220 of the wearable system 200 may present a reticle to the user, the inward-facing imaging system 462 may acquire eye images for eye gaze determination, and an IMU, accelerometer, or gyroscope may determine head pose.

[0157] The wearable system can render a reticle in 3D space for the user. The reticle can be rendered in rig space (which can be represented by a coordinate system relative to the HMD). The reticle can be represented in various graphical forms, which can include 1D, 2D, and 3D images. The reticle can correspond to the user's current position relative to the user's field of view, and can represent, for example, the user's gaze direction. As the user moves around, the reticle can move with the user. In addition, the reticle can point to one or more objects, and the user can select the target object to which the reticle is pointing. In some instances, the location of the reticle in the user's FOV can remain constant as the user's head moves. For example, the reticle can be positioned in the center of the user's field of view and will remain centered even as the user's FOV changes. Thus, in some cases, to select a target object with the reticle, the user must move their head and adjust the FOV so that the center of the FOV (e.g., the location of the reticle) is over the target object.

[0158] However, it may be advantageous for the wearable system to adjust the position of the reticle within the user's FOV. For example, in an instance where the user desires to select a target object that is high in the air (e.g., the ceiling), low on the ground (e.g., the floor), far right, or far left, the wearable system may adjust the position of the reticle within the user's FOV to help the user position the reticle at the location of the target object, but without requiring the user to tilt or turn their head so that the center of the user's FOV is on the target object. Thus, the wearable system may implement process 1600 to adjust the position of the reticle.

[0159] In block 1710, the wearable system may identify or determine one or more ranges of head positions or head orientations that together serve to define one or more ranges of head poses that may correspond to uncomfortable or straining head poses. Boundaries of such ranges may be considered to correspond to thresholds. For example, the wearable system may identify or determine a maximum head pitch threshold and / or a minimum head pitch threshold that may correspond to a degree or angle of head tilt in the forward or backward direction that will result in a maximum or minimum reticle angle adjustment. In some embodiments, the wearable system may identify or determine a maximum head yaw threshold or a minimum head yaw threshold that may correspond to a degree or angle of left or right head rotation before the user's neck is in an uncomfortable or straining position. Similarly, the wearable system may identify or determine a maximum head roll threshold or a minimum head roll threshold that may correspond to a degree or angle of lateral head tilt before the user's neck is in an uncomfortable or straining position. Additionally, the wearable system may identify or determine a neutral head pitch threshold, a neutral head yaw threshold, or a neutral head roll threshold, which may correspond to a neutrally resting state of the head. In some cases, these thresholds may be user input, may be determined during a calibration phase, or may be constants in the system.

[0160] In blocks 1720 and 1730, the wearable system may determine a head pose vector and a reference (or neutral) head pose vector based at least in part on the received head pose data as described herein.

[0161] In block 1740, the wearable system can determine an angle difference between the head pose vector and the neutral head pose vector. For example, as shown in Figure 14, the angle difference can be used to determine an angle corresponding to the degree of tilt or rotation of the user's head relative to a reference head pose. Based at least in part on the angle difference, the wearable system can determine whether the user is looking up or down, or tilting or turning their head.

[0162] In block 1750, the wearable system may calculate a reticle position adjustment. A reticle position adjustment (e.g., an angle adjustment or offset) is associated with each of the thresholds and may serve to, for example, accelerate cursor movement. For example, if the head pose meets a maximum head pitch threshold, a maximum head yaw threshold, or a maximum head roll threshold, the wearable system may adjust the reticle angle or position by a first predetermined amount (e.g., +8 degrees). Similarly, if the head pose does not meet a minimum head pitch threshold, a minimum head yaw threshold, or a minimum head roll threshold, the wearable system may adjust the reticle angle or position by a second predetermined amount (e.g., -12 degrees). In some embodiments, if the head pose meets a minimum threshold (e.g., pitch, yaw, or roll) but does not meet a maximum threshold (e.g., pitch, yaw, or roll), the wearable system may adjust the reticle angle or position using an easing function. For example, if the head pose meets the minimum head pitch threshold but does not meet the neutral head pitch threshold, the wearable system may adjust the angle or position of the reticle using one of the following:

number

[0163] Similarly, if the head pose meets the neutral head pitch threshold but does not meet the maximum head pitch threshold, the wearable system may adjust the reticle angle or position using the following two:

number

[0164] The various blocks described herein can be implemented in various orders, and the wearable system can implement one or more of the blocks in parallel and / or vary the order, as desired. Fewer, more, or different blocks can be used as part of process 1700. For example, process 1700 can include blocks for determining a position of a reticle, providing an indication that a position of a reticle has been updated, etc.

[0165] Additionally, although the process 1700 is logically associated with preventing or reducing the possibility of neck strain, similar techniques can also be utilized to prevent or reduce the possibility of eye fatigue. For example, the system can obtain eye gaze data and determine an eye gaze vector and a neutral eye gaze vector from the eye gaze data. The system can further determine an angle difference between the eye gaze vector and the neutral eye gaze vector and determine an adjustment for at least one of the position of the virtual reticle or the 3D view of the display based on the angle difference and one or more thresholds. (Reticle positioning or adjustment based on head pitch threshold)

[0166] 18 illustrates a flow chart for an exemplary reticle adjustment process. The exemplary process 1800 may be performed, alone or in combination, by one or more components of the wearable system 200, such as, for example, the remote processing module 270 or the local processing and data module 260. The display 220 of the wearable system 200 may present a reticle to the user, the inward-facing imaging system 462 may acquire eye images for eye gaze determination, and an IMU, accelerometer, or gyroscope may determine head pose.

[0167] In block 1810, similar to block 1740 in FIG. 17, the wearable system calculates the angle difference between the head pose vector and the neutral head pose vector.

[0168] In block 1820, the wearable system determines whether the angle difference meets or does not meet the minimum head pitch threshold. As described herein, the head pitch may correspond to tilting the head forward or backward. Thus, the angle difference may not meet the minimum head pitch threshold when the user strains their neck forward or bends (e.g., to look at the ground). For example, the minimum head pitch threshold may correspond to an angle difference of about -30, -35, -40, -45, -50, 55, or -60 degrees (+ / - a few degrees). Thus, in some cases, if the angle difference is at or below the minimum head pitch threshold, the angle difference does not meet the minimum head pitch threshold. However, in some cases, if the angle difference is at or below the minimum head pitch threshold, the angle difference meets the minimum head pitch threshold.

[0169] It should be noted that if the angle between the head pose vector and the neutral head pose vector (e.g., head pose vector - neutral head pose vector) is positive, then the user is looking up relative to the neutral position. Similarly, if the angle is negative, then the user is looking down. Thus, in some cases, the system can determine whether the user is looking up or down and take the absolute value of the angle difference to ensure a positive angle. It will be appreciated that in some embodiments, the system can determine the angle difference (e.g., head pose vector - neutral head pose vector), with a negative value indicating that the user is looking down.

[0170] In block 1830, the wearable system determines that the angular difference does not meet the minimum head pitch threshold. Thus, the wearable system determines an adjusted reticle position using an angular adjustment associated with the minimum head pitch threshold. In some cases, the angular adjustment can be approximately -30, -25, -20, -15, -12, -10, -5, or -2 degrees.

[0171] In block 1840, the wearable system determines whether the angle difference meets or does not meet the maximum head pitch threshold. As described herein, the head pitch may correspond to tilting the head forward or backward. Thus, the angle difference may meet the maximum head pitch threshold when the user is straining their neck backward or bending (e.g., to look at the sky). For example, the maximum head pitch threshold may correspond to an angle difference of about 2, 5, 8, 10, 12, 15, 20, 25, or 30 degrees (+ / - a few degrees). Thus, in some cases, if the angle difference is at or above the maximum head pitch threshold, the angle difference meets the maximum head pitch threshold. However, in some cases, if the angle difference is at or above the maximum head pitch threshold, the angle difference does not meet the maximum head pitch threshold.

[0172] In block 1850, the wearable system determines that the angular difference meets the maximum head pitch threshold. Thus, the wearable system determines an adjusted reticle position using the angular adjustment associated with the maximum head pitch threshold. In some cases, the angular adjustment can be approximately 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 15, or 20 degrees.

[0173] In block 1860, the wearable system determines whether the angle difference meets or does not meet the neutral head pitch threshold. As described herein, the head pitch may correspond to tilting the head forward or backward. The neutral head pitch threshold may correspond to a neutrally placed state of the head. For example, the neutral head pitch threshold may correspond to an angle difference of about -10, -15, -20, -25, or -30 degrees (+ / - a few degrees). Thus, in some cases, if the angle difference is at or above the neutral head pitch threshold, the angle difference meets the neutral head pitch threshold. Similarly, if the angle difference is below the neutral head pitch threshold, the angle difference may not meet the neutral head pitch threshold. However, in some cases, the angle difference may not meet the neutral head pitch threshold if it is at or above the neutral head pitch threshold, and may meet the neutral head pitch threshold if it is below the head pitch threshold.

[0174] In block 1870, the wearable system determines that the angular difference meets the neutral head pitch threshold. Thus, the wearable system may adjust the angle or position of the reticle using Equation 2 above.

[0175] In block 1880, the wearable system determines that the angular difference does not meet the neutral head pitch threshold. Thus, the wearable system may adjust the angle or position of the reticle using Equation 1 above.

[0176] The various blocks described herein can be implemented in various orders, and the wearable system can implement one or more of the blocks in parallel and / or vary the order, as desired. Fewer, more, or different blocks can be used as part of process 1800. For example, process 1800 can include blocks for determining additional thresholds, providing an indication that the position of the reticle has been updated, etc.

[0177] Additionally, although the process 1800 is logically associated with preventing or reducing the possibility of neck strain, similar techniques can also be utilized to prevent or reduce the possibility of eye fatigue. For example, the system can obtain eye gaze data, and from the eye gaze data, can determine an eye gaze vector and a neutral eye gaze vector. The system can further determine an angle difference between the eye gaze vector and the neutral eye gaze vector, and can determine an adjustment for at least one of the position of the virtual reticle or the 3D view of the display based on the angle difference and one or more thresholds. (Example Software Code)

[0178] Appendix A includes an example of code in the C# programming language that calculates head pose and vertical pitch adjustments. The code implements an example of a reticle adjustment process 1700 described with reference to Figure 17. Appendix A is incorporated by reference in its entirety into this specification as if it formed a part of this specification. (Additional Aspects)

[0179] In a first aspect, a system includes a head pose sensor configured to acquire head pose data of a user of the system, a non-transitory memory configured to store the head pose data, a display configured to be positioned in front of an eye of the user, the display configured to project a virtual reticle towards the eye of the user, and a hardware processor in communication with the head pose sensor, the display, and the non-transitory memory, the hardware processor: The system is programmed to: acquire head pose data of a user; identify a head pose of the user based at least in part on the head pose data; determine an adjustment for a position of a virtual reticle based at least in part on a comparison between the user's head pose and a reference head pose; and move the virtual reticle to a fixed position based at least in part on the determined adjustment.

[0180] In a second aspect, the system of aspect 1, wherein the virtual reticle comprises a movable indicator that identifies the user's position within the user's field of view.

[0181] In a third aspect, the system of aspect 1 or aspect 2, wherein the head pose data corresponds to at least one of an indication of yaw, pitch, or roll of the user's head.

[0182] In a fourth aspect, the system of aspect 3, wherein the yaw, pitch, or roll command is with respect to a reference head pose.

[0183] In a fifth aspect, the system of aspect 4, wherein the reference head pose corresponds to a horizontal head pose of the user's head.

[0184] In a sixth aspect, the system of aspect 5, wherein the horizontal head pose comprises a head pose in which a coronal plane of the user's head, a sagittal plane of the user's head, and an axial plane of the user's head are each perpendicular to one another.

[0185] In a seventh aspect, the system of any one of aspects 1-6, wherein the reference head pose comprises a head pose corresponding to a neutral position of the user's head.

[0186] In an eighth aspect, the system of aspect 7, wherein a neutral position of the user's head corresponds to a yaw of -5 to 5 degrees, a roll of -5 to 5 degrees, and a pitch of -15 to -25 degrees relative to a horizontal head posture.

[0187] In a ninth aspect, the system of aspect 8, wherein the reference head pose corresponds to at least one of 0 degrees yaw, 0 degrees roll, or -20 degrees pitch relative to a horizontal head pose.

[0188] In a tenth aspect, the system described in any one of aspects 1-9, wherein the hardware processor is further programmed to identify a head pose vector corresponding to the user's head pose and to identify a reference head pose vector corresponding to a reference head pose.

[0189] In an eleventh aspect, the system described in aspect 10, wherein the hardware processor is further programmed to determine an angular difference between the head pose vector and the reference head pose vector based at least in part on a comparison between the user's head pose and the reference head pose, the angular difference corresponding to at least one of a difference in yaw, pitch, or roll of the user's head pose relative to the reference head pose.

[0190] In a twelfth aspect, the system described in aspect 11, wherein the hardware processor is programmed to compare the determined angular difference with one or more head pose thresholds to determine an adjustment for the position of the virtual reticle.

[0191] In a thirteenth aspect, the system of aspect 12, wherein the one or more head pose thresholds comprise at least one of a maximum head pose threshold or a minimum head pose threshold.

[0192] In a fourteenth aspect, the system of aspect 13, wherein the maximum head pose threshold corresponds to at least one of a maximum head yaw threshold, a maximum head roll threshold, or a maximum head pitch threshold.

[0193] In a fifteenth aspect, the system of aspect 14, wherein the maximum head yaw threshold is 50 degrees, the maximum head roll threshold is 20 degrees, or the maximum head pitch threshold is 5 degrees, relative to the reference head pose.

[0194] In a sixteenth aspect, the system of any one of aspects 13-15, wherein the minimum head pose threshold corresponds to at least one of a minimum head yaw threshold, a minimum head roll threshold, or a minimum head pitch threshold.

[0195] In a seventeenth aspect, the system of aspect 16, wherein the minimum head yaw threshold is -50 degrees, the minimum head roll threshold is -20 degrees, or the minimum head pitch threshold is -45 degrees, relative to the reference head pose.

[0196] In an 18th aspect, the system of any one of aspects 13-17, wherein the hardware processor is further programmed to determine an adjustment for the position of the virtual reticle based at least in part on the first adjustment value in response to a determination that the angular difference does not meet a minimum head pose threshold.

[0197] In a nineteenth aspect, the system of aspect 18, wherein the first adjustment value is approximately -12 degrees.

[0198] In a twentieth aspect, the system of any one of aspects 13-19, wherein the hardware processor is further programmed to determine an adjustment for the position of the virtual reticle based at least in part on the second adjustment value in response to determining that the angular difference satisfies a maximum head pose threshold.

[0199] In a twenty-first aspect, the system of aspect 20, wherein the second adjustment value is approximately +5 degrees.

[0200] In a 22nd aspect, the system of any one of aspects 13-21, wherein the hardware processor is further programmed to determine an adjustment for the position of the virtual reticle based at least in part on the third adjustment value in response to a determination that the angular difference meets the minimum head pose threshold and does not meet the maximum head pose threshold.

[0201] In a twenty-third aspect, the system of aspect 22, wherein the third adjustment value corresponds to an easing function.

[0202] In a twenty-fourth aspect, the system of aspect 22 or aspect 23, wherein the third adjustment value is approximately 0 degrees.

[0203] In a twenty-fifth aspect, the system of any one of aspects 1-24, wherein altering the virtual reticle to a fixed position includes altering the virtual reticle to a position from a default reticle position in the user's field of view.

[0204] In a twenty-sixth aspect, the system of aspect 25, wherein the default reticle position comprises the center of the user's field of view.

[0205] In a twenty-seventh aspect, the system of any one of aspects 1-26, wherein the head posture sensor comprises an inertial measurement unit (IMU), an accelerometer, a gyroscope, or a magnetometer.

[0206] In a twenty-eighth aspect, the wearable system comprises a head-mounted wearable system, the system being described in any one of aspects 1-27.

[0207] In a twenty-ninth aspect, a method for adjusting a position of a virtual reticle that identifies a position of a user within a field of view corresponding to a display of a display system, the method including: acquiring head pose data of a user of the display system from a head pose sensor configured to track the user's head pose; identifying a head pose vector corresponding to the user's head pose based at least in part on the head pose data; identifying a reference head pose vector corresponding to a reference head pose; determining an angular difference between the head pose vector and the reference head pose vector based at least in part on a comparison between the user's head pose and the reference head pose, the angular difference corresponding to at least one of a difference in yaw, pitch, or roll of the user's head pose relative to the reference head pose; and comparing the determined angular difference to one or more head pose thresholds, the one or more head pose thresholds being determined based on the comparison. the threshold comprises at least one of a maximum head pose threshold or a minimum head pose threshold; in response to determining that the angular difference does not satisfy the minimum head pose threshold, determining an adjustment for a position of a virtual reticle based at least in part on a first adjustment value, where the position of the virtual reticle corresponds to a position of a movable indicator identifying a position of a user within a field of view of the user; in response to determining that the angular difference satisfies the maximum head pose threshold, determining an adjustment for the position of the virtual reticle based at least in part on a second adjustment value; in response to determining that the angular difference satisfies the minimum head pose threshold and does not satisfy the maximum head pose threshold, determining an adjustment for the position of the virtual reticle based at least in part on a third adjustment value; and causing the position of the virtual reticle to be adjusted from a default reticle position in the field of view of the user based on the determined adjustment. The method can be implemented under control of a display system, for example by a hardware processor programmed to perform operations of the method.

[0208] In a 30th aspect, the method of aspect 29, wherein the virtual reticle comprises a movable indicator that identifies the user's position within the user's field of view.

[0209] In a thirty-first aspect, the method of aspect 29 or aspect 30, wherein the head pose data corresponds to at least one of an indication of yaw, pitch, or roll of the user's head.

[0210] In a thirty-second aspect, the method of aspect 31, wherein the instruction of yaw, pitch, or roll of the user's head is with respect to a reference head pose.

[0211] In a thirty-third aspect, the method of any one of aspects 29-32, wherein the reference head pose of the user corresponds to a horizontal head pose of the user's head.

[0212] In a thirty-fourth aspect, the method of aspect 33, wherein the horizontal head pose comprises a head pose in which a coronal plane of the user's head, a sagittal plane of the user's head, and an axial plane of the user's head are each perpendicular to one another.

[0213] In a thirty-fifth aspect, the method of any one of aspects 29-34, wherein the reference head pose comprises a head pose corresponding to the user's head in a neutral position.

[0214] In a thirty-sixth aspect, the method described in aspect 35, wherein the neutral position of the user's head corresponds to a yaw of approximately -5 to 5 degrees, a roll of approximately -5 to 5 degrees, or a pitch of approximately -15 to -25 degrees relative to the user's head horizontal.

[0215] In a thirty-seventh aspect, the method of aspect 35 or aspect 36, wherein the reference head pose corresponds to at least one of 0 degrees yaw, 0 degrees roll, or -20 degrees pitch relative to a horizontal head pose.

[0216] In a thirty-eighth aspect, the method of any one of aspects 29-37, wherein the maximum head pose threshold corresponds to at least one of a maximum head yaw threshold, a maximum head roll threshold, or a maximum head pitch threshold.

[0217] In a thirty-ninth aspect, the method of any one of aspects 29-38, wherein, relative to the reference head pose, the maximum head yaw threshold is approximately 50 degrees, the maximum head roll threshold is approximately 20 degrees, and the maximum head pitch threshold is approximately 5 degrees.

[0218] In a 40th aspect, the method of any one of aspects 29-39, wherein the minimum head pose threshold corresponds to at least one of a minimum head yaw threshold, a minimum head roll threshold, or a minimum head pitch threshold.

[0219] In a 41st aspect, the method of any one of aspects 29-40, wherein the minimum head yaw threshold is approximately -50 degrees, the minimum head roll threshold is approximately -20 degrees, and the minimum head pitch threshold is approximately -45 degrees, relative to the reference head pose.

[0220] In a 42nd aspect, the method of any one of aspects 29-41, wherein the first adjustment value is approximately -12 degrees.

[0221] In a 43rd aspect, the method of any one of aspects 29-42, wherein the second adjustment value is about +5 degrees.

[0222] In a 44th aspect, the method of any one of aspects 29-43, wherein the third adjustment value corresponds to an easing function.

[0223] In a 45th aspect, the method of any one of aspects 29-44, wherein the third adjustment value is about 0 degrees.

[0224] In a 46th aspect, the method of any one of aspects 29-45, wherein the default reticle position comprises the center of the user's field of view.

[0225] In a 47th aspect, the method of any one of aspects 29-46, wherein the head posture sensor comprises an inertial measurement unit (IMU), an accelerometer, a gyroscope, or a magnetometer.

[0226] In a forty-eighth aspect, a method for adjusting a position of a virtual reticle that identifies a position of a user within a field of view corresponding to a display of a display system, the method including acquiring head pose data of a user of the display system, identifying a head pose based at least in part on the head pose data, identifying a reference head pose, and determining an adjustment for a position of a virtual reticle projected on a head mounted display based at least in part on a comparison between the head pose and the reference head pose.

[0227] In a forty-ninth aspect, the method of aspect 48 further comprises obtaining head pose data of a user of the display system from a head pose sensor configured to track the user's head pose.

[0228] In a 50th aspect, the method of aspect 48 or aspect 49 further comprises identifying a head pose vector corresponding to the user's head pose and identifying a reference head pose vector corresponding to a reference head pose.

[0229] In a 51st aspect, the method of aspect 50 further includes determining an angular difference between the head pose vector and the reference head pose vector based at least in part on a comparison between the head pose and a reference head pose, the angular difference corresponding to at least one of a difference in yaw, pitch, or roll of the user's head pose relative to the reference head pose.

[0230] In a 52nd aspect, the method of any one of aspects 48-51, wherein determining an adjustment for the position of the virtual reticle is further based on a comparison of the angular difference to one or more head pose thresholds.

[0231] In a 53rd aspect, the method of aspect 52, wherein the one or more head pose thresholds comprise at least one of a maximum head pose threshold or a minimum head pose threshold.

[0232] In a fifty-fourth aspect, a method for adjusting a position of a movable indicator identifying a user's position within a user's field of view relative to a display of a display system, the method including identifying at least one of a maximum head pitch threshold, a minimum head pitch threshold, or a neutral head pitch threshold, identifying a head pose vector corresponding to the user's head pose, identifying a reference head pose vector, calculating an angular difference between the head pose vector and the reference head pose vector, calculating a reticle adjustment based at least in part on the angular difference and at least one of the maximum head pitch threshold, the minimum head pitch threshold, or the neutral head pitch threshold, and determining an adjusted reticle position based at least in part on the calculated reticle adjustment.

[0233] In a fifty-fifth aspect, the method of aspect 54 further includes obtaining head pose data of a user of the display system from a head pose sensor configured to track the user's head pose.

[0234] In a 56th aspect, the method of aspect 54 or aspect 55, wherein determining the angular difference is based at least in part on a comparison between the head pose vector and a reference head pose vector, and the angular difference corresponds to at least one of a difference in yaw, pitch, or roll of the user's head pose relative to the reference head pose.

[0235] In a 57th aspect, the method of any one of aspects 54-56, wherein determining the adjusted reticle position is further based on comparing the angular difference to one or more head pose thresholds.

[0236] In a fifty-eighth aspect, the method of aspect 57, wherein the one or more head pose thresholds comprise at least one of a maximum head pose threshold or a minimum head pose threshold.

[0237] In a fifty-ninth aspect, a method for adjusting a position of a virtual reticle that identifies a user's position within a field of view corresponding to a display of a display system, the method including calculating an angular difference between a head pose vector and a reference head pose vector, where the head pose vector corresponds to a head pose of a user of the display system and the reference head pose vector corresponds to a reference head pose, determining that the angular difference does not satisfy a minimum head pitch threshold, and determining an adjusted reticle position based at least in part on the angular adjustment associated with the minimum head pitch threshold.

[0238] In a 60th aspect, the method of aspect 59 further includes obtaining head pose data of a user of the display system from a head pose sensor configured to track the user's head pose, identifying a head pose vector corresponding to the user's head pose based at least in part on the head pose data, and identifying a reference head pose vector corresponding to a reference head pose.

[0239] In a 61st aspect, the method of aspect 59 or aspect 60, wherein calculating the angular difference is based at least in part on a comparison between the head pose vector and a reference head pose vector, and the angular difference corresponds to at least one of a difference in yaw, pitch, or roll of the user's head pose relative to the reference head pose.

[0240] In a 62nd aspect, a method for adjusting a position of a virtual reticle that identifies a user's position within a field of view corresponding to a display of a display system, the method including calculating an angular difference between a head pose vector and a reference head pose vector, where the head pose vector corresponds to a head pose of a user of the display system and the reference head pose vector corresponds to the reference head pose, determining that the angular difference satisfies a minimum head pitch threshold, determining that the angular difference does not satisfy a maximum head pitch threshold, and determining an adjusted reticle position based at least in part on the angular adjustment associated with the maximum head pitch threshold.

[0241] In aspect 63, the method of aspect 62 further includes obtaining head pose data of a user of the display system from a head pose sensor configured to track the user's head pose, identifying a head pose vector corresponding to the user's head pose based at least in part on the head pose data, and identifying a reference head pose vector corresponding to a reference head pose.

[0242] In a 64th aspect, the method of aspect 62 or aspect 63, wherein calculating the angular difference is based at least in part on a comparison between the head pose vector and a reference head pose vector, and the angular difference corresponds to at least one of a difference in yaw, pitch, or roll of the user's head pose relative to the reference head pose.

[0243] In a sixty-fifth aspect, a method for adjusting a position of a virtual reticle that identifies a user's position within a field of view corresponding to a display of a display system, the method including calculating an angular difference between a head pose vector and a reference head pose vector, where the head pose vector corresponds to a head pose of a user of the display system and the reference head pose vector corresponds to a reference head pose; determining that the angular difference satisfies a minimum head pitch threshold; determining that the angular difference does not satisfy a neutral head pitch threshold; and determining an adjusted reticle position based at least in part on an easing function.

[0244] In a 66th aspect, the method of aspect 65 further includes obtaining head pose data of a user of the display system from a head pose sensor configured to track the user's head pose, identifying a head pose vector corresponding to the user's head pose based at least in part on the head pose data, and identifying a reference head pose vector corresponding to a reference head pose.

[0245] In a 67th aspect, the method of aspect 65, wherein calculating the angular difference is based at least in part on a comparison between the head pose vector and a reference head pose vector, and the angular difference corresponds to at least one of a difference in yaw, pitch, or roll of the user's head pose relative to the reference head pose.

[0246] In a 68th aspect, a method for adjusting a position of a virtual reticle that identifies a user's position within a field of view corresponding to a display of a display system, the method including calculating an angular difference between a head pose vector and a reference head pose vector, where the head pose vector corresponds to a head pose of a user of the display system and the reference head pose vector corresponds to a reference head pose; determining that the angular difference satisfies a neutral head pitch threshold; determining that the angular difference does not satisfy a maximum head pitch threshold; and determining an adjusted reticle position based at least in part on an easing function.

[0247] In a 69th aspect, the method of aspect 68 further includes obtaining head pose data of a user of the display system from a head pose sensor configured to track the user's head pose, identifying a head pose vector corresponding to the user's head pose based at least in part on the head pose data, and identifying a reference head pose vector corresponding to a reference head pose.

[0248] In a seventieth aspect, the method of aspect 68 or aspect 69, wherein calculating the angular difference is based at least in part on a comparison between the head pose vector and a reference head pose vector, the angular difference corresponding to at least one of a difference in yaw, pitch, or roll of the user's head pose relative to the reference head pose. Any of the methods described in any of the preceding aspects can be implemented under control of a hardware processor, for example a hardware processor associated with a head mounted display system.

[0249] In a seventy-first aspect, a system includes a head posture sensor configured to measure head posture data of a user of the system; a non-transitory memory configured to store head posture data corresponding to at least one of an indication of a yaw, pitch, or roll of the user's head; a display configured to be positioned in front of an eye of the user, the display configured to project a virtual reticle toward the eye of the user, the virtual reticle comprising a movable indicator that identifies a position of the user within a field of view; and a hardware processor in communication with the head posture sensor, the display, and the non-transitory memory, the hardware processor performing the steps of: acquiring the user's head posture data; identifying a head posture vector corresponding to the user's head posture based at least in part on the head posture data; identifying a reference head posture vector corresponding to a reference head posture; and determining a head posture vector and a reference head posture vector based at least in part on a comparison between the user's head posture and the reference head posture vector. responsive to determining that the angular difference does not meet the minimum head pose threshold, determine an adjustment for the position of the virtual reticle based at least in part on a first adjustment value; in response to determining that the angular difference meets the maximum head pose threshold, determine an adjustment for the position of the virtual reticle based at least in part on a second adjustment value; in response to determining that the angular difference meets the minimum head pose threshold and does not meet the maximum head pose threshold, determine an adjustment for the position of the virtual reticle based at least in part on a third adjustment value; and causing the position of the virtual reticle to be adjusted from a default reticle position in the user's field of view based on the determined adjustment. (Additional Considerations)

[0250] Each of the processes, methods, and algorithms described herein and / or depicted in the accompanying figures may be embodied in code modules executed by one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, and may be fully or partially automated thereby. For example, the computing system may include a general-purpose computer (e.g., a server) or a special-purpose computer programmed with specific computer instructions, special-purpose circuits, etc. The code modules may be installed in a dynamic link library that is compiled and linked into an executable program, or may be written in a programming language that is interpreted. In some implementations, certain operations and methods may be performed by circuitry that is specific to a given function.

[0251] Furthermore, certain implementations of the functionality of the present disclosure may be sufficiently mathematically, computationally, or technically complex that special purpose hardware (utilizing appropriate specialized executable instructions) or one or more physical computing devices may be required to implement the functionality, e.g., due to the amount or complexity of the calculations involved, or to provide results in substantially real-time. For example, a video may contain many frames, each frame may have millions of pixels, and specially programmed computer hardware is required to process the video data to provide the desired image processing task or application in a commercially reasonable amount of time.

[0252] The code modules or any type of data may be stored on any type of non-transitory computer readable medium, such as physical computer storage devices, including hard drives, solid state memory, random access memory (RAM), read only memory (ROM), optical disks, combinations of volatile or non-volatile storage devices, etc. The methods and modules (or data) may also be transmitted (e.g., as part of a carrier wave or other analog or digital propagated signal) generated over a variety of computer readable transmission media, including wireless-based and wired / cable-based media, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple separate digital packets or frames). The results of the disclosed processes or process steps may be stored persistently or otherwise in any type of non-transitory tangible computer storage device or communicated via a computer readable transmission medium.

[0253] Any process, block, state, step, or functionality in the flow diagrams described herein and / or depicted in the attached figures should be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in the process. The various processes, blocks, states, steps, or functionality can be combined, rearranged, added, deleted, modified, or otherwise altered from the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules may perform some or all of the functionality described herein. The methods and processes described herein are not limited to any particular sequence, and the blocks, steps, or states associated therewith can also be performed in other sequences as appropriate, e.g., serially, in parallel, or in some other manner. Tasks or events can be added to or removed from the disclosed exemplary embodiments. Furthermore, the separation of various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods, and systems may generally be integrated together in a single computer product, or packaged in multiple computer products. Many implementation variations are possible.

[0254] The processes, methods, and systems may be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowdsourced computing networks, the Internet, and the World Wide Web. The network may be a wired or wireless network or any other type of communication network.

[0255] Each of the systems and methods of the present disclosure has several innovative aspects, none of which is solely responsible for or required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of one another or may be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of the present disclosure. Various modifications of the implementations described in the present disclosure may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations without departing from the spirit or scope of the present disclosure. Thus, the claims are not intended to be limited to the implementations shown herein, but are to be accorded the widest scope consistent with the present disclosure, the principles, and novel features disclosed herein.

[0256] Certain features described herein in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation can also be implemented separately in multiple implementations or in any suitable subcombination. Furthermore, although features may be described above as acting in a combination and may even be initially claimed as such, one or more features from the claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination. No single feature or group of features is required or essential to every embodiment.

[0257] In particular, conditional statements used herein, such as "can," "could," "might," "may," "eg," and the like, are generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not, unless specifically stated otherwise or understood otherwise within the context as used. Thus, such conditional statements are generally not intended to suggest that features, elements, and / or steps are in any way required for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are to be included or performed in any particular embodiment, with or without authorial input or prompting. The terms "comprising," "including," "having," and the like, are synonymous and used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, and the like. Also, the term "or" is used in its inclusive sense (and not its exclusive sense), so that, for example, when used to connect a list of elements, the term "or" refers to one, some, or all of the elements in the list. In addition, the articles "a," "an," and "the," as used in this application and the appended claims, should be interpreted to mean "one or more" or "at least one," unless otherwise specified.

[0258] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single elements. As an example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Transitive phrases such as "at least one of X, Y, and Z" are generally understood differently in the context in which they are used to convey that an item, term, etc. may be at least one of X, Y, or Z, unless specifically stated otherwise. Thus, such transitive phrases are generally not intended to suggest that an embodiment requires that each of at least one of X, at least one of Y, and at least one of Z be present.

[0259] Similarly, although operations may be depicted in the figures in a particular order, it should be appreciated that such operations need not be performed in the particular order depicted, or in sequential order, or that all of the depicted operations need not be performed to achieve desirable results. Additionally, the figures may diagrammatically depict one or more exemplary processes in the form of a flow chart. However, other operations not depicted may also be incorporated within the diagrammatically depicted exemplary methods and processes. For example, one or more additional operations may be performed before, after, simultaneously with, or during any of the depicted operations. Additionally, operations may be rearranged or reordered in other implementations. In some circumstances, multitasking and parallel processing may be advantageous. Additionally, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results. (Appendix A) [Header(’Pitcheslimits relative to horizontal’)] public float maxHeadPitch = 5; public float neutralHeadPitch = -20; public float minHeadPitch= -45; [Header(’Angle offsets at the min / max head pitch’)] public float maxHeadPitchDelta = 8; public float minHeadPitchDelta = -12; public Vector3 GetAcceleratedForwardVector() { / / Pitch limitsrelative to horizontal float maxHeadPitch = 5; float neutralHeadPitch = -20; float minHeadPitch = -45; / / Angle offsets at the min / max head pitch float maxHeadPitchDelta = 8; float minHeadPitchDelta = -12; / / Get the camera / headpose forward vector Vector3 forward = Headpose.Forward; / / Get the forward vector projected on to the horizontal plane Vector3 flat = forward; flat.y = 0; flat.Normalize (); / / Calculate the absolute angle between the headpose forward vector and the flattened forward vector float pitch = Mathf.Acos (Vector3.Dot (forward,flat)) * Mathf.Rad2Deg; / / If the forward vector is pointing downwards,then the angle is negative if (forward.y < 0) { pitch = -pitch; } / / pitchAdjustment is the angular deviation from the forward vector float pitchAdjustment = 0; if (pitch >= maxHeadPitch) { / / If pitch > maxHeadPitch,set adjustment to maxHeadPitchDelta pitchAdjustment = maxHeadPitchDelta; } else if (pitch >= neutralHeadPitch) { / / If pitch is between neutralHeadPitch and maxHeadPitch,set adjustment to be an eased value between 0 and maxHeadPitchDelta float t = (pitch - neutralHeadPitch) / (maxHeadPitch - neutralHeadPitch); t = EaseOutSine(t); pitchAdjustment = maxHeadPitchDelta * t; } else if (pitch >= minHeadPitch) { / / If pitch is between neutralHeadPitch and minHeadPitch,set adjustment to be an eased value between 0 and minHeadPitchDelta float t = (pitch - minHeadPitch) / (neutralHeadPitch - minHeadPitch); t = EaseOutSine(1 - t); pitchAdjustment = minHeadPitchDelta * t; } else { / / If pitch < minHeadPitch,set adjustment to minHeadPitchDelta pitchAdjustment = minHeadPitchDelta; } / / If the pitch adjustment is not equal to zero if (pitchAdjustment != 0) { / / Calculate the vector perpendicular to the up and forward vectors Vector3 right = Vector3.Cross (Vector3.up,flat); / / Calculate the rotation adjustment Quaternion adjust = Quaternion.AngleAxis(pitchAdjustment, -right); / / Apply the adjustment to the forward vector to get a new adjusted forward vector forward = adjust * forward; } return forward; } (Copyright) MAGIC LEAP, INC. All rights reserved.

Claims

1. A system, comprising: a sensor configured to acquire a head and / or eye orientation of a user of said system; a non-transitory memory configured to store orientation data; and a display configured to be positioned in front of a user's eye, the display configured to project a virtual reticle within a field of view (FOV) of the user towards the user's eye, the FOV changing as the user's head pose changes; and a hardware processor in communication with the sensor, the display, and the non-transitory memory; Equipped with The hardware processor includes: Recognizing a head and / or eye orientation of the user; determining whether the orientation of the user's head and / or eyes is outside a range of acceptable head and / or eye poses; in response at least in part to determining that the orientation of the user's head and / or eyes is outside the range of acceptable head and / or eye poses, reducing the extent to which the user needs to reorient the user's neck to align the virtual reticle and a target object, said reducing being effected by accelerating movement of the virtual reticle away from a default position toward a position in the direction of the user's head and / or eye movement; A system that is programmed to:

2. The system of claim 1, wherein determining whether the orientation of the user's head and / or eyes is outside the range of acceptable head and / or eye poses further comprises determining whether the acceptable head and / or eye pose meets a first orientation threshold.

3. The system of claim 1, wherein while the orientation of the user's head and / or eyes is outside a range of acceptable head and / or eye poses, the location of the virtual reticle is based at least in part on offset and pose data.

4. The system of claim 1, wherein when the orientation of the user's head is outside a range of acceptable head poses, the location of the virtual reticle is based at least in part on an easing function and pose data.

5. The system of claim 1, wherein the hardware processor is programmed to cause the display to maintain the virtual reticle at a fixed offset position within the FOV offset from the default position while the orientation of the user's head and / or eyes is outside the far-end limit of the range of acceptable head and / or eye postures.

6. The system of claim 1, wherein the virtual reticle comprises a movable indicator that identifies a position within the FOV.

7. The system of claim 1, wherein the head posture is determined in the user's axial plane, the user's coronal plane, and the user's sagittal plane.

8. A system, comprising: a sensor configured to acquire a head and / or eye orientation of a user of said system; a non-transitory memory configured to store orientation data; and a display configured to be positioned in front of a user's eye, the display configured to project a virtual reticle within a field of view (FOV) of the user toward the user's eye, the FOV changing as the user's head pose changes; and a hardware processor in communication with the sensor, the display, and the non-transitory memory; Equipped with The hardware processor includes: Recognizing a head and / or eye orientation of the user; determining whether the orientation of the user's head and / or eyes is outside a range of acceptable head and / or eye poses; in response at least in part to determining that the orientation of the user's head and / or eyes is outside the range of acceptable head and / or eye poses, reducing the extent to which the user needs to reorient the user's neck to align the virtual reticle and a target object, said reducing being effected by moving the virtual reticle away from a default position towards a position in the direction of the user's head and / or eye movement; in response at least in part to determining that the orientation of the user's head and / or eye is within the range of acceptable head and / or eye poses, causing the display to render a virtual reticle at a fixed location within the FOV; A system that is programmed to:

9. The system of claim 8, wherein determining whether the orientation of the user's head and / or eyes is outside a range of acceptable head and / or eye poses further comprises determining whether the acceptable head and / or eye pose meets a first orientation threshold.

10. The system of claim 8, wherein while the orientation of the user's head and / or eyes is outside a range of acceptable head and / or eye poses, the location of the virtual reticle is based at least in part on offset and pose data.

11. The system of claim 8, wherein when the orientation of the user's head is outside a range of acceptable head poses, the location of the virtual reticle is based at least in part on an easing function and pose data.

12. The system of claim 8, wherein the hardware processor is programmed to cause the display to maintain the virtual reticle at a fixed offset position within the FOV offset from the default position while the orientation of the user's head and / or eyes is outside the far-end limit of the range of acceptable head and / or eye postures.

13. The system of claim 8, wherein the virtual reticle comprises a movable indicator that identifies a position within the FOV.

14. The system of claim 8, wherein the head posture is determined in the user's axial plane, the user's coronal plane, and the user's sagittal plane.

15. A method for adjusting a position of a virtual reticle that identifies a position within a field of view (FOV) corresponding to a display of a display system, the method comprising: Recognizing a head and / or eye orientation of the user based at least on data from one or more sensors; determining whether the orientation of the user's head and / or eyes is outside a range of acceptable head and / or eye poses; in response at least in part to determining that the orientation of the user's head and / or eyes is outside the range of acceptable head and / or eye poses, reducing the extent to which the user needs to reorient the user's neck to align the virtual reticle and a target object, said reducing being effected by accelerating movement of the virtual reticle away from a default position toward a position in the direction of the user's head and / or eye movement; in response at least in part to determining that the orientation of the user's head and / or eye is within the range of acceptable head and / or eye poses, causing the display to render a virtual reticle at a fixed location within the FOV; A method comprising:

16. The method of claim 15, wherein determining whether the orientation of the user's head and / or eyes is outside a range of acceptable head and / or eye poses further comprises determining whether the acceptable head and / or eye poses satisfy a first orientation threshold.

17. The method of claim 15, wherein while the orientation of the user's head and / or eyes is outside a range of acceptable head and / or eye poses, the location of the virtual reticle is based at least in part on offset and pose data.

18. The method of claim 15, wherein when the orientation of the user's head is outside a range of acceptable head poses, the location of the virtual reticle is based at least in part on an easing function and pose data.

19. The method of claim 15, further comprising causing the display to maintain the virtual reticle at a fixed offset position within the FOV, offset from the default position, while the orientation of the user's head and / or eyes is outside the far-end limit of the range of acceptable head and / or eye postures.

20. The method of claim 15, wherein the virtual reticle comprises a movable indicator that identifies a position within the FOV.

21. The method of claim 15, wherein the head posture is determined in the user's axial plane, the user's coronal plane, and the user's sagittal plane.

Images