Virtual markings for augmented reality systems

Abstract

Systems and methods for displaying a virtual reticle in an augmented or virtual reality environment via a wearable device are described. The environment may include real or virtual objects that a user interacts with via various gestures (e.g., head gestures, eye gestures or gaze, or body gestures). The user may select an object by changing the gesture or gaze so that the virtual reticle points to the target object. The wearable device may recognize that the orientation of the user's head or eyes is not within an acceptable or comfortable range of head or eye postures, and accelerate the reticle to move away from a default position and toward a position in the direction of the user's head or eye movement, which may reduce the amount of movement of the user's alignment reticle with the target.

Description

[0001] Copyright Notice

[0002] A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyrights whatsoever.

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS

[0004] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 62 / 578,094, filed on October 27, 2017, entitled “VIRTUAL RETICLE FOR AUGMENTED REALITY SYSTEMS,” the entire contents of which are incorporated herein by reference. Technical Field

[0005] The present invention relates to virtual reality and augmented reality imaging and visualization systems, and more particularly, to displaying virtual reticles based on head pose. Background Art

[0006] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality," "augmented reality," or "mixed reality" experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears real or can be perceived as real. Virtual reality or "VR" scenarios typically involve presenting digital or virtual image information opaque to other actual real-world visual input; augmented reality or "AR" scenarios typically involve presenting digital or virtual image information as an enhancement to the visualization of the actual world around the user. Mixed reality or "MR" involves merging the real and virtual worlds to produce a new environment in which physical and virtual objects coexist and interact in real time. It turns out that the human visual perception system is very complex, and it is challenging to produce VR, AR, or MR technologies that facilitate comfortable, natural, and rich presentation of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, or MR technologies. Summary of the invention

[0007] Various embodiments of systems and methods for displaying a virtual reticle are disclosed. An environment may include real or virtual objects that a user interacts with using a virtual reticle. The position or orientation of the user's head, eyes, shoulders, chest, arms, or other body parts may indicate the position or speed of movement of the virtual reticle within the environment, and the user may select or point to an object by directing the virtual reticle toward a target object or focusing the reticle on a target object. The wearable device may identify that the position or orientation of the user's head, one or more eyes, shoulders, chest, one or more arms, etc. is uncomfortable or otherwise undesirable. In response to identifying an uncomfortable or undesirable posture, the system may adjust the position or speed of movement of the virtual reticle to help ideally position the virtual reticle, thereby reducing the likelihood that the user's posture is uncomfortable or undesirable.

[0008] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent through the description, drawings, and claims. The present disclosure and the following detailed description are not intended to define or restrict the scope of the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An illustration of a mixed reality scene with some virtual reality objects and some physical objects viewed by a person is shown.

[0010] Figure 2 An example of a wearable system is schematically shown.

[0011] Figure 3 Aspects of a method of simulating a three-dimensional image using multiple depth planes are schematically illustrated.

[0012] Figure 4 An example of a waveguide stack for outputting image information to a user is schematically shown.

[0013] Figure 5 An example output light beam that may be output by a waveguide is shown.

[0014] Figure 6 is a schematic diagram showing an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem for generating a multi-focal volumetric display, image, or light field.

[0015] Figure 7 is a block diagram of an example of a wearable system.

[0016] Figure 8 is a process flow diagram of an example of a method of presenting virtual content related to an identified object.

[0017] Fig. 9 is a block diagram of another example of a wearable system.

[0018] Fig.10 is a process flow diagram of an example of a method for determining user input for a wearable system.

[0019] Fig.11 is a process flow diagram of an example of a method for interacting with a virtual user interface.

[0020] Fig. 12A Examples of the field of view (FOR), the field of view (FOV) of the world camera, the user's field of view, and the user's fixed field of view are schematically shown.

[0021] Fig. 12B Examples of virtual objects in the user's field of view and virtual objects in the visible area are schematically shown.

[0022] Fig.13A and Fig. 13B shows the vertical change in response to head posture ( Fig.13A ) and level changes ( Fig. 13B ) to accelerate the movement of the marking line.

[0023] Fig.14 An example of adjusting the position of a reticle based on the user's head posture is shown.

[0024] Fig.15 An example relationship between a user's head pose and adjustments to the position of a guideline is shown.

[0025] Fig.16 A flow chart of an example reticle position adjustment process is shown.

[0026] Fig.17 A flow chart of an example reticle adjustment process is shown.

[0027] Fig.18 A flow chart of another example reticle adjustment process is shown.

[0028] Throughout the drawings, reference numerals may be repeatedly used to indicate the corresponding relationship between referenced elements. The drawings are provided to illustrate the example embodiments described herein and are not intended to limit the scope of the present disclosure. In addition, the drawings in the present disclosure are for illustrative purposes and are not drawn to scale. DETAILED DESCRIPTION

[0029] Overview

[0030] The wearable device may include a display for presenting an interactive VR / AR / MR environment. The VR / AR / MR environment may include data elements that a user interacts with through various gestures (e.g., head gestures, eye gaze, or body gestures). The VR / AR / MR environment may include a reticle that corresponds to the user's current position relative to the user's field of view (e.g., the extent of the VR / AR / MR environment seen at any given moment). For example, the reticle may indicate the user's gaze direction. As the user moves around (e.g., by moving the eyes, head, body, or any combination thereof), the reticle may also move with the user. The reticle may point to one or more objects, and the user may select the target object that the reticle is pointing to. For example, a user may move his or her head to point the reticle to the object to be selected, and then click a handheld user input totem to select the object. The wearable display device may perform appropriate operations on the selected object (e.g., moving the object, highlighting or magnifying the object, calling up information about the object, displaying a menu of actions associated with the object that can be performed, etc.).

[0031] At times, a user may desire to reorient his or her head (non-limiting examples: to select an object that is higher in the environment (e.g., a ceiling), an object that is low to the ground (e.g., the floor or the user's feet), an object far to the right, an object far to the left, etc.), which may require the user to tilt, turn, or stretch his or her neck so that the reticle is located over the desired object. In addition to other negative effects, the tilting, turning, and / or stretching of the user's neck can cause neck fatigue or discomfort during use of the wearable device. Therefore, the wearable system can identify that the orientation of the user's head is outside of a range of acceptable (e.g., comfortable, non-fatiguing, etc.) head postures. Therefore, in order to assist the user in moving or adjusting the reticle, the wearable system can modify or accelerate the movement of the reticle (e.g., modify the angle) in a direction corresponding to the direction of the user's head movement. By modifying or accelerating the movement of the reticle, the wearable system advantageously reduces the extent to which the user must tilt, turn, or stretch his or her neck to align the reticle with the target object, thereby reducing the likelihood of neck fatigue or discomfort.

[0032] As an example, if the user is looking upward toward an object near the ceiling, the wearable display device may move the reticle from a default reticle position (non-limiting example: near the center of the display's field of view (FOV)) to another position, such as toward the top of the user's FOV. Similarly, if the user is looking downward, the wearable display device may move the reticle from a default position in the FOV to another position, such as near the bottom of the user's FOV. If the user is looking to the right or left, diagonally to the upper right or upper left, diagonally to the lower right or lower left, etc., the wearable display device may similarly reposition the reticle (e.g., move the reticle to the right or left of the FOV, respectively). The default reticle position is not limited to the center of the FOV. Instead, any position within the FOV may be used as the default reticle position.

[0033] The user may desire to use his or her eyes to adjust the reticle. The wearable device may collect eye data such as eye images (e.g., via an eye camera in an internally facing imaging system of the wearable device). The wearable system may calculate the user's eye gaze direction based on a mapping matrix that provides an association between the user's eye gaze and a gaze vector (which may indicate the user's gaze direction). When the user's eyes move, the wearable system may determine the user's eye gaze direction, and the user's field of view or reticle may move in response to changes in the eye gaze direction. If the user expects to use changes in eye gaze as a substitute or supplement to changes in head posture to view or aim at objects that are high, low, far left, far right, etc., the movement of the user's eyes and / or the gaze of the eyes at a specific position may cause eye fatigue, discomfort, or headaches in addition to other discomforts. Therefore, the wearable system may identify that the orientation of the user's eyes is outside an acceptable (e.g., comfortable, non-fatiguing, etc.) eye posture range. Therefore, in order to assist the user in moving or adjusting the reticle, the wearable system may accelerate the movement of the reticle (e.g., modify the angle) in a direction corresponding to the direction of the user's eye movement. By accelerating the movement of the reticle, the wearable system advantageously reduces the possibility of eye fatigue, headaches, eye discomfort, etc.

[0034] By providing a process by which a user can align a virtual reticule with a target using a combination of head gestures or eye gaze, the wearable system can provide an intuitive process by which the user can more easily align while reducing or minimizing neck strain or eye fatigue.

[0035] Example of 3D display for wearable systems

[0036] A wearable system (also referred to herein as an augmented reality (AR) system) may be configured to present a two-dimensional (2D) or three-dimensional (3D) virtual image to a user. The image may be a still image, a video frame or video, a combination of the above, or the like. A wearable system may include a wearable device that may present a VR, AR, or MR environment, alone or in combination, for user interaction. The wearable device may be a head mounted device (HMD) that may be used interchangeably as an AR device (ARD).

[0037] Figure 1 An illustration of a mixed reality scene with certain virtual reality objects and certain physical objects viewed by a person is shown. Figure 1 , an MR scene 100 is shown in which a user of MR technology sees a real-world park-like setting 110 featuring people, trees, buildings, and a concrete platform 120 in the background. In addition to these items, the user of MR technology also perceives that he "sees" a robotic statue 130 standing on the real-world platform 120, and a cartoon-like avatar character 140 that appears to be an avatar of a bumblebee flying, even though these elements do not exist in the real world.

[0038] In order for a 3D display to produce a realistic sense of depth, and more specifically, a simulated sense of surface depth, it may be necessary for each point in the display field of view to generate an accommodation response corresponding to its virtual depth. If the accommodation response to a displayed point does not correspond to the virtual depth of that point, as determined by the binocular depth cues of convergence and stereoscopic vision, the human eye may experience accommodation conflict, resulting in unstable imaging, harmful eye fatigue, headaches, and, in the absence of accommodation information, an almost complete lack of surface depth.

[0039] VR, AR, and MR experiences can be provided by display systems having displays in which images corresponding to multiple depth planes are provided to a viewer. For each depth plane, the image can be different (e.g., providing a slightly different scene or object presentation) and can be focused on separately by the viewer's eyes, thereby facilitating providing depth cues to the user based on the accommodation of the eyes required to focus on different image features of a scene located at different depth planes, or based on observing different image features at different depth planes that are out of focus. As discussed elsewhere herein, such depth cues provide a reliable sense of depth.

[0040] Figure 2An example of a wearable system 200 is shown. The wearable system 200 includes a display 220, and various mechanical and electronic modules and systems that support the functionality of the display 220. The display 220 can be coupled to a frame 230 that can be worn by a display system user, wearer, or viewer 210. The display 220 can be positioned in front of the eyes of the user 210. The display 220 can present AR / VR / MR content to the user. The display 220 may include a head-mounted display (HMD) worn on the user's head. In some embodiments, a speaker 240 is coupled to the frame 230 and positioned near the user's ear canal (in some embodiments, another speaker (not shown) can be positioned near the user's other ear canal to provide stereo / shapeable sound control).

[0041] The wearable system 200 may include an externally facing imaging system 464 (e.g. Figure 4 The wearable system 200 may also include an internally facing imaging system 462 (as shown in FIG. Figure 4 210 ), the inward-facing imaging system 462 can track the user's eye movements. The inward-facing imaging system can track the movement of one eye or the movement of both eyes. The inward-facing imaging system 462 can be attached to the frame 230 and can be electrically connected to the processing module 260 or 270, which can process the image information obtained by the inward-facing imaging system to determine, for example, the pupil diameter or orientation of the user's 210 eye, eye movement, or eye posture.

[0042] As an example, the wearable system 200 can obtain an image of the user's gesture using the external-facing imaging system 464 or the internal-facing imaging system 462. The image can be a still image, a video frame, or a video, a combination thereof, or the like.

[0043] The display 220 can be operably coupled 250 to a local data processing module 260, such as via a wired lead or a wireless connection, and the local data processing module 260 can be mounted in various configurations, such as fixedly attached to the frame 230, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, in a strap-on configuration).

[0044] The local processing and data module 260 may include a hardware processor, and digital memory, such as non-volatile memory (e.g., flash memory), both of which may be used to assist in the processing, caching, and storage of data. Such data may include a) data captured by sensors (which may, for example, be operably coupled to the frame 230 or otherwise attached to the user 210), such as image capture devices (e.g., cameras in an interior-facing imaging system or an exterior-facing imaging system), microphones, inertial measurement units (IMUs) (e.g., accelerometers, densimeters, magnetometers, etc.), compasses, global positioning system (GPS) units, radios, or gyroscopes; and / or b) data obtained and / or processed using the remote processing module 270 and / or remote data repository 280, which may be transmitted to the display 220 after the aforementioned processing or retrieval is performed. The local processing and data module 260 may be operably connected to a remote processing module 270 and a remote data repository 280 via a communication link 262 or 264 (such as via a wired or wireless communication link) to enable these remote modules to be used as resources for the local processing and data module 260. In addition, the remote processing module 280 and the remote data repository 280 may be operably coupled to each other.

[0045] In some embodiments, the remote processing module 270 may include one or more processors configured to analyze and process data and / or image information. In some embodiments, the remote data repository 280 may include a digital data storage facility that may be obtained via the Internet or other network configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module, allowing for full autonomous use from the remote module.

[0046] The human visual system is complex and providing a realistic sense of depth is challenging. Without being limited by theory, it is believed that a viewer of an object may perceive the object as "three-dimensional" due to a combination of convergence and accommodation. The convergence action of the two eyes relative to each other (i.e., the rotational action of the pupils moving closer or further away from each other to keep the eyes' line of sight focused on an object) is closely related to the focusing (or "accommodation") of the eye lenses. Under normal circumstances, changing the focus of the eye lenses or accommodating the eyes to shift focus from one object to another at a different distance will automatically result in a matching change in convergence to the same distance, according to a relationship known as the "accommodation-convergence reflex." Display systems that provide a better match between accommodation and convergence can form a more realistic and comfortable three-dimensional image simulation.

[0047] Figure 3 Various aspects of a method for simulating a three-dimensional image using multiple depth planes are shown. Figure 3, objects at different distances from 302 and 304 on the z-axis are adjusted by eyes 302 and 304 to bring these objects into focus. Eyes 302 and 304 present a specific state of accommodation to bring objects at different distances on the z-axis into focus. Therefore, a specific accommodation state can be considered to be associated with a specific one of the depth planes 306, and the specific accommodation state has an associated focal length so that when the eye is in an accommodation state for a specific depth plane, the object or part thereof in the specific depth plane is in focus. In some embodiments, a three-dimensional image can be simulated by providing a different presentation of an image for each eye 302 and 304, and also by providing a different presentation of an image corresponding to each depth plane. Although shown as separate for clarity of illustration, it should be understood that the fields of view of eyes 302 and 304 can overlap, for example when the distance along the z-axis increases. In addition, although shown as flat for ease of illustration, it should be understood that the contours of the depth planes can be curved in physical space so that all features in the depth planes are in focus when the eyes are in a specific accommodation state. Without being limited by theory, it is believed that the human eye can generally interpret a limited number of depth planes to provide a sense of depth. Therefore, a highly credible perceived depth simulation can be achieved by providing the eye with a different presentation of an image corresponding to each of these limited number of depth planes.

[0048] Waveguide stack components

[0049] Figure 4 An example of a waveguide stack for outputting image information to a user is shown. The wearable system 400 includes a stack of waveguides or a stacked waveguide assembly 480 that can be used to provide a three-dimensional perception to the eye / brain using multiple waveguides 432b, 434b, 436b, 438b, 4400b. In some embodiments, the wearable system 400 can be used with Figure 2 Corresponding to the wearable system 200, Figure 4 Some parts of the wearable system 200 are schematically shown in more detail. For example, in some embodiments, the waveguide assembly 480 can be integrated into Figure 2 in the display 220 .

[0050] Continue to refer Figure 4 , the waveguide assembly 480 may also include a plurality of features 458, 456, 454, 452 located between the waveguides. In some embodiments, the features 458, 456, 454, 452 may be lenses. In other embodiments, the features 458, 456, 454, 452 may not be lenses. Instead, they are simply spacers (e.g., cladding and / or structures used to form air gaps).

[0051] The waveguides 432b, 434b, 436b, 438b, 440b and / or the plurality of lenses 458, 456, 454, 452 can be configured to send image information to the eye at various levels of wavefront curvature or light divergence. Each waveguide level can be associated with a particular depth plane and can be configured to output image information corresponding to that depth plane. The image injection devices 420, 422, 424, 426, 428 can be used to inject image information into the waveguides 440b, 438b, 436b, 434b, 432b, each of which can be configured to distribute incident light across each respective waveguide for output toward the eye 410. Light exits the output surface of the image injection devices 420, 422, 424, 426, 428 and is injected into the corresponding pair of input edges of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam (e.g., a collimated beam) can be injected into each waveguide to output an entire field of cloned collimated beams directed toward the eye 410 at specific angles (and divergences) corresponding to the depth plane associated with the particular waveguide.

[0052] In some embodiments, the image injection devices 420, 422, 424, 426, 428 are separate displays, each of which generates image information for injection into the corresponding waveguides 440b, 438b, 436b, 434b, 432b, respectively. In some other embodiments, the image injection devices 420, 422, 424, 426, 428 are the outputs of a single multiplexed display, which can, for example, pipe image information to each of the image injection devices 420, 422, 424, 426, 428 via one or more optical conduits (e.g., fiber optic cables).

[0053] Controller 460 controls the operation of stacked waveguide assembly 480 and image injection pens 420, 422, 424, 426, 428. Controller 460 includes programming (e.g., instructions in a non-transitory computer readable medium) that regulates the timing and provision of image information to waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, controller 460 may be a single integrated device, or a distributed system connected via wired or wireless communication channels. In some embodiments, controller 460 may be a processing module 260 or 270 (e.g., Figure 2 shown).

[0054] The waveguides 440b, 438b, 436b, 434b, 432b can be configured to propagate light within each respective waveguide by total internal reflection (TIR). The waveguides 440b, 438b, 436b, 434b, 432b can each be planar or have another shape (e.g., curved) having a top major surface and a bottom major surface and an edge extending between these top major surfaces and bottom major surfaces. In the illustrated configuration, the waveguides 440b, 438b, 436b, 434b, 432b can each include light extraction optical elements 440a, 438a, 436a, 434a, 432a that are configured to extract light from the waveguides by redirecting light propagating within each respective waveguide out of the waveguides, thereby outputting image information to the eye 410. The extracted light can also be referred to as outcoupled light, and the light extraction optical elements can also be referred to as outcoupling optical elements. At the location where light propagating within the waveguide strikes the light redirecting element, an extracted light beam can be output by the waveguide. The light extraction optical elements (440a, 438a, 436a, 434a, 432a) can be, for example, reflective or diffractive optical features. Although for ease of description and drawing clarity, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are shown as being disposed at the bottom major surface of the waveguides 440b, 438b, 436b, 434b, 432b, in some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be disposed on the top major surface or the bottom major surface, and / or can be disposed directly in the body of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed in a material layer attached to a transparent substrate to form the waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, 432b may be a monolithic piece of material, and the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed on a surface and / or within the piece of material.

[0055] Continue to refer Figure 4As discussed herein, each waveguide 440b, 438b, 436b, 434b, 432b is configured to output light to form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye can be configured to transmit collimated light injected into such waveguide 432b to the eye 410. The collimated light can represent an optical infinity focal plane. The next upstream waveguide 434b can be configured to emit collimated light that is transmitted through a first lens 452 (e.g., a negative lens) before reaching the eye 410; the first lens 452 can be configured to produce a slightly convex wavefront curvature so that the eye / brain interprets the light from the next upstream waveguide 434b as coming from a first focal plane that is closer to the eye 410 from optical infinity inward. Similarly, the third upstream waveguide 436b has its output light transmitted through the first lens 452 and the second lens 454 before reaching the eye 410. The combined optical power of the first lens 452 and the second lens 454 can be configured to produce another wavefront curvature increment so that the eye / brain interprets light from the third waveguide 436b as light from a second focal plane further inward from optical infinity and closer to the person, rather than light from the next upstream waveguide 434b.

[0056] The other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured, with the highest waveguide 440b in the stack sending its output through all lenses between it and the eye to obtain a total focal power that represents the focal plane closest to the person. In order to compensate for the lens stack 458, 456, 454, 452 when observing / interpreting light from the world 470 on the other side of the stacked waveguide assembly 480, a compensating lens layer 430 can be provided at the top of the stack to compensate for the total focal power of the lens stack 458, 456, 454, 452 below. This configuration provides as many perceived focal planes as there are available waveguide / lens pairs. Both the light extraction optics of the waveguide and the focusing aspects of the lens can be static (i.e., not dynamic or electro-active). In some embodiments, one or both of them can be dynamic through the use of electro-active features.

[0057] Continue to refer Figure 4, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be configured to redirect light out of their respective waveguides and output the light with an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. Thus, waveguides with different associated depth planes can have differently configured light extraction optical elements that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be volume features or surface features that can be configured to output light at specific angles. For example, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be volume holograms, surface holograms, and / or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication Serial No. 2015 / 0178939, published on June 25, 2015, the entire contents of which are incorporated herein by reference.

[0058] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffractive features, or "diffractive optical elements" (also referred to herein as "DOEs") that form a diffraction pattern. Preferably, the DOE has a relatively low diffraction efficiency so that only a portion of the light beam is deflected toward the eye 410 by each intersection of the DOE, while the remainder continues through the waveguide via total internal reflection. Thus, the light carrying image information can be split into multiple related exit beams that exit the waveguide at multiple locations, and for a particular collimated beam that bounces around within the waveguide, the result is a fairly uniform pattern that is emitted toward the eye 304.

[0059] In some embodiments, one or more DOEs can be switched between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE can include a polymer dispersed liquid crystal layer in which a droplet includes a diffractive pattern in a host medium, and the refractive index of the droplet can be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the droplet can be switched to a refractive index that mismatches the refractive index of the host medium (in which case the pattern actively diffracts incident light).

[0060] In some embodiments, the number and distribution of depth planes or depth of field can be dynamically changed based on the pupil size or orientation of the viewer's eye. The depth of field can change inversely with the viewer's pupil size. Therefore, when the pupil size of the viewer's eye decreases, the depth of field increases, so that the plane that cannot be distinguished due to the position of the plane beyond the focal depth of the eye may become distinguishable and appear more in focus as the pupil size decreases and the corresponding increase in depth of field is commensurate. Similarly, the number of spaced-apart depth planes used to present different images to the viewer can decrease as the pupil size decreases. For example, the viewer may not be able to clearly perceive the details of both the first depth plane and the second depth plane at one pupil size without adjusting the adaptation of the eye away from one depth plane and close to another depth plane. However, the two depth planes can be fully focused on the user at the same time at another pupil size without changing the adaptation.

[0061] In some embodiments, the display system can change the number of waveguides that receive image information based on a determination of pupil size or orientation, or based on receiving an electrical signal indicative of a particular pupil size or orientation. For example, if the user's eye cannot distinguish between two depth planes associated with two waveguides, the controller 460 can be configured or programmed to stop providing image information to one of the waveguides. Advantageously, this can reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments where the DOE for the waveguide is switchable between an on and off state, the DOE can be switched to an off state when the waveguide does receive image information.

[0062] In some embodiments, it may be desirable to have the exit beam satisfy a condition of having a diameter less than the diameter of the viewer's eye. However, satisfying this condition may be challenging given the variation in viewer pupil size. In some embodiments, this condition is satisfied over a wide range of pupil sizes by varying the size of the exit beam in response to a determination of the viewer's pupil size. For example, as the pupil size decreases, the size of the exit beam may also decrease. In some embodiments, a variable aperture may be used to vary the size of the exit beam.

[0063] The wearable system 400 may include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the field of view (FOV) of the world camera, and the imaging system 464 is sometimes referred to as a FOV camera. The entire area available for viewing or imaging by a viewer may be referred to as the field of view (FOR). The FOR may include a solid angle of 4π radians around the wearable system 400, because the wearer can move his or her body, head, or eyes to perceive essentially any direction in space. In other scenarios, the wearer's movements may be more limited, and accordingly, the wearer's FOR may subtend a smaller solid angle. Images obtained from the outward-facing imaging system 464 may be used to track gestures made by the user (e.g., hand or finger gestures), detect objects in the world 470 in front of the user, and the like.

[0064] The wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera) that observes the user's actions, such as eye movements and facial movements. The inward-facing imaging system 466 may be used to capture images of the eye 410 to determine the size and / or orientation of the pupil of the eye 304. The inward-facing imaging system 466 may be used to obtain images used to determine the viewing direction of the user (e.g., eye posture) or for biometric identification of the user (e.g., via iris recognition). In some embodiments, at least one camera may be used for each eye to separately determine the pupil size or eye posture of each eye separately, thereby allowing the image information presentation for each eye to dynamically meet the eye. In some other embodiments, only the pupil diameter or orientation of a single eye 410 is determined (e.g., only a single camera is used for each pair of eyes), and it is assumed to be similar for both eyes of the user. The images obtained by the inward-facing imaging system 466 can be analyzed to determine the user's eye posture or mood, and the wearable system 400 can use this determination to decide the audio or visual content that should be presented to the user. Wearable system 400 may also determine head pose (eg, head position or head orientation) using one or more head pose sensors such as an IMU (which may include an accelerometer, gyroscope, or magnetometer).

[0065] The wearable system 400 may include a user input device 466 through which a user may input commands to the controller 460 to interact with the wearable system 400. For example, the user input device 466 may include a touchpad, a touch screen, a joystick, a multi-degree-of-freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a handheld manipulator, a haptic device, a totem (e.g., used as a virtual user input device), etc. A multi-DOF controller may sense user input in terms of some or all of the possible translations (e.g., left / right, forward / backward, or up / down) or rotations (e.g., yaw, pitch, or roll) of the controller. A multi-DOF controller that supports translational motion may be referred to as 3DOF, while a multi-DOF controller that supports translation and rotation may be referred to as 6DOF. In some cases, a user may use a finger (e.g., a thumb) to press or swipe on a touch-sensitive input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 can be held by the user during use of the wearable system 400. The user input device 466 can communicate with the wearable system 400 by wire or wirelessly.

[0066] Figure 5 An example of an exit beam output by a waveguide is shown. One waveguide is shown, but it should be understood that other waveguides in the waveguide assembly 480 can function similarly, where the waveguide assembly 480 includes multiple waveguides. Light 520 is injected into the waveguide 432b at the input edge 432c of the waveguide 432b and propagates within the waveguide 432b by TIR. At the point where the light 520 impinges on the DOE 432a, a portion of the light exits the waveguide as an exit beam 510. The exit beams 510 are shown as being substantially parallel, but they can also be redirected to propagate to the eye 410 at a certain angle (e.g., to form a diverging exit beam), which angle depends specifically on the depth plane associated with the waveguide 432b. It should be understood that substantially parallel exit beams can be indicative of a waveguide having light extraction optical elements that couple out light to form an image that appears to be disposed at a depth plane that is a large distance away from the eye 410 (e.g., optical infinity). Other waveguides or other light extraction optical element groups may output a more divergent exit beam pattern, which would require the eye 410 to adapt to a closer distance to focus it on the retina, and would be interpreted by the brain as light coming from a distance closer to the eye 410 than optical infinity.

[0067] Figure 6632a( 632b ) is a schematic diagram showing an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem, and the control subsystem is used to produce a multi-focal stereoscopic display, image, or light field. The optical system may include a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem. The optical system may be used to generate a multi-focal volume, image, or light field. The optical system may include one or more primary planar waveguides 632a( Figure 6 Only one is shown) and one or more DOEs 632b associated with each of at least some of the main waveguides 632a. The planar waveguides 632b are similar to those of reference Figure 4 The optical system may use a distributed waveguide device to transmit the optical signal along a first axis ( Figure 6 The distributed waveguide device may include, for example, a distributed planar waveguide 622b and at least one DOE 622a (shown by a double-dashed line) associated with the distributed planar waveguide 622b. The distributed planar waveguide 622b may be similar or identical to the main planar waveguide 632b in at least some aspects, but may be oriented differently therefrom. Similarly, at least one DOE 622a may be similar or identical to the DOE 632a in at least some aspects. For example, the distributed planar waveguide 622b or the DOE 622a may be composed of the same material as the main planar waveguide 632b or the DOE 632a, respectively. Figure 6 The embodiment of the optical display system 600 shown can be integrated into Figure 2 In the wearable system 200 shown.

[0068] The relayed and pupil-expanded light can be optically coupled out of the distributed waveguide device and into one or more main planar waveguides 632b. The main planar waveguides 632b can be arranged along a second axis (e.g., Figure 6 The main planar waveguide 632b can relay and expand light along the vertical axis or Y axis and pass the light to the main planar waveguide 632b, which can relay and expand light along the horizontal axis or X axis. It is worth noting that the second axis can be an axis that is not perpendicular to the first axis. The main planar waveguide 632b expands the effective exit pupil of the light along the second axis (e.g., X axis). For example, the distribution planar waveguide 622b can relay and expand light along the vertical axis or Y axis and pass the light to the main planar waveguide 632b, which can relay and expand light along the horizontal axis or X axis.

[0069] The optical display system can include one or more color light sources (e.g., red, green, and blue lasers) 610 that can be optically coupled to the proximal end of a single-mode optical fiber 640. The distal end of the optical fiber 640 is threaded or received by a hollow tube 642 made of a piezoelectric material. The distal end protrudes from the tube 642 as a non-fixed flexible cantilever 644. The piezoelectric tube 642 can be associated with four quadrant electrodes (not shown). For example, these electrodes can be plated on the outside, outer surface, or outer periphery or outer diameter of the tube 642. The core electrode (not shown) can also be located in the core, center, inner periphery, or inner diameter of the tube 642.

[0070] The drive electronics 650, electrically coupled, for example, via wires 660, drives the opposing electrode pairs to independently bend the piezoelectric tube 642 along two axes. The protruding distal end of the optical fiber 644 has a mechanical resonant mode. The resonant frequency depends on the diameter, length, and material properties of the optical fiber 644. By vibrating the piezoelectric tube 642 near the first mechanical resonant mode of the optical fiber cantilever 632, the optical fiber cantilever 644 can be vibrated and can be swept through large deflections.

[0071] By stimulating the resonance of two axes, the tip of the fiber cantilever 644 is scanned biaxially in an area that fills a two-dimensional (2D) scan. By modulating the intensity of one or more light sources 610 synchronously with the scanning of the fiber cantilever 644, the light emitted from the fiber cantilever 644 can form an image. A description of this arrangement is provided in U.S. Patent Publication No. 2014 / 0003762, the entire contents of which are incorporated herein by reference.

[0072] Components of the optical coupler subsystem can collimate the light emitted by the scanning fiber cantilever 644. The collimated light can be reflected by the reflective mirror 648 into a narrow distributed planar waveguide 622b containing at least one diffractive optical element (DOE) 622a. The collimated light can be vertically (relative to the optical fiber) along the distributed planar waveguide 622b by TIR. Figure 6 The light propagates along the main planar waveguide 632b (view) and in doing so repeatedly intersects the DOE 622a. The DOE 622a preferably has a low diffraction efficiency. This can result in a portion of the light (e.g., 10%) being diffracted toward the edge of the larger main planar waveguide 632b at each intersection with the DOE 622a, and a portion of the light continuing on its original trajectory down the length of the distributed planar waveguide 622b via TIR.

[0073] At each intersection with the DOE 622a, additional light can be diffracted toward the entrance of the main waveguide 632b. By splitting the incident light into multiple outcoupled sets, the exit pupil of light can be expanded vertically in the distribution planar waveguide 622b by the DOE 4. This vertically expanded light coupled out of the distribution planar waveguide 622b can enter the edge of the main planar waveguide 632b.

[0074] The light entering the main waveguide 632b propagates horizontally along the main waveguide 632b via TIR (relative to Figure 6 632a). As the light propagates horizontally along at least a portion of the length of the main waveguide 632b via TIR, the light intersects the DOE 632a at multiple points. The DOE 632a may advantageously be designed or configured to have a phase profile that is a sum of a linear diffraction pattern and a radially symmetric diffraction pattern to produce deflection and focusing of the light. The DOE 632a may advantageously have a low diffraction efficiency (e.g., 10%) so that only a portion of the light beam is deflected toward the direction of the field of view at each intersection point of the DOE 632a, while the remainder of the light continues to propagate through the main waveguide 632b via TIR.

[0075] At each intersection between the propagating light and the DOE 632a, a portion of the light is diffracted toward the adjacent face of the main waveguide 632b, allowing the light to escape TIR and exit from the face of the main waveguide b32b. In some embodiments, the radially symmetric diffraction pattern of the DOE 632a additionally imparts a focus level to the diffracted light, both shaping the optical wavefront of the individual beams (e.g., imparting curvature) and directing the beams at angles that match the designed focus level.

[0076] Thus, these different paths can couple light out of the main planar waveguide 632b through multiple DOEs 632a at different angles, focus levels, and / or producing different fill patterns at the exit pupil. The different fill patterns at the exit pupil can be advantageously used to create a light field display with multiple depth planes. Each layer in the waveguide assembly or set of layers in the stack (e.g., 3 layers) can be used to produce a corresponding color (e.g., red, blue, green). Thus, for example, a first set of layers consisting of three adjacent layers can be used to produce red, blue, and green light at a first focal depth, respectively. A second set of layers consisting of three adjacent layers can be used to produce red, blue, and green light at a second focal depth, respectively. Multiple layer sets can be used to generate full 3D or 4D color image light fields with various focal depths.

[0077] Other components of wearable systems

[0078] In many embodiments, the wearable system may include other components as a supplement or alternative to the components of the wearable system described above. The wearable system may, for example, include one or more haptic devices or components. The haptic device or component can provide a sense of touch to the user through operation. For example, the haptic device or component can provide a sense of touch of pressure or texture when contacting virtual content (e.g., a virtual object, a virtual tool, other virtual constructions). The sense of touch can replicate the feeling of a physical object represented by the virtual object, or can replicate the feeling of an imaginary object or character (e.g., a dragon) represented by the virtual content. In some embodiments, the user can wear a haptic device or component (e.g., the user can wear gloves). In some embodiments, the haptic device or component can be held by the user.

[0079] The wearable system may, for example, include one or more physical objects that can be manipulated by a user to allow input or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as sheets of metal or plastic, walls, desktops. In some embodiments, the totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, the totem may only provide a physical surface, and the wearable system may present a user interface so that the user appears to be on one or more surfaces of the totem. For example, the wearable system may make images of a computer keyboard and trackpad appear to be located on one or more surfaces of the totem. For example, the wearable system may make a virtual computer keyboard and a virtual trackpad appear on the surface of a rectangular thin aluminum plate used as a totem. The rectangular plate itself does not have any physical keys or trackpads or sensors. However, when a selection or input is made via the virtual keyboard or virtual trackpad, the wearable system may detect user manipulation or interaction or touch through the rectangular plate. User input device 466 (such as Figure 4 ) may be an embodiment of a totem, which may include a touchpad, a touch pad, a trigger, a joystick, a trackball, a rocker or a virtual switch, a mouse, a keyboard, a multi-degree-of-freedom controller, or other physical input devices. A user may use a totem alone or in combination with gestures to interact with a wearable system or other users.

[0080] Examples of haptic devices and totems that may be used with the wearable device HMD and display system of the present disclosure are described in U.S. Patent Publication Serial No. 2015 / 0016777, the entire contents of which are incorporated herein by reference.

[0081] Examples of wearable systems, environments, and interfaces

[0082] The wearable system can employ various mapping-related techniques to achieve high depth of field in presenting light fields. When mapping a virtual world, it is advantageous to understand all features and points in the real world to accurately depict virtual objects in relation to the real world. To this end, a FOV image captured from a user of the wearable system can be added to the world model by including a new picture that conveys information about various points and features of the real world. For example, the wearable system can collect mapping points (e.g., 2D points or 3D points) and find new mapping points to present a more accurate version of the world model. The world model of a first user can be transmitted to a second user (e.g., via a network such as a cloud network) so that the second user can experience the world around the first user.

[0083] Figure 7 is a block diagram of an example of an MR environment 700. The MR environment 700 can be configured to receive input (e.g., visual input 702 from the user's wearable system, fixed input 704 such as a room camera, sensor input 706 from various sensors (gestures, totems, eye tracking sensors), user input from input device 466, etc.) from one or more user wearable systems (e.g., wearable system 200 or display system 220) or fixed room systems (e.g., room cameras, etc.). The wearable system can use various sensors (e.g., accelerometers, gyroscopes, temperature sensors, motion sensors, depth sensors, GPS sensors, internal-facing imaging systems, external-facing imaging systems, etc.) to determine the location of the user environment and various other attributes. This information can be further supplemented by information from fixed cameras in the room, which can provide images or various clues from different viewpoints. The image data obtained by the camera (such as a room camera and / or a camera of an external-facing imaging system) can be reduced to a set of mapping points.

[0084] One or more object identifiers 708 can browse the received data (e.g., point sets) and identify or draw points, mark images, and attach semantic information to objects with the help of a map database 710. The map database 710 may include various points collected over a period of time and their corresponding objects. Various devices and map databases may be connected to each other through a network (e.g., LAN, WAN, etc.) to access the cloud.

[0085] Based on this information and the set of points in the mapping database, object identifiers 708a to 708n can identify objects in the environment. For example, an object identifier can identify faces, people, windows, walls, user input devices, televisions, other objects in the user's environment, etc. One or more object identifiers can be dedicated to objects with specific characteristics. For example, object identifier 708a can be used to identify faces, while another object identifier can be used to identify totems.

[0086] Object recognition can be performed using various computer vision techniques. For example, the wearable system can analyze the imagery captured by the outward-facing imaging system 464 (e.g., Figure 4 The images obtained by the computer vision system (as shown) can be used to perform scene reconstruction, event detection, video tracking, object recognition, object pose estimation, learning, indexing, motion estimation or image restoration, etc. One or more computer vision algorithms can be used to perform these tasks. Non-limiting examples of computer vision algorithms include: Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retinal Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces method, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, Visual Simultaneous Localization and Mapping (vSLAM) technology, continuous Bayesian estimation (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, adaptive thresholding method (and other thresholding techniques), iterative closest point (ICP), semi-global matching (SGM), semi-global block matching (SGBM), feature point histograms, various machine learning algorithms (e.g., support vector machine, k-nearest neighbor algorithm, Naive Bayes (Naive Bayes), neural networks (including convolutional or deep neural networks), or other supervised / unsupervised models, etc.

[0087] In some embodiments, object recognition can be performed by various machine learning algorithms. Once trained, the machine learning algorithm can be stored by the HMD. Some examples of machine learning algorithms can include supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., ordinary least squares regression), instance-based algorithms (e.g., learning vector quantization), decision tree algorithms (e.g., classification and regression trees), Bayesian algorithms (e.g., naive Bayes), clustering algorithms (e.g., k-means clustering), association rule learning algorithms (e.g., prior algorithms), artificial neural network algorithms (e.g., perceptrons), deep learning algorithms (e.g., deep Boltzmann machines or deep neural networks), dimensionality reduction algorithms (e.g., principal component analysis), ensemble algorithms (e.g., stack generalization), and / or other machine learning algorithms. In some embodiments, each model can be customized for each data set. For example, the wearable device can generate or store a base model. The base model can be used as a starting point for generating other models that are specific to data types (e.g., a specific user in a telepresence session), data sets (e.g., additional image sets obtained from a user in a telepresence session), conditional situations, or other changes. In some embodiments, the wearable HMD can be configured to utilize a variety of techniques to generate a model for analyzing aggregated data. Other techniques may include using predefined thresholds or data values.

[0088] Based on this information and the point set in the mapping database, object identifiers 708a to 708n can identify objects and supplement objects with semantic information to give them life. For example, if the object identifier identifies a point set as a door, the system can attach some semantic information (e.g., the door has a hinge and has a 90-degree movement around the hinge). If the object identifier identifies a point set as a mirror, the system can attach semantic information about the mirror having a reflective surface that can reflect the image of the object in the room. Over time, as the system (which can reside locally or can be accessed via a wireless network) accumulates more data from the world, the mapping database grows. Once an object is identified, the information can be sent to one or more wearable systems. For example, the MR environment 700 can include information about scenes occurring in California. The environment 700 can be sent to one or more users in New York. Based on data received from the FOV camera and other inputs, the object identifier and other software components can map points collected from various images, identify objects, etc., so that the scene can be accurately "delivered" to a second user who may be located in another part of the world. The environment 700 can also use a topological map to achieve localization.

[0089] Figure 88 is a process flow diagram of an example of a method 800 for presenting virtual content related to an identified object. The method 800 describes how to reproduce a virtual scene to a user of a wearable system. The user may be geographically far away from the scene. For example, the user may be in New York, but may want to watch a scene currently taking place in California, or may want to take a walk with a friend who lives in California.

[0090] At box 810, the wearable system can receive input about the user's environment from the user and other users. This can be achieved through various input devices and knowledge already stored in the mapping database. At box 810, the user's FOV camera, sensor, GPS, eye tracking, etc. transmit information to the system. At box 820, the system can determine sparse points based on this information. The sparse points can be used to determine posture data (e.g., head posture, eye posture, body posture, or gesture), which can be used to display and understand the orientation and position of various objects in the user's surrounding environment. At box 830, the object identifiers 708a to 708n can browse these collection points and use the mapping database to identify one or more objects. Then, at box 840, the information can be transmitted to the user's personal wearable system, and at box 850, the desired virtual scene can be displayed to the user accordingly. For example, the desired virtual scene (e.g., a user located in CA) can be displayed with appropriate orientation, position, etc. relative to various objects and other environments of the user in New York.

[0091] Fig. 9 is a block diagram of another example of a wearable system. In this example, the wearable system 900 includes a map, which may include map data of the world. The map may reside partially locally on the wearable system and partially on a network storage location accessible by a wired or wireless network (e.g., in a cloud system). Posture processing 910 may be performed on a wearable computing architecture (e.g., processing module 260 or controller 460) and utilize data from the map to determine the position and orientation of the wearable computing hardware or user. Posture data may be calculated based on data collected in real time while the user is experiencing the system and performing operations in the world. The data may include images, data from sensors (e.g., an inertial measurement unit (IMU), which may include an accelerometer, a gyroscope, a magnetometer, or a combination of these components), and surface information associated with objects in a real or virtual environment.

[0092] Sparse point representations can be the output of a simultaneous localization and mapping (SLAM or V-SLAM, indicating a configuration where the input is only images / vision) process. The system can be configured to not only figure out where various parts are in the world, but also what the world is made of. Poses can be a building block to achieve many goals, including populating maps and using data from maps.

[0093] In one embodiment, the sparse point locations themselves may not be completely sufficient, and further information may be required to produce a multi-focal AR, VR, or MR experience. The gap can be at least partially filled using a dense representation that typically references depth map information. Such information can be calculated according to what is referred to as stereo imaging processing 940, where depth information is determined using techniques such as triangulation or time-of-flight sensing. Image information and active patterns (such as infrared patterns created using active projectors) can be used as inputs to stereo processing 940. A large amount of depth map information can be fused together, and some of it can be summarized using surface representations. For example, a mathematically definable surface is an effective (e.g., relative to a large point cloud) and understandable input to other processing devices such as a game engine. Therefore, the output of the stereo processing (e.g., a depth map) 940 can be combined by a fusion process 930. Poses can also be input to this fusion process 930, and the output of the fusion 930 becomes the input to fill the mapping process 920. Sub-surfaces can be connected to each other (e.g., in a topographic map) to form a larger surface, and the map becomes a large mixture of points and surfaces.

[0094] To address various aspects of mixed reality processing 960, various inputs may be used. Fig. 9 In the illustrated embodiment, game parameters may be input to determine that the user of the system is playing a monster-killing game, where one or more monsters are located at various locations, monsters die or flee under various conditions (e.g., if the user shoots the monster), walls or other objects are located at various locations, etc. The world map may include information about where these objects are relative to each other, as another valuable input to mixed reality. Posture relative to the world also becomes an input and plays a key role in almost any interactive system.

[0095] Control or input from the user is another input to the wearable system 900. As described herein, user input can include visual input, gestures, totems, audio input, sensory input, and the like. In order to move around or play games, for example, the user may need to indicate to the wearable system 900 what he or she wants to do. In addition to moving oneself in space, various forms of user control can be used. In one embodiment, a totem (e.g., a user input device) or an object such as a toy gun can be held by the user and tracked by the system. The system is preferably configured to know that the user is holding an item and understand what kind of interaction the user is performing with the item (e.g., if the totem or object is a gun, the system can be configured to understand the position and orientation, and whether the user is clicking a trigger or other button or element that may be equipped with a sensor, such as an IMU, which can help determine what is happening, even if such activity is not within the field of view of any camera).

[0096] Gesture tracking or recognition can also provide input information. The wearable system 900 can be configured to track and interpret gestures of button presses, gestures indicating left or right, stop, grab, hold, etc. For example, in one configuration, a user may need to flip through email or a calendar in a non-gaming environment, or "fist" with other people or players. The wearable system 900 can be configured to utilize a minimal amount of gestures, which may or may not be dynamic. For example, a gesture can be a simple static gesture, such as an open hand to indicate stop, a thumb up to indicate perfection, and a thumb down to indicate imperfection; or flipping the hand left or right or up and down to make a directional command.

[0097] Eye tracking is another input (e.g., tracking where the user is looking to control the display technology to render at a specific depth or range). In one embodiment, triangulation can be used to determine the vergence of the eyes, and then using a vergence / accommodation model developed for that particular person, accommodation can be determined.

[0098] Regarding the camera system, Fig. 9 The example wearable system 990 shown may include three camera pairs: a relatively wide FOV camera pair or passive SLAM cameras, which are arranged on the sides of the user's face; and an additional camera pair, which is positioned in front of the user to handle binaural imaging processing 940 and also capture gestures and totems / object tracking in front of the user. The FOV camera and the camera pair for stereo imaging processing 940 may be part of the externally facing imaging system 464 (e.g., Figure 4 The wearable system 900 may include an eye tracking camera (which may be Figure 4 The wearable system 900 may also include one or more texture light projectors (e.g., infrared (IR) projectors) to inject texture into the scene.

[0099] Fig.10 1000 is a process flow diagram of an example of a method 1000 for determining user input to a wearable system. In this example, a user can interact with a totem. The user may have multiple totems. For example, the user may have designated a totem for a social media application, another totem for playing a game, and so on. At block 1010, the wearable system may detect the motion of the totem. The motion of the totem may be recognized by an externally facing system, or may be detected by a sensor (e.g., a tactile glove, an image sensor, a hand tracking device, an eye tracking camera, a head pose sensor, etc.).

[0100] At block 1020, based at least in part on the detected gesture, eye pose, head pose, or input through the totem, the wearable system detects the position, orientation, and / or motion of the totem (or the user's eye pose or head pose or gesture) relative to a reference frame. The reference frame can be a set of mapping points based on which the wearable system converts the motion of the totem (or the user) into an action or command. At block 1030, the user's interaction with the totem is mapped. At block 1040, based on the mapping of the user interaction relative to the reference frame 1020, the system determines the user input.

[0101] For example, a user may move a totem or physical object forward and backward to represent turning a virtual page and moving to the next page or moving from one user interface (UI) display screen to another UI screen. As another example, a user may move their head or eyes to view different real or virtual objects in the user's FOR. If the user gazes at a particular real or virtual object for longer than a threshold time, the real or virtual object may be selected as user input. In some embodiments, the convergence of the user's eyes may be tracked, and an accommodation / convergence model may be used to determine the accommodation state of the user's eyes, which provides information about the depth plane on which the user is focusing. In some embodiments, the wearable system may use ray casting techniques to determine which real or virtual objects are along the direction of the user's head pose or eye pose. In various embodiments, the ray casting technique may include projecting a narrow beam of rays having a substantially small lateral width or projecting rays having a large lateral width (e.g., a cone or frustum).

[0102] The user interface may be provided by a display system (eg, Figure 2 The user interface may be displayed using a display 220 in the system. It may also be displayed using various other technologies, such as one or more projectors. The projector may project the image onto a physical object such as a canvas or a globe. Interactions with the user interface may be tracked using one or more cameras external to or as part of the system (e.g., using internally facing imaging system 462 or externally facing imaging system 464).

[0103] Fig.11 is a process flow diagram of an example of a method 1100 for interacting with a virtual user interface. The method 1100 may be performed by the wearable system described herein.

[0104] At box 1110, the wearable system may identify a specific UI. The type of UI may be predetermined by the user. The wearable system may identify that a specific UI needs to be populated based on user input (e.g., gestures, visual data, audio data, sensory data, direct commands, etc.). At box 1120, the wearable system may generate data for a virtual UI. For example, data associated with the limitations, general structure, shape, etc. of the UI may be generated. In addition, the wearable system may determine the map coordinates of the user's physical location so that the wearable system can display a UI related to the user's physical location. For example, if the UI is centered on the body, the wearable system may determine the coordinates of the user's body posture, head posture, or eye posture so that a ring-shaped UI can be displayed around the user, or a flat UI can be displayed on a wall or in front of the user. If the UI is centered on the hand, the map coordinates of the user's hand may be determined. These mapping points may be derived with the aid of data received through a FOV camera, sensory input, or any other type of collected data.

[0105] At box 1130, the wearable system can send data from the cloud to the display, or can send data from a local database to the display component. At box 1140, the UI is displayed to the user based on the sent data. For example, a light field display can project a virtual UI into one or both eyes of the user. Once the virtual UI is created, at box 1150, the wearable system only needs to wait for commands from the user to generate more virtual content on the virtual UI. For example, the UI can be a body-centered ring around the user's body. The wearable system can then wait for a command (gesture, head or eye movement, input from a user input device, etc.), and if it is recognized (box 1160), the virtual content associated with the command can be displayed to the user (box 1170). As an example, the wearable system can wait for a user's gesture to remix multiple stream tracks.

[0106] Other examples of wearable systems, UIs, and user experiences (UXs) are described in U.S. Patent Publication Serial No. 2015 / 0016777, the entire contents of which are incorporated herein by reference.

[0107] Example objects in FOR and FOV

[0108] Fig. 12A Schematically illustrates an example of a field of view (FOR) 1200, a field of view (FOV) of a world camera 1270, a field of view of a user 1250, and a fixed field of view of a user 1290. Figure 4As shown, FOR 1200 includes a portion of the environment around the user that can be perceived by the user via the wearable system. FOR may include a solid angle of 4π radians around the wearable system 400, because the wearer can move his body, head, or eyes to perceive substantially any direction in space. In other scenarios, the wearer's movement may be more limited, so the wearer's FOR may subtend a smaller solid angle.

[0109] The field of view of world camera 1270 may include a portion of the user's FOR currently observed by outward-facing imaging system 464. Figure 4 , the field of view of world camera 1270 can include the world 470 observed by wearable system 400 at a given time. The size of the FOV of world camera 1270 can depend on the optical properties of external-facing imaging system 464. For example, external-facing imaging system 464 can include a wide-angle camera that can image 190 degrees of space around the user. In some embodiments, the FOV of world camera 1270 can be greater than or equal to the natural FOV of the user's eyes.

[0110] The FOV of user 1250 may include a portion of FOR 1200 that the user perceives at a given time. The FOV may depend on the size or optical properties of the display of the wearable device. For example, an AR display may include optics that only provide AR functionality when the user is viewing a specific portion of the display. FOV 1250 may be associated with the portion of FOR 1200 that the user perceives at a given time. Figure 4 ) or planar waveguide 600( Figure 6 )), the solid angle perceived by the user corresponds. In some embodiments, the FOV of user 1250 can be smaller than the natural FOV of the user's eyes.

[0111] The wearable system can also determine the user's fixed field of view (FOF) 1290. FOF 1290 can include a portion of FOV 1250 that the user's eyes can focus on (e.g., maintain visual focus on this portion). FOF 1290 can be smaller than the user's FOV 1250, for example, the FOF can span between a few degrees to about 5 degrees. Therefore, the user can perceive some virtual objects in FOV 1250 that are not in the fixed field of view 1290, but are in the user's peripheral FOV.

[0112] Fig. 12B Examples of virtual objects in the user's FOV 1250 and virtual objects in the FOR 1200 are schematically shown. Fig. 12B1200 may include a set of objects (e.g., 1210, 1220, 1230, 1242, and 1244) that can be perceived by a user via a wearable system. The objects in FOR 1200 may be virtual objects and / or physical objects. For example, FOR 1200 may include physical objects such as chairs, sofas, walls, and the like. Virtual objects may include operating system objects such as a recycle bin for deleted files, a terminal for inputting commands, a file manager for accessing files or directories, icons, menus, applications for audio or video streaming, notifications from an operating system, and the like. Virtual objects may also include objects in applications, such as avatars, virtual objects in games, graphics or images, and the like. Some virtual objects may be both operating system objects and objects in applications. In some embodiments, a wearable system may add virtual elements to existing physical objects. For example, a wearable system may add a virtual menu associated with a TV in a room, wherein the virtual menu may provide the user with the option of turning on or changing a TV channel using the wearable system.

[0113] A virtual object may be a three-dimensional (3D), two-dimensional (2D), or one-dimensional (1D) object. For example, a virtual object may be a 3D coffee cup (which may represent virtual controls for a physical coffee machine). A virtual object may also be a 2D graphical representation of a clock (which displays the current time to the user). In some embodiments, one or more virtual objects may be displayed within (or associated with) another virtual object. The virtual coffee cup may be displayed inside the user interface plane, although the virtual coffee cup appears to be 3D within the 2D planar virtual space.

[0114] The object in the user's FOR can be a reference Fig. 9 A portion of the world map described. Data associated with an object (e.g., location, semantic information, properties, etc.) can be stored in various data structures, such as arrays, lists, trees, hashes, graphs, etc. The index of each stored object can be determined, for example, by the location of the object when appropriate. For example, a data structure can index an object by a single coordinate such as the distance of the object from a reference position (e.g., how far away from the left or right side of the reference position, how far away from the top or bottom of the reference position, or how much depth (depth-wise) away from the reference position). The reference position can be determined based on the position of the user (such as the position of the user's head or eyes). The reference position can also be determined based on the position of a virtual or physical object (such as a target object) in the user's environment. Therefore, the 3D space in the user's environment can be folded into a 2D user interface, in which virtual objects are arranged according to the distance of the object from the reference position.

[0115] Purpose of markings

[0116] Continue to refer Fig. 12B , the VR / AR / MR system can display a virtual reticle 1256, which can include a movable indicator that identifies the user's position within the FOV 1250. For example, the reticle 1256 can represent the user's gaze direction, such as a fixed field of view, a point affected by input from the user, etc. The appearance of the reticle 1256 can present any of a variety of different colors, outlines, shapes, symbols, sizes, images, graphics, and combinations. For example, the reticle 1256 can take on a variety of shapes, such as a cursor, a geometric cone, a narrow beam, an arrow, an ellipse, a circle, a center of a circle, a polygon, or other 1D, 2D, or 3D shape. The reticle 1256 can be a virtual object fixed in the device (rig) space (e.g., a coordinate system associated with the wearable device, such as a Cartesian xyz coordinate system), and can also be fixed in the user's 3D environment. The reticle 1256 can be represented by a virtual object that the user can drag and drop (e.g., from a location in the device space) to a specific location in the user's 3D space.

[0117] The user may move his or her body, head, or eyes to move reticle 1256. For example, a change in the user's posture (e.g., head posture, body posture, or eye gaze) may cause the position of reticle 1256 to change within FOV 1250 and / or change the content shown or observable in FOV 1250. Similarly, reticle 1256 may be controlled by a user input device such as a touch pad, a touch screen, a joystick, a multi-degree-of-freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a handheld manipulator, a haptic device, a totem (e.g., used as a virtual user input device), etc. For example, reticle 1256 may move from a first position to a second position when the user moves his hand on the user input device.

[0118] The reticle 1256 may be used to select, view, or point to an object, such as one of the objects 1210, 1242, 1244, 1230, 1220, by moving the reticle 1256 so that it hovers over or otherwise points to the target object. For example, to effectively align the target object with the reticle 1256, the user may tilt, turn, or otherwise reorient his or her head to a posture corresponding to the location of the target object. Once the reticle 1256 is aligned with the target object, the user may select the target object that the reticle 1256 will hover or point to. In some embodiments, the target object may also receive a focus indicator (e.g., a virtual ray or other graphical highlight emanating from the reticle 1256 or the selected object).

[0119] Accelerate marking positioning

[0120] FIG. 13A to FIG. 13B An example of accelerating the movement of a reticle in response to a change in head posture is shown. Environments 1300A, 1300B include a user wearing a headset such as Figure 2 The system 200 includes a user 1304 of a wearable system 1320 and further includes a representation of the wearable system 1320 that can view a field of view (FOR) 1310. Figure 4 As described, FOR 1310 includes a portion of the environment surrounding the user 1304 that can be sensed by the user via the wearable system 1320 .

[0121] The user 1304 can move his or her body, head, or eyes to perceive substantially any direction in the FOR 1310. For example, the user's head can have multiple degrees of freedom. When the user's head moves (e.g., tilts or turns), the user's head pose changes. The change in head pose can be quantified by determining the change in angle relative to a reference head pose vector. The reference head pose can be any head pose of the user. For example, in considering Fig.13A and Fig. 13B When the coordinate values ​​in the xyz coordinate system shown are those of the user's head, the reference head posture vector may correspond to the position of the user's head when each of a coronal plane of the user's head (e.g., a vertical plane dividing the body into abdomen and back), a sagittal plane of the user's head (e.g., an anatomical plane dividing the body into right and left parts), and an axial plane of the user's head (e.g., a plane dividing the body into upper and lower parts, roughly perpendicular to the spine) are mutually orthogonal.

[0122] In some applications, the user may look in a specific direction more frequently, and the system can adjust the reference head posture to reduce or alleviate neck fatigue. For example, both games related to shooting birds and stargazing applications that teach users about constellations can involve users looking up periodically or continuously. Similarly, chess games can involve users looking down periodically or continuously. Other games or applications may involve users looking left, right, diagonally, etc. periodically or continuously. Therefore, in some cases, the reference head posture can be updated to offset the head posture associated with the more frequent direction. For example, the reference head posture can be the average or most common user's head posture within a specific time period, or it can be the average or most common user's head posture when the user uses a specific application or plays a specific game. By configuring the reference head posture to offset or match the head posture associated with a common or average head posture, the system can reduce or alleviate neck fatigue.

[0123] Fig.13A and Fig. 13BThe coordinate system in shows three angular degrees of freedom (e.g., yaw, pitch, and roll) that can be used to measure head pose relative to a reference head pose vector. For example, the user's head can be tilted forward and backward (e.g., pitch), rotated left and right (e.g., yaw), or tilted left and right (e.g., roll), and the angular difference between the new head pose and the reference head pose vector can be determined to quantify the change. In other embodiments, other techniques or angle representations for measuring head pose can be used, such as quaternion angles or any type of Euler angle system.

[0124] When the user's posture (e.g., body, head, or eye posture) changes, the user FOV may change accordingly, and any objects within the FOV may also change. Fig.13A Three FOVs 1350a, 1350b, 1350c are shown, which are schematically represented by rectangles and can correspond to a portion of the user's FOR 1310 observed at a particular time period. In each of the three scenes 1350a, 1350b, 1350c shown, the user can perceive a reticle 1356 via the display 1320. In addition, as shown in the FOVs 1350b, 1350c, the user can perceive a target object, such as a cylinder 1382 or a box 1380, which can represent a virtual or physical object located at a given location in the user's environment. When the user adjusts his or her head (e.g., pitch, yaw, or roll), the system can adjust the user's FOV so that the user perceives that he or she is tilting or turning his or her head in the VR / AR / MR environment.

[0125] As a non-limiting example, user 1304 may desire to view or interact with object 1380 and / or object 1382. For simplicity, the initial pose 1348 of the user's head corresponds to a reference head pose vector, as described herein. Additionally, at the initial head pose 1348, the user perceives FOV 1350a. Since no objects are perceived within FOV 1350a, the user may begin scanning FOR 1310 to search for objects 1380, 1382. As the user tilts his or her head downward toward the floor, the user's FOV adjusts accordingly so that the user may eventually perceive object 1382 within FOV 1350b. The user may continue to scan FOR 1310 to search for more objects, for example, by tilting his or her head upward toward the ceiling. At some point thereafter, the user may adjust his or her head pose so that the user perceives FOV 1350c. As shown, object 1380 may be perceptible in FOV 1350c. In some cases, in addition to simply perceiving an object, a user may also use reticle 1356 to select an object.

[0126] Various methods may be used to position the reticle 1356 over the target object. As a first example, the user's FOV may be temporarily or permanently fixed within the FOR. Thus, a change in the user's posture (e.g., head posture, body posture, eye posture) may cause the reticle to move within the user's FOV (e.g., relative to a default reticle position, which may be near the center of the FOV) but may not cause the user's FOV to change. For example, if the user looks up or tilts his or her head back, that movement may cause the reticle to move upward within the FOV, such as near the top of the FOV. Thus, in examples such as these, in order to select an object using the reticle, the object may need to be within the FOV, and the user may need to position his or her head so that the reticle is pointing toward the target object.

[0127] As a second example, the position of the reticle within the FOV can be temporarily or permanently fixed. Thus, a change in the user's posture may change the user's FOV but not change the position of the reticle within the FOV. For example, the reticle may be fixed or locked at the location (e.g., the center) of the user's FOV, such as shown in FOV 1350a. As the user's posture changes, the FOV may change, but the reticle remains at the default location (e.g., the center) of the dynamic FOV. Thus, in order to select an object using the reticle, the user may need to position his or her head so that the target object is located at the center of the FOV (e.g., the location of the reticle).

[0128] As a third example, a combination of the first two examples can be used to position the reticle on the target object. The reticle can be temporarily fixed at a position within the FOV. For example, when the user's head posture is between the minimum head posture threshold and the maximum head posture threshold (e.g., the minimum head posture threshold is met and the maximum head posture threshold is not met), the reticle can be fixed. When the user's posture changes, the user's FOV may change, while the position of the reticle within the FOV remains unchanged. However, when the user's head moves toward an uncomfortable or other undesirable posture (e.g., the user's head posture does not meet the minimum head posture threshold or meets the maximum head posture threshold), the reticle may become unfixed and can move freely within the user's FOV. For example, the wearable system can accelerate the movement of the reticle in a direction corresponding to the direction of the user's head movement. In some cases, this acceleration can reduce the possibility of neck fatigue or discomfort because the reticle moves toward the position where the user moves his or her head, thereby reducing or minimizing the amount of head movement required to position the reticle according to the target object. Although the reticle is accelerated, in some embodiments, the reticle is not accelerated beyond a threshold position (e.g., a position within the FOV). For example, in some cases, the reticle will not be accelerated outside the user's FOV. This advantageously helps reduce the likelihood of neck fatigue or discomfort, while also preserving the user's ability to interact with objects in the FOV via the reticle.

[0129] Sometimes when a user needs to reorient his or her head (non-limiting examples: to select an object high in the air (e.g., the ceiling), an object low to the ground (e.g., the floor), an object far to the right, an object far to the left, etc.), the user may have to bend, twist, or stretch his or her neck to position the reticle 1356 at the desired object. Among other negative effects, the bending, twisting, and / or stretching of the user's neck can cause neck fatigue or discomfort. Therefore, the wearable system can identify that the orientation of the user's head is outside the range of acceptable (e.g., comfortable, non-fatiguing, etc.) head postures (e.g., below a minimum head posture threshold, above a maximum head posture threshold). Therefore, in order to assist the user in moving the reticle 1356, the wearable system can accelerate the movement of the reticle 1356 in a direction corresponding to the direction of the user's head movement (e.g., adjusting the angle relative to a reference head posture vector or adjusting the position within the FOV). By accelerating the movement of the reticle 1356, the wearable system advantageously reduces the degree to which the user must bend, twist, or stretch his or her neck to align the reticle with the target object, thereby reducing the possibility of neck fatigue or discomfort.

[0130] return Fig.13A, a reticle 1356 within FOV 1350a is located at a fixed position at the center 1360 of FOV 1350a. As the user tilts his or her head toward the ceiling or floor, the wearable system can determine the user's head posture in real time. If the user's head posture begins to correspond to any of the undesirable, uncomfortable, or fatigued head posture ranges, the wearable system can begin to accelerate the reticle 1356.

[0131] For example, Fig.13A As shown, the user's real-time head posture can be compared with one or more threshold head postures (e.g., maximum head pitch threshold 1394, minimum head pitch threshold 1396) to determine whether the user's head posture corresponds to or falls within one or more of a desired or undesired head posture range. The wearable system 1320 can determine the user's head posture by calculating one or more angles corresponding to the degree to which the user bends, turns, tilts, or rotates his or her neck. For example, the angular difference between a vector corresponding to the user's current head posture and a reference head posture vector can be calculated. If one or more angles correspond to angles associated with an undesired head posture, the wearable system can accelerate the movement of the reticle 1356, such as by adding an offset to or removing an offset from the angles associated with the reticle to adjust the position of the reticle within the FOV.

[0132] Fig.13A and Fig. 13B An example of offset addition is shown. As a first example, and continuing with the example where the user's initial head pose corresponds to FOV 1350a, the user must tilt his or her head downward toward the ground to perceive FOV 1350b. When the user's head pose is at an angle that causes the user to perceive FOV 1350b, the system can determine an angle 1386 between the user's new head pose and a reference head pose (e.g., the user's initial head pose 1348). Fig.13A As shown, the user's new head posture corresponding to the center 1360 of the FOV 1350b is below (or does not meet) the minimum head pitch threshold 1396. Based on the user's new head posture, the system can determine that the user is looking down and / or the user's head or neck is in or facing an uncomfortable or other undesirable posture. Therefore, the system can accelerate the reticle 1356 in the downward direction so that the position is no longer at the default reticle position (e.g., center) of the FOV. In order to make this reticle position adjustment, the system can add an offset 1364 to the angle 1386 where the reticle is positioned relative to the reference head posture. Various methods known in the art can be used to add an offset (e.g., approximately 12 degrees) to the angle 1386. For example, the offset 1364 can be added to the angle 1386 or removed from the angle 1386 to make the new angle more negative. Similarly, the offset can increase or decrease the absolute value of the new angle.

[0133] As another example, and continuing to assume that the user's initial head pose corresponds to FOV 1350a, the user must tilt his or her head upward toward the ceiling to perceive FOV 1350c. When the user's head pose is at an angle that causes the user to perceive FOV 1350c, the system can determine an angle 1390 between the user's new head pose and the reference head pose (e.g., the user's initial head pose 1348). Fig.13A As shown, the new head pose of the user corresponding to the center 1360 of the FOV 1350c is above (or meets) the maximum head pitch threshold 1394. Based on the new head pose of the user, the system can determine that the user is looking upward and / or that the user's head or neck is in or oriented in an uncomfortable or other undesirable posture. Therefore, the system can accelerate the reticle 1356 in the upward direction so that the position is no longer at the default reticle position relative to the FOV (e.g., the center of the FOV). In order to make this reticle position adjustment, the system can add an offset 1368 to the angle 1390 where the reticle is positioned relative to the reference head pose. Various methods known in the art can be used to add an offset to the angle 1390 (e.g., approximately 8 degrees). For example, the offset 1368 can be added to the angle 1390 or removed from the angle 1390 to correct the new angle. Likewise, the offset can increase or decrease the absolute value of the new angle.

[0134] As another example, something like Fig.13A , Fig. 13B Three FOVs 1350d, 1350e, 1350f are shown, which are schematically represented by rectangles and can correspond to a portion of the user FOR 1310 observed at a particular time period. In each of the three scenes 1350d, 1350e, 1350f shown, the user can perceive a reticle 1356 via the display 1320. Initially, the reticle 1356 is located at a fixed position at the center 1360 of the FOV 1350d. As the user turns his or her head to the right, the wearable system 1320 can determine the user's head posture in real time. When the user's head posture begins to move toward any of the undesirable, uncomfortable, or fatigued head posture ranges, the wearable system begins to accelerate the reticle 1356, thereby moving the reticle away from the fixed position within the FOV.

[0135] As a non-limiting example, user 1304 may desire to view or interact with object 1384. For simplicity, the initial pose 1348 of the user's head corresponds to a reference head pose, as described herein, and the initial FOV perceived by the user is FOV 1350d. As the user's head pose changes, the head pose may be compared to one or more threshold head poses (e.g., maximum head yaw threshold 1374, easing threshold 1372, minimum head yaw threshold, etc.) to determine whether the head pose exceeds the threshold and whether the reticle position should be accelerated.

[0136] Returning to the example, the user may begin scanning FOR 1310 to search for object 1384. As the user turns his or her head to the right, the user's FOV is adjusted accordingly so that the user may eventually perceive FOV 1350e. While perceiving FOV 1350e, the user's head pose (which may correspond to center 1360 of FOV 1350e) satisfies easing threshold 1372. As a result, the system may accelerate the position of reticle 1356 so that it is positioned slightly to the right of center 1360 of FOV 1350e. For example, the system may add an offset 1376 (e.g., corresponding to an easing function) to angle 1392 corresponding to reticle 1356 in FOV 1350e.

[0137] The user may continue scanning FOR 1310 to search for object 1384, such as by turning his or her head further to the right. At some later point in time, the user may adjust his or her head pose so that the user can perceive FOV 1350f. As shown, object 1384 may be perceived in FOV 1350f, and the system may advantageously, for example, accelerate the reticle toward object 1384 to make it easier for the user to select the object. Therefore, the system may determine that the user's head pose (which may correspond to center 1360 of FOV 1350f) satisfies maximum head yaw threshold 1374. In this way, the system may accelerate the position of reticle 1356 so that it is positioned further to the right of the center of FOV 1350e than the reticle positioned in FOV 1350e. For example, the system may add a larger offset 1378 to angle 1388 corresponding to the reticle in FOV 1350f.

[0138] In some cases, to determine the amount of cursor acceleration in the horizontal direction, the system determines the posture of the user's head relative to the user's body, torso, shoulders, etc. For example, if the head posture vector is more offset relative to the body posture vector, neck fatigue is more likely to increase. Therefore, the system can accelerate the reticle at a relatively faster or slower rate depending on the alignment of the user's head, neck, shoulders, body, etc.

[0139] In some cases, an additional inertial measurement unit (IMU) configured to track the posture of the user's body, torso, or shoulders can be added, and the system can determine the angle or position of the user's head relative to the user's body. The additional IMU can include an auxiliary device located on the patient's shoulder, chest, or waist, such as a pin, necklace, backpack, belt bag, totem, etc. In some embodiments, the system can include external or user-facing sensors or cameras that can calculate body vectors, for example, using computer vision processing.

[0140] although Fig.13A and Fig. 13B The example shown corresponds to the user tilting his or her head up or down and turning his or her head to the right, but this is not limiting, and diagonal head movement (e.g., a combination of vertical and horizontal movement) can be measured and if a diagonal threshold (or a combination of vertical and horizontal thresholds) is exceeded, the reticle can be accelerated to the corner of the FOV. In addition, the user can move his or her body, head, or eyes to perceive substantially any direction in space. In some embodiments, similar to adjusting the user's FOV based on head posture changes, the system can adjust the position or location of the reticle based on eye posture changes (e.g., eye gaze). Eye posture can be detected by the interior-facing imaging system 504 (in the Figure 4 The system can use one or more thresholds to determine whether the user's eyes are likely to be fatigued, and can change the reticle position based on this determination.

[0141] Example head pose angle adjustment

[0142] Fig.14 An example of adjusting the position of the reticle based on the user's head posture is shown. Figure 2 A wearable system such as wearable system 200 can determine a user's head pose, for example, by determining a head pose vector corresponding to the user's head pose. Based on the angular difference between the head pose vector and a reference vector, the system can adjust the position of the reticle relative to the user's FOV. As described herein, the wearable system can include one or more head pose sensors, such as an IMU that can be used to determine head pose or an eye tracking camera that can be used to determine eye gaze. The wearable system can use data from such sensors to determine the poses and angles described herein.

[0143] The head posture vector can provide an indication of the user's head posture. For example, the head posture vector can show where the user is looking, how the user's head is oriented, how the user's neck is bent, etc. In some embodiments, the head posture vector can be defined as a vector extending orthogonally from the coronal plane of the user's head (e.g., a frontal plane that divides the user's head into a ventral portion and a dorsal portion). For example, the head posture plane can be a plane parallel (or substantially parallel) to the user's forehead. In some embodiments, the head posture plane can be parallel to the coronal plane and can include a line connecting the user's eyes or other facial features. In some embodiments, the head posture vector can extend orthogonally from the coronal plane (or a plane parallel to the coronal plane) from a center point on the user's head (such as the center of the user's forehead, the center of the user's eyes, the user's nose, etc.). In some embodiments, the head posture vector can extend from any other point corresponding to the user's head or neck. Therefore, when the user's head posture changes, the head posture vector also changes. As some examples, Fig.14 Various example head pose vectors 1408, 1412, 1416, 1420, 1424, 1428 are shown.

[0144] The reference vectors may be a subset of possible head pose vectors and may be used as a reference to determine the angle of tilt or rotation of the user's head. The reference vectors may be vectors corresponding to any head pose. For example, when considering Fig.14 The reference vector 1416 may be equivalent to a head pose vector having a vertical or y-component of zero (sometimes referred to as a horizontal head pose vector 1416) when the coordinate values ​​are in the xyz coordinate system shown. In some cases, the reference vector is determined by identifying a vector in a horizontal plane that is perpendicular to the plane of the display. In some embodiments, the reference vector may be a head pose vector when the user's head is in a natural resting state (e.g., as a neutral head pitch vector 1420). In some embodiments, the horizontal head pose vector corresponds to the position of the user's head when each of a coronal plane of the user's head (e.g., a vertical plane that divides the body into abdomen and back), a sagittal plane of the user's head (e.g., an anatomical plane that divides the body into left and right portions), and an axial plane of the user's head (e.g., a plane that divides the body into upper and lower portions, roughly perpendicular to the spine) are orthogonal to each other. Fig.14As shown, in some cases, the angular difference between the middle head pitch vector 1420 and the horizontal head posture vector 1416 is about -20 degrees (e.g., the middle head pitch vector 1420 is about 20 degrees below the horizontal head posture vector 1416). In some applications, the user may look in a specific direction more frequently, and the system can adjust the reference head posture to reduce or alleviate neck fatigue. For example, both games related to shooting birds and stargazing applications that teach users about constellations can involve users looking up periodically or continuously. Similarly, chess games can involve users looking down periodically or continuously. Other games or applications can involve users looking left, right, diagonally, etc. periodically or continuously. Therefore, in some cases, the reference head posture can be updated to offset the head posture associated with the more frequent direction. For example, the reference head posture can be the average or most common user's head posture over a specific time period, or it can be the average or most common user's head posture when the user uses a specific application or plays a specific game. By configuring the reference head posture to offset or match a head posture associated with a common user's head posture, the system can reduce or alleviate neck fatigue.

[0145] As described herein, a wearable system can identify a range of head positions or head orientations that can be used for one or more head posture ranges corresponding to specific uncomfortable or fatigued head postures. The boundaries of these ranges can be considered to correspond to thresholds. For example, if a user tilts or turns his or her head too far in one direction, the user's head posture may not meet (e.g., be less than) a minimum head pitch, head yaw, or head roll threshold. Similarly, if a user tilts or turns his or her head too far in another direction, the user's head posture may meet (e.g., be greater than) a maximum head pitch, head yaw, or head roll threshold. In cases such as this, head postures that are less than (or do not meet) the minimum head posture threshold and head postures that are greater than (or meet) the maximum head posture threshold may correspond to one or more ranges of head tilt or rotation that may cause neck fatigue or discomfort.

[0146] Fig.14 Three examples of adjusting the position of the reticle, or adjusting the angle of the reticle relative to a reference head pose are shown. In the first example, the user's head pose corresponds to a head pitch vector 1420, which in this example corresponds to an intermediate head pose threshold. As described herein, the intermediate head pose threshold may correspond to a natural resting state of the head, and may represent a more comfortable head pose for the user than, for example, horizontal head pose 1416. Thus, in some cases, the intermediate head pose vector may be used as a reference head pose vector instead of or in addition to the horizontal head pose vector.

[0147] Returning to the first example, there is an angular difference of -20 degrees between head pitch vector 1420 and reference head pose vector 1416, which in this case is also horizontal head pose vector 1416. Based on the determined angular difference of -20 degrees, the system determines that the user's head pose is above (e.g., meets) a minimum pitch threshold of -45 degrees. Additionally, the system determines that the angular difference of -20 degrees is below (e.g., does not meet) a maximum pitch threshold of 5 degrees. Based on these determinations, the system can determine not to adjust the position of the reticle. Thus, as seen in scene 1436, the reticle remains at a default position (e.g., center) within scene 1436.

[0148] In a second example, the user's head pose corresponds to head pose vector 1412. As shown, there is an angular difference of +5 degrees between head pose vector 1412 and reference head pose vector 1416. Based on the determined angular difference of +5 degrees, the system determines that the user's pose is equal to (e.g., meets) a maximum pitch threshold of +5 degrees. Therefore, the system adjusts the angle of the reticle by an offset, in this case 8 degrees. This adjustment can be seen in scene 1432, where the reticle position is presented toward the top of scene 1432, rather than being presented at a default position (e.g., center) of scene 1432.

[0149] In a third example, the user's head pose corresponds to head pose vector 1424. As shown, there is an angular difference of -45 degrees between head pose vector 1424 and reference head pose vector 1416. Therefore, based on the determined angular difference of -45 degrees, the system determines that the user's pose is equal to (e.g., does not meet) a minimum head pitch threshold of -45 degrees. Therefore, the system adjusts the reticle angle by an offset, in this case -12 degrees. This adjustment can be seen in scene 1440, where the reticle position is presented toward the bottom of scene 1440, rather than being presented at a default position (e.g., center) of scene 1440.

[0150] It should be noted that Fig.14The examples shown are illustrative only and should not be construed as limiting. Thus, in some embodiments, a reference vector is used rather than the horizontal head posture vector 1416 to determine the angle difference. The system may use fewer, more, or different thresholds to determine whether the head position exceeds the threshold and adjust the position of the reticle. In addition, the values ​​corresponding to the maximum head pitch threshold 1412, the minimum head pitch threshold 1424, the intermediate head pitch vector 1420, the offset corresponding to the maximum head pitch threshold 1412, and / or the offset corresponding to the minimum head pitch threshold 1424 are for illustrative purposes. In other embodiments, the maximum head pitch threshold 1412 may be in the range of 10 degrees below the horizontal head posture vector 1416 to 25 degrees above the horizontal head pitch vector 1416. The minimum head pitch threshold 1424 may be in the range of 60 degrees below the horizontal head posture vector 1416 to 30 degrees above the horizontal head posture vector 1416. The intermediate head pitch vector 1420 can be in a range from 30 degrees below the horizontal head posture vector 1416 to level with the horizontal head posture vector 1416. The offset corresponding to the maximum pitch threshold 1412 can be in a range of +1 degree to +25 degrees. The offset corresponding to the minimum pitch threshold 1424 can be in a range of -1 degree to -25 degrees. In addition, one or more thresholds can be satisfied in various ways. For example, in some cases, a value equal to a threshold satisfies the threshold, while in other cases, a value equal to a threshold does not satisfy the threshold.

[0151] Example of head pose angle adjustment for the location of the reticule

[0152] Fig.15 An example relationship between a user's head pose and a position adjustment of a reticle is shown. Graph 1500 illustrates example reticle angle adjustments and angle differences. As described herein, an angle difference (e.g., x-axis) can be defined as the difference in angle between a head pose vector and a reference vector (e.g., a horizontal head pose vector, a median head pose vector, etc.). Additionally, adjustments to the reticle angle (e.g., y-axis) can correspond to how the system changes the position of the reticle based on the determined head pose.

[0153] Graph 1500 illustrates several different possible relationships between reticle angle adjustment and angle difference (e.g., lines 1560, 1564, 1568). For example, lines 1560, 1568 illustrate gradually adjusting the reticle angle as the angle difference changes, while line 1564 illustrates not adjusting the reticle angle within a specified range (e.g., -30 degrees to -10 degrees), and then gradually adjusting the reticle angle when the angle difference exceeds the specified range (e.g., below a minimum head pose threshold, above a maximum head pose threshold).

[0154] As shown by each of relationships 1564, 1560, and 1568, if the angle difference is 5 degrees or more (e.g., the user is looking up about 5 degrees or more above the horizontal head pose vector), the system determines that the user's head pose is greater than or equal to (e.g., meets) a maximum head pitch threshold (e.g., +5 degrees). Therefore, the system adjusts the angle of the reticle by a maximum offset of 8 degrees. Additionally, if the angle difference is less than or equal to -45 degrees (e.g., the user is looking down about 45 degrees or more below the horizontal head pose vector), the system determines that the user's head pose is less than or equal to (e.g., does not meet) a minimum head pitch threshold (e.g., -45 degrees). Therefore, the system adjusts the angle of the reticle by a minimum offset of -12 degrees.

[0155] Additionally, if the angle difference is between about -45 degrees and about 5 degrees (e.g., between the minimum head pose threshold and the maximum head pose threshold), the system determines that the user's head pose is greater than (e.g., meets) the minimum head pitch threshold and less than (e.g., does not meet) the maximum head pitch threshold. Accordingly, the system adjusts the angle of the reticle by an offset, which can be determined, for example, by various linear, exponential, piecewise, or easing functions. For example, an easing function can provide a gradually increasing or decreasing reticle angle adjustment.

[0156] Example marking positioning process

[0157] Positioning or adjusting guide lines based on head posture

[0158] Fig.16 A flow chart of an example reticle position adjustment process 1600 is shown. The example process 1600 can be performed by one or more components of the wearable system 200 (e.g., the remote processing module 270 or the local processing and data module 260), alone or in combination. The display 220 of the wearable system 200 can present one or more reticles to the user, the internally facing imaging system 462 can obtain eye images for eye gaze determination, and the IMU, accelerometer, or gyroscope can determine head pose.

[0159] At block 1610, the wearable system may receive data indicating a current head pose of a user. The head pose may describe the position and orientation of the user's head. The data may include the current position and orientation of the user's head, or the movement of the user's head in 3D space. The position may be represented by a translation coordinate value (e.g., Figure 6The orientation of the head may be represented by a coordinate value in an xyz coordinate system as shown. For example, when the user's head tilts or turns, the wearable system can track and record the movement of the user's head. The orientation may be represented by a vector or an angle value relative to the natural resting state of the head. For example, a vector or angle value may represent the head tilting forward and backward (e.g., pitch), turning left and right (e.g., yaw), and tilting from left to right (e.g., roll).

[0160] The wearable system can identify a range of head positions or head orientations that together are used to specify one or more head posture ranges that can correspond to uncomfortable or tiring head postures. The boundaries of these ranges can be considered to correspond to thresholds. For example, a maximum or minimum head pitch, head yaw, or head roll threshold can correspond to one or more head tilts or turns to the left that cause neck fatigue or discomfort. Similarly, an intermediate or reference head posture threshold can correspond to a natural resting state of the head.

[0161] Head poses that fall within these ranges may correspond to head poses for which the position of the reticles may need to be adjusted. For example, the wearable system may render reticles in 3D space for the user. The reticles may be rendered in device space (this space may be represented by a coordinate system relative to the HMD. The reticles may be represented in a variety of graphical forms, including 1D, 2D, and 3D images). The reticles may correspond to the user's current position relative to the user's field of view, and may, for example, represent the user's gaze direction. The reticles may also move with the user as the user moves around. When the user's head pose changes (e.g., causing it to fall within one of the above ranges), the position of the reticles may be adjusted to prevent, alleviate, or reduce neck fatigue or discomfort.

[0162] The wearable system can track head pose using one or more sensors internal to the HMD (e.g., an IMU or an outward-facing imaging system (e.g., to track reflected images of the user's head)) or one or more sensors external to the HMD (e.g., a camera mounted on a wall in the user's room).

[0163] At block 1620, the wearable system may identify or determine a real-time head pose of the user based at least in part on the head pose data obtained from block 1610. For example, the wearable system may identify or determine a head pose vector, as described herein, that corresponds to the head pose of the user. In some cases, the head pose vector is determined using an AR software development kit (SDK).

[0164] At block 1630, the wearable system may identify, access, or determine a reference head pose for the user. For example, the reference head pose may correspond to an intermediate head pose and may be based at least in part on the head pose data obtained from block 1610. The wearable system may determine a reference head pose vector, which may correspond to an intermediate head pose (such as a head pose relative to a natural resting state of the head), as described herein. In some cases, the reference head pose vector is determined using the AR SDK. In some embodiments, the user's reference or intermediate head pose is set to a default value (e.g., at a -20 degree angle), and the wearable system determines the reference head pose by accessing the default value (e.g., by querying the AR SDK).

[0165] At block 1640, the wearable system may determine a position adjustment for a virtual reticle projected on the head mounted display based at least in part on a comparison between the current head pose and the reference head pose. For example, the system may determine a value indicating a difference between the head pose determined from block 1620 and the reference head pose determined from block 1630. For example, the position adjustment may be relative to a coordinate value (e.g., Figure 6 In some embodiments, the wearable system can determine the angular difference between the head posture vector and the reference head posture vector. For example, Fig.14 As shown, the angle difference can be used to determine an angle corresponding to the degree of tilt or rotation of the user's head relative to the reference head posture. Based at least in part on the angle difference, the wearable system can determine whether the user is looking up or down, or tilting or turning his head. In some cases, the angle difference is determined using an AR SDK.

[0166] The wearable system can determine the adjusted reticle position based at least in part on the user's head posture. For example, the wearable system can determine whether the user presents a head posture that falls within one or more of the identified head posture ranges. The wearable system can determine whether the user's head posture is in a position or orientation that results in an adjustment of the position of the reticle. For example, the wearable system can determine whether the user's head posture falls within the identified head position range and the identified head orientation range. The wearable system can make such a determination by comparing the user's head posture with a threshold value that defines the boundaries of the identified head position range (e.g., translation coordinate values), or by comparing a head orientation (e.g., an angle difference) associated with the user's head posture with a threshold value that defines the boundaries of the identified head orientation range (e.g., an angle value).

[0167] refer to Fig.14, based on determining that the user is presenting a head pose corresponding to head pose vector 1412, the wearable system can determine that the angle difference (e.g., the difference from the reference head pose vector) is 5 degrees. Therefore, by comparing the angle difference to a threshold (e.g., the intermediate head pose threshold corresponding to vector 1416), the wearable system can determine whether the position of the reticle needs to be adjusted. For example, returning to Fig.14 In the example of FIG. 14, the wearable system can determine that the angular difference of 5 degrees corresponding to head pose vector 1412 satisfies the maximum head pose threshold. The wearable system can then adjust the position of the reticle so that the head pose vector 1408 pointing to the newly adjusted reticle has an angular difference of 8 degrees. Thus, the wearable system can speed up or slow down the reticle movement based at least in part on the head pose or the change in head pose.

[0168] The various blocks described herein may be implemented in various orders, and the wearable system may implement one or more blocks simultaneously and / or in a different order as desired. Fewer, more, or different blocks may be used as part of process 1600. For example, process 1600 may include blocks for determining a position of a reticle, providing an indication that the position of the reticle is updated, and the like.

[0169] Furthermore, although process 1600 is logically associated with preventing or reducing the likelihood of neck fatigue, similar techniques can be employed to prevent or reduce the likelihood of eye fatigue. For example, the system can obtain eye gaze data, and can determine an eye gaze vector and an intermediate eye gaze vector based on the eye gaze data. The system can further determine an angular difference between the eye gaze vector and the intermediate eye gaze vector, and based on the angular difference and one or more thresholds, can determine an adjustment to at least one of a virtual reticle position or a 3D display view.

[0170] Line positioning or adjustment based on angle difference

[0171] Fig.17 A flow chart of an example reticle adjustment process 1700 is shown. The example process 1700 can be performed by one or more components of the wearable system 200 (e.g., the remote processing module 270 or the local processing and data module 260), alone or in combination. The display 220 of the wearable system 200 can present one or more reticles to the user, the internally facing imaging system 462 can obtain eye images for eye gaze determination, and the IMU, accelerometer, or gyroscope can determine head pose.

[0172] The wearable system can render a reticle in a 3D space for the user. The reticle can be rendered in a device space (this space can be represented by a coordinate system relative to the HMD. The reticle can be represented in various graphical forms, including 1D, 2D, and 3D images). The reticle can correspond to the user's current position relative to the user's field of view, and can, for example, represent the user's gaze direction. The reticle can also move with the user as the user moves around. In addition, the reticle can point to one or more objects, and the user can select the target object to which the reticle points. In some cases, the position of the reticle within the user's FOV may remain constant as the user's head moves. For example, the reticle can be located at the center of the user's field of view and remain at this center even when the user's FOV changes. Therefore, in some cases, in order to select a target object via the reticle, the user must move his or her head to adjust the FOV so that the center of the FOV (e.g., the position of the reticle) is on the target object.

[0173] However, the wearable system can advantageously adjust the position of the reticle within the user's FOV. For example, in situations where the user desires to select an object high in the air (e.g., a ceiling), an object low to the ground (e.g., a floor), an object far to the right, an object far to the left, etc., the wearable system can adjust the position of the reticle within the user's FOV to help the user position the reticle at the location of the target object without requiring the user to tilt or turn his or her head so that the center of the user's FOV is on the target object. Thus, the wearable system can implement process 1600 to adjust the position of the reticle.

[0174] At box 1710, the wearable system can identify or determine one or more head position ranges or one or more head orientation ranges, which together with the orientations are used to specify one or more head posture ranges that can correspond to uncomfortable or fatigued head postures. The boundaries of these ranges can be considered to correspond to thresholds. For example, the wearable system can identify or determine a maximum head pitch threshold and / or a minimum head pitch threshold, which can correspond to the degree or angle of the front-to-back head tilt that will result in the maximum or minimum reticle angle adjustment. In some embodiments, the wearable system can identify or determine a maximum head yaw threshold or a minimum head yaw threshold, which can correspond to the degree or angle of turning the head left and right before the user's neck is in an uncomfortable or fatigued position. Similarly, the wearable system can identify or determine a maximum head roll threshold or a minimum head roll threshold, which can correspond to the degree or angle of tilting the head left and right before the user's neck is in an uncomfortable or fatigued position. In addition, the wearable system can identify or determine an intermediate head pitch threshold, an intermediate head yaw threshold, or an intermediate head roll threshold, which can correspond to the natural resting state of the head. In some cases, these thresholds may be user input, may be determined during a calibration phase, or may be constant within the system.

[0175] At blocks 1720 and 1730, the wearable system can determine a head pose vector and a reference (or intermediate) head pose vector based at least in part on the received head pose data, as described herein.

[0176] At block 1740, the wearable system may determine an angular difference between the head pose vector and the intermediate head pose vector. Fig.14 As shown, the angle difference can be used to determine an angle corresponding to the degree of tilt or rotation of the user's head relative to the horizontal head posture vector. Based at least in part on the angle difference, the wearable system can determine whether the user is looking up or down, or tilting or turning his head.

[0177] At box 1750, the wearable system can calculate the reticle position adjustment. The reticle position adjustment (e.g., an angle adjustment or offset) can be associated with each of the thresholds to help, for example, accelerate cursor movement. For example, if the head posture meets the maximum head pitch threshold, the maximum head yaw threshold, or the maximum head roll threshold, the wearable system can adjust the angle or position of the reticle by a first predetermined amount (e.g., +8 degrees). Similarly, if the head posture does not meet the minimum head pitch threshold, the minimum head yaw threshold, or the minimum head roll threshold, the wearable system can adjust the angle or position of the reticle by a second predetermined amount (e.g., -12 degrees). In some embodiments, if the head posture meets the minimum threshold (e.g., pitch, yaw, or roll) and does not meet the maximum threshold (e.g., pitch, yaw, or roll), the wearable system can use an easing function to adjust the angle or position of the reticle. For example, if the head posture meets the minimum head pitch threshold and does not meet the intermediate head pitch threshold, the wearable system can use the following equation 1 to adjust the angle or position of the reticle:

[0178]

[0179] where VPA is the vertical pitch adjustment, maxHPA is the maximum head pitch adjustment, P is the angle between the head pose vector and the horizontal head pose vector, neutralHP is the angle corresponding to the neutral head pose vector, and ease() is an easing function, such as easeOutSine.

[0180] Similarly, if the head pose satisfies the intermediate head pitch threshold and does not satisfy the maximum head pitch threshold, the wearable system can adjust the angle or position of the reticle using the following Equation 2:

[0181]

[0182] Where VPA is the vertical angle pitch adjustment, minHPA is the minimum head pitch adjustment, P is the angle between the head pose vector and the horizontal head pose vector, neutralHP is the angle corresponding to the neutral head pose vector, and ease() is an easing function, such as easeOutSine.

[0183] The various blocks described herein may be implemented in various orders, and the wearable system may implement one or more blocks simultaneously and / or in a different order as desired. Fewer, more, or different blocks may be used as part of process 1700. For example, process 1700 may include blocks for determining a position of a reticle, providing an indication that the position of the reticle is updated, and the like.

[0184] Furthermore, although process 1700 is logically associated with preventing or reducing the likelihood of neck fatigue, similar techniques can be employed to prevent or reduce the likelihood of eye fatigue. For example, the system can obtain eye gaze data, and can determine an eye gaze vector and an intermediate eye gaze vector based on the eye gaze data. The system can further determine an angular difference between the eye gaze vector and the intermediate eye gaze vector, and based on the angular difference and one or more thresholds, can determine an adjustment to at least one of a virtual reticle position or a 3D display view.

[0185] Reticle positioning or adjustment based on head pitch threshold

[0186] Fig.18 A flow chart of an example reticle adjustment process is shown. The example process 1800 can be performed by one or more components of the wearable system 200, such as the remote processing module 270 or the local processing and data module 260, alone or in combination. The display 220 of the wearable system 200 can present one or more reticles to the user, the internally facing imaging system 462 can obtain eye images for eye gaze determination, and the IMU, accelerometer, or gyroscope can determine head pose.

[0187] At block 1810, similar to Fig.17 At block 1740 , the wearable system calculates an angular difference between the head pose vector and the intermediate head pose vector.

[0188] At block 1820, the wearable system determines whether the angle difference satisfies or does not satisfy a minimum head pitch threshold. As described herein, head pitch may correspond to a forward and backward tilt of the head. Thus, when the user strains or bends his or her neck forward (e.g., looking at the ground), the angle difference may not satisfy the minimum head pitch threshold. For example, the minimum head pitch threshold may correspond to an angle difference of approximately -30, -35, -40, -45, -50, 55, or -60 degrees (+ / - several degrees). Thus, in some cases, if the angle difference is equal to or lower than the minimum head pitch threshold, the angle difference does not satisfy the minimum head pitch threshold. However, in some cases, if the angle difference is equal to or lower than the minimum head pitch threshold, the angle difference satisfies the minimum head pitch threshold.

[0189] It should be noted that if the angle between the head pose vector and the median head pose vector (e.g., head pose vector minus median head pose vector) is positive, then the user is looking up relative to the median position. Likewise, if the angle is negative, then the user is looking down. Thus, in some cases, the system can determine whether the user is looking up or down, and can take the absolute value of the angle difference to ensure a positive angle. In some embodiments, the system can determine the angle difference (e.g., head pose vector minus median head pose vector), and it should be understood that a negative value indicates that the user is looking down.

[0190] At block 1830, the wearable system determines that the angle difference does not satisfy the minimum head pitch threshold. Therefore, the wearable system uses the angle adjustment associated with the minimum head pitch threshold to determine the adjusted reticle position. In some cases, the angle adjustment can be approximately -30 degrees, -25 degrees, -20 degrees, -15 degrees, -12 degrees, -10 degrees, -5 degrees, or -2 degrees.

[0191] At block 1840, the wearable system determines whether the angle difference satisfies or does not satisfy a maximum head pitch threshold. As described herein, head pitch may correspond to a forward and backward tilt of the head. Thus, when the user strains or bends his or her neck backward (e.g., to look at the sky), the angle difference may satisfy the maximum head pitch threshold. For example, the maximum head pitch threshold may correspond to an angle difference of approximately 2, 5, 8, 10, 12, 15, 20, 25, or 30 degrees (+ / - several degrees). Thus, in some cases, if the angle difference is equal to or higher than the maximum head pitch threshold, the angle difference satisfies the maximum head pitch threshold. However, in some cases, if the angle difference is equal to or higher than the maximum head pitch threshold, the angle difference does not satisfy the maximum head pitch threshold.

[0192] At block 1850, the wearable system determines that the angle difference satisfies the maximum head pitch threshold. Therefore, the wearable system uses the angle adjustment associated with the maximum head pitch threshold to determine the adjusted reticle position. In some cases, the angle adjustment can be approximately 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 15, or 20 degrees.

[0193] At box 1860, the wearable system determines whether the angle difference meets or does not meet the intermediate head pitch threshold. As described herein, head pitch can correspond to the front-to-back tilt of the head. The intermediate head pitch threshold can correspond to the natural resting state of the head. For example, the intermediate head pitch threshold can correspond to an angle difference of approximately -10 degrees, -15 degrees, -20 degrees, -25 degrees, or -30 degrees (+ / - several degrees). Therefore, in some cases, if the angle difference is equal to or higher than the intermediate head pitch threshold, the angle difference meets the intermediate head pitch threshold. Similarly, if the angle difference is lower than the intermediate head pitch threshold, the angle difference does not meet the intermediate head pitch threshold. However, in some cases, if the angle difference is equal to or higher than the intermediate head pitch threshold, the angle difference does not meet the intermediate head pitch threshold, and if the angle difference is lower than the intermediate head pitch threshold, the angle difference meets the intermediate head pitch threshold.

[0194] At block 1870, the wearable system determines that the angle difference satisfies the intermediate head pitch threshold. Therefore, the wearable system can use Equation 2 above to adjust the angle or position of the reticle.

[0195] At block 1880, the wearable system determines that the angle difference does not satisfy the intermediate head pitch threshold. Therefore, the wearable system can adjust the angle or position of the reticle using Equation 1 above.

[0196] The various blocks described herein can be implemented in various orders, and the wearable system can implement one or more blocks simultaneously and / or change the order as needed. Fewer, more, or different blocks can be used as part of process 1800. For example, process 1800 may include blocks for determining other thresholds, providing an indication that the position of the reticle is updated, etc.

[0197] Furthermore, although process 1800 is logically associated with preventing or reducing the likelihood of neck fatigue, similar techniques can be employed to prevent or reduce the likelihood of eye fatigue. For example, the system can obtain eye gaze data, and can determine an eye gaze vector and an intermediate eye gaze vector based on the eye gaze data. The system can further determine an angular difference between the eye gaze vector and the intermediate eye gaze vector, and based on the angular difference and one or more thresholds, can determine an adjustment to at least one of the position of the virtual reticle or the 3D display view.

[0198] Software Code Examples

[0199] Appendix A includes examples of code in the C# programming language for calculating head pose and vertical pitch adjustment. This code is implemented in the reference Fig.17 An example of a reticle adjustment process 1700 is described. The entire contents of Appendix A are hereby incorporated by reference into this document and constitute a part of this specification.

[0200] Other aspects

[0201] In a first aspect, a system comprises: a head posture sensor configured to obtain head posture data of a user of the system; a non-transitory memory configured to store the head posture data; a display configured to be located in front of the user's eyes, the display configured to project a virtual reticle to the user's eyes; a hardware processor communicating with the head posture sensor, the display and the non-transitory memory, the hardware processor being programmed to: obtain the head posture data of the user; identify the head posture of the user based at least in part on the head posture data; determine an adjustment to a position of the virtual reticle based at least in part on a comparison between the user's head posture and a reference head posture; and change the position of the virtual reticle based at least in part on the determined adjustment.

[0202] In a second aspect, the system of aspect 1, wherein the virtual reticle comprises a movable indicator identifying the position of the user within the user's field of view.

[0203] In a 3rd aspect, the system of aspect 1 or aspect 2, wherein the head posture data corresponds to at least one of an indication of yaw, pitch or roll of the user's head.

[0204] In a 4th aspect, the system of aspect 3, wherein the indication of yaw, pitch or roll is relative to the reference head pose.

[0205] In a 5th aspect, according to the system of aspect 4, wherein the reference head posture corresponds to a horizontal head posture of the user's head.

[0206] In a 6th aspect, according to the system of aspect 5, the horizontal head posture includes a head posture in which a coronal plane of the user's head, a sagittal plane of the user's head, and an axial plane of the user's head are orthogonal to each other.

[0207] In aspect 7, the system according to any one of aspects 1 to 6, wherein the reference head posture comprises a head posture corresponding to a natural resting state of the user's head.

[0208] In an 8th aspect, the system of aspect 7, wherein the natural resting state of the user's head corresponds to between -5 and 5 degrees yaw, between -5 and 5 degrees roll, and between -15 and -25 degrees pitch relative to the horizontal head posture.

[0209] In a 9th aspect, the system of aspect 8, wherein the reference head pose corresponds to at least one of 0 degrees yaw, 0 degrees roll, or -20 degrees pitch relative to the horizontal head pose.

[0210] In aspect 10, the system of any one of aspects 1 to 9, wherein the hardware processor is further programmed to identify a head posture vector corresponding to the head posture of the user, and to identify a reference head posture vector corresponding to the reference head posture.

[0211] In aspect 11, the system of aspect 10, wherein the hardware processor is further programmed to: determine an angular difference between the head posture vector and the reference head posture vector based at least in part on a comparison between the head posture of the user and the reference head posture, wherein the angular difference corresponds to at least one of a yaw difference, a pitch difference, or a roll difference of the head posture of the user relative to the reference head posture.

[0212] In a 12th aspect, the system of aspect 11, wherein to determine the adjustment to the position of the virtual guideline, the hardware processor is programmed to compare the determined angular difference to one or more head posture thresholds.

[0213] In a 13th aspect, the system of aspect 12, wherein the one or more head posture thresholds include at least one of a maximum head posture threshold or a minimum head posture threshold.

[0214] In a 14th aspect, the system of aspect 13, wherein the maximum head posture threshold corresponds to at least one of a maximum head yaw threshold, a maximum head roll threshold, or a maximum head pitch threshold.

[0215] In a 15th aspect, the system of aspect 14, wherein, relative to the reference head posture, the maximum head yaw threshold is 50 degrees, the maximum head roll threshold is 20 degrees, or the maximum head pitch threshold is 5 degrees.

[0216] In aspect 16, a system according to any one of aspects 13 to 15, wherein the minimum head posture threshold corresponds to at least one of a minimum head yaw threshold, a minimum head roll threshold, or a minimum head pitch threshold.

[0217] In a 17th aspect, the system of aspect 16, wherein the minimum head yaw threshold is -50 degrees, the minimum head roll threshold is -20 degrees, or the minimum head pitch threshold is -45 degrees relative to the reference head posture.

[0218] In aspect 18, the system of any one of aspects 13 to 17, wherein the hardware processor is further programmed to: in response to determining that the angular difference does not satisfy the minimum head posture threshold, determine an adjustment to the position of the virtual reticle based at least in part on a first adjustment value.

[0219] In a 19th aspect, the system of aspect 18, wherein the first adjustment value is approximately -12 degrees.

[0220] In aspect 20, the system of any one of aspects 13 to 19, wherein the hardware processor is further programmed to: in response to determining that the angular difference satisfies the maximum head posture threshold, determine an adjustment to the position of the virtual reticle based at least in part on a second adjustment value.

[0221] In a 21st aspect, the system of aspect 20, wherein the second adjustment value is approximately +5 degrees.

[0222] In aspect 22, the system of any one of aspects 13 to 21, wherein the hardware processor is further programmed to: in response to determining that the angular difference satisfies the minimum head posture threshold and does not satisfy the maximum head posture threshold, determine an adjustment to the position of the virtual marking based at least in part on a third adjustment value.

[0223] In aspect 23, the system of aspect 22, wherein the third adjustment value corresponds to an easing function.

[0224] In aspect 24, the system of aspect 22 or aspect 23, wherein the third adjustment value is approximately 0 degrees.

[0225] In a 25th aspect, the system of any one of aspects 1 to 24, wherein causing the virtual reticle to reposition comprises causing the virtual reticle to reposition from a default reticle position of the user's field of view.

[0226] In aspect 26, the system of aspect 25, wherein the default reticle position comprises a center of the user's field of view.

[0227] In aspect 27, the system of any one of aspects 1 to 26, wherein the head posture sensor comprises an inertial measurement unit (IMU), an accelerometer, a gyroscope, or a magnetometer.

[0228] In aspect 28, the system of any one of aspects 1 to 27, wherein the wearable system comprises a head-mounted wearable system.

[0229] In aspect 29, a method for adjusting the position of a virtual reticle that identifies a user's position within a field of view corresponding to a display of the display system, the method comprising: obtaining head posture data of a user of the display system from a head posture sensor configured to track the user's head posture; identifying a head posture vector corresponding to the user's head posture based at least in part on the head posture data; identifying a reference head posture vector corresponding to a reference head posture; determining an angular difference between the head posture vector and the reference head posture vector based at least in part on a comparison between the user's head posture and the reference head posture, wherein the angular difference corresponds to at least one of a yaw difference, a pitch difference, or a roll difference of the user's head posture relative to the reference head posture; comparing the determined angular difference with one or more head posture thresholds, wherein the one or more head posture thresholds include at least one of a maximum head posture threshold or a minimum head posture threshold; in response to determining that the angular difference does not satisfy the minimum head posture threshold, determining an adjustment to a position of a virtual reticle based at least in part on a first adjustment value, wherein the position of the virtual reticle corresponds to a position of a movable indicator that identifies the position of the user within the user's field of view; in response to determining that the angular difference satisfies the maximum head posture threshold, determining an adjustment to the position of the virtual reticle based at least in part on a second adjustment value; in response to determining that the angular difference satisfies the minimum head posture threshold and does not satisfy the maximum head posture threshold, determining an adjustment to the position of the virtual reticle based at least in part on a third adjustment value; and based on the determined adjustment, causing the position of the virtual reticle to be adjusted from a default reticle position of the user's field of view. The method may be performed under control of the display system, such as by a hardware processor programmed to perform the operations of the method.

[0230] In a 30th aspect, the method of aspect 29, wherein the virtual reticle comprises a movable indicator identifying the position of the user within the user's field of view.

[0231] In aspect 31, a method according to aspect 29 or aspect 30, wherein the head posture data corresponds to at least one of an indication of yaw, pitch, or roll of the user's head.

[0232] In a 32nd aspect, the method of aspect 31, wherein the indication of yaw, pitch, or roll of the user's head is relative to the reference head pose.

[0233] In aspect 33, the method of any one of aspects 29 to 32, wherein the reference head posture of the user corresponds to a horizontal head posture of the user's head.

[0234] In aspect 34, the method according to aspect 33, wherein the horizontal head posture includes a head posture in which a coronal plane of the user's head, a sagittal plane of the user's head, and an axial plane of the user's head are orthogonal to each other.

[0235] In aspect 35, the method of any one of aspects 29 to 34, wherein the reference head posture comprises a head posture corresponding to a natural resting state of the user's head.

[0236] In aspect 36, the method of aspect 35, wherein the natural resting state of the user's head corresponds to a yaw of between approximately -5 and 5 degrees, a roll of between approximately -5 and 5 degrees, or a pitch of between approximately -15 and -25 degrees relative to the user's horizontal head.

[0237] In aspect 37, the method of aspect 35 or aspect 36, wherein the reference head pose corresponds to at least one of 0 degrees yaw, 0 degrees roll, or -20 degrees pitch relative to the horizontal head pose.

[0238] In aspect 38, the method of any one of aspects 29 to 37, wherein the maximum head posture threshold corresponds to at least one of a maximum head yaw threshold, a maximum head roll threshold, or a maximum head pitch threshold.

[0239] In a 39th aspect, a method according to any one of aspects 29 to 38, wherein, relative to the reference head posture, the maximum head yaw threshold is approximately 50 degrees, the maximum head roll threshold is approximately 20 degrees, and the maximum head pitch threshold is approximately 5 degrees.

[0240] In aspect 40, the method of any one of aspects 29 to 39, wherein the minimum head posture threshold corresponds to at least one of a minimum head yaw threshold, a minimum head roll threshold, or a minimum head pitch threshold.

[0241] In a 41st aspect, a method according to any one of aspects 29 to 40, wherein, relative to the reference head posture, the minimum head yaw threshold is approximately -50 degrees, the minimum head roll threshold is approximately -20 degrees, and the minimum head pitch threshold is approximately -45 degrees.

[0242] In aspect 42, the method of any one of aspects 29 to 41, wherein the first adjustment value is approximately -12 degrees.

[0243] In a 43rd aspect, the method of any one of aspects 29 to 42, wherein the second adjustment value is approximately +5 degrees.

[0244] In aspect 44, the method according to any one of aspects 29 to 43, wherein the third adjustment value corresponds to an easing function.

[0245] In aspect 45, the method of any one of aspects 29 to 44, wherein the third adjustment value is approximately 0 degrees.

[0246] In aspect 46, the method of any one of aspects 29 to 45, wherein the default reticle position comprises a center of the user's field of view.

[0247] In aspect 47, the method of any one of aspects 29 to 46, wherein the head posture sensor comprises an inertial measurement unit (IMU), an accelerometer, a gyroscope, or a magnetometer.

[0248] In aspect 48, a method for adjusting the position of a virtual reticle that identifies a user's position within a field of view corresponding to a display of a display system, the method comprising: obtaining head posture data of a user of the display system; identifying a head posture based at least in part on the head posture data; identifying a reference head posture; and determining an adjustment to the position of a virtual reticle projected on a head-mounted display based at least in part on a comparison between the head posture and the reference head posture.

[0249] In a 49th aspect, the method of aspect 48 further comprises obtaining the head pose data of the user of the display system from a head pose sensor configured to track the head pose of the user.

[0250] In aspect 50, the method according to aspect 48 or aspect 49 further includes: identifying a head posture vector corresponding to the head posture of the user; and identifying a reference head posture vector corresponding to the reference head posture.

[0251] In aspect 51, the method according to any one of aspect 50 further includes: determining the angular difference between the head posture vector and the reference head posture vector based at least in part on the comparison between the head posture and the reference head posture, wherein the angular difference corresponds to at least one of the yaw difference, pitch difference or roll difference of the user's head posture relative to the reference head posture.

[0252] In aspect 52, the method of any one of aspects 48 to 51, wherein the determining of the adjustment to the position of the virtual guideline is further based on a comparison of the angular difference to one or more head posture thresholds.

[0253] In a 53rd aspect, the method of aspect 52, wherein the one or more head posture thresholds include at least one of a maximum head posture threshold or a minimum head posture threshold.

[0254] In aspect 54, a method for adjusting the position of a movable indicator that identifies a user's position within a field of view of the user corresponding to a display of a display system, the method comprising: identifying at least one of a maximum head pitch threshold, a minimum head pitch threshold, or an intermediate head pitch threshold; identifying a head posture vector corresponding to the user's head posture; identifying a reference head posture vector; calculating an angular difference between the head posture vector and the reference head posture vector; calculating a reticle adjustment based at least in part on the angular difference and at least one of the maximum head pitch threshold, the minimum head pitch threshold, or the intermediate head pitch threshold; and determining an adjusted reticle position based at least in part on the calculated reticle adjustment.

[0255] In a 55th aspect, the method of aspect 54 further comprises obtaining head pose data of a user of the display system from a head pose sensor configured to track the head pose of the user.

[0256] In aspect 56, a method according to aspect 54 or aspect 55, wherein the determination of the angular difference is based at least in part on a comparison between the head posture vector and the reference head posture vector, wherein the angular difference is related to at least one of a yaw difference, a pitch difference, or a roll difference of the head posture of the user relative to the reference head posture.

[0257] In aspect 57, the method of any one of aspects 54 to 56, wherein the determining the adjusted reticle position is further based on a comparison of the angular difference with one or more head posture thresholds.

[0258] In aspect 58, the method of aspect 57, wherein the one or more head posture thresholds include at least one of a maximum head posture threshold or a minimum head posture threshold.

[0259] In aspect 59, a method for adjusting the position of a virtual reticle that identifies a position of a user within a field of view corresponding to a display of a display system, the method comprising: calculating an angular difference between a head posture vector and a reference head posture vector, wherein the head posture vector corresponds to a head posture of a user of the display system, wherein the reference head posture vector corresponds to a reference head posture; determining that the angular difference does not satisfy a minimum head pitch threshold; and adjusting the position of the adjusted reticle based at least in part on an angle associated with the minimum head pitch threshold.

[0260] In aspect 60, the method according to aspect 59 further includes: obtaining head posture data of the user of the display system from a head posture sensor configured to track the head posture of the user; and identifying the head posture vector corresponding to the head posture of the user based at least in part on the head posture data.

[0261] In aspect 61, a method according to aspect 59 or aspect 60, wherein the calculation of the angular difference is based at least in part on a comparison between the head posture vector and the reference head posture vector, wherein the angular difference corresponds to at least one of a yaw difference, a pitch difference, or a roll difference of the user's head posture relative to the reference head posture.

[0262] In aspect 62, a method for adjusting the position of a virtual reticle that identifies a user's position within a field of view corresponding to a display of a display system, the method comprising: calculating an angular difference between a head posture vector and a reference head posture vector, wherein the head posture vector corresponds to a head posture of a user of the display system, wherein the reference head posture vector corresponds to a reference head posture; determining that the angular difference satisfies a minimum head pitch threshold; determining that the angular difference does not satisfy a maximum head pitch threshold; and determining an adjusted reticle position based at least in part on the angular adjustment associated with the maximum head pitch threshold.

[0263] In aspect 63, the method according to aspect 62 further includes: obtaining head posture data of the user of the display system from a head posture sensor configured to track the head posture of the user; identifying the head posture vector corresponding to the head posture of the user based at least in part on the head posture data; and identifying the reference head posture vector corresponding to the reference head posture.

[0264] In aspect 64, a method according to aspect 62 or aspect 63, wherein the calculation of the angular difference is based at least in part on a comparison between the head posture vector and the reference head posture vector, wherein the angular difference corresponds to at least one of a yaw difference, a pitch difference, or a roll difference of the user's head posture relative to the reference head posture.

[0265] In aspect 65, a method for adjusting the position of a virtual reticle that identifies a user's position within a field of view corresponding to a display of a display system, the method comprising: calculating an angular difference between a head posture vector and a reference head posture vector, wherein the head posture vector corresponds to a head posture of a user of the display system, wherein the reference head posture vector corresponds to a reference head posture; determining that the angular difference satisfies a minimum head pitch threshold; determining that the angular difference does not satisfy an intermediate head pitch threshold; and determining the adjusted reticle position based at least in part on an easing function.

[0266] In aspect 66, the method according to aspect 65 further includes: obtaining head posture data of the user of the display system from a head posture sensor configured to track the head posture of the user; identifying the head posture vector corresponding to the head posture of the user based at least in part on the head posture data; and identifying the reference head posture vector corresponding to the reference head posture.

[0267] In aspect 67, a method according to aspect 65, wherein the calculation of the angular difference is based at least in part on a comparison between the head posture vector and the reference head posture vector, wherein the angular difference corresponds to at least one of a yaw difference, a pitch difference, or a roll difference of the user's head posture relative to the reference head posture.

[0268] In aspect 68, a method for adjusting the position of a virtual reticle that identifies a user's position within a field of view corresponding to a display of a display system, the method comprising: calculating an angular difference between a head posture vector and a reference head posture vector, wherein the head posture vector corresponds to a head posture of a user of the display system, wherein the reference head posture vector corresponds to a reference head posture; determining that the angular difference satisfies an intermediate head pitch threshold; determining that the angular difference does not satisfy a maximum head pitch threshold; and determining an adjusted reticle position based at least in part on an easing function.

[0269] In aspect 69, the method according to aspect 68 further includes: obtaining head posture data of the user of the display system from a head posture sensor configured to track the head posture of the user; identifying the head posture vector corresponding to the head posture of the user based at least in part on the head posture data; and identifying the reference head posture vector corresponding to the reference head posture.

[0270] In aspect 70, the method of aspect 68 or aspect 69, wherein the calculating the angular difference is based at least in part on a comparison between the head pose vector and the reference head pose vector, wherein the angular difference corresponds to at least one of a yaw difference, a pitch difference, or a roll difference of the user's head pose relative to the reference head pose. Any of the methods of any of the above aspects may be performed under the control of a hardware processor, such as a hardware processor associated with a head mounted display system.

[0271] In aspect 71, a system comprises: a head posture sensor configured to measure head posture data of a user of the system; a non-transitory memory configured to store head posture data corresponding to at least one of an indication of yaw, pitch, or roll of the user's head; a display configured to be located in front of the user's eyes, the display configured to project a virtual reticle to the user's eyes, wherein the virtual reticle includes a movable indicator that identifies the user's position within a field of view; a hardware processor that communicates with the head posture sensor, the display, and the non-transitory memory, the hardware processor being programmed to: obtain the user's head posture data; identify a head posture vector corresponding to the user's head posture based at least in part on the head posture data; identify a reference head posture vector corresponding to a reference head posture; determine a correspondence between the head posture vector and the reference head posture based at least in part on a comparison between the user's head posture and the reference head posture determining an angle difference between two head posture vectors of the user's head posture relative to the reference head posture, wherein the angle difference corresponds to at least one of a yaw difference, a pitch difference, or a roll difference of the user's head posture relative to the reference head posture; comparing the determined angle difference with one or more head posture thresholds, wherein the one or more head posture thresholds include at least one of a maximum head posture threshold or a minimum head posture threshold; in response to determining that the angle difference does not satisfy the minimum head posture threshold, determining an adjustment to the position of the virtual reticle based at least in part on a first adjustment value; in response to determining that the angle difference satisfies the maximum head posture threshold, determining an adjustment to the position of the virtual reticle based at least in part on a second adjustment value; in response to determining that the angle difference satisfies the minimum head posture threshold and does not satisfy the maximum head posture threshold, determining an adjustment to the position of the virtual reticle based at least in part on a third adjustment value; and adjusting the position of the virtual reticle from a default reticle position of the user's field of view based on the determined adjustment.

[0272] Other Considerations

[0273] Each of the processes, methods, and algorithms described herein and / or illustrated in the accompanying drawings may be embodied in and fully or partially automated by: code modules executed by one or more physical computing systems, hardware computer processors, dedicated circuits; and / or electronic hardware configured to execute specific and particular computer instructions. For example, a computing system may include a general-purpose computer (e.g., a server) or a dedicated computer, dedicated circuit, etc., programmed with specific computer instructions. Code modules may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpretive programming language. In some embodiments, specific operations and methods may be performed by circuits dedicated to a given function.

[0274] Furthermore, certain implementations of the functionality of the present disclosure are sufficiently mathematically, computationally, or technically complex that dedicated hardware or one or more physical computing devices (with appropriate dedicated executable instructions) may be necessary to perform the functionality (e.g., due to the volume or complexity of the computations involved or to provide results in substantially real time). For example, a video may include many frames, each with millions of pixels, and specially programmed computer hardware may be required to process the video data to provide the desired image processing task or application in a commercially reasonable time.

[0275] Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as physical computer storage devices, including hard drives, solid-state memory, random access memory (RAM), read-only memory (ROM), optical disks, volatile or non-volatile memory, and combinations of the same and / or similar elements. Methods and modules (or data) may also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media, including wireless-based media and wire / cable-based media, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital data packets or frames). The results of the disclosed processes or processing steps may be stored persistently or otherwise in any type of non-transitory physical computer memory, or may be transmitted via a computer-readable transmission medium.

[0276] Any process, frame, state, step or function in the flowchart described herein and / or shown in the drawings should be understood as potentially representing a code module, code segment or code portion, which includes one or more executable instructions for implementing a specific function (e.g., logic or arithmetic function) or step in the process. Various processes, frames, states, steps or functions can perform the following operations together with the illustrative examples provided herein: combination, rearrangement, addition, deletion, modification or other changes. In some embodiments, additional or different computing systems or code modules can perform part or all of the functions described herein. The methods and processes described herein are also not limited to any specific order, and the frames, steps or states associated therewith can be performed in other appropriate sequences, such as serial, parallel or in some other way. Tasks or events can be added to or removed from the disclosed example embodiments. In addition, the separation of various system components in the embodiments described herein is for illustrative purposes and should not be understood as requiring such separation in all embodiments. It should be understood that the described program components, methods and systems can generally be integrated together in a single computer product or packaged into multiple computer products. Many variants of the embodiments are feasible.

[0277] The processes, methods and systems can be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowd-sourced computing networks, the Internet, and the World Wide Web. The network can be a wired or wireless network, or any other type of communication network.

[0278] The systems and methods of the present disclosure each have several innovative aspects, which cannot independently constitute the desired attributes disclosed herein. The various features and processes described above can be used independently of each other, or can be used in combination in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. Various modifications to the embodiments described in the present disclosure are obvious to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the claims are not intended to be limited to the embodiments shown herein, but should be given the widest scope consistent with the present disclosure, the principles disclosed herein, and the novel features.

[0279] Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually in multiple embodiments, or in any suitable sub-combination. In addition, although features may function in certain combinations as described above, and may even be initially claimed in this manner, one or more features in a claimed combination may be excluded from the combination in certain circumstances, and a claimed combination may involve sub-combinations or variations of sub-combinations. No single feature or group of features is essential or indispensable for each embodiment.

[0280] Unless otherwise specifically stated, or otherwise understood in the context of use, conditional terms such as "may", "can", "possibly", "can", "for example", etc. used herein are generally intended to express that certain embodiments include, while other embodiments do not include certain features, elements and / or steps. Therefore, such conditional terms are generally not intended to imply that features, elements and / or steps are necessary for one or more embodiments in any way, nor are they intended to imply that, with or without author input or prompting, one or more embodiments must include logic for determining whether to include these features, elements and / or steps or whether to perform these features, elements and / or steps in any particular embodiment. The terms "include", "comprise", "have", etc. are synonymous and are used inclusively in an open manner and do not exclude other elements, features, actions, operations, etc. In addition, the term "or" has an inclusive meaning (rather than an exclusive meaning) when used, so when used, for example, to connect a list of elements, the term "or" means one, some or all of the list elements. In addition, unless otherwise stated, the articles "one", "an" and "said" used in this application and the appended claims should be interpreted as meaning "one or more" or "at least one".

[0281] As used herein, phrases referring to "at least one" of a list of items refer to any combination of those items, including single members. As an example, "at least one of A, B, or C" is intended to cover: A, B, C, A and B, A and C, B and C, and A, B and C. Conjunctions such as the phrase "at least one of X, Y, and Z" are understood by common context, unless otherwise specifically stated, to convey that an item, term, etc. may be at least one of X, Y, and Z. Thus, such conjunctions are generally not intended to imply that certain embodiments require that each of at least one X, at least one Y, and at least one Z be present.

[0282] Similarly, although the operations are shown in the drawings as taking a specific order, it should be appreciated that these operations do not need to be performed in the specific order shown or in a sequential order, or all the illustrated operations need to be performed to achieve the desired result. In addition, the drawings may schematically illustrate one or more example processes in the form of a flow chart. However, other operations not shown may be incorporated into the schematically illustrated example methods and processes. For example, one or more additional operations may be performed before, after, at the same time, or during any of the operations shown. In addition, in other embodiments, the operations may be rearranged or sorted. In some cases, multitasking and parallel processing are advantageous. In addition, the separation of various system components described in the above embodiments should not be understood as requiring such separation in all embodiments, and it is necessary to understand that the described program components and systems can generally be integrated together in a single software product or encapsulated in multiple software products. In addition, other embodiments are within the scope of the following claims. In some cases, the actions listed in the claims may be performed in different orders and still achieve the desired results.

[0283] APPENDIX A

[0284] [Header("Pitches limits relative to horizontal")]

[0285] public float maxHeadPitch=5;

[0286] public float neutralHeadPitch=-20;

[0287] public float minHeadPitch=-45;

[0288] [Header("Angle offsets at the min / max head pitch")]

[0289] public float maxHeadPitchDelta=8;

[0290] public float minHeadPitchDelta=-12;

[0291] public Vector3 GetAcceleratedForwardVector(){

[0292] / / Pitch limits relative to horizontal

[0293] float maxHeadPitch=5;

[0294] float neutralHeadPitch=-20;

[0295] float minHeadPitch=-45;

[0296] / / Angle offsets at the min / max head pitch

[0297] float maxHeadPitchDelta=8;

[0298] float minHeadPitchDelta=-12;

[0299] / / Get the camera / headpose forward vector

[0300] Vector3 forward=Headpose.Forward;

[0301] / / Get the forward vector projected on to the horizontal plane

[0302] Vector3 flat=forward;

[0303] flat.y=0;

[0304] flat.Normalize();

[0305] / / Calculate the absolute angle between the headpose forward vectorand the flattened forward vector

[0306] float pitch=Mathf.Acos(Vector3.Dot(forward,flat))*Mathf.Rad2Deg;

[0307] / / If the forward vector is pointing downwards,then the angle isnegative

[0308] if(forward.y<0)

[0309] {

[0310] pitch=-pitch;

[0311] }

[0312] / / pitchAdjustment is the angular deviation from the forward vector

[0313] float pitchAdjustment=0;

[0314] if(pitch>=maxHeadPitch)

[0315] {

[0316] / / If pitch>maxHeadPitch,set adjustment to maxHeadPitchDelta

[0317] pitchAdjustment=maxHeadPitchDelta;

[0318] }

[0319] else if(pitch>=neutralHeadPitch)

[0320] {

[0321] / / If pitch is between neutralHeadPitch and maxHeadPitch,setadjustment to be an eased value between 0 and ma

[0322] xHeadPitchDelta

[0323] float t=(pitch-neutralHeadPitch) / (maxHeadPitch-neutralHeadPitch);

[0324] t=EaseOutSine(t);

[0325] pitchAdjustment=maxHeadPitchDelta*t;

[0326] }

[0327] else if(pitch>=minHeadPitch)

[0328] {

[0329] / / If pitch is between neutralHeadPitch and minHeadPitch,setadjustment to be an eased value between 0 and min

[0330] HeadPitchDelta

[0331] float t=(pitch-minHeadPitch) / (neutralHeadPitch-minHeadPitch);

[0332] t=EaseOutSine(1-t);

[0333] pitchAdjustment=minHeadPitchDelta*t;

[0334] }

[0335] else

[0336] {

[0337] / / If pitch<minHeadPitch,set adjustment to minHeadPitchDelta

[0338] pitchAdjustment=minHeadPitchDelta;

[0339] }

[0340] / / If the pitch adjustment is not equal to zero

[0341] if(pitchAdjustment!=0)

[0342] {

[0343] / / Calculate the vector perpendicular to the up and forward vectors

[0344] Vector3 right=Vector3.Cross(Vector3.up,flat);

[0345] / / Calculate the rotation adjustment

[0346] Quaternion adjust=Quaternion.AngleAxis(pitchAdjustment,-right);

[0347] / / Apply the adjustment to the forward vector to get a new adjustedforward vector

[0348] forward=adjust*forward;

[0349] }

[0350] return forward;

[0351] }

[0352] INC.All rights reserved.

Claims

1. A wearable system comprising: a head posture sensor configured to obtain head posture data of a user of the wearable system; a non-transitory memory configured to store the head posture data; a display configured to be located in front of a user's eyes, the display configured to project a virtual reticle to the user's eyes within the user's field of view, wherein when the user's head posture changes, the user's field of view changes accordingly; a hardware processor in communication with the head pose sensor, the display, and the non-transitory memory, the hardware processor being programmed to: Obtaining the head posture data of the user; identifying a head pose of the user based at least in part on the head pose data; determining a difference between the head pose of the user and a reference head pose; When the difference does not satisfy a head posture threshold, causing the display to update the user's field of view in response to the change in the head posture and maintain the virtual reticle at a default position within the user's field of view; and When the difference satisfies the head posture threshold, the display is caused to move the virtual reticle away from the default position within the user's field of view, thereby rendering the virtual reticle at a position different from the default position within the user's field of view based on the user's head posture to reduce the possibility of neck fatigue or discomfort while still allowing the user to interact with objects toward the edge of the user's field of view.

2. The wearable system according to claim 1, wherein: The virtual reticle includes a movable indicator that identifies the user's position within the user's field of view.

3. The wearable system according to claim 1, wherein: The head pose data corresponds to at least one of an indication of yaw, pitch, or roll of the user's head.

4. The wearable system according to claim 1, wherein: The reference head pose corresponds to at least one of 0 degrees yaw, 0 degrees roll, or -20 degrees pitch relative to a horizontal head pose of the user.

5. The wearable system according to claim 4, wherein: The horizontal head posture includes a head posture in which a coronal plane of the user's head, a sagittal plane of the user's head, and an axial plane of the user's head are orthogonal to each other.

6. The wearable system according to claim 4, wherein: The reference head pose comprises a head pose corresponding to a natural resting state of the user's head.

7. The wearable system according to claim 6, wherein: The natural resting state of the user's head corresponds to between -5 and 5 degrees yaw, between -5 and 5 degrees roll, and between -15 and -25 degrees pitch relative to the horizontal head posture.

8. The wearable system according to claim 1, wherein: The hardware processor is further programmed to identify a head pose vector corresponding to the head pose of the user and to identify a reference head pose vector corresponding to the reference head pose.

9. The wearable system according to claim 8, wherein: The hardware processor is further programmed to determine an angular difference between the head posture vector and the reference head posture vector, wherein the angular difference corresponds to at least one of a yaw difference, a pitch difference, or a roll difference of the head posture of the user relative to the reference head posture.

10. The wearable system according to claim 9, wherein: The hardware processor is further programmed to determine an adjustment to the position of the virtual reticle by comparing the determined angular difference to the reference head pose.

11. The wearable system according to claim 10, wherein: The reference head pose includes at least one of a maximum head pose threshold or a minimum head pose threshold.

12. The wearable system according to claim 11, wherein: The maximum head posture threshold corresponds to at least one of a maximum head yaw threshold, a maximum head roll threshold, or a maximum head pitch threshold.

13. The wearable system according to claim 12, wherein: Relative to the reference head posture, the maximum head yaw threshold is 50 degrees, the maximum head roll threshold is 20 degrees, or the maximum head pitch threshold is 5 degrees.

14. The wearable system according to claim 11, wherein: The minimum head posture threshold corresponds to at least one of a minimum head yaw threshold, a minimum head roll threshold, or a minimum head pitch threshold.

15. The wearable system according to claim 14, wherein: Relative to the reference head posture, the minimum head yaw threshold is -50 degrees, the minimum head roll threshold is -20 degrees, or the minimum head pitch threshold is -45 degrees.

16. The wearable system according to claim 11, wherein: The hardware processor is further programmed to: In response to determining that the angular difference does not satisfy the minimum head posture threshold, an adjustment to the position of the virtual reticle is determined based at least in part on a first adjustment value.

17. The wearable system according to claim 16, wherein: The first adjustment value is approximately -12 degrees.

18. The wearable system according to claim 11, wherein: The hardware processor is further programmed to, in response to determining that the angular difference satisfies the maximum head pose threshold, determine an adjustment to the position of the virtual reticle based at least in part on a second adjustment value.

19. The wearable system according to claim 18, wherein: The second adjustment value is approximately +5 degrees.

20. The wearable system according to claim 11, wherein: The hardware processor is further programmed to: In response to determining that the angular difference satisfies the minimum head pose threshold and does not satisfy the maximum head pose threshold, an adjustment to the position of the virtual reticle is determined based at least in part on a third adjustment value.

21. The wearable system according to claim 20, wherein: The third adjustment value corresponds to the easing function.

22. The wearable system according to claim 20, wherein: The third adjustment value is approximately 0 degrees.

23. The wearable system of claim 1, wherein: The default reticle position includes the center of the user's field of view.

24. The wearable system of claim 1, wherein: The head posture sensor includes an inertial measurement unit (IMU), an accelerometer, a gyroscope, or a magnetometer.

25. The wearable system of claim 1, wherein: The wearable system includes a head-mounted wearable system.

Images