Display system and method for determining alignment between a display and a user's eyes - Patents.com

Abstract

To provide suitable display systems and methods for determining alignment between a display and eyes of a user.SOLUTION: A wearable device may include a head-mounted display (HMD) for rendering a three-dimensional (3D) virtual object which appears to be located in an ambient environment of a user of the display. The relative positions of the HMD and one or more eyes of the user may not be in desired positions to receive, or register, image information outputted by the HMD. For example, the HMD-to-eye alignment vary for different users and may change over time (e.g., as a given user moves around or as the HMD slips or otherwise becomes displaced). The wearable device may determine a relative position or alignment between the HMD and the user's eyes by determining whether features of the eye are at certain vertical positions relative to the HMD.SELECTED DRAWING: Figure 20

Description

Technical Field

[0001] (Claim of Priority) This application claims the priority of U.S. Provisional Patent Application No. 62 / 702,866, filed on Jul. 24, 2019, entitled "DISPLAY SYSTEMS AND METHODS FOR DETERMINING REGISTRATION BETWEEN A DISPLAY AND EYES OF A USER", which is hereby incorporated by reference in its entirety.

[0002] (Incorporation by Reference) This application incorporates by reference in its entirety each of the following patent applications and publications: U.S. Patent Application No. 14 / 555,585, filed November 27, 2014 and published as U.S. Patent Publication No. 2015 / 0205126 on July 23, 2015; U.S. Patent Application No. 14 / 690,401, filed April 18, 2015 and published as U.S. Patent Publication No. 2015 / 0302652 on October 22, 2015; U.S. Patent Application No. 14 / 212,961, filed March 14, 2014 and issued as U.S. Patent No. 9,417,452 on August 16, 2016; U.S. Patent Application No. 14 / 331,218, filed July 14, 2014 and published as U.S. Patent Publication No. 2015 / 0309263 on October 29, 2015; U.S. Patent Publication No. 2016 / 0270656; U.S. Patent Publication No. 2015 / 0178939, published June 25, 2015; U.S. Patent Publication No. 2015 / 0016777; U.S. Patent Application No. 15 / 274,823; U.S. Patent Application No. 15 / 296,869; U.S. Patent Application No. 15 / 717,747, filed September 27, 2017; U.S. Patent Application No. 15 / 497,726, filed April 26, 2017; U.S. Patent Publication No. 2017 / 0053165, published February 23, 2017; U.S. Patent Publication No. 2017 / 0053166, published February 23, 2017; U.S. Patent Application No. 15 / 341,760, filed November 2, 2016 and published as U.S. Patent Publication No. 2017 / 0122725 on May 4, 2017; U.S. Patent Application No. 15 / 341,822, filed November 2, 2016 and published as U.S. Patent Publication No. 2017 / 0124928 on May 4, 2017; U.S. Provisional Patent Application No. 62 / 618,559, filed January 17, 2018; U.S. Provisional Patent Application No. 62 / 642,761, filed March 14, 2018; and U.S. Provisional Patent Application No. 62 / 644,321, filed March 16, 2018.

[0003] The present disclosure relates to a display system, including virtual reality and augmented reality display systems, and more particularly, to a system and method for evaluating the fit of a display on a user.

Background Art

[0004] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality", "augmented reality", or "mixed reality" experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner that appears or is perceived to be real. Virtual reality, i.e., the "VR" scenario, typically involves the presentation of digital or virtual image information without transparency to other actual real-world visual inputs. Augmented reality, i.e., the "AR" scenario, typically involves the presentation of digital or virtual image information as an augmentation to the visualization of the actual world around the user. Mixed reality or "MR" relates to the fusion of the real and virtual worlds to create a new environment in which physical and virtual objects coexist and interact in real time. In summary, the human visual perception system is very complex, and the production of VR, AR, or MR technologies that facilitate a comfortable, natural, and rich presentation of virtual image elements among other virtual or real-world image elements is difficult. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies.

Summary of the Invention

Means for Solving the Problems

[0005] In some embodiments, a display system is configured to project light into a user's eyes and display virtual image content. The display system includes a frame configured to be supported on the user's head, a head-mounted display disposed on the frame, one or more eye-tracking cameras configured to image the user's eyes, and processing electronics that communicate with the head-mounted display and the one or more eye-tracking cameras. The display is configured to project light into the user's eyes and display virtual image content with different amounts of wavefront divergence, presenting virtual image content that appears to be located at different depths at different time periods. The processing electronics are configured to determine whether the head-mounted display is properly aligned with the user's eyes by determining whether the imaged features of the eyes are within a predetermined range of vertical positions relative to the head-mounted display, and to provide feedback to the user if the head-mounted display is not properly adjusted to fit the user.

[0006] In some other embodiments, a method is provided for evaluating the alignment of virtual image content from a head-mounted display system by the user's eyes. The method includes imaging the eyes, determining whether the imaged features of the eyes are within a predetermined range of vertical positions relative to the head-mounted display, and providing a notification based on the determined position of the imaged features. The notification indicates at least that the head-mounted display and the eyes are not properly aligned.

[0007] Additional examples of embodiments are listed below.

[0008] (Example 1) A display system configured to project light into a user's eyes and display virtual image content, a frame configured to be supported on the user's head, and A head-mounted display disposed on a frame, which projects light into the user's eyes, displays virtual image content with different amounts of wavefront divergence, and is configured to present virtual image content that appears to be located at different depths at different time periods, a display; One or more eye-tracking cameras configured to image the user's eyes; A processing electronic device that communicates with the head-mounted display and the one or more eye-tracking cameras, By determining whether the imaged feature of the eye is within a predetermined range of the vertical position relative to the head-mounted display, determine whether the head-mounted display is properly aligned with the user's eyes, If the head-mounted display is not properly adjusted to fit the user, provide feedback to the user. A processing electronic device configured as such; A display system comprising.

[0009] (Example 2) The one or more eye-tracking cameras are configured to image the user's left eye and the user's right eye, The processing electronic device is further configured to determine a left-eye tracking reliability score indicating the reliability level of the position of the user's left eye, and determine a right-eye tracking reliability score indicating the reliability level of the position of the user's right eye. When one of the reliability scores exceeds the other, the processing electronic device is further configured to determine whether the head-mounted display is properly aligned based on the user's left eye or right eye associated with the larger reliability score. The display system according to Example 1.

[0010] (Example 3) The one or more eye-tracking cameras are configured to image the user's left eye and the user's right eye, When the user's left and right eyes are vertically offset from each other by less than a first predetermined threshold, the processing electronic device is further configured to determine whether the head-mounted display is properly aligned based on the positions of the user's left and right eyes that are farthest from the desired vertical position. The display system according to Example 1.

[0011] (Example 4) When the user's left and right eyes are vertically offset from each other by less than a second predetermined threshold that is less than the first predetermined threshold, the processing electronic device is further configured to determine whether the head-mounted display is properly aligned based on the average position of the user's left and right eyes. The display system according to Example 3.

[0012] (Example 5) When the user's left and right eyes are vertically offset from each other by more than the first predetermined threshold, the processing electronic device is further configured to determine whether the head-mounted display is properly aligned based on the average position of the user's left and right eyes. The display system according to Example 4.

[0013] (Example 6) The display system according to Example 1, further comprising at least one interchangeable fitting part removably mounted on the frame and configured to adjust the fit of the frame.

[0014] (Example 7) The interchangeable fitting part includes an interchangeable nose bridge configured to adjust the fit of the frame between the frame and the user's nose bridge. The display system according to Example 6.

[0015] (Example 8) The interchangeable fit component is a display system according to Example 6, including an interchangeable forehead pad configured to adjust the fit of the frame between the frame and the user's forehead.

[0016] (Example 9) The interchangeable fit component is a display system according to Example 6, including an interchangeable rear pad configured to adjust the fit of the frame between the frame and the back of the user's head.

[0017] (Example 10) When the head-mounted display is not properly adjusted to fit the user, the step of providing feedback to the user includes the step of providing a proposal to the user to replace the currently installed interchangeable fit component with another interchangeable fit component, which is a display system according to Example 1.

[0018] (Example 11) A method for evaluating the alignment of virtual image content from a head-mounted display system by the user's eyes, The step of imaging the eyes, and The step of determining whether the imaged features of the eyes are within a predetermined range of the vertical position relative to the head-mounted display, and Based on the determined position of the imaged features, the step of providing a notification, where the notification indicates at least that the head-mounted display and the eyes are not properly aligned. A method including the above.

[0019] (Example 12) The step of determining whether the imaged features of the eyes are within a predetermined range of the vertical position includes the step of determining the position of the eye flash, which is a method according to Example 11.

[0020] (Example 13) The method according to Example 12, further comprising the step of determining the position of the pupil of the eye based on the flash of the eye.

[0021] (Example 14) The head-mounted display system is configured to project light into the eyes and display virtual image content within the user's field of view, and the step of providing a notification includes the step of displaying the notification as virtual image content, according to the method described in Example 11.

[0022] (Example 15) The method according to Example 11, further comprising the step of automatically tracking the pupil of the eye over time and notifying the user when the center of rotation of the eye moves outside a predetermined range of vertical positions.

[0023] (Example 16) When the position of the eye is outside the display alignment volume, further comprising the step of changing from the first field of view of the head-mounted display system to the second field of view of the head-mounted display system, The head-mounted display system includes at least one display having a first field of view when the position of the eye is within the display alignment volume, and the at least one display has a second field of view when the position of the eye is outside the display alignment volume, and the second field of view is smaller than the first field of view. The method according to Example 11.

[0024] (Example 17) The head-mounted display system includes at least one interchangeable fitting part, and the step of providing a notification includes indicating that the wearable system is not properly fitted to the user and proposing or instructing the user to replace the currently worn interchangeable fitting part with an alternative interchangeable fitting part, according to the method described in Example 11.

[0025] (Example 18) The method of Example 17, comprising at least one interchangeable fitting component selected from the group consisting of a nose bridge pad, a forehead pad, and a back pad extending between the wearable system and the back of the user's head.

[0026] (Example 19) The method of Example 18, wherein the at least one interchangeable fitting component includes at least one interchangeable nose bridge pad, further comprising determining that the head-mounted display is too low relative to the eyes, and the step of providing a notification to the user further includes prompting the user to wear a larger nose bridge pad.

[0027] (Example 20) Identifying a plurality of pixels of a display of a head-mounted display system, wherein the user is expected to perceive that the first position of the eye is darkened as a result of being outside the display alignment volume; Increasing the brightness of the plurality of pixels of the display relative to other pixels within the display to reduce the expected darkening; The method of Example 11, further comprising.

[0028] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will be apparent from the description, the drawings, and the claims. Neither this summary nor the following detailed description purports to define or limit the scope of the subject matter of the invention. This specification also provides, for example, the following items. (Item 1) A display system configured to project light onto a user's eyes and display virtual image content, the display system comprising: A frame configured to be supported on the user's head; A head-mounted display disposed on the frame, wherein the display projects light into the user's eyes, displays virtual image content with different amounts of wavefront divergence, and is configured to present virtual image content that appears to be located at different depths at different time periods. A display, One or more eye-tracking cameras configured to image the user's eyes, A processing electronic device that communicates with the head-mounted display and the one or more eye-tracking cameras, wherein the processing electronic device, Determine whether the head-mounted display is properly aligned with the user's eyes by determining whether the imaged features of the eyes are within a predetermined range of the vertical position relative to the head-mounted display, Provide feedback to the user if the head-mounted display is not properly adjusted to fit the user, A processing electronic device configured to perform, A display system comprising. (Item 2) The one or more eye-tracking cameras are configured to image the user's left eye and the user's right eye, The processing electronic device is further configured to determine a left-eye tracking reliability score indicating the reliability level of the position of the user's left eye and determine a right-eye tracking reliability score indicating the reliability level of the position of the user's right eye, When one of the reliability scores exceeds the other, the processing electronic device is further configured to determine whether the head-mounted display is properly aligned based on the user's left eye or right eye associated with the larger reliability score, The display system according to item 1. (Item 3) The one or more eye-tracking cameras are configured to image the user's left eye and the user's right eye, When the left and right eyes of the user are vertically offset from each other by less than a first predetermined threshold value, the processing electronic device is further configured to determine whether the head-mounted display is properly aligned based on the positions of the left and right eyes of the user that are farthest from the desired vertical position. The display system according to item 1. (Item 4) When the left and right eyes of the user are vertically offset from each other by less than a second predetermined threshold value that is less than the first predetermined threshold value, the processing electronic device is further configured to determine whether the head-mounted display is properly aligned based on the average position of the left and right eyes of the user. The display system according to item 3. (Item 5) When the left and right eyes of the user are vertically offset from each other by more than the first predetermined threshold value, the processing electronic device is further configured to determine whether the head-mounted display is properly aligned based on the average position of the left and right eyes of the user. The display system according to item 4. (Item 6) The display system according to item 1, further comprising at least one interchangeable fitting part removably mounted on the frame and configured to adjust the fit of the frame. (Item 7) The interchangeable fitting part includes an interchangeable nose bridge configured to adjust the fit of the frame between the frame and the user's nose bridge. The display system according to item 6. (Item 8) The interchangeable fitting part includes an interchangeable forehead pad configured to adjust the fit of the frame between the frame and the user's forehead. The display system according to item 6. (Item 9) The display system according to item 6, wherein the interchangeable fitting component includes an interchangeable rear pad configured to adjust the fit of the frame between the frame and the back of the user's head. (Item 10) The processing electronic device is further configured to include providing the user with a proposal to replace the currently installed interchangeable fitting component with another interchangeable fitting component in order to provide the user with feedback that the head-mounted display is not properly adjusted to fit the user. The display system according to item 1. (Item 11) A method for evaluating the alignment of virtual image content from a head-mounted display system by the user's eyes, the method comprising: Imaging the eye; Determining whether the imaged feature of the eye is within a predetermined range of vertical position relative to the head-mounted display; Based on the determined position of the imaged feature, providing a notification, the notification indicating at least that the head-mounted display and the eye are not properly aligned; A method comprising. (Item 12) Determining whether the imaged feature of the eye is within a predetermined range of vertical position includes determining the position of the flash of the eye. The method according to item 11. (Item 13) The method according to item 12, further comprising determining the position of the pupil of the eye based on the flash of the eye. (Item 14) The head-mounted display system is configured to project light into the eye and display virtual image content within the user's field of view, and providing the notification includes displaying the notification as virtual image content. The method according to item 11. (Item 15) The method according to item 11, further comprising automatically tracking the pupil of the eye over time and notifying the user when the center of rotation of the eye moves outside a predetermined range of the vertical position. (Item 16) When the position of the eye is outside the display alignment volume, further comprising changing from a first field of view of the head-mounted display system to a second field of view of the head-mounted display system, The head-mounted display system includes at least one display having the first field of view when the position of the eye is within the display alignment volume, and the at least one display has the second field of view when the position of the eye is outside the display alignment volume, and the second field of view is smaller than the first field of view. The method according to item 11. (Item 17) The head-mounted display system includes at least one interchangeable fitting component, and providing the notification indicates that the wearable system is not properly fitted to the user, and includes proposing or instructing the user to replace the currently worn interchangeable fitting component with an alternative interchangeable fitting component. The method according to item 11. (Item 18) The at least one interchangeable fitting component includes at least one fitting component selected from the group consisting of a nose bridge pad, a forehead pad, and a back pad extending between the wearable system and the back of the user's head. The method according to item 17. (Item 19) The at least one interchangeable fitting component includes at least one interchangeable nose bridge pad, and further includes determining that the head-mounted display is too low with respect to the eye, and providing the notification to the user further includes prompting the user to wear a larger nose bridge pad. The method according to item 18. (Item 20) Identifying a plurality of pixels of a display of the head-mounted display system that are expected to be perceived by the user as being darkened as a result of the first position of the eye being outside a display alignment volume; Increasing the brightness of the plurality of pixels of the display relative to other pixels within the display to reduce the expected darkening; The method of claim 11, further comprising. BRIEF DESCRIPTION OF THE DRAWINGS

[0029]

Figure 1

[0030]

Figure 2

[0031]

Figure 3

[0032]

Figure 4

[0033]

Figure 5

[0034]

Figure 6

[0035]

Figure 7A

[0036]

Figure 7B

[0037]

Figure 7C

[0038]

Figure 8A

[0039]

Figure 8B

[0040]

Figure 8C

Figure 8D

Figure 8E

[0041]

Figure 9A

Figure 9B

Figure 9C

[0042]

Figure 9D

Figure 9E

Figure 9F

Figure 9G

[0043]

Figure 10

[0044]

Figure 11

[0045]

Figure 12A

Figure 12B

[0046]

Figure 13A

Figure 13B

[0047]

Figure 13C

Figure 13D

[0048]

Figure 14

[0049]

Figure 15A

Figure 15B

[0050]

Figure 16

[0051]

Figure 17A

Figure 17B

Figure 17C

Figure 17D

Figure 17E

Figure 17F

Figure 17G

Figure 17H

[0052]

Figure 18

[0053]

Figure 19

[0054]

Figure 20

[0055]

Figure 21

[0056]

Figure 22A

Figure 22B

Figure 22C

[0057] Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. The drawings are provided to illustrate the exemplary embodiments described herein and are not intended to limit the scope of the present disclosure.

DETAILED DESCRIPTION OF THE INVENTION

[0058] The display portion of the display system may include a head-mounted display (HMD) that can display three-dimensional (3D) virtual objects such that the objects appear to be located within the user's surrounding environment. As a result, the 3D virtual objects can be perceived by the user in a manner similar to real-world objects.

[0059] The HMD can display an image by outputting spatially modulated light to the user, where the light corresponds to the virtual object. The spatially modulated light containing the image information may be referred to as image light. For the image light to be perceived by the user, the image light travels from the HMD to the user's eyes, propagates through the pupils, and impinges on the retina of the eyes. It should be understood that if all or a portion of the image light for the image does not enter the eye's pupil and / or does not impinge on the retina of the eye, the viewer will not be able to see the image, or the quality of the image may be degraded. As used herein, alignment relates to the relative positioning of the display and the user's eyes. For example, the display can be said to be properly aligned when the user's eyes and the display are positioned relative to each other such that a desired amount of image light enters the eyes. An alignment observer (e.g., a computer program) within the display device may be programmed to monitor whether the display is properly aligned or whether the eyes are positioned to receive the image light from the display.

[0060] For example, in order to appropriately display content to a user by positioning the user's eyes to receive image light, the user's eyes may need to be positioned within a particular region or volume of space relative to the HMD. This volume may be referred to as the display alignment volume. If the user's eyes are outside the display alignment volume, the display quality may be degraded (e.g., there may be dimming and / or display content that does not reach the user's eyes). Various factors may be combined to determine the position of the user's eyes relative to the HMD and thus whether the user's eyes are positioned within the desired display alignment volume. As an example, anatomical variations between users may mean that a head-mounted display fits some users such that their eyes are placed outside the display alignment volume. As another example, the HMD may not be tightly attached to the user's head and may shift over time on the user's head, particularly when the user moves around. In a particular example, the HMD may slip off the user's nose or may be tilted relative to the line between the user's eyes (interpupillary axis), such that the HMD may be unable to provide the desired virtual content due to misalignment of the display relative to the user's eyes (e.g., without some undesirable degradation).

[0061] The various systems and techniques described herein are at least in part directed to solving problems related to proper alignment of a display to enable a viewer to view image content as desired. In some embodiments, a head-mounted display system may be configured to determine the position of a user's eyes. The display system may then determine whether the position of the eyes is within a display alignment volume of the head-mounted display system. The step of determining the position of the eyes may include determining the position of a representative pointer volume associated with the eyes, such as the center of rotation of the eyes. The step of determining whether the position of the eyes is within the display alignment volume may include determining whether the center of rotation of the eyes is within the display alignment volume. As discussed herein, the center of rotation of the eyes may be determined using an inward-facing imaging system configured to image the eyes. Additionally, in some embodiments, the display alignment volume is an imaginary volume associated with a proper fit of the head-mounted display system to the user's eyes. For example, the display alignment volume may be a volume defined by a projection from the surface of the head-mounted display system that outputs image light. More specifically, the display alignment volume may be a three-dimensional geometric shape that tapers from a base to a vertex. The shape of the base of the display alignment volume may be at least partially defined by the geometry of the display, and the depth of the display alignment volume (i.e., the distance from the base to the vertex along the z-axis) may be at least partially defined by the field of view (FOV) of the display. For example, a rounded or circular display may result in a conical display alignment volume, and a polygonal display may result in a pyramidal display alignment volume. As an additional example, a display with a larger FOV may result in a display alignment volume having a smaller depth than a display with a smaller FOV. In some embodiments, the display alignment volume may have the general shape of a truncated cone or pyramid.For example, the display alignment volume may have a general shape of a frustum of a pyramid, such as a frustum of a rectangular pyramid or the like.

[0062] In some embodiments, the imaging system facing inward of the head-mounted display system may obtain an image of the user's face, including the eyes. The imaging system facing inward may be an eye-tracking system that can be mounted on the frame of the head-mounted display. The head-mounted display system may analyze the image and determine the relative position of the user's eyes and the HMD, and whether the respective positions of the user's eyes are within the display alignment volume related to the eyes. Based on this information, the head-mounted display system may notify the user to adjust the fit of the HMD. For example, the notification may inform the user that the device is slipping and needs adjustment, or provide a proposal for adjusting the HMD. In some embodiments, the head-mounted display system may take steps to reduce any display degradation caused by the misalignment of the HMD with the user, such as increasing the brightness in the area that would otherwise be darkened by inconsistent or moving virtual content. Thus, such embodiments of the HMD can assist the user in properly fitting the HMD and reducing problems caused by an improper fit of the HMD, such as when the HMD slips, moves, or tilts relative to the user's head.

[0063] Advantageously, the alignment analysis may be automatically performed using the image obtained from the imaging system facing inward and the information regarding the display alignment volume stored or accessible by the display system. As a result, the fit of the HMD may be corrected initially, optionally in response to using the HMD, and during the course of continued use of the HMD, ensuring a high level of image quality in the use of the head-mounted display system.

[0064] Accordingly, various implementations of systems and methods for observing the alignment of a head-mounted display system and taking an action in response to the observed alignment are provided herein. For example, the display system may be configured to observe the alignment by determining the center of rotation of the user's eyes, determining the boundaries or locations of the alignment volume of the display system, and determining whether the center of rotation is within that alignment volume. It should be understood that the alignment volume may be calculated by the display system and / or provided as predetermined information accessible by the display system. In some embodiments, in response to the observed alignment, the display system may provide feedback to the user regarding whether the alignment can be improved and / or how.

[0065] As another example, the display system may be configured to observe the alignment and provide feedback to the user by imaging the eyes and estimating whether the center of rotation is within the alignment volume without specifically calculating the position of the center of rotation relative to the alignment volume. Rather, the display system may be configured to image the eyes and, based on those images, determine deviations of various eye features from a desired orientation of those features. The display system may then make a determination of a particular adjustment that can be made to the fit of the display system. The adjustment may be correlated with the associated deviation of the various eye features from the desired orientation. The desired orientation may be the orientation of those features when the center of rotation of the eyes is within the alignment volume. The adjustment may, in some embodiments, be an adjustment correlated with steps to address a particular deviation and may include changing a physical part of the display system such that the display system is seated on the user's head and the center of rotation of the eyes can be assumed to be within the desired alignment volume.

[0066] Reference is now made to the drawings, where like reference numerals refer to like parts throughout. Unless otherwise indicated, the drawings are schematic and are not necessarily drawn to scale.

[0067] Example of a 3D display of a wearable system A wearable system (also referred to herein as a head-mounted display system or an augmented reality (AR) system) may be configured to present 2D or 3D virtual images to a user. The images may be still images, video frames, or video in a combination or equivalent. At least a portion of the wearable system may be implemented on a wearable device that can present a VR, AR, or MR environment, alone or in combination, for user interaction. The wearable device may be used synonymously with an AR device (ARD). Further, for the purposes of the present disclosure, the term “AR” is used synonymously with the term “MR”.

[0068] FIG. 1 depicts an illustration of a composite reality scenario with a virtual reality object and a physical object as viewed by a person. In FIG. 1, an MR scene 100 is depicted, and to a user of MR technology, a real-world park-like setting 110 is visible, featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, a user of MR technology also “sees” a robot figure 130 standing on the real-world platform 120 and an avatar character 140 in the form of a flying cartoon that appears anthropomorphic like a honeybee, although these elements do not exist in the real world.

[0069] For a 3D display to generate a true sense of depth, more specifically a simulated sense of surface depth, it may be desirable for each point within the field of view of the display to generate a focusing response corresponding to its virtual depth. If the focusing response for a display point does not correspond to the virtual depth of that point such that it is determined by both the convergence and stereoscopic binocular depth cues, the human eye may experience a focusing conflict, resulting in unstable imaging, harmful eye strain, headaches, and in the absence of focusing information, a near-complete lack of surface depth.

[0070] VR, AR, and MR experiences may be provided by a display system having a display that provides an image viewer with images corresponding to a plurality of depth planes. The images may vary for each depth plane (e.g., providing a somewhat different presentation of a scene or object), and are separately focused by the viewer's eyes, thereby based on the eye focusing required to focus on different image features for scenes located on different depth planes, or based on observing different image features on different depth planes that are out of focus, which can help provide the user with depth cues. As discussed anywhere herein, such depth cues provide a credible perception of depth.

[0071] Figure 2 illustrates an embodiment of a wearable system 200, which may be configured to provide an AR / VR / MR scenario. The wearable system 200 may also be referred to as an AR system 200. The wearable system 200 includes a display 220 and various mechanical and electronic modules and systems for supporting the functions of the display 220. The display 220 may be coupled to a frame 230 that can be worn by a user, wearer, or viewer 210. The display 220 may be positioned in front of the eyes of the user 210. The display 220 may present AR / VR / MR content to the user. Since the display 220 can be worn on the head of the user 210, it may also be referred to as a head-mounted display (HMD), and the wearable system 200 including the display 220 may also be referred to as a head-mounted display system.

[0072] In some embodiments, a speaker 240 is coupled to the frame 230 and positioned adjacent to the user's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / formable acoustic control). The display 220 may include an audio sensor (e.g., a microphone) 232 to detect an audio stream from the environment and capture ambient sound. In some embodiments, one or more other audio sensors, not shown, are positioned to provide stereo sound reception. Stereo sound reception may be used to determine the location of the sound source. The wearable system 200 may perform voice or speech recognition on the audio stream.

[0073] The wearable system 200 may include an outward-facing imaging system 464 (shown in FIG. 4) that observes the world within the environment around the user. The wearable system 200 may also include an inward-facing imaging system 462 (shown in FIG. 4) that can track the user's eye movements. The inward-facing imaging system can track either the movement of one eye or the movement of both eyes. The inward-facing imaging system 462 may be attached to the frame 230 and may communicate electrically with a processing module 260 or 270 that processes the image information obtained by the inward-facing imaging system and can determine, for example, the pupil diameter or orientation, eye movement, or eye pose of the user's eyes. The inward-facing imaging system 462 may include one or more cameras. For example, at least one camera may be used to image each eye. The images obtained by the cameras may be used to determine the pupil size or eye pose separately for each eye, thereby enabling the presentation of image information to each eye to be dynamically adjusted with respect to that eye.

[0074] As an example, the wearable system 200 may use the outward-facing imaging system 464 or the inward-facing imaging system 462 to obtain an image of the user's pose. The image may be a still image, a video frame, or a video.

[0075] The display 220 may be operably coupled to the local data processing module 260 (250) and may be mounted in various configurations, such as fixed to the frame 230 by a wired conductor or wireless connection, fixed to a helmet or hat worn by the user, incorporated within headphones, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, in a belt attachment configuration).

[0076] The local processing and data module 260 may include a hardware processor and digital memory such as non-volatile memory (e.g., flash memory), both of which can be used to assist in data processing, caching, and storage. The data may be a) data captured from sensors such as an image capture device (e.g., a camera within an inward-facing imaging system and / or an outward-facing imaging system), an audio sensor (e.g., a microphone), an inertial measurement unit (IMU), an accelerometer, a compass, a global positioning system (GPS) unit, a wireless device, or a gyroscope (e.g., operably coupled to the frame 230 or otherwise attachable to the user 210), or b) data that may potentially be obtained or processed using the remote processing module 270 or the remote data repository 280 for passage to the display 220 after processing or reading. The local processing and data module 260 may be operably coupled to the remote processing module 270 or the remote data repository 280 by a communication link 262 or 264 via a wired or wireless communication link or the like such that these remote modules are available as resources to the local processing and data module 260. Additionally, the remote processing module 280 and the remote data repository 280 may be operably coupled to each other.

[0077] In some embodiments, the remote processing module 270 may include one or more processors configured to analyze and process data or image information. In some embodiments, the remote data repository 280 may include a digital data storage facility, which may be available through other networking configurations in an Internet or "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module, enabling complete autonomy from the remote modules.

[0078] Exemplary Components of a Wearable System FIG. 3 schematically illustrates exemplary components of a wearable system. FIG. 3 shows a wearable system 200, which can include a display 220 and a frame 230. Stretch view 202 schematically illustrates various components of the wearable system 200. In one implementation, one or more of the components illustrated in FIG. 3 may be part of the display 220. The various components may collect various data (e.g., auditory or visual data, etc.) associated with the user or the user's environment of the wearable system 200, either alone or in combination. It should be understood that other embodiments may have additional or fewer components depending on the use for which the wearable system is employed. Note that FIG. 3 provides some of the various components and a basic concept of the types of data that can be collected, analyzed, and stored through the wearable system.

[0079] Figure 3 shows an exemplary wearable system 200, which may include a display 220. The display 220 may include a display lens 226 that can be mounted on a housing or frame 230 corresponding to the user's head or the frame 230. The display lens 226 may include one or more transparent mirrors positioned in front of the user's eyes 302, 304 by the housing 230, bounce the projected light 338 into the eyes 302, 304, and be configured to allow transmission of at least some light from the local environment while promoting beam shaping. The wavefront of the projected light beam 338 may be bent or focused to match the desired focal length of the projected light. As shown, two wide-field-of-view machine vision cameras 316 (also referred to as world cameras) may be coupled to the housing 230 to image the environment around the user. These cameras 316 may be dual-capture visible / non-visible (e.g., infrared) light cameras. The cameras 316 may be part of an outward-facing imaging system 464 shown in FIG. 4. Images obtained by the world cameras 316 may be processed by a pose processor 336. For example, the pose processor 336 may implement one or more object recognition devices 708 (e.g., shown in FIG. 7) and identify the pose of the user or another person within the user's environment, or identify physical objects within the user's environment.

[0080] Continuing to refer to FIG. 3, shown is a pair of scanning laser shaped wavefront light projector modules (e.g., for depth) with a display mirror and an optical system configured to project light 338 into eyes 302, 304. The depicted figure also shows two small infrared cameras 324 paired with an infrared light source 326 (such as a light emitting diode “LED”, etc.) configured to track the user's eyes 302, 304 and support rendering and user input. Camera 324 may be part of an inward-facing imaging system 462 shown in FIG. 4. Wearable system 200 may further feature a sensor assembly 339, which has X, Y, and Z axis accelerometer capabilities as well as magnetic compass and X, Y, and Z axis gyroscope capabilities and preferably can provide data at a relatively high frequency such as 200 Hz. Sensor assembly 339 may be part of an IMU described with reference to FIG. 2A. The depicted system 200 may also include a head pose processor 336 such as an ASIC (application specific integrated circuit), FPGA (field programmable gate array), or ARM processor (advanced reduced instruction set machine), which may be configured to calculate from wide field of view image information output from a capture device 316 of real-time or near real-time user head pose. Head pose processor 336 may be a hardware processor and may be implemented as part of a local processing and data module 260 shown in FIG. 2A.

[0081] The wearable system may also include one or more depth sensors 234. Depth sensor 234 may be configured to measure the distance between objects in the environment and the wearable device. Depth sensor 234 may include a laser scanner (e.g., LIDAR), an ultrasonic depth sensor, or a depth sensing camera. In some implementations where camera 316 has depth sensing capabilities, camera 316 may also be considered a depth sensor 234.

[0082] Also shown is a processor 332 configured to perform digital or analog processing and derive from gyroscope, compass, or accelerometer data from sensor assembly 339. Processor 332 may be part of local processing and data module 260, shown in FIG. 2. Wearable system 200 may also include a positioning system, such as, for example, GPS 337 (Global Positioning System), as shown in FIG. 3, to assist with attitude and positioning analysis. Additionally, the GPS may further provide remote base (e.g., cloud-based) information about the user's environment. This information may be used to recognize objects or information within the user's environment.

[0083] The wearable system may combine data obtained by GPS 337 and a remote computing system (e.g., remote processing module 270, another user's ARD, etc.), which may provide more information about the user's environment. As one example, the wearable system may determine the user's location based on GPS data and read out a world map that includes virtual objects associated with the user's location (e.g., by communicating with remote processing module 270). As another example, wearable system 200 may use world camera 316 (which may be part of an outward-facing imaging system 464 shown in FIG. 4) to monitor the environment. Based on an image obtained by world camera 316, wearable system 200 may detect objects within the environment (e.g., by using one or more object recognition devices 708 shown in FIG. 7). The wearable system may further interpret characters using data obtained by GPS 337.

[0084] The wearable system 200 may also include a rendering engine 334, which may be configured to provide rendering information local to the user for the view of a user worldwide and to facilitate the operation of the scanner and the imaging into the user's eyes. The rendering engine 334 may be implemented by a hardware processor (e.g., a central processing unit or a graphics processing unit, etc.). In some embodiments, the rendering engine is part of the local processing and data module 260. The rendering engine 334 may be communicatively coupled to other components of the wearable system 200 (e.g., via a wired or wireless link). For example, the rendering engine 334 may be coupled to the eye camera 324 via a communication link 274 and may be coupled to the projection subsystem 318 (which may project light into the user's eyes 302, 304 via a scanning laser array in a manner similar to a retinal scanning display) via a communication link 272. The rendering engine 334 may also communicate with other processing units, such as the sensor attitude processor 332 and the image attitude processor 336, via links 276 and 294, respectively.

[0085] A camera 324 (e.g., a small infrared camera) may be used to track eye pose and support rendering and user input. Some exemplary eye poses may include where the user is looking or the depth at which the user is focused (which may be estimated using eye convergence / divergence motion). A GPS 337, gyroscope, compass, and accelerometer 339 may be used to provide gross or high-speed pose estimation. One or more of the cameras 316 may obtain images and poses, which may be used to map the local environment and share the user view with others in conjunction with data from associated cloud computing resources.

[0086] The exemplary components depicted in FIG. 3 are for illustrative purposes only. For ease of illustration and explanation, a plurality of sensors and other functional modules are shown together. Some embodiments may include only one or a subset of these sensors or modules. Further, the locations of these components are not limited to the positions depicted in FIG. 3. Some components may be mounted or stored within other components, such as belt-mounted components, handheld components, or helmet components. As an example, the image pose processor 336, the sensor pose processor 332, and the rendering engine 334 may be positioned within a belt pack and configured to communicate with other components of the wearable system via wireless communication such as ultra-wideband, Wi-Fi, Bluetooth®, or via wired communication. The depicted housing 230 is preferably head-mountable and wearable by a user. However, some components of the wearable system 200 may be worn on other parts of the user's body. For example, the speaker 240 may be inserted into the user's ear to provide sound to the user.

[0087] Regarding the projection of light 338 into the user's eyes 302, 304, in some embodiments, the camera 324 may generally be utilized to measure the location where the center of the user's eye is geometrically converged, which generally coincides with the focal position or "depth of focus" of the eye. The three-dimensional surface of all points where the eyes converge may be referred to as the "horopter." The focal distance may take a finite number of depths or may vary infinitely. Light projected from the convergence / divergence movement distance appears to be focused on the target eyes 302, 304, while light in front of or behind the convergence / divergence movement distance is blurred. Embodiments of the wearable system and other display systems of the present disclosure are also described in U.S. Patent Publication No. 2016 / 0270656, which is incorporated herein by reference in its entirety.

[0088] The human visual system is complex and it is difficult to provide a realistic perception of depth. An object viewer can perceive an object in three dimensions due to a combination of vergence movement and accommodation. The vergence movement of two eyes relative to each other (e.g., the rotation of the pupils such that the pupils move towards each other or away from each other, converging the lines of sight of the eyes to fixate on an object) is closely associated with the focusing (or "accommodation") of the eye's lens. Under normal conditions, the change in focus of the eye's lens or the eye's accommodation to change the focus from one object to another object at a different distance will automatically cause a coordinated change in vergence under a relationship known as the "accommodation-vergence reflex" at the same distance. Similarly, a change in vergence will, under normal conditions, induce a coordinated change in accommodation. A display system that provides better coordination between accommodation and vergence can form a more realistic and comfortable simulation of a three-dimensional image.

[0089] Furthermore, spatially coherent light with a beam diameter of less than about 0.7 millimeters can be correctly resolved by the human eye regardless of where the eye is focused. Thus, in order to create an illusion of appropriate depth of focus, the eye's convergence / divergence movements may be tracked using camera 324, and rendering engine 334 and projection subsystem 318 may be utilized to focus and render all objects on or near the monocular trajectory and to render all other objects with a varying degree of defocus (e.g., using intentionally created blur). Preferably, system 220 renders to the user at a frame rate of about 60 frames per second or greater. As described above, preferably, camera 324 may be utilized for eye tracking, and the software may be configured to take into account not only the convergence / divergence movement geometry but also a focus location queue for serving as user input. Preferably, such a display system is configured with brightness and contrast suitable for daytime or nighttime use.

[0090] In some embodiments, the display system preferably has a latency of less than about 20 milliseconds, an angular alignment of less than about 0.1 degrees, and a resolution of about 1 arc minute for visual object alignment, which, while not limited by theory, is thought to be near the limit of the human eye. Display system 220 may be integrated with a positioning system, which may involve GPS elements, optical tracking, a compass, an accelerometer, or other data sources and may assist in position and orientation determination. The positioning information may be utilized to facilitate accurate rendering within the user's view of the relevant world (e.g., such information would facilitate glasses in determining their location relative to the real world).

[0091] In some embodiments, the wearable system 200 is configured to display one or more virtual images based on the user's eye accommodation. Different from the conventional 3D display approach that forces the user to focus on the location where the image is projected, in some embodiments, the wearable system automatically varies the focus of the projected virtual content and is configured to enable more comfortable viewing of the one or more images presented to the user. For example, if the user's eye has a current focus of 1 m, the image may be projected to match the user's focus. If the user shifts the focus to 3 m, the image is projected to match the new focus. Thus, rather than forcing a predetermined focus on the user, the wearable system 200 in some embodiments allows the user's eyes to function in a more natural manner.

[0092] Such a wearable system 200 can eliminate or reduce the incidence of eye strain, headaches, and other physiological symptoms typically observed with virtual reality devices. To achieve this, various embodiments of the wearable system 200 are configured to project virtual images at variable focal distances through one or more variable focus elements (VFEs). In one or more embodiments, 3D perception may be achieved through a multi-plane focus system that projects the image onto a fixed focal plane from the user. Other embodiments employ variable plane focus, and the focal plane is reciprocally moved in the z-direction to match the current state of the user's focus.

[0093] In both multi-plane focus systems and variable plane focus systems, the wearable system 200 may employ eye tracking, determine the convergence / divergence movement of the user's eyes, determine the user's current focus, and project a virtual image onto the determined focus. In other embodiments, the wearable system 200 includes a light modulator that variably projects a variable focus light beam in a raster pattern across the retina through a fiber scanner or other light generation source. Thus, the display capabilities of the wearable system 200 that project an image at a variable focal length not only facilitate depth adjustment for the user to visually perceive an object in 3D, but may also be used to compensate for the user's eye abnormalities, as further described in U.S. Patent Publication No. 2016 / 0270656, which is incorporated herein by reference in its entirety. In some other embodiments, the spatial light modulator may project an image to the user through various optical components. For example, as further described below, the spatial light modulator may project an image onto one or more waveguides, which then transmit the image to the user.

[0094] Waveguide stack assembly FIG. 4 illustrates an example of a waveguide stack for outputting image information to a user. The wearable system 400 includes a stack or stacked waveguide assembly 480 of waveguides that can be utilized to provide 3D perception to the eye / brain using a plurality of waveguides 432b, 434b, 436b, 438b, 4400b. In some embodiments, the wearable system 400 may correspond to the wearable system 200 of FIG. 2, and FIG. 4 schematically shows some portions of that wearable system 200 in more detail. For example, in some embodiments, the waveguide assembly 480 may be integrated into the display 220 of FIG. 2.

[0095] Continuing to refer to FIG. 4, waveguide assembly 480 may also include a plurality of features 458, 456, 454, 452 between waveguides. In some embodiments, features 458, 456, 454, 452 may be lenses. In other embodiments, features 458, 456, 454, 452 may not be lenses. Rather, they may simply be spacers (e.g., a cladding layer or structure for forming an air gap).

[0096] Waveguides 432b, 434b, 436b, 438b, 440b or a plurality of lenses 458, 456, 454, 452 may be configured to transmit image information to the eye using various levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane. Image input devices 420, 422, 424, 426, 428 may each be utilized to input image information into waveguides 440b, 438b, 436b, 434b, 432b such that the incident light is dispersed across each individual waveguide for output toward the eye 410. Light exits from the output surfaces of image input devices 420, 422, 424, 426, 428 and is input into the corresponding input edges of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam of light (e.g., a collimated beam) may be input into each waveguide and output an entire field of cloned collimated beams that are directed toward the eye 410 at a particular angle (and divergence amount) corresponding to the depth plane associated with that particular waveguide.

[0097] In some embodiments, the image input devices 420, 422, 424, 426, 428 are discrete displays that generate image information for input into their respective corresponding waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the image input devices 420, 422, 424, 426, 428 are the output ends of a single multiplexed display that can send image information, for example, via one or more optical waveguides (such as optical fiber cables), to each of the image input devices 420, 422, 424, 426, 428.

[0098] The controller 460 controls the operation of the stacked waveguide assembly 480 and the image input devices 420, 422, 424, 426, 428. The controller 460 includes programming (such as instructions in a non-transitory computer-readable medium) that adjusts the timing and provision of image information to the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the controller 460 may be a single integrated device or a distributed system connected by a wired or wireless communication channel. The controller 460 may, in some embodiments, be part of the processing module 260 or 270 (illustrated in FIG. 2).

[0099] Waveguides 440b, 438b, 436b, 434b, 432b may be configured to propagate light within each individual waveguide by total internal reflection (TIR). The waveguides 440b, 438b, 436b, 434b, 432b may each be planar, or have another shape (e.g., curved), with a major top surface and a major bottom surface and an edge extending between their major top and major bottom surfaces. In the illustrated configuration, the waveguides 440b, 438b, 436b, 434b, 432b each include light extraction optical elements 440a, 438a, 436a, 434a, 432a configured to extract light from the waveguides by redirecting the light, propagating it within each individual waveguide, and outputting image information from the waveguides to the eye 410. The extracted light may also be referred to as external coupled light, and the light extraction optical elements may also be referred to as external coupling optical elements. The beam of extracted light is output by the waveguide at the location where the light propagating within the waveguide impinges on the light redirecting element. The light extraction optical elements (440a, 438a, 436a, 434a, 432a) may be, for example, reflective or diffractive optical features. For ease of explanation and clarity of the drawings, they are shown disposed on the bottom major surfaces of the waveguides 440b, 438b, 436b, 434b, 432b, but in some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be disposed on the top major surface or the bottom major surface, or may be disposed directly within the volume of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be attached to a transparent substrate and formed within a layer of the material forming the waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, 432b may be monolithic material components, and the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed on and / or within the surface of the material component.

[0100] Continuing to refer to FIG. 4, as discussed herein, each of the waveguides 440b, 438b, 436b, 434b, 432b is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 as it is input into such waveguide 432b. The collimated light may represent an optically infinite focal plane. The next upper waveguide 434b may be configured to output collimated light that passes through a first lens 452 (e.g., a negative lens) before reaching the eye 410. The first lens 452 may be configured to generate a somewhat convex wavefront curvature such that the eye / brain interprets the light arising from its next upper waveguide 434b as arising from a first focal plane that is closer inwardly toward the eye 410 from the optically infinite. Similarly, the third upper waveguide 436b passes its output light through both the first lens 452 and the second lens 454 before reaching the eye 410. The combined refractive power of the first and second lenses 452 and 454 may be configured to generate another incremental amount of wavefront curvature such that the eye / brain interprets the light arising from the third waveguide 436b as arising from a second focal plane that is even closer inwardly toward the person from the optically infinite than the light from the next upper waveguide 434b was.

[0101] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured, and using the highest waveguide 440b in the stack, its output is sent through all of the lenses between it and the eye for the aggregated focusing power representing the focal plane closest to the person. When viewing / interpreting light originating from the world 470 on the other side of the stacked waveguide assembly 480, a compensation lens layer 430 may be disposed at the top of the stack to compensate for the stack of lenses 458, 456, 454, 452. (The compensation lens layer 430 and the stacked waveguide assembly 480 may be configured such that light originating from the world 470 is transmitted to the eye 410 with substantially the same level of divergence (or collimation) as the light had when first received by the stacked waveguide assembly 480.) Such a configuration provides the same number of perceived focal planes as there are available waveguide / lens pairs. Both the light extraction optical elements of the waveguides and the focusing sides of the lenses may be static (e.g., not dynamic or electroactive). In some alternative embodiments, one or both may be dynamic using electroactive features.

[0102] Continuing to refer to FIG. 4, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be configured to redirect light from their respective waveguides for a particular depth plane associated with the waveguide and output the light with an appropriate amount of divergence or collimation. As a result, waveguides having different associated depth planes may have light extraction optical elements of different configurations that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be three-dimensional or surface features configured to output light at a specific angle. For example, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be a volume hologram, a surface hologram, and / or a diffraction grating. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published Jun. 25, 2015, which is incorporated herein by reference in its entirety.

[0103] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffraction features or “diffractive optical elements” (also referred to herein as “DOEs”) that form a diffraction pattern. Preferably, the DOE has a relatively low diffraction efficiency such that only a portion of the light of the beam is deflected towards the eye 410 at each intersection of the DOE, while the remainder continues to travel through the waveguide via total internal reflection. The light carrying the image information is thus split into several associated output beams that exit the waveguide at multiple locations, resulting in a very uniform pattern of output emission towards the eye 304 with respect to this particular collimated beam that bounces within the waveguide.

[0104] In some embodiments, one or more DOEs may be switchable between an “on” state that actively diffracts and an “off” state that does not significantly diffract. For example, a switchable DOE may comprise a layer of polymer-dispersed liquid crystal, in which microdroplets have a diffraction pattern in a host medium, and the refractive index of the microdroplets may be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract the incident light), or the microdroplets may be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts the incident light).

[0105] In some embodiments, the number and distribution of depth planes or depth of field may be varied dynamically based on the pupil size or orientation of the viewer's eye. The depth of field may vary inversely with the pupil size of the viewer. As a result, as the pupil size of the viewer's eye decreases, one plane that is indistinguishable because its location in the plane is beyond the depth of focus of the eye becomes distinguishable, and as the pupil size decreases and the depth of field correspondingly increases, it may increase to appear more in focus. Similarly, the number of spaced-apart depth planes used to present different images to the viewer may decrease with a decreased pupil size. For example, a viewer may not be able to clearly perceive the details of both a first depth plane and a second depth plane at one pupil size without adjusting the eye's focusing from one depth plane to the other. However, these two depth planes may be sufficient for the user to focus at another pupil size without changing the focusing adjustment simultaneously.

[0106] In some embodiments, the display system may vary the number of waveguides that receive image information based on a determination of pupil size or orientation, or in response to receiving an electrical signal indicative of a particular pupil size or orientation. For example, if the user's eye is indistinguishable between two depth planes associated with two waveguides, the controller 460 (which may be an embodiment of the local processing and data module 260) may be configured or programmed to stop providing image information to one of these waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments where the DOE for the waveguide is switchable between on and off states, the DOE may be switched to the off state when the waveguide receives image information.

[0107] In some embodiments, it may be desirable to satisfy the condition that the outgoing beam has a diameter less than the diameter of the viewer's eye. However, satisfying this condition may be difficult in view of the variability of the viewer's pupil size. In some embodiments, this condition is satisfied over a wide range of pupil sizes by varying the size of the outgoing beam in response to a determination of the viewer's pupil size. For example, as the pupil size decreases, the size of the outgoing beam may also decrease. In some embodiments, the outgoing beam size may be varied using a variable aperture.

[0108] The wearable system 400 may include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the field of view (FOV) of the world camera, and the imaging system 464 is sometimes also referred to as the FOV camera. The FOV of the world camera may or may not be the same as the FOV of the viewer 210 and encompasses a portion of the world 470 that the viewer 210 perceives at a given time. For example, in some situations, the FOV of the world camera may be larger than the field of view of the viewer 210 of the wearable system 400. The entire area available for viewing or imaging by the viewer may be referred to as the field of regard (FOR). The FOR may include a solid angle of 4π steradians surrounding the wearable system 400 so that the wearer can move their body, head, or eyes and perceive substantially any direction in space. In other contexts, the movement of the wearer may be more restricted, and accordingly, the wearer's FOR may touch a smaller solid angle. Images obtained from the outward-facing imaging system 464 may be used, for example, to track gestures made by the user (e.g., hand or finger gestures) and detect objects within the world 470 in front of the user.

[0109] The wearable system 400 may include an audio sensor 232, such as a microphone, to capture ambient sound. As described above, in some embodiments, one or more other audio sensors may be positioned to provide stereo sound reception useful for determining the location of the source of speech. As another example, the audio sensor 232 may comprise a directional microphone, which may also provide such useful directional information regarding the location where the audio source is located. The wearable system 400 may use information from both the outward-facing imaging system 464 and the audio sensor 230 when locating the source of speech or determining the active speaker at a particular instant. For example, the wearable system 400 may use speech recognition, alone or in combination with a reflected image of the speaker (such as seen in a mirror), to determine the identification of the speaker. As another example, the wearable system 400 may be able to determine the location of a speaker within the environment based on sound obtained from a directional microphone. The wearable system 400 may use a speech recognition algorithm to analyze the sound resulting from the location of the speaker, determine the content of the speech, and use speech recognition techniques to determine the identification of the speaker (such as a name or other demographic information).

[0110] The wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera) that observes user movements such as eye movements and face movements. The inward-facing imaging system 466 may capture an image of the eye 410 and may be used to determine the size and / or orientation of the pupil of the eye 304. The inward-facing imaging system 466 may be used to obtain an image for use in determining the direction in which the user is looking (e.g., eye posture), or for biometric identification of the user (e.g., via iris identification). In some embodiments, at least one camera is used to independently determine the pupil size or eye posture of each eye separately for each eye, thereby enabling the presentation of image information to each eye to be dynamically adjusted with respect to that eye. In some other embodiments, only the pupil diameter or orientation of a single eye 410 (e.g., using only a single camera per pair of eyes) is determined and assumed to be similar for both eyes of the user. The images obtained by the inward-facing imaging system 466 may be analyzed to determine the user's eye posture or mood, which may be used by the wearable system 400 to determine the audio or visual content to be presented to the user. The wearable system 400 may also use sensors such as an IMU, accelerometer, gyroscope, etc. to determine the head posture (e.g., head position or head orientation).

[0111] The wearable system 400 may include a user input device 466 through which a user can input commands to the controller 460 and interact with the wearable system 400. For example, the user input device 466 may include a trackpad, a touch screen, a joystick, a multi-degree-of-freedom (DOF) controller, a capacitance sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a wand, a tactile device, a totem (e.g., functioning as a virtual user input device), etc. A multi-DOF controller may sense user input in translational (e.g., left / right, forward / backward, or up / down) or rotational (e.g., yaw, pitch, or roll) movements that are possible for some or all of the controller. A multi-DOF controller that supports translational movement may be referred to as 3DOF, while a multi-DOF controller that supports both translational and rotational movement may be referred to as 6DOF. In some cases, the user may use a finger (e.g., the thumb) to press or swipe on a touch sensor-based input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held by the user's hand during use of the wearable system 400. The user input device 466 may communicate with the wearable system 400 either wired or wirelessly.

[0112] Other components of the wearable system In many implementations, a wearable system may include, in addition to or instead of the components of the wearable system described above, other components. The wearable system may include, for example, one or more haptic devices or components. The haptic device or component may be operable to provide a haptic sensation to the user. For example, the haptic device or component may provide a haptic sensation of pressure or texture when touching virtual content (e.g., virtual objects, virtual tools, other virtual structures). The haptic sensation may reproduce the sensation of a physical object represented by the virtual object, or may reproduce the sensation of an imaginary object or character (e.g., a dragon) represented by the virtual content. In some implementations, the haptic device or component may be worn by the user (e.g., a user-wearable glove). In some implementations, the haptic device or component may be held by the user.

[0113] A wearable system may include, for example, one or more physical objects that are operable by a user and enable input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as, for example, pieces of metal or plastic, walls, table surfaces, etc. In some implementations, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, a totem may simply provide a physical surface, and the wearable system may render a user interface so as to appear to the user to be on one or more surfaces of the totem. For example, the wearable system may render an image of a computer keyboard and trackpad so as to appear to be resident on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and virtual trackpad so as to appear on the surface of a thin rectangular plate of aluminum that serves as a totem. The rectangular plate itself does not have any physical keys or trackpads or sensors. However, the wearable system may detect user operations or interactions or touches using the rectangular plate as selections or inputs made via the virtual keyboard or virtual trackpad. The user input device 466 (shown in FIG. 4) may be an embodiment of a totem that may include a trackpad, touchpad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. The user may use the totem alone or in combination with a gesture to interact with the wearable system or other users.

[0114] Examples of haptic devices and totems that can be used with the wearable devices, HMDS, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated herein by reference in its entirety.

[0115] Example of an eye image FIG. 5 illustrates an image of an eye 500 with an eyelid 504, a sclera 508 (“white of the eye”), an iris 512, and a pupil 516. Curve 516a indicates the pupil boundary between the pupil 516 and the iris 512, and curve 512a indicates the edge boundary between the iris 512 and the sclera 508. The eyelid 504 includes an upper eyelid 504a and a lower eyelid 504b. The eye 500 is illustrated in a natural rest position (e.g., oriented such that both the user's face and line of sight will be directed at a distant object directly in front of the user). The natural rest position of the eye 500 can be indicated by a natural rest direction 520 that is in a position (e.g., out of the page with respect to the eye 500 shown in FIG. 5) that is perpendicular to the surface of the eye 500 when centered within the pupil 516 in the present example.

[0116] As the eye 500 moves to look at different objects, the eye pose will change relative to the natural rest direction 520. The current eye pose may be determined with reference to an eye pose direction 524 that is a direction perpendicular to the surface of the eye (and centered within the pupil 516) but oriented toward the object at which the eye is currently directed. Referring to the exemplary coordinate system shown in FIG. 5A, the pose of the eye 500 can be represented as two angular parameters that both indicate the azimuth deviation and zenith deviation of the eye pose direction 524 of the eye relative to the natural rest direction 520 of the eye. For illustrative purposes, these angular parameters can be represented as θ (azimuth deviation, determined from a reference azimuth) and φ (zenith deviation, sometimes also referred to as polar deviation). In some implementations, the angular roll of the eye around the eye pose direction 524 may be included in the determination of the eye pose and the angular roll may be included in the following analysis. In other implementations, other techniques for determining the eye pose may be used, such as a pitch, yaw, and optionally, a roll system.

[0117] Eye images may be obtained from video using any suitable process, for example, a video processing algorithm that can extract the image from one or more sequential frames. The eye pose may be determined from the eye image using various eye tracking techniques. For example, the eye pose may be determined by considering the lens effect of the cornea on the provided light source. Any suitable eye tracking technique may be used to determine the eye pose in the eyelid shape estimation technique described herein.

[0118] Example of an eye tracking system FIG. 6 illustrates a schematic diagram of a wearable or head-mounted display system 600 that includes an eye-tracking system. The head-mounted display system 600 may include, in at least some embodiments, components located within a head-mounted unit 602 and components located within a non-head-mounted unit 604. The non-head-mounted unit 604 may be, by way of example, a belt-mounted component, a handheld component, a component within a backpack, a remote component, etc. Incorporating some of the components of the head-mounted display system 600 within the non-head-mounted unit 604 can help reduce the size, weight, complexity, and cost of the head-mounted unit 602. In some implementations, some or all of the functionality described as being implemented by one or more components of the head-mounted unit 602 and / or the non-head-mounted 604 may be provided using one or more components included anywhere within the head-mounted display system 600. For example, some or all of the functionality described below in connection with the CPU 612 of the head-mounted unit 602 may be provided using the CPU 616 of the non-head-mounted unit 604, and vice versa. In some embodiments, some or all of such functionality may be provided using a peripheral device of the head-mounted display system 600. Further, in some implementations, some or all of such functionality may be provided using one or more cloud computing devices or other remotely located computing devices in a manner similar to that described above with reference to FIG. 2.

[0119] As shown in FIG. 6, the head-mounted display system 600 may include an eye-tracking system that includes a camera 324 that captures an image of the user's eye 610. Optionally, the eye-tracking system may also include light sources 326a and 326b (such as light-emitting diodes "LEDs"). The light sources 326a and 326b may generate a flash (i.e., a reflection from the user's eye that appears in the image of the eye captured by the camera 324). The positions of the light sources 326a and 326b relative to the camera 324 may be known, such that the position of the flash in the image captured by the camera 324 may be used when tracking the user's eye (as will be discussed in more detail below in connection with FIGS. 7-11). In at least one embodiment, there may be one light source 326 and one camera 324 associated with one of the user's eyes 610. In another embodiment, there may be one light source 326 and one camera 324 associated with each of the user's eyes 610. In yet another embodiment, there may be one or more cameras 324 and one or more light sources 326 associated with one or each of the user's eyes 610. As a specific example, there may be two light sources 326a and 326b and one or more cameras 324 associated with each of the user's eyes 610. As another example, there may be three or more light sources such as light sources 326a and 326b and one or more cameras 324 associated with each of the user's eyes 610.

[0120] The eye tracking module 614 may receive an image from the eye tracking camera 324, analyze the image, and extract various information. As an example, the eye tracking module 614 may detect the user's eye pose, the three-dimensional position of the user's eyes relative to the eye tracking camera 324 (and the head-mounted unit 602), the direction of one or both of the user's focused eyes 610, the user's vergence / accommodation motion depth (i.e., the depth from the user at which the user is focused), the position of the user's pupils, the position of the user's corneas and corneal spheres, the respective centers of rotation of the user's eyes, and the respective centers of the user's eye viewpoints. The eye tracking module 614 may extract such information using the techniques described below in connection with FIGS. 7-11. As shown in FIG. 6, the eye tracking module 614 may be a software module implemented using the CPU 612 within the head-mounted unit 602.

[0121] Data from the eye tracking module 614 may be provided to other components within the wearable system. As an example, such data may be transmitted to components within a non-head-mounted unit 604, such as the CPU 616, including software modules for the light field rendering controller 618 and the alignment observer 620, which may be configured to evaluate whether the display of the head-mounted display system 600 is properly aligned with the user's eyes.

[0122] The rendering controller 618 may adjust the image displayed to the user using the information from the eye tracking module 614 by the rendering engine 622 (which may be a software module within the GPU 621 and may provide images to the display 220). As an example, the rendering controller 618 may adjust the image displayed to the user based on the center of rotation or the center of the user's viewpoint. In particular, the rendering controller 618 may use the information regarding the center of the user's viewpoint to simulate a rendering camera (i.e., simulate the collection of an image from the user's viewpoint), and may adjust the image displayed to the user based on the simulated rendering camera.

[0123] Sometimes referred to as a "pinhole perspective projection camera" (or simply, a "perspective projection camera") or a "virtual pinhole camera" (or simply, a "virtual camera"), a "rendering camera" is a simulated camera potentially used for rendering virtual image content from a database of objects within a virtual world. The objects may have locations and orientations with respect to a user or wearer and, potentially, real objects within an environment surrounding the user or wearer. In other words, a rendering camera may represent a viewpoint within a rendering space from which a user or wearer is to visually perceive 3D virtual content (e.g., virtual objects) within the rendering space. A rendering camera may be managed by a rendering engine and may render a virtual image based on a database of virtual objects to be presented to the eye. The virtual image may be rendered as if it were captured from the viewpoint of the user or wearer. For example, the virtual image may be rendered as if it were captured by a pinhole camera (corresponding to the "rendering camera") having a specific set of intrinsic parameters (e.g., focal length, camera pixel size, principal point coordinates, distortion parameters, etc.) and a specific set of extrinsic parameters (e.g., translation and rotation components with respect to the virtual world). The virtual image is captured from the viewpoint of such a camera having the position and orientation of the rendering camera (e.g., the extrinsic parameters of the rendering camera). The system is then to be able to define and / or adjust the intrinsic and extrinsic rendering camera parameters. For example, the system may define a specific set of extrinsic rendering camera parameters such that the virtual image is rendered as if it were captured from the viewpoint of a camera having a specific location with respect to the user or wearer's eye so that the image appears as if it were from the viewpoint of the user or wearer. The system may later dynamically adjust the extrinsic rendering camera parameters on the fly to maintain alignment with the specific location. Similarly, the intrinsic rendering camera parameters may also be defined and dynamically adjusted over time.In some implementations, the image is rendered as if it were captured from the viewpoint of a camera having an aperture (e.g., a pinhole) at a specific location relative to the user's or wearer's eye (such as the center of the viewpoint or the center of rotation or other locations).

[0124] In some embodiments, the system may create or dynamically reposition and / or reorient one rendering camera for the user's left eye and a different rendering camera for the user's right eye as the user's eyes are physically separated from each other and thus consistently located at different positions. In at least some implementations, virtual content rendered from the viewpoint of the rendering camera associated with the viewer's left eye may be presented to the user through the left eyepiece on the left side of a head-mounted display (e.g., head-mounted unit 602), and virtual content rendered from the viewpoint of the rendering camera associated with the user's right eye may be presented to the user through the right eyepiece on the right side of such a head-mounted display. Further details regarding the creation, adjustment, and use of the rendering cameras in the rendering process are provided in U.S. Patent Application No. 15 / 274,823, entitled "METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTURAL FEATURES IN 3D RECONSTRUCTION", which is hereby expressly incorporated by reference in its entirety for all purposes.

[0125] In some embodiments, one or more modules (or components) of system 600 (e.g., light field rendering controller 618, rendering engine 622, etc.) may determine the position and orientation of a rendering camera within a rendering space based on the position and orientation of the user's head and eyes (e.g., as determined based on head pose and eye tracking data, respectively). That is, system 600 effectively maps the position and orientation of the user's head and eyes to a particular location and angular position within a 3D virtual environment, positions and orients a rendering camera at the particular location and angular position within the 3D virtual environment, and may render virtual content for the user as would be captured by the rendering camera. Further details discussing the real-world / virtual-world mapping process are provided in U.S. Patent Application No. 15 / 296,869, entitled "SELECTING VIRTUAL OBJECTS IN A THREE-DIMENSIONAL SPACE", which is hereby expressly incorporated by reference in its entirety for all purposes. As an example, the rendering controller 618 may adjust the depth at which an image is displayed by selecting the depth plane (or depth planes) to be utilized at any given time for displaying the image. In some implementations, such depth plane switching may be performed through adjustment of one or more intrinsic rendering camera parameters.

[0126] The alignment observer 620 may identify whether the head-mounted unit 602 is properly positioned on the user's head using information from the eye tracking module 614. As an example, the eye tracking module 614 may provide eye location information such as the position of the center of rotation of the user's eyes, indicating the three-dimensional position of the user's eyes relative to the camera 324, and the head-mounted unit 602 and the eye tracking module 614 may use the location information to determine whether the display 220 is properly aligned within the user's field of view, or whether the head-mounted unit 602 (or headset) has slipped or is otherwise misaligned with the user's eyes. As an example, the alignment observer 620 may determine whether the head-mounted unit 602 has slipped from the user's nasal bridge and thus moved the display 220 away from and downward from the user's eyes (which may not be desirable), whether the head-mounted unit 602 has moved above the user's nasal bridge and thus moved the display 220 closer to and upward from the user's eyes, whether the head-mounted unit 602 has shifted left or right relative to the user's nasal bridge, whether the head-mounted unit 602 has been lifted above the user's nasal bridge, or whether the head-mounted unit 602 has moved away from the desired position or range of positions in these or other ways. Generally, the alignment observer 620 may generally be able to determine whether the head-mounted unit 602, and in particular the display 220, is properly positioned in front of the user's eyes. In other words, the alignment observer 620 may determine whether the left display in the display system 220 is properly aligned with the user's left eye and whether the right display in the display system 220 is properly aligned with the user's right eye. The alignment observer 620 may determine whether the head-mounted unit 602 is properly positioned by determining whether the head-mounted unit 602 is positioned and oriented within the desired range of positions and / or orientations relative to the user's eyes.

[0127] In at least some embodiments, the alignment observer 620 may generate user feedback in the form of an alert, message, or other content. Such feedback may be provided to the user and inform the user of any misalignment of the head-mounted unit 602, along with optional feedback regarding how to correct the misalignment (such as suggestions for adjusting the head-mounted unit 602 in a particular manner).

[0128] Exemplary alignment observation and feedback techniques that may be utilized by the alignment observer 620 are described in U.S. Patent Application No. 15 / 717,747, filed on September 27, 2017 (Attorney Docket No. MLEAP.052A2), which is incorporated herein by reference in its entirety.

[0129] Example of an eye tracking module A detailed block diagram of an exemplary eye tracking module 614 is shown in FIG. 7A. As shown in FIG. 7A, the eye tracking module 614 may include various different sub-modules, may provide various different outputs, and may utilize various available data when tracking the user's eyes. By way of example, the eye tracking module 614 may utilize available data including the geometric arrangement of the light source 326 and the eye tracking camera 324 relative to the head-mounted unit 602, an assumed eye dimension 704 such as a typical distance of about 4.7 mm between the center of the user's corneal curvature and the average center of rotation of the user's eye, or a typical distance between the user's center of rotation and the center of the point of view, and calibration data 706 specific to each user such as the user's interpupillary distance. Additional examples of incidental, intrinsic, and other information that may be employed by the eye tracking module 614 are described in U.S. Patent Application No. 15 / 497,726, filed on April 26, 2017 (Attorney Docket No. MLEAP.023A7), which is incorporated herein by reference in its entirety.

[0130] The image preprocessing module 710 may receive an image from an eye camera such as the eye camera 324, and may perform one or more preprocessing (i.e., adjustment) operations on the received image. As an example, the image preprocessing module 710 may apply Gaussian blur to the image, may downsample the image to a lower resolution, may apply an unsharp mask, may apply an edge sharpening algorithm, or may apply other suitable filters that assist in subsequent detection, localization, and labeling of flashes, pupils, or other features within the image from the eye camera 324. The image preprocessing module 710 may remove high-frequency noise from the pupil boundary 516a (see FIG. 5), etc., and thereby may remove noise that may interfere with pupil and flash determination, and may apply a low-pass filter such as an open filter or a morphological filter. The image preprocessing module 710 may output the preprocessed image to the pupil identification module 712 and the flash detection and labeling module 714.

[0131] The pupil recognition module 712 may receive the pre - processed image from the image pre - processing module 710 and may identify the regions of those images that contain the user's pupils. In some embodiments, the pupil recognition module 712 may determine the coordinates of the position of the user's pupil in the eye - tracking image from the camera 324, i.e., the coordinates of the center or centroid. In at least some embodiments, the pupil recognition module 712 may identify the contour (e.g., the contour of the pupil - iris boundary) in the eye - tracking image, identify the contour moment (i.e., the center of mass), apply the starburst pupil detection and / or Canny edge detection algorithm, exclude outliers based on intensity values, identify sub - pixel boundary points, correct for eye camera distortion (i.e., the distortion in the images captured by the eye camera 324), apply the random sample consensus (RANSAC) iterative algorithm, fit an ellipse to the boundary in the eye - tracking image, apply a tracking filter to the image, and identify the sub - pixel image coordinates of the user's pupil centroid. The pupil recognition module 712 may output pupil recognition data, which may indicate the region of the pre - processed image module 712 identified as showing the user's pupil, to the flash detection and labeling module 714. The pupil recognition module 712 may provide the 2D coordinates of the user's pupil (i.e., the 2D coordinates of the user's pupil centroid) in each eye - tracking image to the flash detection module 714. In at least some embodiments, the pupil recognition module 712 may also provide the same type of pupil recognition data to the coordinate system normalization module 718.

[0132] Pupil detection techniques that may be utilized by the pupil recognition module 712 are described in U.S. Patent Publication No. 2017 / 0053165, published on February 23, 2017, and U.S. Patent Publication No. 2017 / 0053166, published on February 23, 2017, each of which is incorporated herein by reference in its entirety.

[0133] The flash detection and labeling module 714 may receive the pre - processed image from module 710 and the pupil identification data from module 712. The flash detection module 714 may use this data to detect and / or identify a flash (i.e., the reflection of light from the light source 326 off the user's eye) within the region of the pre - processed image that indicates the user's pupil. As an example, the flash detection module 714 may search for bright regions in the eye - tracking image that are near the user's pupil and are sometimes also referred to herein as "blobs" or local intensity maxima. In at least some embodiments, the flash detection module 714 may re - scale (e.g., enlarge) the pupil ellipse to include additional flashes. The flash detection module 714 may filter the flashes by size and / or intensity. The flash detection module 714 may also determine the 2D position of each flash within the eye - tracking image. In at least some examples, the flash detection module 714 may determine the 2D position of the flash relative to the user's pupil, which may also be referred to as a pupil - flash vector. The flash detection and labeling module 714 may label the flashes and output the pre - processed image with the labeled flashes to the 3D corneal center estimation module 716. The flash detection and labeling module 714 may also transmit data such as the pre - processed image from module 710 and the pupil identification data from module 712.

[0134] Pupil and flash detection, such as that implemented by modules such as modules 712 and 714, may use any suitable technique. As an example, edge detection may be applied to the eye image to identify the flash and the pupil. Edge detection may be applied by various edge detectors, edge detection algorithms, or filters. For example, a Canny edge detector may be applied to the image to detect edges such as lines in the image. The edges may include points located along a line corresponding to a local maximum derivative. For example, the pupil boundary 516a (see FIG. 5) may be located using a Canny edge detector. Once the location of the pupil is determined, various image processing techniques may be used to detect the "pose" of the pupil 116. The determination of the eye pose of the eye image may also be referred to as the detection of the eye pose of the eye image. The pose may also be referred to as the line of sight, the direction being looked at, or the orientation of the eye. For example, the pupil may be looking left towards an object, and the pose of the pupil may be classified as a left-facing pose. Other methods may also be used to detect the location of the pupil or the flash. For example, concentric rings may be located within the eye image using a Canny edge detector. As another example, an integral differential operator may be used to find the corneal limbus boundary of the pupil or the iris. For example, a Daugman integral differential operator, a Hough transform, or other iris segmentation techniques may be used to return a curve that estimates the boundary of the pupil or the iris.

[0135] The 3D corneal center estimation module 716 may receive a pre-processed image, including the detected flash data and pupil identification data, from modules 710, 712, 714. The 3D corneal center estimation module 716 may use this data to estimate the 3D position of the user's cornea. In some embodiments, the 3D corneal center estimation module 716 may estimate the 3D position of the center of the corneal curvature of the eye or the center of the user's corneal sphere, i.e., generally the center of an imaginary sphere having a surface portion coextensive with the user's cornea. The 3D corneal center estimation module 716 may provide data indicating the estimated 3D coordinates of the corneal sphere and / or the user's cornea to the coordinate system normalization module 718, the optical axis determination module 722, and / or the light field rendering controller 618. Further details of the operation of the 3D corneal center estimation module 716 are provided herein in connection with FIGS. 8A-8E. Techniques for estimating the position of eye features such as the cornea or corneal sphere, which may be utilized by the 3D corneal center estimation module 716 and other modules within the wearable system of the present disclosure, are discussed in U.S. Patent Application No. 15 / 497,726, filed Apr. 26, 2017 (Attorney Docket No. MLEAP.023A7), which is incorporated herein by reference in its entirety.

[0136] The coordinate system normalization module 718 may optionally be included within the eye tracking module 614 (as indicated by its dashed outline). The coordinate system normalization module 718 may receive data indicating the estimated 3D coordinates of the center of the user's cornea (and / or the center of the user's corneal sphere) from the 3D corneal center estimation module 716, and may also receive data from other modules. The coordinate system normalization module 718 may normalize the eye camera coordinate system, which may help compensate for slippage of the wearable device (e.g., slippage of the head-mounted component from its normal resting position on the user's head, which may be identified by the alignment observer 620). The coordinate system normalization module 718 may rotate the coordinate system and align the z-axis of the coordinate system (i.e., the convergence / divergence motion depth axis) with the corneal center (e.g., as indicated by the 3D corneal center estimation module 716), and may translate the camera center (i.e., the origin of the coordinate system) to a predetermined distance away from the corneal center, such as 30 mm (i.e., the module 718 may expand or contract the eye tracking image depending on whether the eye camera 324 is determined to be closer or farther than the predetermined distance). By using this normalization process, the eye tracking module 614 may be able to establish a consistent orientation and distance within the eye tracking data, relatively independently of variations in the headset positioned on the user's head. The coordinate system normalization module 718 may provide the 3D coordinates of the center of the cornea (and / or corneal sphere), pupil identification data, and the preprocessed eye tracking image to the 3D pupil center locator module 720. Further details of the operation of the coordinate system normalization module 718 are provided herein in connection with FIGS. 9A-9C.

[0137] The 3D pupil center locator module 720 may receive data including the 3D coordinates of the center of the user's cornea (and / or corneal sphere), pupil location data, and pre - processed eye - tracking images, in a normalized or non - normalized coordinate system. The 3D pupil center locator module 720 may analyze such data to determine the 3D coordinates of the center of the user's pupil in a normalized or non - normalized eye camera coordinate system. The 3D pupil center locator module 720 may determine the location of the user's pupil in three dimensions based on the 2D position of the pupil centroid (as determined by module 712), the 3D position of the corneal center (as determined by module 716), assumed eye dimensions 704 such as the size of the typical user's corneal sphere and the typical distance from the corneal center to the pupil center, and the optical properties of the eye such as the refractive index of the cornea (relative to the refractive index of air), or any combination of these. Further details of the operation of the 3D pupil center locator module 720 are provided herein in connection with FIGS. 9D - 9G. Techniques for estimating the position of eye features such as the pupil, which may be utilized by the 3D pupil center locator module 720 and other modules within the wearable system of the present disclosure, are discussed in U.S. Patent Application No. 15 / 497,726, filed on April 26, 2017 (Attorney Docket No. MLEAP.023A7), which is incorporated herein by reference in its entirety.

[0138] The optical axis determination module 722 may receive data indicating the 3D coordinates of the center of the user's cornea and the center of the user's pupil from modules 716 and 720. Based on such data, the optical axis determination module 722 may identify a vector from the position of the corneal center (i.e., from the center of the corneal sphere) to the center of the user's pupil that may define the optical axis of the user's eye. As an example, the optical axis determination module 722 may provide an output defining the user's optical axis to modules 724, 728, 730, and 732.

[0139] The center of rotation (CoR) estimation module 724 may receive data from module 722 that includes parameters of the optical axis of the user's eye (i.e., data indicating the direction of the optical axis within a coordinate system with a known relationship to the head-mounted unit 602). The CoR estimation module 724 may estimate the center of rotation of the user's eye (i.e., the point around which the user's eye rotates when the user's eye rotates left, right, up, and / or down). Assume that a single point may be sufficient even if the eye cannot rotate perfectly around a single point. In at least some embodiments, the CoR estimation module 724 may estimate the center of rotation of the eye by moving the center of the pupil (identified by module 720) or the center of curvature of the cornea (as identified by module 716) a specific distance along the optical axis (identified by module 722) toward the retina. This specific distance may be the assumed eye dimension 704. As an example, the specific distance between the center of curvature of the cornea and the CoR may be about 4.7 mm. This distance may be varied for a particular user based on any relevant data including the user's age, gender, vision prescription, other relevant characteristics, etc.

[0140] In at least some embodiments, the CoR estimation module 724 may refine over time the estimated value of the respective center of rotation of the user's eyes. As an example, over time, the user may eventually rotate the eye (to look at something else, closer, farther away, or at some point left, right, up, or down), causing an offset along the respective optical axis of the eye. The CoR estimation module 724 may then analyze the two (or more) optical axes identified by module 722 and locate the 3D point at the intersection of those optical axes. The CoR estimation module 724 may then determine the center of rotation at the 3D point of that intersection. Such techniques may provide an estimated value of the center of rotation with improved accuracy over time. Various techniques may be employed to increase the accuracy of the CoR estimation module 724 as well as the determined CoR positions of the left and right eyes. As an example, the CoR estimation module 724 may estimate the CoR by finding the average point of the intersections of the optical axes determined over time for various different eye postures. As an additional example, module 724 may filter or average the estimated CoR positions over time, calculate a moving average of the estimated CoR positions over time, and / or apply a Kalman filter and the known dynamics of the eye and eye tracking system to estimate the CoR position over time. As a specific example, module 724 may slowly move over time from a hypothesized CoR position (i.e., 4.7 mm behind the center of the corneal curvature of the eye) to a somewhat different location within the user's eye as eye tracking data regarding the user is acquired, thereby enabling per-user refinement of the CoR position, by calculating a weighted average of the determined points of the optical axis intersection and the hypothesized CoR position (such as 4.7 mm from the center of the corneal curvature of the eye).

[0141] The interpupillary distance (IPD) estimation module 726 may receive data indicating the estimated 3D positions of the centers of rotation of the user's left and right eyes from the CoR estimation module 724. The IPD estimation module 726 may then estimate the user's IPD by measuring the 3D distance between the centers of rotation of the user's left and right eyes. Generally, the distance between the estimated CoR of the user's left eye and the estimated CoR of the user's right eye may be approximately equal to the distance between the centers of the user's pupils when the user is looking at optical infinity (i.e., the optical axes of the user's eyes are substantially parallel to each other), which is the typical definition of the interpupillary distance (IPD). The user's IPD may be used by various components and modules within the wearable system. As an example, the user's IPD may be provided to the alignment observer 620 and used when assessing the extent to which the wearable device is aligned with the user's eyes (e.g., whether the left and right display lenses are appropriately spaced according to the user's IPD). As another example, the user's IPD may be provided to the convergence / divergence motion depth estimation module 728 and used when determining the user's convergence / divergence motion depth. The module 726 may employ various techniques such as those discussed in relation to the CoR estimation module 724 to increase the accuracy of the estimated IPD. As an example, the IPD estimation module 724 may apply filtering, averaging over time, weighted averaging, including an assumed IPD distance, a Kalman filter, etc. as part of the estimation of the user's IPD in an accurate manner.

[0142] The convergence-divergence motion depth estimation module 728 may receive data from various modules and sub-modules within the eye tracking module 614 (as shown in connection with FIG. 7A). In particular, the convergence-divergence motion depth estimation module 728 may employ data indicative of the estimated 3D position of the pupil center (e.g., as provided by module 720 described above), one or more determined parameters of the optical axis (e.g., as provided by module 722 described above), the estimated 3D position of the center of rotation (e.g., as provided by module 724 described above), the estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the centers of rotation) (e.g., as provided by module 726 described above), and / or one or more determined parameters of the optical axis and / or visual axis (e.g., as provided by module 722 and / or module 730 described below). The convergence-divergence motion depth estimation module 728 may detect or otherwise obtain a measurement of the user's convergence-divergence motion depth, which may be the distance from the user at which the user's eyes are focused. As an example, when the user is looking at an object 3 feet from their front, the user's left and right eyes have a convergence-divergence motion depth of 3 feet, while when the user is looking at a distant landscape (i.e., the optical axes of the user's eyes are substantially parallel to each other such that the distance between the centers of the user's pupils can be approximately equal to the distance between the centers of rotation of the user's left and right eyes), the user's left and right eyes have an infinite convergence-divergence motion depth. In some implementations, the convergence-divergence motion depth estimation module 728 may utilize data indicative of the estimated center of the user's pupil (e.g., as provided by module 720) and determine the 3D distance between the estimated centers of the user's pupils. The convergence-divergence motion depth estimation module 728 may obtain a measurement of the convergence-divergence motion depth by comparing such a determined 3D distance between pupil centers with the estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the centers of rotation) (e.g., as indicated by module 726 described above).In addition to the 3D distance between the pupil centers and the estimated IPD, the convergence / divergence motion depth estimation module 728 may calculate the convergence / divergence motion depth using known, assumed, estimated, and / or determined geometries. As an example, module 728 may combine the 3D distance between the pupil centers, the estimated IPD, and the 3D CoR position in the trigonometric calculation to estimate (i.e., determine) the user's convergence / divergence motion depth. In fact, the evaluation of such a determined 3D distance between the pupil centers relative to the estimated IPD can serve to indicate a measured value of the user's current convergence / divergence motion depth relative to optical infinity. In some embodiments, the convergence / divergence motion depth estimation module 728 may simply receive or access data indicating the estimated 3D distance between the estimated centers of the user's pupils for the purpose of obtaining such a measured value of the convergence / divergence motion depth. In some embodiments, the convergence / divergence motion depth estimation module 728 may estimate the convergence / divergence motion depth by comparing the user's left and right optical axes. In particular, the convergence / divergence motion depth estimation module 728 may estimate the convergence / divergence motion depth by identifying the distance from the user at which the user's left and right optical axes intersect (or the projections of the user's left and right optical axes on a plane such as a horizontal plane intersect). Module 728 may utilize the user's IPD in this calculation by setting zero depth to be the depth at which the user's left and right optical axes are separated by the user's IPD. In at least some embodiments, the convergence / divergence motion depth estimation module 728 may determine the convergence / divergence motion depth by triangulating the eye tracking data with known or derived spatial relationships.

[0143] In some embodiments, the vergence / accommodation depth estimation module 728 may estimate the user's vergence / accommodation depth based on the intersection of the user's visual axes (instead of their optical axes), which may provide a more accurate indication of the distance at which the user is focused. In at least some embodiments, the eye tracking module 614 may include an optical axis / visual axis mapping module 730. As will be discussed in more detail in connection with FIG. 10, the user's optical axis and visual axis generally do not coincide. The visual axis is the axis along which a person is looking, while the optical axis is defined by the centers of the person's lens and pupil and may pass through the center of the person's retina. In particular, the user's visual axis is generally offset from the center of the user's retina, thereby resulting in different optical and visual axes, as defined by the location of the user's fovea. In at least some of these embodiments, the eye tracking module 614 may include an optical axis / visual axis mapping module 730. The optical axis / visual axis mapping module 730 may correct for the difference between the user's optical axis and visual axis and provide information regarding the user's visual axis to other components within the wearable system, such as the vergence / accommodation depth estimation module 728 and the light field rendering controller 618. In some examples, the module 730 may use an assumed eye dimension 704 that includes a typical offset of approximately 5.2° inward (towards the nose, towards the user's nose) between the optical axis and the visual axis. In other words, the module 730 may shift the user's left optical axis 5.2° to the right towards the nose (nasally) and the user's right optical axis 5.2° to the left towards the nose (nasally) to estimate the directions of the user's left and right optical axes. In other examples, the module 730 may utilize per-user calibration data 706 when mapping the optical axis (e.g., as indicated by the module 722 described above) to the visual axis. As an additional example, the module 730 may shift the user's optical axis nasally by an arbitrary range formed by, for example, 4.0° - 6.5°, 4.5° - 6.0°, 5.0° - 5.4°, etc., or any of these values.In some arrays, the module 730 may apply an offset, at least in part, based on characteristics of a particular user such as their age, gender, visual prescription, or other relevant characteristics, and / or at least in part based on a calibration process for a particular user (i.e., to determine the optical axis - visual axis offset for a particular user). In at least some embodiments, the module 730 may also offset the origin of the left and right optical axes and correspond to the user's CoP (as determined by the module 732) instead of the user's CoR.

[0144] When an optional center of perspective (CoP) estimation module 732 is provided, it may estimate the locations of the user's left and right centers of perspective (CoP). The CoP is a useful location for a wearable system and, in at least some embodiments, can be the location directly in front of the pupil. In at least some embodiments, the CoP estimation module 732 may estimate the locations of the user's left and right centers of perspective based on the 3D location of the user's pupil center, the 3D location of the center of the user's corneal curvature, or such suitable data, or any combination thereof. As an example, the user's CoP can be approximately 5.01 mm in front of the center of the corneal curvature (i.e., 5.01 mm in the direction along the optical axis from the center of the corneal sphere towards the eye's cornea) and can be approximately 2.97 mm behind the outer surface of the user's cornea along the optical or visual axis. The user's center of perspective can be directly in front of the center of their pupil. As an example, the user's CoP can be less than about 2.0 mm from the user's pupil, less than about 1.0 mm from the user's pupil, or less than about 0.5 mm from the user's pupil, or any range between these values. As another example, the center of perspective can correspond to a location within the anterior chamber of the eye. In other examples, the CoP can be between 1.0 mm and 2.0 mm, about 1.0 mm, between 0.25 mm and 1.0 mm, between 0.5 mm and 1.0 mm, or between 0.25 mm and 0.5 mm.

[0145] (As the potentially desirable position of the pinhole of the rendering camera and the anatomical position within the user's eye) The center of perspective described herein can be a position that serves to reduce and / or eliminate undesirable parallax shifts. In particular, the optical system of the user's eye closely approximates a theoretical system formed by the projection of the front pinhole of the lens onto the screen, where the pinhole, lens, and screen roughly correspond to the user's pupil / iris, lens, and retina, respectively. Further, when two point light sources (or objects) at different distances from the user's eye rotate precisely about the opening of the pinhole (e.g., rotated along a radius of curvature equal to its individual distance from the opening of the pinhole), it may be desirable for there to be little or no parallax shift. Thus, the CoP would be considered to be located at the center of the pupil of the eye (and such a CoP may be used in some embodiments). However, the human eye includes the cornea, which, in addition to the pinholes of the lens and pupil, imparts additional refractive power to the light propagating towards the retina. Thus, the anatomical equivalent of the pinhole within the theoretical system described in this paragraph can be the region of the user's eye located between the outer surface of the user's eye's cornea and the center of the user's eye's pupil or iris. For example, the anatomical equivalent of the pinhole can correspond to a region within the anterior chamber of the user's eye. For various reasons discussed herein, it may be desirable to set the CoP at such a position within the anterior chamber of the user's eye.

[0146] As discussed above, the eye tracking module 614 may provide data such as the estimated 3D positions of the left and right eye centers of rotation (CoR), the vergence / accommodation depth, the left and right eye optical axes, the 3D positions of the user's eyes, the 3D positions of the left and right centers of the user's corneal curvature, the 3D positions of the user's left and right pupil centers, the 3D positions of the user's left and right fixation centers, the user's IPD, etc. to other components such as the light field rendering controller 618 and the alignment observer 620 within the wearable system. The eye tracking module 614 may also include other sub-modules that detect and generate data associated with other aspects of the user's eyes. As an example, the eye tracking module 614 may include a blink detection module that provides a flag or other alert each time the user blinks, and a saccade detection module that provides a flag or other alert each time the user's eyes saccade (i.e., rapidly shift focus to another point).

[0147] Example of a rendering controller A detailed block diagram of an exemplary light field rendering controller 618 is shown in FIG. 7B. As shown in FIGS. 6 and 7B, the rendering controller 618 may receive eye tracking information from the eye tracking module 614 and may provide an output to the rendering engine 622, which may generate an image to be displayed for viewing by a user of the wearable system. As an example, the rendering controller 618 may receive information regarding the vergence / accommodation depth, the left and right eye centers of rotation (and / or fixation centers), and other eye data such as blink data, saccade data, etc.

[0148] The depth plane selection module 750 may receive the convergence / divergence motion depth information, and based on such data, cause the rendering engine 622 to provide the user with the content in a state where the content appears to be located on a specific depth plane (i.e., a specific perspective adjustment or focal length). As discussed in relation to FIG. 4, the wearable system may include a plurality of discrete depth planes formed by a plurality of waveguides that each transmit image information with a variable level of wavefront curvature. In some embodiments, the wearable system may include one or more variable depth planes, such as optical elements, that transmit image information with a level of wavefront curvature that varies over time. In these and other embodiments, the depth plane selection module 750 may cause the rendering engine 622 to transmit the content to the user at a selected depth, at least in part, based on the user's convergence / divergence motion depth (i.e., cause the rendering engine 622 to instruct the display 220 to switch depth planes). In at least some embodiments, the depth plane selection module 750 and the rendering engine 622 may render the content at different depths, and also generate and / or provide depth plane selection data to a display hardware such as the display 220. The display hardware such as the display 220 may perform electrical depth plane switching in response to depth plane selection data (which may be control signals) generated and / or provided by modules such as the depth plane selection module 750 and the rendering engine 622.

[0149] Generally, it may be desirable for the depth plane selection module 750 to select a depth plane that matches the user's current convergence / divergence motion depth so that the user is provided with an accurate perspective adjustment cue. However, it may also be desirable to switch depth planes in a careful and unobtrusive manner. As an example, it may be desirable to avoid excessive switching between depth planes and / or to switch depth planes at times when the user is less likely to notice the switch, such as during a blink or an eye saccade.

[0150] The hysteresis band intersection detection module 752 can be particularly useful for avoiding excessive switching between depth planes, especially when the depth of the user's convergence / divergence movement varies at the midpoint or transition point between two depth planes. In particular, module 752 may cause the depth plane selection module 750 to exhibit hysteresis in its selection of the depth plane. As an example, module 752 may cause the depth plane selection module 750 to switch from a first, more distant depth plane to a second, closer depth plane only after the depth of the user's convergence / divergence movement has passed a first threshold. Similarly, module 752 may cause the depth plane selection module 750 (and thus, can instruct a display such as display 220) to switch to the first, more distant depth plane only after the depth of the user's convergence / divergence movement has passed a second threshold that is farther from the user than the first threshold. In the overlapping region between the first threshold and the second threshold, module 750 may cause the depth plane selection module 750 to maintain whichever depth plane is currently selected as the selected depth plane, and thus, may avoid excessive switching between depth planes.

[0151] The eye event detection module 750 may receive other eye data from the eye tracking module 614 of FIG. 7A, and may delay several depth plane switches to the depth plane selection module 750 until an eye event occurs. As an example, the eye event detection module 750 may delay the planned depth plane switch to the depth plane selection module 750 until a user blink is detected, and may receive data indicating that the user is currently blinking from a blink detection component within the eye tracking module 614, and in response, may cause the depth plane selection module 750 to perform the planned depth plane switch during the blink event (such as by instructing the module 750 to perform the depth plane switch on the display 220 during the blink event). In at least some embodiments, the wearable system may be able to shift the content onto a new depth plane during a blink event such that the user is less likely to perceive the shift. As another example, the eye event detection module 750 may delay the planned depth plane switch until an eye saccade is detected. As discussed in connection with eye blinks, such an arrangement may facilitate discrete shifts in the depth plane.

[0152] Optionally, the depth plane selection module 750 may delay the planned depth plane switch only for a limited time period before performing the depth plane switch even in the absence of an eye event. Similarly, the depth plane selection module 750 may perform the depth plane switch even in the absence of an eye event when the user's convergence / divergence motion depth is substantially outside the currently selected depth plane (i.e., when the user's convergence / divergence motion depth exceeds a predetermined threshold that exceeds the normal threshold for depth plane switching). These arrangements may help ensure that the eye event detection module 754 does not indefinitely delay the depth plane switch and does not delay the delayed depth plane switch when there is a large depth of field adjustment error.

[0153] The rendering camera controller 758 may provide information indicating the locations of the user's left and right eyes to the rendering engine 622. The rendering engine 622 may then generate content by simulating the camera at the locations of the user's left and right eyes and generating the content based on the viewpoints of the simulated cameras. As discussed above, the rendering camera may potentially be a simulated camera for use in rendering virtual image content from a database of objects within a virtual world. The objects may have locations and orientations relative to the user or wearer, potentially relative to real objects within an environment surrounding the user or wearer. The rendering camera may be included within the rendering engine and may render a virtual image based on a database of virtual objects to be presented to the eyes. The virtual image may be rendered as if it were captured from the viewpoint of the user or wearer. For example, the virtual image may be rendered as if it were captured by a camera (corresponding to the "rendering camera") having an aperture, lens, and detector for viewing objects within the virtual world. The virtual image is captured from the viewpoint of such a camera having the position of the "rendering camera". For example, the virtual image may be rendered as if it were captured from a camera viewpoint having a specific location relative to the user or wearer's eyes such that the virtual image provides an image that appears to be from the viewpoint of the user or wearer. In some implementations, the image may be rendered as if it were captured from a camera viewpoint having an aperture at a specific location relative to the user or wearer's eyes (such as a viewpoint center or rotation center or other location as discussed herein).

[0154] The rendering camera controller 758 may determine the positions of the left and right cameras based on the left and right centers of rotation (CoR) determined by the CoR estimation module 724 and / or based on the left and right centers of perspective (CoP) determined by the CoP estimation module 732. In some embodiments, the rendering camera controller 758 may switch between the CoR location and the CoP location based on various factors. As an example, the rendering camera controller 758, in various modes, may constantly align the rendering camera to the CoR location, constantly align the rendering camera to the CoP location, toggle between aligning the rendering camera to the CoR location and aligning the rendering camera to the CoP location over time based on various factors, or discretely switch, or dynamically align the rendering camera to any of a range of different positions along the optical (or visual) axis between the CoR location and the CoP location over time based on various factors. The CoR and CoP positions may optionally pass through a smoothing filter 756 (in any of the aforementioned modes for rendering camera positioning), which may average the CoR and CoP locations over time, reduce noise at these positions, and prevent jitter when rendering the simulated rendering camera.

[0155] In at least some embodiments, the rendering camera may be simulated as a pinhole camera with a pinhole located at the estimated CoR or CoP position identified by the eye tracking module 614. Since the CoP is offset from the CoR, whenever the position of the rendering camera is based on the user's CoP, the locations of both the rendering camera and its pinhole are offset as the user's eyes rotate. In contrast, whenever the position of the rendering camera is based on the user's CoR, the location of the pinhole of the rendering camera does not move with eye rotation, but the rendering camera (behind the pinhole) may move with eye rotation in some embodiments. In other embodiments where the position of the rendering camera is based on the user's CoR, the rendering camera may not move (i.e., rotate) with the user's eyes.

[0156] Example of an alignment observer A block diagram of an exemplary alignment observer 620 is shown in FIG. 7C. As shown in FIGS. 6, 7A, and 7C, the alignment observer 620 may receive eye tracking information from the eye tracking module 614 (FIGS. 6 and 7A). As an example, the alignment observer 620 may receive information regarding the user's left and right eye rotation centers (e.g., the three-dimensional positions of the user's left and right eye rotation centers that may be on a common coordinate system or have a reference common frame with the head-mounted display system 600). As another example, the alignment observer 620 may receive display attendant properties, fit tolerance, and an eye tracking valid indicator. The display attendant properties may include information regarding the display (e.g., the display 200 of FIG. 2) such as the field of view of the display, the size of one or more display surfaces, and the position of the display surface relative to the head-mounted display system 600. The fit tolerance may include information regarding the display alignment volume that may indicate the distance by which the user's left and right eyes may move from their nominal positions before the display performance is affected. Additionally, the fit tolerance may indicate the amount of display performance impact expected as a function of the position of the user's eyes.

[0157] As shown in FIG. 7C, the alignment observer 620 may include a 3D position fit sense module 770. The position fit sense module 770 may, as an example, acquire and analyze various data including the 3D position of the left eye rotation center (e.g., CoR left), the 3D position of the right eye rotation center (e.g., CoR right), the display-related properties, and the fit sense tolerance. The 3D position fit sense module 770 may determine the distances of the user's left and right eyes from the individual left and right eye nominal positions (e.g., may calculate the 3D left error and the 3D right error), and may provide the error distances (e.g., the 3D left error and the 3D right error) to the device 3D fit sense module 772.

[0158] The 3D position fit sense module 770 may also compare the error distances to the display-related properties and the fit sense tolerance to determine whether the user's eyes are within a nominal volume, a partially degraded volume (e.g., a volume in which the performance of the display 220 is partially degraded), or a completely degraded or almost completely degraded volume (e.g., a volume in which it is substantially impossible for the display 220 to provide content to the user's eyes). In at least some embodiments, the 3D position fit sense module 770 or the 3D fit sense module 772 may provide an output that qualitatively describes the fit sense of the HMD on the user, such as the fit sense quality output shown in FIG. 7C. As an example, the module 770 may provide an output indicating whether the current fit sense of the HMD on the user is good, within tolerance, or a failure. A good fit sense may correspond to a fit sense that enables the user to view at least a certain percentage (such as 90%) of the image, a within-tolerance fit sense may enable the user to view at least a lower percentage (such as 80%) of the image, while a failed fit sense may be a fit sense in which only an even lower percentage of the image is visible to the user.

[0159] As another example, the 3D position fit feeling module 770 and / or the device 3D fit feeling module 772 may calculate a visible area metric, which may be a percentage of the overall area (or pixels) of the image displayed by the display 220 visible to the user. Modules 770 and 772 may use one or more models (e.g., mathematical or geometric models), one or more look-up tables, or other techniques for determining the percentage of the image visible to the user as a function of the position of the user's eyes, or a combination of these and other techniques, to calculate the visible area metric by evaluating the positions of the user's left and right eyes relative to the display 220 (e.g., which may be based on the center of rotation of the user's eyes). Additionally, modules 770 and 772 may determine the area or portion of the image displayed by the display 220 that is expected to be visible to the user as a function of the position of the user's eyes.

[0160] The alignment observer 620 may also include a device 3D fit feeling module 772. Module 772 may receive data from the 3D position fit feeling module 770 and may also receive an eye tracking valid indicator, which may be provided by the eye tracking module 614 and may indicate whether the eye tracking system is currently tracking the position of the user's eyes or whether the eye tracking data is unavailable or under error conditions (e.g., determined to be unreliable). The device 3D fit feeling module 772 may, if desired, modify the fit feeling quality data received from the 3D position fit feeling module 770 according to the state of the eye tracking valid data. For example, if data from the eye tracking system is shown to be unavailable or have an error, the device 3D fit feeling module 772 may provide a notification of the existence of the error and / or may not provide an output to the user regarding the quality of the fit feeling or the fit feeling error.

[0161] In at least some embodiments, the alignment observer 620 may provide feedback to the user regarding the quality of the fit as well as details of the nature and magnitude of the error. By way of example, a head-mounted display system may provide feedback to the user during a calibration or fitting process (e.g., as part of a setup procedure), and may also provide feedback during operation (e.g., if the fit degrades due to slippage, the alignment observer 620 may prompt the user to readjust the head-mounted display system). In some embodiments, the alignment analysis may be performed automatically (e.g., during use of the head-mounted display system), and the feedback may be provided without user input. These are merely illustrative examples.

[0162] Example of identifying the user's cornea using an eye tracking system FIG. 8A is a schematic view of an eye showing the corneal sphere of the eye. As shown in FIG. 8A, the user's eye 810 may have a cornea 812, a pupil 822, and a lens 820. The cornea 812 may have a generally spherical shape, represented by the corneal sphere 814. The corneal sphere 814 may have a center point 816, also referred to as the corneal center, and a radius 818. The hemispherical cornea of the user's eye may curve around the corneal center 816.

[0163] FIGS. 8B-8E illustrate an example of identifying the user's corneal center 816 using the 3D corneal center estimation module 716 and the eye tracking module 614.

[0164] As shown in FIG. 8B, the 3D corneal center estimation module 716 may receive an eye tracking image 852 that includes a corneal flash 854. The 3D corneal center estimation module 716 may then simulate the known 3D positions of the eye camera 324 and the light source 326 (which may be obtained based on data in the eye tracking incidental and intrinsic property database 702, the assumed eye dimension database 704, and / or the per-user calibration data 706) within the eye camera coordinate system 850 in order to project a light ray 856 within the eye camera coordinate system. In at least some embodiments, the eye camera coordinate system 850 may have its origin at the 3D position of the eye tracking camera 324.

[0165] In FIG. 8C, the 3D corneal center estimation module 716 simulates a corneal sphere 814a (which may be obtained based on the assumed eye dimensions from the database 704) and a corneal curvature center 816a at a first position. The 3D corneal center estimation module 716 may then check whether the corneal sphere 814a would appropriately reflect light from the light source 326 to the flash position 854. As shown in FIG. 8C, the first position does not match because the light ray 860a does not intersect the light source 326.

[0166] Similar to FIG. 8D, the 3D corneal center estimation module 716 simulates a corneal sphere 814b and a corneal curvature center 816b at a second position. The 3D corneal center estimation module 716 then checks whether the corneal sphere 814b would appropriately reflect light from the light source 326 to the flash position 854. As shown in FIG. 8D, the second position also does not match.

[0167] As shown in FIG. 8E, the 3D corneal center estimation module 716 can ultimately determine that the correct position of the corneal sphere is the corneal sphere 814c and the corneal curvature center 816c. The 3D corneal center estimation module 716 checks that the light from the source 326 will be properly reflected from the corneal sphere and imaged at the correct location of the flash 854 on the image 852 by the camera 324, thereby confirming that the illustrated position is correct. Using this arrangement, and the known 3D positions of the light source 326, the camera 324, and the optical properties (such as focal length) of the camera, the 3D corneal center estimation module 716 can determine the 3D location of the center 816 of the corneal curvature (with respect to the wearable system).

[0168] The processes described herein, at least in relation to FIGS. 8C - 8E, can in fact be an iterative, repetitive, or optimization process for identifying the 3D position of the user's corneal center. Thus, any of a plurality of techniques (e.g., iterative, optimization techniques, etc.) may be used to efficiently and quickly sort through, or reduce the search space of possible positions. Further, in some embodiments, the system may include two, three, four, or more light sources such as the light source 326, some of all of which may be arranged at different positions, resulting in multiple flashes such as the flash 854 located at different positions on the image 852 and multiple light rays such as the light ray 856 having different origins and directions. Such embodiments can improve the accuracy of the 3D corneal center estimation module 716 because the module 716 can search for corneal positions that result in some or all of the flashes and light rays being properly reflected between their individual light sources and their individual positions on the image 852. In other words, in these embodiments, the positions of some or all of the light sources can rely on the 3D corneal positioning (e.g., iterative, optimization techniques, etc.) process of FIGS. 8B - 8E.

[0169] Example of normalizing the coordinate system of the eye tracking image Figures 9A-9C illustrate an exemplary normalization of the coordinate system of an eye-tracking image by components within a wearable system, such as the coordinate system normalization module 718 of FIG. 7A. Normalization of the coordinate system of the eye-tracking image with respect to the location of the user's pupil may compensate for slippage of the wearable system with respect to the user's face (i.e., headset slippage), and such normalization may establish a consistent orientation and distance between the eye-tracking image and the user's eye.

[0170] As shown in FIG. 9A, the coordinate system normalization module 718 may receive the estimated 3D coordinates 900 of the center of rotation of the user's cornea and may receive an unnormalized eye-tracking image, such as image 852. The eye-tracking image 852 and the coordinates 900 may be in an unnormalized coordinate system 850, based on the location of the eye-tracking camera 324, as an example.

[0171] In a first normalization step, the coordinate system normalization module 718 may rotate the coordinate system 850 into a rotated coordinate system 902 such that the z-axis of the coordinate system (i.e., the vergence / accommodation depth axis) may be aligned with the vector between the origin of the coordinate system and the corneal curvature center coordinates 900, as shown in FIG. 9B. In particular, the coordinate system normalization module 718 may rotate the eye-tracking image 850 into a rotated eye-tracking image 904 until the coordinates 900 of the user's corneal curvature center are normal to the plane of the rotated image 904.

[0172] As a second normalization step, the coordinate system normalization module 718 may translate the rotated coordinate system 902 into a normalized coordinate system 910 such that the corneal curvature center coordinates 900 are at a standard normalized distance 906 from the origin of the normalized coordinate system 910, as shown in FIG. 9C. In particular, the coordinate system normalization module 718 may translate the rotated eye-tracking image 904 into a normalized eye-tracking image 912. In at least some embodiments, the standard normalized distance 906 may be about 30 millimeters. Optionally, the second normalization step may be performed prior to the first normalization step.

[0173] Example of identifying the center of gravity of a user's pupil using an eye tracking system Figures 9D - 9G illustrate an example of using the 3D pupil center locator module 720 and the eye tracking module 614 to identify the center of the user's pupil (i.e., the center of the user's pupil 822 as shown in Figure 8A).

[0174] As shown in Figure 9D, the 3D pupil center locator module 720 may receive a normalized eye tracking image 912 that includes the center of gravity 913 of the pupil (i.e., the center of the user's pupil as identified by the pupil identification module 712). The 3D pupil center locator module 720 may then simulate the normalized 3D position 910 of the eye camera 324 and project a light ray 914 through the center of gravity 913 within the normalized coordinate system 910.

[0175] In Figure 9E, the 3D pupil center locator module 720 may simulate a corneal spherical surface such as the corneal spherical surface 901 having a center of curvature 900 based on data from the 3D corneal center estimation module 716 (and as discussed in more detail in relation to Figures 8B - 8E). As an example, the corneal spherical surface 901 may be positioned within the normalized coordinate system 910 based on the location of the center of curvature 816c identified in relation to Figure 8E and based on the normalization process of Figures 9A - 9C. In addition, the 3D pupil center locator module 720 may identify a first intersection 916 between the light ray 914 (i.e., the light ray between the origin of the normalized coordinate system 910 and the normalized location of the user's pupil) and the simulated cornea as shown in Figure 9E.

[0176] As shown in FIG. 9F, the 3D pupil center locator module 720 may determine a pupil sphere 918 based on the corneal sphere 901. The pupil sphere 918 shares a common center of curvature with the corneal sphere 901, but may have a smaller radius. The 3D pupil center locator module 720 may determine the distance between the corneal center 900 and the pupil sphere 918 (i.e., the radius of the pupil sphere 918) based on the distance between the corneal center and the pupil center. In some embodiments, the distance between the pupil center and the corneal center of curvature may be determined from the assumed eye dimensions 704 of FIG. 7A, from the eye tracking incidental and intrinsic property database 702, and / or from the per-user calibration data 706. In other embodiments, the distance between the pupil center and the corneal center of curvature may be determined from the per-user calibration data 706 of FIG. 7A.

[0177] As shown in FIG. 9G, the 3D pupil center locator module 720 may locate the 3D coordinates of the user's pupil center based on various inputs. As an example, the 3D pupil center locator module 720 may use the 3D coordinates and radius of the pupil sphere 918, the 3D coordinates of the intersection 916 between the simulated corneal sphere 901 and the ray 914 associated with the pupil centroid 913 in the normalized eye tracking image 912, information regarding the refractive index of the cornea, and other relevant information such as the refractive index of air (which may be stored in the eye tracking incidental and intrinsic property database 702) to determine the 3D coordinates of the center of the user's pupil. In particular, in the simulation, the 3D pupil center locator module 720 may bend the ray 916 into the refracted ray 922 based on the refractive difference between air (at a first refractive index of approximately 1.00) and the corneal material (at a second refractive index of approximately 1.38). After considering the refraction caused by the cornea, the 3D pupil center locator module 720 may determine the 3D coordinates of the first intersection 920 between the refracted ray 922 and the pupil sphere 918. The 3D pupil center locator module 720 may determine that the user's pupil center 920 is located around the first intersection 920 between the refracted ray 922 and the pupil sphere 918. By using this arrangement, the 3D pupil center locator module 720 may determine the 3D location of the pupil center 920 (with respect to the wearable system) within the normalized coordinate system 910. Optionally, the wearable system may denormalize the coordinates of the pupil center 920 into the original eye camera coordinate system 850. The pupil center 920 may be used together with the corneal curvature center 900 to determine, among other things, the user's optical axis using the optical axis determination module 722 and the user's convergence / divergence motion depth using the convergence / divergence motion depth estimation module 728.

[0178] Example of the difference between the optical axis and the visual axis As discussed in connection with the optical axis / view axis mapping module 730 of FIG. 7A, the user's optical axis and view axis are generally not aligned, in part because the user's view axis is defined by the fovea, which is generally not at the center of a person's retina. Thus, when a person desires to focus on a particular object, the person aligns their view axis with the object, ensuring that light from the object strikes the fovea while their optical axis (defined by the center of their pupil and the center of curvature of their cornea) is actually slightly offset from the object. FIG. 10 is an example of an eye 1000 illustrating the optical axis 1002 of the eye, the view axis 1004 of the eye, and the offset between these axes. Additionally, FIG. 10 illustrates the pupil center 1006 of the eye, the center of curvature 1008 of the eye's cornea, and the average center of rotation (CoR) 1010 of the eye. In at least some populations, the center of curvature 1008 of the eye's cornea can be approximately 4.7 mm in front of the average center of rotation (CoR) 1010 of the eye, as indicated by dimension 1012. Additionally, the center of the eye's point of regard 1014 can be approximately 5.01 mm in front of the center of curvature 1008 of the eye's cornea, approximately 2.97 mm behind the outer surface 1016 of the user's cornea, and / or directly in front of the user's pupil center 1006 (e.g., corresponding to a location within the anterior chamber of the eye 1000). As an additional example, dimension 1012 can be in the range of 3.0 mm to 7.0 mm, 4.0 to 6.0 mm, 4.5 to 5.0 mm, or 4.6 to 4.8 mm, or any range between any value within these ranges. The center of the eye's point of regard (CoP) 1014 can be a useful location for a wearable system in at least some embodiments, as aligning a rendering camera with the CoP can help reduce or eliminate parallax artifacts.

[0179] FIG. 10 also illustrates such a location within the human eye 1000 that can be aligned with the pinhole of the rendering camera. As shown in FIG. 10, the pinhole of the rendering camera may be aligned with a location 1014 along the optical axis 1002 or visual axis 1004 of the human eye 1000 that is closer to the outer surface of the cornea than both (a) the center of the pupil or iris 1006 and (b) the center 1008 of the corneal curvature of the human eye 1000. For example, as shown in FIG. 10, the pinhole of the rendering camera may be aligned with a location 1014 along the optical axis 1002 of the human eye 1000 that is about 2.97 millimeters rearward from the outer surface of the cornea and about 5.01 millimeters forward from the center 1008 of the corneal curvature. The location 1014 of the pinhole of the rendering camera and / or the anatomical region of the human eye 1000 corresponding to the location 1014 therewith may be regarded as representing the center of the viewing point of the human eye 1000. The optical axis 1002 of the human eye 1000 as shown in FIG. 10 represents the shortest line passing through the center 1008 of the corneal curvature and the center of the pupil or iris 1006. The visual axis 1004 of the human eye 1000 is different from the optical axis 1002 because it represents a line extending from the fovea of the human eye 1000 to the center of the pupil or iris 1006.

[0180] Exemplary process for rendering content and checking alignment based on eye tracking FIG. 11 is a process flow diagram of an exemplary method 1100 for providing feedback regarding alignment within a wearable device using eye tracking when rendering content. The method 1100 may be implemented by the wearable system described herein. Embodiments of the method 1100 may be used by the wearable system to render content and provide feedback regarding alignment (i.e., the fit of the wearable device with the user) based on data from an eye tracking system.

[0181] In block 1110, the wearable system may capture an image of one or both of the user's eyes. The wearable system may capture the eye image using one or more eye cameras 324, at least as shown in the embodiment of FIG. 3. Optionally, the wearable system may also include one or more light sources 326 configured to shine IR light on the user's eyes and produce a corresponding flash within the eye image captured by the eye camera 324. As discussed herein, the flash may be used by the eye tracking module 614 to derive various information about the user's eyes, including where the eyes are looking.

[0182] In block 1120, the wearable system may detect the flash and the pupil within the eye image captured in block 1110. As an example, block 1120 may include processing the eye image by a flash detection and labeling module 714 to identify the 2D position of the flash within the eye image, and processing the eye image by a pupil identification module 712 to identify the 2D position of the pupil within the eye image.

[0183] In block 1130, the wearable system may estimate the 3D positions of the user's left and right corneas relative to the wearable system. As an example, the wearable system may estimate the positions of the centers of curvature of the user's left and right corneas and the distances between those centers of curvature and the user's left and right corneas. Block 1130 may be accompanied by a 3D corneal center estimation module 716 that identifies the positions of the centers of curvature, at least as described herein in connection with FIGS. 7A and 8A-8E.

[0184] In block 1140, the wearable system may estimate the 3D positions of the user's left and right pupil centers relative to the wearable system. As an example, the wearable system and a 3D pupil center locator module 720 may estimate the positions of the user's left and right pupil centers as part of block 1140, particularly as described in connection with FIGS. 7A and 9D-9G.

[0185] In block 1150, the wearable system may estimate the three-dimensional position of the user's left and right centers of rotation (CoR) relative to the wearable system. As an example, the wearable system and the CoR estimation module 724 may estimate the position of the CoR with respect to the user's left and right eyes, particularly as described in connection with at least FIGS. 7A and 10. In a particular example, the wearable system may find the eye's CoR by tracing back along the optical axis from the center of curvature of the cornea towards the retina.

[0186] In block 1160, the wearable system may estimate the user's IPD, vergence / accommodation depth, center of perspective (CoP), optical axis, visual axis, and other desired attributes from the eye-tracking data. As an example, as part of block 1160, the IPD estimation module 726 may estimate the user's IPD by comparing the 3D positions of the left and right CoRs, the vergence / accommodation depth estimation module 728 may estimate the user's depth by finding the intersection (or near-intersection) of the left and right optical axes or the intersection of the left axis and the right visual axis, the optical axis determination module 722 may identify the left and right optical axes over time, the optical axis / visual axis mapping module 730 may identify the left and right visual axes over time, and the CoP estimation module 732 may identify the left and right centers of perspective.

[0187] In block 1170, the wearable system may render content and optionally provide feedback regarding alignment (i.e., the fit of the wearable system to the user's head) based in part on the eye-tracking data identified in blocks 1120 - 1160. As an example, the wearable system may identify a suitable location for the rendering camera, as discussed in relation to the light field rendering controller 618 (FIG. 7B) and the rendering engine 622, and then generate content for the user based on the location of the rendering camera. As another example, the wearable system may determine whether it is properly fitted to the user or has slipped from its proper location relative to the user, as discussed in relation to the alignment observer 620 and as discussed in relation to block 1608 of FIG. 16, and may optionally provide the user with feedback indicating whether adjustment of the fit of the device is necessary. In some embodiments, the wearable system may adjust the rendered content based on improper or sub-optimal alignment in an attempt to reduce, minimize, or compensate for the effects of improper or misaligned alignment, as discussed in relation to block 1610 of FIG. 16.

[0188] Overview of Device Alignment The wearable system 200 described herein, in order to output high-perception high-quality images, the display 220 of the wearable system 200 (FIG. 2) is preferably appropriately fitted to the user (e.g., the input and output of the system 200 interface appropriately with the corresponding parts of the user's head and the device is positioned and oriented with respect to the user's head so as to be stable and comfortable for wearing and use). As an example, for the display 220 to provide visual content to the user's eyes, the display 220 is preferably positioned in front of the user's eyes, and depending on the relevant properties of the display 220, the user's eyes are preferably positioned within a specific volume (see further discussion associated with FIGS. 13A and 13B). As an additional example, the speaker 240 is preferably positioned near, on, or in the user's ear to provide high-quality audio content to the user, the audio sensor (e.g., microphone) 232 is preferably positioned within a specific area to receive sound from the user, and the inward-facing imaging system 462 (which may include one or more cameras 324 and one or more infrared light sources 326) is preferably positioned in an appropriate position and orientation for acquiring a clear and unobstructed image of the user's eyes (which may be part of an eye tracking system). These are simply examples of various reasons why the wearable system 200 is preferably appropriately fitted to the user.

[0189] To ensure that the wearable system 200 is properly aligned with the user, the wearable system 200 may include an alignment observer such as the alignment observer 620 of FIG. 6. In some embodiments, a properly aligned wearable system 200 includes a display positioned such that one or both of the user's eyes receive sufficient image light and can view substantially the entire field of view provided by the display 220 of the wearable display system 200. For example, a properly aligned display may allow an image to be visible across about 80% or more, about 85% or more, about 90% or more, or about 95% or more of the field of view of the display, with a brightness uniformity of 80% or more, about 85% or more, about 90% or more, or about 95% or more. Brightness uniformity may be equal to 100% multiplied by the minimum luminance divided by the maximum luminance across the entire field of view of the display when the display shows the same content across the entire field of view (100%×L min / L max ). It should be understood that it can be equal to).

[0190] The alignment observer 620 may use various sensors to determine the extent to which the wearable system 200 fits on the user (e.g., whether the display 220 of the wearable system 200 is properly positioned on the user). As an example, the alignment observer 620 may use an inward-facing imaging system 462, which may include an eye-tracking system, to determine the extent to which relevant portions of the wearable system 200 are spatially oriented relative to the user, particularly the user's eyes, ears, mouth, or other portions that interface with the wearable system 200.

[0191] The alignment observer 620 may assist in the calibration process of an initial or subsequent configuration or setting, etc. of the wearable system 200 for a particular user. As an example, the alignment observer 620 may provide feedback to the user during the configuration or setting of the wearable system 200 for that particular user. Additionally, or alternatively, the alignment observer 620 may continuously or intermittently monitor the alignment of the wearable system 200 on the user, check for continuous proper alignment during use, and may provide on-the-fly user feedback. The alignment observer 620 may provide user feedback indicating when the wearable system 200 is properly aligned and when it is not, either as part of the configuration process or as part of the alignment monitoring during use. The alignment observer 620 may also provide specific recommendations for how the user may correct any alignment misalignment and achieve proper alignment. As an example, after detecting slippage of the wearable device (such as below the user's nasal bridge), the alignment observer 620 may recommend that the user push the wearable device back up, and may recommend that the user adjust some adjustable components of the wearable device (such as as described herein in connection with FIGS. 15A and 15B), etc.

[0192] Example of an alignment coordinate system FIGS. 12A-12B illustrate an exemplary eye position coordinate system that may be used to define the three-dimensional positions of a user's left and right eyes relative to the display of the wearable system described herein. As an example, the coordinate system may include axes x, y, and z. The axis z of the coordinate system may correspond to a depth (e.g., in the direction normal to the plane of the front of the user's face) such as the distance between the plane in which the user's eyes are located and the plane in which the display 220 is located. The axis x of the coordinate system may correspond to the left-right direction such as the distance between the user's left and right eyes. The axis y of the coordinate system may correspond to the up-down direction, which may be the vertical direction when the user is standing upright.

[0193] FIG. 12A illustrates a side view of the user's eye 1200 and a display surface 1202 (which may be part of the display 220 of FIG. 2), while FIG. 12B illustrates a top-down view of the user's eye 1200 and the display surface 1202. The display surface 1202 may be positioned in front of the user's eyes and output image light to the user's eyes. As an example, the display surface 1202 may include one or more externally coupled light elements, active or pixel display elements, and may be part of a stack of waveguides such as the stacked waveguide assembly 480 of FIG. 4. In some embodiments, the display surface 1202 may be planar. In some other embodiments, the display surface 1202 may have other topologies (e.g., curved). It should be understood that the display surface 1202 may be the physical surface of the display or simply a plane or other imaginary surface from which image light is understood to propagate from the display 220 to the user's eyes.

[0194] As shown in FIG. 12A, the user's eye 1200 may have an actual position 1204 offset from the nominal position 1206, and the display surface 1202 may be at position 1214. FIG. 12A also illustrates the corneal apex 1212 of the user's eye 1200. The user's line of sight (e.g., its optical axis and / or visual axis) may be substantially along the line between the actual position 1204 and the corneal apex 1212. As shown in FIGS. 12A and 12B, the actual position 1204 may be offset from the nominal position 1206 by a z-offset 1210, a y-offset 1208, and an x-offset 1209. The nominal position 1206 may represent a preferred position for the user's eye 1200 relative to the display surface 1202 (sometimes also referred to as the design position, which can generally be centered within a desired volume). As the user's eye 1200 moves away from the nominal position 1206, the performance of the display surface 1202 may degrade, as discussed herein in connection with FIG. 14, for example.

[0195] It should be understood that a point or volume associated with the user's eye 1200 can be used to represent the position of the user's eye in the alignment analysis herein. The representative point or volume can preferably be any point or volume associated with the eye 1200 that is used consistently. For example, the point or volume may be on or within the eye 1200 or may be arranged away from the eye 1200. In some embodiments, the point or volume is the center of rotation of the eye 1200. The center of rotation can be determined as described herein and has the advantage of simplifying the alignment analysis to allow a single display alignment volume that is generally symmetrically arranged on various axes within the eye 1200 and aligned with the optical axis to be utilized for the analysis.

[0196] FIG. 12A also illustrates that the display surface 1202 can be centered below the user's horizontal line (as seen along the y-axis when the user is looking straight ahead with their optical axis parallel to the ground) and can be tilted (with respect to the y-axis). In particular, the display surface 1202 can be positioned slightly below the user's horizontal line such that the user would need to look down at approximately angle 1216 to view the center of the display surface 1202 when the eye 1200 is at position 1206. This can promote a more natural and comfortable interaction with the display surface 1202, especially when viewing content rendered at a lower depth (or distance from the user), as the user may find it more comfortable to view content below their horizontal line than above it. Additionally, the display surface 1202 can be tilted at an angle 1218, etc. (with respect to the y-axis) such that the display surface 1202 is substantially perpendicular to the user's line of sight when the user is looking at the center of the display surface 1202 (e.g., looking slightly below the user's horizontal line). In at least some embodiments, the display surface 1202 may also be offset left or right (e.g., along the x-axis) with respect to the nominal position of the user's eyes. As an example, when the user's line of sight is focused at a distance less than infinity, the left-eye display surface may be offset to the right and the right-eye display surface may be offset to the left (e.g., the display surfaces 1202 may be offset towards each other) such that the line of sight strikes the center of the display surface, which can increase user comfort during typical use on a wearable device.

[0197] Example of a display alignment volume Figures 13A - 13B illustrate an exemplary display alignment volume 1302a. The display alignment volume 1302a may represent a volumetric space in which an eye 1200 is positioned to receive image light from a display device. In some embodiments, the center of rotation of the user's eye is preferably positioned such that the eye aligns with, or receives, image information from the display device. In some embodiments, when the center of rotation of the user's eye is within the display alignment volume 1302a, the user can view the entire image output by the display device with high brightness uniformity. For example, as described herein, a properly aligned display can enable an image to be seen across about 80% or more, about 85% or more, about 90% or more, or about 95% or more of the display's field of view with a brightness uniformity of 80% or more, about 85% or more, about 90% or more, or about 95% or more. In other words, a display with "good" alignment (as determined by, for example, module 772 in FIG. 7C) may have a brightness uniformity of 90% or more, a display with "acceptable" alignment may have a brightness uniformity of 80% or more, and a display with a "failed" alignment may have a brightness uniformity of less than 80%.

[0198] Also, as described herein, the center of rotation 1204 can serve as a convenient reference point for referencing and determining the three-dimensional position of the user's eye. The respective centers of rotation of the user's eyes may be determined using the techniques described herein, such as by tracing back along the user's optical axis from the center of curvature of the cornea to the center of rotation (CoR). However, in general, any desired reference point associated with the user's eye may be utilized in the processes and systems described herein. The display alignment volume 1203 may represent a volume of space within which the display surface 1202 can operate with substantially full potential (e.g., without significant degradation of the type described in connection with FIGS. 15A and 15B of the performance of the display surface 1202). If the user's eye (e.g., the center of rotation 1204 of the user's eye) is not within the alignment volume 1302a, the user may experience degraded performance, and some or all of the content provided by the display surface 1202 may be partially darkened or completely invisible to the user.

[0199] As shown in FIG. 13A, the alignment volume 1302a may have a frustum shape, where its upper portion is typically the portion of a pyramid that remains after being cut by a plane parallel to its base. In other words, the alignment volume 1302a can become larger along the x and y axes when the user's eye is closer to the display surface 1202 (e.g., see FIGS. 12A and 12B), and can become smaller along the x and y axes when the user's eye is farther from the display surface 1202. The frustum is an example of a truncated top where the shear plane within it (e.g., the line where a portion of the original shape is cut) is parallel to the base of the volume. In general, an alignment volume such as volume 1302a may take the shape of a volume that is truncated in any manner, such as by one or more non-parallel shear planes (e.g., as shown in FIG. 13B), or by one or more non-planar shears.

[0200] The dimensions of the alignment volume may depend on the specific implementation of the display surface 1202 and other elements of the wearable system. As an example, FIG. 13B illustrates that the alignment volume 1302b may be angled with respect to the display surface 1202. In the example of FIG. 13B, the portion of the alignment volume 1302b closest to the display surface 1202 is angled away from the display surface 1202 such that as the user's eye moves vertically (in the y direction) at the front of the volume (the z position closest to the display surface 1202), the user's eye would need to move away from the display surface (along the z axis) and remain inside the alignment volume 1302b. In some embodiments, the shape of the alignment volume 1302b may be based on the capabilities of the eye tracking system such that it may not be possible to track the user's eye outside of the angled volume 1302b of FIG. 13B.

[0201] The dimensions and shape of the alignment volume can also depend on the nature of the various parts of the display 220, which may include the display surface 1202. As an example, the display 220 can include one or more waveguides (which can be stacked and can provide multiple convergent / divergent motion cues to the user), an internal coupling element that receives light from an image input device and couples the light into the waveguide, a light dispersing element (sometimes also referred to as an orthogonal pupil expander (OPE)) that is disposed on the waveguide and disperses the light to an external coupling element, and an external coupling element (sometimes also referred to as an exit pupil expander (EPE)) that directs the light towards the viewer's eye, and can be a light field display. In some embodiments, as described herein, the display surface 1202 is the surface or a portion of the surface from which light with image information is output from the display system and forms an image in the user's eye. For example, the display surface 1202 can be an area on the waveguide surface defined by the external coupling element or EPE, and the perimeter of the display surface 1202 is the perimeter of the area defined by the external coupling element or EPE. Further examples and details of light field displays and components of such displays are also described in connection with at least FIGS. 9A-9C of U.S. Provisional Patent Application No. 62 / 642,761, filed Mar. 14, 2018, which is incorporated herein by reference in its entirety.

[0202] In some embodiments, the x dimension of the alignment volume 1302a may range from about 3.0 mm, 3.0 mm, 3.5 mm, 4.0 mm, 4.5 mm, 4.7 mm, 5.0 mm, 5.5 mm, or 6.0 mm, or be less than 3.0 mm or greater than 6.0 mm, along the back of the volume (e.g., the maximum distance along the z-axis from the display surface). Similarly, the y dimension of the alignment volume 1302a may range from about 2.5 mm, 3.0 mm, 3.5 mm, 3.9 mm, 4.0 mm, 4.5 mm, 5.0 mm, 5.5 mm, or 6.0 mm, or be less than 2.5 mm or greater than 6.0 mm, along the back of the volume. At the nominal x and y positions, the z dimension of the alignment volume 1302a may range from about 7.0 mm, 7.5 mm, 8.0 mm, 8.5 mm, 9.0 mm, 9.5 mm, 10.0 mm, 10.5 mm, or 11.0 mm, or be less than 7.0 mm or greater than 11.0 mm. The x and y dimensions may be larger at the front of the volume. As an example, the x and y dimensions of the alignment volume at the front of the volume may range from about 7.0 mm, 7.5 mm, 8.0 mm, 8.5 mm, 8.9 mm, 9.0 mm, 9.5 mm, 10.0 mm, 10.0 mm, 10.5 mm, 11.0 mm, 11.4 mm, 11.5 mm, 12.0 mm, or 12.5 mm, or be less than 7.0 mm or greater than 12.5 mm. As a specific example, the dimensions of the alignment volume may include a z-dimension of about 9 mm, an x-dimension of about 4.7 mm at the back of the volume and about 11.4 mm at the surface of the volume, and a y-dimension of about 3.9 mm at the back of the volume and about 8.9 mm at the front of the volume.

[0203] In at least some embodiments, there may be a plurality of alignment volumes, such as volumes 1302b and 1304, each associated with a different minimum level of display performance. As an example, volume 1304 in FIG. 13B may be smaller than volume 1302 and may represent a volume in which a user perceives all of the content provided by display surface 1202 with 100% brightness uniformity, while the larger volume 1302b may represent a volume in which a user perceives at least 90% of the content provided by display surface 1202 with 100% brightness uniformity.

[0204] FIGS. 13C and 13D illustrate exemplary display alignment volumes configured to use the center of rotation of the eye as a reference point indicating the position of the user's eye and the eye with respect to the display surface. In particular, FIG. 13C illustrates an exemplary positioning of a display alignment volume, such as alignment volume 1302b, within user's eye 1200. In the example of FIG. 13C, the center of rotation 1204 of eye 1200 is generally centered within alignment volume 1302b. Additionally, alignment volume 1302b is illustrated with an exemplary depth of about 9 mm and exemplary dimensions of about 3.5 mm in width and 3 mm in height at the midpoint of the depth axis. As discussed herein, the dimensions of the alignment volume can vary and can be related to the nature of the various components of the wearable system. FIG. 13C also illustrates an eye structure 1340, which may be the lens or pupil of eye 1200.

[0205] FIG. 13D shows a larger context in which user's eye 1200 is generally positioned within alignment volume 1302b and is viewing virtual content 1350 through display surface 1202. As discussed herein, virtual content, such as virtual content 1350, can be provided to the user with convergence / divergence motion and accommodation cues associated with a depth that exceeds the depth of display surface 1202. In other words, virtual content 1350 can appear to a user with eye 1200 to be at a distance away from the user than display 1202. Such an arrangement is illustrated in the example of FIG. 13D.

[0206] Continuing to refer to FIG. 13D, it should be understood that the display alignment volume 1302b can be an imaginary volume having a boundary defined by a projection from the perimeter of the display surface 1202 to a point inside the eye 1200. For example, the projection can define a pyramid, and the display alignment volume 1302b can be a frustum of that pyramid. Thus, the cross-sectional shape of the display alignment volume 1302b along a plane facing the display surface 1202 on the optical axis is similar to the shape created by the perimeter of the display surface 1202. For example, as shown, if the display surface 1202 is square, the cross-sectional shape of the display alignment volume 1302b is also square. Additionally, as also shown, if the center of the display surface 1202 is below the user's horizontal line, the frustum can also be tilted so that the center of the front of the display alignment volume 1302b is also below the user's horizontal line. In some embodiments, it should be understood that the relevant perimeter of the display surface 1202 is the perimeter of the area of the display over which image light or display content is output.

[0207] In some embodiments, the center of rotation of the eye 1204 is centered within the frustum that defines the display alignment volume 1302b. However, it should be understood that the nominal location of the center of rotation of the eye 1204 and / or the overall shape of the frustum can be determined experimentally or selected using criteria other than the projection from the display surface 1202 so that the display system can provide accurate feedback regarding proper alignment of the display, the quality of the alignment, and an acceptable level of alignment, even if not ideal.

[0208] Examples of display performance at various alignment positions FIG. 14 illustrates how the performance of display surface 1202 can vary with the position of user's eye 1200. As shown, the light rays from display surface 1202 can be directed at the eye at an angle such that the light rays from the edge of display surface 1202 propagate inwards towards eye 1200. Thus, cone 1202’ represents the cone of light that is output by display surface 1202 and forms an image at eye 1200.

[0209] As a result, as display surface 1202 shifts with respect to eye 1200, the exit pupils of the pixels corresponding to the individual portions of the visual field do not reach the retina of eye 1200, and the image appears dark in those portions of the visual field. The positions 1204a, 1204b, 1204c, and 1204d of the center of rotation of the eye are in fact offset with respect to the idealized position 1204’ of the center of rotation. That is, the movement of display surface 1202 with respect to eye 1200 can potentially move the center of rotation of the eye outside of the display alignment volumes 1302a, 1304, 1302b (FIGS. 13A and 13B) for display surface 1202. As discussed herein, the display alignment volumes can be associated with display surface 1202, for example, the display alignment volumes can be defined by the projection from display surface 1202. As a result, as display surface 1202 moves with respect to eye 1200, the display alignment volumes 1302a, 1302b (FIGS. 13A and 13B) also move. FIG. 14 illustrates the various positions (e.g., positions 1204a, 1204b, 1204c, and 1204d) of the center of rotation of the user's eye, the relative position of display surface 1202, and the representation (e.g., representations 1400a, 1400b, 1400c, and 1400d) of how the content provided by display surface 1202 will be perceived by the user at each of the various positions.

[0210] In Example 1400a, the center of rotation of the user's eye can be at position 1204a, which can be centered within an alignment volume such as alignment volume 1300b (for example, a volume with high image quality therein because the eye 1200 receives substantially all of the image light output by the display surface 1202 on its retina). Representation 1400a can represent the user's perception (or view) of the content provided by the display surface 1202 when the user's eye is at position 1204a. As shown by representation 1400a, the luminance for substantially all of the content across the display surface 1202 can be uniform and can be at a full or nearly full brightness level.

[0211] In Example 1400b, the center of rotation of the user's eye can be at position 1204b, which is outside a preferred display alignment volume such as volume 1304 (FIG. 13B), but can be within a secondary alignment volume such as volume 1302b (for example, a volume in which the display performance is only slightly degraded). Representation 1400b can represent the user's perception (or view) of the content provided by the display surface 1202 when the center of rotation of the user's eye is at position 1204b. As shown by representation 1400b, portion 1402 of the image along the right side of the display surface 1202 can have a perceived reduced luminance (for example, 50% luminance) due to a misalignment of the user's eye with respect to the display surface 1202.

[0212] In Example 1400c, the center of rotation of the user's eye can be at position 1204c, which can be outside (or on the outer edge of) a second alignment volume such as volume 1302b (FIG. 13B). Representation 1400c can represent the user's perception (or view) of the content provided by display surface 1202 when the center of rotation of the user's eye is at position 1204c. As shown by representation 1400c, portion 1406 along the edge of the displayed image that the user perceives appears completely (or almost completely) darkened due to misalignment and thus cannot be seen by the user. In an array where some of the pixels of the display therein are below the perceived luminance level, the display can provide a reduced field of view (e.g., the user may not be able to perceive the full field of view that the display would otherwise present). Additionally, there can be a band or portion 1404 of the image having a gradually reduced lightness between the dark portion 1406 and the rest of the representation.

[0213] In Example 1400d, the center of rotation of the user's eye can be at position 1204d, which can be significantly outside the desired alignment volume. Representation 1400d can represent the user's perception (or view) of the content provided by display surface 1202 when the center of rotation of the user's eye is at position 1204d. As shown by representation 1400d, due to a significant misalignment, most of the image 1410 can appear completely (or almost completely) dark to the user, and a substantial portion 1408 of the image can appear darkened.

[0214] Example of a replaceable fit component for a wearable system Figures 15A and 15B show exploded perspective views of a wearable system 220 that may include components for interchangeable fits. In particular, FIG. 15A illustrates a manner in which the wearable system 200 may include interchangeable back padding such as pads 1500a, 1500b, and 1500c, while FIG. 15B illustrates a manner in which the system 200 may include interchangeable forehead pads such as pad 1502 and interchangeable bridge pads such as pad 1504. These interchangeable pads may be used to adjust the fit of the wearable system 200 for individual users, which may have varying anatomical attributes (e.g., the degree to which the display 220 and frame 230 fit different users). As an example, a user with a relatively small head may benefit from attaching relatively large back pads 1500a, 1500b, and 1500c to the frame 230, while a user with a relatively large head may achieve better results (e.g., better optical performance and stability of the frame 300 on their head) by attaching relatively small back pads or even omitting the back pads altogether. Similarly, a user with a prominent nose and / or forehead may benefit from smaller forehead pads 1502 and / or bridge pads 1504, while a user with a less prominent nose and / or forehead may benefit from larger forehead pads 1502 and / or bridge pads 1504. These are merely illustrative examples, and generally, determining the set of interchangeable pads that will provide the best fit for any given user can be complex. As described herein, the display system may show a user a notification indicating that different interchangeable fit components may desirably be provided to properly align the display with the user.

[0215] Exemplary Process for Observing Device Alignment FIG. 16 is a process flow diagram of an exemplary method 1600 for observing device alignment and providing feedback regarding the alignment or compensation for misalignment within a wearable device. Method 1600 may be implemented by a wearable system described herein. Embodiments of method 1600 may be used by a wearable system to provide feedback regarding alignment (i.e., the fit of the wearable device to the user), adjust the display, and attempt to compensate for fit errors (e.g., misalignment).

[0216] In block 1602, the wearable system may obtain a fit tolerance. The fit tolerance may include information associated with a display alignment volume such as volumes 1302a, 1302b, or 1304. In particular, the fit tolerance may include information associated with a nominal (e.g., normal) position of the user's eye relative to the wearable device, and may include information associated with the degree to which variance from the nominal position affects device performance. As an example, the fit tolerance may include information regarding a range of nominal positions within which the wearable device can interface with the user, with at least a desired amount of performance (e.g., with less than 50% dimming on any pixel within the display).

[0217] In block 1604, the wearable system may acquire alignment data. The alignment data may include the spatial relationship between various components of the wearable system and the associated parts of the user. As an example, the alignment data may include one or more of the three-dimensional position of the user's left eye relative to the left-eye display of the wearable system, the 3D position of the user's right eye relative to the right-eye display, and the 3D position of the user's ear relative to the audio output (e.g., speakers, headphones, headsets, etc.) of the wearable system. The wearable system may acquire the alignment data using any suitable mechanism. As an example, the wearable system may use an eye-tracking camera 324 of the type shown in FIG. 3 (or other cameras, which may or may not be inward-facing cameras) to capture images of one or both of the user's eyes and determine the relative position of the user's eyes and the wearable system. As another example, the wearable system may include a depth sensor, a pressure sensor, a temperature sensor, a light sensor, an audio sensor, or other sensors to measure or acquire alignment data such as the position of the wearable device relative to the user.

[0218] In block 1606, the wearable system may determine the fit characteristic. As an example, the wearable system may determine whether the user's left eye is within the left eye alignment volume (such as one of volumes 1302a, 1302b, or 1304 for the left eye) and whether the user's right eye is within the right eye alignment volume (such as one of volumes 1302a, 1302b, or 1304 for the right eye). Block 1606 may also involve the step of determining the distance of the user's eye (or other body part) from its nominal position. As an example, in block 1606, the wearable system may determine that at least one of the user's eyes is outside its individual display alignment volume and the extent and direction to which the user's eye is outside its display alignment volume. Information regarding the direction and magnitude of the misalignment (e.g., the distance between the alignment volume or nominal position and the actual position of the user's eye or other body part) may beneficially be utilized in blocks 1608 and 1610.

[0219] In block 1608, the wearable system may provide feedback to the user (or some other entity) regarding the fit characteristic determined in block 1608. As an example, if in block 1606 the wearable system determines that the wearable device is too low relative to the user's eyes, the wearable system may provide the user with a notification in block 1608 suggesting that the user utilize an appropriate nose bridge pad 1504 (e.g., add a nose bridge pad if neither were previously attached, or replace an existing nose bridge pad with a larger or higher nose bridge pad). Conversely, if the wearable device determines that it is too high relative to the user's eyes, the system may provide the user with a proposal to use a smaller nose bridge pad or to remove the pad completely (if designed to be wearable without a pad). As another example, the wearable system may provide feedback to the user proposing changes to the forehead pad such as pad 1502, changes to the back pads such as pads 1500a - 1500c, changes to other adjustable components of the wearable system, or changes to the way the user is wearing the wearable system (e.g., instructions to move or rotate the system in a particular orientation relative to the user). Generally, the user feedback may be generated based on other metrics such as the position of the user's eyes relative to the display or the portion of the visible image identified by the system. As an example, if the system determines that the user's eyes are above the alignment volume, the system may recommend to the user that the user push the wearable device up along their nose to correct the alignment misalignment.

[0220] User feedback may be provided using any suitable device. As an example, user feedback may be provided via video presented by a display within a wearable device or an external display, or via audio presented by a wearable device or an external device. In various embodiments, the wearable device may provide a two-way guide to assist the user in obtaining proper alignment in a relatively intuitive manner. As an example, the wearable device may display two virtual targets, one representing the position of the user's eyes and the other representing the nominal alignment position. Then, as the user moves the wearable device and adjusts its fit, the user can perceive the extent to which the adjustment affects the alignment, and the user can quickly and intuitively achieve proper alignment.

[0221] In an arrangement where user feedback is provided by an output device such as a display that is part of a wearable device, the wearable device may provide user feedback in a manner that ensures the user can perceive the feedback. As an example, consider rendition 1400d of FIG. 14. In such an example, the wearable system may move user feedback into a portion of the displayed image that is perceived by the user, for example, the left half of the display, as opposed to the non-visible right half of the display of example 1400d of FIG. 14.

[0222] In some embodiments, feedback of the type described herein may be provided to a salesperson in a retail environment, and the feedback may be communicated via a network to the salesperson's computer or mobile device.

[0223] In block 1608, the wearable system may adjust its output and input and compensate for an uncorrected fit error. In some embodiments, block 1608 may be implemented only after the user has failed to correct the fit error in response to feedback. In other embodiments, block 1608 may be implemented until the user corrects the fit error. In some embodiments, block 1608 may be implemented each time the user decides to continue using the wearable system with a fit error. In some embodiments, block 1608 may be omitted.

[0224] As an example, the wearable system may adjust its output and input in block 1608 by adjusting a portion of the displayed image (e.g., to compensate for misregistration-induced darkening of the type shown in FIG. 14), by adjusting the microphone input (e.g., increasing the microphone gain when the user is too far from the microphone or decreasing the microphone gain when the user is too close to the microphone), and by adjusting the speaker output (e.g., increasing or decreasing the speaker volume when the user is too close to or too far from the speaker, respectively, within the wearable device). As one particular example, the wearable system may selectively increase the luminance of a portion of an image, such as portions 1402, 1404, or 1408 in FIG. 14, in an attempt to reduce misregistration-induced darkening. In some other embodiments, the wearable system may recognize that certain portions of the image, such as portions 1406 or 1410 in FIG. 14, are not visible to the user and may reduce the light output within those regions to reduce the energy consumption by the wearable system. For example, in a configuration where different portions of the image may have dedicated selectively activatable light sources or portions of light sources, one or more light sources or portions of light sources associated with the non-visible portions of the image may have their light output reduced or turned off.

[0225] Example of identifying a display alignment volume Figures 17A - 17H illustrate a diagram of a light field projected by a display and how the intersection of the light fields can partially define a display alignment volume. FIG. 18 illustrates a top - down view of an overlapping light field projected by a display and how the intersection of the light fields can partially define a display alignment volume. As illustrated in FIGS. 17A - 17H and 18, the size and shape of the display alignment volume can depend, in part, on the geometry of the display (which can be the display 220 of FIG. 2) and the angle at which externally coupled light propagates out of the display (e.g., out of a waveguide display). It should be understood that the angle at which light is output can define the FOV of the display. That is, the larger the angle with respect to the normal, the larger the FOV provided. In some embodiments, the display surface can output an angle large enough to provide the desired FOV.

[0226] FIGS. 17A - 17H and 18 illustrate a display 220 that can be a light field display (forming a display surface 1202, also illustrated in various other figures herein including FIGS. 12A - 14) that includes elements such as a waveguide 1701, an internal coupling element 1702, an orthogonal pupil expander (OPE) 1704, and an exit pupil expander (EPE) 1706. As an example, the internal coupling element 1702 can receive light from an image source and couple the light into the waveguide 1701. The waveguide 1701 can transmit the light to the OPE 1704, which can provide pupil expansion and direct the light to the EPE 1706, and the EPE 1706 (which can be provided on the display surface 1202) can provide further pupil expansion and transmit the light to the user's eye. Further examples and details of light field displays and components of such displays are also described in relation to at least FIGS. 9A - 9C of U.S. Provisional Patent Application No. 62 / 642,761, filed on Mar. 14, 2018, which is incorporated herein by reference in its entirety.

[0227] FIG. 17A illustrates an example where display 220 projects light 1710 associated with virtual image content at optical infinity and in the rightmost region of the FOV of the display (e.g., the rightmost pixel). In contrast, FIG. 17B illustrates an example where display 220 projects light 1712 associated with an object at optical infinity and in the leftmost region of the FOV of the display (e.g., the leftmost pixel). FIG. 17C illustrates the overlapping region 1714 of the light 1710 of FIG. 17A and the light 1712 of FIG. 17B. Region 1714 can be a horizontal alignment volume. In particular, when the user's eye is placed within region 1714 of FIG. 17C, the user can perceive the object in both the rightmost region of the FOV (as in FIG. 17A) and the leftmost region of the FOV (as in FIG. 17B) (e.g., display 220 can provide light from there to the user).

[0228] FIGS. 17D - F are similar to those of FIGS. 17A - 17E but illustrate examples in the vertical direction. In particular, FIG. 17D illustrates an example where display 220 projects light 1716 associated with an object at optical infinity and in the bottommost region of the FOV of the display (e.g., the bottommost pixel), while FIG. 17E illustrates an example where display 220 projects light 1718 associated with an object at optical infinity and in the uppermost region of the FOV of the display (e.g., the uppermost pixel). Similarly, FIG. 17F illustrates the overlapping region 1720 of the light 1716 of FIG. 17D and the light 1718 of FIG. 17E. Region 1720 can be a vertical alignment volume. In particular, when the user's eye is placed within region 1720 of FIG. 17F, the user can perceive the object in both the bottommost region of the FOV (as in FIG. 17D) and the uppermost region of the FOV (as in FIG. 17E) (e.g., display 220 can provide light from there to the user).

[0229] Figures 17G and 17H illustrate the intersection of region 1714 of FIG. 17C and region 1720 of FIG. 17F (as region 1722). In particular, FIG. 17G illustrates region 1722 where light from objects at the four corners of the FOV of display 220 overlaps. FIG. 17H simply illustrates the contour of region 1722. As should be clear, when the user's eye is placed within region 1722, the user is able to perceive an object at any location within the FOV of the display (e.g., display 220 is able to provide light from there to the user).

[0230] In some embodiments, increasing the FOV 220 of the display (horizontally, vertically, or in a combination thereof) while keeping other attributes (such as display size) constant can have the effect of shrinking the associated alignment volume (e.g., horizontal volume 1714, vertical volume 1720, or combined alignment volume 1722). As an example, consider FIGS. 17A - C and the horizontal FOV and alignment volume 1714. An increase in the horizontal FOV of display 220 means that light 1710 from an object on the right - hand horizontal edge is projected at a more acute angle (e.g., a larger angle from the normal to display surface 1202) by display surface 1202 (e.g., EPE 1706). Similarly, light 1712 from an object on the left - hand horizontal edge is also projected at a more acute angle. Thus, from the perspective of FIG. 17C, the apex of the horizontal alignment volume 1714 moves towards display surface 1202 with an increase in the horizontal FOV, thereby shrinking volume 1714. Similar considerations can apply in some embodiments to the vertical FOV and vertical alignment volume as well as the overall FOV and overall alignment volume.

[0231] Figure 18 shows a top view of display 220, including display surface 1202, which may have a rectangular shape and a specific FOV, and the light rays produced by the display. Generally, the alignment volume of display 220 in Figure 18 may be volume 1802, which appears triangular in the top and bottom viewpoints of Figure 18. Volume 1802 may represent the volume where the various light fields formed by the light shown in Figures 17A - 17G overlap. It should be understood that when the user's eye is located outside volume 1802 (e.g., within volume 1804), the light of the light field from at least a part of display 220 cannot reach the user's eye and will result in partial or complete darkening of a part of the FOV.

[0232] The side view of the display and the alignment volume will have substantially the same appearance as shown in Figure 18 (at least with respect to a rectangular display), but it should be noted that the dimension shown for display 220 will be the height of display 220 rather than its width, and the FOV shown will be the vertical FOV rather than the horizontal FOV shown in Figure 18. Thus, volume 1802 may actually have a substantially pyramidal shape. In other embodiments, the display may have a non - rectangular shape such as a circular shape, an elliptical shape, a free - form shape, or any other desired shape. In such embodiments, the corresponding alignment volume may be determined by projecting the light fields onto the relevant FOV and identifying the locations where those light fields intersect (which may correspond to volume 1802) and the locations where the light fields do not intersect (which may correspond to volume 1804).

[0233] As discussed herein, the "base" of the pyramid may be truncated (which can help move the user's eye away from the display so that when properly aligned, the user's eyelashes do not affect the display), and the "top" of the pyramid may also be truncated (otherwise it can be useful in reducing the influence of noise in determining the location of the user's eye, where the location can quickly become aligned or deviate from the alignment in the "top" of the pyramidal alignment volume). It should be understood that the "top" is close to the apex of volume 1802 and the "base" is close to waveguide 1701. When the user's eye is located within region 1804 outside alignment volume 1802, the user may perceive dimming of some or all of the pixels of display 220 as discussed herein (see, e.g., FIG. 14).

[0234] Generally, the alignment volume may be adjusted in any number of ways for various reasons (e.g., truncated or otherwise reduced). As an example, the alignment volume may be truncated such that the volume has a minimum distance from display 220 and prevents the user's eyelashes or eyelids from affecting display 220. As another example, the wearable system may have an eye tracking system that includes elements such as camera 324 and light source 326 of FIG. 6 that can track only the user's eye when the user's eye is within the eye tracking volume where it cannot exactly overlap with the display alignment volume.

[0235] An exemplary process for observing device alignment and providing user feedback to improve alignment In some embodiments, the wearable system may use eye-tracking information such as the position of the user's eye along the y-axis and z-axis (see FIGS. 12A and 12B, which illustrate the y-axis, for example, the vertical direction or the direction of gravity when the user and the wearable system are upright, and the z-axis, which is a horizontal direction extending along the user's line of sight when they are looking forward) to determine device alignment and provide feedback regarding the alignment. In some embodiments, the wearable system may be configured to track the position of the user's pupil along the y-axis and the center of rotation of the eye along the z-axis. The wearable system may determine whether the user's pupil is vertically offset (e.g., along the y-axis) from a desired position, or whether the eye (e.g., the center of rotation of the eye) is too close to or too far from the wearable system (e.g., along the z-axis), and based on any of these offsets, may provide customized feedback to the user. The feedback may include instructions for modifying various physical wearable system components that affect the position of the display relative to the user. For example, the instructions may include steps for prompting the user to don or remove various components connected to the display 220 and frame 230 (FIG. 2). In some embodiments, the user may be prompted to don or remove a thicker forehead pad, a thinner forehead pad, a higher nose pad, a lower nose pad, another fitting component, another back pad, or a combination or sub-combination of fitting components to improve the fit of the device (e.g., adjust the position of the user's pupil towards a desired position). As another example, the wearable system may include physically adjustable or movable parts, and the instructions may include steps for prompting the user to make adjustments to those parts (including steps for prompting the user using specific instructions such as how to adjust those parts, the amount by which those parts should be adjusted, and / or the direction or manner in which those parts should be adjusted).As a specific example, the wearable system may include a dynamic IPD adjustment mechanism that physically adjusts the separation between the left and right displays of the wearable system. The wearable system may prompt the user to adjust the dynamic IPD adjustment mechanism when it determines that the user's IPD does not match the current setting of the mechanism. For example, the wearable system may have a slide mechanism that joins the left and right displays, and the displays may be moved closer or farther apart by crushing or extending the slide mechanism.

[0236] As shown in the embodiment of FIG. 19, the wearable system may vertically identify the location of each of the user's pupils within a plurality of vertical regions. As an example, the wearable system may determine whether each of the user's pupils is within one of regions T1 or T2 (e.g., the wearable system is too low relative to the user's pupils), one of regions M1 or M2 (e.g., the wearable system is within the desired vertical range relative to the user's pupils), or one of regions B1 or B2 (e.g., the wearable system is too high relative to the user's pupils).

[0237] The scale along the right side of FIG. 19 may be in units of pixels and may refer to an image captured by the eye tracking system. Thus, it can be said that an eye pupil within the upper 80 pixels of the eye tracking image is within region T2. Similarly, if the eye pupil is within the next 40 pixels, it may be within region T2 (80 - 120 pixels), if within the following 90 pixels, it may be within region M1 (120 - 210 pixels), if within the following 90 pixels, it may be within region M2 (210 - 300 pixels), if within the following 100 pixels, it may be within region B1 (300 - 400 pixels), and if within the lower pixels (over 400), it may be within region B2. It should be understood that the association of each region with specific pixels may vary depending on the resolution of the image captured by the eye tracking system, the location of the eye imaging camera relative to the display, and the like. Further, different numbers and sizes of regions may be utilized. Preferably, a sufficient number of regions are provided to distinguish an eye pupil at an acceptable position from an eye pupil at a position that is considered too high or too low or otherwise unacceptable or undesirable.

[0238] In various embodiments, the wearable system may determine the respective vertical locations of the user's pupils based on images captured by an eye tracking system, such as the inward-facing imaging system 462 of FIG. 4 (which may include one or more cameras 324 and may also include one or more infrared light sources 326 of FIG. 3). As shown in FIG. 19, the inward-facing imaging system may capture an image of the eye, including flashes such as flashes 1900a, 1900b, 1900c, and 1900d. The flashes are reflections from the user's eye that appear in the image of the eye captured by a camera such as camera 324. The positions of light sources such as light sources 326a and 326b relative to camera 324 can be ascertained, and as a result, the positions of the flashes in the image captured by camera 324 may be used in tracking the user's eye, including the step of determining the vertical location of the user's pupil (e.g., the region of FIG. 19 where the user's pupil is located) (as discussed in further detail herein in connection with FIGS. 7-11). For example, flashes 1900a, 1900b, 1900c, and 1900d may be utilized as reference points, and the position of the user's pupil may be determined based on the location of the pupil relative to these flashes, as disclosed herein.

[0239] FIG. 20 is an example of a process flow diagram for observing device alignment and providing feedback regarding alignment with a head-mounted wearable system. The schematic illustrates a method 2000 that may be implemented by the wearable system described herein. Method 2000 may be used by the wearable system to provide feedback regarding alignment (the position of the wearable device relative to the user) based on data from the eye tracking system.

[0240] In block 2002, the wearable system may provide the user with an initial wearing instruction. In some embodiments, the initial wearing instruction may guide the user regarding how to initially properly place the wearable system on their head. As an example, the wearable system may display a screen, an example of which is illustrated via the screenshot of FIG. 21, which includes visual aids (e.g., text, photos, animations, videos, etc.) that assist the user in initially placing the system on their head. Additionally, or alternatively, the wearable system may provide auditory aids (e.g., verbal instructions and / or feedback) for the same purpose.

[0241] As shown in the embodiment of FIG. 21, the user may be instructed to raise or lower the back of the wearable system until the markings 2100 are visible at each corner of the display. It should be understood that raising or lowering the back of the wearable system changes the pitch of the display 220 (FIG. 2) with respect to the user's eyes. This changes the range of the vertical dimension of the field of view (FOV) of the display visible to the user. The markings 2100 are preferably located at positions delimiting the desired vertical range of the FOV (also referred to as the vertical FOV), alternatively, may take any desired shape and have any desired size. As a result, in some embodiments, the markings 2100 may simply take the form of horizontal bars at the top and bottom of the desired vertical FOV and / or vertical bars on the left and right of the desired horizontal FOV.

[0242] In some embodiments, the marking 2100 may be located at the polar angle of the FOV of the wearable system such that a user who can see all of the markings can see 100% of the nominal vertical FOV of the wearable system with 100% pixel saturation. Pixel saturation may refer to the apparent brightness of a pixel from the user's perspective. 100% pixel saturation may refer to a situation where the user can perceive a given pixel at 100% of that brightness. In situations where the user's eye is not properly aligned with the display of the wearable system, the user may perceive some pixels with reduced brightness and may not be able to perceive other pixels. Further discussion of pixel saturation is provided in connection with FIG. 14, which discusses how pixel saturation or pixel brightness may be reduced in the case of misalignment. In some embodiments, the marking 2100 may be located at a certain distance away from the edge of the nominal vertical FOV. When the marking 2100 is set away from the edge of the nominal vertical FOV, it may be easier for the user to adjust the wearable system so that all of the marking 2100 can be seen. Given the diversity of facial geometries among different people, it can be difficult to provide a wearable system with a large vertical FOV such that 100% of users can see 100% of the vertical FOV. When the marking 2100 is installed at the edge of the vertical FOV, some users may find it difficult or impossible to adjust the wearable device so that all of the marking is visible, which can lead to user dropout and / or calibration that is not completed. When the marking 2100 is installed to cover slightly less than the vertical FOV of the wearable system, most, if not all, users may be able to adjust the wearable system so that all of the marking is visible.On the other hand, when the markings 2100 are placed to cover much less than the amount of the vertical FOV, the user may be able to see the markings 2100 even when the wearable system is not worn properly, and may decide to skip proper wearing, thereby making it impossible to view the content displayed near the ends of the vertical FOV. As an example, the markings 2100 may be placed to cover 95%, 90%, 85%, or 80% of the nominal vertical FOV. In some embodiments, the markings 2100 cover 95% of the vertical FOV of the wearable system. Preferably, the amount of the vertical FOV covered enables the entire user interface of the wearable system to be visible to the user.

[0243] It should be understood that changing the pitch of the wearable system cannot significantly modify the horizontal extent of the FOV visible to the user. Thus, the horizontal separation between the markings 2100 may cover a percentage of the horizontal FOV that is different from the vertical FOV. For example, the horizontal separation between the markings 2100 may be selected such that the horizontal separation does not prevent the user from seeing the markings 2100 throughout the range of possible pitch adjustments.

[0244] Referring again to FIG. 20, in block 2004, the wearable system may determine the vertical position of the user's eyes and may also determine an eye tracking reliability level. The vertical position of the user's eyes may be determined with reference to the regions of FIG. 19. In other words, the wearable system may determine the vertical position of the user's eyes relative to the wearable system and may also determine the region (e.g., T2, T1, M1, M2, B1, or B2) in which the user's eyes and / or a part thereof are located. The wearable system may also determine the z-axis position of the user's eyes relative to the wearable system. In some embodiments, the z-axis position of the user's eyes may be understood to be the z-axis position of the center of rotation of the user's eyes, which may be determined as disclosed herein. Block 2004 may also involve the step of determining whether a Y-offset exists between the user's two eyes. For example, the wearable system may determine whether one of the user's eyes is located within a different one of the regions (T2, T1, M1, M2, B1, or B2) than the user's other eye.

[0245] Continuing to refer to block 2004, the eye tracking reliability level may be an indication of the reliability of the eye tracking data (e.g., the level of confidence the wearable system has that each eye is actually within the region suggested by the eye tracking data). The eye tracking reliability level may be based on factors such as whether a clear image of the eye can be captured, whether a flash can be properly detected, and whether the pupil can be properly detected. Specific examples of reliability factors include the following. · Whether the user is blinking · The number of flashes detected (e.g., detection of 3 or 4 flashes may indicate good reliability, while detection of 2 flashes may reduce the reliability score, and detection of no or only 1 flash may result in a zero reliability score) · Difficulty in pupil detection · Pupil aspect ratio (a high aspect ratio may indicate poor pupil detection) · Whether the pupil is on the image boundary (if the pupil touches the upper or bottom edge of the eye tracking image, a reliability penalty may be incurred) · Eye movement factors (e.g., if the eye center has recently moved, e.g., since the last captured image, a reliability penalty may exist)

[0246] In some embodiments, the aforementioned factors may each be equally weighted. For example, the reliability (C) may be given by the equation C = 1.0 - F f / 10, where F f = the number of flag factors. Higher numerical values for C indicate higher reliability levels, and lower numerical values for C indicate lower reliability levels. It should be understood that the various factors used to determine reliability may vary depending on the calculations and methods used to perform eye tracking. As a result, in some embodiments, reliability determination may utilize more or fewer factors than those listed above. In some embodiments, the aforementioned factors may each (or some of them) not be equally weighted.

[0247] It should be understood that the eye tracking reliability level is used in method 2000 to provide a comparison between successive lower reliability levels (e.g., high, low, and poor) for the left and right eyes. In some embodiments, the reliability factors and their weighting and calculation may differ from the above examples as long as a relative comparison of high, low, or poor levels between the left and right eyes can be obtained.

[0248] In block 2006, the wearable system may determine whether the current fit is satisfactory. As an example, the wearable system may determine whether the user's eyes are within one of the desired vertical ranges of FIG. 19 (e.g., M1 or M2) and within the desired range of z-axis positions. Additionally, in block 2006, the wearable system may determine whether there is a point at which the fit should be considered satisfactory due to the exhaustion of the available fit adjustments. As an example, if the previously recommended fit adjustments have been made and no further useful adjustments can be made, the wearable system may determine that the fit is acceptable even when the user's eyes are outside the desired vertical range and desired Z-position.

[0249] If it is determined that the current fit is not satisfactory, the method may continue with block 2008. In block 2008, the wearable system may generate and provide to the user one or more fit adjustment recommendations based on the pupil (eye) position and the eye tracking confidence level determined in block 2004. Block 2008 may also sometimes involve generating and providing to the user one or more fit adjustment recommendations based on the eye center position, also sometimes referred to as the center of rotation (CoR) position. As an example, the wearable system may determine that the wearable device is seated too low on the user's face (e.g., along the y-axis in FIGS. 12A and 12B), and thus may generate a recommendation to wear a higher nose pad. As another example, the wearable system may determine that the user's eyes are "too close" to the wearable device. As a specific example, referring to FIGS. 12A and 12B, the wearable system may determine that the user's eyes 1200 are too close to the display surface 1202 along the z-axis, and thus may provide a fit adjustment recommendation that includes switching to a thicker forehead pad. The recommendation to switch to a thicker forehead pad may be made regardless of the y-position or vertical offset. As another example, the wearable system may determine whether the wearable device is seated too far to the right or left of the user's face (e.g., along the x-axis in FIGS. 12A and 12B), and then may provide an appropriate fit adjustment recommendation. Additional details and examples are discussed in connection with FIGS. 22A, 22B, and 22C. After providing the fit adjustment recommendation to the user in block 2008, method 2000 may return to block 2004. In some embodiments, multiple iterations of the fit check in block 2004 and the fit adjustment recommendation in block 2008 may be performed to achieve the desired fit.

[0250] Once it is determined that the current fit is satisfactory in block 2006, the wearable system may end the wearing process in block 2010. As an example, the wearable system may display or otherwise provide a message to the user indicating that they have completed the wearing process.

[0251] Figures 22A, 22B, and 22C are process flow diagrams of an example of the details of a method for observing device alignment and providing feedback regarding alignment within a wearable device. Figures 22A, 22B, and 22C illustrate different portions of method 2200 that may be implemented by the wearable system described herein. Embodiments of method 2200 may be used by a wearable system to provide feedback regarding alignment (i.e., the fit of the wearable device to the user) based on data from an eye tracking system.

[0252] Figures 22A, 22B, and 22C include various "external page references" to simplify the flow diagrams. As an example, the method splits into two paths in block 2208, one external page reference 1 and the other external page reference 4. These references on Figure 22A (similar references also appear on Figure 22B) correspond to the external page references on Figure 22C. Thus, it should be understood that external page reference 1 from block 2208 leads to external page reference 1 on Figure 22C and is associated with block 2291. Similarly, it should be understood that external page reference 4 from block 2208 leads to external page reference 4 on Figure 22C and is associated with block 2294.

[0253] Referring to Figure 22A, in block 2202, the wearable system may provide the user with an initial wearing instruction (e.g., a way to tilt the device so that marking 2100 is visible). Block 2202 corresponds to block 2002 of Figure 20, and the additional details of block 2202 are thus described in relation to block 2002 of Figure 20.

[0254] In block 2204, the wearable system may acquire the eye position and the eye tracking reliability level. In particular, the wearable system may determine the y-axis and z-axis positions of each of the user's eyes (the pupil and the center of rotation of each eye, respectively), and may also determine a reliability level associated with the eye tracking data for each of the user's eyes (including the reliability level for the left eye and the reliability level for the right eye). Block 2204 corresponds to block 2004 of FIG. 20, and additional details of block 2204 are described in connection with block 2004 of FIG. 20.

[0255] In block 2206, method 2200 may split according to the eye-tracking reliability level. When the eye-tracking reliability level is (1) poor for both eyes or (2) poor for one eye and low for the other eye, method 2200 may proceed to block 2208. In block 2208, the wearable system may determine whether the user is already using a thicker forehead pad. Generally, the wearable system may assist in wearing a plurality of different forehead pads with variable thicknesses. In some embodiments, the wearable system may assist in wearing a limited number of different forehead pads with variable thicknesses. As a specific example, the wearable system may assist in wearing two forehead pads, one being relatively thin (which may be referred to herein as the thinner forehead pad) and the other being relatively thick (which may be referred to herein as the thicker forehead pad). If three or more forehead pads of different thicknesses are available, the wearable system may determine in block 2208 whether the user is already using a forehead pad of a specific thickness (e.g., the thicker forehead pad). In some embodiments, the wearable system may determine that the user is already using a thicker forehead pad based on previous recommendations to the user for wearing a thicker forehead pad. In some embodiments, the wearable system may ask the user whether they are already using a thicker forehead pad. In some embodiments, the wearable system may include a sensor that detects the presence of a thicker forehead pad. When the user is already using a thicker forehead pad, method 2200 may proceed to block 2294 (e.g., as shown by external page reference 4 in FIG. 22A coupled to the "yes" branch of block 2208 and external page reference 4 in FIG. 22C coupled to block 2294). When the user has not yet used a thicker forehead pad, method 220 may proceed to block 2291 (FIG. 22C).

[0256] When the eye tracking reliability level is low or poor for one eye and high for the other eye, method 2200 may proceed from block 2206 to block 2212. At block 2212, the wearable system may determine to continue based on the eye position of the eye with the high eye tracking reliability score. In particular, method 2200 may utilize the eye position of the eye that has a high reliability score with respect to the blocks shown on FIG. 22B.

[0257] When the eye tracking reliability level is high for both eyes or low for both eyes, method 2200 may proceed from block 2206 to block 2214. At block 2214, the wearable system may determine whether a Y-offset exists between the user's two eyes. For example, the wearable system may determine whether one of the user's eyes is located within a given one of the vertical regions of FIG. 19 (e.g., T1, T2, M1, M2, B1, or B2) while the other eye is located within a different one of the vertical regions of FIG. 19.

[0258] If the Y-offset does not exist between the user's eyes, method 2200 continues to block 2216. At block 2216, the wearable system may determine to continue based on the average position of the user's eyes (e.g., the system may average the position of the user's right eye and the position of the user's left eye). For example, after block 2216, method 2200 may utilize the average position to perform the blocks shown on FIG. 22B.

[0259] When the Y-offset is present between the user's eyes, method 2200 continues at block 2218. At block 2218, the wearable system may determine whether the Y-offset is in only one region or more than one region. For example, the system may determine whether the user's eyes are located within adjacent vertical regions (i.e., having only one region offset) or non-adjacent vertical regions (i.e., having an offset of more than one region). As an example, the user's eyes may have two Y-offsets when the user's left eye is located within T2 and the user's right eye is located within M1.

[0260] When the Y-offset is a region of more than one, method 2200 continues at block 2216 (described above). When the Y-offset is a region of only one, method 2200 continues at block 2220. At block 2220, the wearable system may continue based on the eye position of the more offset eye (e.g., the eye farther away from the desired range of M1 or M2). In particular, method 2200 may utilize the position regarding the more offset eye for the blocks shown on Figure 22B.

[0261] At block 2230 (Figure 22B), method 2200 splits based on the y-axis location of the pupil center. As described above, block 2230 may utilize the location of either (1) the confident eye (as in block 2212), (2) the average of the two eyes (as in block 2216), or (3) the more offset eye (as in block 2220).

[0262] When the relevant eye position is within region T2, method 2200 may continue at block 2232. At block 2232, the wearable system may determine whether the Z - position of the relevant eye (e.g., the confident eye, the average of the eyes, or a more offset eye) exceeds a threshold (is above), e.g., whether the eye is extremely far from the wearable display, or whether the eye - tracking reliability level is low for both eyes. As described above, the Z - position for method 2200 may be the position of the center of rotation of the relevant eye. If neither condition exists, method 2200 continues at block 2292 (as shown by external page reference 2). If either condition exists, method 2200 continues at block 2234. At block 2234, the wearable system may determine whether the user is already using a thicker forehead pad. When the user is already using a thicker forehead pad, method 2200 may proceed to block 2292. When the user is not yet using a thicker forehead pad, method 2200 may proceed to block 2291.

[0263] When the relevant eye position is within region T1, method 2200 may continue at block 2236. At block 2236, the wearable system may determine whether the Z - position of the relevant eye (e.g., the confident eye, the average of the eyes, or a more offset eye) exceeds a threshold (is above), e.g., whether the eye is extremely far from the wearable display, or whether the eye - tracking reliability level is low for both eyes. If neither condition exists, method 2200 continues at block 2294 (as shown by external page reference 4). If either condition exists, method 2200 continues at block 2238. At block 2238, the wearable system may determine whether the user is already using a thicker forehead pad. When the user is already using a thicker forehead pad, method 2200 may proceed to block 2294. When the user is not yet using a thicker forehead pad, method 2200 may proceed to block 2293.

[0264] When the relevant eye position is within region B1, method 2200 may continue at block 2240. At block 2240, the wearable system may determine whether the Z-position of the relevant eye (e.g., the confident eye, the average of the eyes, or a more offset eye) exceeds a threshold (e.g., whether the eye is extremely far from the wearable display) or whether the eye tracking reliability level is low for both eyes. If neither condition exists, method 2200 may continue at block 2296 (as shown by external page reference 6). If either condition exists, method 2200 may continue at block 2242. At block 2242, the wearable system may determine whether the user is already using a thicker forehead pad. When the user is already using a thicker forehead pad, method 2200 may proceed to block 2296. When the user is not yet using a thicker forehead pad, method 2200 may proceed to block 2295.

[0265] When the relevant eye position is within region B2, method 2200 may continue at block 2244. At block 2244, the wearable system may determine whether the Z-position of the relevant eye (e.g., the confident eye, the average of the eyes, or a more offset eye) exceeds a threshold (e.g., whether the eye is extremely far from the wearable display) or whether the eye tracking reliability level is low for both eyes. If neither condition exists, method 2200 may continue at block 2298 (as shown by external page reference 8). If either condition exists, method 2200 may continue at block 2246. At block 2246, the wearable system may determine whether the user is already using a thicker forehead pad. When the user is already using a thicker forehead pad, method 2200 may proceed to block 2298. When the user is not yet using a thicker forehead pad, method 2200 may proceed to block 2297.

[0266] When the associated eye position is within region M1 or region M2, method 2200 may continue at block 2250. At block 2250, the wearable system may determine whether the Z-position of the associated eye (e.g., the fiducial eye, the average of the eyes, or a more offset eye) exceeds a threshold (e.g., whether the eye is extremely far from the wearable display) or whether the eye-tracking reliability level is low for both eyes. If neither condition exists, method 2200 continues at block 2252. If either condition exists, method 2200 continues at block 2254. At block 2254, the wearable system may determine whether the user is already using a thicker forehead pad. When the user is already using a thicker forehead pad, method 2200 may proceed to block 2252. When the user has not yet used a thicker forehead pad, method 2200 may proceed to block 2293.

[0267] At block 2252, method 2200 may complete (e.g., end the wearing process). Optionally, the wearable system may provide feedback to the user indicating that the wearing process has been completed. Optionally, the wearable system may provide the user with an indication of the quality of the fit (e.g., an indication of the success of the wearing process in achieving an appropriate fit or improving the fit).

[0268] In each of blocks 2291 - 2298, the wearable system may provide the user with recommendations for improving the fit or position of the wearable system on the user. The recommendations may be based on the measured eye position (e.g., the vertical position of the user's eyes as discussed in relation to FIG. 19). In the embodiments of FIGS. 21A - 21C, the recommendations include replacement of the forehead pad with a thicker forehead pad and replacement of the nose pad with a nose pad that is one or two sizes higher or lower. A higher nose pad can generally raise the display relative to the user's eyes (e.g., along the y - axis), but can also modify the z - axis position of the user's eyes. A thicker forehead pad can generally move the display away from the user's eyes (e.g., along the z - axis), but can also modify the y - axis position of the user's eyes. In some embodiments, there are two forehead pads, including a thicker forehead pad and a standard forehead pad (which may simply be the absence of the thicker forehead pad). In some embodiments, there may be various forehead pads of variable thickness. These are merely illustrative examples and can vary depending on the availability of additional fitting components and fit adjustment mechanisms. Additional discussion of interchangeable fitting components can be found above in relation to FIGS. 15A and 15B.

[0269] As described above, FIG. 22C provides a specific set of recommendations based on a particular deviation of the display from the desired position. Various ones of these recommendations have been described above and are also listed below.

[0270] In block 2291, the wearable system recommends to the user to wear a thicker forehead pad and a nose pad that is two sizes higher.

[0271] In block 2292, the wearable system recommends to the user to wear a nose pad that is two sizes higher.

[0272] In block 2293, the wearable system recommends that the user wear a thicker forehead pad and a nose pad that is one size higher.

[0273] In block 2294, the wearable system recommends that the user wear a nose pad that is one size higher.

[0274] In block 2295, the wearable system recommends that the user wear a thicker forehead pad.

[0275] In block 2296, the wearable system recommends that the user wear a nose pad that is one size lower.

[0276] In block 2297, the wearable system recommends that the user wear a thicker forehead pad and a nose pad that is one size lower.

[0277] In block 2298, the wearable system recommends that the user wear a nose pad that is two sizes lower.

[0278] After any one of blocks 2291 - 2298, method 2200 may return to block 2204 (as shown by external page reference 0 on FIGS. 22A and 22C). In particular, after providing the user with a fit adjustment recommendation (as part of one of blocks 2291 - 2298) and enabling the user to don new fit components, method 2200 may determine a new eye position and eye - tracking reliability level. Method 2200 may continue to recommend additional fit adjustment recommendations until a certain final condition is met. As an example, method 2200 may continue until an appropriate fit is achieved, the user ends the wearing process, or the number of times is sufficient such that the likelihood of further improvement in fit is low. As an example, method 2200 may continue over a maximum of 3 iterations, a maximum of 4 iterations, or a maximum of 5 iterations. Optionally, method 2200 may return to block 2202 instead of block 2204 during one or more of the iterations (e.g., to remind the user of how to properly place the wearable system on their head).

[0279] Computer vision for detecting objects in the surrounding environment As discussed above, the display system may be configured to detect objects or their properties in the environment surrounding the user. The detection may be accomplished using a variety of techniques, including a variety of environmental sensors (e.g., cameras, audio sensors, temperature sensors, etc.) as discussed herein.

[0280] In some embodiments, objects present in the environment may be detected using computer vision techniques. For example, as disclosed herein, a camera facing forward of the display system may be configured to image the surrounding environment, and the display system may be configured to perform image analysis on the image to determine the presence of objects in the surrounding environment. The display system may analyze an image obtained by an outward-facing imaging system to perform scene reconstruction, event detection, video tracking, object recognition, object pose estimation, learning, indexing, motion estimation, or image restoration, etc. As another example, the display system may be configured to perform face and / or eye recognition to determine the presence and location of faces and / or human eyes within the user's field of view. One or more computer vision algorithms may be used to perform these tasks. Non-limiting examples of computer vision algorithms include Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retina Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, Visual Simultaneous Localization and Mapping (vSLAM) techniques, Sequential Bayesian estimators (e.g., Kalman filter, Extended Kalman filter, etc.), Bundle adjustment, Adaptive thresholding (and other thresholding techniques), Iterative Closest Point (ICP), Semi-Global Matching (SGM), Semi-Global Block Matching (SGBM), Feature Point Histogram, various machine learning algorithms (e.g., Support Vector Machine, k-Nearest Neighbor algorithm, Naive Bayes, Neural Network (including convolutional or deep neural network), or other supervised / unsupervised models, etc.), etc.

[0281] One or more of these computer vision techniques may also be used in combination with data obtained from other environmental sensors (e.g., microphones, etc.) to detect and determine various properties of the objects detected by the sensors.

[0282] As discussed herein, objects in the ambient environment may be detected based on one or more criteria. When a display system uses a computer vision algorithm or uses data received from one or more sensor assemblies (which may or may not be part of the display system) to detect the presence or absence of a criterion in the ambient environment, the display system may then signal the presence of the object.

[0283] Machine learning Various machine learning algorithms may be used to learn to identify the presence of objects in the surrounding environment. Once trained, the machine learning algorithms may be stored by the display system. Some examples of machine learning algorithms may include supervised or unsupervised machine learning algorithms, regression algorithms (e.g., ordinary least squares regression, etc.), instance-based algorithms (e.g., learning vector quantization, etc.), decision tree algorithms (e.g., classification and regression trees, etc.), Bayesian algorithms (e.g., naive Bayes, etc.), clustering algorithms (e.g., k-means clustering, etc.), association rule learning algorithms (e.g., Apriori algorithm, etc.), artificial neural network algorithms (e.g., Perceptron, etc.), deep learning algorithms (e.g., Deep Boltzmann Machine, i.e., deep neural network, etc.), dimensionality reduction algorithms (e.g., principal component analysis, etc.), ensemble algorithms (e.g., Stacked Generalization, etc.), or other machine learning algorithms. In some embodiments, the individual models may be customized for individual datasets. For example, a wearable device may generate or store a base model. The base model may be used as a starting point and generate additional models specific to the data type (e.g., a particular user), the dataset (e.g., a set of additional images to be acquired), the conditional situation, or other variations. In some embodiments, the display system may be configured to generate models for the analysis of aggregated data using multiple techniques. Other techniques may include using predefined thresholds or data values.

[0284] The criteria for detecting an object may include one or more threshold conditions. When the analysis of data obtained by an environmental sensor indicates that a threshold condition has been reached, the display system may provide a signal indicating the detection of the presence of an object in the surrounding environment. The threshold conditions may involve quantitative and / or qualitative measurement values. For example, the threshold conditions may include a score or percentage associated with the likelihood of reflection and / or the presence of an object in the environment. The display system may compare a score calculated from the environmental sensor data with a threshold score. If the score is higher than the threshold level, the display system may detect the presence of a reflection and / or an object. In some other embodiments, the display system may signal the presence of an object in the environment if the score is lower than the threshold. In some embodiments, the threshold conditions may be determined based on the user's emotional state and / or interaction with the user's surrounding environment.

[0285] In some embodiments, the threshold conditions, machine learning algorithms, or computer vision algorithms may be specialized for a specific context. For example, in a diagnostic context, the computer vision algorithm may be specialized to detect a certain response to a stimulus. As another example, the display system may execute a face recognition algorithm and / or an event tracing algorithm as discussed herein to sense the user's reaction to a stimulus.

[0286] The processes, methods, and algorithms described herein and / or depicted in the accompanying figures are each embodied in code modules that are executed by one or more physical computing systems, hardware computer processors, application - specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, whereby they can be fully or partially automated. For example, a computing system can include a general - purpose computer (e.g., a server) or a dedicated computer, a dedicated circuit, etc., programmed with specific computer instructions. The code modules can be installed in a dynamic - link library that can be compiled and linked into an executable program, or can be written in an interpreted - type programming language. In some implementations, certain operations and methods can be implemented by circuits specific to a given function.

[0287] Furthermore, the functional embodiments of the present disclosure are sufficiently mathematically, computationally, or technically complex that specialized hardware or one or more physical computing devices (utilizing appropriate specialized executable instructions) may be required to implement the functionality, for example, due to the amount or complexity of the calculations involved or to provide results substantially in real - time. For example, a video can contain many frames, each frame can have millions of pixels, and specifically programmed computer hardware is required to process the video data to provide the desired image - processing tasks or applications in a commercially reasonable amount of time.

[0288] A code module or any type of data can be stored on any type of non-transitory computer-readable medium such as a physical computer storage device including a hard drive, solid state memory, random access memory (RAM), read only memory (ROM), optical disk, volatile or non-volatile storage device, a combination of the same, and / or equivalents. In some embodiments, the non-transitory computer-readable medium may be part of one or more of local processing and data module (140), remote processing module (150), and remote data repository (160). The method and module (or data) may also be transmitted as a data signal generated on various computer-readable transmission media including wireless-based and wired / cable-based media (e.g., as part of a carrier wave or other analog or digital propagated signal) and may take various forms (e.g., as part of a single or multiplexed analog signal or as multiple discrete digital packets or frames). The result of the disclosed process or process step can be persistently or otherwise stored within any type of non-transitory tangible computer storage device or communicated via a computer-readable transmission medium.

[0289] Any process, block, state, step, or functionality in the flow diagrams described in and / or depicted in the accompanying figures is to be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in a process. The various processes, blocks, states, steps, or functionality may be combined, rearranged, added, deleted, modified, or otherwise changed from the exemplary embodiments provided herein. In some embodiments, additional or different computing systems or code modules may implement some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the associated blocks, steps, or states may be performed in other suitable sequences, e.g., sequentially, in parallel, or in some other manner. Tasks or events may be added to or removed from the disclosed exemplary embodiments. Further, the separation of the various system components in the embodiments described herein is for purposes of illustration and should not be understood as requiring such separation in all implementations. It is to be understood that the described program components, methods, and systems may generally be integrated together in a single computer product or packaged in multiple computer products.

[0290] Other considerations The processes, methods, and algorithms described herein and / or depicted in the accompanying figures are each embodied in code modules that are executed by one or more physical computing systems, hardware computer processors, application - specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, thereby being fully or partially automated. For example, a computing system may include a general - purpose computer (e.g., a server) or a special - purpose computer, a dedicated circuit, etc., programmed with specific computer instructions. The code modules can be compiled and linked into an executable program, installed in a dynamic - link library, or written in an interpreted - type programming language. In some implementations, certain operations and methods can be performed by circuits specific to a given function.

[0291] Furthermore, the functional implementations of the present disclosure are sufficiently mathematically, computationally, or technically complex that special - purpose hardware or one or more physical computing devices (utilizing appropriate specialized executable instructions) may be required to implement the functionality, for example, due to the amount or complexity of the calculations involved or to provide results in substantially real - time. For example, a video or video stream may contain many frames, each frame may have millions of pixels, and specifically programmed computer hardware is required to process the video data to provide the desired image - processing tasks or applications within a commercially reasonable amount of time.

[0292] A code module or any type of data can be stored on any type of non-transitory computer-readable medium, such as a physical computer storage device including a hard drive, solid state memory, random access memory (RAM), read only memory (ROM), optical disk, volatile or non-volatile storage device, combinations of the same, and / or equivalents. The methods and modules (or data) can also be transmitted as data signals generated on various computer-readable transmission media, including wireless-based and wired / cable-based media (e.g., as part of a carrier wave or other analog / digital propagation signal), and can take various forms (e.g., as part of a single or multiplexed analog signal or as a plurality of discrete digital packets or frames). The results of the disclosed process or process steps can be persistently or otherwise stored within any type of non-transitory tangible computer storage device or communicated via a computer-readable transmission medium.

[0293] Any process, block, state, step, or functionality in the flow diagrams described in and / or depicted in the accompanying figures should be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in a process. The various processes, blocks, states, steps, or functionalities may be combined, rearranged, added, deleted, modified, or otherwise changed from the exemplary embodiments provided herein. In some embodiments, additional or different computing systems or code modules may implement some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the associated blocks, steps, or states can be performed in a suitable other sequence, e.g., sequentially, in parallel, or in some other manner. Tasks or events may be added to or removed from the disclosed exemplary embodiments. Further, the separation of the various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods, and systems can generally be integrated together in a single computer product or packaged in multiple computer products. Many implementation variations are possible.

[0294] The present process, method, and system may be implemented in a network (or distributed) computing environment. The network environment may include an enterprise-wide computer network, an intranet, a local area network (LAN), a wide area network (WAN), a personal area network (PAN), a cloud computing network, a cloud source computing network, the Internet, and the World Wide Web. The network may be a wired or wireless network or any other type of communication network.

[0295] The systems and methods of the present disclosure each have several innovative aspects, none of which alone contribute to or are required for the desirable attributes disclosed herein. The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. Various modifications of the implementations described in this disclosure may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations without departing from the spirit or scope of the present disclosure. Accordingly, the claims are not intended to be limited to the implementations shown herein, but should be accorded the widest scope consistent with the present disclosure, the principles, and the novel features disclosed herein.

[0296] Certain features described herein in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, the various features described in the context of a single implementation may also be implemented separately in multiple implementations or in any suitable sub-combination. Further, a feature may be described above as acting in a certain combination and may further be claimed as such, but one or more features from the claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a sub-combination or a variation of a sub-combination. No single feature or group of features is necessary or essential to any embodiment.

[0297] In particular, conditional statements used herein such as "can", "could", "might", "may", "e.g.", and equivalents, generally convey that while one embodiment includes a certain feature, element, or step, another embodiment does not, unless specifically described otherwise or understood otherwise within the context in which it is used. Thus, such conditional statements are not generally intended to imply that a feature, element, and / or step is required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps should be included or implemented in any particular embodiment, regardless of the author's input or prompting. The terms "comprising", "including", "having", and equivalents are synonyms and are used inclusively in a non-limiting manner, excluding no additional elements, features, acts, operations, etc. Also, the term "or" is used in its inclusive sense (and not in its exclusive sense), and thus, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Additionally, the articles "a", "an", and "the" as used in this application and the appended claims should be construed to mean "one or more" or "at least one" unless otherwise defined.

[0298] As used herein, the phrase referring to a list of items "at least one of" refers to any combination of those items, including a single element. As an example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Connective phrases such as "at least one of X, Y, and Z" are generally understood in a context such that, unless specifically described otherwise, they are used to convey that an item, term, etc. can be at least one of X, Y, or Z. Thus, such connective phrases are generally not intended to suggest that an embodiment requires that at least one of each of X, at least one of Y, and at least one of Z be present.

[0299] Similarly, operations may be depicted in the drawings in a particular order, but it should be recognized that this is for achieving a desired result, and that such operations need not be performed in the particular order shown, or in a sequential order, or that all of the illustrated operations need to be performed. Further, the drawings may schematically depict one or more exemplary processes in the form of flowcharts. However, other operations not depicted may also be incorporated within the exemplary methods and processes schematically illustrated. For example, one or more additional operations may be performed before, after, simultaneously with, or during any of the illustrated operations. Additionally, operations may be rearranged or reordered in other implementations. In some situations, multitasking and parallel processing may be advantageous. Further, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product, or packaged into multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result.

Claims

1. 1. A display system, comprising: a wearable display configured to project light into an eye of a user while the display system is worn by the user to display virtual image content; and an eye tracking camera system configured to image the user's eyes, including a left eye and a right eye; at least one light source associated with each of the left eye and the right eye; processing electronics comprising one or more processors; a non-transitory computer readable medium storing instructions; Equipped with The instructions, when executed by the processing electronics, executing an eye tracking module that generates eye tracking data including a position of the left eye and a position of the right eye based at least in part on one or more eye tracking images of the eyes captured by the eye tracking camera system; executing an alignment observer, the alignment observer calculating, based at least in part on the eye tracking data generated by the eye tracking module, a left eye tracking confidence score indicative of a confidence level in a position of the left eye relative to the display system and a right eye tracking confidence score indicative of a confidence level in a position of the right eye relative to the display system; executing a rendering controller, the rendering controller determining, based at least in part on the eye tracking data, the left eye tracking confidence score, and the right eye tracking confidence score generated by the eye tracking module, at least one rendering camera position at which to position at least one rendering camera for rendering the virtual image content such that the virtual image content appears as seen from the user's perspective; causing the display system to render the virtual image content based on the at least one rendering camera position; causing said processing electronics to perform operations including an initial value of the left eye tracking confidence score is 1 and the left eye tracking confidence score is determined to be lower based on one or more of: detecting less than three flashes in the left eye from the at least one light source, where detecting two flashes reduces the left eye tracking confidence score and detecting one or zero flashes results in a left eye tracking confidence score of zero; detecting that the user is blinking; detecting that the left eye has moved since a last captured image; and detecting that the pupil of the left eye touches an edge of an image of the left eye captured by the eye tracking camera system; A display system, wherein the initial value of the right eye tracking confidence score is 1 and the right eye tracking confidence score is determined to be lower based on one or more of: detecting less than three flashes in the right eye from the at least one light source, where detecting two flashes reduces the right eye tracking confidence score and detecting one or zero flashes results in a right eye tracking confidence score of zero; detecting that the user is blinking; detecting that the right eye has moved since the last captured image; and detecting that the pupil of the right eye touches an edge of an image of the right eye captured by the eye tracking camera system.

2. 10. The display system of claim 1 , wherein the wearable display comprises a stack of waveguides configured to output light and display the virtual image content, each waveguide comprising an in-coupling optical element and an out-coupling optical element.

3. 3. The display system of claim 2, wherein one of the waveguides outputs light with a different level of wavefront divergence than at least one other of the waveguides, the different level of wavefront divergence corresponding to a different depth plane.

4. The display system of claim 1 , wherein the at least one rendering camera includes a first rendering camera for rendering the virtual image content viewed by the left eye and a second rendering camera for rendering the virtual image content viewed by the right eye.

5. 10. The display system of claim 1, wherein the eye tracking camera system comprises a first camera for imaging the left eye and a second camera for imaging the right eye.

6. The display system of claim 1 , wherein the at least one rendering camera position includes a center of rotation for at least one of the eyes.

7. The display system of claim 1 , wherein the at least one rendering camera position includes a center of viewpoint of at least one of the eyes.

8. The display system of claim 1 , wherein the at least one rendering camera position comprises a position along an axis between a center of viewpoint and a center of rotation of at least one of the eyes.

9. The operation includes:

2. The display system of claim 1, further comprising: determining an alignment quality indicating the degree to which the wearable display is properly aligned to the eye of the user based on at least one of the left eye tracking confidence score and the right eye tracking confidence score, and wherein the at least one rendering camera position is based at least in part on the alignment quality.

10. 10. The display system of claim 9, wherein when the left and right eyes are vertically offset from one another by less than a first predetermined threshold, determining the alignment quality is based on the left and right eye positions that are furthest from a desired vertical position.

11. 11. The display system of claim 10, wherein when the left and right eyes are vertically offset from one another by less than a second predetermined threshold that is less than the first predetermined threshold, determining the alignment quality is based on an average position of the left eye and an average position of the right eye.

12. 11. The display system of claim 10, wherein determining the alignment quality is based on an average position of the left and right eyes when the left and right eyes are vertically offset from one another by more than the first predetermined threshold.

13. 10. The display system of claim 9, wherein when one of the left eye tracking confidence score or the right eye tracking confidence score exceeds the other, determining the alignment quality is based on the user's eye associated with the greater confidence score, while the user's eye associated with a lower confidence score is not used in determining the alignment quality.

14. 1. A method for controlling a display system including a wearable display configured to project light into an eye of a user to display virtual image content while the display system is worn by the user, the eyes including a left eye and a right eye, the method being performed by processing electronics of the display system, and comprising: executing an eye tracking module that generates eye tracking data including a position of the left eye and a position of the right eye based at least in part on one or more eye tracking images of the eyes captured by an eye tracking camera system of the display system; executing an alignment observer, the alignment observer calculating, based at least in part on the eye tracking data generated by the eye tracking module, a left eye tracking confidence score indicative of a confidence level in a position of the left eye relative to the display system and a right eye tracking confidence score indicative of a confidence level in a position of the right eye relative to the display system; executing a rendering controller, the rendering controller determining, based at least in part on the eye tracking data, the left eye tracking confidence score, and the right eye tracking confidence score generated by the eye tracking module, at least one rendering camera position at which to position at least one rendering camera for rendering the virtual image content such that the virtual image content appears as seen from the user's perspective; causing the display system to render the virtual image content based on the at least one rendering camera position; Including, an initial value of the left eye tracking confidence score is 1 and the left eye tracking confidence score is determined to be lower based on one or more of: detecting less than three flashes at the left eye from at least one light source of the display system associated with the left eye, where detecting two flashes reduces the left eye tracking confidence score and detecting one or zero flashes results in a left eye tracking confidence score of zero; detecting that the user is blinking; detecting that the left eye has moved since a last captured image; and detecting that the pupil of the left eye touches an edge of an image of the left eye captured by the eye tracking camera system; a right eye tracking confidence score of 1; and a right eye tracking confidence score of 2; a right eye tracking confidence score of 3; a right eye tracking confidence score of 4; a right eye tracking confidence score of 5; a right eye tracking confidence score of 6; a right eye tracking confidence score of 7; a right eye tracking confidence score of 8; a right eye tracking confidence score of 9; a right eye tracking confidence score of 10; a right eye tracking confidence score of 11;

15. 15. The method of claim 14, wherein the wearable display comprises a stack of waveguides, the waveguides configured to output light and display the virtual image content, each waveguide comprising an in-coupling optical element and an out-coupling optical element.

16. 16. The method of claim 15, wherein one of the waveguides outputs light with a different level of wavefront divergence than at least one other of the waveguides, the different level of wavefront divergence corresponding to a different depth plane.

17. The method of claim 14 , wherein the at least one rendering camera position includes a center of rotation of at least one of the eyes.

18. The method of claim 14 , wherein the at least one rendering camera position comprises a center of viewpoint of at least one of the eyes.

19. The method of claim 14 , wherein the at least one rendering camera position comprises a position along an axis between a center of viewpoint and a center of rotation of at least one of the eyes.

Images