Eye rotation center determination, depth plane selection, and rendering camera positioning in display system

Abstract

The invention relates to eye rotation center determination, depth plane selection, and rendering camera positioning in a display system. A display system may include a head-mounted display configured to project light to an eye of a user, displaying virtual image content with different divergence and collimation amounts. The display system may include: an inward-facing imaging system that images an eye of a user; and processing electronics in communication with the inward-facing imaging system and configured to obtain an estimate of a center of rotation of the user's eye. The display system may render virtual image content through a rendering camera positioned at the determined center of rotation position of the eye.

Description

[0001] This application is a divisional application of the application with the filing date of January 17, 2019, PCT international application number PCT / US2019 / 014052, Chinese national phase application number 201980019514.2, and invention title "Eye Rotation Center Determination, Depth Plane Selection, and Rendering Camera Positioning in a Display System".

[0002] Cross - reference to related applications

[0003] This application claims the priority of U.S. Provisional Patent Application No. 62 / 618559, titled "EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS", filed on January 17, 2018. This application also claims the priority of U.S. Provisional Patent Application No. 62 / 702849, titled "EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS", filed on July 24, 2018. The entire content of each of the above - mentioned applications is incorporated herein by reference.

[0004] This application further incorporates by reference U.S. Patent Publication 2018 / 0018515, titled "IRIS BOUNDARY ESTIMATION USING CORNEA CURVATURE", published on January 18, 2018. Technical field

[0005] The present disclosure relates to display systems, virtual reality, and augmented reality imaging and visualization systems, and more particularly to depth plane selection based at least in part on a user's inter - pupillary distance. Background art

[0006] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality", "augmented reality", or "mixed reality" experiences, where digitally reproduced images or portions thereof are presented to a user in a manner that appears to be or can be perceived as real. Virtual reality or "VR" scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs; augmented reality or "AR" scenarios typically involve the presentation of digital or virtual image information as an enhancement to the visualization of the real world surrounding the user; and mixed reality or "MR" involves the merging of the real and virtual worlds to create a new environment where physical and virtual objects coexist and interact in real time. The human visual perception system has proven to be very complex, and it is challenging to produce VR, AR, or MR technologies that facilitate the comfortable, natural, and rich presentation of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies. Summary of the Invention

[0007] Various examples of depth plane selection in a mixed reality system are disclosed.

[0008] A display system may be configured to project light into a user's eyes to display virtual image content in the user's field of view. The user's eyes may have a cornea, iris, pupil, lens, retina, and an optical axis extending through the lens, pupil, and cornea. The display system may include: a frame configured to be supported on the user's head; a head-mounted display disposed on the frame, the display configured to project light into the user's eyes to display virtual image content in the user's field of view with at least one of different amounts of divergence and collimation, whereby the displayed virtual image content appears to originate from different depths at different times; one or more eye-tracking cameras configured to image the user's eyes; and processing electronics communicatively coupled to the display and the one or more eye-tracking cameras, the processing electronics configured to obtain an estimate of the center of rotation of the eyes based on images of the eyes obtained by the one or more eye-tracking cameras.

[0009] Various examples of a display system that projects light into one or more of a user's eyes to display virtual image content in the user's field of view are described herein, such as the examples listed below:

[0010] Example 1: A display system configured to project light into a user's eyes to display virtual image content in the user's field of view, the eyes having a cornea, iris, pupil, lens, retina, and an optical axis extending through the lens, pupil, and cornea, the display system including:

[0011] A frame configured to be supported on the head of the user;

[0012] A head-mounted display disposed on the frame, the display configured to project light into the eyes of the user to display virtual image content in the user's field of view with at least one of different amounts of divergence and collimation, whereby the displayed virtual image content appears to originate from different depths at different times;

[0013] One or more eye-tracking cameras configured to image the eyes of the user; and

[0014] A processing electronic device in communication with the display and the one or more eye-tracking cameras, the processing electronic device configured to obtain an estimate of the center of rotation of the eyes based on images of the eyes obtained by the one or more eye-tracking cameras.

[0015] Example 2: The display system according to Example 1, further comprising one or more light sources disposed on the frame relative to the eyes of the user to illuminate the eyes of the user, the one or more eye-tracking cameras using the light from the one or more light sources to form images of the eyes.

[0016] Example 3: The display system according to Example 1 or 2, wherein the one or more light sources comprise at least two light sources disposed on the frame relative to the eyes of the user to illuminate the eyes of the user.

[0017] Example 4: The display system according to Example 1 or 3, wherein the one or more light sources comprise infrared light emitters.

[0018] Example 5: The display system according to any one of Examples 1 to 4, wherein the one or more light sources form one or more glints on the eyes, and the processing electronic device is configured to determine the position of the cornea based on the one or more glints.

[0019] Example 6: The display system according to any one of Examples 1 to 5, wherein the cornea has an associated corneal sphere having a center of curvature, and the electronic processing device is configured to determine the position of the center of curvature of the corneal sphere.

[0020] Example 7: The display system according to Example 5, wherein the cornea has an associated corneal sphere having a center of curvature, and the processing electronic device is configured to determine the position of the center of curvature of the corneal sphere based on the one or more glints.

[0021] Example 8: The display system according to any one of the above examples, wherein the one or more eye-tracking cameras are configured to image the pupil of the eye.

[0022] Example 9: The display system according to any one of the above examples, wherein the processing electronic device is configured to determine the position of the center of the pupil.

[0023] Example 10: The display system according to any one of the above examples, wherein the processing electronic device is configured to determine at least a part of the boundary between the iris and the pupil.

[0024] Example 11: The display system according to Example 10, wherein the processing electronic device is configured to determine the center of the boundary between the iris and the pupil.

[0025] Example 12: The display system according to any one of the above examples, wherein the processing electronic device is configured to determine the position of the center of the pupil relative to the center of curvature of the cornea in three-dimensional space.

[0026] Example 13: The display system according to any one of the above examples, wherein the processing electronic device is configured to determine the position and orientation of the optical axis.

[0027] Example 14: The display system according to Example 12, wherein the processing electronic device is configured to determine the position and orientation of the optical axis based on the position of the center of the pupil in three-dimensional space.

[0028] Example 15: The display system according to any one of the above examples, wherein the processing electronic device is configured to determine the position and orientation of the optical axis based on the position of the center of the pupil relative to the center of curvature of the cornea in three-dimensional space.

[0029] Example 16: The display system according to any one of the above examples, wherein the processing electronic device is configured to determine the position of the center of rotation of the eye based on the center of curvature of the cornea.

[0030] Example 17: The display system according to any one of the above examples, wherein the processing electronic device is configured to determine the position of the center of rotation of the eye based on the center of curvature of the cornea and the position and orientation of the optical axis.

[0031] Example 18: The display system according to Example 17, wherein the processing electronic device is configured to determine the position of the center of rotation of the eye by translating a specific distance along the optical axis from the center of curvature of the cornea.

[0032] Example 19: The display system according to Example 18, wherein the specific distance from the center of curvature to the center of rotation is between 4.0 mm and 6.0 mm.

[0033] Example 20: The display system according to Example 18 or 19, wherein the specific distance from the center of curvature to the center of rotation is about 4.7 mm.

[0034] Example 21: The display system according to Example 18 or 19, wherein the specific distance is fixed.

[0035] Example 22: The display system according to Example 18 or 19, wherein the processing electronic device is configured to determine the specific distance based at least on one or more images of the eye previously obtained by the one or more eye tracking cameras.

[0036] Example 23: The display system according to any of the above examples, wherein the processing electronic device is configured to determine the position and orientation of the visual axis offset from the optical axis based on the position and orientation of the optical axis.

[0037] Example 24: The display system according to any of the above examples, wherein the processing electronic device is configured to determine the position and orientation of the visual axis based on the angular rotation relative to the optical axis.

[0038] Example 25: The display system according to any of the above examples, wherein the processing electronic device is configured to determine the position and orientation of the visual axis based on an angular rotation between 4.0° and 6.5° relative to the optical axis.

[0039] Example 26: The display system according to any of the above examples, wherein the processing electronic device is configured to determine the position and orientation of the visual axis based on an angular rotation of about 5.2° relative to the optical axis.

[0040] Example 27: The display system according to any of the above examples, wherein the processing electronic device is configured to determine the position and orientation of the visual axis based at least on one or more images of the eye previously obtained by the one or more eye tracking cameras.

[0041] Example 28: The display system according to any of the above examples, wherein the processing electronic device is configured to determine the center of rotation of the eye based on a plurality of determinations of the position of the optical axis or the visual axis during a period of eye rotation.

[0042] Example 29: The display system according to any of the above examples, wherein the processing electronic device is configured to determine the center of rotation by identifying an intersection, convergence, or adjacent region of a plurality of determinations of the position of the optical axis or the visual axis during a period of eye rotation.

[0043] Example 30: The display system according to any of the above examples, wherein the processing electronic device is configured to determine the vergence distance of the user at the point where the user's left and right eyes are fixated based on the determination of the positions and orientations of the optical axes of the user's left and right eyes.

[0044] Example 31: The display system according to any of the above examples, wherein the processing electronic device is configured to determine the vergence distance of the user at the point where the user's left and right eyes are fixated based on the determination of the positions and orientations of the visual axes of the user's left and right eyes.

[0045] Example 32: The display system according to any of the above examples, wherein the processing electronic device is configured to determine the vergence distance of the user at the point where the user's left and right eyes are fixated based on identifying the region of intersection, convergence, or proximity of the visual axes of the user's left and right eyes.

[0046] Example 33: The display system according to any of the above examples, wherein the processing electronic device is configured to project the visual axes of the user's left and right eyes onto a horizontal plane and identify the region of intersection, convergence, or proximity of the projections of the visual axes of the left and right eyes on the horizontal plane to determine the vergence distance of the user at the point where the user's left and right eyes are fixated.

[0047] Example 34: The display system according to any of the above examples, wherein the processing electronic device is configured to determine at least one of the relative amounts of divergence and collimation for projecting image content based on the determination of the vergence distance.

[0048] Example 35: The display system according to any of the above examples, wherein the processing electronic device includes an electronic device located on the frame.

[0049] Example 36: The display system according to any of the above examples, wherein the processing electronic device includes an electronic device located on the frame and an electronic device disposed at a position remote from the frame.

[0050] Example 37: The display system according to any of the above examples, wherein the processing electronic device includes an electronic device located on the frame and an electronic device located on a belt pack.

[0051] Example 38: The display system according to any of the above examples, wherein at least a portion of the display is transparent and is disposed at a position in front of the user's eyes when the user wears the head-mounted display, such that the transparent portion transmits light from a portion of the user's front environment and the head-mounted display to the user's eyes to provide a view of the portion of the user's front environment and the head-mounted display.

[0052] Example 39: The display system according to any of the above examples, wherein the head-mounted display receives light from a portion of the user's front environment at a first divergence amount and transmits light from the portion of the user's front environment to the user's eyes at a second divergence amount that is substantially the same as the first divergence amount.

[0053] Example 40: The display system according to any of the above examples, wherein the processing electronic device is configured to obtain the estimate of the center of rotation by filtering, averaging, applying a Kalman filter, or performing any combination of the above items on a plurality of estimated center of rotation positions.

[0054] Example 41: The display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture located at the determined center of rotation position of the user's eyes.

[0055] Example 42: The display system according to any of the above examples, wherein the processing electronic device is configured to render the virtual image to be presented to the eyes using a rendering camera located at the center of rotation.

[0056] Example 43: The display system according to any of the above examples, wherein the processing electronic device is configured to use a rendering camera, and the rendering camera is configured to render the virtual image to be presented to the eyes, and the virtual image is rendered as if captured by a camera having an aperture closer to the center of rotation than the retina of the eyes.

[0057] Example 44: The display system according to any of the above examples, wherein the processing electronic device is configured to use a rendering camera, and the rendering camera is configured to render the virtual image to be presented to the eyes, and the virtual image is rendered as if captured by a camera having an aperture located at the center of rotation of the eyes.

[0058] Example 45: The display system according to any of the above examples, wherein the processing electronic device is configured to render a virtual image to be presented to the eye using a rendering camera located at the center of rotation, and the rendering camera is modeled to have an aperture located at the center of rotation of the eye.

[0059] Example 46: A display system configured to project light onto a user's eye to display virtual image content in the user's field of view, the eye having a cornea, iris, pupil, lens, retina, and an optical axis extending through the lens, pupil, and cornea, the display system comprising:

[0060] A frame configured to be supported on the user's head;

[0061] A head-mounted display disposed on the frame, the display configured to project light into the user's eye to display virtual image content in the user's field of view with at least one of different amounts of divergence and collimation, whereby the displayed virtual image content appears to originate from different depths at different times;

[0062] One or more eye-tracking cameras configured to image the user's eye; and

[0063] A processing electronic device in communication with the display and the one or more eye-tracking cameras, the processing electronic device configured to obtain an estimate of the position of the center of perspective of the eye based on an image of the eye obtained by the one or more eye-tracking cameras, the center of perspective being estimated to be near the pupil of the eye or between the cornea and the pupil of the eye,

[0064] wherein the processing electronic device is configured to present the virtual image content to the user's eye, and the virtual image content is rendered by a rendering camera located at the center of perspective.

[0065] Example 47: The display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eye, and the virtual image content is rendered as if captured by a camera having an aperture closer to the center of perspective than the retina.

[0066] Example 48: The display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eye, and the virtual image content is rendered as if captured by a camera having an aperture closer to the center of perspective than the center of rotation of the eye.

[0067] Example 49: The display system according to any of the preceding examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture located at the perspective center.

[0068] Example 50: The display system according to any of the preceding examples, wherein the perspective center is not located at the pupil of the eye.

[0069] Example 51: The display system according to any of the preceding examples, wherein the processing electronic device is configured to obtain an estimate of the user's eye pose over time, and wherein the processing electronic device adjusts the position of the rendering camera at least in part based on the user's eye pose.

[0070] Example 52: The display system according to any of the preceding examples, wherein the processing electronic device is configured to track the user's eye pose over time, and wherein the position of the rendering camera over time is adjusted in response to a change in the user's eye pose over time.

[0071] Example 53: The display system according to any of the preceding examples, wherein the processing electronic device is configured to obtain an estimate of the perspective center by filtering a plurality of estimated perspective center positions.

[0072] Example 54: The display system according to any of the preceding examples, wherein the processing electronic device is configured to obtain an estimate of the perspective center by averaging a plurality of estimated perspective center positions and / or applying a Kalman filter.

[0073] Example 55: The display system according to any of the preceding examples, wherein the perspective center includes a position within the anterior chamber of the user's eye.

[0074] Example 56: The display system according to any of the preceding examples, wherein the perspective center includes a position in front of the pupil of the user's eye.

[0075] Example 57: The display system according to any of the preceding examples, wherein the perspective center includes a position between 1.0 mm and 2.0 mm in front of the pupil of the user's eye.

[0076] Example 58: The display system according to any of the preceding examples, wherein the perspective center includes a position approximately 1.0 mm in front of the pupil of the user's eye.

[0077] Example 59: The display system according to any of the above examples, wherein the perspective center includes a position between 0.25 mm and 1.0 mm in front of the pupil of the user's eye.

[0078] Example 60: The display system according to any of the above examples, wherein the perspective center includes a position between 0.5 mm and 1.0 mm in front of the pupil of the user's eye.

[0079] Example 61: The display system according to any of the above examples, wherein the perspective center includes a position between 0.25 mm and 0.5 mm in front of the pupil of the user's eye.

[0080] Example 62: The display system according to any of the above examples, wherein the perspective center is positioned along the optical axis of the eye, and wherein the processing electronic device is further configured to obtain an estimated position of the perspective center by obtaining an estimated position of the optical axis of the eye.

[0081] Example 63: The display system according to any of the above examples, wherein the perspective center is located along the optical axis of the eye at a position between the outer surface of the cornea of the eye and the pupil, and wherein the processing electronic device is further configured to obtain an estimated position of the perspective center by obtaining an estimated position of the optical axis of the eye.

[0082] Example 64: The display system according to any of the above examples, wherein the perspective center is located along the optical axis of the eye at a position between the outer surface of the cornea of the eye and the pupil, and wherein the processing electronic device is further configured to obtain an estimated position of the perspective center by obtaining an estimated position of the optical axis of the eye and an estimated position of the center of rotation of the eye, the cornea of the eye, the iris of the eye, the retina of the eye, and the pupil of the eye, or any combination thereof.

[0083] Example 65: The display system according to any of the above examples, wherein the processing electronic device includes an electronic device located on the frame.

[0084] Example 66: The display system according to any of the above examples, wherein the processing electronic device includes an electronic device located on the frame and an electronic device disposed at a position remote from the frame.

[0085] Example 67: The display system according to any of the above examples, wherein the processing electronic device includes an electronic device located on the frame and an electronic device located on a waist pack.

[0086] Example 68: The display system according to any of the above examples, wherein at least a portion of the display is transparent and is disposed in front of the user's eyes when the user wears the head-mounted display, such that the transparent portion transmits light from a portion of the user's front environment and the head-mounted display to the user's eyes to provide a view of the portion of the user's front environment and the head-mounted display.

[0087] Example 69: The display system according to any of the above examples, further comprising one or more light sources, the one or more light sources being disposed on the frame relative to the user's eyes to illuminate the user's eyes, and the one or more eye tracking cameras capturing images of the eyes using the light from the one or more light sources.

[0088] Example 70: The display system according to any of the above examples, wherein the one or more light sources comprise at least two light sources, the at least two light sources being disposed on the frame relative to the user's eyes to illuminate the user's eyes.

[0089] Example 71: The display system according to any of the above examples, wherein the one or more light sources comprise at least three light sources, the at least three light sources being disposed on the frame relative to the user's eyes to illuminate the user's eyes.

[0090] Example 72: The display system according to any of the above examples, wherein the one or more light sources comprise infrared light emitters.

[0091] Example 73: The display system according to any of the above examples, wherein the one or more light sources form one or more bright spots on the eyes, and the processing electronic device is configured to determine the position of the center of curvature of the cornea based on the one or more bright spots.

[0092] Example 74: The display system according to any of the above examples, wherein the one or more light sources form one or more bright spots on the eyes, and the processing electronic device is configured to determine the three-dimensional position of the center of curvature of the cornea based on the one or more bright spots.

[0093] Example 75: The display system according to any of the above examples, wherein the one or more eye tracking cameras are further configured to image the pupil of the user's eyes, and wherein the processing electronic device is further configured to determine the position of the pupil of the eyes based at least on the images of the pupil from the one or more eye tracking cameras.

[0094] Example 76: The display system according to any of the preceding examples, wherein the one or more eye-tracking cameras are further configured to image the pupil of the user's eye, and wherein the processing electronic device is further configured to determine a three-dimensional position of the pupil of the eye based at least on an image of the pupil from the one or more eye-tracking cameras.

[0095] Example 77: The display system according to any of the preceding examples, wherein the one or more eye-tracking cameras are further configured to image the pupil of the user's eye, and wherein the processing electronic device is further configured to determine the position of the pupil of the eye based on the position of the center of curvature of the cornea and on an image of the pupil from the one or more eye-tracking cameras.

[0096] Example 78: The display system according to any of the preceding examples, wherein the processing electronic device is configured to determine an optical axis of the eye based on a three-dimensional position of the center of curvature of the cornea and on a three-dimensional position of the pupil.

[0097] Example 79: The display system according to any of the preceding examples, wherein the processing electronic device is configured to determine a visual axis of the eye based on the optical axis.

[0098] Example 80: The display system according to any of the preceding examples, wherein the processing electronic device is configured to determine the visual axis of the eye based on at least one of the center of curvature of the cornea and the pupil or both the three-dimensional positions of the center of curvature of the cornea and the pupil and the optical axis.

[0099] Example 81: The display system according to any of the preceding examples, wherein the processing electronic device is configured to determine a three-dimensional position of a center of rotation of the eye based on a three-dimensional position of the center of curvature of the cornea.

[0100] Example 82: The display system according to any of the preceding examples, wherein the processing electronic device is configured to determine a three-dimensional position of a center of rotation of the eye based on a three-dimensional position of the center of curvature of the cornea and on the optical axis.

[0101] Example 83: The display system according to any of the preceding examples, wherein the processing electronic device is configured to determine a distance between the eye and the user's other eye based at least on a three-dimensional position of the center of rotation of the eye.

[0102] Example 84: The display system according to any one of the above examples, wherein the processing electronic device is configured to determine the interpupillary distance between the eye and the user's other eye at least based on the three-dimensional position of the center of rotation of the eye.

[0103] Example 85: The display system according to any one of the above examples, wherein the processing electronic device is configured to determine the convergence distance of the user at least based on the optical axis of the eye.

[0104] Example 86: The display system according to any one of the above examples, wherein the processing electronic device is configured to determine the convergence distance of the user at least based on the optical axis of the eye and based on the determined optical axis of the user's other eye.

[0105] Example 87: The display system according to any one of the above examples, wherein the processing electronic device is configured to determine the convergence distance of the user at least based on the visual axis of the eye and based on the determined visual axis of the user's other eye.

[0106] Example 88: The display system according to any one of the above examples, wherein the display is configured to project collimated light into the user's eyes.

[0107] Example 89: The display system according to any one of the above examples, wherein the display is configured to project collimated light corresponding to image pixels into the user's eyes during a first time period, and to project divergent light corresponding to the image pixels into the user's eyes during a second time period.

[0108] Example 90: The display system according to any one of the above examples, wherein the display is configured to project light having a first divergence amount corresponding to image pixels into the user's eyes during a first time period, and to project light having a second divergence amount greater than the first divergence amount corresponding to the image pixels into the user's eyes during a second time period.

[0109] Example 91: A method of rendering virtual image content in a display system, the display system being configured to project light into a user's eyes to display the virtual image content in the user's field of view, the eyes having a cornea, iris, pupil, lens, retina, and an optical axis extending through the lens, pupil, and cornea, the method comprising:

[0110] Determining the position of the center of rotation of the eye by one or more eye tracking cameras configured to image the user's eyes to track the movement of the eyes;

[0111] Rendering virtual image content using a rendering camera located at the center of rotation of the eye, the rendering camera being configured to render a virtual image to be presented to the eye; and

[0112] Projecting light into the user's eyes through a head-mounted display to display the rendered virtual image content in the user's field of view with different divergence amounts such that the virtual image content appears to originate from different depths at different times.

[0113] Example 92: The method according to any of the above examples, wherein the rendering camera is configured to render a virtual image to be presented to the eye, and the virtual image is rendered as if captured by a camera having an aperture closer to the center of rotation than the retina of the eye.

[0114] Example 93: The method according to any of the above examples, wherein the rendering camera is configured to render a virtual image to be presented to the eye, and the virtual image is rendered as if captured by a camera having an aperture located at the center of rotation.

[0115] Example 94: The method according to any of the above examples, wherein the rendering camera is modeled as having an aperture located at the center of rotation of the eye.

[0116] Example 95: The method according to any of the above examples, wherein the rendering camera is modeled as having an aperture, a lens, and a detector.

[0117] Example 96: The method according to any of the above examples, wherein the rendering camera has an aperture at a position along a line between (i) the determined position of the center of rotation of the eye and (ii) the determined position of at least one of the iris or the pupil.

[0118] Example 97: The method according to any of the above examples, further comprising:

[0119] Determining the position of the perspective center of the user's eye by the one or more eye tracking cameras, wherein the perspective center of the user's eye is located at a position less than about 1.0 mm from the pupil of the user's eye; and

[0120] Rendering the virtual image content using the rendering camera by the rendering engine,

[0121] wherein the rendering camera has an aperture at the determined position of the perspective center of the user's eye.

[0122] Example 98: The method according to any of the above examples, further comprising:

[0123] Using the rendering engine, rendering the virtual image content with the rendering camera, wherein the rendering camera has an aperture at a position on a line between (i) the determined center of rotation position of the eye and (ii) the determined perspective center position of the user's eye.

[0124] Example 99: The method according to any of the above examples, further comprising:

[0125] Determining, by a processing electronic device in communication with the one or more eye tracking cameras, a measure of the change over time of the determined perspective center position of the user's eye; and

[0126] By the processing electronic device, if it is determined that the measure of the change over time exceeds a first threshold, instructing the rendering engine to render the virtual content with the rendering camera, wherein the rendering camera has an aperture at the determined center of rotation position of the eye.

[0127] Example 100: The method according to any of the above examples, further comprising:

[0128] By the processing electronic device, if it is determined that the measure of the change over time is below a second threshold, instructing the rendering engine to render the virtual content with the rendering camera, wherein the rendering camera has an aperture at the determined perspective center position of the eye, and wherein the first threshold indicates a higher level of the change over time of the determined perspective center position of the user's eye compared to the second threshold.

[0129] Example 101: The method according to any of the above examples, further comprising:

[0130] By the processing electronic device, if it is determined that the measure of the change over time is below a second threshold, instructing the rendering engine to render the virtual content with the rendering camera, wherein the rendering camera has an aperture at the determined perspective center position of the eye.

[0131] Example 102: The method according to any of the above examples, further comprising:

[0132] By the processing electronic device, if it is determined that the measure of the change over time is between the first threshold and the second threshold, instructing the rendering engine to render the virtual content with the rendering camera, wherein the rendering camera has an aperture at a point on a line between (i) the determined center of rotation position of the eye and (ii) the determined perspective center position of the eye.

[0133] Example 103: The method according to any of the above examples further comprises:

[0134] Transmitting light from a part of the user's front environment and the head-mounted display through at least a part of the display, the part being transparent and disposed at a position in front of the user's eyes when the user wears the head-mounted display, to provide a view of the part of the user's front environment and the head-mounted display to the user's eyes.

[0135] Example 104: The method according to any of the above examples further comprises:

[0136] Determining the position of at least one of the iris, pupil or lens by the one or more eye tracking cameras.

[0137] Example 105: The method according to any of the above examples further comprises:

[0138] Rendering the virtual image content by the rendering engine using the rendering camera, the rendering camera being configured to present a virtual image to the eye, the virtual image being rendered as if captured by a camera having an aperture at a position on a line between (i) the determined position of the center of rotation of the eye and (ii) the determined position of at least one of the iris or pupil.

[0139] Example 106: The method according to any of the above examples further comprises:

[0140] Determining the position of the perspective center of the user's eye by the one or more eye tracking cameras, wherein the perspective center of the user's eye is located at a position less than about 1.0 mm from the pupil of the user's eye; and

[0141] Rendering the virtual image content by the rendering engine using the rendering camera, the rendering camera being configured to present a virtual image to the eye, the virtual image being rendered as if captured by a camera having an aperture at the determined perspective center position of the user's eye.

[0142] Example 107: The method according to any of the above examples further comprises:

[0143] Using the rendering engine, render the virtual image content using the rendering camera, the rendering camera being configured to present a virtual image to the eye, the virtual image being rendered as if captured by a camera having an aperture located at a position along a line between (i) the determined center of rotation position of the eye and (ii) the determined perspective center position of the user's eye.

[0144] Example 108: The method according to any of the above examples, further comprising:

[0145] Determining, by a processing electronic device in communication with the one or more eye tracking cameras, a measure of the change over time of the determined perspective center position of the user's eye; and

[0146] By the processing electronic device, if it is determined that the measure of change over time exceeds a first threshold, instruct the rendering engine to render the virtual content using the rendering camera as if captured by a camera having an aperture located at the determined center of rotation position of the eye.

[0147] Example 109: The method according to any of the above examples, further comprising:

[0148] By the processing electronic device, if it is determined that the measure of change over time is below a second threshold, instruct the rendering engine to render the virtual content using the rendering camera as if captured by a camera having an aperture located at the determined perspective center position of the eye, wherein the first threshold indicates a higher level of change over time of the determined perspective center position of the user's eye compared to the second threshold.

[0149] Example 110: The method according to any of the above examples, further comprising:

[0150] By the processing electronic device, if it is determined that the measure of change over time is below a second threshold, instruct the rendering engine to render the virtual content using the rendering camera as if captured by a camera having an aperture located at the determined perspective center position of the eye.

[0151] Example 111: The method according to any of the above examples, further comprising:

[0152] By the processing electronic device, if it is determined that the measure of change over time is between the first threshold and the second threshold, instruct the rendering engine to render the virtual content using the rendering camera as if captured by a camera having an aperture located at a point along a line between (i) the determined center of rotation position of the eye and (ii) the determined perspective center position of the eye.

[0153] Example 112: A display system configured to project light into a user's eyes to display virtual image content in the user's field of view, the eyes having a cornea, iris, pupil, lens, retina, and an optical axis extending through the lens, pupil, and cornea, the display system comprising:

[0154] A frame configured to be supported on the user's head;

[0155] A head-mounted display disposed on the frame, the display configured to project light into the user's eyes to display virtual image content in the user's field of view with different divergence amounts, whereby the displayed virtual image content appears to originate from different depths at different times, wherein the head-mounted display is configured to project light with a first divergence amount into the user's eyes during a first time period and is configured to project light with a second divergence amount into the user's eyes during a second time period, wherein the first divergence amount is different from the second divergence amount;

[0156] One or more eye tracking cameras configured to image the user's eyes; and

[0157] A processing electronic device in communication with the display and the one or more eye tracking cameras, the processing electronic device configured to obtain an estimate of the center of rotation of the eyes based on an image of the eyes obtained by the one or more eye tracking cameras, obtain an estimate of the user's vergence distance based on an image of the eyes obtained by the one or more eye tracking cameras, and transform from projecting light with the first divergence amount into the user's eyes to projecting light with the second divergence amount based on the estimated vergence distance of the user.

[0158] Example 113: The display system according to any of the above examples, wherein at least a portion of the display is transparent and is disposed at a position in front of the user's eyes when the user wears the head-mounted display, such that the transparent portion transmits light from a portion of the user's front environment and the head-mounted display to the user's eyes to provide a view of the portion of the user's front environment and the head-mounted display.

[0159] Example 114: The display system according to any of the above examples, wherein the processing electronic device is further configured to detect a blink of the eyes based on an image of the eyes obtained using the one or more eye tracking cameras.

[0160] Example 115: The display system according to any of the preceding examples, wherein the processing electronic device is further configured to detect a saccade of the eye based on an image of the eye obtained by using the one or more eye tracking cameras.

[0161] Example 116: The display system according to any of the preceding examples, wherein the processing electronic device is configured to transform from projecting light into the user's eyes with the first divergence amount to projecting light into the user's eyes with the second divergence amount based on the determined vergence distance of the user and based on whether the processing electronic device detects a blink of the eye.

[0162] Example 117: The display system according to any of the preceding examples, wherein the processing electronic device is configured to transform from projecting light into the user's eyes with the first divergence amount to projecting light into the user's eyes with the second divergence amount based on the determined vergence distance of the user and based on whether the processing electronic device detects a saccade of the eye.

[0163] Example 118: The display system according to any of the preceding examples, wherein the processing electronic device is configured to transform from projecting light into the user's eyes with the first divergence amount to projecting light into the user's eyes with the second divergence amount based on the determined vergence distance of the user and based on whether the processing electronic device detects at least one of a saccade or a blink of the eye.

[0164] Example 119: The display system according to any of the preceding examples, wherein the first divergence amount is associated with a vergence distance within a first range, and wherein the second divergence amount is associated with a vergence distance within a second range.

[0165] Example 120: The display system according to any of the preceding examples, wherein the first divergence amount is associated with a vergence distance within a first range, the second divergence amount is associated with a vergence distance within a second range, and wherein the first range and the second range overlap but are not equal.

[0166] Example 121: The display system according to any of the preceding examples, wherein the processing electronic device is configured to transform from projecting light into the user's eyes with the first divergence amount to projecting light into the user's eyes with the second divergence amount when determining that the vergence distance of the user is outside the first range and within the second range.

[0167] Example 122: The display system according to any one of the above examples, wherein the processing electronic device is configured to, when determining that the vergence distance of the user is outside the second range and within the first range, change from projecting light into the user's eyes with the first divergence amount to projecting light into the user's eyes with the second divergence amount.

[0168] Example 123: The display system according to any one of the above examples, wherein the processing electronic device is configured to, when determining that the vergence distance of the user is outside the first range and within the second range and detecting a blink of the eye, change from projecting light into the user's eyes with the first divergence amount to projecting light into the user's eyes with the second divergence amount.

[0169] Example 124: The display system according to any one of the above examples, wherein the processing electronic device is configured to, when determining that the vergence distance of the user is outside the first range and within the second range and detecting a saccade of the eye, change from projecting light into the user's eyes with the first divergence amount to projecting light into the user's eyes with the second divergence amount.

[0170] Example 125: The display system according to any one of the above examples, wherein the processing electronic device is configured to, when determining that the vergence distance of the user is outside the first range and within the second range for a time longer than a predetermined time period, change from projecting light into the user's eyes with the first divergence amount to projecting light into the user's eyes with the second divergence amount.

[0171] Example 126: The display system according to any one of the above examples, wherein the processing electronic device is configured to, when determining that the vergence distance of the user is outside the first range and within the second range for a time longer than a predetermined time period of at least 10 seconds, change from projecting light into the user's eyes with the first divergence amount to projecting light into the user's eyes with the second divergence amount.

[0172] Example 127: The display system according to any one of the above examples, wherein the head-mounted display includes a first display element configured to project light with the first divergence amount and a second display element configured to project light with the second divergence amount.

[0173] Example 128: The display system according to any of the preceding examples, wherein the display is configured to project light into the user's eyes to display virtual image content in a discrete display mode, in which the display is configured to project light associated with a plurality of consecutive frames using only one of the first display elements.

[0174] Example 129: The display system according to any of the preceding examples, wherein the display is configured to project light into the user's eyes to display virtual image content in a mixed display mode, in which the display is configured to project light associated with the plurality of consecutive frames using both the first display element and the second display element for each of the plurality of consecutive frames.

[0175] Example 130: The display system according to any of the preceding examples, wherein the display is configured to project light into the user's eyes to display virtual image content in a mixed display mode, in which the display is configured to project light associated with the plurality of consecutive frames using both the first display element and the second display element for each of the plurality of consecutive frames, and wherein in the mixed display mode, the display is configured to project light using the first display element and the second display element, the light being perceived by the user as having a given divergence amount between the first divergence amount and the second divergence amount.

[0176] Example 131: The display system according to any of the preceding examples, wherein the display is configured to project light into the user's eyes to display virtual image content in a multi - focal display mode, in which the display is configured to project light associated with the plurality of consecutive frames using both the first display element and the second display element for each of the plurality of consecutive frames, wherein in the multi - focal display mode, the display is configured to project light associated with first virtual image content with a third divergence amount and project light associated with second virtual image content with a fourth divergence amount, and wherein the third divergence amount is different from the fourth divergence amount.

[0177] Example 132: The display system according to any of the preceding examples, wherein the third divergence amount and the fourth divergence amount are each between the first divergence amount and the second divergence amount.

[0178] Example 133: The display system according to any of the preceding examples, wherein at least one of the third divergence amount and the fourth divergence amount is between the first divergence amount and the second divergence amount.

[0179] Example 134: The display system according to any one of the above examples, wherein the third divergence amount and the fourth divergence amount are respectively equal to the first divergence amount and the second divergence amount.

[0180] Example 135: The display system according to any one of the above examples, wherein the display is configured to project light associated with the first virtual image in a first region of the user's field of view and project light associated with the second virtual image in a second region of the user's field of view, and wherein the first region and the second region are different.

[0181] Example 136: The display system according to any one of the above examples, wherein the display is configured to project light associated with the first virtual image in a first region of the user's field of view and project light associated with the second virtual image in a second region of the user's field of view, and wherein the first region and the second region do not overlap.

[0182] Example 137: A display system configured to project light into a user's left and right eyes to display virtual image content in the user's field of view, each of the eyes having a cornea, iris, pupil, lens, retina, and an optical axis extending through the lens, pupil, and cornea, the display system comprising:

[0183] A frame configured to be supported on the user's head;

[0184] A head-mounted display disposed on the frame, the display configured to project light into the user's left and right eyes to display virtual image content in the user's field of view with at least one of different amounts of divergence and collimation, whereby the displayed virtual image content appears to originate from different distances relative to the user's left and right eyes at different times;

[0185] A first eye-tracking camera configured to image the user's left eye;

[0186] A second eye-tracking camera configured to image the user's right eye; and

[0187] A processing electronic device in communication with the display and the first and second eye-tracking cameras, the processing electronic device configured to obtain an estimate of the interpupillary distance between the user's left and right eyes based on images of the left and right eyes obtained by the first and second eye-tracking cameras.

[0188] Example 138: The display system according to any of the above examples further includes one or more light sources, the one or more light sources being disposed on the frame relative to the user's eyes to illuminate the user's eyes, and the one or more eye tracking cameras forming an image of the eyes using the light from the one or more light sources.

[0189] Example 139: The display system according to any of the above examples, wherein the one or more light sources include at least two light sources, the at least two light sources being disposed on the frame relative to the user's eyes to illuminate the user's eyes.

[0190] Example 140: The display system according to any of the above examples, wherein the one or more light sources include infrared light emitters.

[0191] Example 141: The display system according to any of the above examples, wherein the one or more light sources form one or more bright spots on the eyes, and the processing electronic device is configured to determine the position of the cornea based on the one or more bright spots.

[0192] Example 142: The display system according to any of the above examples, wherein the cornea has an associated corneal sphere, the corneal sphere has a center of curvature, and the electronic processing device is configured to determine the position of the center of curvature of the corneal sphere.

[0193] Example 143: The display system according to any of the above examples, wherein the cornea has an associated corneal sphere, the corneal sphere has a center of curvature, and the processing electronic device is configured to determine the position of the center of curvature of the corneal sphere based on the one or more bright spots.

[0194] Example 144: The display system according to any of the above examples, wherein the one or more eye tracking cameras are configured to image the pupil of the eyes.

[0195] Example 145: The display system according to any of the above examples, wherein the processing electronic device is configured to determine the position of the center of the pupil.

[0196] Example 146: The display system according to any of the above examples, wherein the processing electronic device is configured to determine at least a part of the boundary between the iris and the pupil.

[0197] Example 147: The display system according to any of the above examples, wherein the processing electronic device is configured to determine the center of the boundary between the iris and the pupil.

[0198] Example 148: The display system according to any of the preceding examples, wherein the processing electronic device is configured to determine the position of the center of the pupil relative to the center of curvature of the cornea in three-dimensional space.

[0199] Example 149: The display system according to any of the preceding examples, wherein the processing electronic device is configured to determine the position and orientation of the optical axis.

[0200] Example 150: The display system according to any of the preceding examples, wherein the processing electronic device is configured to determine the position and orientation of the optical axis based on the position of the center of the pupil in three-dimensional space.

[0201] Example 151: The display system according to any of the preceding examples, wherein the processing electronic device is configured to determine the position and orientation of the optical axis based on the position of the center of the pupil relative to the center of curvature of the cornea in three-dimensional space.

[0202] Example 152: The display system according to any of the preceding examples, wherein the processing electronic device is configured to determine the position of the center of rotation of the eye based on the center of curvature of the cornea.

[0203] Example 153: The display system according to any of the preceding examples, wherein the processing electronic device is configured to determine the position of the center of rotation of the eye based on the center of curvature of the cornea and the position and orientation of the optical axis.

[0204] Example 154: The display system according to any of the preceding examples, wherein the processing electronic device is configured to determine the position of the center of rotation of the eye by translating a specific distance along the optical axis from the center of curvature of the cornea.

[0205] Example 155: A method of rendering virtual image content in a display system, the display system being configured to project light onto a user's left and right eyes to display the virtual image content in the user's field of view, each of the eyes having a cornea, iris, pupil, lens, retina, and an optical axis extending through the lens, pupil, and cornea, the method comprising:

[0206] Determining the position of the center of rotation of the left eye and the position of the center of rotation of the right eye by one or more eye tracking cameras configured to image the user's eyes to track eye movement;

[0207] Estimating the user's interpupillary distance by a processing electronic device in communication with the one or more eye tracking cameras based on the determined positions of the centers of rotation of the left and right eyes;

[0208] Determine a current left-eye pose and a current right-eye pose via the one or more eye tracking cameras; and

[0209] Estimate a current vergence distance of the user by the processing electronic device by comparing an estimated interpupillary distance and the determined current left-eye pose and the determined current right-eye pose.

[0210] Example 156: The method according to any of the preceding examples, wherein determining the current left-eye pose and the current right-eye pose includes: estimating a position of a pupil of the user's left eye and a position of a pupil of the user's right eye via the one or more eye tracking cameras.

[0211] Example 157: The method according to any of the preceding examples, wherein determining the current left-eye pose and the current right-eye pose includes: estimating a position of a cornea of the user's left eye and a position of a cornea of the user's right eye via the one or more eye tracking cameras.

[0212] Example 158: The method according to any of the preceding examples, wherein determining the current left-eye pose and the current right-eye pose includes: estimating a position of an iris of the user's left eye and a position of an iris of the user's right eye via the one or more eye tracking cameras.

[0213] Example 159: The method according to any of the preceding examples, wherein determining the current left-eye pose and the current right-eye pose includes: estimating a position of a lens of the user's left eye and a position of a lens of the user's right eye via the one or more eye tracking cameras.

[0214] Example 160: The method according to any of the preceding examples, wherein estimating the current vergence distance of the user includes:

[0215] Estimating a distance between positions of the irises of the user's left eye and right eye by the processing electronic device; and

[0216] Estimating the current vergence distance of the user by the processing electronic device based on a comparison of the estimated interpupillary distance and the estimated distance between the positions of the irises of the user's left eye and right eye.

[0217] Example 161: The method according to any of the preceding examples, further comprising: projecting light into the user's eyes via a head-mounted display and displaying rendered virtual image content to the user's field of view with different divergence amounts such that the virtual image content appears to originate from different depths at different time periods.

[0218] Example 162: The method according to any of the preceding examples further includes: transmitting, through at least a part of the display, the part being transparent and disposed at a position in front of the user's eyes when the user wears the head-mounted display, light from a part of the user's front environment and the head-mounted display to the user's eyes to provide a view of the part of the user's front environment and the head-mounted display.

[0219] Example 163: A display system configured to project light into a user's eyes to display virtual image content in the user's field of view, the eyes having a cornea, iris, pupil, lens, retina, and an optical axis extending through the lens, pupil, and cornea, the display system comprising:

[0220] A frame configured to be supported on the user's head;

[0221] A head-mounted display disposed on the frame, the display configured to project light into the user's eyes to display virtual image content in the user's field of view with at least one of different amounts of divergence and collimation, whereby the displayed virtual image content appears to originate from different depths at different times;

[0222] One or more eye tracking cameras configured to image the user's eyes; and

[0223] A processing electronic device in communication with the display and the one or more eye tracking cameras, the processing electronic device configured to obtain an estimated position of the center of rotation of the eyes based on an image of the eyes obtained by the one or more eye tracking cameras, and to obtain an estimated direction of the optical axis of the eyes based on the image,

[0224] wherein the processing electronic device is configured to present the virtual image content to the user's eyes, the virtual image content being rendered as if captured by a camera having an aperture, the aperture being disposed along the optical axis and spaced 6.0 mm to 13.0 mm from the estimated position of the center of rotation of the eyes in a direction away from the retina.

[0225] Example 164: The display system according to any of the preceding examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, the virtual image content being rendered as if captured by a camera having an aperture, the aperture being disposed along the optical axis and spaced 7.0 mm to 12.0 mm from the estimated position of the center of rotation of the eyes in a direction away from the retina.

[0226] Example 165: The display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture, the aperture being disposed along the optical axis and spaced 8.0 mm to 11.0 mm from the estimated position of the center of rotation of the eye in a direction away from the retina.

[0227] Example 166: The display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture, the aperture being disposed along the optical axis and spaced 9.0 mm to 10.0 mm from the estimated position of the center of rotation of the eye in a direction away from the retina.

[0228] Example 167: The display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture, the aperture being disposed along the optical axis and spaced 9.5 mm to 10.0 mm from the estimated position of the center of rotation of the eye in a direction away from the retina.

[0229] Example 168: The display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture, the aperture being disposed along the optical axis and spaced approximately 9.7 mm from the estimated position of the center of rotation of the eye.

[0230] Example 169: The display system according to any of the above examples, wherein the processing electronic device includes an electronic device located on the frame.

[0231] Example 170: The display system according to any of the above examples, wherein the processing electronic device includes an electronic device located on the frame and an electronic device disposed at a position remote from the frame.

[0232] Example 171: The display system according to any of the above examples, wherein the processing electronic device includes an electronic device located on the frame and an electronic device located on a waist pack.

[0233] Example 172: A display system according to any of the above examples, wherein at least a portion of the display is transparent and is positioned in front of the user's eyes when the user wears the head-mounted display, such that the transparent portion transmits light from a portion of the user's front environment and the head-mounted display to the user's eyes to provide a view of the portion of the user's front environment and the head-mounted display.

[0234] Example 173: A display system according to any of the above examples, further comprising one or more light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes, and one or more eye tracking cameras that capture images of the eyes using the light from the one or more light sources.

[0235] Example 174: A display system according to any of the above examples, wherein the one or more light sources include at least two light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes.

[0236] Example 175: A display system according to any of the above examples, wherein the one or more light sources include at least three light sources positioned on the frame relative to the user's eyes to illuminate the user's eyes.

[0237] Example 176: A display system according to any of the above examples, wherein the one or more light sources include infrared light emitters.

[0238] Example 177: A display system configured to project light into a user's eyes to display virtual image content in the user's field of view, the eyes having a cornea, iris, pupil, lens, retina, and an optical axis extending through the lens, pupil, and cornea, the display system comprising:

[0239] A frame configured to be supported on the user's head;

[0240] A head-mounted display disposed on the frame, the display configured to project light into the user's eyes to display virtual image content in the user's field of view with at least one of different amounts of divergence and collimation, whereby the displayed virtual image content appears to originate from different depths at different times;

[0241] One or more eye tracking cameras configured to image the user's eyes; and

[0242] A processing electronic device in communication with the display and the one or more eye tracking cameras,

[0243] wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered by a rendering camera located at the pupil of the eye or between the pupil and the cornea of the eye.

[0244] Example 178: The display system according to any one of the above examples, wherein the rendering camera is located at a position between 1.0 mm and 2.0 mm in front of the pupil of the user's eye.

[0245] Example 179: The display system according to any one of the above examples, wherein the rendering camera is located at a position approximately 1.0 mm in front of the pupil of the user's eye.

[0246] Example 180: The display system according to any one of the above examples, wherein the rendering camera is located at a position between 0.25 mm and 1.0 mm in front of the pupil of the user's eye.

[0247] Example 181: The display system according to any one of the above examples, wherein the rendering camera is located at a position between 0.5 mm and 1.0 mm in front of the pupil of the user's eye.

[0248] Example 182: The display system according to any one of the above examples, wherein the rendering camera is located at a position between 0.25 mm and 0.5 mm in front of the pupil of the user's eye.

[0249] Example 183: The display system according to any one of the above examples, wherein the rendering camera is located at the pupil of the eye.

[0250] Example 184: The display system according to any one of the above examples, wherein the rendering camera is not located at the pupil of the eye.

[0251] Example 185: The display system according to any one of the above examples, wherein the rendering camera includes a pinhole camera.

[0252] Example 186: The display system according to any one of the above examples, wherein the aperture includes the pinhole of the pinhole camera.

[0253] Example 187: A method of rendering virtual image content in a display system, the display system being configured to project light onto a user's eye to display the virtual image content in the user's field of view, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, the pupil, and the cornea, the method comprising:

[0254] Determine a position based on imaging of the eye achieved by one or more eye cameras configured to image the eye of the user;

[0255] Render virtual image content by a rendering engine using a rendering camera located at a position based on the determined position, the rendering camera being configured to render a virtual image to be presented to the eye; and

[0256] Project light into the eye of the user through a head-mounted display to display the rendered virtual image content in the user's field of view.

[0257] Example 188: The method according to any of the above examples, wherein the position is the center of rotation of the eye.

[0258] Example 189: The method according to any of the above examples, wherein the position of the rendering camera is located at the center of rotation of the eye.

[0259] Example 190: The method according to any of the above examples, wherein the position is the perspective center of the eye.

[0260] Example 191: The method according to any of the above examples, wherein the position of the rendering camera is located at the perspective center of the eye.

[0261] Example 192: The method according to any of the above examples, wherein the rendering camera is configured to render a virtual image to be presented to the eye, and the virtual image is rendered as if captured by a camera having an aperture closer to the center of rotation than the retina of the eye.

[0262] Example 193: The method according to any of the above examples, wherein the rendering camera is configured to render a virtual image to be presented to the eye, and the virtual image is rendered as if captured by a camera having an aperture located at the center of rotation.

[0263] Example 194: The method according to any of the above examples, wherein the rendering camera is modeled as having an aperture located at the center of rotation of the eye.

[0264] Example 195: The method according to any of the above examples, wherein the rendering camera is modeled as having an aperture, a lens, and a detector.

[0265] Example 196: The method according to any of the above examples, wherein the rendering camera has an aperture at a position along a line between (i) the determined position of the center of rotation of the eye and (ii) the determined position of at least one of the iris or the pupil.

[0266] Example 197: The method according to any of the preceding examples further comprises:

[0267] Determining, by the one or more cameras, a position of a perspective center of the user's eyes, wherein the perspective center of the user's eyes is located at a position less than about 1.0 mm from the pupil of the user's eyes; and

[0268] Rendering, by the rendering engine, the virtual image content using the rendering camera,

[0269] wherein the rendering camera has an aperture located at the determined perspective center position of the user's eyes.

[0270] Example 198: The method according to any of the preceding examples further comprises:

[0271] Rendering, by the rendering engine, the virtual image content using the rendering camera, wherein the rendering camera has an aperture located at a position along a line between (i) the determined center of rotation position of the eyes and (ii) the determined perspective center position of the user's eyes.

[0272] Example 199: The method according to any of the preceding examples further comprises:

[0273] Determining, by a processing electronic device in communication with the one or more cameras, a measure of the change over time of the determined perspective center position of the user's eyes; and

[0274] Indicating, by the processing electronic device, if it is determined that the measure of the change over time exceeds a first threshold, that the rendering engine renders the virtual content using the rendering camera, wherein the rendering camera has an aperture located at the determined center of rotation position of the eyes.

[0275] Example 200: The method according to any of the preceding examples further comprises:

[0276] Indicating, by the processing electronic device, if it is determined that the measure of the change over time is below a second threshold, that the rendering engine renders the virtual content using the rendering camera, wherein the rendering camera has an aperture located at the determined perspective center position of the eyes, and wherein the first threshold indicates a higher level of change over time of the determined perspective center position of the user's eyes compared to the second threshold.

[0277] Example 201: The method according to any of the preceding examples further comprises:

[0278] By the processing electronic device, if it is determined that the metric varying over time is below a second threshold, then instruct the rendering engine to render the virtual content using the rendering camera, wherein the rendering camera has an aperture located at the determined perspective center position of the eye.

[0279] Example 202: The method according to any of the above examples, further comprising:

[0280] By the processing electronic device, if it is determined that the metric varying over time is between the first threshold and the second threshold, then instruct the rendering engine to render the virtual content using the rendering camera, wherein the rendering camera has an aperture located at a point on a line between (i) the determined center of rotation position of the eye and (ii) the determined perspective center position of the eye.

[0281] Example 203: The method according to any of the above examples, further comprising:

[0282] Through at least a portion of the display, the portion being transparent and disposed in front of the user's eyes when the user wears the head-mounted display, transmit light from a portion of the user's front environment and the head-mounted display to the user's eyes to provide a view of the portion of the user's front environment and the head-mounted display.

[0283] Example 204: The method according to any of the above examples, further comprising:

[0284] Determine the position of at least one of the iris, pupil, or lens by the one or more cameras.

[0285] Example 205: The method according to any of the above examples, further comprising:

[0286] By the rendering engine, render the virtual image content using the rendering camera, the rendering camera being configured to present a virtual image to the eye, the virtual image being rendered as if captured by a camera having an aperture located at a position on a line between (i) the determined center of rotation position of the eye and (ii) the determined position of at least one of the iris or pupil.

[0287] Example 206: The method according to any of the above examples, further comprising:

[0288] Determine the position of the perspective center of the user's eye by the one or more cameras, wherein the perspective center of the user's eye is located at a position less than about 1.0 mm from the pupil of the user's eye; and

[0289] Through the rendering engine, the virtual image content is rendered using the rendering camera, which is configured to present the virtual image to the eye, and the virtual image is rendered as if captured by a camera having an aperture located at the determined perspective center position of the user's eye.

[0290] Example 207: The method according to any of the above examples, further comprising:

[0291] Through the rendering engine, the virtual image content is rendered using the rendering camera, which is configured to present the virtual image to the eye, and the virtual image is rendered as if captured by a camera having an aperture located at a position along the line between (i) the determined center of rotation position of the eye and (ii) the determined perspective center position of the user's eye.

[0292] Example 208: The method according to any of the above examples, wherein projecting light into the user's eye through the head-mounted display to display the rendered virtual image content in the user's field of view comprises: projecting light into the user's eye to display the rendered virtual image content in the user's field of view with different divergence amounts such that the virtual image content appears to originate from different depths at different times.

[0293] Example 209: The method according to any of the above examples, wherein the different divergence amounts include zero divergence.

[0294] Example 210: The method according to any of the above examples, wherein the different divergence amounts include collimation.

[0295] Example 211: A display system configured to project light into a user's eye to display virtual image content in the user's field of view, the eye having a cornea, iris, pupil, lens, retina, and an optical axis extending through the lens, pupil, and cornea, the display system comprising:

[0296] A frame configured to be supported on the user's head;

[0297] A head-mounted display disposed on the frame, the display configured to project light into the user's eye to display virtual image content in the user's field of view;

[0298] One or more cameras configured to image the user's eye; and

[0299] A processing electronic device communicates with the display and the one or more cameras, and the processing electronic device is configured to obtain the position of the eye based on an image of the eye obtained by the one or more cameras.

[0300] Wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered by a rendering camera at a position based on the determined position.

[0301] Example 212: The display system according to any of the above examples, wherein the position is an estimate of the center of rotation of the eye.

[0302] Example 213: The display system according to any of the above examples, wherein the position of the rendering camera is located at the estimated center of rotation of the eye.

[0303] Example 214: The display system according to any of the above examples, wherein the position is an estimate of the perspective center of the eye.

[0304] Example 215: The display system according to any of the above examples, wherein the position of the rendering camera is located at the estimated perspective center of the eye.

[0305] Example 216: The display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture closer to the perspective center than the retina.

[0306] Example 217: The display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture closer to the center of rotation than the retina.

[0307] Example 218: The display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture located at the center of rotation.

[0308] Example 219: The display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture closer to the center of rotation than the perspective center.

[0309] Example 220: The display system according to any of the preceding examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture closer to the perspective center than the retina.

[0310] Example 221: The display system according to any of the preceding examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture closer to the perspective center than the center of rotation of the eye.

[0311] Example 222: The display system according to any of the preceding examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture located at the perspective center.

[0312] Example 223: The display system according to any of the preceding examples, wherein the perspective center is not located at the pupil of the eye.

[0313] Example 224: The display system according to any of the preceding examples, wherein the processing electronic device is configured to obtain an estimate of the user's eye pose over time, and wherein the processing electronic device adjusts the position of the rendering camera at least in part based on the user's eye pose.

[0314] Example 225: The display system according to any of the preceding examples, wherein the processing electronic device is configured to track the user's eye pose over time, and wherein the position of the rendering camera is adjusted in response to a change in the user's eye pose over time.

[0315] Example 226: The display system according to any of the preceding examples, wherein the processing electronic device is configured to obtain an estimate of the perspective center by filtering a plurality of estimated perspective center positions.

[0316] Example 227: The display system according to any of the preceding examples, wherein the processing electronic device is configured to obtain an estimate of the perspective center by averaging a plurality of estimated perspective center positions and / or applying a Kalman filter.

[0317] Example 228: The display system according to any of the preceding examples, wherein the perspective center includes a position within the anterior chamber of the user's eye.

[0318] Example 229: The display system according to any of the above examples, wherein the perspective center includes a position in front of the pupil of the user's eye.

[0319] Example 230: The display system according to any of the above examples, wherein the perspective center includes a position between 1.0 mm and 2.0 mm in front of the pupil of the user's eye.

[0320] Example 231: The display system according to any of the above examples, wherein the perspective center includes a position approximately 1.0 mm in front of the pupil of the user's eye.

[0321] Example 232: The display system according to any of the above examples, wherein the perspective center includes a position between 0.25 mm and 1.0 mm in front of the pupil of the user's eye.

[0322] Example 233: The display system according to any of the above examples, wherein the perspective center includes a position between 0.5 mm and 1.0 mm in front of the pupil of the user's eye.

[0323] Example 234: The display system according to any of the above examples, wherein the perspective center includes a position between 0.25 mm and 0.5 mm in front of the pupil of the user's eye.

[0324] Example 235: The display system according to any of the above examples, wherein the perspective center is positioned along the optical axis of the eye, and wherein the processing electronic device is further configured to obtain an estimated position of the perspective center by obtaining an estimated position of the optical axis of the eye.

[0325] Example 236: The display system according to any of the above examples, wherein the perspective center is located along the optical axis of the eye at a position between the outer surface of the cornea and the pupil of the eye, and wherein the processing electronic device is further configured to obtain an estimated position of the perspective center by obtaining an estimated position of the optical axis of the eye.

[0326] Example 237: The display system according to any of the above examples, wherein the perspective center is located along the optical axis of the eye at a position between the outer surface of the cornea and the pupil of the eye, and wherein the processing electronic device is further configured to obtain an estimated position of the perspective center by obtaining an estimated position of the optical axis of the eye and an estimated position of the center of rotation of the eye, the cornea of the eye, the iris of the eye, the retina of the eye, and the pupil of the eye, or any combination thereof.

[0327] Example 238: The display system according to any one of the above examples, wherein the processing electronic device includes an electronic device located on the frame.

[0328] Example 239: The display system according to any one of the above examples, wherein the processing electronic device includes an electronic device located on the frame and an electronic device disposed at a position away from the frame.

[0329] Example 240: The display system according to any one of the above examples, wherein the processing electronic device includes an electronic device located on the frame and an electronic device located on a hip pack.

[0330] Example 241: The display system according to any one of the above examples, wherein at least a portion of the display is transparent and is disposed at a position in front of the user's eyes when the user wears the head-mounted display, such that the transparent portion transmits light from a portion of the user's front environment and the head-mounted display to the user's eyes to provide a view of the portion of the user's front environment and the head-mounted display.

[0331] Example 242: The display system according to any one of the above examples, further comprising one or more light sources, the one or more light sources being disposed on the frame relative to the user's eyes to illuminate the user's eyes, and the one or more cameras capturing images of the eyes using the light from the one or more light sources.

[0332] Example 243: The display system according to any one of the above examples, wherein the one or more light sources include at least two light sources, the at least two light sources being disposed on the frame relative to the user's eyes to illuminate the user's eyes.

[0333] Example 244: The display system according to any one of the above examples, wherein the one or more light sources include at least three light sources, the at least three light sources being disposed on the frame relative to the user's eyes to illuminate the user's eyes.

[0334] Example 245: The display system according to any one of the above examples, wherein the one or more light sources include an infrared light emitter.

[0335] Example 246: The display system according to any one of the above examples, wherein the one or more light sources form one or more bright spots on the eyes, and the processing electronic device is configured to determine the position of the center of curvature of the cornea based on the one or more bright spots.

[0336] Example 247: The display system according to any of the above examples, wherein the one or more light sources form one or more bright spots on the eye, and the processing electronic device is configured to determine a three-dimensional position of the center of curvature of the cornea based on the one or more bright spots.

[0337] Example 248: The display system according to any of the above examples, wherein the one or more cameras are further configured to image the pupil of the user's eye, and wherein the processing electronic device is further configured to determine a position of the pupil of the eye based at least on an image of the pupil from the one or more cameras.

[0338] Example 249: The display system according to any of the above examples, wherein the one or more cameras are further configured to image the pupil of the user's eye, and wherein the processing electronic device is further configured to determine a three-dimensional position of the pupil of the eye based at least on an image of the pupil from the one or more cameras.

[0339] Example 250: The display system according to any of the above examples, wherein the one or more cameras are further configured to image the pupil of the user's eye, and wherein the processing electronic device is further configured to determine a position of the pupil of the eye based on a position of the center of curvature of the cornea and based on an image of the pupil from the one or more cameras.

[0340] Example 251: The display system according to any of the above examples, wherein the processing electronic device is configured to determine an optical axis of the eye based on the three-dimensional position of the center of curvature of the cornea and based on the three-dimensional position of the pupil.

[0341] Example 252: The display system according to any of the above examples, wherein the processing electronic device is configured to determine a visual axis of the eye based on the optical axis.

[0342] Example 253: The display system according to any of the above examples, wherein the processing electronic device is configured to determine a visual axis of the eye based on at least one of the center of curvature of the cornea and the pupil or both the three-dimensional positions of the center of curvature of the cornea and the pupil and the optical axis.

[0343] Example 254: The display system according to any of the above examples, wherein the processing electronic device is configured to determine a three-dimensional position of a center of rotation of the eye based on the three-dimensional position of the center of curvature of the cornea.

[0344] Example 255: The display system according to any one of the above examples, wherein the processing electronic device is configured to determine a three-dimensional position of the center of rotation of the eye based on the three-dimensional position of the center of curvature of the cornea and based on the optical axis.

[0345] Example 256: The display system according to any one of the above examples, wherein the processing electronic device is configured to determine a distance between the eye and the other eye of the user based at least on the three-dimensional position of the center of rotation of the eye.

[0346] Example 257: The display system according to any one of the above examples, wherein the processing electronic device is configured to determine an interpupillary distance between the eye and the other eye of the user based at least on the three-dimensional position of the center of rotation of the eye.

[0347] Example 258: The display system according to any one of the above examples, wherein the processing electronic device is configured to determine a convergence distance of the user based at least on the optical axis of the eye.

[0348] Example 259: The display system according to any one of the above examples, wherein the processing electronic device is configured to determine the convergence distance of the user based at least on the optical axis of the eye and the determined optical axis of the other eye of the user.

[0349] Example 260: The display system according to any one of the above examples, wherein the processing electronic device is configured to determine the convergence distance of the user based at least on the visual axis of the eye and the determined visual axis of the other eye of the user.

[0350] Example 261: The display system according to any one of the above examples, wherein the display is configured to project collimated light into the user's eye.

[0351] Example 262: The display system according to any one of the above examples, wherein the display is configured to project collimated light corresponding to image pixels into the user's eye during a first time period, and to project divergent light corresponding to the image pixels into the user's eye during a second time period.

[0352] Example 263: The display system according to any one of the above examples, wherein the display is configured to project light having a first divergence amount corresponding to image pixels into the user's eye during a first time period, and to project light having a second divergence amount greater than the first divergence amount corresponding to the image pixels into the user's eye during a second time period.

[0353] Example 264: The display system according to any of the above examples, wherein the perspective center is estimated to be close to the pupil of the eye.

[0354] Example 265: The display system according to any of the above examples, wherein the perspective center is estimated to be located between the cornea and the pupil of the eye.

[0355] Example 266: The display system according to any of the above examples, wherein the display is configured to project light into the user's eye to display virtual image content in the user's field of view with at least one of different amounts of divergence and collimation, whereby the displayed virtual image content appears to originate from different depths at different times.

[0356] Example 267: A display system configured to project light into a user's eye to display virtual image content in the user's field of view, the eye having a cornea, iris, pupil, lens, retina, the display system comprising:

[0357] A frame configured to be supported on the user's head;

[0358] A head-mounted display disposed on the frame, the display configured to project light into the user's eye to display virtual image content in the user's field of view with at least one of different amounts of divergence and collimation, whereby the displayed virtual image content appears to originate from different depths at different times;

[0359] One or more eye tracking cameras configured to image the user's eye; and

[0360] A processing electronic device in communication with the display and the one or more eye tracking cameras, the processing electronic device configured to:

[0361] Based on the image of the eye obtained by the one or more eye tracking cameras, obtain an estimate of the position and orientation of the optical axis extending through the lens, pupil, and cornea of the user's eye;

[0362] Identify a specific position along an axis registered with the optical axis of the eye in a rendering space; and

[0363] Present the virtual image content to the user's eye, the virtual image content being rendered by a virtual rendering camera located at the specific position in the rendering space.

[0364] Example 268: The display system according to Example 267, wherein the head-mounted display is configured to project light having a first divergence amount into the user's eyes during a first time period, and is configured to project light having a second divergence amount into the user's eyes during a second time period, wherein the first divergence amount is different from the second divergence amount, and wherein the processing electronic device is further configured to obtain an estimate of the user's vergence distance based on an image of the eyes obtained by the one or more eye tracking cameras, and to transform from projecting light having the first divergence amount into the user's eyes to projecting light having the second divergence amount into the user's eyes based on the estimated vergence distance of the user.

[0365] Example 269: The display system according to Example 267, wherein the head-mounted display is configured to project light having a first divergence amount into the user's eyes during a first time period, and is configured to project light having a second divergence amount into the user's eyes during a second time period, wherein the first divergence amount is different from the second divergence amount, and wherein the processing electronic device is further configured to obtain an estimate of the user's vergence distance based on an image of the eyes obtained by the one or more eye tracking cameras, and to transform from projecting light having the first divergence amount into the user's eyes to projecting light having the second divergence amount into the user's eyes based on the estimated vergence distance of the user.

[0366] Example 270: The display system according to Example 267, wherein the processing electronic device is further configured to determine a position along the optical axis at which the center of rotation of the eyes is estimated to be located based on an image of the eyes obtained by one or more eye tracking cameras, and

[0367] wherein the processing electronic device is configured to identify a specific position along the axis in the rendering space that is registered with the optical axis of the eyes based on the position along the optical axis at which the center of rotation of the eyes is estimated to be located.

[0368] Example 271: The display system according to Example 267, wherein the specific position along the axis in the rendering space includes a position along the axis in the rendering space at which the parallax shift in the rendering space is determined to be minimal.

[0369] Example 272: The display system according to Example 267, wherein the specific position along the axis in the rendering space includes a position along the axis in the rendering space at which the parallax shift in the rendering space is determined to be minimal.

[0370] Example 273: The display system according to any of the above examples, wherein the processing electronic device is configured to obtain the estimation of the center of rotation of the eye based on the determination of a plurality of fixation directions of the user's eyes during a period in which the eyes rotate, according to an image of the eyes obtained by the one or more eye tracking cameras.

[0371] Example 274: The display system according to Example 273, wherein the processing electronic device is configured to determine the fixation direction based on changes in the shape of one or more of the pupil, iris, and limbus of the user's eyes in an image obtained by the one or more eye tracking cameras during a period in which the eyes rotate.

[0372] Example 275: The display system according to any of the above examples, wherein the processing electronic device is configured to determine a position array based on a plurality of spatial positions on an image of the user's eyes obtained by the one or more eye tracking cameras.

[0373] Example 276: The display system according to Example 275, wherein the position array corresponds to at least a portion of an ellipse.

[0374] Example 277: The display system according to Example 275 or 276, wherein the processing electronic device is configured to determine the position array by fitting a curve to the plurality of spatial positions on the image of the user's eyes.

[0375] Example 278: The display system according to Example 277, wherein the curve includes an ellipse.

[0376] Example 279: The display system according to any one of Examples 275 to 278, wherein the plurality of spatial positions on the image include spatial positions on the limbus of the user's eyes in the image.

[0377] Example 280: The display system according to any one of Examples 275 to 279, wherein the plurality of spatial positions on the image include spatial positions on the boundary between the iris and the sclera of the user's eyes in the image.

[0378] Example 281: The display system according to any one of Examples 275 to 279, wherein the plurality of spatial positions on the image include spatial positions on the boundary between the cornea and the sclera of the user's eyes in the image obtained by the one or more eye tracking cameras.

[0379] Example 282: The display system according to any one of Examples 275 to 281, wherein the processing electronic device is configured to determine a plurality of linear paths extending from a position on a first side of the position array through the position array to an opposite second side of the position array.

[0380] Example 283: The display system according to Example 282, wherein the processing electronic device is configured to determine a circular region based on the plurality of linear paths, the circular region having a radius R.

[0381] Example 284: The display system according to Example 283, wherein the radius R corresponds to the average radius of the limbus.

[0382] Example 285: The display system according to Example 283, wherein the radius R corresponds to the measured radius of the limbus of the user's eye.

[0383] Example 286: The display system according to Example 283, wherein the radius R corresponds to the average radius of the pupil.

[0384] Example 287: The display system according to Example 283, wherein the radius R corresponds to the measured radius of the pupil of the user's eye.

[0385] Example 288: The display system according to any one of Examples 282 to 287, wherein the processing electronic device is configured to determine the position and orientation of a normal line passing through a central portion of the circular region.

[0386] Example 289: The display system according to any one of Examples 282 to 288, wherein the processing electronic device is configured to determine the respective positions and orientations of a plurality of normal lines passing through central portions of respective circular regions based on a plurality of images of the eye previously obtained by the one or more eye tracking cameras.

[0387] Example 290: The display system according to Example 289, wherein the processing electronic device is configured to determine the position where the plurality of normal lines converge or intersect.

[0388] Example 291: The display system according to Example 289, wherein the processing electronic device is configured to obtain an estimate of the center of rotation of the user's eye by identifying a region of intersection, convergence, or adjacency of a plurality of the normal lines determined based on images of the user's eye obtained during a period in which the eye rotates.

[0389] Example 292: The display system according to Example 289, wherein the processing electronic device is configured to obtain an estimate of the center of rotation of the user's eye based on the positions and orientations of multiple ones of the plurality of normal vectors determined based on an image of the user's eye obtained during a period of rotation of the eye.

[0390] Example 293: The display system according to any one of Examples 282 to 292, wherein a position on the first side of the array position corresponds to the origin of the coordinate system of one of the one or more eye tracking cameras.

[0391] Example 294: The display system according to any of the above examples, wherein the processing electronic device is configured to obtain an estimate of the center of rotation by filtering, averaging, applying a Kalman filter, or performing any combination of the above items on multiple estimated center of rotation positions.

[0392] Various additional examples of a display system that projects light into one or more eyes of a user to display virtual image content in the user's field of view are described herein, such as the additional examples listed below:

[0393] Additional Example 1: A display system configured to project light into a user's eye to display virtual image content in the user's field of view, the eye having a cornea, iris, pupil, lens, retina, and an optical axis extending through the lens, pupil, and cornea, the display system comprising:

[0394] A frame configured to be supported on the user's head;

[0395] A head-mounted display disposed on the frame, the display configured to project light into the user's eye to display virtual image content to the user's field of view, at least a portion of the display being transparent and disposed at a position in front of the user's eye when the user wears the frame, such that the transparent portion transmits light from a portion of the user's front environment and the head-mounted display to the user's eye to provide a view of the portion of the user's front environment and the head-mounted display;

[0396] An inward-facing imaging system configured to image the user's eye; and

[0397] A processing electronic device in communication with the inward-facing imaging system, the processing electronic device configured to obtain an estimate of the center of rotation of the eye based on a plurality of images of the eye obtained by the inward-facing imaging system, the processing electronic device configured to determine a change in a calculated value of the center of rotation and select a statistically determined estimate of the center of rotation based on the change.

[0398] Additional Example 2: The display system according to Additional Example 1, wherein a reduced variation is used to identify the statistically determined center of rotation.

[0399] Additional Example 3: The display system according to Additional Example 1 or 2, wherein a first set of estimates of the center of rotation is calculated based on a first value of a parameter used to calculate the center of rotation, and a first variation is determined based on the first set of estimates.

[0400] Additional Example 4: The display system according to Additional Example 3, wherein a second set of estimates of the center of rotation is calculated based on a second value of the parameter, and a second variation is determined based on the second set of estimates.

[0401] Additional Example 5: The display system according to Additional Example 4, wherein the first variation and the second variation are compared to determine which set has a reduced variation, and the determination of the estimate of the statistically determined center of rotation is based on this comparison.

[0402] Additional Example 6: The display system according to Additional Example 1 or 2, wherein multiple sets of values of the center of rotation are calculated based on multiple corresponding values of the parameter and corresponding variations determined for different corresponding sets.

[0403] Additional Example 7: The display system according to Additional Example 6, wherein the corresponding variations are compared to determine which set has a reduced variation, and the determination of the estimate of the statistically determined center of rotation is based on this comparison.

[0404] Additional Example 8: The display system according to Additional Example 6 or 7, wherein the value of the parameter for the set having the minimum variation is used to calculate the estimate of the statistically determined center of rotation.

[0405] Additional Example 9: The display system according to Additional Example 6, 7 or 8, wherein the set having the minimum variation is used to calculate the estimate of the statistically determined center of rotation.

[0406] Additional Example 10: The display system according to any one of Additional Examples 3 to 9, wherein the parameter includes the distance from the center of curvature of the cornea to the center of rotation.

[0407] Additional Example 11: The display system according to any one of Additional Examples 3 to 9, wherein the parameter includes the distance along the optical axis from the center of curvature of the cornea to the center of rotation.

[0408] Additional Example 12: The display system according to any of the above additional examples, wherein the variation includes variance and / or standard deviation.

[0409] Additional Example 13: The display system according to any one of the above additional examples further includes one or more light sources, the one or more light sources being disposed on the frame relative to the user's eyes to illuminate the user's eyes, and the inward-facing imaging system forming an image of the eyes using the light from the one or more light sources.

[0410] Additional Example 14: The display system according to Additional Example 13, wherein the one or more light sources include at least two light sources, the at least two light sources being disposed on the frame relative to the user's eyes to illuminate the user's eyes.

[0411] Additional Example 15: The display system according to Additional Example 13 or 14, wherein the one or more light sources include infrared light emitters.

[0412] Additional Example 16: The display system according to any one of Additional Examples 13 to 15, wherein the one or more light sources form one or more bright spots on the eyes, and the processing electronic device is configured to determine the position of the cornea based on the one or more bright spots.

[0413] Additional Example 17: The display system according to any one of Additional Examples 13 to 16, wherein the cornea has a corneal sphere associated therewith, the corneal sphere has a center of curvature, and the processing electronic device is configured to determine the position of the center of curvature of the corneal sphere.

[0414] Additional Example 18: The display system according to Additional Example 17, wherein the cornea has a corneal sphere associated therewith, the corneal sphere has a center of curvature, and the processing electronic device is configured to determine the position of the center of curvature of the corneal sphere based on the one or more bright spots.

[0415] Additional Example 19: The display system according to any one of the above additional examples, wherein the inward-facing imaging system is configured to image the pupil of the eyes.

[0416] Additional Example 20: The display system according to any one of the above additional examples, wherein the processing electronic device is configured to determine the position of the center of the pupil.

[0417] Additional Example 21: The display system according to any one of the above additional examples, wherein the processing electronic device is configured to determine at least a part of the boundary between the iris and the pupil.

[0418] Additional Example 22: The display system according to Additional Example 21, wherein the processing electronic device is configured to determine the center of the boundary between the iris and the pupil.

[0419] Additional Example 23: The display system according to any one of the above additional examples, wherein the processing electronic device is configured to determine the position of the center of the pupil in three-dimensional space relative to the center of curvature of the cornea.

[0420] Additional Example 24: The display system according to any one of the above additional examples, wherein the processing electronic device is configured to determine the position and orientation of the optical axis.

[0421] Additional Example 25: The display system according to Additional Example 24, wherein the processing electronic device is configured to determine the position and orientation of the optical axis based on the position of the center of the pupil in three-dimensional space.

[0422] Additional Example 26: The display system according to any one of the above additional examples, wherein the processing electronic device is configured to determine the position and orientation of the optical axis based on the position of the center of the pupil in three-dimensional space relative to the center of curvature of the cornea.

[0423] Additional Example 27: The display system according to any one of the above additional examples, wherein the processing electronic device is configured to determine the position of the center of rotation of the eye based on the center of curvature of the cornea.

[0424] Additional Example 28: The display system according to any one of the above additional examples, wherein the processing electronic device is configured to determine the position of the center of rotation of the eye based on the center of curvature of the cornea and the position and orientation of the optical axis.

[0425] Additional Example 29: The display system according to Additional Example 28, wherein the processing electronic device is configured to determine the position of the center of rotation of the eye by translating a specific distance along the optical axis from the center of curvature of the cornea.

[0426] Additional Example 30: The display system according to Additional Example 29, wherein the specific distance from the center of curvature to the center of rotation is between 4.0 mm and 6.0 mm.

[0427] Additional Example 31: The display system according to Additional Example 29 or 30, wherein the specific distance from the center of curvature to the center of rotation is about 4.7 mm.

[0428] Additional Example 32: The display system according to Additional Example 20 or 30, wherein the processing electronic device is configured to determine the specific distance based at least on one or more images of the eye previously obtained by the inward-facing imaging system.

[0429] Additional Example 33: The display system according to any one of the above additional examples, wherein the processing electronic device includes an electronic device located on the frame.

[0430] Additional Example 34: The display system according to any one of the above additional examples, wherein the processing electronic device includes an electronic device located on the frame and an electronic device disposed at a position away from the frame.

[0431] Additional Example 35: The display system according to any one of the above additional examples, wherein the processing electronic device includes an electronic device located on the frame and an electronic device located on a waist pack.

[0432] Additional Example 36: The display system according to any one of the above additional examples, wherein at least a portion of the display is transparent and is disposed in front of the user's eyes when the user wears the head-mounted display, such that the transparent portion transmits light from a portion of the user's front environment and the head-mounted display to the user's eyes to provide a view of the portion of the user's front environment and the head-mounted display.

[0433] Additional Example 37: The display system according to any one of the above additional examples, wherein the head-mounted display receives light from a portion of the user's front environment with a first divergence amount and transmits light from a portion of the user's front environment to the user's eyes with a second divergence amount that is substantially the same as the first divergence amount.

[0434] Additional Example 38: The display system according to any one of the above additional examples, wherein the processing electronic device is configured to obtain an estimate of the center of rotation by filtering, averaging, applying a Kalman filter, or performing any combination of the above items on a plurality of estimated center of rotation positions.

[0435] Additional Example 39: The display system according to any one of the above additional examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture located at the determined center of rotation position of the user's eyes.

[0436] Additional Example 40: The display system according to any one of the above additional examples, wherein the processing electronic device is configured to render a virtual image to be presented to the eyes using a rendering camera located at the center of rotation.

[0437] Additional Example 41: The display system according to any of the above additional examples, wherein the processing electronic device is configured to use a rendering camera, the rendering camera is configured to render a virtual image to be presented to the eye, and the virtual image is rendered as if captured by a camera having an aperture closer to the center of rotation than the retina of the eye.

[0438] Additional Example 42: The display system according to any of the above additional examples, wherein the processing electronic device is configured to use a rendering camera, the rendering camera is configured to render a virtual image to be presented to the eye, and the virtual image is rendered as if captured by a camera having an aperture located at the center of rotation of the eye.

[0439] Additional Example 43: The display system according to any of the above additional examples, wherein the processing electronic device is configured to use a rendering camera located at the center of rotation to render a virtual image to be presented to the eye, and the rendering camera is modeled as having an aperture located at the center of rotation of the eye.

[0440] Additional Example 44: The display system according to any one of Additional Examples 1 to 9, wherein the processing electronic device is configured to select the estimated statistically determined center of rotation based on the change during a calibration process.

[0441] Additional Example 45: The display system according to any of the above examples or additional examples, wherein the processing electronic device is configured to obtain the estimate of the center of rotation of the eye based on the determination of a plurality of fixation directions of the user's eye during a period of eye rotation according to an image of the eye obtained by the one or more eye tracking cameras.

[0442] Additional Example 46: The display system according to any of the above examples or additional examples, wherein the processing electronic device is configured to determine the fixation direction based on a change in the shape of one or more of the pupil, iris, and limbus of the user's eye in an image obtained by the one or more eye tracking cameras during a period of eye rotation.

[0443] Additional Example 47: The display system according to any of the above examples or additional examples, wherein the processing electronic device is configured to determine a position array based on a plurality of spatial positions on an image of the user's eye obtained by the one or more eye tracking cameras.

[0444] Additional Example 48: The display system according to any of the above examples or additional examples, wherein the position array corresponds to at least a part of an ellipse.

[0445] Additional Example 49: The display system according to any one of the above examples or additional examples, wherein the processing electronic device is configured to determine the position array by fitting a curve to the plurality of spatial positions on the image of the user's eyes.

[0446] Additional Example 50: The display system according to any one of the above examples or additional examples, wherein the curve includes an ellipse.

[0447] Additional Example 51: The display system according to any one of the above examples or additional examples, wherein the plurality of spatial positions on the image include spatial positions on the limbus of the user's eyes in the image.

[0448] Additional Example 52: The display system according to any one of the above examples or additional examples, wherein the plurality of spatial positions on the image include spatial positions on the boundary between the iris and the sclera of the user's eyes in the image.

[0449] Additional Example 53: The display system according to any one of the above examples or additional examples, wherein the plurality of spatial positions on the image include spatial positions on the boundary between the cornea and the sclera of the user's eyes in the image obtained by the one or more eye tracking cameras.

[0450] Additional Example 54: The display system according to any one of the above examples or additional examples, wherein the processing electronic device is configured to determine a plurality of linear paths extending from positions on a first side of the position array through the position array to an opposite second side of the position array.

[0451] Additional Example 55: The display system according to any one of the above examples or additional examples, wherein the processing electronic device is configured to determine a circular region based on the plurality of linear paths, the circular region having a radius R.

[0452] Additional Example 56: The display system according to any one of the above examples or additional examples, wherein the radius R corresponds to the average radius of the limbus.

[0453] Additional Example 57: The display system according to any one of the above examples or additional examples, wherein the radius R corresponds to the measured radius of the limbus of the user's eyes.

[0454] Additional Example 58: The display system according to any one of the above examples or additional examples, wherein the radius R corresponds to the average radius of the pupil.

[0455] Additional Example 59: The display system according to any one of the above examples or additional examples, wherein the radius R corresponds to the measured radius of the pupil of the user's eyes.

[0456] Additional example 60: The display system according to any of the above examples or additional examples, wherein the processing electronic device is configured to determine the position and direction of the normal line passing through the central portion of the circular area.

[0457] Additional example 61: The display system according to any of the above examples or additional examples, wherein the processing electronic device is configured to determine the respective positions and directions of multiple normal lines passing through the central portions of the respective circular areas based on multiple images of the eyes previously obtained by the one or more eye tracking cameras.

[0458] Additional example 62: The display system according to any of the above examples or additional examples, wherein the processing electronic device is configured to determine the position where the multiple normal lines converge or intersect.

[0459] Additional example 63: The display system according to any of the above examples or additional examples, wherein the processing electronic device is configured to obtain an estimate of the center of rotation of the user's eyes by identifying an area of intersection, convergence, or adjacency among multiple of the normal lines determined based on images of the user's eyes obtained during a period when the eyes rotate.

[0460] Additional example 64: The display system according to any of the above examples or additional examples, wherein the processing electronic device is configured to obtain the estimate of the center of rotation of the user's eyes based on the positions and directions of multiple of the normal lines determined based on images of the user's eyes obtained during a period when the eyes rotate.

[0461] Additional example 65: The display system according to any of the above examples or additional examples, wherein the position on the first side of the array position corresponds to the origin of the coordinate system of one of the one or more eye tracking cameras.

[0462] Additional example 66: The display system according to any of the above examples or additional examples, wherein the processing electronic device is configured to obtain the estimate of the center of rotation by filtering, averaging, applying a Kalman filter, or performing any combination of the above on multiple estimated center of rotation positions.

[0463] Any of the above examples or additional examples can be combined. Additionally, any of the above examples or additional examples can be integrated with a head-mounted display. Additionally, any of the above examples or additional examples can be implemented in a single depth plane and / or one or more variable depth planes (e.g., one or more elements with variable focusing capabilities that provide accommodation cues that vary over time).

[0464] In addition, apparatuses and methods for determining various values, parameters, etc. (such as but not limited to anatomical, optical, and geometric features, positions, and orientations, etc.) are disclosed herein. Examples of such parameters include, for example, the center of rotation of the eye, the center of curvature of the cornea, the center of the pupil, the boundary of the pupil, the center of the iris, the boundary of the iris, the boundary of the limbus, the optical axis of the eye, the visual axis of the eye, the perspective center, but are not limited to these. The determination of such values, parameters, etc. described herein includes their estimation and does not necessarily need to be exactly consistent with the actual values. For example, the determination of the center of rotation of the eye, the center of curvature of the cornea, the center or boundary of the pupil or iris, the boundary of the limbus, the optical axis of the eye, the visual axis of the eye, the perspective center, etc. can be an estimate, approximation, or value that is close to but not equal to the actual (e.g., anatomical, optical, or geometric) value or parameter. In some cases, for example, root mean square estimation techniques are used to obtain an estimate of such values. As an example, some of the techniques described herein involve identifying the position or point at which rays or vectors intersect. However, such rays or vectors may not intersect. In this example, the position or point can be estimated. For example, the position or point can be determined based on root mean square or other estimation techniques (e.g., the position or point can be estimated as being close to or closest to the ray or vector). Other processes can also be used to estimate, approximate, or otherwise provide values that may not be consistent with the actual values. Therefore, the terms "determine" and "estimate" or "determined" and "estimated" are used interchangeably herein. Thus, a reference to such determined values can include values that are estimated, approximated, or close to the actual value. Therefore, the "determining a parameter or value" referred to above or elsewhere in this document should not be precisely limited to the actual value, but can include values that are estimated, approximated, or close to the actual value.

[0465] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. The present summary and the following detailed description are not intended to limit or restrict the scope of the subject matter of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0466] Figure 1 An illustration showing a mixed reality scene having a specific virtual reality object and a specific physical object viewed by a person is shown.

[0467] Figure 2 An example of a wearable system is schematically shown.

[0468] Figure 3 An example of the components of a wearable system is schematically shown.

[0469] Figure 4 An example of a waveguide stack of a wearable device for outputting image information to a user is schematically shown.

[0470] Figure 5 An example of an eye is schematically shown.

[0471] Figure 5A An example coordinate system for determining the eye pose of an eye is schematically shown.

[0472] Figure 6 It is a schematic diagram of a wearable system including an eye tracking system.

[0473] Figure 7A It is a block diagram of a wearable system that may include an eye tracking system.

[0474] Figure 7B It is a block diagram of a rendering controller in a wearable system.

[0475] Figure 8A It is an eye schematic diagram showing the corneal globe of an eye.

[0476] Figure 8B An example corneal bright spot detected by an eye tracking camera is shown.

[0477] Figures 8C to 8E An example stage of locating the corneal center of a user by an eye tracking module in a wearable system is shown.

[0478] Figures 9A to 9C An example normalization of the coordinate system of an eye tracking image is shown.

[0479] Figures 9D to 9G An example stage of locating the pupil center of a user by an eye tracking module in a wearable system is shown.

[0480] Figure 9H An example calculated position of the pupil center with and without considering corneal refraction is shown.

[0481] Figures 9I to 9M An example experimental variation of the calculated eye rotation center based on different calculated pupil centers is shown.

[0482] Figure 10 An example of an eye is shown, including the optical axis and visual axis of the eye and the rotation center of the eye.

[0483] Figure 11 It is a process flow diagram of an example of a method for rendering content and providing registration feedback in a wearable device using eye tracking.

[0484] Figure 12 It is a set of example diagrams showing how a wearable system can switch depth planes in response to a user's eye movement.

[0485] Figure 13 A mixed reality system is shown in which a particular virtual object can be discretized in depth into one or more depth planes.

[0486] Figure 14A and 14B An example is shown in which the mixed reality system uses a single depth plane to display a virtual object.

[0487] Figure 14C and 14D An example is shown in which the mixed reality system uses two adjacent depth planes to display a virtual object to generate an accommodation cue between the two adjacent depth planes.

[0488] Figure 14E and 14F An example is shown in which the mixed reality system uses two or more depth planes to display a virtual object to simultaneously generate two or more accommodation cues.

[0489] Figure 15A An example is shown in which the mixed reality system, in the presence of misalignment of the perspective center, Figure 14A and 14B displays a virtual object in a certain way.

[0490] Figure 15B An example is shown in which the mixed reality system, in the presence of misalignment of the perspective center, Figure 14C and 14D displays a virtual object in a certain way.

[0491] Figure 15C An example is shown in which the mixed reality system, in the presence of misalignment of the perspective center, Figure 14E and 14F displays a virtual object in a certain way.

[0492] Figure 16A and 16B An example is shown in which the pinhole of the rendering camera is aligned with the perspective center of the eye or approximately with the pupil of the eye.

[0493] Figure 16C Another example is shown in which the pinhole of the rendering camera is aligned with the perspective center of the eye or approximately with the pupil of the eye.

[0494] Figure 17A and 17B An example is shown in which the pinhole of the rendering camera is aligned with the center of rotation of the eye.

[0495] Figure 17C Another example is shown in which the pinhole of the rendering camera is aligned with the center of rotation of the eye.

[0496] Figure 18A and 18B are a set of example diagrams showing eye-tracking data related to the perspective center of a user.

[0497] Figure 18C are example diagrams showing how the rendering camera position can vary with eye-tracking data.

[0498] Figure 18D is a schematic diagram of a user's eye, showing various rendering camera positions that can be used as a function of eye-tracking data.

[0499] Figure 19 shows a graphical representation of determining an estimate of the CoR using an ellipse projected onto an image of the user's eye (e.g., the limbus). Rays can be traced through the projected ellipse to form a cone in which a circle is fitted. In certain embodiments, the CoR can be estimated using the normal to one or more circles.

[0500] Figure 20 is a flowchart of an example process for determining the center of rotation of an eye based on the limbus of the eye.

[0501] Figure 21A shows a first ellipse (hereinafter referred to as the "projected ellipse") projected onto an image of the user's limbus determined based on the user's first fixation. Rays passing through the projected ellipse are shown forming a cone in which a circle is fitted.

[0502] Figure 21B shows a second projected ellipse determined based on a second fixation of the user's eye. A second cone and the circle fitted to it are also shown.

[0503] Figure 21C shows a third projected ellipse determined based on a third fixation of the user's eye. A third cone and the circle fitted to it are also shown.

[0504] Figure 21D shows determining the center of rotation based on the circles obtained using the above projected ellipses.

[0505] Figure 22 shows an optical system including two point light sources, an aperture, a lens, and a projection screen.

[0506] Figure 23A shows an embodiment of the optical system in a first stage.

[0507] Figure 23B shows the optical system in a second stage.

[0508] Figure 24A shows another embodiment of the optical system in a first stage.

[0509] Figure 24B shows the optical system in the second stage.

[0510] In all the figures, reference numerals may be reused to indicate the correspondence between the elements being referenced. The figures are provided to illustrate example embodiments described herein and are not intended to limit the scope of the disclosure. Detailed Description

[0511] Referring now to the figures, in all the figures, like reference numerals represent like components. Unless otherwise indicated, the figures are schematic and not necessarily drawn to scale.

[0512] Example of 3D display of a wearable system

[0513] A wearable system (also referred to herein as an augmented reality (AR) system) may be configured to present 2D or 3D virtual images to a user. The images may be still images, video frames, or videos, combinations of the foregoing, and so on. At least a portion of the wearable system may be implemented on a wearable device that is capable of presenting VR, AR, or MR environments, alone or in combination, for user interaction. The wearable device may be used interchangeably with an AR device (ARD). Additionally, for the purposes of this disclosure, the term "AR" may be used interchangeably with the term "MR".

[0514] Figure 1 shows an illustration of a mixed reality scene having a particular virtual reality object and a particular physical object as viewed by a person. In Figure 1 it, an MR scene 100 is shown, where a user of MR technology sees a real-world park-like setting 110 featuring a person, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of MR technology also perceives that he "sees" a robotic statue 130 standing on the real-world platform 120, and a flying cartoonish avatar character 140 that looks like a bumblebee, even though these elements do not exist in the real world.

[0515] To make a three-dimensional (3D) display produce a realistic sense of depth, and more specifically, to produce a simulated sense of surface depth, it may be desirable for each point in the field of view of the display to produce an accommodative response corresponding to its virtual depth. If the accommodative response to a display point does not correspond to the virtual depth of that point (determined by the binocular depth cues of convergence and stereopsis), the human eye may experience an accommodation conflict, resulting in unstable imaging, harmful eye fatigue, headaches, and, in the absence of accommodation information, an almost complete lack of surface depth.

[0516] VR, AR, and MR experiences can be provided by a display system having a display in which images corresponding to multiple depth planes are provided to a viewer. The images for each depth plane may be different (e.g., providing slightly different scene or object presentations), and can be individually focused by the viewer's eyes, thereby contributing to providing depth cues to the user based on the eye's accommodation (which is required to focus different image features of scenes located at different depth planes), or based on observing different image features at different depth planes being out of focus. As discussed elsewhere herein, such depth cues provide a reliable sense of depth.

[0517] Figure 2 An example of a wearable system 200 that can be configured to provide an AR / VR / MR scene is shown. The wearable system 200 may also be referred to as the AR system 200. The wearable system 200 includes a display 220 and various mechanical and electronic modules and systems that support the functionality of the display 220. The display 220 can be coupled to a frame 230 that can be worn by a user, wearer, or viewer 210. The display 220 can be positioned in front of the eyes of the user 210. The display 220 can present AR / VR / MR content to the user. The display 220 can include a head-mounted display (HMD) worn on the user's head.

[0518] In some embodiments, a speaker 240 is coupled to the frame 230 and is positioned near the user's ear canal (in some embodiments, another speaker (not shown) is positioned near the user's other ear canal to provide stereo / plastic sound control). The display 220 can include an audio sensor (e.g., a microphone) 232 to detect an audio stream from the environment and capture ambient sound. In some embodiments, one or more other audio sensors (not shown) are placed to provide stereo reception. Stereo reception can be used to determine the location of a sound source. The wearable system 200 can perform speech or voice recognition on the audio stream.

[0519] The wearable system 200 can include an outward-facing imaging system 464 (as Figure 4 shown) that observes the world in the user's surrounding environment. The wearable system 200 can also include an inward-facing imaging system 462 (as Figure 4as shown), which can track the user's eye movements. The inward-facing imaging system can track the movement of one eye or the movement of both eyes. The inward-facing imaging system 462 can be attached to the frame 230 and can be in electrical communication with the processing module 260 or 270, which can process the image information acquired by the inward-facing imaging system to determine, for example, the pupil diameter or orientation, eye movement, or eye pose of the user 210's eyes. The inward-facing imaging system 462 can include one or more cameras. For example, at least one camera can be used to image each eye. The images acquired by the cameras can be used to separately determine the pupil size or eye pose of each eye, thereby allowing image information dynamically customized for that eye to be presented to each eye.

[0520] As an example, the wearable system 200 can use the outward-facing imaging system 464 or the inward-facing imaging system 462 to acquire images of the user's pose. The images can be still images, video frames, or videos.

[0521] The display 220 can be operably coupled 250 to the local data processing module 260, for example, via a wired lead or a wireless connection. The local data processing module 260 can be installed in various configurations, such as fixedly attached to the frame 230, fixedly attached to a helmet or hat worn by the user, embedded in the earphone, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, in a belt-coupled configuration).

[0522] The local processing and data module 260 can include a hardware processor and a digital memory such as non-volatile memory (e.g., flash memory), both of which can be used to assist in the processing, caching, and storage of data. The data can include the following data: a) data captured from environmental sensors, which can be operably coupled to the frame 230 or otherwise attached to the user 210, such as image capture devices (e.g., cameras in the inward-facing imaging system or the outward-facing imaging system), audio sensors (e.g., microphones), inertial measurement units (IMUs), accelerometers, compasses, global positioning system (GPS) units, radios, or gyroscopes; or b) data acquired or processed using the remote processing module 270 or the remote data repository 280, which may be passed to the display 220 after such processing or retrieval. The local processing and data module 260 can be operably coupled to the remote processing module 270 or the remote data repository 280 via a communication link 262 or 264 (such as via a wired or wireless communication link), such that these remote modules are available as resources to the local processing and data module 260. Additionally, the remote processing module 280 and the remote data repository 280 can be operably coupled to each other.

[0523] In some embodiments, the remote processing module 270 may include one or more processors configured to analyze and process data or image information. In some embodiments, the remote data repository 280 may include a digital data storage facility that may be made available in a "cloud" resource configuration via the Internet or other network. In some embodiments, all data is stored and all computations are performed in the local processing and data module, which allows for full autonomous use from the remote module.

[0524] Example components of a wearable system

[0525] Figure 3 Exemplary components of a wearable system are schematically shown. Figure 3 A wearable system 200 is shown, which may include a display 220 and a frame 230. An enlarged view 202 schematically shows the various components of the wearable system 200. In certain embodiments, Figure 3 One or more of the components shown may be part of the display 220. The various components, individually or in combination, may collect various data (e.g., audio or visual data) associated with the user of the wearable system 200 or the user's environment. It should be understood that other embodiments may have more or fewer components depending on the application for which the wearable system is used. Nevertheless, Figure 3 The basic concept of the various components and some of the types of data that may be collected, analyzed, and stored by the wearable system are provided.

[0526] Figure 3 An exemplary wearable system 200 is shown, which may include a display 220. The display 220 may include a display lens 226, which may be mounted to the user's head or to a housing or frame 230 corresponding to the frame 230. The display lens 226 may include one or more transparent lenses that are positioned by the housing 230 in front of the user's eyes 302, 304 and may be configured to reflect the projected light 338 into the eyes 302, 304 and facilitate beam shaping while also allowing transmission of at least some light from the local environment. The wavefront of the projected light beam 338 may be bent or focused to be consistent with the desired focal length of the projected light. As shown, two wide field of view machine vision cameras 316 (also referred to as world cameras) may be coupled to the housing 230 to image the environment around the user. These cameras 316 may be dual-shot visible / non-visible (e.g., infrared) light cameras. The cameras 316 may be Figure 4A portion of the outward-facing imaging system 464 shown in the figure. The images acquired by the world camera 316 can be processed by the pose processor 336. For example, the pose processor 336 can implement one or more object recognizers 708 (e.g., as shown in FIG. 7) to recognize the pose of the user or another person in the user's environment, or to recognize physical objects in the user's environment.

[0527] Continuing to refer to Figure 3 , a pair of scanning laser shaping wavefront (e.g., for depth) light projector modules are shown, which have display mirrors and optics configured to project light 338 into the eyes 302, 304. The depicted view also shows two miniature infrared cameras 324 paired with an infrared light source 326 (e.g., a light-emitting diode "LED"), which are configured to be able to track the user's eyes 302, 304 to support rendering and user input. The cameras 324 can be Figure 4 A portion of the inward-facing imaging system 462 shown in the figure. The wearable system 200 can also have a sensor assembly 339, which can include X, Y, and Z-axis accelerometer functionality, a magnetic compass, and X, Y, and Z-axis gyroscope functionality, preferably providing data at a relatively high frequency (e.g., 200 Hz). The sensor assembly 339 can be part of the IMU described in Figure 2 . The depicted system 200 can also include a head pose processor 336, such as an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), or an ARM processor (advanced reduced instruction set machine), which can be configured to calculate the real-time or near-real-time head pose of the user based on the wide-field-of-view image information output from the capture device 316. The head pose processor 336 can be a hardware processor and can be implemented as Figure 2 A portion of the local processing and data module 260 shown in the figure.

[0528] The wearable system can also include one or more depth sensors 234. The depth sensors 234 can be configured to measure the distance between an object in the environment and the wearable device. The depth sensors 234 can include a laser scanner (e.g., lidar), an ultrasonic depth sensor, or a depth-sensing camera. In certain embodiments, if the camera 316 has depth-sensing functionality, the camera 316 can also be considered a depth sensor 234.

[0529] A processor 332 is also shown, which is configured to perform digital or analog processing to derive a pose from gyroscope, compass, or accelerometer data from the sensor assembly 339. The processor 332 can be Figure 2 A portion of the local processing and data module 260 shown in the figure. Figure 3The wearable system 200 shown may also include a positioning system such as GPS 337 (Global Positioning System) to assist in pose and positioning analysis. Additionally, the GPS may further provide remote (e.g., cloud-based) information about the user's environment. This information can be used to identify objects or information in the user's environment.

[0530] The wearable system can combine data obtained by the GPS 337 with a remote computing system (e.g., remote processing module 270, another user's ARD, etc.) that can provide more information about the user's environment. As an example, the wearable system can determine the user's location based on GPS data and retrieve a world map (e.g., by communicating with the remote processing module 270), where the world map includes virtual objects associated with the user's location. As another example, the wearable system 200 can use the world camera 316 (which can be Figure 4 part of the outward-facing imaging system 464 shown) to monitor the environment. Based on the images obtained by the world camera 316, the wearable system 200 can detect objects in the environment (e.g., by using one or more object recognizers 708 shown in FIG. 7). The wearable system can further use the data obtained by the GPS 337 to interpret characters.

[0531] The wearable system 200 may also include a rendering engine 334, which can be configured to provide rendering information local to the user to facilitate the operation of the scanner and imaging into the user's eyes for the user to view the world. The rendering engine 334 can be implemented by a hardware processor (e.g., a central processing unit or a graphics processing unit). In some embodiments, the rendering engine is part of the local processing and data module 260. The rendering engine 334 can be communicatively (e.g., via a wired or wireless link) coupled to other components of the wearable system 200. For example, the rendering engine 334 can be coupled to the eye camera 324 via the communication link 274, and to the projection subsystem 318 (which can project light into the user's eyes 302, 304 via a scanning laser device in a manner similar to a retinal scanning display) via the communication link 272. The rendering engine 334 can also communicate with other processing units (e.g., the sensor pose processor 332 and the image pose processor 336) via the links 276 and 294, respectively.

[0532] A camera 324 (e.g., a miniature infrared camera) can be used to track eye pose to support rendering and user input. Some example eye poses can include where the user is looking. Or at what depth he or she is focusing (which can be estimated by eye convergence). The GPS 337, gyroscope, compass, and accelerometer 339 can be used to provide a rough or quick pose estimate. One or more of the cameras 316 can acquire images and poses, which, combined with data from associated cloud computing resources, can be used to map the local environment and share the user's view with other users.

[0533] Figure 3 The example components shown are for illustrative purposes only. For ease of illustration and description, multiple sensors and other functional modules are shown together. Some embodiments may include only one or a subset of these sensors or modules. Additionally, the locations of these components are not limited to Figure 3 the locations shown. Certain components can be mounted to or housed within other components (e.g., a strap-mounted component, a hand-held component, or a helmet component). As an example, the image pose processor 336, the sensor pose processor 332, and the rendering engine 334 can be located in a waist pack and configured to communicate with other components of the wearable system via wireless communication (e.g., ultra-wideband, Wi-Fi, Bluetooth, etc.) or via wired communication. The shown housing 230 is preferably user-wearable on the head. However, certain components of the wearable system 200 can be worn on other parts of the user's body. For example, the speaker 240 can be inserted into the user's ear to provide sound to the user.

[0534] Regarding the projection of light 338 towards the user's eyes 302, 304, in some embodiments, the position that the center of the user's eyes geometrically tends towards can be measured using the camera 324, which generally coincides with the focusing position or "depth of focus" of the eyes. The three-dimensional surface of all the points that the eyes tend towards can be referred to as the "horopter". The focal length can exhibit a finite number of depths or can vary infinitely. Examples of the wearable device and other display systems of the present disclosure are also described in U.S. Patent Publication No. 2016 / 0270656, the entire content of which is incorporated herein by reference.

[0535] The human visual system is complex, and it is challenging to provide a realistic perception of depth. A viewer of an object may perceive the object as "three-dimensional" due to the combination of convergence and accommodation. The convergence movement of the two eyes relative to each other (e.g., the rolling movement of the pupils towards or away from each other to converge the lines of sight of the eyes to fixate on an object) is closely related to the focusing (or "accommodation") of the eye lenses. Under normal circumstances, changing the focus of the eye lenses or accommodating the eyes to change the focus from one object to another at a different distance will automatically result in a matching change in convergence to the same distance under what is known as the "accommodation-vergence reflex" relationship. Similarly, under normal circumstances, a change in convergence will trigger a matching change in accommodation. A display system that provides a better match between accommodation and convergence can form a more realistic and comfortable three-dimensional image simulation.

[0536] Regardless of where the eyes are focused, spatially coherent light with a beam diameter of less than about 0.7 millimeters can be correctly resolved by the human eye. Thus, to create the illusion of an appropriate depth of focus, the eye convergence can be tracked by camera 324, and the rendering engine 334 and projection subsystem 318 can be utilized to render all objects on or near the binocular field of view in focus, as well as all other objects with varying degrees of defocus (e.g., using deliberately induced blur). Preferably, system 220 renders to the user at a frame rate of approximately 60 frames per second or higher. As described above, preferably, camera 324 can be used for eye tracking, and the software can be configured to pick up not only the convergence geometry but also focus position cues to be used as user input. Preferably, such a display system is configured with brightness and contrast suitable for use during the day or at night.

[0537] In some embodiments, the display system preferably has a visual object alignment delay of less than about 20 milliseconds, an angular alignment delay of less than about 0.1 degrees, and a resolution delay of about 1 arc minute, without being limited by theory, which is considered to be close to the limit of the human eye. The display system 220 can be integrated with a positioning system that may include GPS elements, optical tracking, a compass, an accelerometer, or other data sources to assist in determining position and orientation; the positioning information can be used to facilitate accurate rendering in the view of the relevant world that the user sees (e.g., such information will help the glasses understand their position relative to the real world).

[0538] In some embodiments, the wearable system 200 is configured to display one or more virtual images based on the accommodation of the user's eyes. Different from existing 3D display methods that force the user to focus on the image projection location, in some embodiments, the wearable system is configured to automatically change the focus of the projected virtual content to allow for a more comfortable viewing of the one or more images presented to the user. For example, if the current focus of the user's eyes is 1m, the image can be projected to coincide with the user's focus. If the user moves the focus to 3m, the image is projected to coincide with the new focus. Thus, the wearable system 200 of some embodiments does not force the user to reach a predetermined focus, but rather allows the user's eyes to function in a more natural way.

[0539] Such a wearable system 200 can eliminate or reduce the occurrence of eye fatigue, headaches, and other physiological symptoms commonly observed with virtual reality devices. To achieve this, various embodiments of the wearable system 200 are configured to project virtual images with a varying focal length through one or more variable focal elements (VFE). In one or more embodiments, 3D perception can be achieved through a multi-plane focusing system that projects images at a fixed focal plane away from the user. Other embodiments employ a variable plane focus, where the focal plane moves back and forth in the z direction to coincide with the user's current focusing state.

[0540] In both the multi-plane focusing system and the variable plane focusing system, the wearable system 200 can employ eye tracking to determine the convergence of the user's eyes, determine the user's current focus, and project the virtual image to a predetermined focus. In other embodiments, the wearable system 200 includes a light modulator that variably projects a zoom beam in a raster pattern across the entire retina through a fiber optic scanner or other light generating source. Thus, as further described in U.S. Patent Publication No. 2016 / 0270656, which is incorporated herein by reference in its entirety, the ability of the display of the wearable system 200 to project images with a varying focal length not only alleviates the accommodation of the user when viewing 3D objects, but can also be used to compensate for the user's visual anomalies. In some other embodiments, a spatial light modulator can project an image to the user through various optical components. For example, as further described below, the spatial light modulator can project an image onto one or more waveguides, which then transmit the image to the user.

[0541] Waveguide stack assembly

[0542] Figure 4Shows an example of a waveguide stack for outputting image information to a user. Wearable system 400 includes a waveguide stack or stacked waveguide assembly 480, which can be used to provide three-dimensional perception to the eye / brain using a plurality of waveguides 432b, 434b, 436b, 438b, 4400b. In some embodiments, wearable system 400 can correspond to Figure 2 wearable system 200 of Figure 4 Some parts of the wearable system 200 are shown schematically in more detail. For example, in some embodiments, waveguide assembly 480 can be integrated into Figure 2 display 220 of

[0543] Continuing to refer to Figure 4 , waveguide assembly 480 can also include a plurality of features 458, 456, 454, 452 located between the waveguides. In some embodiments, features 458, 456, 454, 452 can be lenses. In other embodiments, features 458, 456, 454, 452 can not be lenses. Instead, they can simply be spacers (e.g., cladding or structures for forming air gaps).

[0544] Waveguides 432b, 434b, 436b, 438b, 440b or a plurality of lenses 458, 456, 454, 452 can be configured to send image information to the eye with various levels of wavefront curvature or light divergence. Each waveguide level can be associated with a specific depth plane and can be configured to output image information corresponding to that depth plane. Image injection devices 420, 422, 424, 426, 428 can be used to inject image information into waveguides 440b, 438b, 436b, 434b, 432b, and each waveguide 440b, 438b, 436b, 434b, 432b can be configured to distribute incident light through each corresponding waveguide for output towards eye 410. Light exits from the output surface of image injection devices 420, 422, 424, 426, 428 and is injected into the corresponding input edges of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single light beam (e.g., a collimated beam) can be injected into each waveguide to output the entire field of cloned collimated beams, and these cloned collimated beams are directed towards eye 410 at a specific angle (and divergence amount) corresponding to the depth plane associated with a particular waveguide.

[0545] In some embodiments, the image injection devices 420, 422, 424, 426, 428 are discrete displays, each generating image information for injection into a respective waveguide 440b, 438b, 436b, 434b, 432b. In some other embodiments, the image injection devices 420, 422, 424, 426, 428 are output ports of a single multiplexed display, which can pipe image information to each of the image injection devices 420, 422, 424, 426, 428 via, for example, one or more light pipes (such as fiber optic cables).

[0546] A controller 460 controls the operation of the stacked waveguide assembly 480 and the image injection devices 420, 422, 424, 426, 428. The controller 460 includes programming (e.g., instructions in a non-transitory computer-readable medium) that regulates the timing and provision of image information to the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the controller 460 can be a single monolithic device or a distributed system connected via wired or wireless communication channels. In some embodiments, the controller 460 can be part of the processing module 260 or 270 ( Figure 2 shown in).

[0547] Waveguides 440b, 438b, 436b, 434b, 432b can be configured to propagate light within each respective waveguide by total internal reflection (TIR). Waveguides 440b, 438b, 436b, 434b, 432b can each be planar or have other shapes (e.g., curved), having top and bottom major surfaces and edges extending between these top and bottom major surfaces. In the illustrated configuration, waveguides 440b, 438b, 436b, 434b, 432b can each include light extraction optical elements 440a, 438a, 436a, 434a, 432a, which are configured to extract light out of the waveguide by redirecting the light propagating within each respective waveguide to output image information to the eye 410. The extracted light can also be referred to as coupled-out light, and the light extraction optical elements can also be referred to as coupled-out optical elements. The extracted light beam is output from the waveguide at a location where the light propagating in the waveguide irradiates the light redirecting element. The light extraction optical elements (440a, 438a, 436a, 434a, 432a) can be, for example, reflective or diffractive optical features. Although illustrated at the bottom major surface of waveguides 440b, 438b, 436b, 434b, 432b for ease of description and clarity of the drawing, in some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be provided at the top or bottom major surface, or can be provided directly within the volume of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be formed in a material layer attached to a transparent substrate to form waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, waveguides 440b, 438b, 436b, 434b, 432b can be a monolithic material, and the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be formed on the surface or within the interior of the piece of material.

[0548] Continue to refer to Figure 4, as discussed herein, each of the waveguides 440b, 438b, 436b, 434b, 432b is configured to output light to form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye can be configured to transmit collimated light, such as light injected into such waveguide 432b, to the eye 410. The collimated light can represent an optically infinite focal plane. The next upstream waveguide 434b can be configured to emit collimated light that has passed through the first lens 452 (e.g., a negative lens) before it can reach the eye 410. The first lens 452 can be configured to produce a slightly convex wavefront curvature such that the eye / brain interprets the light from this next upstream waveguide 434b as coming from a first focal plane that is closer to the eye 410 inward from optically infinite distance. Similarly, the third upstream waveguide 436b transmits the output light through the first lens 452 and the second lens 454 before it reaches the eye 410. The combined optical power of the first lens 452 and the second lens 454 can be configured to produce another increment of wavefront curvature such that the eye / brain interprets the light from the third waveguide 436b as coming from a second focal plane that is closer to the person inward from optically infinite distance than the light from the said next upstream waveguide 434b.

[0549] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are configured similarly, where the highest waveguide 440b in the stack sends its output through all the lenses between it and the eye for an aggregate focal power representative of the focal plane closest to the person. When viewing / interpreting light from the world 470 on the other side of the stacked waveguide assembly 480, to compensate for the stack of lenses 458, 456, 454, 452, a compensating lens layer 430 can be provided on top of the stack to compensate for the total focal power of the underlying lens stack 458, 456, 454, 452 (the compensating lens layer 430 and the stacked waveguide assembly 480 as a whole can be configured such that light from the world 470 is transmitted to the eye 410 at a divergence (or collimation) level that is substantially the same as when it was initially received by the stacked waveguide assembly 480). This configuration provides as many perceived focal planes as there are available waveguide / lens pairings. The light extraction optics of the waveguides and the focusing aspects of the lenses can be static (e.g., not dynamic or electrically actuated). In some alternative embodiments, one or both of them can be dynamic using electrically actuated features.

[0550] Continuing reference Figure 4, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be configured to redirect light out of their respective waveguides and output the light with an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes can have different light extraction optical element configurations that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be volume or surface features that can be configured to output light at a particular angle. For example, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be volume holograms, surface holograms, and / or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published on June 25, 2015, which is hereby incorporated by reference in its entirety.

[0551] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffraction features or “diffractive optical elements” (also referred to herein as “DOEs”) that form a diffraction pattern. Preferably, the DOE has a relatively low diffraction efficiency such that only a portion of the light beam is deflected towards the eye 410 through each intersection of the DOE, while the remainder continues to travel through the waveguide via total internal reflection. The light carrying the image information can thus be split into multiple related outgoing light beams that exit the waveguide at multiple locations, and the result is a relatively uniform pattern of outgoing emission towards the eye 304 for that particular collimated light beam bouncing within the waveguide.

[0552] In some embodiments, one or more DOEs can be switchable between an “on” state in which they actively diffract and an “off” state in which they do not significantly diffract. For example, a switchable DOE can include a polymer dispersed liquid crystal layer where microdroplets contain a diffraction pattern within a matrix medium, and the refractive index of the microdroplets can be switched to substantially match the refractive index of the matrix material (in which case the pattern does not significantly diffract the incident light), or the microdroplets can be switched to a refractive index that does not match the refractive index of the matrix medium (in which case the pattern actively diffracts the incident light).

[0553] In some embodiments, the number and distribution of depth planes or the depth of field may be dynamically changed based on the pupil size or orientation of the viewer's eyes. The depth of field may change inversely with the pupil size of the viewer. Thus, as the pupil size of the viewer's eyes decreases, the depth of field increases, such that a plane that was indistinguishable due to its position being outside the depth of focus of the eyes may become distinguishable and appears more focused with the decrease in pupil size, and commensurate with the increase in the depth of field. Similarly, the number of spaced-apart depth planes for presenting different images to the viewer may decrease as the pupil size decreases. For example, a viewer may not be able to clearly perceive the details of both a first depth plane and a second depth plane at one pupil size without adjusting the accommodation of the eyes from one depth plane to the other. However, these two depth planes may be simultaneously in sufficient focus for a user at another pupil size without changing the accommodation.

[0554] In some embodiments, the display system may change the number of waveguides receiving image information based on the determination of the pupil size or orientation, or upon receiving an electrical signal indicating a particular pupil size or orientation. For example, if the user's eyes cannot distinguish between two depth planes associated with two waveguides, the controller 460 (which may be an embodiment of the local processing and data module 206) may be configured or programmed to stop providing image information to one of these waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments where the DOE for a waveguide can be switched between an on and an off state, the DOE may be switched to the off state when the waveguide does receive image information.

[0555] In some embodiments, it may be desirable to have the outgoing light beam satisfy the condition that its diameter is smaller than the diameter of the viewer's eyes. However, given the variability of the viewer's pupil size, satisfying such a condition may be challenging. In some embodiments, by changing the size of the outgoing light beam in response to the determination of the viewer's pupil size, this condition is satisfied over a wide range of pupil sizes. For example, as the pupil size decreases, the size of the outgoing light beam may also decrease. In some embodiments, a variable aperture may be used to change the size of the outgoing light beam.

[0556] The wearable system 400 may include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the field of view (FOV) of the world camera, and the imaging system 464 is sometimes referred to as the FOV camera. The FOV of the world camera may be the same as or different from the FOV of the viewer 210, where the FOV of the viewer 210 includes the portion of the world 470 that the viewer 210 perceives at a given moment. For example, in some cases, the FOV of the world camera may be larger than the viewer 210 of the wearable system 400. The entire area available for viewing or imaging by the viewer may be referred to as the field of regard (FOR). Since the wearer can move their body, head, or eyes to perceive substantially any direction in space, the FOR may include 4π steradians of solid angle around the wearable system 400. In other cases, the movement of the wearer may be more restricted, and correspondingly, the wearer's FOR may subtend a smaller solid angle. Images obtained from the outward-facing imaging system 464 can be used to track gestures made by the user (e.g., the pose of a hand or finger), detect objects in the world 470 in front of the user, and so on.

[0557] The wearable system 400 may include an audio sensor 232, such as a microphone, to capture ambient sound. As described above, in some embodiments, one or more other audio sensors may be placed to provide stereo reception useful for determining the location of a voice source. The audio sensor 232 may include a directional microphone. As another example, the directional microphone may also provide this useful directional information about the location of the audio source. The wearable system 400 may use information from both the outward-facing imaging system 464 and the audio sensor 230 to locate a voice source or determine the speaker, etc., at a particular moment. For example, the wearable system 400 may use speech recognition alone or in combination with a reflected image of the speaker (e.g., a reflected image seen in a mirror) to determine the identity of the speaker. As another example, the wearable system 400 may determine the location of the speaker in the environment based on the sound obtained from the directional microphone. The wearable system 400 may parse the sound from the speaker's location through a speech recognition algorithm to determine the speech content and use speech recognition techniques to determine the identity of the speaker (e.g., name or other demographic information).

[0558] The wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera) that observes the user's movements, such as eye movements and facial movements. The inward-facing imaging system 466 can be used to capture an image of the eye 410 to determine the size and / or orientation of the pupil of the eye 304. The inward-facing imaging system 466 can be used to obtain images for determining the direction in which the user is looking (e.g., eye pose) or for biometric identification of the user (e.g., via iris recognition). In some embodiments, at least one camera can be used for each eye to independently determine the pupil size or eye pose of each eye separately, thereby allowing the presentation of image information to each eye to be dynamically adapted to that eye. In some other embodiments, only the pupil diameter or orientation of one eye 410 (e.g., using only one camera for each pair of eyes) is determined and assumed to be similar for the user's two eyes. The images obtained through the inward-facing imaging system 466 can be analyzed to determine the user's eye pose or mood, and the wearable system 400 can use the user's eye pose or mood to decide which audio or visual content should be presented to the user. The wearable system 400 can also use sensors such as an IMU, accelerometer, gyroscope, etc. to determine the head pose (e.g., head position or head orientation).

[0559] The wearable system 400 can include a user input device 466 through which the user can input commands to the controller 460 to interact with the wearable system 400. For example, the user input device 466 can include a touchpad, a touch screen, a joystick, a multi-degree-of-freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a D-pad, a wand, a haptic device, a totem (e.g., used as a virtual user input device), and so on. The multi-DOF controller can sense user input in some or all of the possible translational (e.g., left / right, forward / backward, or up / down) or rotational (e.g., yaw, pitch, or roll) aspects of the controller. A multi-DOF controller that supports translational motion can be referred to as 3DOF, while a multi-DOF controller that supports both translational and rotational motion can be referred to as 6DOF. In some cases, the user can use a finger (e.g., the thumb) to press or swipe on the touch-sensitive input device to provide input to the wearable system 400 (e.g., to provide user input to the user interface provided by the wearable system 400). The user input device 466 can be held by the user's hand during use of the wearable system 400. The user input device 466 can communicate with the wearable system 400 in a wired or wireless manner.

[0560] Other components of a wearable system

[0561] In many embodiments, the wearable system may include other components as a supplement or alternative to the components of the wearable system described above. The wearable system may include, for example, one or more haptic devices or components. The haptic device or component may be used to provide haptic sensations to the user. For example, the haptic device or component may provide a haptic sensation of pressure or texture when touching virtual content (e.g., virtual objects, virtual tools, other virtual constructs). The haptic sensation may replicate the feel of a physical object represented by the virtual object, or may replicate the feel of an imaginary object or character (e.g., a dragon) represented by the virtual content. In some embodiments, the haptic device or component may be worn by the user (e.g., a glove wearable by the user). In some embodiments, the haptic device or component may be held by the user.

[0562] The wearable system may include, for example, one or more physical objects manipulable by the user to allow input or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects such as, for example, blocks of metal or plastic, the surface of a wall, a table. In certain embodiments, the totem may not actually have any physical input structure (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, the totem may simply provide a physical surface, and the wearable system may present a user interface such that it appears to the user to be on one or more surfaces of the totem. For example, the wearable system may cause an image of a computer keyboard and touchpad to appear to reside on one or more surfaces of the totem. For example, the wearable system may cause a virtual computer keyboard and virtual touchpad to appear on the surface of a thin rectangular plate of aluminum that is a totem. The rectangular plate itself does not have any physical keys or touchpad or sensors. However, the wearable system may detect the user manipulating or interacting with or touching the rectangular plate as a selection or input via the virtual keyboard or virtual touchpad. The user input device 466 (shown in Figure 4 may be an example of a totem and may include a touchpad, a touch panel, a trigger, a joystick, a trackball, a rocker, or a virtual switch, a mouse, a keyboard, a multi-degree-of-freedom controller, or another physical input device. The user may use the totem alone or in combination with a gesture to interact with the wearable system or with other users.

[0563] Examples of haptic devices and totems that may be used in the wearable devices, HMDs, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, the entire contents of which are incorporated herein by reference.

[0564] Example of an eye image

[0565] Figure 5An image of an eye 500 is shown, having an eyelid 504, a sclera 508 (the "white part" of the eye), an iris 512, and a pupil 516. Curve 516a shows the pupil boundary between the pupil 516 and the iris 512, and curve 512a shows the limbus boundary between the iris 512 and the sclera 508. The eyelid 504 includes an upper eyelid 504a and a lower eyelid 504b. The shown eye 500 is in a natural rest pose (e.g., where the user's face and gaze are both directed towards a distant object straight ahead of the user). The natural rest pose of the eye 500 can be indicated by a natural rest direction 520, which is the direction orthogonal to the surface of the eye 500 when the eye 500 is in the natural rest pose (e.g., directly outwards from Figure 5 the plane of the shown eye 500), which in this example is located at the center of the pupil 516.

[0566] When the eye 500 moves to look at different objects, the eye pose will change relative to the natural rest direction 520. The current eye pose can be determined with reference to an eye pose direction 524, which is the direction orthogonal to the surface of the eye (and located at the center within the pupil 516), but towards the object at which the eye is currently looking. Referring to Figure 5A the example coordinate system shown, the pose of the eye 500 can be represented as two angular parameters indicating the azimuthal deflection and the zenithal deflection of the eye pose direction 524 of the eye, both deflections relative to the natural rest direction 520 of the eye. For illustrative purposes, these angular parameters can be represented as θ (azimuthal deflection, determined according to a reference azimuth) and φ (zenithal deflection, sometimes also referred to as polar angle deflection). In some embodiments, the angular roll of the eye around the eye pose direction 524 can be included in the determination of the eye pose, and moreover, the angular roll can be included in the following analysis. In other embodiments, other techniques for determining the eye pose can be used, such as pitch, yaw, and optionally roll systems.

[0567] Any suitable process can be used to obtain an eye image from a video, such as using a video processing algorithm that can extract an image from one or more consecutive frames. Various eye tracking techniques can be used to determine the eye pose from the eye image. For example, the eye pose can be determined by considering the lensing effect of the cornea on the provided light source. In the eyelid shape estimation techniques described herein, any suitable eye tracking technique can be used to determine the eye pose.

[0568] Example of an eye tracking system

[0569] Figure 6FIG. 0 shows a schematic diagram of a wearable system 600 including an eye tracking system. In at least some embodiments, the wearable system 600 may include components located in a head-mounted unit 602 and components located in a non-head-mounted unit 604. The non-head-mounted unit 604 may be, for example, a strap-mounted component, a hand-held component, a component in a backpack, a remote component, etc. Including certain components of the wearable system 600 in the non-head-mounted unit 604 helps to reduce the size, weight, complexity, and cost of the head-mounted unit 602. In some implementations, some or all of the functions described as being performed by one or more components of the head-mounted unit 602 and / or the non-head-mounted unit 604 may be provided by one or more components included elsewhere in the wearable system 600. For example, some or all of the functions associated with the CPU 612 of the head-mounted unit 602 below may be provided by the CPU 616 of the non-head-mounted unit 604, and vice versa. In some examples, some or all of such functions may be provided by a peripheral device of the wearable system 600. Additionally, in some implementations, some or all of such functions may be provided by one or more cloud computing devices or other remotely located computing devices in a manner similar to that described above with reference to Figure 2 described.

[0570] As Figure 6 shown, the wearable system 600 may include an eye tracking system that includes a camera 324 that captures an image of the user's eye 610. If desired, the eye tracking system may also include light sources 326a and 326b (such as light-emitting diodes "LEDs"). The light sources 326a and 326b may produce bright spots (e.g., reflections from the user's eye that appear in the eye image captured by the camera 324). The positions of the light sources 326a and 326b relative to the camera 324 may be known, and thus, the positions of the bright spots within the image captured by the camera 324 can be used to track the user's eye (as will be discussed in more detail below in connection with FIGS. 7-11). In at least one embodiment, there may be one light source 326 and one camera 324 associated with a single eye of the user's eyes 610. In another embodiment, there may be one light source 326 and one camera 324 associated with each of the user's eyes 610. In additional embodiments, there may be one or more cameras 324 and one or more light sources 326 associated with one or each of the user's eyes 610. As a specific example, there may be two light sources 326a and 326b and one or more cameras 324 associated with each of the user's eyes 610. As another example, there may be three or more light sources (such as light sources 326a and 326b) and one or more cameras 324 associated with each of the user's eyes 610.

[0571] The eye tracking module 614 can receive images from the eye tracking camera 324 and can analyze the images to extract various pieces of information. As an example, the eye tracking module 614 can detect the user's eye pose, the three-dimensional position of the user's eyes relative to the eye tracking camera 324 (and relative to the head-mounted unit 602), the focusing direction of one or both of the user's eyes 610, the user's vergence depth (e.g., the depth relative to the user at which the user is focusing), the position of the user's pupils, the position of the user's corneas and corneal globes, the center of rotation of each of the user's eyes, and the perspective center of each of the user's eyes. The eye tracking module 614 can use the techniques described below in connection with FIGS. 7 through 11 to extract such information. As Figure 6 shown, the eye tracking module 614 can be a software module implemented using the CPU 612 in the head-mounted unit 602.

[0572] Data from the eye tracking module 614 can be provided to other components in the wearable system. For example, such data can be transmitted to components in the non-head-mounted unit 604, such as the CPU 616, which includes software modules for the light field rendering controller 618 and the registration viewer 620.

[0573] The rendering controller 618 can use the information from the eye tracking module 614 to adjust the images displayed to the user by the rendering engine 622 (e.g., a software module in the GPU 620 that can provide images to the display 220). As an example, the rendering controller 618 can adjust the images displayed to the user based on the center of rotation or the perspective center of the user. Specifically, the rendering controller 618 can use the information about the user's perspective center to simulate a rendering camera (e.g., simulate collecting images from the user's perspective), and can adjust the images displayed to the user based on the simulated rendering camera.

[0574] A "rendering camera", sometimes also referred to as a "pinhole perspective camera" (or simply "perspective camera") or a "virtual pinhole camera" (or simply "virtual camera"), is an analog camera used to render virtual image content that may be from a database of objects in a virtual world. The objects can have positions and orientations relative to the user or wearer and possibly relative to real objects in the user's or wearer's surrounding environment. In other words, the rendering camera can represent a perspective within a rendering space from which the user or wearer views 3D virtual content (e.g., virtual objects) in the rendering space. The rendering camera can be managed by a rendering engine to render a virtual image based on a database of virtual objects to be presented to the eyes. The virtual image can be rendered as if taken from the perspective of the user or wearer. For example, the virtual image can be rendered as if captured by a pinhole camera (corresponding to the "rendering camera") having a set of specific intrinsic parameters (e.g., focal length, camera pixel size, principal point coordinates, skew / distortion parameters, etc.) and a set of specific extrinsic parameters (e.g., translational and rotational components relative to the virtual world). The virtual image is taken from the perspective of such a camera having the position and orientation of the rendering camera (e.g., the extrinsic parameters of the rendering camera). Thus, the system can define and / or adjust the intrinsic and extrinsic rendering camera parameters. For example, the system can define a set of specific extrinsic rendering camera parameters such that the virtual image is rendered as if captured from the perspective of a camera having a specific position relative to the user's or wearer's eyes, thereby providing an image that appears to be from the perspective of the user or wearer. The system can then dynamically adjust the extrinsic rendering camera parameters on-the-fly in order to maintain registration with the specific position. Similarly, the intrinsic rendering camera parameters can be defined and dynamically adjusted over time. In some embodiments, the image is rendered as if captured from the perspective of a camera having an aperture (e.g., a pinhole) located at a specific position (e.g., the perspective center or the center of rotation or other position) relative to the user's or wearer's eyes.

[0575] In some embodiments, the system can create or dynamically reposition and / or reorient one rendering camera for the user's left eye and another rendering camera for the user's right eye, as the user's eyes are physically separated from each other and thus are always located at different positions. Accordingly, in at least some implementations, virtual content rendered from the perspective of the rendering camera associated with the viewer's left eye can be presented to the user through the left eyepiece of a head-mounted display (e.g., head-mounted unit 602), and virtual content rendered from the perspective of the rendering camera associated with the viewer's right eye can be presented to the user through the right eyepiece of such a head-mounted display. More details regarding creating, adjusting, and using rendering cameras during the rendering process are provided in U.S. Patent Application No. 15 / 274,823, entitled "METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTURAL FEATURES IN 3D RECONSTRUCTION", the entire content of which is hereby expressly incorporated by reference for all purposes.

[0576] In some examples, one or more modules (or components) of system 600 (e.g., light field rendering controller 618, rendering engine 620, etc.) can determine the position and orientation of the rendering camera within the rendering space based on the position and orientation of the user's head and eyes (e.g., determined based on head pose and eye tracking data, respectively). That is, system 600 can effectively map the position and orientation of the user's head and eyes to a particular position and angular orientation within the 3D virtual environment, place and orient the rendering camera at the particular position and angular orientation within the 3D virtual environment, and render virtual content for the user as if it were captured by the rendering camera. More details regarding the real-world to virtual-world mapping process are provided in U.S. Patent Application No. 15 / 296,869, entitled "SELECTING VIRTUAL OBJECTS IN A THREE-DIMENSIONAL SPACE", the entire content of which is hereby expressly incorporated by reference for all purposes. As an example, the rendering controller 618 can adjust the depth at which an image is displayed by selecting which depth plane (or which depth planes) to utilize at any given time. In some implementations, such depth plane switching can be performed by adjusting one or more intrinsic rendering camera parameters. For example, when performing depth plane switching or adjustment, the light field rendering controller 618 can adjust the focal length of the rendering camera. As described in further detail below, the depth plane can be switched based on the user's determined vergence or fixation depth.

[0577] The registration viewer 620 can use information from the eye tracking module 614 to identify whether the head-mounted unit 602 is correctly positioned on the user's head. As an example, the eye tracking module 614 can provide eye position information, such as the position of the center of rotation of the user's eyes, which indicates the three-dimensional position of the user's eyes relative to the camera 324 and the head-mounted unit 602. The eye tracking module 614 can use this position information to determine whether the display 220 is correctly aligned in the user's field of view, or whether the head-mounted unit 602 (or the head-mounted device) has slipped or otherwise become misaligned with the user's eyes. As an example, the registration viewer 620 can determine whether the head-mounted unit 602 has slipped down from the user's nose bridge, thereby moving the display 220 away from and downward from the user's eyes (which is undesirable); whether the head-mounted unit 602 has moved upward along the user's nose bridge, thereby moving the display 220 upward and closer to the user's eyes; whether the head-mounted unit 602 has moved left or right relative to the user's nose bridge; whether the head-mounted unit 602 has been raised above the user's nose bridge; or whether the head-mounted unit 602 has been moved away from the desired position or range of positions in these or other ways. Generally, the registration viewer 620 can determine whether the general head-mounted unit 602 and specifically the display 220 are correctly positioned in front of the user's eyes. In other words, the registration viewer 620 can determine whether the left display in the display system 220 is properly aligned with the user's left eye and whether the right display in the display system 220 is properly aligned with the user's right eye. The registration viewer 620 can determine whether the head-mounted unit 602 is properly positioned by determining whether the head-mounted unit 602 is positioned and oriented within the desired position and / or orientation range relative to the user's eyes.

[0578] In at least some embodiments, the registration viewer 620 can generate user feedback in the form of an alert, a message, or other content. Such feedback can be provided to the user to notify the user of any misalignment of the head-mounted unit 602 and optional feedback on how to correct the misalignment (such as a suggestion to adjust the head-mounted unit 602 in a particular manner).

[0579] Example registration viewing and feedback techniques that the registration viewer 620 can use are described in U.S. Patent Application No. 15 / 717,747, filed on September 27, 2017 (Attorney Docket No. MLEAP.052A2) and U.S. Provisional Patent Application No. 62 / 644,321, filed on March 16, 2018 (Attorney Docket No. MLEAP.195PR), the entire contents of both of which are incorporated herein by reference.

[0580] Example of an eye tracking module

[0581] A detailed block diagram of an example eye tracking module 614 is shown in Figure 7A . As Figure 7A shown, the eye tracking module 614 can include various different sub-modules, can provide various different outputs, and can utilize various available data to track a user's eye. By way of example, the eye tracking module 614 can utilize available data including the following: extrinsic and intrinsic features of eye tracking, such as the geometric arrangement of an eye tracking camera 324 relative to a light source 326 and a head-mounted unit 602; a hypothesized eye size 704, such as a typical distance of about 4.7 mm between the corneal curvature center of a user and the average center of rotation of the user's eye, or a typical distance between the center of rotation and the perspective center of the user's eye; and per-user calibration data 706, such as the interpupillary distance of a particular user. Additional examples of extrinsic features, intrinsic features, and other information that can be employed by the eye tracking module 614 are described in U.S. Patent Application No. 15 / 497,726, filed Apr. 26, 2017 (Attorney Docket No. MLEAP.023A7), the entire content of which is incorporated herein by reference.

[0582] An image preprocessing module 710 can receive images from an eye camera (such as eye camera 324) and can perform one or more preprocessing (e.g., conditioning) operations on the received images. By way of example, the image preprocessing module 710 can apply a Gaussian blur to the images, can downsample the images to a lower resolution, can apply an unsharp mask, can apply an edge sharpening algorithm, or can apply other suitable filters that assist in later detection, localization, and marking of bright spots, pupils, or other features in images from the eye camera 324. The image preprocessing module 710 can apply a low-pass filter or a morphological filter (such as an opening filter), which can remove high-frequency noise such as from a pupil boundary 516a (see Figure 5 ), thereby removing noise that may impede pupil and bright spot determination. The image preprocessing module 710 can output the preprocessed images to a pupil recognition module 712 and a bright spot detection and marking module 714.

[0583] The pupil recognition module 712 can receive the preprocessed images from the image preprocessing module 710 and can recognize the regions in these images that include the user's pupils. In some embodiments, the pupil recognition module 712 can determine the coordinates of the position of the user's pupil or the coordinates of its center or centroid in the eye tracking images from the camera 324. In at least some embodiments, the pupil recognition module 712 can recognize the contours in the eye tracking images (e.g., the contours of the pupil-iris boundary), recognize the contour moments (e.g., the centroid), apply the starburst pupil detection and / or Canny edge detection algorithms, discard the outliers based on the intensity values, recognize the sub-pixel boundary points, correct the eye camera distortion (e.g., the distortion in the images captured by the eye camera 324), apply the random sample consensus (RANSAC) iterative algorithm to fit an ellipse to the boundary in the eye tracking image, apply a tracking filter to the image, and recognize the sub-pixel image coordinates of the centroid of the user's pupil. The pupil recognition module 712 can output pupil recognition data to the bright spot detection and marking module 714, and the pupil recognition data can indicate which regions of the preprocessing image module 712 are recognized as showing the user's pupils. The pupil recognition module 712 can provide the 2D coordinates of the user's pupil in each eye tracking image (e.g., the 2D coordinates of the centroid of the user's pupil) to the bright spot detection module 714. In at least some embodiments, the pupil recognition module 712 can also provide the same type of pupil recognition data to the coordinate system normalization module 718.

[0584] The pupil detection techniques that the pupil recognition module 712 can utilize are described in U.S. Patent Publication No. 2017 / 0053165, published on February 23, 2017, and U.S. Patent Publication No. 2017 / 0053166, published on February 23, 2017, and the entire contents of each of these two patent publications are incorporated herein by reference.

[0585] The bright spot detection and marking module 714 can receive the preprocessed image from module 710, and the pupil recognition data from module 712. The bright spot detection module 714 can use this data to detect and / or identify bright spots (e.g., the reflection of light from light source 326 from the user's eye) within the region of the preprocessed image that shows the user's pupil. As an example, the bright spot detection module 714 can search for bright regions within the eye tracking image, sometimes referred to herein as "spots" or local intensity maxima, which are located near the user's pupil. In at least some embodiments, the bright spot detection module 714 can rescale (e.g., magnify) the pupil ellipse to include additional bright spots. The bright spot detection module 714 can filter bright spots by size and / or intensity. The bright spot detection module 714 can also determine the 2D position of each bright spot within the eye tracking image. In at least some examples, the bright spot detection module 714 can determine the 2D position of the bright spot relative to the user's pupil, which can also be referred to as the pupil-bright spot vector. The bright spot detection and marking module 714 can mark the bright spots and output the preprocessed image with the marked bright spots to the 3D corneal center estimation module 716. The bright spot detection and marking module 714 can also transfer data, such as the preprocessed image from module 710 and the pupil recognition data from module 712. In some implementations, the bright spot detection and marking module 714 can determine which light source (e.g., determine from multiple light sources of a system including infrared light sources 326a and 326b) produces each identified bright spot. In these examples, the bright spot detection and marking module 714 can mark the bright spots with information identifying the associated light source and output the preprocessed image with the marked bright spots to the 3D corneal center estimation module 716.

[0586] Pupil and bright spot detection performed by modules such as modules 712 and 714 can use any suitable technique. As an example, edge detection can be applied to the eye image to identify bright spots and pupils. Edge detection can be applied through various edge detectors, edge detection algorithms, or filters. For example, the Canny edge detector can be applied to the image to detect edges such as in image lines. Edges can include points corresponding to local maximum derivatives located along the line. For example, the Canny edge detector can be used to locate the pupil boundary 516a (see Figure 5)。Once the pupil position is determined, various image processing techniques can be used to detect the "pose" of the pupil 116. Determining the eye pose of an eye image can also be referred to as detecting the eye pose of the eye image. The pose can also be referred to as the gaze, pointing direction, or orientation of the eye. For example, the pupil may be looking at an object to the left, and the pose of the pupil can be classified as a left pose. Other methods can be used to detect the position of the pupil or bright spot. For example, a Canny edge detector can be used to locate the concentric rings in the eye image. As another example, an integral-differential operator can be used to find the pupil or limbus boundary of the iris. For example, a Daugman integral-differential operator, Hough transform, or other iris segmentation techniques can be used to return a curve that estimates the boundary of the pupil or iris.

[0587] The 3D corneal center estimation module 716 can receive the preprocessed images including the detected bright spot data and pupil recognition data from the modules 710, 712, 714. The 3D corneal center estimation module 716 can use this data to estimate the 3D position of the user's cornea. In some embodiments, the 3D corneal center estimation module 716 can estimate the corneal curvature center of the eye or the 3D position of the user's corneal sphere, e.g., the center of a hypothetical sphere, the surface portion of which substantially coextends with the user's cornea. The 3D corneal center estimation module 716 can provide data indicating the estimated 3D coordinates of the corneal sphere and / or the user's cornea to the coordinate system normalization module 718, the optical axis determination module 722, and / or the light field rendering controller 618. More operational details of the 3D corneal center estimation module 716 are provided herein in connection with Figures 8A to 8E the present disclosure. Techniques for estimating the position of eye features (e.g., the cornea or corneal sphere) that can be utilized by the 3D corneal center estimation module 716 and other modules in the wearable system of the present disclosure are discussed in U.S. Patent Application No. 15 / 497,726 (Attorney Docket No. MLEAP.023A7) filed on April 26, 2017, the entire content of which is incorporated herein by reference.

[0588] The coordinate system normalization module 718 can optionally (as shown by its dashed box) be included in the eye tracking module 614. The coordinate system normalization module 718 can receive data indicating the estimated 3D coordinates of the user's corneal center (and / or the center of the user's corneal sphere) from the 3D corneal center estimation module 716, and can also receive data from other modules. The coordinate system normalization module 718 can normalize the eye camera coordinate system, which helps to compensate for slippage of the wearable device (e.g., the head-mounted component slips from its normal resting position on the user's head, which can be identified by the registration viewer 620). The coordinate system normalization module 718 can rotate the coordinate system to align the z-axis of the coordinate system (e.g., the vergence depth axis) with the corneal center (e.g., indicated by the 3D corneal center estimation module 716), and can translate the camera center (e.g., the origin of the coordinate system) to a predetermined distance away from the corneal center, such as 30 mm (e.g., the module 718 can zoom in or out on the eye tracking image, depending on whether the eye camera 324 is determined to be closer or farther than the predetermined distance). Through this normalization process, the eye tracking module 614 can establish a consistent orientation and distance in the eye tracking data relatively independently of changes in the positioning of the head-mounted device on the user's head. The coordinate system normalization module 718 can provide the 3D coordinates of the corneal (and / or corneal sphere) center, pupil identification data, and pre-processed eye tracking images to the 3D pupil center locator module 720. More operational details of the coordinate system normalization module 718 are provided herein in connection with Figures 9A to 9C provided.

[0589] The 3D pupil center locator module 720 can receive data in a normalized or un-normalized coordinate system, the data including the 3D coordinates of the user's corneal (and / or corneal sphere) center, pupil position data, and pre-processed eye tracking images. The 3D pupil center locator module 720 can analyze such data to determine the 3D coordinates of the user's pupil center in the normalized or un-normalized eye camera coordinate system. The 3D pupil center locator module 720 can determine the position of the user's pupil in three dimensions based on the 2D position of the pupil centroid (as determined by module 712), the 3D position of the corneal center (as determined by module 716), the assumed eye size 704 (e.g., the size of a typical user's corneal sphere and the typical distance from the corneal center to the pupil center), and the optical properties of the eye (e.g., the corneal refractive index (relative to the air refractive index)) or any combination of these. More operational details of the 3D pupil center locator module 720 are provided herein in connection with Figures 9D to 9GProvided. The techniques for estimating the position of an eye feature (e.g., the pupil) that can be utilized by the 3D pupil center locator module 720 and other modules in the wearable system of the present disclosure are discussed in U.S. Patent Application No. 15 / 497,726 (Attorney Docket No. MLEAP.023A7), filed on Apr. 26, 2017, the entire content of which is incorporated herein by reference.

[0590] The optical axis determination module 722 can receive data indicating the 3D coordinates of the user's corneal center and the user's pupil from modules 716 and 720. Based on such data, the optical axis determination module 722 can identify a vector from the position of the corneal center (e.g., from the corneal spherical center) to the pupil center of the user, which vector can define the optical axis of the user's eye. As an example, the optical axis determination module 722 can provide an output specifying the optical axis of the user to modules 724, 728, 730, and 732.

[0591] The center of rotation (CoR) estimation module 724 can receive data from module 722 that includes parameters of the optical axis of the user's eye (e.g., data indicating the direction of the optical axis in a coordinate system, which direction has a known relationship with the head-mounted unit 602). For example, the CoR estimation module 724 can estimate the center of rotation of the user's eye. The center of rotation can indicate the point about which the user's eye rotates when the user's eye rotates left, right, up, and / or down. Although the eye may not rotate perfectly about a singular point, it is sufficient to assume a singular point. In at least some embodiments, the CoR estimation module 724 can estimate the center of rotation of the eye by moving a specific distance along the optical axis (identified by module 722) from the pupil center (identified by module 720) or the corneal curvature center (identified by module 716) towards the retina. The specific distance can be the assumed eye size 704. As an example, the specific distance between the corneal curvature center and the CoR can be about 4.7 mm. This distance can be changed for a particular user based on any relevant data, including the user's age, gender, vision prescription, other relevant characteristics, etc. Other discussions of the estimated value of 4.7 mm as the distance between the corneal curvature center and the CoR are provided in Appendix (Part III) that forms a part of this application.

[0592] In at least some embodiments, the CoR estimation module 724 may refine its estimate of the center of rotation of each of the user's eyes over time. As an example, over time, the user eventually rotates their eyes (looks at objects closer, farther away, or looks left, right, up, or down at some point), causing the optical axes of each of their eyes to shift. The CoR estimation module 724 then analyzes the two (or more) optical axes identified by module 722 and locates the 3D intersection of these optical axes. The CoR estimation module 724 can then determine that the center of rotation is located at this 3D intersection. This technique can provide an estimate of the center of rotation whose accuracy improves over time.

[0593] Various techniques can be employed to increase the accuracy of the CoR estimation module 724 and the determined CoR positions of the left and right eyes. As an example, the CoR estimation module 724 can estimate the CoR by finding the average intersection of the optical axes determined for various different eye poses over time. As an additional example, module 724 can filter or average the CoR positions estimated over time, can calculate a moving average of the CoR positions estimated over time, and / or can apply a Kalman filter along with the known dynamics of the eye and the eye tracking system to estimate the CoR position over time. In some implementations, the least squares method can be employed to determine one or more intersections of the optical axes. In such an implementation, the system can identify, at a given point in time, the position at which the sum of the squared distances to a given set of optical axes is reduced or minimized as the intersection of the optical axes. As a specific example, module 724 can calculate a weighted average of the determined optical axis intersection and a hypothesized CoR position (e.g., 4.7 mm from the corneal curvature center of the eye) such that the determined CoR can slowly drift over time from the hypothesized CoR position (e.g., 4.7 mm behind the corneal curvature center of the eye) to a slightly different position within the user's eye as eye tracking data for that user is obtained, enabling per-user refinement of the CoR position.

[0594] Under ideal conditions, when a user moves their eyes (e.g., when the user's eyes rotate about their center of rotation), the true CoR of the user's eyes relative to the 3D position of the HMD should change by a negligible or minimal amount over time. In other words, for a given set of eye movements, the 3D position of the true CoR of the user's eyes (relative to the HMD) is assumed to change less over time than any other point along the optical axis of the user's eyes. Thus, it can be concluded that when a user moves their eyes, the farther a point along the optical axis is from the true CoR of the user's eyes, the greater the change or variance in its 3D position will be over time. In some embodiments, the CoR estimation module 724 and / or other sub-modules of the eye tracking module 614 can utilize this statistical relationship to improve CoR estimation accuracy. In such embodiments, the CoR estimation module 724 and / or other sub-modules of the eye tracking module 614 can refine their estimation of the CoR 3D position over time by identifying changes in the CoR estimation that have low variation (e.g., low variance or standard deviation).

[0595] As a first example and in embodiments where the CoR estimation module 724 estimates the CoR based on the intersection of multiple different optical axes (each optical axis associated with the user looking in a different direction), the CoR estimation module 724 can utilize this statistical relationship (i.e., the true CoR should have low variance) by: introducing a common offset in the direction of each optical axis (e.g., offsetting each axis by some same amount), and determining whether the offset optical axes intersect at an intersection with low variation (e.g., low variance or standard deviation). This can correct for small systematic errors in calculating the direction of the optical axes and help refine the estimated position of the CoR to be closer to the true CoR.

[0596] As a second example and in embodiments where the CoR estimation module 724 estimates the CoR by moving a particular distance (e.g., the distance between the corneal curvature center and the CoR) along the optical axis (or other axis), the system can vary, optimize, modulate, or otherwise adjust the particular distance between the corneal curvature center and the CoR over time (e.g., for a large set of eye images captured at different times) in a manner that reduces or minimizes the variation (e.g., variance and / or standard deviation) in the estimated CoR position. For example, if the CoR estimation module 724 initially uses a particular distance value of 4.7 mm (along the optical axis relative to the corneal curvature center) to obtain a CoR position estimate, but the true CoR of a given user's eye may be located 4.9 mm behind (along the optical axis) the corneal curvature center of the eye, then an initial set of CoR position estimates obtained by the CoR estimation module 724 may exhibit a relatively high amount of variation (e.g., variance or standard deviation). In response to detecting such a relatively high amount of variation (e.g., variance or standard deviation), the CoR estimation module 724 can search for and identify one or more points along the optical axis that have a lower amount of variation (e.g., variance or standard deviation), and may identify the 4.9 mm distance as having the lowest amount of variation (e.g., variance or standard deviation), and thus can adjust the particular distance value utilized to 4.9 mm.

[0597] The CoR estimation module 724 can search for alternative CoR estimates having a lower amount of variation (e.g., variance and / or standard deviation) in response to detecting that the current CoR estimate has a relatively high amount of variation (e.g., variance or standard deviation), or can routinely search for alternative CoR estimates having a lower amount of variation (e.g., variance or standard deviation) after obtaining an initial CoR estimate. In some examples, such optimization / adjustment can occur gradually over time, while in other examples, such optimization / adjustment can occur during an initial user calibration session. In examples where such a process occurs during the calibration process, the CoR estimation module 724 may not initially assume / observe any hypothesized particular distance, but rather collect a set of eye tracking data over time, perform a statistical analysis on the set of eye tracking data, and determine a particular distance value based on the statistical analysis that yields a CoR position estimate having the minimum (e.g., globally minimum) possible amount of variation (e.g., variance or standard deviation).

[0598] Appendix (Part III) forming a part of this application provides additional discussion of the above statistical relationships (e.g., the true CoR should have a low variance or standard deviation), as well as the importance of considering corneal refraction when determining pupil position.

[0599] The interpupillary distance (IPD) estimation module 726 may receive data indicating the estimated 3D positions of the centers of rotation of the user's left and right eyes from the CoR estimation module 724. The IPD estimation module 726 may then estimate the user's IPD by measuring the 3D distance between the centers of rotation of the user's left and right eyes. Generally, when the user is looking at optical infinity (e.g., the optical axes of the user's eyes are substantially parallel to each other), the distance between the estimated CoR of the user's left eye and the estimated CoR of the user's right eye may be approximately equal to the distance between the user's pupil centers, which is the typical definition of the interpupillary distance (IPD). The user's IPD may be used by various components and modules in the wearable system. For example, the user's IPD may be provided to the registration viewer 620 and used to evaluate the alignment of the wearable device with the user's eyes (e.g., whether the left and right display lenses are properly spaced according to the user's IPD). As another example, the user's IPD may be provided to the vergence depth estimation module 728 and used to determine the user's vergence depth. The module 726 may employ various techniques, such as those discussed in conjunction with the CoR estimation module 724, to improve the accuracy of the estimated IPD. As an example, the IPD estimation module 724 may apply filtering, averaging over time, weighted averaging including hypothesized IPD distances, Kalman filters, etc., as part of estimating the user's IPD in an accurate manner.

[0600] The vergence depth estimation module 728 may receive from the various modules and sub-modules in the eye tracking module 614, such as in conjunction with Figure 7Areceives data as shown. Specifically, the vergence depth estimation module 728 may employ data indicative of the following: the estimated 3D position of the pupil center (e.g., provided by the aforementioned module 720), one or more determined parameters of the optical axis (e.g., provided by the aforementioned module 722), the estimated 3D position of the center of rotation (e.g., provided by the aforementioned module 724), the estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the centers of rotation) (e.g., provided by the aforementioned module 726), and / or one or more determined parameters of the optical axis and / or visual axis (e.g., provided by module 722 and / or the module 730 described below). The vergence depth estimation module 728 may detect or otherwise obtain a vergence depth metric of the user, which may be the distance relative to the user at which the user's eyes are focused. For example, when the user is looking at an object three feet in front of him / her, the vergence depth of the user's left and right eyes is three feet; and when the user is looking at a distant landscape (e.g., the optical axes of the user's eyes are substantially parallel to each other such that the distance between the user's pupil centers is approximately equal to the distance between the centers of rotation of the user's left and right eyes), the vergence depth of the user's left and right eyes is at infinity. In some embodiments, the vergence depth estimation module 728 may utilize data indicative of the estimated center of the user's pupil (e.g., provided by module 720) to determine the 3D distance between the estimated centers of the user's pupils. The vergence depth estimation module 728 may obtain a metric of the vergence depth by comparing such determined 3D distance between the pupil centers with the estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the centers of rotation) (e.g., indicated by the aforementioned module 726). In addition to the 3D distance between the pupil centers and the estimated IPD, the vergence depth estimation module 728 may also utilize known, assumed, estimated, and / or determined geometries to calculate the vergence depth. As an example, module 728 may combine the 3D distance between the pupil centers, the estimated IPD, and the 3D CoR position in a trigonometric calculation to estimate (e.g., determine) the user's vergence depth. In fact, calculating such determined 3D distance between the pupil centers for the estimated IPD may be used to indicate a metric of the user's current vergence depth relative to optical infinity. In some examples, the vergence depth estimation module 728 may simply receive or access data indicative of the estimated 3D distance between the estimated user pupil centers in order to obtain such a metric of the vergence depth. In some embodiments, the vergence depth estimation module 728 may estimate the vergence depth by comparing the user's left and right optical axes. Specifically, the vergence depth estimation module 728 may estimate the vergence depth by locating the distance relative to the user at which the user's left and right optical axes intersect (or the projections of the user's left and right optical axes intersect on a plane (e.g., the horizontal plane)). By setting zero depth as the depth at which the user's left and right optical axes are separated by the user's IPD, module 728 may utilize the user's IPD in this calculation.In at least some embodiments, the vergence depth estimation module 728 may determine the vergence depth by triangulating the eye tracking data along with known or derived spatial relationships.

[0601] In some embodiments, the vergence depth estimation module 728 may estimate the user's vergence depth based on the intersection of the user's visual axes (as opposed to their optical axes), which may provide a more accurate indication of the distance at which the user is focused. In at least some embodiments, the eye tracking module 614 may include an optical axis to visual axis mapping module 730. As discussed in more detail Figure 10 The user's optical axis and visual axis typically do not align. The visual axis is the axis along which a person is gazing, while the optical axis is defined by the centers of the person's lens and pupil and may pass through the center of the person's retina. In particular, the user's visual axis is typically defined by the position of the user's fovea, which may be offset from the center of the user's retina, resulting in different optical and visual axes. In at least some of these embodiments, the eye tracking module 614 may include an optical axis to visual axis mapping module 730. The optical axis to visual axis mapping module 730 may correct for the difference between the user's optical axis and visual axis and provide information about the user's visual axis to other components in the wearable system (e.g., the vergence depth estimation module 728 and the light field rendering controller 618). In some examples, the module 730 may use a hypothesized eye size 704, which includes a typical offset of approximately 5.2° inward (towards the nose, towards the user's nose) between the optical axis and the visual axis. In other words, the module 730 may shift the user's left optical axis 5.2° to the right (towards the nose) and the user's right optical axis 5.2° to the left (towards the nose) in order to estimate the direction of the user's left and right optical axes. In other examples, the module 730 may utilize user-specific calibration data 706 when mapping the optical axis (e.g., as indicated by the module 722 above) to the visual axis. As an additional example, the module 730 may shift the user's optical axis towards the nose by 4.0° to 6.5°, 4.5° to 6.0°, 5.0° to 5.4°, etc., or any range formed by any of these values. In some arrangements, the module 730 may apply the shift at least in part based on characteristics of a particular user (e.g., their age, gender, vision prescription, or other relevant characteristics), and / or may apply the shift at least in part based on a calibration process for a particular user (e.g., determining the optical axis - visual axis offset for a particular user). In at least some embodiments, the module 730 may also shift the origin of the left and right optical axes to correspond to the user's CoP (determined by the module 732) rather than the user's CoR.

[0602] When an optional center of perspective (CoP) estimation module 732 is provided, the CoP estimation module 732 can estimate the positions of the user's left and right centers of perspective (CoP). The CoP can be a useful location for the wearable system and, in at least some embodiments, the CoP is the position directly in front of the pupil. In at least some embodiments, the CoP estimation module 732 can estimate the positions of the user's left and right centers of perspective based on the 3D position of the user's pupil center, the 3D position of the user's corneal curvature center, or such suitable data or any combination thereof. As an example, the user's CoP can be approximately 5.01 mm in front of the corneal curvature center (e.g., 5.01 mm from the corneal sphere center of the eye in the direction towards the cornea of the eye and along the optical axis) and can be approximately 2.97 mm behind the outer surface of the user's cornea along the optical axis or visual axis. The user's center of perspective can be directly in front of their pupil center. For example, the user's CoP can be less than approximately 2.0 mm from the user's pupil, less than approximately 1.0 mm from the user's pupil, less than approximately 0.5 mm from the user's pupil, or any range between any of these values. As another example, the center of perspective can correspond to a position within the anterior chamber of the eye. As other examples, the CoP can be between 1.0 mm and 2.0 mm, approximately 1.0 mm, between 0.25 mm and 1.0 mm, between 0.5 mm and 1.0 mm, or between 0.25 mm and 0.5 mm.

[0603] The center of perspective described herein (which may be the desired location of the rendering camera pinhole and an anatomical location within the user's eye) can be a location for reducing and / or eliminating unwanted parallax shifts. Specifically, the optical system of the user's eye is very approximately equivalent to a theoretical system formed by a pinhole in front of a lens projecting onto a screen, where the pinhole, lens, and screen approximately correspond to the user's pupil / iris, lens, and retina, respectively. Additionally, when two point light sources (or objects) at different distances from the user's eye rotate strictly around the pinhole opening (e.g., rotate along a radius of curvature equal to the respective distance of the point light sources from the pinhole opening), it is desirable to have little or no parallax shift. Thus, it might seem that the CoP should be located at the pupil center of the eye (and in some embodiments, such a CoP can be used). However, the human eye includes a cornea in addition to the pinhole of the lens and pupil, and the cornea imparts additional optical power to the light propagating towards the retina. Therefore, in the theoretical system described in this paragraph, the equivalent anatomical portion of the pinhole can be the region of the user's eye located between the outer surface of the user's cornea and the center of the user's pupil or iris. For example, the equivalent anatomical portion of the pinhole can correspond to the region within the anterior chamber of the user's eye. For various reasons discussed herein, it is desirable to set the CoP to such a position within the anterior chamber of the user's eye. The derivation and importance of the CoP are described in more detail below with reference to Figures 22 to 24B describe the derivation and importance of the CoP in more detail.

[0604] As described above, the eye tracking module 614 can provide data such as the estimated 3D positions of the centers of rotation (CoR) of the left and right eyes, vergence depth, left and right eye optical axes, the 3D position of the user's eyes, the 3D positions of the centers of curvature of the user's left and right corneas, the 3D positions of the centers of the user's left and right pupils, the 3D positions of the centers of perspective of the user, the user's IPD, etc. to other components in the wearable system, such as the light field rendering controller 618 and the registration viewer 620. The eye tracking module 614 can also include other sub-modules that detect and generate data associated with other aspects of the user's eyes. As an example, the eye tracking module 614 can include a blink detection module that provides a flag or other alert each time the user blinks and a saccade detection module that provides a flag or other alert each time the user's eyes saccade (e.g., quickly shift focus to another point).

[0605] Example of a rendering controller

[0606] A detailed block diagram of an example light field rendering controller 618 is shown in Figure 7B As Figure 6 and 7B shown, the rendering controller 618 can receive eye tracking information from the eye tracking module 614 and can provide an output to the rendering engine 622, which can generate an image to be displayed for viewing by the user of the wearable system. As an example, the rendering controller 618 can receive vergence depth, left and right eye centers of rotation (and / or centers of perspective), and other eye data, such as blink data, saccade data, etc.

[0607] The depth plane selection module 750 can receive vergence depth information and other eye data and, based on this data, can cause the rendering engine 622 to deliver content to the user through a specific depth plane (e.g., at a specific accommodation or focal length). As described in connection with Figure 4As discussed, the wearable system can include multiple discrete depth planes formed by a plurality of waveguides, each depth plane transmitting image information at a varying level of wavefront curvature. In some embodiments, the wearable system can include one or more variable depth planes, such as optical elements that transmit image information at a wavefront curvature level that varies over time. In these and other embodiments, the depth plane selection module 750 can cause the rendering engine 622 to transmit content to the user at a selected depth, at least in part based on the user's vergence depth (e.g., cause the rendering engine 622 to instruct the display 220 to switch depth planes). In at least some embodiments, the depth plane selection module 750 and the rendering engine 622 can render content at different depths and also generate depth plane selection data and / or provide depth plane selection data to display hardware, such as the display 220. Display hardware, such as the display 220, can perform electrical depth plane switching in response to depth plane selection data (which can be control signals) generated and / or provided by modules such as the depth plane selection module 750 and the rendering engine 622.

[0608] Generally, it is desirable for the depth plane selection module 750 to select a depth plane that matches the user's current vergence depth in order to provide the user with accurate accommodation cues. However, it is also desirable to switch depth planes in a discreet and unobtrusive manner. As an example, it is desirable to avoid excessive switching between depth planes and / or to switch depth planes when the user is less likely to notice the switch (e.g., during a blink or an eye saccade).

[0609] The cross-lag band detection module 752 can help avoid excessive switching between depth planes, particularly when the user's vergence depth fluctuates at or near the midpoint or transition point between two depth planes. Specifically, the module 752 can cause the depth plane selection module 750 to exhibit hysteresis in its depth plane selection. As an example, the module 752 can cause the depth plane selection module 750 to switch from a first, more distant depth plane to a second, closer depth plane only after the user's vergence depth crosses a first threshold. Similarly, the module 752 can cause the depth plane selection module 750 (which in turn instructs a display such as the display 220) to switch to the first, more distant depth plane only after the user's vergence depth crosses a second threshold that is further from the user than the first threshold. In the overlapping region between the first and second thresholds, the module 750 can cause the depth plane selection module 750 to maintain the currently selected depth plane as the selected depth plane, thereby avoiding excessive switching between depth planes.

[0610] The eye event detection module 750 can obtain from Figure 7AThe eye tracking module 614 receives other eye data and can cause the depth plane selection module 750 to delay certain depth plane switches until an eye event occurs. As an example, the eye event detection module 750 can cause the depth plane selection module 750 to delay a planned depth plane switch until a user blink is detected; can receive data indicating the time at which the user is currently blinking from a blink detection component in the eye tracking module 614; and in response, can cause the depth plane selection module 750 to perform the planned depth plane switch during the blink event (e.g., by causing module 750 to instruct the display 220 to perform a depth plane switch during the blink event). In at least some embodiments, the wearable system is capable of shifting content to a new depth plane during a blink event such that the user is less likely to perceive the shift. As another example, the eye event detection module 750 can delay a planned depth plane switch until an eye saccade is detected. As discussed in connection with blinks, this arrangement can facilitate discrete shifting of depth planes.

[0611] If desired, even in the absence of an eye event, the depth plane selection module 750 can delay a planned depth plane switch by only a limited time period before performing the depth plane switch. Similarly, even in the absence of an eye event, the depth plane selection module 750 can perform a depth plane switch when the user's vergence depth is substantially outside the currently selected depth plane (e.g., when the user's vergence depth has exceeded a conventional threshold for depth plane switching by a predetermined threshold). These arrangements can help ensure that the eye event detection module 754 does not indefinitely delay depth plane switches and does not delay depth plane switches when there is a large accommodation error. More details regarding the operation of the depth plane selection module 750 and how the module times depth plane switches are provided herein in connection with Figure 12 provided.

[0612] The rendering camera controller 758 may provide information indicating the positions of the user's left and right eyes to the rendering engine 622. The rendering engine 622 may then generate content by simulating cameras at the positions of the user's left and right eyes and generating content based on the perspectives of the simulated cameras. As described above, the rendering camera is a simulated camera for rendering virtual image content, which may be from a database of objects in a virtual world. The objects may have positions and orientations relative to the user or wearer, and possibly relative to real objects in the user's or wearer's surrounding environment. The rendering camera may be included in the rendering engine to render virtual images based on a database of virtual objects to be presented to the eyes. The virtual images may be rendered as if taken from the perspective of the user or wearer. For example, the virtual images may be rendered as if captured by a camera (corresponding to the "rendering camera") having an aperture, lens, and detector for viewing objects in the virtual world. The virtual images are taken from the perspective of such a camera at the "rendering camera" position. For example, the virtual images may be rendered as if captured from the perspective of a camera having a specific position relative to the user's or wearer's eyes, thereby providing an image that appears to be from the perspective of the user or wearer. In some embodiments, the images are rendered as if captured from the perspective of a camera having an aperture at a specific position relative to the user's or wearer's eyes (e.g., the center of perspective or center of rotation as discussed herein or elsewhere).

[0613] The rendering camera controller 758 may determine the positions of the left and right cameras based on the center of rotation (CoR) of the left and right eyes determined by the CoR estimation module 724 and / or based on the center of perspective (CoP) of the left and right eyes determined by the CoP estimation module 732. In some embodiments, the rendering camera controller 758 may switch between the CoR and CoP positions based on various factors. As an example, the rendering camera controller 758 may, in various modes, always register the rendering camera to the CoR position, always register the rendering camera to the CoP position, switch or discretely change between registering the rendering camera to the CoR position and registering the rendering camera to the CoP position over time based on various factors, or dynamically register the rendering camera to any one of a series of different positions along the optical axis (or line of sight) between the CoR and CoP positions over time based on various factors. The CoR and CoP positions may optionally be passed through a smoothing filter 756 (for any of the above modes of rendering camera positioning), which may average the CoR and CoP positions over time to reduce noise in these positions and prevent jitter in the rendered simulated rendering camera.

[0614] In at least some embodiments, the rendering camera can be simulated as a pinhole camera, where the pinhole is set at the position of the estimated CoR or CoP identified by the eye tracking module 614. In the case where the CoP deviates from the CoR, whenever the position of the rendering camera is based on the user's CoP, both the rendering camera and its pinhole are shifted as the user's eyes rotate (see, for example, how the rendering camera linearly translates as the eyes rotate as shown in Figure 16A and 16B ). Conversely, whenever the position of the rendering camera is based on the user's CoR, the position of the rendering camera pinhole does not move as the eyes rotate, although in some embodiments, the rendering camera (located behind the pinhole) can move as the eyes rotate. In other embodiments where the position of the rendering camera is based on the user's CoR, the rendering camera may not move (e.g., rotate) with the user's eyes (see, for example, how the rendering camera does not move or linearly translate as the eyes rotate as shown in Figure 17A and 17B ).

[0615] Example of localizing a user's cornea through an eye tracking system

[0616] Figure 8A is an eye schematic showing the corneal globe of the eye. As shown in Figure 8A , the user's eye 810 can have a cornea 812, a pupil 822, and a lens 820. The cornea 812 can have an approximately spherical shape shown by the corneal globe 814. The corneal globe 814 can have a center point 816 (also known as the corneal center) and a radius 818. The hemispherical cornea of the user's eye can curve around the corneal center 816.

[0617] Figures 8B to 8E shows an example of using the 3D corneal center estimation module 716 and the eye tracking module 614 to locate the corneal center 816 of the user.

[0618] As shown in Figure 8B , the 3D corneal center estimation module 716 can receive an eye tracking image 852 including a corneal bright spot 854. Then, the 3D corneal center estimation module 716 can simulate the known 3D positions of the eye camera 324 and the light source 326 in the eye camera coordinate system 850 (which can be based on data in the eye tracking extrinsic and intrinsic feature database 702, the assumed eye size database 704, and / or the per-user calibration data 706) in order to project rays 856 in the eye camera coordinate system. In at least some embodiments, the origin of the eye camera coordinate system 850 can be at the 3D position of the eye tracking camera 324.

[0619] In Figure 8CIn [the figure], the 3D corneal center estimation module 716 simulates the corneal sphere 814a (which may be based on the assumed eye size from the database 704) and the corneal curvature center 816a at the first position. The 3D corneal center estimation module 716 then can check whether the corneal sphere 814a reflects the light from the light source 326 properly to the bright spot position 854. As Figure 8C shown, when the ray 860a does not intersect with the light source 326, the first position does not match.

[0620] Similarly, in Figure 8D [the figure], the 3D corneal center estimation module 716 simulates the corneal sphere 814b and the corneal curvature center 816b at the second position. Then, the 3D corneal center estimation module 716 checks whether the corneal sphere 814b reflects the light from the light source 326 properly to the bright spot position 854. As Figure 8D shown, the second position also does not match.

[0621] As Figure 8E shown, the 3D corneal center estimation module 716 can finally determine that the correct position of the corneal sphere is the corneal sphere 814c and the corneal curvature center 816c. The 3D corneal center estimation module 716 confirms that the shown position is correct by checking that the light from the light source 326 is properly reflected from the corneal sphere at the correct position of the bright spot 854 on the image 852 and imaged by the camera 324. With this arrangement, and through the known 3D positions of the light source 326, the camera 324, and the optical characteristics (focal length, etc.) of the camera, the 3D corneal center estimation module 716 can determine the 3D position of the corneal curvature center 816 (relative to the wearable system).

[0622] The process described herein at least in conjunction with Figures 8C to 8E can be effectively used as an iterative, repetitive, or optimization process to identify the 3D position of the user's corneal center. In this way, any of a variety of techniques (e.g., iterative techniques, optimization techniques, etc.) can be used to effectively and quickly prune or reduce the search space of possible positions. Additionally, in some embodiments, the system may include two, three, four, or more light sources (e.g., the light source 326), and some of all these light sources may be set at different positions, resulting in multiple bright spots (e.g., the bright spot 584) at different positions on the image 852 and multiple rays (e.g., the ray 856) with different origins and directions. Such embodiments can enhance the accuracy of the 3D corneal center estimation module 716 because the module 716 seeks to identify the corneal position that causes some or all of the bright spots and rays to be properly reflected between their respective light sources and their respective positions on the image 852. In other words, in these embodiments, the positions of some or all of the light sources are dependent on Figures 8B to 8EThe process of 3D corneal position determination (e.g., iterative, optimization techniques, etc.). In some embodiments, the system may determine the vector or ray (i.e., 2D corneal center position) where the corneal center is located before performing the optimization process. In such embodiments, the 3D corneal center estimation module 716 may search for the corneal position only along such a vector, which can provide computational and / or time savings when the optimization process is executed. In at least some of these embodiments, before determining such a vector, the system may first (i) define a first plane between the origin of the eye camera coordinate system 850, the first light source (e.g., light source 326a), and the first bright spot (e.g., bright spot 854a) generated by the first light source, and (ii) define a second plane between the origin of the eye camera coordinate system 850, the second light source (e.g., light source 326b), and the second bright spot (e.g., bright spot 854b) generated by the second light source. Then, the system can simply calculate the cross product of the first plane and the second plane to determine the vector or ray (i.e., 2D corneal center position) where the corneal center is located.

[0623] Example of normalizing the coordinate system of an eye tracking image

[0624] Figures 9A to 9C illustrates an example of the normalization of the coordinate system of the eye tracking image by components (e.g., Figure 7A the coordinate system normalization module 718) in the wearable system. Normalizing the coordinate system of the eye tracking image with respect to the user's pupil position can compensate for the slip of the wearable system relative to the user's face (e.g., headset slip), and such normalization can establish a consistent orientation and distance between the eye tracking image and the user's eye.

[0625] As Figure 9A shown, the coordinate system normalization module 718 may receive the estimated 3D coordinates 900 of the user's corneal rotation center and may receive the unnormalized eye tracking image, such as image 852. As an example, the eye tracking image 852 and the coordinates 900 may be in the unnormalized coordinate system 850 based on the position of the eye tracking camera 324.

[0626] As a first normalization step, the coordinate system normalization module 718 may rotate the coordinate system 850 to a rotated coordinate system 902 such that the z-axis (e.g., the convergence depth axis) of the coordinate system can be aligned with the vector between the origin of the coordinate system and the corneal curvature center coordinates 900, as Figure 9B shown. Specifically, the coordinate system normalization module 718 may rotate the eye tracking image 850 to a rotated eye tracking image 904 until the coordinates 900 of the user's corneal curvature center are perpendicular to the plane of the rotated image 904.

[0627] As a second normalization step, the coordinate system normalization module 718 may translate the rotated coordinate system 902 into a normalized coordinate system 910 such that the corneal curvature center coordinates 900 are a standard normalized distance 906 from the origin of the normalized coordinate system 910, as Figure 9C shown. Specifically, the coordinate system normalization module 718 may translate the rotated eye tracking image 904 into a normalized eye tracking image 912. In at least some embodiments, the standard normalized distance 906 may be approximately 30 millimeters. The second normalization step may be performed before the first normalization step if desired.

[0628] Example of localizing the centroid of a user's pupil through an eye tracking system

[0629] Figures 9D to 9G An example of using the 3D pupil center locator module 720 and the eye tracking module 614 to locate the user's pupil center (e.g., the center of the user's pupil 822 as Figure 8A shown) is illustrated.

[0630] As Figure 9D shown, the 3D pupil center locator module 720 may receive the normalized eye tracking image 912 including the pupil centroid 913 (e.g., the center of the user's pupil identified by the pupil identification module 712). The 3D pupil center locator module 720 may then simulate the normalized 3D position 910 of the eye camera 324 to project a ray 914 through the pupil centroid 913 in the normalized coordinate system 910.

[0631] In Figure 9E , the 3D pupil center locator module 720 may simulate a corneal sphere based on data from the 3D corneal center estimation module 716, such as a corneal sphere 901 having a curvature center 900 (as discussed in more detail in connection with Figures 8B to 8E ). As an example, the corneal sphere 901 may be positioned in the normalized coordinate system 910 based on the position of the curvature center 816c identified in connection with Figure 8E and the normalization process based on Figures 9A to 9C . Additionally, as Figure 9E shown, the 3D pupil center locator module 720 may identify a first intersection 916 between the ray 914 (e.g., the ray between the origin of the normalized coordinate system 910 and the normalized position of the user's pupil) and the simulated cornea.

[0632] As Figure 9FAs shown, the 3D pupil center locator module 720 may determine a pupil sphere 918 based on a corneal sphere 901. The pupil sphere 918 may share a common center of curvature with the corneal sphere 901, but have a smaller radius. The 3D pupil center locator module 720 may determine the distance between the corneal center 900 and the pupil sphere 918 (e.g., the radius of the pupil sphere 918) based on the distance between the corneal center and the pupil center. In some embodiments, the distance between the pupil center and the corneal center of curvature may be determined according to Figure 7A an assumed eye size 704, according to an eye tracking extrinsic and intrinsic feature database 702, and / or according to user calibration data 706. In other embodiments, the distance between the pupil center and the corneal center of curvature may be determined according to Figure 7A user calibration data 706.

[0633] As Figure 9G shown, the 3D pupil center locator module 720 may locate the 3D coordinates of a user's pupil center based on various inputs. As an example, the 3D pupil center locator module 720 may utilize the 3D coordinates and radius of the pupil sphere 918, the 3D coordinates of the intersection 916 between the simulated corneal sphere 901 in the normalized eye tracking image 912 and a ray 914 associated with the pupil centroid 913, information regarding the refractive index of the cornea, and other relevant information such as the refractive index of air (which may be stored in the eye tracking extrinsic and intrinsic feature database 702) to determine the 3D coordinates of the user's pupil center. Specifically, the 3D pupil center locator module 720 may bend the ray 916 into a refracted ray 922 based on the refractive index difference between air (having a first refractive index of approximately 1.00) and the corneal material (having a second refractive index of approximately 1.38) during simulation. After considering the refraction caused by the cornea, the 3D pupil center locator module 720 may determine the 3D coordinates of a first intersection 920 between the refracted ray 922 and the pupil sphere 918. The 3D pupil center locator module 720 may determine that the user's pupil center 920 is located at an approximate first intersection 920 between the refracted ray 922 and the pupil sphere 918. With this arrangement, the 3D pupil center locator module 720 may determine the 3D position of the pupil center 920 (relative to the wearable system) in the normalized coordinate system 910. If desired, the wearable system may denormalize the coordinates of the pupil center 920 to the original eye camera coordinate system 850. The pupil center 920 may be used together with the corneal center of curvature 900 to determine, among other things, the user's optical axis using the optical axis determination module 722 and the user's vergence depth using the vergence depth estimation module 728.

[0634] Considering corneal refraction may result in a more stable determined pupil position than the pupil position determined based on the first intersection point 916 between the ray 914 (i.e., the ray between the origin of the normalized coordinate system 910 and the normalized position of the user's pupil) and the simulated cornea, as Figure 9E shown. This is partially correct because although it is easier to calculate, the first intersection point 916 may not correspond to the physical characteristics of the eye and thus cannot move with the eye as an entity. Instead, even though there is still some variation due to the viewing angle, calculating the pupil center 920 by considering corneal refraction can better correspond to the physical pupil position of the eye. Therefore, in various embodiments, determining the optical axis of the eye thus involves calculating the true pupil center rather than the first intersection point 916 between the ray 914 and the simulated cornea.

[0635] When the center of rotation (CoR) is estimated as a point having a fixed distance along the optical axis of the eye relative to the corneal center, including corneal refraction will have significant benefits. Specifically, including corneal refraction when determining the pupil position can significantly reduce the variation when calculating the center of rotation for different eye orientations. For example, when the eye moves as a whole in the camera coordinate system (e.g., during reinstallation of the head-mounted device), it may cause variation because the eye as a whole will be oriented differently relative to the reinstalled head-mounted device. Since the pupil center 920 better corresponds to the physical pupil position of the eye, the variation of the CoR can be smaller when the eye moves as a whole in the camera coordinate system. Advantageously, including corneal surface refraction can result in a more stable and accurate CoR, which can be used to determine when to reposition the head-mounted device onto the user's head, can allow for a more correct placement of the rendering camera, can allow for other novel gaze tracking algorithms or any combination thereof. Additionally, the CoR, as a stable and slowly changing eye feature, can potentially be tracked by a multi-frame Kalman-type time filter to provide a geometric reference position for other applications. Figure 9H Examples of the calculated positions of the pupil center due to including and not including corneal refraction are shown. When calculating the pupil center without considering the effect of corneal refraction, the result can be the external pupil 960. The external pupil center 960 can correspond to the first intersection point 916 between the ray 914 (i.e., the ray between the origin of the normalized coordinate system 910 and the normalized position of the user's pupil) and the simulated cornea, as Figure 9E shown. When calculating the pupil center considering corneal refraction, the result can be the refracted pupil center 962. The refracted pupil center 962 can correspond to the pupil center 920, as Figure 9GAs shown. Different positions of the outer pupil center 960 and the refractive pupil center 962 can result in different calculations of the eye rotation center, which is determined as a fixed distance along the optical axis of the eye relative to the corneal center 964. For example, the outer pupil center 960 can result in an outer rotation center 968. The outer rotation center 968 can be significantly different from the rotation center 966 calculated based on the refractive pupil center 962.

[0636] Figures 9I to 9L Shows an example experimental variation of the calculated eye rotation center based on different calculated pupil centers using a set of over 400 datasets. The data selected are only those frames with four bright spots and valid pupil centers (i.e., not all three x, y, z coordinate components are equal to zero) to exclude obvious pipeline failure cases. Different rotation center (CoR) to corneal curvature center distance values (R) were studied, and it was found that R = 4.7 mm gave nearly the best average result. However, specific user distance values can be adjusted to provide a smaller CoR coordinate variation.

[0637] Figure 9I Shows the x, y, and z coordinates of different pupil centers, and a three-dimensional plot of the corresponding CoR calculated using different pupil centers for the above datasets. Figure 9J Shows the data in the XY projection Figure 9I The cluster 970 corresponds to the coordinates of the outer pupil center 960, the cluster 972 corresponds to the coordinates of the correct refractive pupil center 962, the cluster 974 corresponds to the corneal position (e.g., the three-dimensional corneal center position), the cluster 978 corresponds to the CoR using the correct refractive pupil center 962, and the cluster 980 corresponds to the CoR using the outer pupil center 960. The size of the cluster 978 (especially the size in the x direction) is smaller than that of the cluster 980, indicating that the variation of the CoR when using the refractive pupil center 962 is smaller than that when using the outer pupil center 960.

[0638] Table 1

[0639]

[0640] As can be seen from Table 1, when calculating using the refractive pupil center 962, the standard deviation (or sigma) of the x component of the CoR is reduced by approximately half compared to using the outer pupil center 960. By using the refractive pupil center 962, the total three-dimensional standard deviation (sigma 3d ) is also significantly reduced.

[0641] Figure 9K and 9LSeparate plots show the mean and median CoR standard deviations as a function of CoR to corneal curvature center distance for a collection of over 400 datasets. Plots 991L and 991R show the mean three-dimensional CoR standard deviation (sigma 3d ) as a function of CoR to corneal curvature center distance for the left and right eyes, respectively. Plots 992L and 992R show the median three-dimensional CoR standard deviation (sigma 3d ) as a function of CoR to corneal curvature center distance for the left and right eyes, respectively. Curve 982A corresponds to the total left eye mean sigma 3d , curve 982B corresponds to the total right eye mean sigma 3d , curve 982C corresponds to the total left eye median sigma 3d , and curve 982D corresponds to the total right eye median sigma 3d . Curves 984A-D correspond to the x-components of the respective sigmas 3d , curves 990A-D correspond to the y-components of the respective sigmas 3d , and curves 986A-D correspond to the z-components of the respective sigmas 3d .

[0642] Figure 9M shows an example distribution of the optimal radius calculated separately for each user dataset as the value of CoR to corneal distance that provides the minimum sigma 3d . The mean of the distribution was found to be at R = 4.9 mm, with a standard deviation of 1.5 mm. Users with extremely small radii (R ~ 1 mm) were found to have extremely poor gaze tracking and were therefore excluded.

[0643] Example of the difference between the optical axis and the visual axis

[0644] As discussed in connection with Figure 7A the optical axis to visual axis mapping module 730, the optical axis and visual axis of a user are typically misaligned, in part because the user's visual axis is defined by their fovea, and the fovea is typically not at the center of the human retina. Thus, when a person wishes to focus their attention on a particular object, they align their visual axis with the object to ensure that light from the object falls on their fovea, while their optical axis (defined by the center of their pupil and the center of their corneal curvature) is actually slightly offset from the object. Figure 10 is an example of an eye 1000 showing the eye optical axis 1002, the eye visual axis 1004, and the offset between these axes. Additionally, Figure 10The pupil center 1006 of the eye, the corneal curvature center 1008 of the eye, and the average center of rotation (CoR) 1010 of the eye are shown. In at least some populations, the corneal curvature center 1008 of the eye can be located approximately 4.7 mm in front of the average center of rotation (CoR) 1010 of the eye, as shown by dimension 1012. Additionally, the perspective center 1014 of the eye can be located approximately 5.01 mm in front of the corneal curvature center 1008 of the eye, approximately 2.97 mm behind the outer surface 1016 of the user's cornea, and / or just in front of the pupil center 1006 of the user (e.g., corresponding to a position within the anterior chamber of the eye 1000). As an additional example, dimension 1012 can be between 3.0 mm and 7.0 mm, between 4.0 mm and 6.0 mm, between 4.5 mm and 5.0 mm, or between 4.6 mm and 4.8 mm, or within any range between any values, and be any value within any of these ranges. The perspective center (CoP) 1014 of the eye can be a useful location for a wearable system because, in at least some embodiments, registering the rendering camera at the CoP helps reduce or eliminate parallax artifacts.

[0645] Figure 10 Also shown is such within the human eye 1000 that can be aligned with the pinhole of the rendering camera. As Figure 1 shown. As Figure 10 shown, the pinhole of the rendering camera can be registered with position 1014, which is along the optical axis 1002 or visual axis 1004 of the human eye 1000, and which is closer to the outer surface of the cornea than both (a) the center 1006 of the pupil or iris and (b) the corneal curvature center 1008 of the human eye 1000. For example, as Figure 10 shown, the pinhole of the rendering camera can be registered with position 1014, which is along the optical axis 1002 of the human eye 1000, approximately 2.97 mm behind the outer surface 1016 of the cornea, and approximately 5.01 mm in front of the corneal curvature center 1008. The position 1014 of the pinhole of the rendering camera and / or the anatomical region of the human eye 1000 corresponding to position 1014 can be considered to represent the perspective center of the human eye 1000. Figure 10 The optical axis 1002 of the human eye 1000 shown represents the most direct line passing through the corneal curvature center 1008 and the center 1006 of the pupil or iris. The visual axis 1004 of the human eye 1000 is different from the optical axis 1002 because it represents the line extending from the fovea of the human eye 1000 to the center 1006 of the pupil or iris.

[0646] Example process of rendering content and checking registration based on eye tracking

[0647] Figure 11is a process flow diagram of an example method 1100 for rendering content and providing registration feedback in a wearable device using eye tracking. Method 1100 may be performed by a wearable system described herein. Embodiments of method 1100 may be used by a wearable system to render content and provide registration feedback (e.g., fit of the wearable device to the user) based on data from an eye tracking system.

[0648] At block 1110, the wearable system may capture an image of one or more of the user's eyes. The wearable system may use one or more eye cameras 324 as shown at least in Figure 3 the example of. If desired, the wearable system may also include one or more light sources 326 configured to shine IR light on the user's eyes and create corresponding bright spots in the eye images captured by the eye cameras 324. As discussed herein, the eye tracking module 614 may use the bright spots to derive various pieces of information about the user's eyes, including where the eyes are looking.

[0649] At block 1120, the wearable system may detect bright spots and pupils in the eye images captured at block 1110. As an example, block 1120 may include processing the eye images by a bright spot detection and marking module 714 to identify the two-dimensional positions of the bright spots in the eye images, and processing the eye images by a pupil identification module 712 to identify the two-dimensional positions of the pupils in the eye images.

[0650] At block 1130, the wearable system may estimate the three-dimensional positions of the user's left and right corneas relative to the wearable system. As an example, the wearable system may estimate the positions of the centers of curvature of the user's left and right corneas and the distances between these centers of curvature and the user's left and right corneas. Block 1130 may include a 3D corneal center estimation module 716 identifying the positions of the centers of curvature, as at least in combination with Figure 7A and Figures 8A to 8E described.

[0651] At block 1140, the wearable system may estimate the three-dimensional positions of the user's left and right pupil centers relative to the wearable system. As an example, the wearable system and a 3D pupil center locator module 720 may specifically estimate the positions of the user's left and right pupil centers, as at least in combination with Figure 7A and Figures 9D to 9G described, as part of block 1140.

[0652] At block 1150, the wearable system may estimate the three-dimensional positions of the user's left and right centers of rotation (CoR) relative to the wearable system. As an example, the wearable system and a CoR estimation module 724 may specifically estimate the positions of the CoRs of the user's left and right eyes, as at least in combination with Figure 7A andFigure 10 As a specific example, the wearable system can find the CoR of the eye by returning from the corneal curvature center along the optical axis to the retina.

[0653] At block 1160, the wearable system can estimate the user's IPD, convergence depth, center of perspective (CoP), optical axis, visual axis, and other desired attributes based on the eye tracking data. As an example, the IPD estimation module 726 can estimate the user's IPD by comparing the 3D positions of the left and right CoRs, the convergence depth estimation module 728 can estimate the user's depth by finding the intersection (or near intersection) of the left and right optical axes or the intersection of the left and right visual axes, the optical axis determination module 722 can identify the left and right optical axes over time, the optical axis to visual axis mapping module 730 can identify the left and right visual axes over time, and the CoP estimation module block 732 can identify the left and right centers of perspective, as part of block 1160.

[0654] At block 1170, the wearable system can render content based in part on the eye tracking data identified in blocks 1120 to 1160 and can optionally provide registration feedback (e.g., the fit of the wearable system to the user's head). As an example, the wearable system can identify the appropriate position of the rendering camera and then generate content for the user based on the position of the rendering camera, as discussed in connection with the light field rendering controller 618, Figure 7B and the rendering camera 622. As another example, the wearable system can determine whether it is properly fitted to the user or whether it has slipped from its correct position relative to the user and can provide the user with optional feedback indicating whether the device fit needs to be adjusted, as discussed in connection with the registration viewer 620. In some embodiments, the wearable system can adjust the rendered content based on improper or less than ideal registration to attempt to reduce, minimize, or compensate for the effects of improper or incorrect registration.

[0655] Example figure of rendering content in response to a user's eye movement

[0656] Figure 12 including a set of example figures 1200a - 1200j, which illustrate how the wearable system can switch depth planes in response to user eye movements. As described herein in connection with Figure 4As discussed in Sections 6 and 7, a wearable system may include multiple depth planes, where each depth plane is configured to present content to a user at a different simulated depth or via different accommodation cues (e.g., via various wavefront curvatures or light divergence levels). As an example, the wearable system may include a first depth plane configured to simulate a first depth range and a second depth plane configured to simulate a second depth range, although the two ranges may overlap as needed to facilitate switching hysteresis, but the second depth range generally may extend to a greater distance relative to the user. In such an embodiment, the wearable system may track the user's vergence depth, saccadic movements, and blinks in order to switch between the first and second depth planes in a manner that avoids excessive depth plane switching, excessive accommodation-vergence mismatch, excessive accommodation-vergence mismatch periods, and seeks to reduce the visibility of depth plane switching (e.g., by transitioning depth planes during blinks and saccades).

[0657] Graph 1200a shows an example of a user's vergence depth over time. Graph 1200b shows an example of a user's saccade signal or eye movement speed over time.

[0658] Graph 1200c may show the vergence depth data generated by the eye tracking module 614, specifically, the data generated by the vergence depth estimation module 728. As shown in Graphs 1200c - 1200h, the eye tracking data may be sampled within the eye tracking module 614 at a rate of approximately 60 Hz. As shown between Graphs 1200b and 1200c, the eye tracking data within the eye tracking module 614 may lag the user's actual eye movements by a latency 1202. As an example, at time t 1 , the user's vergence depth may cross the hysteresis threshold 1210a, but the cross-hysteresis band detection module 752 does not identify the event until time t 2 after the latency 1202.

[0659] Graph 1200c also shows various thresholds 1210a, 1210b, 1210c within the hysteresis band, which may be associated with the first and second depth planes (e.g., Figure 12associated with the transition between depth planes #1 and #0). In some embodiments, whenever the user's vergence depth is greater than threshold 1210b, the wearable system may attempt to display content in depth plane #1; whenever the user's vergence depth is less than threshold 1210b, the wearable system may attempt to display content in depth plane #0. However, to avoid excessive switching, the wearable system may implement hysteresis such that the wearable system does not switch from depth plane #1 to depth plane #0 until the user's vergence depth crosses an outer threshold 1210c. Similarly, the wearable system does not switch from depth plane #0 to depth plane #1 until the user's vergence depth crosses an outer threshold 1210a.

[0660] Graph 1200d shows an internal flag that can be generated by depth plane selection module 750 or cross-hysteresis band detection module 752, which indicates whether the user's vergence depth is in the volume typically associated with depth plane #1 or in the volume typically associated with depth plane #2 (e.g., whether the user's vergence depth is greater than or less than threshold 1210b).

[0661] Graph 1200e shows an internal hysteresis band flag that can be generated by depth plane selection module 750 or cross-hysteresis band detection module 752, which indicates whether the user's vergence depth has crossed an outer threshold, such as threshold 1210a or 1210c. Specifically, graph 1200e shows a flag that indicates whether the user's vergence depth has completely crossed the hysteresis band and entered a region outside the volume of the active depth plane (e.g., entered a region associated with a depth plane other than the active depth plane), thus potentially causing an unwanted accommodation-vergence mismatch (AVM).

[0662] Graph 1200f shows an internal AVM flag that can be generated by depth plane selection module 750 or cross-hysteresis band detection module 752, which indicates whether the time that the user's vergence is outside the volume of the active depth plane exceeds a predetermined time. Thus, the AVM flag can identify when the user experiences an unwanted accommodation-vergence mismatch during a nearly excessive or excessive period. Additionally or alternatively, the internal AVM flag can also indicate whether the user's vergence has exceeded a predetermined distance from the volume of the active depth plane, thus causing a potential excessive accommodation-vergence mismatch. In other words, the AVM flag can indicate when the user's vergence exceeds an additional threshold that is even farther from threshold 1210b than thresholds 1210a and 1210c.

[0663] Graph 1200g shows an internal blink flag that can be generated by eye event detection module 754, which can determine when the user has blinked or is blinking. As described herein, it is desirable to switch depth planes when the user blinks to reduce the likelihood that the user perceives the depth plane switch.

[0664] Graph 1200h shows an example output from the depth plane selection module 750. Specifically, Graph 1200h shows that the depth plane selection module 750 can output instructions to the rendering engine (e.g., rendering engine 622, see Figure 6 ) that utilize the selected depth plane, which changes over time.

[0665] Graphs 1200i and 1200j show the latencies that may exist in the wearable system, including the latency for the rendering engine 622 to switch depth planes and the latency for the display 220 to provide light associated with a new image frame in the new depth plane to effect the depth plane change.

[0666] Now, events shown at different times (t 0 -t 10 ) will be referenced with respect to Graphs 1200a - 1200j.

[0667] At some time around time t 0 , the user's vergence depth may cross threshold 1210a, which may be an external hysteresis threshold. After the latency associated with image capture and signal processing, the wearable system can generate a signal, as shown in Graph 1200e, that indicates the user's vergence depth is within the hysteresis band. In the example of Graph 1200e, one or more modules of the light field rendering controller 618 can present a hysteresis band exceeded flag at approximately time t 1 .

[0668] From time t 0 to approximately time t 4 , the user's vergence depth continuously decreases and then increases.

[0669] At time t 1 , the user's vergence depth crosses threshold 1210b, which may be the midpoint between two depth planes (e.g., depth plane #1 and #0). After the processing latency 1202, the eye tracking module 614 can change an internal flag indicating that the user's vergence depth has moved from the volume typically associated with depth plane #1 to the volume typically associated with depth plane #0 (as shown in Graph 1200d).

[0670] At time t 3At this point, one or more modules of the light field rendering controller 618 may determine that the user's vergence depth (as shown in graph 1200a) has fully moved through the hysteresis band and beyond the outer threshold 1210c. Accordingly, one or more modules of the light field rendering controller 618 may generate a signal indicating that the user's vergence depth is outside the hysteresis band (as shown in graph 1200e). In at least some embodiments, one or more modules of the light field rendering controller 618 (e.g., the depth plane selection module 750) switch between the first and second depth planes only when the user's vergence depth is outside the hysteresis band located between the two depth planes.

[0671] In at least some embodiments, one or more modules of the light field rendering controller 618 may be configured to switch depth planes at time t 3 Specifically, one or more modules of the light field rendering controller 618 may be configured to switch depth planes based on determining that the vergence depth has moved from the volume of the currently selected depth plane (depth plane #1 shown in graph 1200h) to the volume of another depth plane (depth plane #0) and has fully crossed the hysteresis band. In other words, whenever the hysteresis band is exceeded (graph 1200e is high) and an accommodation-vergence mismatch based on time or amount of mismatch is detected (graph 1200f is high), one or more modules of the light field rendering controller 618 may effect a depth plane switch. In such an embodiment, one or more modules of the light field rendering controller 618 may provide a signal to the rendering engine 622 indicating that the rendering engine 622 switch to another depth plane (depth plane #0). However, in Figure 12 an example, one or more modules of the light field rendering controller 618 may be configured to delay the depth plane switch until at least one other condition is met. These additional conditions may include, for example, a blink condition, an accommodation-vergence mismatch timeout condition, and an accommodation-vergence amount condition.

[0672] At time t 4 and at Figure 12In the example of, one or more modules of the light field rendering controller 618 may be configured to switch depth planes. Specifically, one or more modules of the light field rendering controller 618 may determine that the user's vergence has been within the volume associated with depth plane #0 for a time longer than a predetermined time threshold (and optionally, outside the hysteresis band during that time period). Examples of the predetermined time threshold include 5 seconds, 10 seconds, 20 seconds, 30 seconds, 1 minute, and 90 seconds, as well as any range between any of these values. Based on such a determination, one or more modules of the light field rendering controller 618 may generate an AVM flag, as shown in graph 1200f, and instruct the rendering engine 622 to switch to depth plane #0, as shown in graph 1200h. In some embodiments, if it is detected that the user's vergence depth is greater than a threshold distance from the currently selected depth volume, one or more modules of the light field rendering controller 618 may generate an AVM flag and instruct the rendering engine 622 to switch depth planes.

[0673] At time t 5 and after a delay 1204, the rendering engine 622 may begin rendering content at the newly selected depth plane #0. After a delay 1206 associated with the rendering and after transmitting light to the user through the display 220, the display 220 may fully switch to the newly selected depth plane #0 before time t 6

[0674] Thus, graphs 1200a-j show between time t 0 and t 6 how the system responds to the user's changing vergence and how to switch depth planes after the user's vergence has moved away from the previous depth volume for more than a predetermined time period. Graphs 1200a-j show between time t 7 and t 10 how the system responds to the user's changing vergence and how to switch depth planes when a user blink is detected (possibly before the predetermined time period).

[0675] At time t 7 the one or more modules of the light field rendering controller 618 may detect that the user's vergence depth has entered the hysteresis region located between depth planes #0 and #1 (e.g., the user's vergence depth has crossed the outer threshold 1210c). In response, one or more modules of the light field rendering controller 618 may change the hysteresis flag, as shown in graph 1200e.

[0676] At time t 8 ​At this point, one or more modules of the light field rendering controller 618 may detect that the user's vergence depth has crossed the threshold 1210b and moved from the volume typically associated with depth plane #0 to the volume typically associated with depth plane #1. Accordingly, one or more modules of the light field rendering controller 618 may change the depth volume flag, as shown in graph 1200d.

[0677] At time t 9 At this point, one or more modules of the light field rendering controller 618 may detect that the user's vergence depth has crossed the threshold 1210a and moved out of the hysteresis volume and into the volume typically associated only with depth plane #1. In response, one or more modules of the light field rendering controller 618 may change the hysteresis flag, as shown in graph 1200e.

[0678] At time t 10 Around this time, the user blinks, and one or more modules of the light field rendering controller 618 may detect the blink. As an example, the eye event detection module 754 may detect the user's blink. In response, one or more modules of the light field rendering controller 618 may generate a blink flag, as shown in graph 1200h. In at least some embodiments, whenever the hysteresis band is exceeded (graph 1200e is high) and a blink is detected (graph 1200g is high), one or more modules of the light field rendering controller 618 may effect a depth plane switch. Accordingly, one or more modules of the light field rendering controller 618 may instruct the rendering engine 622 to switch depth planes at time t 10 At this point.

[0679] Example rendering modes in a mixed reality system with multiple depth planes

[0680] In a mixed reality system, a computer-generated (rendered) scene may be delivered to the human eye such that real and virtual objects are spatially aligned (from the user's perspective). To provide the user with a visual perception of the spatial alignment between the real and virtual objects, the perspective from which the computer-generated scene is rendered and presented may preferably correspond to the perspective (e.g., position and orientation) of the user's eyes. As an example, when the "real world" frame (in which real objects are present) and the "rendered world" frame (in which virtual objects are present) are accurately aligned with each other, the user may perceive the real and virtual objects to be spatially aligned in the desired manner.

[0681] Digital light field display devices (e.g., Figure 2The wearable system 200 including the display 220) is an example of a mixed reality system, where one or more depth planes can be used to provide a light field representing 3D virtual content (virtual objects) to the user at various depths. The depth planes can be compared to one or more virtual screens located at different distances relative to the user, on which the virtual content can be projected or displayed, transformed into virtual pixels, and provided to the user. In this way, the mixed reality system can be optically equivalent to a system with one or more transparent floating screens located at different distances relative to the user. In this manner, the digitized light field is projected through the user's iris onto the user's retina and forms an image of the 3D virtual content (e.g., the user perceives an image of the 3D virtual content).

[0682] Figure 13 FIG. shows a mixed reality system 1300, where one or more depth planes are used to provide a light field representing 3D virtual content including virtual objects to the user's eyes. The depth planes can be optical structures on the wearable device that simulate virtual screens at various distances separated from the user and the wearable device. The mixed reality system 1300 can include an eyepiece 1310, which can represent the eyepiece of a head-mounted digital light field display device (e.g Figure 3 the display 220) or a part thereof. Such a system can be configured to project a light field representing the 3D virtual object 1330 onto the retina 1303 of the user's eye 1302 through the eyepiece 1310.

[0683] Figure 13 Also shown are depth planes 1321 - 1323 on which the virtual object 1330 and other virtual content can be projected or displayed and transformed into virtual pixels. In Figure 13 the specific example shown, the virtual object 1330 is projected onto the depth plane 1322 and thus transformed into virtual pixels 1332. As a result, the light generated by the eyepiece 1310 (e.g., the display 220) can provide accommodation cues to the user's eye 1302 as if the virtual object 1330 were provided on a physical display or projector screen at a certain distance from the user from the depth plane 1322. The head-mounted digital display device can generate a digitized light field representing the virtual pixels 1332 and project such a light field onto the retina 1303 of the user's eye 1302 through the eyepiece 1310.

[0684] As will be discussed in more detail below, in a mixed reality system (e.g Figure 13In the mixed reality system 1300), different rendering modes are adopted to provide different eye accommodation cues across the user's entire field of view for different content and / or at different time periods. As an example, the mixed reality system can adopt a discrete zoom mode in which a single depth plane is used at a time to display virtual objects (as shown in Figures 14A to 14B ), a hybrid zoom mode can be adopted in which two adjacent depth planes are used to display virtual objects to generate accommodation cues between the two depth planes (as shown in Figures 14C to 14D ), and a multi-focus mode can be adopted in which two or more depth planes are used to display virtual objects to simultaneously generate two or more accommodation cues (as shown in Figures 14E to 14F ). Generally, in response to various conditions, the mixed reality system can switch between these and other rendering modes during operation. As an example, the mixed reality system can use a first rendering mode (e.g., discrete zoom mode) when displaying a first type of content (e.g., text that can be provided at a single depth), and a second different rendering mode (e.g., multi-focus mode) when displaying a second type of content (e.g., content that can be provided at multiple depths simultaneously). In at least some embodiments, Figure 6 's light field rendering controller 618 can be configured to select which rendering mode to adopt at any given time based on the various inputs and conditions discussed herein.

[0685] Single depth plane rendering mode (discrete zoom mode)

[0686] As Figure 14A and 14B shown, the wearable system described herein can adopt what is referred to herein as a single depth plane rendering mode (also known as discrete zoom mode) to render virtual reality objects using a single depth plane at a time. In the discrete zoom mode, the wearable system can use a single depth plane across the entire field of view (FOV) of the display to display all currently rendered virtual objects (even if some of these objects are set at depths other than the depth associated with that depth plane). In other words, the wearable system can provide a single focus or accommodation cue across the entire FOV. Of course, the wearable system can switch the depth plane used to render content over time, thereby changing the accommodation cues of the rendered virtual content over time (in response to changes in user convergence depth, virtual content depth, and other factors described in more detail herein). Generally, in the discrete zoom mode, the mixed reality system adopts only one depth plane at any given moment.

[0687] Figures 14A to 14B Show a top view and a side / iso view of the mixed reality system 1400A operating in discrete zoom mode to present content to the user's eyes 1402, respectively. Figures 14A to 14BDepth planes 1421, 1422, and 1423 are also shown, onto which virtual content can be projected and transformed into virtual pixels. When operating in a discrete zoom mode, the virtual content can only be projected onto one of the depth planes 1421 - 1423 at a given time. In Figures 14A to 14B the example of, the mixed reality system 1400A has switched to a state where the virtual content is projected onto depth plane 1422 but not onto either of depth planes 1421 or 1423. As shown, the virtual content projected onto depth plane 1422 is transformed into virtual pixels 1432A.

[0688] As discussed in more detail in connection with Figure 4 , 6 , 7A, 7B, and 12, which depth all pixels are placed at (e.g., onto which depth plane the virtual content is projected) and the switching timing between depth planes can be selected based on eye tracking and virtual content information. As an example, a wearable system operating in a discrete zoom mode can be based on the user's vergence depth, based on the user's vergence depth but with depth plane switching delayed until a trigger event (e.g., when the user blinks or saccades, after a predetermined AVM mismatch timeout, as discussed in connection with Figure 12 etc.), based on the depth of the virtual content, based on the depth of the virtual content but with depth plane switching delayed until a trigger event (e.g., when the user blinks or saccades, after a predetermined AVM mismatch timeout, as discussed in connection with Figure 12 etc.) or any combination of the above conditions, to switch which active depth plane. In at least some embodiments, one or more of the depth plane selection module 750, the cross-lag band detection module 752, and the eye event detection module 754 can be configured, either individually or in combination, to implement the desired depth plane switching scheme.

[0689] Mixed depth plane rendering mode (mixed zoom mode)

[0690] As Figure 14C and 14DAs shown, the wearable system described herein can use two adjacent depth planes to render virtual reality objects to generate accommodation or focus cues located between the depth planes. In some embodiments, such a hybrid zoom mode can enable the wearable system to generate accommodation or focus cues at any distance (between and including the depths provided by a set of depth planes in the wearable system). In other words, if the system includes three depth planes, the first depth plane provides accommodation cues for 1 foot, the second depth plane provides accommodation cues for 10 feet, and the third depth plane provides accommodation cues for optical infinity, the wearable system can provide accommodation cues for the user at any position from the 1-foot depth of the first plane to the optical infinity depth of the third depth plane.

[0691] Figures 14C to 14D A top view and a side / iso view of a mixed reality system 1400C operating in a hybrid zoom mode to present content to the user's eyes 1402 are shown. The mixed reality system 1400C can, for example, have the same architecture as the mixed reality system 1400A described above with reference to Figures 14A to 14B When operating in the hybrid zoom mode, virtual content can be projected onto two or more of the depth planes 1421-1423 at any given point in time to generate accommodation cues between the planes. In Figures 14C to 14D the example, the mixed reality system 1400C has switched to a state in which virtual content is projected onto the depth planes 1421 and 1422. As shown, the virtual content projected onto the depth planes 1421 and 1422 is transformed into virtual pixel sets 1431C and 1432C, respectively. The virtual pixel sets 1431C and 1432C can be mixed together and provide accommodation cues for the user somewhere between the depth planes 1421 and 1422 based on their relative intensities.

[0692] In the hybrid zoom mode, the wearable system can provide the same focus or accommodation cues for all pixels in the FOV of the display, and the accommodation cues can vary continuously between the depths of any pair of adjacent depth planes. By mixing pixel intensities between two depth planes, the wearable system can achieve continuously variable accommodation cues. As an example, the wearable system can display a virtual object with accommodation cues between the depth planes 1422 and 1421 by rendering the virtual object in two depth planes 1421 and 1421. In another example where the virtual object is closer to the depth of the depth plane 1421, the wearable system can render the virtual object in the depth plane 1421 with a greater light intensity (e.g., brightness) than in the depth plane 1422. In such an arrangement, the light from the two depth planes can be mixed so that the user perceives that the virtual object has accommodation cues located near the depth plane 1421 (but still between the planes 1421 and 1422).

[0693] In the hybrid zoom mode, the wearable system is configured to select which adjacent depth planes to blend to provide the desired accommodation cues. However, since the accommodation cues can vary continuously between planes by continuously changing the brightness, the timing of the depth plane switch may not be as critical as in the discrete zoom mode. Thus, the wearable system can be configured to switch the two depth planes that form a pair of adjacent depth planes without waiting for a trigger event such as a user blink, saccade, or AVM timeout. Instead, the wearable system can smoothly change the provided accommodation cues and the depth planes used over time in response to the user's vergence depth, the depth of the virtual content, or a combination of these and other inputs as needed.

[0694] Multi-depth plane rendering mode (multi-focus mode)

[0695] As Figure 14E and 14F shown, the wearable systems described herein can use two or more depth planes to render virtual reality objects in a multi-focus mode, thereby generating two or more accommodation cues simultaneously. In other words, the virtual content in a given frame can be presented across multiple depths simultaneously. As an example, the multi-focus mode can include using two or more depth planes to provide a first hybrid accommodation cue in a manner described in conjunction with the hybrid zoom mode; using two or more depth planes to provide a second hybrid accommodation cue; using a single depth plane to provide a third accommodation cue; using a second single depth plane to provide a fourth accommodation cue; or using a combination of these and other focus modes to provide various accommodation cues.

[0696] Figures 14E to 14F FIGS. show a top view and a side / iso view of a mixed reality system 1400E operating in a multi-focus mode to present content to a user's eyes 1402. The mixed reality system 1400E can, for example, have the same architecture as the mixed reality systems 1400A and / or 1400C described above with reference to Figures 14A to 14D When operating in the multi-focus mode, the virtual content can be projected onto two or more depth planes 1421-1423 simultaneously at any given point in time to generate two or more different focus cues. In the Figures 14E to 14F example, the virtual content is projected onto depth planes 1421, 1422, and 1423 simultaneously. As shown, the virtual content projected onto the depth planes 1421-1423 is transformed into virtual pixel sets 1431E-1433E.

[0697] Generally, when operating in the multi-focus mode (or when operating in the hybrid zoom mode), the wearable system can provide both hybrid and non-hybrid accommodation cues. As Figure 14FAs shown, depth planes 1421 and 1422 can be configured to use pixels 1431E on dept...

Claims

1. A method implemented by a display system configured to project light into a user's left and right eyes to display virtual image content in the user's field of view, each of the eyes having a cornea, iris, pupil, lens, retina, and an optical axis extending through the lens, the pupil, and the cornea, the method comprising: displaying virtual image content in the user's field of view by the display system such that the displayed virtual image content appears to originate from different depths at different times; determining, by one or more eye tracking cameras configured to image the user's eyes to track eye movement, the position of the center of rotation of the left eye and the position of the center of rotation of the right eye based on an image of the eyes, wherein determining the position of the center of rotation of one of the left eye or the right eye comprises: determining a position array based on a plurality of spatial positions on an image of the left eye or the right eye obtained by the one or more eye tracking cameras, wherein the position array corresponds to at least a portion of an ellipse, and determining a plurality of linear paths extending from positions on a first side of the position array through the position array to an opposite second side of the position array.

2. The method according to claim 1, wherein, displaying virtual image content in the user's field of view by the display system comprises projecting light into the user's eyes to display virtual image content in the user's field of view with different divergence amounts such that the displayed virtual image content appears to originate from different depths.

3. The method according to claim 1, wherein, displaying virtual image content in the user's field of view by the display system comprises projecting a portion of the light into the user's eyes to display diverging virtual image content in the user's field of view such that the displayed virtual image content appears to originate from a first depth, and projecting a portion of the light into the user's eyes to display collimated virtual image content in the user's field of view such that the displayed virtual image content appears to originate from a second depth different from the first depth.

4. The method according to claim 1, wherein, the cornea has an associated corneal sphere having a center of curvature, and wherein the method further comprises determining the position of the center of curvature of the corneal sphere.

5. The method according to claim 1, wherein, the method further comprises determining the position and orientation of the optical axis.

6. The method according to claim 5, wherein, determining the position and orientation of the optical axis is based on the position of the center of the pupil in three-dimensional space relative to the center of curvature of the cornea.

7. The method according to claim 1, wherein, the method further comprises determining the position and orientation of the visual axis offset from the optical axis based on the position and orientation of the optical axis.

8. The method according to claim 1, wherein, The method further includes presenting the virtual image content to the left eye or the right eye of the user, the virtual image content being rendered as if captured by a camera having an aperture at the determined center of rotation position of the left eye or the right eye of the user.

9. The method according to claim 1, wherein, the method further includes obtaining the center of rotation by filtering a plurality of estimated center of rotation positions, averaging the plurality of estimated center of rotation positions, applying a Kalman filter to the plurality of estimated center of rotation positions, or performing any combination of the foregoing items on the plurality of estimated center of rotation positions.

10. The method according to claim 1, wherein, the method further includes using a rendering camera configured to render a virtual image to be presented to the left eye or the right eye, the virtual image being rendered as if captured by a camera having an aperture closer to the center of rotation than the retina of the left eye or the right eye.

Images