Eye rotation center determination, depth plane selection, and rendering camera positioning in display systems

Abstract

A display system may include a head mounted display configured to project light into a user's eyes to display virtual image content with different amounts of divergence and collimation. The display system may include an inward-facing imaging system that images the user's eyes; and processing electronics that communicate with the inward-facing imaging system and are configured to obtain an estimate of the center of rotation of the user's eyes. The display system may render the virtual image content via a rendering camera positioned at the determined center of rotation location of the eyes.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 618,559, filed on January 17, 2018, entitled “EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS.” This application also claims priority to U.S. Provisional Patent Application No. 62 / 702,849, filed on July 24, 2018, entitled “EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING INDISPLAY SYSTEMS.” The entire contents of each of the above applications are incorporated herein by reference.

[0003] This application further incorporates by reference U.S. Patent Publication 2018 / 0018515, entitled “IRIS BOUNDARY ESTIMATION USING CORNEA CURVATURE,” published on January 18, 2018. Technical Field

[0004] The present disclosure relates to display systems, virtual reality and augmented reality imaging and visualization systems, and more particularly to depth plane selection based at least in part on a user's interpupillary distance. Background Art

[0005] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality," "augmented reality," or "mixed reality" experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears to be real or can be perceived as real. Virtual reality or "VR" scenarios generally involve the presentation of digital or virtual image information without being transparent to other actual real-world visual input; augmented reality or "AR" scenarios generally involve the presentation of digital or virtual image information as an enhancement to the visualization of the real world around the user; and mixed reality or "MR" involves the merging of the real and virtual worlds to produce a new environment in which physical and virtual objects coexist and interact in real time. The human visual perception system has proven to be very complex, and it has been challenging to produce VR, AR, or MR technologies that facilitate the comfortable, natural, and rich presentation of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies. Summary of the Invention

[0006] Various examples of depth plane selection in a mixed reality system are disclosed.

[0007] A display system may be configured to project light into a user's eye to display virtual image content in the user's field of view. The user's eye may have a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea. The display system may include: a frame configured to be supported on the user's head; a head-mounted display disposed on the frame, the display configured to project light into the user's eye to display virtual image content to the user's field of view with at least one of different amounts of divergence and collimation, such that the displayed virtual image content appears to originate from different depths at different time periods; one or more eye-tracking cameras configured to image the user's eye; and processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics configured to obtain an estimate of a rotational center of the eye based on images of the eye obtained by the one or more eye-tracking cameras.

[0008] Various examples of display systems that project light into one or more eyes of a user to display virtual image content in the user's field of view are described herein, such as the following examples:

[0009] Example 1: A display system configured to project light into an eye of a user to display virtual image content in the user's field of view, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea, the display system comprising:

[0010] a frame configured to be supported on the user's head;

[0011] a head-mounted display disposed on the frame, the display configured to project light into the user's eyes to display virtual image content to the user's field of view with different amounts of at least one of divergence and collimation, whereby the displayed virtual image content appears to originate from different depths at different time periods;

[0012] one or more eye-tracking cameras configured to image the user's eyes; and

[0013] Processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics configured to obtain an estimate of a center of rotation of the eye based on images of the eye obtained by the one or more eye-tracking cameras.

[0014] Example 2: The display system according to Example 1 further includes one or more light sources, which are arranged on the frame relative to the user's eyes to illuminate the user's eyes, and the one or more eye tracking cameras use the light from the one or more light sources to form an image of the eye.

[0015] Example 3: A display system according to Example 1 or 2, wherein the one or more light sources include at least two light sources, and the at least two light sources are arranged on the frame relative to the user's eyes to illuminate the user's eyes.

[0016] Example 4: A display system according to Example 1 or 3, wherein the one or more light sources include infrared light emitters.

[0017] Example 5: A display system according to any one of Examples 1 to 4, wherein one or more light sources form one or more glints on the eye, and the processing electronic device is configured to determine the position of the cornea based on the one or more glints.

[0018] Example 6: A display system according to any of Examples 1 to 5, wherein the cornea has a corneal sphere associated therewith, the corneal sphere having a center of curvature, and the electronic processing device is configured to determine the position of the center of curvature of the corneal sphere.

[0019] Example 7: A display system according to Example 5, wherein the cornea has a corneal sphere associated therewith, the corneal sphere having a center of curvature, and the processing electronic device is configured to determine the position of the center of curvature of the corneal sphere based on the one or more bright spots.

[0020] Example 8: A display system according to any of the above examples, wherein the one or more eye tracking cameras are configured to image the pupil of the eye.

[0021] Example 9: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the location of the center of the pupil.

[0022] Example 10: The display system of any of the above examples, wherein the processing electronics are configured to determine at least a portion of a boundary between the iris and the pupil.

[0023] Example 11: The display system of Example 10, wherein the processing electronics are configured to determine a center of the boundary between the iris and the pupil.

[0024] Example 12: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the position of the center of the pupil relative to the center of curvature of the cornea in three-dimensional space.

[0025] Example 13: The display system of any of the above examples, wherein the processing electronics are configured to determine the position and orientation of the optical axis.

[0026] Example 14: A display system according to Example 12, wherein the processing electronics are configured to determine the position and orientation of the optical axis based on the position of the center of the pupil in three-dimensional space.

[0027] Example 15: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the position and orientation of the optical axis based on the position of the center of the pupil relative to the center of curvature of the cornea in three-dimensional space.

[0028] Example 16: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the position of the center of rotation of the eye based on the center of curvature of the cornea.

[0029] Example 17: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the position of the center of rotation of the eye based on the center of curvature of the cornea and the position and orientation of the optical axis.

[0030] Example 18: A display system according to Example 17, wherein the processing electronic device is configured to determine the position of the rotation center of the eye by translating a specific distance along the optical axis from the center of curvature of the cornea.

[0031] Example 19: A display system according to Example 18, wherein the specific distance from the center of curvature to the center of rotation is between 4.0 mm and 6.0 mm.

[0032] Example 20: A display system according to Example 18 or 19, wherein the specific distance from the center of curvature to the center of rotation is approximately 4.7 mm.

[0033] Example 21: The display system of Example 18 or 19, wherein the specific distance is fixed.

[0034] Example 22: A display system according to Example 18 or 19, wherein the processing electronic device is configured to determine the specific distance based at least on one or more images of the eye previously obtained by the one or more eye tracking cameras.

[0035] Example 23: The display system of any of the above examples, wherein the processing electronics are configured to determine a position and orientation of a viewing axis offset from the optical axis based on the position and orientation of the optical axis.

[0036] Example 24: The display system of any of the above examples, wherein the processing electronics are configured to determine the position and orientation of the viewing axis based on an angular rotation relative to the optical axis.

[0037] Example 25: The display system of any of the above examples, wherein the processing electronics are configured to determine the position and orientation of the viewing axis based on an angular rotation between 4.0° and 6.5° relative to the optical axis.

[0038] Example 26: The display system of any of the above examples, wherein the processing electronics are configured to determine the position and orientation of the viewing axis based on an angular rotation of approximately 5.2° relative to the optical axis.

[0039] Example 27: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the position and orientation of the visual axis based at least on one or more images of the eye previously obtained by the one or more eye tracking cameras.

[0040] Example 28: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the center of rotation of the eye based on multiple determinations of the position of the optical axis or visual axis over a time period during which the eye rotates.

[0041] Example 29: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the center of rotation by identifying multiple determined intersecting, converging, or adjacent areas of the position of the optical axis or visual axis within a time period of rotation of the eye.

[0042] Example 30: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the vergence distance of the user at which the left eye and right eye of the user are looking based on determining the position and orientation of the optical axes of the left eye and right eye of the user.

[0043] Example 31: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the convergence distance of the user at which the left eye and right eye of the user are looking based on the determination of the position and orientation of the visual axes of the left eye and right eye of the user.

[0044] Example 32: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the convergence distance at which the user's left eye and right eye are looking based on identifying the area where the visual axes of the user's left eye and right eye intersect, converge, or are in close proximity.

[0045] Example 33: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the convergence distance at which the left and right eyes of the user are looking by projecting the visual axes of the left and right eyes of the user onto a horizontal plane and identifying the area where the projections of the visual axes of the left and right eyes on the horizontal plane intersect, converge, or are in close proximity.

[0046] Example 34: The display system of any of the above examples, wherein the processing electronics are configured to determine a relative amount of at least one of divergence and collimation for projecting image content based on the determination of the convergence distance.

[0047] Example 35: The display system of any of the above examples, wherein the processing electronics comprises electronics located on the frame.

[0048] Example 36: The display system of any of the above examples, wherein the processing electronics include electronics located on the frame and electronics located at a location remote from the frame.

[0049] Example 37: The display system of any of the above examples, wherein the processing electronics include electronics located on the frame and electronics located on a belt pack.

[0050] Example 38: A display system according to any of the above examples, wherein at least a portion of the display is transparent and is positioned in front of the user's eyes when the user wears the head-mounted display, so that the transparent portion transmits light from a portion of the user's front environment and the head-mounted display to the user's eyes to provide a view of the portion of the user's front environment and the head-mounted display.

[0051] Example 39: A display system according to any of the above examples, wherein the head-mounted display receives light from a portion of the user's front environment at a first divergence and transmits light from the portion of the user's front environment to the user's eyes at a second divergence that is substantially the same as the first divergence.

[0052] Example 40: A display system according to any of the above examples, wherein the processing electronic device is configured to obtain the estimate of the rotation center by filtering, averaging, applying a Kalman filter, or performing any combination of the above items to multiple estimated rotation center positions.

[0053] Example 41: A display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture located at a determined rotational center position of the user's eyes.

[0054] Example 42: A display system according to any of the above examples, wherein the processing electronic device is configured to render the virtual image to be presented to the eye using a rendering camera located at the rotation center.

[0055] Example 43: A display system according to any of the above examples, wherein the processing electronic device is configured to use a rendering camera, the rendering camera being configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture closer to the center of rotation than the retina of the eye.

[0056] Example 44: A display system according to any of the above examples, wherein the processing electronic device is configured to use a rendering camera, the rendering camera being configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture located at the rotation center of the eye.

[0057] Example 45: A display system according to any of the above examples, wherein the processing electronic device is configured to render a virtual image to be presented to the eye using a rendering camera located at the rotation center, and the rendering camera is modeled to have an aperture located at the rotation center of the eye.

[0058] Example 46: A display system configured to project light into an eye of a user to display virtual image content in the user's field of view, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea, the display system comprising:

[0059] a frame configured to be supported on the user's head;

[0060] a head-mounted display disposed on the frame, the display configured to project light into the user's eyes to display virtual image content to the user's field of view with different amounts of at least one of divergence and collimation, whereby the displayed virtual image content appears to originate from different depths at different time periods;

[0061] one or more eye-tracking cameras configured to image the user's eyes; and

[0062] processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics configured to obtain a position estimate of a center of perspective of the eye based on images of the eye obtained by the one or more eye-tracking cameras, the center of perspective estimated to be proximate to the pupil of the eye or between the cornea of ​​the eye and the pupil,

[0063] The processing electronic device is configured to present the virtual image content to the user's eyes, the virtual image content being rendered by a rendering camera located at the perspective center.

[0064] Example 47: A display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture closer to the perspective center than the retina.

[0065] Example 48: A display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture closer to the perspective center than the rotation center of the eye.

[0066] Example 49: A display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture located at the perspective center.

[0067] Example 50: A display system according to any of the above examples, wherein the perspective center is not located at the pupil of the eye.

[0068] Example 51: A display system according to any of the above examples, wherein the processing electronic device is configured to obtain an estimate of the user's eye posture over time, and wherein the processing electronic device adjusts the position of the rendering camera at least in part based on the user's eye posture.

[0069] Example 52: A display system according to any of the above examples, wherein the processing electronic device is configured to track the user's eye posture over time, and wherein the position of the rendering camera is adjusted over time in response to changes in the user's eye posture over time.

[0070] Example 53: The display system of any of the above examples, wherein the processing electronics are configured to obtain the estimate of the perspective center by filtering a plurality of estimated perspective center positions.

[0071] Example 54: A display system according to any of the above examples, wherein the processing electronic device is configured to obtain the estimate of the perspective center by averaging multiple estimated perspective center positions and / or applying a Kalman filter.

[0072] Example 55: A display system according to any of the above examples, wherein the perspective center includes a location within the anterior chamber of the user's eye.

[0073] Example 56: A display system according to any of the above examples, wherein the perspective center includes a position in front of the pupil of the user's eye.

[0074] Example 57: A display system according to any of the above examples, wherein the perspective center includes a position between 1.0 mm and 2.0 mm in front of the pupil of the user's eye.

[0075] Example 58: A display system according to any of the above examples, wherein the perspective center includes a position approximately 1.0 mm in front of the pupil of the user's eye.

[0076] Example 59: A display system according to any of the above examples, wherein the perspective center includes a position between 0.25 mm and 1.0 mm in front of the pupil of the user's eye.

[0077] Example 60: A display system according to any of the above examples, wherein the perspective center includes a position between 0.5 mm and 1.0 mm in front of the pupil of the user's eye.

[0078] Example 61: A display system according to any of the above examples, wherein the perspective center includes a position between 0.25 mm and 0.5 mm in front of the pupil of the user's eye.

[0079] Example 62: A display system according to any of the above examples, wherein the perspective center is positioned along the optical axis of the eye, and wherein the processing electronic device is further configured to obtain a position estimate of the perspective center by obtaining a position estimate of the optical axis of the eye.

[0080] Example 63: A display system according to any of the above examples, wherein the perspective center is located at a position along the optical axis of the eye between the outer surface of the cornea of ​​the eye and the pupil, and wherein the processing electronic device is further configured to obtain a position estimate of the perspective center by obtaining a position estimate of the optical axis of the eye.

[0081] Example 64: A display system according to any of the above examples, wherein the perspective center is located at a position along the optical axis of the eye between the outer surface of the cornea of ​​the eye and the pupil, and wherein the processing electronic device is further configured to obtain a position estimate of the perspective center by obtaining a position estimate of the optical axis of the eye and a position estimate of the rotation center of the eye, the cornea of ​​the eye, the iris of the eye, the retina of the eye, and the pupil of the eye, or any combination thereof.

[0082] Example 65: A display system according to any of the above examples, wherein the processing electronics include electronics located on the frame.

[0083] Example 66: The display system of any of the above examples, wherein the processing electronics include electronics located on the frame and electronics located at a location remote from the frame.

[0084] Example 67: The display system of any of the above examples, wherein the processing electronics include electronics located on the frame and electronics located on the waist pack.

[0085] Example 68: A display system according to any of the above examples, wherein at least a portion of the display is transparent and is positioned in front of the user's eyes when the user wears the head-mounted display, so that the transparent portion transmits light from a portion of the user's front environment and the head-mounted display to the user's eyes to provide a view of the portion of the user's front environment and the head-mounted display.

[0086] Example 69: The display system according to any of the above examples further includes one or more light sources, which are arranged on the frame relative to the user's eyes to illuminate the user's eyes, and the one or more eye tracking cameras use the light from the one or more light sources to capture an image of the eye.

[0087] Example 70: A display system according to any of the above examples, wherein the one or more light sources include at least two light sources, and the at least two light sources are arranged on the frame relative to the user's eyes to illuminate the user's eyes.

[0088] Example 71: A display system according to any of the above examples, wherein the one or more light sources include at least three light sources, and the at least three light sources are arranged on the frame relative to the user's eyes to illuminate the user's eyes.

[0089] Example 72: A display system according to any of the above examples, wherein the one or more light sources include infrared light emitters.

[0090] Example 73: A display system according to any of the above examples, wherein the one or more light sources form one or more bright spots on the eye, and the processing electronic device is configured to determine the position of the center of curvature of the cornea based on the one or more bright spots.

[0091] Example 74: A display system according to any of the above examples, wherein the one or more light sources form one or more bright spots on the eye, and the processing electronic device is configured to determine the three-dimensional position of the center of curvature of the cornea based on the one or more bright spots.

[0092] Example 75: A display system according to any of the above examples, wherein the one or more eye tracking cameras are further configured to image the pupil of the user's eye, and wherein the processing electronic device is further configured to determine the position of the pupil of the eye based at least on the image of the pupil from the one or more eye tracking cameras.

[0093] Example 76: A display system according to any of the above examples, wherein the one or more eye tracking cameras are further configured to image the pupil of the user's eye, and wherein the processing electronic device is further configured to determine the three-dimensional position of the pupil of the eye based at least on the image of the pupil from the one or more eye tracking cameras.

[0094] Example 77: A display system according to any of the above examples, wherein the one or more eye tracking cameras are further configured to image the pupil of the user's eye, and wherein the processing electronic device is further configured to determine the position of the pupil of the eye based on the position of the center of curvature of the cornea and based on the image of the pupil from the one or more eye tracking cameras.

[0095] Example 78: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the optical axis of the eye based on the three-dimensional position of the center of curvature of the cornea and based on the three-dimensional position of the pupil.

[0096] Example 79: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the visual axis of the eye based on the optical axis.

[0097] Example 80: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the visual axis of the eye based on the three-dimensional position of at least one of the center of curvature of the cornea and the pupil, or both the center of curvature of the cornea and the pupil, and the optical axis.

[0098] Example 81: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the three-dimensional position of the center of rotation of the eye based on the three-dimensional position of the center of curvature of the cornea.

[0099] Example 82: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the three-dimensional position of the center of rotation of the eye based on the three-dimensional position of the center of curvature of the cornea and based on the optical axis.

[0100] Example 83: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the distance between the eye and the user's other eye based at least on the three-dimensional position of the rotation center of the eye.

[0101] Example 84: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the pupil distance between the eye and the user's other eye based at least on the three-dimensional position of the rotation center of the eye.

[0102] Example 85: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the convergence distance of the user based at least on the optical axis of the eye.

[0103] Example 86: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the convergence distance of the user based at least on the optical axis of the eye and based on the determined optical axis of the user's other eye.

[0104] Example 87: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the convergence distance of the user based at least on the visual axis of the eye and based on the determined visual axis of the user's other eye.

[0105] Example 88: A display system according to any of the above examples, wherein the display is configured to project collimated light into the user's eyes.

[0106] Example 89: A display system according to any of the above examples, wherein the display is configured to project collimated light corresponding to the image pixels into the user's eyes in a first time period, and to project divergent light corresponding to the image pixels into the user's eyes in a second time period.

[0107] Example 90: A display system according to any of the above examples, wherein the display is configured to project light having a first divergence corresponding to image pixels into the user's eyes in a first time period, and to project light having a second divergence corresponding to the image pixels that is greater than the first divergence into the user's eyes in a second time period.

[0108] Example 91: A method of rendering virtual image content in a display system, the display system being configured to project light into an eye of a user to display the virtual image content in the user's field of view, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea, the method comprising:

[0109] determining, with one or more eye-tracking cameras configured to image the eyes of the user to track movement of the eyes, a position of a center of rotation of the eyes;

[0110] rendering, by a rendering engine, virtual image content using a rendering camera located at the rotation center of the eye, the rendering camera being configured to render a virtual image to be presented to the eye; and

[0111] Light is projected into the user's eyes through a head-mounted display, and the rendered virtual image content is displayed to the user's field of view with different divergences, so that the virtual image content appears to originate from different depths at different time periods.

[0112] Example 92: A method according to any of the above examples, wherein the rendering camera is configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture closer to the center of rotation than the retina of the eye.

[0113] Example 93: A method according to any of the above examples, wherein the rendering camera is configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture located at the center of rotation.

[0114] Example 94: A method according to any of the above examples, wherein the rendering camera is modeled to have an aperture located at the rotation center of the eye.

[0115] Example 95: The method of any of the preceding examples, wherein the rendering camera is modeled as having an aperture, a lens, and a detector.

[0116] Example 96: A method according to any of the above examples, wherein the rendering camera has an aperture located at a position along a line between (i) the determined center of rotation position of the eye and (ii) the determined position of at least one of the iris or pupil.

[0117] Example 97: The method according to any of the above examples further comprising:

[0118] determining, by the one or more eye-tracking cameras, a location of a perspective center of the user's eye, wherein the perspective center of the user's eye is located less than approximately 1.0 mm from the pupil of the user's eye; and

[0119] Rendering the virtual image content using the rendering engine and the rendering camera,

[0120] The rendering camera has an aperture located at the determined perspective center position of the user's eye.

[0121] Example 98: The method according to any of the above examples further comprising:

[0122] The virtual image content is rendered, by the rendering engine, using the rendering camera, wherein the rendering camera has an aperture located at a position along a line between (i) the determined rotational center position of the eye and (ii) the determined perspective center position of the user's eye.

[0123] Example 99: The method according to any of the above examples further comprises:

[0124] determining, by processing electronics in communication with the one or more eye-tracking cameras, a measure of change over time in the determined perspective center position of the user's eyes; and

[0125] If it is determined that the time-varying metric exceeds a first threshold, the rendering engine is instructed, by the processing electronic device, to render the virtual content using the rendering camera, wherein the rendering camera has an aperture located at the determined rotation center position of the eye.

[0126] Example 100: The method according to any of the above examples further includes:

[0127] If it is determined that the metric of change over time is below a second threshold, the rendering engine is instructed, by the processing electronic device, to render the virtual content using the rendering camera, wherein the rendering camera has an aperture located at the determined perspective center position of the eye, and wherein the first threshold indicates a higher level of change over time in the determined perspective center position of the user's eye than the second threshold.

[0128] Example 101: The method according to any of the above examples further includes:

[0129] If it is determined that the time-varying metric is below a second threshold, the rendering engine is instructed, by the processing electronic device, to render the virtual content using the rendering camera, wherein the rendering camera has an aperture located at the determined perspective center position of the eye.

[0130] Example 102: The method according to any of the above examples further includes:

[0131] If it is determined, by the processing electronic device, that the time-varying metric is between the first threshold and the second threshold, the rendering engine is instructed to render the virtual content using the rendering camera, wherein the rendering camera has an aperture located at a point along a line between (i) the determined rotational center position of the eye and (ii) the determined perspective center position of the eye.

[0132] Example 103: The method according to any of the above examples further includes:

[0133] and transmitting light from a portion of the environment in front of the user and the head-mounted display to the user's eyes through at least a portion of the display, the portion being transparent and being positioned in front of the user's eyes when the user is wearing the head-mounted display, to provide a view of the portion of the environment in front of the user and the head-mounted display.

[0134] Example 104: The method according to any of the above examples further includes:

[0135] A position of at least one of the iris, pupil, or lens is determined by the one or more eye tracking cameras.

[0136] Example 105: The method according to any of the above examples further includes:

[0137] The virtual image content is rendered, by the rendering engine, using the rendering camera, the rendering camera being configured to present a virtual image to the eye, the virtual image being rendered as if captured by a camera having an aperture located at a position along a line between (i) the determined center of rotation position of the eye and (ii) the determined position of at least one of the iris or pupil.

[0138] Example 106: The method according to any of the above examples further includes:

[0139] determining, by the one or more eye-tracking cameras, a location of a perspective center of the user's eye, wherein the perspective center of the user's eye is located less than approximately 1.0 mm from the pupil of the user's eye; and

[0140] The virtual image content is rendered, by the rendering engine, using the rendering camera, the rendering camera being configured to present a virtual image to the eye, the virtual image being rendered as if captured by a camera having an aperture located at the determined perspective center position of the user's eye.

[0141] Example 107: The method according to any of the above examples further comprises:

[0142] The virtual image content is rendered, by the rendering engine, using the rendering camera, the rendering camera being configured to present a virtual image to the eye, the virtual image being rendered as if captured by a camera having an aperture located at a position along a line between (i) the determined rotational center position of the eye and (ii) the determined perspective center position of the user's eye.

[0143] Example 108: The method according to any of the above examples further includes:

[0144] determining, by processing electronics in communication with the one or more eye-tracking cameras, a measure of change over time in the determined perspective center position of the user's eyes; and

[0145] If it is determined, by the processing electronic device, that the time-varying metric exceeds a first threshold, the rendering engine is instructed to render the virtual content using the rendering camera as if captured by a camera having an aperture located at the determined rotational center position of the eye.

[0146] Example 109: The method according to any of the above examples further includes:

[0147] The processing electronic device instructs the rendering engine to render the virtual content using the rendering camera as if captured by a camera having an aperture located at the determined perspective center position of the eye if it is determined that the metric of change over time is below a second threshold, wherein the first threshold indicates a higher level of change over time in the determined perspective center position of the user's eye than the second threshold.

[0148] Example 110: The method according to any of the above examples further comprises:

[0149] If it is determined, by the processing electronic device, that the time-varying metric is below a second threshold, the rendering engine is instructed to render the virtual content using the rendering camera as if captured by a camera having an aperture located at the determined perspective center position of the eye.

[0150] Example 111: The method according to any of the above examples further includes:

[0151] If it is determined, by the processing electronic device, that the time-varying metric is between the first threshold and the second threshold, the rendering engine is instructed to render the virtual content using the rendering camera as if captured by a camera having an aperture located at a point along a line between (i) the determined rotational center position of the eye and (ii) the determined perspective center position of the eye.

[0152] Example 112: A display system configured to project light into an eye of a user to display virtual image content in the user's field of view, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea, the display system comprising:

[0153] a frame configured to be supported on the user's head;

[0154] a head-mounted display disposed on the frame, the display configured to project light into the user's eyes to display virtual image content to the user's field of view with different divergences, whereby the displayed virtual image content appears to originate from different depths at different time periods, wherein the head-mounted display is configured to project light having a first divergence into the user's eyes during a first time period and to project light having a second divergence into the user's eyes during a second time period, wherein the first divergence is different from the second divergence;

[0155] one or more eye-tracking cameras configured to image the user's eyes; and

[0156] and processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics configured to obtain an estimate of a center of rotation of the eye based on images of the eye obtained by the one or more eye-tracking cameras, obtain an estimate of a vergence distance of the user based on images of the eye obtained by the one or more eye-tracking cameras, and transition from projecting light into the user's eye at the first divergence to projecting light into the user's eye at the second divergence based on the user's estimated vergence distance.

[0157] Example 113: A display system according to any of the above examples, wherein at least a portion of the display is transparent and is positioned in front of the user's eyes when the user wears the head-mounted display, so that the transparent portion transmits light from a portion of the user's front environment and the head-mounted display to the user's eyes to provide a view of the portion of the user's front environment and the head-mounted display.

[0158] Example 114: A display system according to any of the above examples, wherein the processing electronic device is further configured to detect a blink of the eye based on an image of the eye obtained using the one or more eye tracking cameras.

[0159] Example 115: A display system according to any of the above examples, wherein the processing electronic device is further configured to detect a saccade of the eye based on an image of the eye obtained using the one or more eye tracking cameras.

[0160] Example 116: A display system according to any of the above examples, wherein the processing electronic device is configured to change from projecting light into the user's eyes with the first divergence to projecting light into the user's eyes with the second divergence based on the user's determined convergence distance and based on whether the processing electronic device detects a blink of the eye.

[0161] Example 117: A display system according to any of the above examples, wherein the processing electronic device is configured to change from projecting light into the user's eyes with the first divergence to projecting light into the user's eyes with the second divergence based on the user's determined convergence distance and based on whether the processing electronic device detects a glance of the eye.

[0162] Example 118: A display system according to any of the above examples, wherein the processing electronic device is configured to change from projecting light into the user's eyes with the first divergence to projecting light into the user's eyes with the second divergence based on the user's determined convergence distance and based on whether the processing electronic device detects at least one of a glance or a blink of the eye.

[0163] Example 119: The display system of any of the above examples, wherein the first divergence is associated with a convergence distance within a first range, and wherein the second divergence is associated with a convergence distance within a second range.

[0164] Example 120: A display system according to any of the above examples, wherein the first divergence is associated with a convergence distance within a first range, wherein the second divergence is associated with a convergence distance within a second range, and wherein the first range and the second range overlap but are not equal.

[0165] Example 121: A display system according to any of the above examples, wherein the processing electronic device is configured to change from projecting light into the user's eyes with the first divergence to projecting light into the user's eyes with the second divergence when it is determined that the convergence distance of the user is outside the first range and within the second range.

[0166] Example 122: A display system according to any of the above examples, wherein the processing electronic device is configured to change from projecting light into the user's eyes with the first divergence to projecting light into the user's eyes with the second divergence when it is determined that the convergence distance of the user is outside the second range and within the first range.

[0167] Example 123: A display system according to any of the above examples, wherein the processing electronic device is configured to change from projecting light into the user's eyes with the first divergence to projecting light into the user's eyes with the second divergence when it is determined that the convergence distance of the user is outside the first range and within the second range and a blink of the eye is detected.

[0168] Example 124: A display system according to any of the above examples, wherein the processing electronic device is configured to change from projecting light into the user's eyes with the first divergence to projecting light into the user's eyes with the second divergence when it is determined that the convergence distance of the user is outside the first range and within the second range and a glance of the eyes is detected.

[0169] Example 125: A display system according to any of the above examples, wherein the processing electronic device is configured to change from projecting light into the user's eyes with the first divergence to projecting light into the user's eyes with the second divergence when it is determined that the convergence distance of the user is outside the first range and within the second range for a time longer than a predetermined time period.

[0170] Example 126: A display system according to any of the above examples, wherein the processing electronic device is configured to switch from projecting light into the user's eyes with the first divergence to projecting light into the user's eyes with the second divergence when it is determined that the convergence distance of the user is outside the first range and within the second range for a predetermined time period longer than at least 10 seconds.

[0171] Example 127: A display system according to any of the above examples, wherein the head-mounted display includes a first display element configured to project light having the first divergence and a second display element configured to project light having the second divergence.

[0172] Example 128: A display system according to any of the above examples, wherein the display is configured to project light into the user's eyes to display virtual image content in a discrete display mode, and in the discrete display mode, the display is configured to project light associated with multiple consecutive frames using only one of the first display elements.

[0173] Example 129: A display system according to any of the above examples, wherein the display is configured to project light into the user's eyes to display virtual image content in a hybrid display mode, in which the display is configured to project light associated with the multiple consecutive frames using both the first display element and the second display element for each frame of the multiple consecutive frames.

[0174] Example 130: A display system according to any of the above examples, wherein the display is configured to project light into the eyes of the user to display virtual image content in a hybrid display mode, wherein the display is configured to project light associated with the multiple consecutive frames using both the first display element and the second display element for each frame of the multiple consecutive frames, and wherein in the hybrid display mode, the display is configured to project light using the first display element and the second display element, and the light is perceived by the user as having a given divergence between the first divergence and the second divergence.

[0175] Example 131: A display system according to any of the above examples, wherein the display is configured to project light into the eyes of the user to display virtual image content in a multi-focus display mode, wherein the display is configured to project light associated with the multiple consecutive frames using both the first display element and the second display element for each frame of the multiple consecutive frames, wherein in the multi-focus display mode, the display is configured to project light associated with the first virtual image content with a third divergence and light associated with the second virtual image content with a fourth divergence, and wherein the third divergence is different from the fourth divergence.

[0176] Example 132: The display system of any of the above examples, wherein the third divergence and the fourth divergence are each between the first divergence and the second divergence.

[0177] Example 133: The display system of any of the above examples, wherein at least one of the third divergence and the fourth divergence is between the first divergence and the second divergence.

[0178] Example 134: The display system of any of the above examples, wherein the third divergence and the fourth divergence are equal to the first divergence and the second divergence, respectively.

[0179] Example 135: A display system according to any of the above examples, wherein the display is configured to project light associated with the first virtual image in a first area of ​​the user's field of view and to project light associated with the second virtual image in a second area of ​​the user's field of view, and wherein the first area and the second area are different.

[0180] Example 136: A display system according to any of the above examples, wherein the display is configured to project light associated with the first virtual image in a first area of ​​the user's field of view and to project light associated with the second virtual image in a second area of ​​the user's field of view, and wherein the first area and the second area do not overlap.

[0181] Example 137: A display system configured to project light into a left eye and a right eye of a user to display virtual image content in the user's field of view, each of the eyes having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea, the display system comprising:

[0182] a frame configured to be supported on the user's head;

[0183] a head-mounted display disposed on the frame, the display configured to project light into the user's left and right eyes to display virtual image content to the user's field of view with different amounts of at least one of divergence and collimation, whereby the displayed virtual image content appears to originate from different distances relative to the user's left and right eyes at different time periods;

[0184] a first eye-tracking camera configured to image a left eye of the user;

[0185] a second eye-tracking camera configured to image a right eye of the user; and

[0186] processing electronics in communication with the display and the first and second eye-tracking cameras, the processing electronics configured to obtain an estimate of the pupil distance between the user's left and right eyes based on images of the left and right eyes obtained by the first and second eye-tracking cameras.

[0187] Example 138: The display system according to any of the above examples further includes one or more light sources, which are arranged on the frame relative to the user's eyes to illuminate the user's eyes, and the one or more eye tracking cameras use the light from the one or more light sources to form an image of the eye.

[0188] Example 139: A display system according to any of the above examples, wherein the one or more light sources include at least two light sources, and the at least two light sources are arranged on the frame relative to the user's eyes to illuminate the user's eyes.

[0189] Example 140: A display system according to any of the above examples, wherein the one or more light sources include infrared light emitters.

[0190] Example 141: A display system according to any of the above examples, wherein the one or more light sources form one or more bright spots on the eye, and the processing electronic device is configured to determine the position of the cornea based on the one or more bright spots.

[0191] Example 142: A display system according to any of the above examples, wherein the cornea has a corneal sphere associated therewith, the corneal sphere having a center of curvature, and the electronic processing device is configured to determine the position of the center of curvature of the corneal sphere.

[0192] Example 143: A display system according to any of the above examples, wherein the cornea has a corneal sphere associated therewith, the corneal sphere having a center of curvature, and the processing electronic device is configured to determine the position of the center of curvature of the corneal sphere based on the one or more bright spots.

[0193] Example 144: A display system according to any of the above examples, wherein the one or more eye tracking cameras are configured to image the pupil of the eye.

[0194] Example 145: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the position of the center of the pupil.

[0195] Example 146: A display system according to any of the above examples, wherein the processing electronic device is configured to determine at least a portion of a boundary between the iris and the pupil.

[0196] Example 147: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the center of the boundary between the iris and the pupil.

[0197] Example 148: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the position of the center of the pupil relative to the center of curvature of the cornea in three-dimensional space.

[0198] Example 149: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the position and orientation of the optical axis.

[0199] Example 150: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the position and orientation of the optical axis based on the position of the center of the pupil in three-dimensional space.

[0200] Example 151: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the position and orientation of the optical axis based on the position of the center of the pupil relative to the center of curvature of the cornea in three-dimensional space.

[0201] Example 152: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the position of the center of rotation of the eye based on the center of curvature of the cornea.

[0202] Example 153: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the position of the center of rotation of the eye based on the center of curvature of the cornea and the position and orientation of the optical axis.

[0203] Example 154: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the position of the rotation center of the eye by translating a specific distance along the optical axis from the center of curvature of the cornea.

[0204] Example 155: A method of rendering virtual image content in a display system, the display system being configured to project light into a left eye and a right eye of a user to display the virtual image content in the user's field of view, each of the eyes having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea, the method comprising:

[0205] determining, with one or more eye-tracking cameras configured to image the eyes of the user to track movement of the eyes, a position of a center of rotation of the left eye and a position of a center of rotation of the right eye;

[0206] estimating, by processing electronics in communication with the one or more eye-tracking cameras, an interpupillary distance of the user based on the determined positions of the centers of rotation of the left eye and the right eye;

[0207] determining, by the one or more eye-tracking cameras, a current left eye pose and a current right eye pose; and

[0208] The processing electronic device estimates a current vergence distance of the user by comparing the estimated pupil distance with the determined current left eye posture and the determined current right eye posture.

[0209] Example 156: A method according to any of the above examples, wherein determining the current left eye posture and the current right eye posture includes: estimating the position of the pupil of the user's left eye and the position of the pupil of the user's right eye through the one or more eye tracking cameras.

[0210] Example 157: A method according to any of the above examples, wherein determining the current left eye posture and the current right eye posture includes: estimating the position of the cornea of ​​the user's left eye and the position of the cornea of ​​the user's right eye through the one or more eye tracking cameras.

[0211] Example 158: A method according to any of the above examples, wherein determining the current left eye posture and the current right eye posture includes: estimating the position of the iris of the user's left eye and the position of the iris of the user's right eye through the one or more eye tracking cameras.

[0212] Example 159: A method according to any of the above examples, wherein determining the current left eye posture and the current right eye posture includes: estimating the position of the lens of the user's left eye and the position of the lens of the user's right eye through the one or more eye tracking cameras.

[0213] Example 160: The method of any of the above examples, wherein estimating the current convergence distance of the user comprises:

[0214] estimating, by processing electronic equipment, a distance between the positions of the irises of the left and right eyes of the user; and

[0215] The current vergence distance of the user is estimated, by the processing electronics, based on a comparison of the estimated pupil distance and the estimated distance between the positions of the irises of the left and right eyes of the user.

[0216] Example 161: The method according to any of the above examples further includes: projecting light into the eyes of the user through a head-mounted display, displaying the rendered virtual image content to the user's field of view with different divergence amounts, so that the virtual image content appears to originate from different depths at different time periods.

[0217] Example 162: The method according to any of the above examples further includes: transmitting light from a portion of the user's front environment and the head-mounted display to the user's eyes through at least a portion of the display, which is transparent and is positioned in front of the user's eyes when the user wears the head-mounted display, to provide a view of the portion of the user's front environment and the head-mounted display.

[0218] Example 163: A display system configured to project light into an eye of a user to display virtual image content in the user's field of view, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea, the display system comprising:

[0219] a frame configured to be supported on the user's head;

[0220] a head-mounted display disposed on the frame, the display configured to project light into the user's eyes to display virtual image content to the user's field of view with different amounts of at least one of divergence and collimation, whereby the displayed virtual image content appears to originate from different depths at different time periods;

[0221] one or more eye-tracking cameras configured to image the user's eyes; and

[0222] processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics configured to obtain an estimate of the position of the center of rotation of the eye based on images of the eye obtained by the one or more eye-tracking cameras, and to obtain an estimate of the direction of the optical axis of the eye based on the images,

[0223] wherein the processing electronic device is configured to present the virtual image content to the user's eye, the virtual image content being rendered as if captured by a camera having an aperture disposed along the optical axis and spaced 6.0 mm to 13.0 mm from the estimated position of the center of rotation of the eye in a direction away from the retina.

[0224] Example 164: A display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eye, and the virtual image content is rendered as if captured by a camera having an aperture, the aperture being arranged along the optical axis and spaced 7.0 mm to 12.0 mm from the estimated position of the rotation center of the eye in a direction away from the retina.

[0225] Example 165: A display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eye, and the virtual image content is rendered as if captured by a camera having an aperture, the aperture being arranged along the optical axis and spaced 8.0 mm to 11.0 mm from the estimated position of the rotation center of the eye in a direction away from the retina.

[0226] Example 166: A display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eye, and the virtual image content is rendered as if captured by a camera having an aperture, the aperture being arranged along the optical axis and spaced 9.0 mm to 10.0 mm from the estimated position of the rotation center of the eye in a direction away from the retina.

[0227] Example 167: A display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eye, and the virtual image content is rendered as if captured by a camera having an aperture, the aperture being arranged along the optical axis and spaced 9.5 mm to 10.0 mm from the estimated position of the rotation center of the eye in a direction away from the retina.

[0228] Example 168: A display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture, the aperture being arranged along the optical axis and spaced approximately 9.7 mm from the estimated position of the rotation center of the eye.

[0229] Example 169: The display system of any of the above examples, wherein the processing electronics comprises electronics located on the frame.

[0230] Example 170: The display system of any of the preceding examples, wherein the processing electronics include electronics located on the frame and electronics disposed at a location remote from the frame.

[0231] Example 171: The display system of any of the above examples, wherein the processing electronics include electronics located on the frame and electronics located on the waist pack.

[0232] Example 172: A display system according to any of the above examples, wherein at least a portion of the display is transparent and is positioned in front of the user's eyes when the user wears the head-mounted display, so that the transparent portion transmits light from a portion of the user's front environment and the head-mounted display to the user's eyes to provide a view of the portion of the user's front environment and the head-mounted display.

[0233] Example 173: The display system according to any of the above examples further includes one or more light sources, which are arranged on the frame relative to the user's eyes to illuminate the user's eyes, and the one or more eye tracking cameras use the light from the one or more light sources to capture an image of the eye.

[0234] Example 174: A display system according to any of the above examples, wherein the one or more light sources include at least two light sources, and the at least two light sources are arranged on the frame relative to the user's eyes to illuminate the user's eyes.

[0235] Example 175: A display system according to any of the above examples, wherein the one or more light sources include at least three light sources, and the at least three light sources are arranged on the frame relative to the user's eyes to illuminate the user's eyes.

[0236] Example 176: A display system according to any of the above examples, wherein the one or more light sources include infrared light emitters.

[0237] Example 177: A display system configured to project light into an eye of a user to display virtual image content in the user's field of view, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea, the display system comprising:

[0238] a frame configured to be supported on the user's head;

[0239] a head-mounted display disposed on the frame, the display configured to project light into the user's eyes to display virtual image content to the user's field of view with different amounts of at least one of divergence and collimation, whereby the displayed virtual image content appears to originate from different depths at different time periods;

[0240] one or more eye-tracking cameras configured to image the user's eyes; and

[0241] processing electronics in communication with the display and the one or more eye-tracking cameras,

[0242] The processing electronic device is configured to present the virtual image content to the user's eye, the virtual image content being rendered by a rendering camera located at the pupil of the eye or between the pupil and the cornea of ​​the eye.

[0243] Example 178: A display system according to any of the above examples, wherein the rendering camera is located between 1.0 mm and 2.0 mm in front of the pupil of the user's eye.

[0244] Example 179: A display system according to any of the above examples, wherein the rendering camera is located approximately 1.0 mm in front of the pupil of the user's eye.

[0245] Example 180: A display system according to any of the above examples, wherein the rendering camera is located between 0.25 mm and 1.0 mm in front of the pupil of the user's eye.

[0246] Example 181: A display system according to any of the above examples, wherein the rendering camera is located between 0.5 mm and 1.0 mm in front of the pupil of the user's eye.

[0247] Example 182: A display system according to any of the above examples, wherein the rendering camera is located between 0.25 mm and 0.5 mm in front of the pupil of the user's eye.

[0248] Example 183: A display system according to any of the above examples, wherein the rendering camera is located at the pupil of the eye.

[0249] Example 184: A display system according to any of the above examples, wherein the rendering camera is not located at the pupil of the eye.

[0250] Example 185: A display system according to any of the above examples, wherein the rendering camera comprises a pinhole camera.

[0251] Example 186: A display system according to any of the above examples, wherein the aperture comprises a pinhole of a pinhole camera.

[0252] Example 187: A method of rendering virtual image content in a display system, the display system being configured to project light into an eye of a user to display the virtual image content in the user's field of view, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea, the method comprising:

[0253] determining, by one or more eye cameras configured to image the eye of the user, a position based on the imaging of the eye achieved with the one or more cameras;

[0254] rendering, by a rendering engine, virtual image content using a rendering camera located at a position based on the determined position, the rendering camera being configured to render a virtual image to be presented to the eye; and

[0255] Light is projected into the user's eyes through a head-mounted display to display the rendered virtual image content to the user's field of view.

[0256] Example 188: A method according to any of the above examples, wherein the position is the center of rotation of the eye.

[0257] Example 189: A method according to any of the above examples, wherein the position of the rendering camera is located at the rotation center of the eye.

[0258] Example 190: A method according to any of the above examples, wherein the position is the perspective center of the eye.

[0259] Example 191: A method according to any of the above examples, wherein the position of the rendering camera is at the perspective center of the eye.

[0260] Example 192: A method according to any of the above examples, wherein the rendering camera is configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture closer to the center of rotation than the retina of the eye.

[0261] Example 193: A method according to any of the above examples, wherein the rendering camera is configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture located at the center of rotation.

[0262] Example 194: The method of any preceding example, wherein the rendering camera is modeled to have an aperture located at the center of rotation of the eye.

[0263] Example 195: The method of any of the preceding examples, wherein the rendering camera is modeled as having an aperture, a lens, and a detector.

[0264] Example 196: A method according to any of the above examples, wherein the rendering camera has an aperture located at a position along a line between (i) the determined center of rotation position of the eye and (ii) the determined position of at least one of the iris or pupil.

[0265] Example 197: The method according to any of the above examples further comprising:

[0266] determining, by the one or more cameras, a location of a perspective center of the user's eye, wherein the perspective center of the user's eye is located less than approximately 1.0 mm from the pupil of the user's eye; and

[0267] Rendering the virtual image content using the rendering engine and the rendering camera,

[0268] The rendering camera has an aperture located at the determined perspective center position of the user's eye.

[0269] Example 198: The method according to any of the above examples further comprising:

[0270] The virtual image content is rendered, by the rendering engine, using the rendering camera, wherein the rendering camera has an aperture located at a position along a line between (i) the determined rotational center position of the eye and (ii) the determined perspective center position of the user's eye.

[0271] Example 199: The method according to any of the above examples further comprises:

[0272] determining, by processing electronics in communication with the one or more cameras, a measure of change over time in the determined perspective center position of the user's eyes; and

[0273] If it is determined that the time-varying metric exceeds a first threshold, the rendering engine is instructed, by the processing electronic device, to render the virtual content using the rendering camera, wherein the rendering camera has an aperture located at the determined rotation center position of the eye.

[0274] Example 200: The method according to any of the above examples further comprising:

[0275] If it is determined that the metric of change over time is below a second threshold, the rendering engine is instructed, by the processing electronic device, to render the virtual content using the rendering camera, wherein the rendering camera has an aperture located at the determined perspective center position of the eye, and wherein the first threshold indicates a higher level of change over time in the determined perspective center position of the user's eye than the second threshold.

[0276] Example 201: The method according to any of the above examples further includes:

[0277] If it is determined that the time-varying metric is below a second threshold, the rendering engine is instructed, by the processing electronic device, to render the virtual content using the rendering camera, wherein the rendering camera has an aperture located at the determined perspective center position of the eye.

[0278] Example 202: The method according to any of the above examples further comprising:

[0279] If it is determined, by the processing electronic device, that the time-varying metric is between the first threshold and the second threshold, the rendering engine is instructed to render the virtual content using the rendering camera, wherein the rendering camera has an aperture located at a point along a line between (i) the determined rotational center position of the eye and (ii) the determined perspective center position of the eye.

[0280] Example 203: The method according to any of the above examples further includes:

[0281] and transmitting light from a portion of the environment in front of the user and the head-mounted display to the user's eyes through at least a portion of the display, the portion being transparent and positioned in front of the user's eyes when the user is wearing the head-mounted display, to provide a view of the portion of the environment in front of the user and the head-mounted display.

[0282] Example 204: The method according to any of the above examples further comprising:

[0283] The position of at least one of the iris, pupil, or lens is determined by the one or more cameras.

[0284] Example 205: The method according to any of the above examples further comprising:

[0285] The virtual image content is rendered, by the rendering engine, using the rendering camera, the rendering camera being configured to present a virtual image to the eye, the virtual image being rendered as if captured by a camera having an aperture located at a position along a line between (i) the determined center of rotation position of the eye and (ii) the determined position of at least one of the iris or pupil.

[0286] Example 206: The method according to any of the above examples further comprising:

[0287] determining, by the one or more cameras, a location of a perspective center of the user's eye, wherein the perspective center of the user's eye is located less than approximately 1.0 mm from the pupil of the user's eye; and

[0288] The virtual image content is rendered, by the rendering engine, using the rendering camera, the rendering camera being configured to present a virtual image to the eye, the virtual image being rendered as if captured by a camera having an aperture located at the determined perspective center position of the user's eye.

[0289] Example 207: The method according to any of the above examples further comprising:

[0290] The virtual image content is rendered, by the rendering engine, using the rendering camera, the rendering camera being configured to present a virtual image to the eye, the virtual image being rendered as if captured by a camera having an aperture located at a position along a line between (i) the determined rotational center position of the eye and (ii) the determined perspective center position of the user's eye.

[0291] Example 208: A method according to any of the above examples, wherein projecting light into the user's eyes through the head-mounted display to display the rendered virtual image content to the user's field of view includes: projecting light into the user's eyes to display the rendered virtual image content to the user's field of view with different divergences, so that the virtual image content appears to originate from different depths at different time periods.

[0292] Example 209: The method of any of the above examples, wherein the different amounts of divergence include zero divergence.

[0293] Example 210: The method of any of the above examples, wherein the different amounts of divergence include collimation.

[0294] Example 211: A display system configured to project light into an eye of a user to display virtual image content in the user's field of view, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea, the display system comprising:

[0295] a frame configured to be supported on the user's head;

[0296] a head-mounted display disposed on the frame, the display configured to project light into the user's eyes to display virtual image content to the user's field of view;

[0297] one or more cameras configured to image the user's eyes; and

[0298] processing electronics in communication with the display and the one or more cameras, the processing electronics configured to obtain a position of the eye based on images of the eye obtained by the one or more cameras,

[0299] The processing electronic device is configured to present the virtual image content to the user's eyes, the virtual image content being rendered by a rendering camera at a position based on the determined position.

[0300] Example 212: A display system according to any of the above examples, wherein the position is an estimate of the center of rotation of the eye.

[0301] Example 213: A display system according to any of the above examples, wherein the position of the rendering camera is located at the estimated rotation center of the eye.

[0302] Example 214: A display system according to any of the above examples, wherein the position is an estimate of the perspective center of the eye.

[0303] Example 215: A display system according to any of the above examples, wherein the rendering camera is located at the estimated perspective center of the eye.

[0304] Example 216: A display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture closer to the perspective center than the retina.

[0305] Example 217: A display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture closer to the center of rotation than the retina.

[0306] Example 218: A display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture located at the center of rotation.

[0307] Example 219: A display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture closer to the rotation center than the perspective center.

[0308] Example 220: A display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture closer to the perspective center than the retina.

[0309] Example 221: A display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture closer to the perspective center than the rotation center of the eye.

[0310] Example 222: A display system according to any of the above examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture located at the perspective center.

[0311] Example 223: A display system according to any of the above examples, wherein the perspective center is not located at the pupil of the eye.

[0312] Example 224: A display system according to any of the above examples, wherein the processing electronic device is configured to obtain an estimate of the user's eye posture over time, and wherein the processing electronic device adjusts the position of the rendering camera based at least in part on the user's eye posture.

[0313] Example 225: A display system according to any of the above examples, wherein the processing electronic device is configured to track the user's eye posture over time, and wherein the position of the rendering camera is adjusted in response to changes in the user's eye posture over time.

[0314] Example 226: The display system of any of the preceding examples, wherein the processing electronics are configured to obtain an estimate of the perspective center by filtering a plurality of estimated perspective center positions.

[0315] Example 227: A display system according to any of the above examples, wherein the processing electronic device is configured to obtain the estimate of the perspective center by averaging multiple estimated perspective center positions and / or applying a Kalman filter.

[0316] Example 228: A display system according to any of the above examples, wherein the perspective center includes a position within the anterior chamber of the user's eye.

[0317] Example 229: A display system according to any of the above examples, wherein the perspective center includes a position in front of the pupil of the user's eye.

[0318] Example 230: A display system according to any of the above examples, wherein the perspective center includes a position between 1.0 mm and 2.0 mm in front of the pupil of the user's eye.

[0319] Example 231: A display system according to any of the above examples, wherein the perspective center includes a position approximately 1.0 mm in front of the pupil of the user's eye.

[0320] Example 232: A display system according to any of the above examples, wherein the perspective center includes a position between 0.25 mm and 1.0 mm in front of the pupil of the user's eye.

[0321] Example 233: A display system according to any of the above examples, wherein the perspective center includes a position between 0.5 mm and 1.0 mm in front of the pupil of the user's eye.

[0322] Example 234: A display system according to any of the above examples, wherein the perspective center includes a position between 0.25 mm and 0.5 mm in front of the pupil of the user's eye.

[0323] Example 235: A display system according to any of the above examples, wherein the perspective center is positioned along the optical axis of the eye, and wherein the processing electronic device is further configured to obtain a position estimate of the perspective center by obtaining a position estimate of the optical axis of the eye.

[0324] Example 236: A display system according to any of the above examples, wherein the perspective center is located at a position along the optical axis of the eye between the outer surface of the cornea and the pupil of the eye, and wherein the processing electronic device is further configured to obtain a position estimate of the perspective center by obtaining a position estimate of the optical axis of the eye.

[0325] Example 237: A display system according to any of the above examples, wherein the perspective center is located at a position along the optical axis of the eye between the outer surface of the cornea and the pupil of the eye, and wherein the processing electronic device is further configured to obtain a position estimate of the perspective center by obtaining a position estimate of the optical axis of the eye and a position estimate of the rotation center of the eye, the cornea of ​​the eye, the iris of the eye, the retina of the eye, and the pupil of the eye, or any combination thereof.

[0326] Example 238: A display system according to any of the above examples, wherein the processing electronics include electronics located on the frame.

[0327] Example 239: The display system of any of the above examples, wherein the processing electronics include electronics located on the frame and electronics disposed at a location remote from the frame.

[0328] Example 240: The display system of any of the above examples, wherein the processing electronics include electronics located on the frame and electronics located on the waist pack.

[0329] Example 241: A display system according to any of the above examples, wherein at least a portion of the display is transparent and is positioned in front of the user's eyes when the user wears the head-mounted display, so that the transparent portion transmits light from a portion of the user's front environment and the head-mounted display to the user's eyes to provide a view of the portion of the user's front environment and the head-mounted display.

[0330] Example 242: The display system according to any of the above examples further includes one or more light sources, which are arranged on the frame relative to the user's eyes to illuminate the user's eyes, and the one or more cameras use the light from the one or more light sources to capture images of the eyes.

[0331] Example 243: A display system according to any of the above examples, wherein the one or more light sources include at least two light sources, and the at least two light sources are arranged on the frame relative to the user's eyes to illuminate the user's eyes.

[0332] Example 244: A display system according to any of the above examples, wherein the one or more light sources include at least three light sources, and the at least three light sources are arranged on the frame relative to the user's eyes to illuminate the user's eyes.

[0333] Example 245: A display system according to any of the above examples, wherein the one or more light sources include infrared light emitters.

[0334] Example 246: A display system according to any of the above examples, wherein the one or more light sources form one or more bright spots on the eye, and the processing electronic device is configured to determine the position of the center of curvature of the cornea based on the one or more bright spots.

[0335] Example 247: A display system according to any of the above examples, wherein the one or more light sources form one or more bright spots on the eye, and the processing electronic device is configured to determine the three-dimensional position of the center of curvature of the cornea based on the one or more bright spots.

[0336] Example 248: A display system according to any of the above examples, wherein the one or more cameras are further configured to image the pupil of the user's eye, and wherein the processing electronic device is further configured to determine the position of the pupil of the eye based at least on the image of the pupil from the one or more cameras.

[0337] Example 249: A display system according to any of the above examples, wherein the one or more cameras are further configured to image the pupil of the user's eye, and wherein the processing electronic device is further configured to determine the three-dimensional position of the pupil of the eye based at least on the image of the pupil from the one or more cameras.

[0338] Example 250: A display system according to any of the above examples, wherein the one or more cameras are further configured to image the pupil of the user's eye, and wherein the processing electronic device is further configured to determine the position of the pupil of the eye based on the position of the center of curvature of the cornea and based on the image of the pupil from the one or more cameras.

[0339] Example 251: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the optical axis of the eye based on the three-dimensional position of the center of curvature of the cornea and based on the three-dimensional position of the pupil.

[0340] Example 252: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the visual axis of the eye based on the optical axis.

[0341] Example 253: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the visual axis of the eye based on the three-dimensional position of at least one of the center of curvature of the cornea and the pupil, or both the center of curvature of the cornea and the pupil, and the optical axis.

[0342] Example 254: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the three-dimensional position of the center of rotation of the eye based on the three-dimensional position of the center of curvature of the cornea.

[0343] Example 255: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the three-dimensional position of the center of rotation of the eye based on the three-dimensional position of the center of curvature of the cornea and based on the optical axis.

[0344] Example 256: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the distance between the eye and the user's other eye based at least on the three-dimensional position of the rotation center of the eye.

[0345] Example 257: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the pupil distance between the eye and the user's other eye based at least on the three-dimensional position of the rotation center of the eye.

[0346] Example 258: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the user's convergence distance based at least on the optical axis of the eye.

[0347] Example 259: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the convergence distance of the user based at least on the optical axis of the eye and the determined optical axis of the user's other eye.

[0348] Example 260: A display system according to any of the above examples, wherein the processing electronic device is configured to determine the convergence distance of the user based at least on the visual axis of the eye and the determined visual axis of the user's other eye.

[0349] Example 261: A display system according to any of the above examples, wherein the display is configured to project collimated light into the user's eyes.

[0350] Example 262: A display system according to any of the above examples, wherein the display is configured to project collimated light corresponding to the image pixels into the user's eyes in a first time period, and to project divergent light corresponding to the image pixels into the user's eyes in a second time period.

[0351] Example 263: A display system according to any of the above examples, wherein the display is configured to project light having a first divergence corresponding to image pixels into the user's eyes in a first time period, and to project light having a second divergence corresponding to the image pixels that is greater than the first divergence into the user's eyes in a second time period.

[0352] Example 264: A display system according to any of the above examples, wherein the perspective center is estimated to be close to the pupil of the eye.

[0353] Example 265: A display system according to any of the above examples, wherein the perspective center is estimated to be located between the cornea and the pupil of the eye.

[0354] Example 266: A display system according to any of the above examples, wherein the display is configured to project light into the user's eyes to display virtual image content to the user's field of view with at least one of different amounts of divergence and collimation, whereby the displayed virtual image content appears to originate from different depths at different time periods.

[0355] Example 267: A display system configured to project light into an eye of a user to display virtual image content in the user's field of view, the eye having a cornea, an iris, a pupil, a lens, and a retina, the display system comprising:

[0356] a frame configured to be supported on the user's head;

[0357] a head-mounted display disposed on the frame, the display configured to project light into the user's eyes to display virtual image content to the user's field of view with different amounts of at least one of divergence and collimation, whereby the displayed virtual image content appears to originate from different depths at different time periods;

[0358] one or more eye-tracking cameras configured to image the user's eyes; and

[0359] processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics configured to:

[0360] obtaining a position and orientation estimate of an optical axis extending through the lens, pupil, and cornea of ​​the user's eye based on images of the eye obtained by the one or more eye-tracking cameras;

[0361] identifying a specific location in rendering space along an axis that is aligned with the optical axis of the eye; and

[0362] The virtual image content is presented to the eyes of the user, the virtual image content being rendered by a virtual rendering camera located at the specific position in a rendering space.

[0363] Example 268: A display system according to Example 267, wherein the head-mounted display is configured to project light having a first divergence into the user's eyes during a first time period and is configured to project light having a second divergence into the user's eyes during a second time period, wherein the first divergence is different from the second divergence, and wherein the processing electronic device is further configured to obtain an estimate of the user's convergence distance based on an image of the eye obtained by the one or more eye tracking cameras, and to switch from projecting light into the user's eyes with the first divergence to projecting light into the user's eyes with the second divergence based on the user's estimated convergence distance.

[0364] Example 269: A display system according to Example 267, wherein the head-mounted display is configured to project light having a first divergence into the user's eyes during a first time period and is configured to project light having a second divergence into the user's eyes during a second time period, wherein the first divergence is different from the second divergence, and wherein the processing electronic device is further configured to obtain an estimate of the user's convergence distance based on an image of the eye obtained by the one or more eye tracking cameras, and to switch from projecting light into the user's eyes with the first divergence to projecting light into the user's eyes with the second divergence based on the user's estimated convergence distance.

[0365] Example 270: The display system of Example 267, wherein the processing electronics are further configured to determine a position along the optical axis at which the center of rotation of the eye is estimated to be located based on images of the eye obtained by one or more eye tracking cameras, and

[0366] Wherein the processing electronics are configured to identify a particular position in rendering space along the axis that is aligned with the optical axis of the eye based on the position along the optical axis where the center of rotation of the eye is estimated to be located.

[0367] Example 271: A display system according to Example 267, wherein the particular position along the axis in the rendering space includes a position along the axis in the rendering space at which a parallax shift in the rendering space is determined to be minimum.

[0368] Example 272: A display system according to Example 267, wherein the particular position along the axis in the rendering space includes a position along the axis in the rendering space at which the parallax offset in the rendering space is determined to be minimum.

[0369] Example 273: A display system according to any of the above examples, wherein the processing electronic device is configured to obtain the estimate of the center of rotation of the eye based on determining multiple gaze directions of the user's eye during the time period of rotation of the eye based on images of the eye obtained by the one or more eye tracking cameras.

[0370] Example 274: A display system according to Example 273, wherein the processing electronic device is configured to determine the gaze direction based on changes in the shape of one or more of the pupil, iris, and limbus of the user's eye in images obtained by the one or more eye tracking cameras over a period of time during which the eye rotates.

[0371] Example 275: A display system according to any of the above examples, wherein the processing electronic device is configured to determine a position array based on a plurality of spatial positions on an image of the user's eyes obtained by the one or more eye tracking cameras.

[0372] Example 276: A display system according to Example 275, wherein the position array corresponds to at least a portion of an ellipse.

[0373] Example 277: A display system according to Example 275 or 276, wherein the processing electronic device is configured to determine the array of positions by fitting a curve to the multiple spatial positions on the image of the user's eye.

[0374] Example 278: A display system according to Example 277, wherein the curve comprises an ellipse.

[0375] Example 279: A display system according to any of Examples 275 to 278, wherein the multiple spatial locations on the image include spatial locations on the corneal limbus of the user's eye in the image.

[0376] Example 280: A display system according to any of Examples 275 to 279, wherein the multiple spatial locations on the image include spatial locations on the boundary between the iris and sclera of the user's eye in the image.

[0377] Example 281: A display system according to any of Examples 275 to 279, wherein the multiple spatial locations on the image include spatial locations on a boundary between the cornea and the sclera of the user's eye in the image obtained by the one or more eye tracking cameras.

[0378] Example 282: A display system according to any of Examples 275 to 281, wherein the processing electronic device is configured to determine a plurality of linear paths extending from positions on a first side of the position array through the position array to an opposite second side of the position array.

[0379] Example 283: A display system according to Example 282, wherein the processing electronic device is configured to determine a circular area based on the multiple linear paths, the circular area having a radius R.

[0380] Example 284: A display system according to Example 283, wherein the radius R corresponds to the average radius of the corneal limbus.

[0381] Example 285: A display system according to Example 283, wherein the radius R corresponds to a measured radius of the corneal limbus of the user's eye.

[0382] Example 286: A display system according to Example 283, wherein the radius R corresponds to the average radius of the pupil.

[0383] Example 287: A display system according to Example 283, wherein the radius R corresponds to a measured radius of the pupil of the user's eye.

[0384] Example 288: A display system according to any of Examples 282 to 287, wherein the processing electronic device is configured to determine the position and direction of a normal passing through a central portion of the circular area.

[0385] Example 289: A display system according to any of Examples 282 to 288, wherein the processing electronic device is configured to determine the corresponding positions and directions of multiple normals passing through the central portion of each circular area based on multiple images of the eye previously obtained by the one or more eye tracking cameras.

[0386] Example 290: A display system according to Example 289, wherein the processing electronic device is configured to determine a location where the multiple normals converge or intersect.

[0387] Example 291: A display system according to Example 289, wherein the processing electronic device is configured to obtain an estimate of the center of rotation of the user's eye by identifying an area where multiple of the normals determined based on images of the user's eye obtained during a time period of rotation of the eye intersect, converge, or are in close proximity.

[0388] Example 292: A display system according to Example 289, wherein the processing electronic device is configured to obtain an estimate of the center of rotation of the user's eye based on the position and direction of multiple of the multiple normals determined based on images of the user's eye obtained during the time period of the eye rotation.

[0389] Example 293: A display system according to any of Examples 282 to 292, wherein the position on the first side of the array position corresponds to the origin of the coordinate system of one of the one or more eye tracking cameras.

[0390] Example 294: A display system according to any of the above examples, wherein the processing electronic device is configured to obtain an estimate of the center of rotation by filtering, averaging, applying a Kalman filter, or performing any combination of the above items on multiple estimated center of rotation positions.

[0391] Various additional examples of display systems that project light into one or more eyes of a user to display virtual image content in the user's field of view are described herein, such as the following additional examples:

[0392] Additional Example 1: A display system configured to project light into an eye of a user to display virtual image content in the user's field of view, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea, the display system comprising:

[0393] a frame configured to be supported on the user's head;

[0394] a head-mounted display disposed on the frame, the display configured to project light into the user's eyes to display virtual image content to the user's field of view, at least a portion of the display being transparent and disposed in front of the user's eyes when the user wears the frame, such that the transparent portion transmits light from a portion of the user's front environment and the head-mounted display to the user's eyes to provide a view of the portion of the user's front environment and the head-mounted display;

[0395] an inward-facing imaging system configured to image the user's eyes; and

[0396] and processing electronics in communication with the inward-facing imaging system, the processing electronics configured to obtain an estimate of a center of rotation of the eye based on a plurality of images of the eye obtained by the inward-facing imaging system, the processing electronics configured to determine a variation in the calculated value of the center of rotation and to select a statistically determined estimate of the center of rotation based on the variation.

[0397] Additional Example 2: The display system of Additional Example 1, wherein reduced variation is used to identify the statistically determined center of rotation.

[0398] Additional Example 3: The display system of Additional Example 1 or 2, wherein a first set of estimates of the center of rotation is calculated based on first values ​​of parameters used to calculate the center of rotation, and the first change is determined based on the first set of estimates.

[0399] Additional Example 4: The display system of Additional Example 3, wherein a second set of estimates of the center of rotation is calculated based on a second value of the parameter, and a second change is determined based on the second set of estimates.

[0400] Additional Example 5: A display system according to Additional Example 4, wherein the first change and the second change are compared to determine which group has a reduced change, and the determination of the estimate of the statistically determined center of rotation is based on the comparison.

[0401] Additional Example 6: The display system of Additional Example 1 or 2, wherein the multiple sets of values ​​for the center of rotation are calculated based on multiple corresponding values ​​of the parameter and corresponding changes determined for different corresponding sets.

[0402] Additional Example 7: The display system of Additional Example 6, wherein the corresponding changes are compared to determine which group has a reduced change, and the determination of the estimate of the statistically determined center of rotation is based on the comparison.

[0403] Additional Example 8: The display system of Additional Example 6 or 7, wherein the value of the parameter for the group with the smallest variation is used to calculate the estimate of the statistically determined center of rotation.

[0404] Additional Example 9: The display system of Additional Example 6, 7, or 8, wherein the group with the smallest variation is used to calculate the estimate of the statistically determined center of rotation.

[0405] Additional Example 10: The display system of any of Additional Examples 3 to 9, wherein the parameter comprises a distance from a center of curvature of the cornea to the center of rotation.

[0406] Additional Example 11: The display system of any of Additional Examples 3 to 9, wherein the parameter comprises a distance from the center of curvature of the cornea to the center of rotation along the optical axis.

[0407] Additional Example 12: The display system of any of the additional examples above, wherein the variation includes variance and / or standard deviation.

[0408] Additional Example 13: The display system according to any of the above additional examples further includes one or more light sources, which are arranged on the frame relative to the user's eyes to illuminate the user's eyes, and the inward-facing imaging system uses the light from the one or more light sources to form an image of the eye.

[0409] Additional Example 14: A display system according to Additional Example 13, wherein the one or more light sources include at least two light sources, and the at least two light sources are arranged on the frame relative to the user's eyes to illuminate the user's eyes.

[0410] Additional Example 15: The display system of Additional Example 13 or 14, wherein the one or more light sources include infrared light emitters.

[0411] Additional Example 16: A display system according to any one of Additional Examples 13 to 15, wherein the one or more light sources form one or more bright spots on the eye, and the processing electronic device is configured to determine the position of the cornea based on the one or more bright spots.

[0412] Additional Example 17: A display system according to any of Additional Examples 13 to 16, wherein the cornea has a corneal sphere associated therewith, the corneal sphere having a center of curvature, and the processing electronic device is configured to determine the position of the center of curvature of the corneal sphere.

[0413] Additional Example 18: A display system according to Additional Example 17, wherein the cornea has a corneal sphere associated therewith, the corneal sphere having a center of curvature, and the processing electronic device is configured to determine the position of the center of curvature of the corneal sphere based on the one or more bright spots.

[0414] Additional Example 19: The display system of any of the above additional examples, wherein the inward-facing imaging system is configured to image the pupil of the eye.

[0415] Additional Example 20: The display system of any of the additional examples above, wherein the processing electronic device is configured to determine the location of the center of the pupil.

[0416] Additional Example 21: The display system of any of the additional examples above, wherein the processing electronic device is configured to determine at least a portion of a boundary between the iris and the pupil.

[0417] Additional Example 22: The display system of Additional Example 21, wherein the processing electronics are configured to determine a center of the boundary between the iris and the pupil.

[0418] Additional Example 23: The display system of any of the additional examples above, wherein the processing electronic device is configured to determine the position of the center of the pupil in three-dimensional space relative to the center of curvature of the cornea.

[0419] Additional Example 24: The display system of any of the above additional examples, wherein the processing electronics are configured to determine the position and orientation of the optical axis.

[0420] Additional Example 25: The display system of Additional Example 24, wherein the processing electronic device is configured to determine the position and orientation of the optical axis based on the position of the center of the pupil in three-dimensional space.

[0421] Additional Example 26: A display system according to any of the above additional examples, wherein the processing electronic device is configured to determine the position and orientation of the optical axis based on the position of the center of the pupil relative to the center of curvature of the cornea in three-dimensional space.

[0422] Additional Example 27: The display system of any of the additional examples above, wherein the processing electronic device is configured to determine the position of the center of rotation of the eye based on the center of curvature of the cornea.

[0423] Additional Example 28: A display system according to any of the above additional examples, wherein the processing electronic device is configured to determine the position of the rotation center of the eye based on the center of curvature of the cornea and the position and orientation of the optical axis.

[0424] Additional Example 29: A display system according to Additional Example 28, wherein the processing electronic device is configured to determine the position of the rotation center of the eye by translating a specific distance along the optical axis from the center of curvature of the cornea.

[0425] Additional Example 30: The display system of Additional Example 29, wherein the specific distance from the center of curvature to the center of rotation is between 4.0 mm and 6.0 mm.

[0426] Additional Example 31: The display system of Additional Example 29 or 30, wherein the specific distance from the center of curvature to the center of rotation is approximately 4.7 mm.

[0427] Additional Example 32: The display system of Additional Example 20 or 30, wherein the processing electronic device is configured to determine the specific distance based at least on one or more images of the eye previously obtained by the inward-facing imaging system.

[0428] Additional Example 33: The display system of any of the above additional examples, wherein the processing electronics comprises electronics located on the frame.

[0429] Additional Example 34: The display system of any of the additional examples above, wherein the processing electronic device includes electronic devices located on the frame and electronic devices disposed at a location away from the frame.

[0430] Additional Example 35: The display system of any of the above additional examples, wherein the processing electronics include electronics located on the frame and electronics located on the waist pack.

[0431] Additional Example 36: A display system according to any of the additional examples above, wherein at least a portion of the display is transparent and is positioned in front of the user's eyes when the user wears the head-mounted display, so that the transparent portion transmits light from a portion of the user's front environment and the head-mounted display to the user's eyes to provide a view of the portion of the user's front environment and the head-mounted display.

[0432] Additional Example 37: A display system according to any of the above additional examples, wherein the head-mounted display receives light from a portion of the user's front environment at a first divergence and transmits light from a portion of the user's front environment to the user's eyes at a second divergence that is substantially the same as the first divergence.

[0433] Additional Example 38: A display system according to any of the above additional examples, wherein the processing electronic device is configured to obtain an estimate of the center of rotation by filtering, averaging, applying a Kalman filter, or performing any combination of the above items to multiple estimated center of rotation positions.

[0434] Additional Example 39: A display system according to any of the above additional examples, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture located at a determined rotational center position of the user's eyes.

[0435] Additional Example 40: The display system of any of the additional examples above, wherein the processing electronic device is configured to render a virtual image to be presented to the eye using a rendering camera located at the center of rotation.

[0436] Additional Example 41: A display system according to any of the above additional examples, wherein the processing electronic device is configured to use a rendering camera, the rendering camera being configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture closer to the center of rotation than the retina of the eye.

[0437] Additional Example 42: A display system according to any of the above additional examples, wherein the processing electronic device is configured to use a rendering camera, the rendering camera being configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture located at the rotation center of the eye.

[0438] Additional Example 43: A display system according to any of the above additional examples, wherein the processing electronic device is configured to render a virtual image to be presented to the eye using a rendering camera located at the rotation center, and the rendering camera is modeled as having an aperture located at the rotation center of the eye.

[0439] Additional Example 44: The display system of any of Additional Examples 1 to 9, wherein the processing electronics are configured to select the estimate of the statistically determined center of rotation based on the variation during a calibration process.

[0440] Additional Example 45: A display system according to any of the above examples or additional examples, wherein the processing electronic device is configured to obtain the estimate of the center of rotation of the eye based on determining multiple gaze directions of the user's eye during the time period of the eye rotation based on images of the eye obtained by the one or more eye tracking cameras.

[0441] Additional Example 46: A display system according to any of the above examples or additional examples, wherein the processing electronic device is configured to determine the gaze direction based on changes in the shape of one or more of the pupil, iris, and limbus of the user's eye in the image obtained by the one or more eye tracking cameras over a time period during which the eye rotates.

[0442] Additional Example 47: A display system according to any of the above examples or additional examples, wherein the processing electronic device is configured to determine a position array based on multiple spatial positions on the image of the user's eyes obtained by the one or more eye tracking cameras.

[0443] Additional Example 48: The display system of any of the above or additional examples, wherein the array of positions corresponds to at least a portion of an ellipse.

[0444] Additional Example 49: A display system according to any of the above examples or additional examples, wherein the processing electronic device is configured to determine the position array by fitting a curve to the multiple spatial positions on the image of the user's eye.

[0445] Additional Example 50: The display system of any of the above examples or additional examples, wherein the curve comprises an ellipse.

[0446] Additional Example 51: A display system according to any of the above examples or additional examples, wherein the multiple spatial positions on the image include spatial positions on the corneal limbus of the user's eye in the image.

[0447] Additional Example 52: A display system according to any of the above examples or additional examples, wherein the multiple spatial positions on the image include spatial positions on the boundary between the iris and sclera of the user's eye in the image.

[0448] Additional Example 53: A display system according to any of the above examples or additional examples, wherein the multiple spatial positions on the image include spatial positions on the boundary between the cornea and the sclera of the user's eye in the image obtained by the one or more eye tracking cameras.

[0449] Additional Example 54: A display system according to any of the above examples or additional examples, wherein the processing electronic device is configured to determine multiple linear paths extending from positions on a first side of the position array through the position array to an opposite second side of the position array.

[0450] Additional Example 55: The display system of any of the above examples or additional examples, wherein the processing electronic device is configured to determine a circular area based on the multiple linear paths, the circular area having a radius R.

[0451] Additional Example 56: A display system according to any of the above examples or additional examples, wherein the radius R corresponds to the average radius of the corneal limbus.

[0452] Additional Example 57: A display system according to any of the above examples or additional examples, wherein the radius R corresponds to the measured radius of the corneal limbus of the user's eye.

[0453] Additional Example 58: A display system according to any of the above examples or additional examples, wherein the radius R corresponds to the average radius of the pupil.

[0454] Additional Example 59: A display system according to any of the above examples or additional examples, wherein the radius R corresponds to the measured radius of the pupil of the user's eye.

[0455] Additional Example 60: The display system of any of the above examples or additional examples, wherein the processing electronic device is configured to determine a position and direction of a normal passing through a center portion of the circular area.

[0456] Additional Example 61: A display system according to any of the above examples or additional examples, wherein the processing electronic device is configured to determine the corresponding positions and directions of multiple normals passing through the central portions of the respective circular areas based on multiple images of the eye previously obtained by the one or more eye tracking cameras.

[0457] Additional Example 62: The display system of any of the above or additional examples, wherein the processing electronic device is configured to determine a location where the plurality of normals converge or intersect.

[0458] Additional Example 63: A display system according to any of the above examples or additional examples, wherein the processing electronic device is configured to obtain an estimate of the center of rotation of the user's eye by identifying an area where multiple of the normals determined based on images of the user's eye obtained during a time period of rotation of the eye intersect, converge, or are in close proximity.

[0459] Additional Example 64: A display system according to any of the above examples or additional examples, wherein the processing electronic device is configured to obtain the estimate of the center of rotation of the user's eye based on the position and direction of multiple of the multiple normals determined based on images of the user's eye obtained during the time period of the eye rotation.

[0460] Additional Example 65: The display system of any of the above or additional examples, wherein the position on the first side of the array positions corresponds to the origin of a coordinate system of one of the one or more eye tracking cameras.

[0461] Additional Example 66: A display system according to any of the above examples or additional examples, wherein the processing electronic device is configured to obtain the estimate of the rotation center by filtering, averaging, applying a Kalman filter, or performing any combination of the above items on multiple estimated rotation center positions.

[0462] Any of the above examples or additional examples may be combined. Additionally, any of the above examples or additional examples may be integrated with a head-mounted display. Additionally, any of the above examples or additional examples may be implemented with a single depth plane and / or one or more variable depth planes (e.g., one or more elements with variable focus capabilities that provide time-varying accommodation cues).

[0463] Furthermore, disclosed herein are apparatuses and methods for determining various values, parameters, and the like (e.g., but not limited to, anatomical, optical, and geometric features, positions, and orientations). Examples of such parameters include, but are not limited to, the center of rotation of the eye, the center of curvature of the cornea, the center of the pupil, the boundary of the pupil, the center of the iris, the boundary of the iris, the boundary of the limbus, the optical axis of the eye, the visual axis of the eye, and the perspective center. The determination of such values, parameters, and the like described herein includes estimates thereof and need not necessarily correspond exactly to the actual values. For example, the determination of the center of rotation of the eye, the center of curvature of the cornea, the center or boundary of the pupil or iris, the boundary of the limbus, the optical axis of the eye, the visual axis of the eye, and the perspective center may be an estimate, approximation, or value that is close to, but not equal to, the actual (e.g., anatomical, optical, or geometric) value or parameter. In some cases, for example, estimates of such values ​​are obtained using root mean square estimation techniques. By way of example, certain techniques described herein involve identifying locations or points where rays or vectors intersect. However, such rays or vectors may not intersect. In such examples, the locations or points may be estimated. For example, a position or point can be determined based on a root mean square or other estimation technique (e.g., a position or point can be estimated to be close to or closest to a ray or vector). Other processes can also be used to estimate, approximate, or otherwise provide a value that may not be consistent with the actual value. Therefore, the terms "determine" and "estimate" or "determined" and "estimated" are used interchangeably herein. Therefore, references to such determined values ​​can include values ​​that are estimated, approximated, or close to the actual value. Therefore, references above or elsewhere herein to "determined parameters or values" should not be limited to exactly actual values, but can include values ​​that are estimated, approximated, or close to actual values.

[0464] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Additional features, aspects, and advantages will become apparent upon reading the description, drawings, and claims. Neither this summary nor the following detailed description is intended to define or restrict the scope of the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0465] Figure 1 An illustration of a mixed reality scene with certain virtual reality objects and certain physical objects viewed by a person is shown.

[0466] Figure 2 An example of a wearable system is schematically shown.

[0467] Figure 3 Example components of a wearable system are schematically shown.

[0468] Figure 4 An example of a waveguide stack for a wearable device that outputs image information to a user is schematically shown.

[0469] Figure 5 An example of an eye is schematically shown.

[0470] Figure 5A An example coordinate system for determining the eye pose of an eye is schematically shown.

[0471] Figure 6 is a schematic diagram of a wearable system including an eye tracking system.

[0472] Figure 7A is a block diagram of a wearable system that may include an eye tracking system.

[0473] Figure 7B This is a block diagram of the rendering controller in a wearable system.

[0474] Figure 8A is a schematic diagram of an eye showing the cornea of ​​the eye.

[0475] Figure 8B An example corneal bright spot detected by an eye-tracking camera is shown.

[0476] Figures 8C to 8E An example stage of locating the center of a user's cornea by an eye tracking module in a wearable system is shown.

[0477] Figures 9A to 9C An example normalization of the coordinate system of an eye-tracking image is shown.

[0478] Figures 9D to 9G An example stage of locating the pupil center of a user by an eye tracking module in a wearable system is shown.

[0479] Figure 9H Example calculated positions of the pupil center are shown with and without accounting for corneal refraction.

[0480] Figures 9I to 9M Example experimental variations of the calculated eye rotation center based on different calculated pupil centers are shown.

[0481] Figure 10 An example of an eye is shown, including the optical and visual axes of the eye, and the center of rotation of the eye.

[0482] Figure 11 is a process flow diagram of an example of a method for rendering content in a wearable device using eye tracking and providing registration feedback.

[0483] Figure 12 is a set of example diagrams showing how a wearable system can switch depth planes in response to a user's eye movements.

[0484] Figure 13 A mixed reality system is shown in which a particular virtual object can be discretized in depth into one or more depth planes.

[0485] Figure 14A and 14B An example is shown in which a mixed reality system displays virtual objects using a single depth plane.

[0486] Figure 14C and 14D An example is shown in which a mixed reality system displays a virtual object using two adjacent depth planes to generate accommodation cues between the two adjacent depth planes.

[0487] Figure 14E and 14F An example is shown in which a mixed reality system displays a virtual object using two or more depth planes to simultaneously generate two or more accommodation cues.

[0488] Figure 15A A mixed reality system is shown in which a perspective center misalignment is present. Figure 14A and 14B An example of displaying a virtual object in a .

[0489] Figure 15B A mixed reality system is shown in which a perspective center misalignment is present. Figure 14C and 14D An example of displaying a virtual object in a .

[0490] Figure 15C A mixed reality system is shown in which a perspective center misalignment is present. Figure 14E and 14F An example of displaying a virtual object in a .

[0491] Figure 16A and 16B An example is shown where the pinhole of the rendering camera is aligned with the perspective center of the eye, or approximately with the pupil of the eye.

[0492] Figure 16C Another example is shown where the pinhole of the rendering camera is aligned with the perspective center of the eye, or approximately with the pupil of the eye.

[0493] Figure 17A and 17B An example is shown where the pinhole of the rendering camera is aligned with the rotation center of the eye.

[0494] Figure 17C Another example is shown where the pinhole of the rendering camera is aligned with the rotation center of the eye.

[0495] Figure 18A and 18B is a set of example graphs illustrating eye tracking data relative to a user's perspective center.

[0496] Figure 18C is an example diagram showing how the rendering camera position can change with eye tracking data.

[0497] Figure 18D is a schematic diagram of a user's eye showing various rendered camera positions that may be used as a function of eye tracking data.

[0498] Figure 19 A graphical representation of determining an estimate of CoR using an ellipse projected onto an image of a user's eye (e.g., the limbus) according to various techniques described herein is shown. Rays can be traced through the projected ellipse to form a cone within which a circle is fitted. In some embodiments, a normal passing through one or more circles can be used to estimate CoR.

[0499] Figure 20 is a flow chart of an example process for determining the center of rotation of an eye based on the limbus of the eye.

[0500] Figure 21A A first ellipse (hereinafter referred to as the "projected ellipse") determined based on the user's first gaze and projected onto the image of the user's corneal limbus is shown. Rays passing through the projected ellipse are shown as forming cones that fit the circle.

[0501] Figure 21B A second projected ellipse determined based on a second gaze of the user's eyes is shown. Also shown are a second cone and a circle fitted thereto.

[0502] Figure 21C A third projected ellipse determined based on a third gaze of the user's eyes is shown. Also shown is a third cone and a circle fitted thereto.

[0503] Figure 21D Determination of the center of rotation based on a circle obtained using the above-mentioned projected ellipse is shown.

[0504] Figure 22 An optical system comprising two point light sources, an aperture, a lens and a projection screen is shown.

[0505] Figure 23A An embodiment of an optical system is shown in a first stage.

[0506] Figure 23B The optical system is shown in the second stage.

[0507] Figure 24A Another embodiment of an optical system is shown in a first stage.

[0508] Figure 24B The optical system is shown in the second stage.

[0509] Throughout the drawings, reference numerals may be repeated to indicate correspondence between referenced elements.The drawings are provided to illustrate the example embodiments described herein and are not intended to limit the scope of the disclosure. DETAILED DESCRIPTION

[0510] Referring now to the drawings, in which like reference numerals represent like parts throughout, unless otherwise indicated, the drawings are schematic and not necessarily drawn to scale.

[0511] Example of 3D display for wearable systems

[0512] A wearable system (also referred to herein as an augmented reality (AR) system) can be configured to present a 2D or 3D virtual image to a user. The image can be a still image, a video frame, or a video, a combination of the above, or the like. At least a portion of the wearable system can be implemented on a wearable device that is capable of presenting a VR, AR, or MR environment, alone or in combination, for user interaction. A wearable device can be used interchangeably with an AR device (ARD). Furthermore, for the purposes of this disclosure, the term "AR" can be used interchangeably with the term "MR."

[0513] Figure 1 An illustration of a mixed reality scene with certain virtual reality objects and certain physical objects viewed by a person is shown. Figure 1 , an MR scene 100 is shown in which a user of MR technology sees a real-world park-like setting 110 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of MR technology also perceives that he "sees" a robotic statue 130 standing on the real-world platform 120, and a flying, cartoon-like avatar character 140 that looks like an avatar of a bumblebee, even though these elements do not exist in the real world.

[0514] In order for a 3D display to produce a realistic sense of depth, and more specifically, to produce a simulated sense of surface depth, it may be desirable for each point in the display's field of view to produce an accommodation response corresponding to its virtual depth. If the accommodation response to a displayed point does not correspond to the point's virtual depth (determined by binocular depth cues from convergence and stereopsis), the human eye may experience accommodation conflict, leading to unstable imaging, unwanted eye strain, headaches, and, in the absence of accommodation information, a near-total lack of surface depth.

[0515] VR, AR, and MR experiences can be provided by a display system having a display in which images corresponding to multiple depth planes are provided to a viewer. The images in each depth plane may be different (e.g., providing a slightly different representation of a scene or object) and can be focused separately by the viewer's eyes, thereby facilitating eye-based accommodation (which is required to bring different image features of a scene located at different depth planes into focus) or providing depth cues to the user based on observing that different image features at different depth planes are out of focus. As discussed elsewhere in this document, such depth cues provide a reliable sense of depth.

[0516] Figure 2 An example of a wearable system 200 that can be configured to provide an AR / VR / MR scene is shown. The wearable system 200 may also be referred to as an AR system 200. The wearable system 200 includes a display 220 and various mechanical and electronic modules and systems that support the functions of the display 220. The display 220 can be coupled to a frame 230, which can be worn by a user, wearer, or viewer 210. The display 220 can be positioned in front of the eyes of the user 210. The display 220 can present AR / VR / MR content to the user. The display 220 may include a head-mounted display (HMD) worn on the user's head.

[0517] In some embodiments, a speaker 240 is coupled to the frame 230 and positioned near the user's ear canal (in some embodiments, another speaker (not shown) is positioned near the user's other ear canal to provide stereo / shapeable sound control). The display 220 may include an audio sensor (e.g., a microphone) 232 to detect an audio stream from the environment and capture ambient sound. In some embodiments, one or more other audio sensors (not shown) are positioned to provide stereo reception. Stereo reception can be used to determine the location of a sound source. The wearable system 200 can perform voice or speech recognition on the audio stream.

[0518] The wearable system 200 may include an outward-facing imaging system 464 (e.g., Figure 4 ), which observes the world around the user. The wearable system 200 may also include an inward-facing imaging system 462 (as shown in FIG. Figure 4), which can track the user's eye movements. The inward-facing imaging system can track the movement of one eye or both eyes. Inward-facing imaging system 462 can be attached to frame 230 and can be in electrical communication with processing module 260 or 270, which can process image information acquired by the inward-facing imaging system to determine, for example, the pupil diameter or orientation, eye movement, or eye pose of the user's 210 eyes. Inward-facing imaging system 462 can include one or more cameras. For example, at least one camera can be used to image each eye. The images acquired by the cameras can be used to individually determine the pupil size or eye pose of each eye, thereby allowing image information dynamically tailored for that eye to be presented to each eye.

[0519] As an example, the wearable system 200 can acquire an image of the user's gesture using the outward-facing imaging system 464 or the inward-facing imaging system 462. The image can be a still image, a video frame, or a video.

[0520] The display 220 can be operably coupled 250 to a local data processing module 260, for example, via a wired lead or a wireless connection, and the local data processing module 260 can be mounted in various configurations, such as fixedly attached to the frame 230, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, in a belt-coupled configuration).

[0521] The local processing and data module 260 may include a hardware processor and digital storage such as non-volatile memory (e.g., flash memory), both of which may be used to assist in the processing, caching, and storage of data. The data may include: a) data captured from environmental sensors (which may, for example, be operably coupled to the frame 230 or otherwise attached to the user 210), such as image capture devices (e.g., cameras in an inward-facing imaging system or an outward-facing imaging system), audio sensors (e.g., microphones), inertial measurement units (IMUs), accelerometers, compasses, global positioning systems (GPS) units, radios, or gyroscopes; or b) data acquired or processed using the remote processing module 270 or remote data repository 280, possibly after such processing or retrieval, and passed to the display 220. The local processing and data module 260 may be operably coupled to the remote processing module 270 or remote data repository 280 via communication links 262 or 264 (e.g., via a wired or wireless communication link), making these remote modules available as resources to the local processing and data module 260. Additionally, the remote processing module 280 and the remote data repository 280 may be operably coupled to each other.

[0522] In some embodiments, the remote processing module 270 may include one or more processors configured to analyze and process data or image information. In some embodiments, the remote data repository 280 may include a digital data storage facility that may be available via the Internet or other network configuration as a "cloud" resource. In some embodiments, all data is stored and all calculations are performed in the local processing and data module, allowing for fully autonomous use from the remote module.

[0523] Example components of a wearable system

[0524] Figure 3 Example components of a wearable system are schematically shown. Figure 3 A wearable system 200 is shown, which may include a display 220 and a frame 230. An enlarged view 202 schematically illustrates various components of the wearable system 200. In some embodiments, Figure 3 One or more of the components shown may be part of the display 220. The various components, alone or in combination, may collect various data (e.g., audio or visual data) associated with the user of the wearable system 200 or the user's environment. It should be understood that other embodiments may have more or fewer components depending on the application for which the wearable system is used. Nonetheless, Figure 3 Provides a basic idea of ​​the various components and some of the types of data that can be collected, analyzed, and stored by a wearable system.

[0525] Figure 3 An example wearable system 200 is shown, which may include a display 220. The display 220 may include a display lens 226, which may be mounted to the user's head or to a housing or frame 230 corresponding to the frame 230. The display lens 226 may include one or more transparent mirrors positioned by the housing 230 in front of the user's eyes 302, 304 and may be configured to reflect projected light 338 into the eyes 302, 304 and facilitate beam shaping, while also allowing transmission of at least some light from the local environment. The wavefront of the projected light beam 338 may be bent or focused to coincide with the desired focal length of the projected light. As shown, two wide field of view machine vision cameras 316 (also known as world cameras) may be coupled to the housing 230 to image the environment surrounding the user. These cameras 316 may be dual shot visible light / invisible (e.g., infrared) light cameras. The cameras 316 may be Figure 47 . The images acquired by world camera 316 may be processed by gesture processor 336. For example, gesture processor 336 may implement one or more object recognizers 708 (e.g., as shown in FIG. 7 ) to recognize gestures of the user or another person in the user's environment, or to recognize physical objects in the user's environment.

[0526] Continue to refer Figure 3 , shows a pair of scanning laser shaping wavefront (e.g., for depth) light projector modules having display mirrors and optics configured to project light 338 into eyes 302, 304. The depicted view also shows two miniature infrared cameras 324 paired with infrared light sources 326 (e.g., light emitting diodes "LEDs") that are configured to track the user's eyes 302, 304 to support rendering and user input. The cameras 324 may be Figure 4 The wearable system 200 may also have a sensor assembly 339 that may include X, Y, and Z axis accelerometer functionality as well as a magnetic compass and X, Y, and Z axis gyroscope functionality, preferably providing data at a relatively high frequency (e.g., 200 Hz). The sensor assembly 339 may be a reference Figure 2 The depicted system 200 may also include a head pose processor 336, such as an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or an ARM processor (Advanced Reduced Instruction Set Machine), which may be configured to calculate the user's head pose in real time or near real time based on the wide field of view image information output from the capture device 316. The head pose processor 336 may be a hardware processor and may be implemented as Figure 2 A portion of the local processing and data module 260 is shown.

[0527] The wearable system may also include one or more depth sensors 234. Depth sensor 234 may be configured to measure the distance between an object in the environment and the wearable device. Depth sensor 234 may include a laser scanner (e.g., a lidar), an ultrasonic depth sensor, or a depth sensing camera. In some embodiments, if camera 316 has depth sensing capabilities, camera 316 may also be considered depth sensor 234.

[0528] Also shown is a processor 332 configured to perform digital or analog processing to derive gesture from gyroscope, compass, or accelerometer data from sensor assembly 339. Processor 332 may be Figure 2 A portion of the local processing and data module 260 is shown. Figure 3The wearable system 200 shown may also include a positioning system such as GPS 337 (Global Positioning System) to assist in posture and location analysis. In addition, GPS can further provide remote-based (e.g., cloud-based) information about the user's environment. This information can be used to identify objects or information in the user's environment.

[0529] The wearable system can combine data acquired by GPS 337 with data from a remote computing system (e.g., remote processing module 270, another user's ARD, etc.) that can provide more information about the user's environment. As one example, the wearable system can determine the user's location based on GPS data and retrieve a world map (e.g., by communicating with remote processing module 270) that includes virtual objects associated with the user's location. As another example, the wearable system 200 can use world camera 316 (which can be Figure 4 7 ). The wearable system 200 may further use data acquired by GPS 337 to interpret characters.

[0530] The wearable system 200 may also include a rendering engine 334, which may be configured to provide local rendering information to the user to facilitate operation of the scanner and imaging into the user's eyes for the user to view the world. The rendering engine 334 may be implemented by a hardware processor (e.g., a central processing unit or a graphics processing unit). In some embodiments, the rendering engine is part of the local processing and data module 260. The rendering engine 334 may be communicatively coupled (e.g., via a wired or wireless link) to other components of the wearable system 200. For example, the rendering engine 334 may be coupled to the eye camera 324 via a communication link 274, and to the projection subsystem 318 via a communication link 272 (which may project light into the user's eyes 302, 304 via a scanning laser device in a manner similar to a retinal scanning display). The rendering engine 334 may also communicate with other processing units (e.g., the sensor gesture processor 332 and the image gesture processor 336) via links 276 and 294, respectively.

[0531] Camera 324 (e.g., a miniature infrared camera) can be used to track eye poses to support rendering and user input. Some example eye poses can include where the user is looking. Or what depth he or she is focusing on (this can be estimated by eye convergence). GPS 337, gyroscope, compass, and accelerometer 339 can be used to provide rough or fast pose estimates. One or more of cameras 316 can acquire images and poses that, combined with data from associated cloud computing resources, can be used to map the local environment and share the user's view with other users.

[0532] Figure 3 The example components shown are for illustration purposes only. For ease of illustration and description, multiple sensors and other functional modules are shown together. Some embodiments may include only one or a subset of these sensors or modules. In addition, the location of these components is not limited to Figure 3 The position shown. Certain components may be mounted to or housed in other components (e.g., a belt-mounted component, a handheld component, or a helmet component). As an example, the image pose processor 336, the sensor pose processor 332, and the rendering engine 334 may be located in a waist pack and configured to communicate with other components of the wearable system via wireless communication (e.g., ultra-wideband, Wi-Fi, Bluetooth, etc.) or via wired communication. The housing 230 shown is preferably head-mountable and wearable by the user. However, certain components of the wearable system 200 may be worn to other parts of the user's body. For example, the speaker 240 may be inserted into the user's ear to provide sound to the user.

[0533] With respect to the projection of light 338 into the user's eyes 302, 304, in some embodiments, a camera 324 can be used to measure where the center of the user's eyes are geometrically directed, which generally coincides with the focus position or "depth of focus" of the eyes. The three-dimensional surface of all points that the eyes are directed toward can be referred to as the "horopter." The focal length can present a finite number of depths, or can vary infinitely. Light projected from the convergence distance appears to be focused onto the target eye 302, 304, while light in front of or behind the convergence distance becomes blurred. Examples of wearable devices and other display systems of the present disclosure are also described in U.S. Patent Publication No. 2016 / 0270656, the entire contents of which are incorporated herein by reference.

[0534] The human visual system is complex, and providing a realistic perception of depth is challenging. A viewer of an object may perceive the object as "three-dimensional" due to a combination of vergence and accommodation. The vergence movement of the two eyes relative to each other (e.g., the rolling movement of the pupils toward or away from each other to converge the eyes' line of sight to look at an object) is closely related to the focusing (or "accommodation") of the eye lenses. Under normal circumstances, changing the focus of the eye lenses, or accommodating the eyes, to change focus from one object to another at a different distance will automatically result in a matching change in vergence to the same distance, under a relationship known as the "accommodation-vergence reflex." Likewise, under normal circumstances, a change in vergence will trigger a matching change in accommodation. A display system that provides a better match between accommodation and vergence can form a more realistic and comfortable simulation of a three-dimensional image.

[0535] Spatially coherent light having a beam diameter of less than about 0.7 mm can be correctly resolved by the human eye regardless of where the eye is focused. Thus, to create the illusion of proper depth of focus, eye convergence can be tracked by camera 324, and all objects on or near the binocular field of view in focus, as well as all other objects in varying degrees of defocus (e.g., using intentionally generated blur), can be rendered using rendering engine 334 and projection subsystem 318. Preferably, system 220 renders to the user at a frame rate of about 60 frames per second or higher. As described above, preferably, eye tracking can be performed using camera 324, and the software can be configured to pick up not only the convergence geometry but also focus position cues for use as user input. Preferably, such a display system is configured with brightness and contrast suitable for daytime or nighttime use.

[0536] In some embodiments, the display system preferably has a visual object alignment delay of less than about 20 milliseconds, an angular alignment delay of less than about 0.1 degrees, and a resolution delay of about 1 arc minute, which, without being limited by theory, is believed to be close to the limits of the human eye. The display system 220 can be integrated with a positioning system that can include a GPS element, optical tracking, a compass, an accelerometer, or other data sources to assist in determining position and posture; the positioning information can be used to facilitate accurate rendering in the user's view of the world (e.g., such information will help the glasses understand their position relative to the real world).

[0537] In some embodiments, the wearable system 200 is configured to display one or more virtual images based on the user's eye adjustments. Unlike existing 3D display methods that force the user to focus on the image projection location, in some embodiments, the wearable system is configured to automatically change the focus of the projected virtual content to allow for more comfortable viewing of the one or more images presented to the user. For example, if the user's eye is currently focused at 1 meter, the image can be projected to coincide with the user's focus. If the user moves the focus to 3 meters, the image is projected to coincide with the new focus. Thus, rather than forcing the user to reach a predetermined focus, the wearable system 200 of some embodiments allows the user's eyes to function in a more natural manner.

[0538] Such a wearable system 200 can eliminate or reduce the occurrence of eyestrain, headaches, and other physiological symptoms commonly observed with virtual reality devices. To achieve this, various embodiments of the wearable system 200 are configured to project virtual images at varying focal lengths via one or more variable focus elements (VFEs). In one or more embodiments, 3D perception can be achieved via a multi-plane focusing system that projects images at a fixed focal plane away from the user. Other embodiments employ variable plane focus, where the focal plane moves back and forth in the z-direction to align with the user's current focus state.

[0539] In both multi-plane focusing systems and variable plane focusing systems, the wearable system 200 can employ eye tracking to determine the convergence of the user's eyes, determine the user's current focus, and project a virtual image at a predetermined focus. In other embodiments, the wearable system 200 includes a light modulator that variably projects a zoom beam in a raster pattern across the retina via a fiber scanner or other light generating source. Thus, as further described in U.S. Patent Publication No. 2016 / 0270656, the entire contents of which are incorporated herein by reference, the ability of the display of the wearable system 200 to project images at varying focal lengths not only eases the user's adjustment when viewing 3D objects, but can also be used to compensate for visual anomalies in the user. In some other embodiments, the spatial light modulator can project an image to the user via various optical components. For example, as further described below, the spatial light modulator can project an image onto one or more waveguides, which then transmit the image to the user.

[0540] Waveguide stack components

[0541] Figure 4An example of a waveguide stack for outputting image information to a user is shown. The wearable system 400 includes a waveguide stack or stacked waveguide assembly 480 that can be used to provide a three-dimensional perception to the eye / brain using a plurality of waveguides 432b, 434b, 436b, 438b, 4400b. In some embodiments, the wearable system 400 may correspond to Figure 2 Wearable system 200, Figure 4 Some parts of the wearable system 200 are schematically shown in more detail. For example, in some embodiments, the waveguide assembly 480 can be integrated into Figure 2 in the display 220 .

[0542] Continue to refer Figure 4 , the waveguide assembly 480 can also include a plurality of features 458, 456, 454, 452 located between the waveguides. In some embodiments, the features 458, 456, 454, 452 can be lenses. In other embodiments, the features 458, 456, 454, 452 can not be lenses. Instead, they can simply be spacers (e.g., cladding or structures used to form air gaps).

[0543] The waveguides 432b, 434b, 436b, 438b, 440b or the plurality of lenses 458, 456, 454, 452 can be configured to deliver image information to the eye at various levels of wavefront curvature or light divergence. Each waveguide level can be associated with a particular depth plane and can be configured to output image information corresponding to that depth plane. Image injection devices 420, 422, 424, 426, 428 can be used to inject image information into the waveguides 440b, 438b, 436b, 434b, 432b, each of which can be configured to distribute incident light through each corresponding waveguide for output toward the eye 410. Light exits the output surfaces of the image injection devices 420, 422, 424, 426, 428 and is injected into the respective input edges of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam (e.g., a collimated beam) can be injected into each waveguide to output an entire field of cloned collimated beams directed toward the eye 410 at a specific angle (and divergence) corresponding to the depth plane associated with the particular waveguide.

[0544] In some embodiments, image injection devices 420, 422, 424, 426, 428 are separate displays, each generating image information for injection into a corresponding waveguide 440b, 438b, 436b, 434b, 432b, respectively. In some other embodiments, image injection devices 420, 422, 424, 426, 428 are outputs of a single multiplexed display, which can pipe image information to each of image injection devices 420, 422, 424, 426, 428, for example, via one or more light conduits (e.g., fiber optic cables).

[0545] Controller 460 controls the operation of stacked waveguide assembly 480 and image injection devices 420, 422, 424, 426, 428. Controller 460 includes programming (e.g., instructions in a non-transitory computer readable medium) that regulates the timing and provision of image information to waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, controller 460 may be a single, integral device or a distributed system connected via wired or wireless communication channels. In some embodiments, controller 460 may be a processing module 260 or 270 ( Figure 2 ) part.

[0546] The waveguides 440b, 438b, 436b, 434b, 432b can be configured to propagate light within each respective waveguide by total internal reflection (TIR). The waveguides 440b, 438b, 436b, 434b, 432b can each be planar or have other shapes (e.g., curved), having top and bottom major surfaces and edges extending between these top and bottom major surfaces. In the configuration shown, the waveguides 440b, 438b, 436b, 434b, 432b can each include light extraction optical elements 440a, 438a, 436a, 434a, 432a that are configured to extract light out of each respective waveguide by redirecting the light propagating within the waveguide to output image information to the eye 410. The extracted light can also be referred to as outcoupled light, and the light extraction optical elements can also be referred to as outcoupling optical elements. The extracted light beam is output from the waveguide at the location where the light propagating in the waveguide strikes the light redirecting element. The light extraction optical elements (440a, 438a, 436a, 434a, 432a) can be, for example, reflective or diffractive optical features. Although illustrated as being positioned at the bottom major surface of the waveguides 440b, 438b, 436b, 434b, 432b for ease of description and clarity of drawing, in some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be positioned at the top or bottom major surface, or can be positioned directly within the volume of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be formed in a layer of material attached to a transparent substrate to form waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, 432b can be a single piece of material, and the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be formed on a surface of the piece of material or within the interior of the piece of material.

[0547] Continue to refer Figure 4As discussed herein, each waveguide 440b, 438b, 436b, 434b, 432b is configured to output light to form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye can be configured to transmit collimated light, such as that injected into such waveguide 432b, to the eye 410. The collimated light can represent a focal plane at optical infinity. The next upstream waveguide 434b can be configured to emit the collimated light transmitted through a first lens 452 (e.g., a negative lens) before it reaches the eye 410. The first lens 452 can be configured to produce a slightly convex wavefront curvature so that the eye / brain interprets the light from the next upstream waveguide 434b as coming from a first focal plane that is closer to the eye 410 from optical infinity inward. Similarly, the third upstream waveguide 436b transmits the output light through the first lens 452 and the second lens 454 before reaching the eye 410. The combined optical power of the first lens 452 and the second lens 454 can be configured to produce another increment of wavefront curvature so that the eye / brain interprets light from the third waveguide 436b as coming from a second focal plane that is further inward toward the person from optical infinity than the light from the next upstream waveguide 434b.

[0548] The other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured, with the highest waveguide 440b in the stack sending its output through all the lenses between it and the eye for an aggregate focal power representing the focal plane closest to the person. To compensate for the stack of lenses 458, 456, 454, 452 when viewing / interpreting light from the world 470 on the other side of the stacked waveguide assembly 480, a compensating lens layer 430 can be provided at the top of the stack to compensate for the aggregate focal power of the underlying lens stack 458, 456, 454, 452 (the compensating lens layer 430 and the stacked waveguide assembly 480 as a whole can be configured such that light from the world 470 is delivered to the eye 410 with substantially the same level of divergence (or collimation) as when it was initially received by the stacked waveguide assembly 480). This configuration provides as many perceived focal planes as there are available waveguide / lens pairs. The light extraction optics of the waveguide and the focusing aspects of the lens can be static (e.g., not dynamic or electrically activated). In some alternative embodiments, one or both can be dynamic using electrically activated features.

[0549] Continue to refer Figure 4, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be configured to redirect light out of their respective waveguides and output the light with an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes can have different light extraction optical element configurations that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be volume or surface features that can be configured to output light at specific angles. For example, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be volume holograms, surface holograms and / or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published on June 25, 2015, which is incorporated herein by reference in its entirety.

[0550] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffractive features or "diffractive optical elements" (also referred to herein as "DOEs") that form a diffraction pattern. Preferably, the DOEs have a relatively low diffraction efficiency so that only a portion of the light beam is deflected toward the eye 410 by each intersection of the DOEs, while the remainder continues through the waveguide via total internal reflection. The light carrying image information can thus be split into multiple related exit beams that exit the waveguide at multiple locations, and the result is a fairly uniform pattern of exit emissions toward the eye 304 for that particular collimated light beam that bounces within the waveguide.

[0551] In some embodiments, one or more DOEs can be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE can include a polymer dispersed liquid crystal layer in which droplets contain a diffraction pattern in a matrix medium, and the refractive index of the droplets can be switched to substantially match the refractive index of the matrix material (in which case the pattern does not significantly diffract incident light), or the droplets can be switched to a refractive index that does not match the refractive index of the matrix medium (in which case the pattern actively diffracts incident light).

[0552] In some embodiments, the number and distribution of depth planes, or depth of field, can dynamically change based on the pupil size or orientation of the viewer's eye. The depth of field can change inversely proportional to the viewer's pupil size. Thus, as the pupil size of the viewer's eye decreases, the depth of field increases, such that a plane that was not discernible due to its location beyond the eye's depth of focus may become discernible and appear more focused as the pupil size decreases, commensurate with the increase in depth of field. Similarly, the number of spaced-apart depth planes used to present different images to the viewer can decrease as pupil size decreases. For example, a viewer may not be able to clearly perceive details in both a first depth plane and a second depth plane at one pupil size without adjusting the eye's accommodation from one depth plane to the other. However, both depth planes may be simultaneously in sufficient focus for a user at another pupil size without changing accommodation.

[0553] In some embodiments, the display system can change the number of waveguides receiving image information based on a determination of pupil size or orientation, or upon receiving an electrical signal indicative of a particular pupil size or orientation. For example, if the user's eye cannot distinguish between two depth planes associated with two waveguides, the controller 460 (which can be an embodiment of the local processing and data module 206) can be configured or programmed to stop providing image information to one of these waveguides. Advantageously, this can reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments where the DOE for a waveguide is switchable between on and off states, the DOE can be switched to the off state when the waveguide does receive image information.

[0554] In some embodiments, it may be desirable to have the exit beam meet the condition of having a diameter smaller than the diameter of the viewer's eye. However, given the variability of viewer pupil size, meeting this condition can be challenging. In some embodiments, by varying the size of the exit beam in response to a determination of the viewer's pupil size, this condition is met over a wide range of pupil sizes. For example, as pupil size decreases, the size of the exit beam may also decrease. In some embodiments, a variable aperture may be used to vary the exit beam size.

[0555] Wearable system 400 may include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of world 470. This portion of world 470 may be referred to as the world camera's field of view (FOV), and imaging system 464 is sometimes referred to as a FOV camera. The world camera's FOV may be the same as or different from the viewer 210's FOV, which encompasses the portion of world 470 perceived by viewer 210 at a given moment. For example, in some cases, the world camera's FOV may be larger than that of viewer 210 of wearable system 400. The total area viewable or imaged by the viewer may be referred to as the field of view (FOR). Because the wearer can move their body, head, or eyes to perceive essentially any direction in space, the FOR may include 4π steradians of solid angle around the wearable system 400. In other cases, the wearer's movement may be more restricted, and accordingly, the wearer's FOR may subtend a smaller solid angle. Images obtained from outward-facing imaging system 464 may be used to track gestures made by the user (eg, hand or finger poses), detect objects in world 470 in front of the user, and so on.

[0556] The wearable system 400 may include an audio sensor 232, such as a microphone, to capture ambient sounds. As described above, in some embodiments, one or more additional audio sensors may be positioned to provide stereo reception useful for determining the location of a speech source. As another example, the audio sensor 232 may include a directional microphone, which can also provide useful directional information regarding the location of an audio source. The wearable system 400 may use information from both the outward-facing imaging system 464 and the audio sensor 230 to locate the source of speech, determine who is speaking at a particular moment, and so on. For example, the wearable system 400 may use voice recognition alone or in combination with a reflected image of the speaker (e.g., a reflected image viewed in a mirror) to determine the speaker's identity. As another example, the wearable system 400 may determine the speaker's location in the environment based on the sound captured by the directional microphone. The wearable system 400 may parse the sound from the speaker's location using a voice recognition algorithm to determine the speech content and use voice recognition technology to determine the speaker's identity (e.g., name or other demographic information).

[0557] Wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera) that observes the user's movements, such as eye and facial movements. Inward-facing imaging system 466 may be used to capture images of eye 410 to determine the size and / or orientation of the pupil of eye 304. Inward-facing imaging system 466 may be used to acquire images used to determine the direction the user is looking (e.g., eye pose) or for biometric identification of the user (e.g., via iris recognition). In some embodiments, at least one camera may be used for each eye to independently determine the pupil size or pose of each eye, allowing the image information presented to each eye to be dynamically tailored to that eye. In some other embodiments, the pupil diameter or orientation of only one eye 410 (e.g., using only one camera for each pair of eyes) is determined and assumed to be similar for both eyes of the user. Images obtained by inward-facing imaging system 466 may be analyzed to determine the user's eye pose or emotion, which the wearable system 400 may use to determine which audio or visual content should be presented to the user. The wearable system 400 may also use sensors such as an IMU, accelerometer, gyroscope, etc. to determine head pose (eg, head position or head orientation).

[0558] Wearable system 400 may include a user input device 466 through which a user can input commands to controller 460 to interact with wearable system 400. For example, user input device 466 may include a trackpad, a touch screen, a joystick, a multi-degree-of-freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, a totem (e.g., used as a virtual user input device), and the like. A multi-DOF controller may sense user input in terms of some or all of the controller's possible translations (e.g., left / right, forward / backward, or up / down) or rotations (e.g., yaw, pitch, or roll). A multi-DOF controller that supports translational motion may be referred to as 3DOF, while a multi-DOF controller that supports both translation and rotation may be referred to as 6DOF. In some cases, a user may use a finger (e.g., a thumb) to press or swipe on a touch-sensitive input device to provide input to wearable system 400 (e.g., to provide user input to a user interface provided by wearable system 400). The user input device 466 can be held by a user's hand during use of the wearable system 400. The user input device 466 can communicate with the wearable system 400 in wired or wireless communication.

[0559] Other components of wearable systems

[0560] In many embodiments, the wearable system may include other components in addition to or instead of the components of the wearable system described above. A wearable system may, for example, include one or more haptic devices or components. A haptic device or component may be used to provide a sense of touch to a user. For example, a haptic device or component may provide a tactile sensation of pressure or texture when touching virtual content (e.g., a virtual object, a virtual tool, other virtual constructs). The sense of touch may replicate the feel of a physical object represented by the virtual object, or may replicate the feel of an imaginary object or character (e.g., a dragon) represented by the virtual content. In some embodiments, the haptic device or component may be wearable by the user (e.g., a glove wearable by the user). In some embodiments, the haptic device or component may be held by the user.

[0561] A wearable system may, for example, include one or more physical objects that can be manipulated by a user to allow input or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as, for example, a metal or plastic block, a wall, or the surface of a table. In some embodiments, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, a totem may simply provide a physical surface, and the wearable system may present a user interface so that it appears to the user to be on one or more surfaces of the totem. For example, a wearable system may make images of a computer keyboard and trackpad appear to reside on one or more surfaces of a totem. For example, a wearable system may make a virtual computer keyboard and virtual trackpad appear to be on the surface of a thin rectangular plate of aluminum that serves as a totem. The rectangular plate itself does not have any physical keys or trackpads or sensors. However, the wearable system may detect user manipulation or interaction or touching of the rectangular plate as a selection or input made via a virtual keyboard or virtual trackpad. User input device 466 (in Figure 4 ) can be an embodiment of a totem, which can include a trackpad, a touchpad, a trigger, a joystick, a trackball, a rocker or virtual switch, a mouse, a keyboard, a multi-degree-of-freedom controller, or another physical input device. A user can use the totem alone or in combination with gestures to interact with the wearable system or other users.

[0562] Examples of haptic devices and totems that may be used with the wearable devices, HMDs, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated herein by reference in its entirety.

[0563] Example of an eye image

[0564] Figure 5An image of an eye 500 is shown having an eyelid 504, a sclera 508 (the "white part" of the eye), an iris 512, and a pupil 516. Curve 516a shows the pupil boundary between the pupil 516 and the iris 512, and curve 512a shows the limbus boundary between the iris 512 and the sclera 508. The eyelid 504 includes an upper eyelid 504a and a lower eyelid 504b. The eye 500 is shown in a natural resting pose (e.g., where the user's face and gaze are both toward a distant object directly in front of the user). The natural resting pose of the eye 500 can be indicated by a natural resting direction 520, which is a direction orthogonal to the surface of the eye 500 when the eye 500 is in the natural resting pose (e.g., from Figure 5 The plane of the eye 500 is shown facing directly outward), in this example, at the center of the pupil 516.

[0565] As the eye 500 moves to look at different objects, the eye pose will change relative to the natural resting direction 520. The current eye pose can be determined with reference to the eye pose direction 524, which is a direction normal to the surface of the eye (and centered within the pupil 516), but towards the object the eye is currently looking at. Figure 5A In the example coordinate system shown, the pose of the eye 500 can be expressed as two angular parameters indicating the azimuthal deflection and zenithal deflection of the eye's eye pose direction 524, both relative to the eye's natural resting direction 520. For illustrative purposes, these angular parameters can be expressed as θ (azimuthal deflection, determined based on a reference orientation) and φ (zenithal deflection, sometimes also referred to as polar deflection). In some embodiments, the angular roll of the eye about the eye pose direction 524 can be included in the determination of the eye pose, and the angular roll can be included in the following analysis. In other embodiments, other techniques for determining eye pose can be used, such as pitch, yaw, and optional roll systems.

[0566] Eye images can be obtained from a video using any suitable process, such as using a video processing algorithm that can extract images from one or more consecutive frames. Eye pose can be determined from the eye images using various eye tracking techniques. For example, eye pose can be determined by considering the lens effect of the cornea on a provided light source. In the eyelid shape estimation techniques described herein, any suitable eye tracking technique can be used to determine eye pose.

[0567] Example of an eye tracking system

[0568] Figure 6A schematic diagram of a wearable system 600 including an eye tracking system is shown. In at least some embodiments, the wearable system 600 can include components located in a head-mounted unit 602 and components located in a non-head-mounted unit 604. The non-head-mounted unit 604 can be, for example, a strap-mounted unit, a handheld unit, a component in a backpack, a remote unit, etc. Including certain components of the wearable system 600 in the non-head-mounted unit 604 can help reduce the size, weight, complexity, and cost of the head-mounted unit 602. In some embodiments, some or all of the functionality described as being performed by one or more components of the head-mounted unit 602 and / or the non-head-mounted unit 604 can be provided by one or more components included elsewhere in the wearable system 600. For example, some or all of the functionality associated below with the CPU 612 of the head-mounted unit 602 can be provided by the CPU 616 of the non-head-mounted unit 604, and vice versa. In some examples, some or all of such functionality can be provided by peripheral devices of the wearable system 600. Furthermore, in some embodiments, some or all of such functionality may be provided by one or more cloud computing devices or other remotely located computing devices, similar to the above referenced Figure 2 Described in the manner provided.

[0569] like Figure 6 As shown, wearable system 600 may include an eye tracking system including a camera 324 that captures images of a user's eyes 610. If desired, the eye tracking system may also include light sources 326a and 326b (such as light emitting diodes (LEDs)). Light sources 326a and 326b may generate bright spots (e.g., reflections from the user's eyes that appear in the eye images captured by camera 324). The positions of light sources 326a and 326b relative to camera 324 may be known, so the positions of the bright spots within the images captured by camera 324 may be used to track the user's eyes (as discussed in more detail below in conjunction with Figures 7 to 11). In at least one embodiment, there may be one light source 326 and one camera 324 associated with a single one of the user's eyes 610. In another embodiment, there may be one light source 326 and one camera 324 associated with each of the user's eyes 610. In yet other embodiments, there may be one or more cameras 324 and one or more light sources 326 associated with one or each of the user's eyes 610. As a specific example, there may be two light sources 326a and 326b and one or more cameras 324 associated with each of the user's eyes 610. As another example, there may be three or more light sources (such as light sources 326a and 326b) and one or more cameras 324 associated with each of the user's eyes 610.

[0570] The eye tracking module 614 can receive images from the eye tracking camera 324 and can analyze the images to extract various pieces of information. As examples, the eye tracking module 614 can detect the user's eye pose, the three-dimensional position of the user's eyes relative to the eye tracking camera 324 (and relative to the head mounted unit 602), the focus direction of one or both of the user's eyes 610, the user's depth of convergence (e.g., the depth relative to the user that the user is focusing), the position of the user's pupils, the position of the user's cornea and corneal sphere, the center of rotation of each of the user's eyes, and the perspective center of each of the user's eyes. The eye tracking module 614 can extract this information using the techniques described below in conjunction with Figures 7 through 11. Figure 6 As shown, the eye tracking module 614 can be a software module implemented using the CPU 612 in the head-mounted unit 602.

[0571] Data from the eye tracking module 614 can be provided to other components in the wearable system. For example, such data can be transmitted to components in the non-head-mounted unit 604, such as the CPU 616, which includes software modules for the light field rendering controller 618 and the registration observer 620.

[0572] The rendering controller 618 can use information from the eye tracking module 614 to adjust the image displayed to the user by the rendering engine 622 (e.g., a rendering engine that can be a software module in the GPU 620 and can provide images to the display 220). As an example, the rendering controller 618 can adjust the image displayed to the user based on the user's rotation center or perspective center. Specifically, the rendering controller 618 can use information about the user's perspective center to simulate a rendering camera (e.g., simulate collecting images from the user's perspective) and can adjust the image displayed to the user based on the simulated rendering camera.

[0573] A "rendering camera," sometimes also referred to as a "pinhole perspective camera" (or simply "perspective camera") or a "virtual pinhole camera" (or simply "virtual camera"), is a simulated camera used to render virtual image content, which may be drawn from a database of objects in a virtual world. Objects may have positions and orientations relative to the user or wearer, and possibly relative to real objects in the user's or wearer's surrounding environment. In other words, the rendering camera may represent a perspective within a rendering space from which the user or wearer views 3D virtual content (e.g., virtual objects) in the rendering space. The rendering camera may be managed by a rendering engine to render virtual images based on a database of virtual objects to be presented to the eye. Virtual images may be rendered as if captured from the user's or wearer's perspective. For example, virtual images may be rendered as if captured by a pinhole camera (corresponding to the "rendering camera") having a specific set of intrinsic parameters (e.g., focal length, camera pixel size, principal point coordinates, skew / distortion parameters, etc.) and a specific set of extrinsic parameters (e.g., translation and rotation components relative to the virtual world). The virtual image is captured from the perspective of such a camera having a position and orientation of the rendering camera (e.g., extrinsic parameters of the rendering camera). Thus, the system can define and / or adjust intrinsic and extrinsic rendering camera parameters. For example, the system can define a specific set of extrinsic rendering camera parameters such that the virtual image is rendered as if captured from the perspective of a camera having a specific position relative to the user's or wearer's eyes, thereby providing an image that appears to be from the user's or wearer's perspective. The system can then dynamically adjust the extrinsic rendering camera parameters on the fly to maintain registration with the specific position. Similarly, intrinsic rendering camera parameters can be defined and dynamically adjusted over time. In some embodiments, the image is rendered as if captured from the perspective of a camera having an aperture (e.g., a pinhole) located at a specific position relative to the user's or wearer's eyes (e.g., a perspective center or a rotational center or other position).

[0574] In some embodiments, the system can create or dynamically reposition and / or reorient one rendering camera for the user's left eye and another rendering camera for the user's right eye, since the user's eyes are physically separated from each other and therefore always positioned at different locations. Thus, in at least some embodiments, virtual content rendered from the perspective of the rendering camera associated with the viewer's left eye can be presented to the user through an eyepiece on the left side of a head-mounted display (e.g., head-mounted unit 602), and virtual content rendered from the perspective of the rendering camera associated with the viewer's right eye can be presented to the user through an eyepiece on the right side of such a head-mounted display. Further details discussing the creation, adjustment, and use of rendering cameras in the rendering process are provided in U.S. patent application Ser. No. 15 / 274,823, entitled “METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTURAL FEATURES IN 3D RECONSTRUCTION,” the entire contents of which are expressly incorporated herein by reference for all purposes.

[0575] In some examples, one or more modules (or components) of system 600 (e.g., light field rendering controller 618, rendering engine 620, etc.) can determine the position and orientation of a rendering camera within a rendering space based on the position and orientation of a user's head and eyes (e.g., determined based on head pose and eye tracking data, respectively). That is, system 600 can effectively map the position and orientation of the user's head and eyes to a specific position and angular orientation within the 3D virtual environment, place and orient the rendering camera at the specific position and angular orientation within the 3D virtual environment, and render virtual content for the user as if captured by the rendering camera. Further details discussing the real-world to virtual-world mapping process are provided in U.S. patent application Ser. No. 15 / 296,869, entitled “SELECTING VIRTUAL OBJECTS IN A THREE-DIMENSIONAL SPACE,” the entire contents of which are expressly incorporated herein by reference for all purposes. As an example, the rendering controller 618 can adjust the depth at which an image is displayed by selecting which depth plane (or depth planes) to use to display the image at any given time. In some embodiments, this depth plane switching can be performed by adjusting one or more intrinsic rendering camera parameters. For example, when performing depth plane switching or adjustment, the light field rendering controller 618 can adjust the focal length of the rendering camera. As described in further detail below, the depth plane can be switched based on the user's determined vergence or gaze depth.

[0576] The registration observer 620 can use information from the eye tracking module 614 to identify whether the head mounted unit 602 is correctly positioned on the user's head. As an example, the eye tracking module 614 can provide eye position information, such as the location of the center of rotation of the user's eyes, which indicates the three-dimensional position of the user's eyes relative to the camera 324 and the head mounted unit 602. The eye tracking module 614 can use this position information to determine whether the display 220 is correctly aligned in the user's field of view, or whether the head mounted unit 602 (or headset) has slipped off or is otherwise misaligned with the user's eyes. By way of example, the registration observer 620 can determine whether: the head mounted unit 602 has slid down the bridge of the user's nose, thereby moving the display 220 away from the user's eyes and downward (which is undesirable); whether the head mounted unit 602 has moved up along the bridge of the user's nose, thereby moving the display 220 upward closer to the user's eyes; whether the head mounted unit 602 has moved left or right relative to the bridge of the user's nose; whether the head mounted unit 602 has risen above the bridge of the user's nose; or whether the head mounted unit 602 has been moved from a desired position or range of positions in these or other ways. In general, the registration observer 620 can determine whether the head mounted unit 602 in general, and the display 220 in particular, are properly positioned in front of the user's eyes. In other words, the registration observer 620 can determine whether the left display in the display system 220 is properly aligned with the user's left eye, and whether the right display in the display system 220 is properly aligned with the user's right eye. The registration observer 620 can determine whether the head mounted unit 602 is properly positioned by determining whether the head mounted unit 602 is positioned and oriented within a desired range of positions and / or orientations relative to the user's eyes.

[0577] In at least some embodiments, the registration viewer 620 can generate user feedback in the form of alerts, messages, or other content. This feedback can be provided to the user to inform the user of any misalignment of the head mounted unit 602, as well as optional feedback on how to correct the misalignment (such as suggestions for adjusting the head mounted unit 602 in a particular manner).

[0578] Example registration observation and feedback techniques that may be used by the registration observer 620 are described in U.S. patent application Ser. No. 15 / 717,747, filed on Sep. 27, 2017 (Attorney Docket No. MLEAP.052A2), and U.S. Provisional Patent Application Ser. No. 62 / 644,321, filed on Mar. 16, 2018 (Attorney Docket No. MLEAP.195PR), both of which are incorporated herein by reference in their entireties.

[0579] Example of the eye tracking module

[0580] A detailed block diagram of an example eye tracking module 614 is provided in Figure 7A As shown in Figure 7A As shown, eye tracking module 614 can include various submodules, can provide various outputs, and can utilize various available data to track the user's eyes. By way of example, eye tracking module 614 can utilize available data including: eye tracking extrinsic and intrinsic features, such as the geometric arrangement of eye tracking camera 324 relative to light source 326 and head-mounted unit 602; assumed eye size 704, such as a typical distance of approximately 4.7 mm between the center of corneal curvature of a user and the average center of rotation of the user's eye, or a typical distance between the center of rotation and the center of perspective of a user; and per-user calibration data 706, such as the pupil distance of a particular user. Additional examples of extrinsic features, intrinsic features, and other information that can be utilized by eye tracking module 614 are described in U.S. patent application Ser. No. 15 / 497,726, filed on April 26, 2017 (Attorney Docket No. MLEAP.023A7), the entire contents of which are incorporated herein by reference.

[0581] The image pre-processing module 710 may receive an image from an eye camera (such as the eye camera 324) and may perform one or more pre-processing (e.g., conditioning) operations on the received image. By way of example, the image pre-processing module 710 may apply a Gaussian blur to the image, may downsample the image to a lower resolution, may apply an unsharp mask, may apply an edge sharpening algorithm, or may apply other suitable filters that facilitate later detection, location, and labeling of bright spots, pupils, or other features in the image from the eye camera 324. The image pre-processing module 710 may apply a low-pass filter or a morphological filter (e.g., an open filter) that may remove artifacts such as those from the pupil boundary 516a (see FIG. Figure 5 ), thereby removing noise that may hinder pupil and bright spot determination. The image preprocessing module 710 can output the preprocessed image to the pupil recognition module 712 and the bright spot detection and marking module 714.

[0582] The pupil identification module 712 can receive pre-processed images from the image pre-processing module 710 and can identify regions in these images that include the user's pupils. In some embodiments, the pupil identification module 712 can determine the coordinates of the location of the user's pupils or the coordinates of their centers or centroids in the eye-tracking images from the camera 324. In at least some embodiments, the pupil identification module 712 can identify contours (e.g., the contours of the pupil-iris boundary) in the eye-tracking images, identify contour moments (e.g., the centroid), apply starburst pupil detection and / or Canny edge detection algorithms, discard outliers based on intensity values, identify sub-pixel boundary points, correct for eye camera distortion (e.g., distortion in images captured by the eye camera 324), apply a random sample consensus (RANSAC) iterative algorithm to fit an ellipse to the boundary in the eye-tracking images, apply a tracking filter to the images, and identify sub-pixel image coordinates of the centroid of the user's pupils. The pupil identification module 712 can output pupil identification data to the bright spot detection and labeling module 714. The pupil identification data can indicate which areas of the image pre-processing module 712 were identified as displaying the user's pupils. The pupil identification module 712 can provide the 2D coordinates of the user's pupils (e.g., the 2D coordinates of the centroid of the user's pupils) in each eye-tracking image to the bright spot detection module 714. In at least some embodiments, the pupil identification module 712 can also provide the same type of pupil identification data to the coordinate system normalization module 718.

[0583] Pupil detection technology that can be utilized by pupil identification module 712 is described in U.S. Patent Publication No. 2017 / 0053165, published on February 23, 2017, and U.S. Patent Publication No. 2017 / 0053166, published on February 23, 2017, the entire contents of each of which are incorporated herein by reference.

[0584] Blob detection and marking module 714 can receive the pre-processed image from module 710 and pupil identification data from module 712. Blob detection module 714 can use this data to detect and / or identify blob (e.g., reflections of light from light source 326 from the user's eye) within the area of ​​the pre-processed image that displays the user's pupil. By way of example, blob detection module 714 can search for bright areas within the eye-tracking image, sometimes referred to herein as "blobs," or local intensity maxima, that are located near the user's pupil. In at least some embodiments, blob detection module 714 can rescale (e.g., enlarge) the pupil ellipse to include the additional blob. Blob detection module 714 can filter blob by size and / or intensity. Blob detection module 714 can also determine the 2D position of each blob within the eye-tracking image. In at least some examples, blob detection module 714 can determine the 2D position of the blob relative to the user's pupil, which can also be referred to as a pupil-blob vector. The blotch detection and marking module 714 can mark the blotches and output a pre-processed image with the marked blotches to the 3D cornea center estimation module 716. The blotch detection and marking module 714 can also pass data, such as the pre-processed image from module 710 and pupil identification data from module 712. In some embodiments, the blotch detection and marking module 714 can determine which light source (e.g., from a system of multiple light sources including infrared light sources 326a and 326b) produced each identified blotch. In these examples, the blotch detection and marking module 714 can mark the blotches with information identifying the associated light source and output a pre-processed image with the marked blotches to the 3D cornea center estimation module 716.

[0585] The pupil and spot detection performed by modules such as modules 712 and 714 may use any suitable technique. As an example, edge detection may be applied to an eye image to identify spots and pupils. Edge detection may be applied using various edge detectors, edge detection algorithms, or filters. For example, a Canny edge detector may be applied to an image to detect edges such as in lines in the image. An edge may include points along a line corresponding to local maximum derivatives. For example, a Canny edge detector may be used to locate the pupil boundary 516a (see Figure 5). Once the pupil position is determined, various image processing techniques can be used to detect the "pose" of the pupil 116. Determining the eye pose of an eye image can also be referred to as detecting the eye pose of an eye image. The pose can also be referred to as the gaze, pointing direction, or orientation of the eye. For example, the pupil may be looking at an object to the left, and the pose of the pupil can be classified as a left pose. Other methods can be used to detect the position of the pupil or bright spot. For example, a Canny edge detector can be used to locate concentric rings in the eye image. As another example, an integral-differential operator can be used to find the pupil or corneal limbus boundary of the iris. For example, a Daugman integral-differential operator, a Hough transform, or other iris segmentation techniques can be used to return a curve that estimates the boundary of the pupil or iris.

[0586] The 3D cornea center estimation module 716 may receive pre-processed images including detected bright spot data and pupil identification data from modules 710, 712, 714. The 3D cornea center estimation module 716 may use this data to estimate the 3D position of the user's cornea. In some embodiments, the 3D cornea center estimation module 716 may estimate the 3D position of the center of corneal curvature of the eye or the user's corneal sphere, for example, the center of an imaginary sphere whose surface portion is approximately coextensive with the user's cornea. The 3D cornea center estimation module 716 may provide data indicating the estimated 3D coordinates of the corneal sphere and / or the user's cornea to the coordinate system normalization module 718, the optical axis determination module 722, and / or the light field rendering controller 618. Further operational details of the 3D cornea center estimation module 716 are provided herein in conjunction with Figures 8A to 8E Techniques for estimating the position of eye features (e.g., the cornea or corneal sphere) that can be utilized by the 3D cornea center estimation module 716 and other modules in the wearable system of the present disclosure are discussed in U.S. patent application Ser. No. 15 / 497,726 (Attorney Docket No. MLEAP.023A7), filed on April 26, 2017, the entire contents of which are incorporated herein by reference.

[0587] A coordinate system normalization module 718 can optionally be included in the eye tracking module 614 (as shown by its dashed box). The coordinate system normalization module 718 can receive data indicating the estimated 3D coordinates of the user's corneal center (and / or the user's corneal sphere center) from the 3D corneal center estimation module 716, and can also receive data from other modules. The coordinate system normalization module 718 can normalize the eye camera coordinate system, which helps compensate for slippage of the wearable device (e.g., the head-mounted component slips from its normal resting position on the user's head, which can be identified by the registration observer 620). The coordinate system normalization module 718 can rotate the coordinate system to align the z-axis of the coordinate system (e.g., the convergence depth axis) with the center of the cornea (e.g., as indicated by the 3D cornea center estimation module 716), and can translate the camera center (e.g., the origin of the coordinate system) to a predetermined distance away from the center of the cornea, such as 30 mm (e.g., the module 718 can zoom in or out of the eye tracking image depending on whether the eye camera 324 is determined to be closer or farther than the predetermined distance). Through this normalization process, the eye tracking module 614 is able to establish consistent orientation and distance in the eye tracking data relatively independent of changes in the positioning of the headset on the user's head. The coordinate system normalization module 718 can provide the 3D coordinates of the center of the cornea (and / or corneal sphere), pupil identification data, and pre-processed eye tracking images to the 3D pupil center locator module 720. Further operational details of the coordinate system normalization module 718 are incorporated herein by reference. Figures 9A to 9C supply.

[0588] The 3D pupil center locator module 720 can receive data in a normalized or unnormalized coordinate system, the data including the 3D coordinates of the center of the user's cornea (and / or corneal sphere), pupil position data, and pre-processed eye tracking images. The 3D pupil center locator module 720 can analyze such data to determine the 3D coordinates of the user's pupil center in a normalized or unnormalized eye camera coordinate system. The 3D pupil center locator module 720 can determine the position of the user's pupil in three dimensions based on the 2D position of the pupil centroid (as determined by module 712), the 3D position of the corneal center (as determined by module 716), the assumed eye size 704 (e.g., the size of a typical user's corneal sphere and the typical distance from the corneal center to the pupil center), and the optical properties of the eye (e.g., the corneal refractive index (relative to the air refractive index)), or any combination of these. Further operational details of the 3D pupil center locator module 720 are incorporated herein by reference. Figures 9D to 9GTechniques for estimating the position of eye features (e.g., pupils) that can be utilized by the 3D pupil center locator module 720 and other modules in the wearable system of the present disclosure are discussed in U.S. patent application Ser. No. 15 / 497,726 (Attorney Docket No. MLEAP.023A7), filed on April 26, 2017, the entire contents of which are incorporated herein by reference.

[0589] The optical axis determination module 722 can receive data indicating the 3D coordinates of the center of the user's cornea and the user's pupil from modules 716 and 720. Based on this data, the optical axis determination module 722 can identify a vector from the location of the corneal center (e.g., from the center of the corneal sphere) to the center of the user's pupil, which can define the optical axis of the user's eye. As an example, the optical axis determination module 722 can provide output specifying the user's optical axis to modules 724, 728, 730, and 732.

[0590] Center of rotation (CoR) estimation module 724 can receive data from module 722, including parameters of the optical axis of the user's eye (e.g., data indicating the orientation of the optical axis in a coordinate system, which orientation has a known relationship with the head-mounted unit 602). For example, CoR estimation module 724 can estimate the center of rotation of the user's eye. The center of rotation can indicate the point about which the user's eye rotates when the user's eye rotates left, right, up, and / or down. While the eye may not rotate perfectly around a singular point, assuming a singular point is sufficient. In at least some embodiments, CoR estimation module 724 can estimate the center of rotation of the eye by moving a specific distance along the optical axis (identified by module 722) from the center of the pupil (identified by module 720) or the center of corneal curvature (identified by module 716) toward the retina. The specific distance can be the assumed eye size 704. As an example, the specific distance between the center of corneal curvature and the CoR can be approximately 4.7 mm. This distance can be modified for a specific user based on any relevant data, including the user's age, gender, vision prescription, other relevant characteristics, etc. Further discussion of the value of 4.7 mm as an estimate of the distance between the center of corneal curvature and the CoR is provided in the Appendix (Section 3) forming a part of this application.

[0591] In at least some embodiments, the CoR estimation module 724 can refine its estimate of the center of rotation of each of the user's eyes over time. As an example, over time, the user eventually rotates their eyes (to look elsewhere, at objects that are closer or farther away, or to the left, right, up, or down at some point), causing the optical axis of each of their eyes to shift. The CoR estimation module 724 then analyzes the two (or more) optical axes identified by module 722 and locates the 3D intersection of these optical axes. The CoR estimation module 724 can then determine that the center of rotation is located at this 3D intersection. This technique can provide an estimate of the center of rotation whose accuracy improves over time.

[0592] Various techniques can be employed to increase the accuracy of the CoR estimation module 724 and the determined CoR positions for the left and right eyes. As an example, the CoR estimation module 724 can estimate the CoR by finding the average intersection of the optical axes determined over time for various eye poses. As additional examples, the module 724 can filter or average the estimated CoR positions over time, can calculate a moving average of the estimated CoR positions over time, and / or can apply a Kalman filter and the known dynamics of the eyes and eye tracking system to estimate the CoR positions over time. In some embodiments, a least squares method can be employed to determine one or more intersections of the optical axes. In such embodiments, the system can identify, at a given point in time, a location where the sum of the squared distances to a given set of optical axes is minimized or minimized as the intersection of the optical axes. As a specific example, module 724 can calculate a weighted average of the determined optical axis intersection and an assumed CoR position (e.g., 4.7 mm from the center of corneal curvature of the eye) such that the determined CoR can slowly drift over time from the assumed CoR position (e.g., 4.7 mm behind the center of corneal curvature of the eye) to a slightly different position within the user's eye as eye tracking data is obtained for that user, thereby enabling per-user refinement of the CoR position.

[0593] Under ideal conditions, as the user moves their eyes (e.g., as the user's eyes rotate about their center of rotation), the 3D position of the user's eyes' true CoR relative to the HMD should change over time by a negligible or minimal amount. In other words, for a given set of eye movements, the 3D position of the user's eyes' true CoR (relative to the HMD) should hypothetically change less over time than any other point along the optical axis of the user's eyes. Thus, it follows that as the user moves their eyes, the further a point along the optical axis is from the user's eyes' true CoR, the greater the variation or variance in its 3D position over time. In some embodiments, the CoR estimation module 724 and / or other submodules of the eye tracking module 614 can exploit this statistical relationship to improve CoR estimation accuracy. In such embodiments, the CoR estimation module 724 and / or other submodules of the eye tracking module 614 can refine its estimate of the CoR 3D position over time by identifying changes in its CoR estimate that have low variation (e.g., low variance or standard deviation).

[0594] As a first example, and in embodiments where the CoR estimation module 724 estimates the CoR based on the intersection of multiple different optical axes (each optical axis associated with a different direction the user is looking), the CoR estimation module 724 can exploit this statistical relationship (i.e., that the true CoR should have low variance) by introducing a common offset in the direction of each optical axis (e.g., offsetting each axis by some same amount) and determining whether the offset optical axes intersect each other in an intersection point with low variation (e.g., low variance or standard deviation). This can correct for small systematic errors in the calculation of the optical axis directions and help refine the estimated position of the CoR to be closer to the true CoR.

[0595] As a second example, and in embodiments in which the CoR estimation module 724 estimates the CoR by moving a particular distance (e.g., the distance between the center of corneal curvature and the CoR) along the optical axis (or other axis), the system can change, optimize, modulate, or otherwise adjust the particular distance between the center of corneal curvature and the CoR over time (e.g., for a large set of eye images captured at different times) in a manner that reduces or minimizes variation (e.g., variance and / or standard deviation) in the estimated CoR position. For example, if the CoR estimation module 724 initially uses a particular distance value of 4.7 mm (along the optical axis relative to the center of corneal curvature) to obtain CoR position estimates, but the true CoR of a given user's eye may be located 4.9 mm behind the center of corneal curvature of the eye (along the optical axis), then the initial set of CoR position estimates obtained by the CoR estimation module 724 may exhibit a relatively high amount of variation (e.g., variance or standard deviation). In response to detecting this relatively high amount of variation (e.g., variance or standard deviation), the CoR estimation module 724 can seek and identify one or more points along the optical axis having a lower amount of variation (e.g., variance or standard deviation), can identify the 4.9 mm distance as having the lowest amount of variation (e.g., variance or standard deviation), and can thereby adjust the particular distance value utilized to 4.9 mm.

[0596] The CoR estimation module 724 may seek an alternative CoR estimate with lower variance (e.g., variance and / or standard deviation) in response to detecting that the current CoR estimate has a relatively high variance (e.g., variance or standard deviation), or may seek an alternative CoR estimate with lower variance (e.g., variance or standard deviation) as usual after obtaining an initial CoR estimate. In some examples, such optimization / adjustment may occur gradually over time, while in other examples, such optimization / adjustment may be performed during an initial user calibration session. In examples where such a process is performed during calibration, the CoR estimation module 724 may not initially prescribe / adhere to any assumed specific distance, but may instead collect a set of eye tracking data over time, perform a statistical analysis on the set of eye tracking data, and determine a specific distance value based on the statistical analysis that produces a CoR position estimate with the smallest (e.g., global minimum) possible variance (e.g., variance or standard deviation).

[0597] The Appendix (Section III), which forms a part of this application, provides further discussion of the statistical relationships discussed above (eg, that the true CoR should have a low variance or standard deviation), and the importance of considering corneal refraction when determining pupil position.

[0598] The interpupillary distance (IPD) estimation module 726 may receive data indicating the estimated 3D positions of the centers of rotation of the user's left and right eyes from the CoR estimation module 724. The IPD estimation module 726 may then estimate the user's IPD by measuring the 3D distance between the centers of rotation of the user's left and right eyes. Typically, when the user is looking at optical infinity (e.g., the optical axes of the user's eyes are substantially parallel to each other), the distance between the estimated CoR of the user's left eye and the estimated CoR of the user's right eye may be approximately equal to the distance between the centers of the user's pupils, which is the typical definition of interpupillary distance (IPD). The user's IPD may be used by various components and modules in the wearable system. For example, the user's IPD may be provided to the registration observer 620 and used to assess the degree to which the wearable device is aligned with the user's eyes (e.g., whether the left and right display lenses are correctly spaced according to the user's IPD). As another example, the user's IPD may be provided to the vergence depth estimation module 728 and used to determine the user's vergence depth. Module 726 may employ various techniques, such as those discussed in conjunction with CoR estimation module 724, to improve the accuracy of the estimated IPD. As an example, IPD estimation module 724 may apply filtering, averaging over time, weighted averaging including hypothesized IPD distances, a Kalman filter, etc., as part of estimating the user's IPD in an accurate manner.

[0599] The convergence depth estimation module 728 can be used from the various modules and submodules in the eye tracking module 614 (such as the combination of Figure 7ASpecifically, the vergence depth estimation module 728 may employ data indicating an estimated 3D position of a pupil center (e.g., provided by the module 720 described above), one or more determined parameters of an optical axis (e.g., provided by the module 722 described above), an estimated 3D position of a rotation center (e.g., provided by the module 724 described above), an estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the rotation centers) (e.g., provided by the module 726 described above), and / or one or more determined parameters of an optical axis and / or visual axis (e.g., provided by the module 722 and / or the module 730 described below). The vergence depth estimation module 728 may detect or otherwise obtain a vergence depth metric for the user, which may be a distance relative to the user at which the user's eyes are focused. For example, when a user is looking at an object three feet in front of them, the user's left and right eyes have a vergence depth of three feet; and when the user is looking at a distant scene (e.g., the optical axes of the user's eyes are substantially parallel to each other, such that the distance between the centers of the user's pupils is approximately equal to the distance between the centers of rotation of the user's left and right eyes), the user's left and right eyes have a vergence depth of infinite distance. In some embodiments, the vergence depth estimation module 728 can use data indicating the estimated centers of the user's pupils (e.g., provided by module 720) to determine the 3D distance between the estimated centers of the user's pupils. The vergence depth estimation module 728 can obtain a measure of vergence depth by comparing this determined 3D distance between the pupil centers with an estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the centers of rotation) (e.g., as indicated by module 726 above). In addition to the 3D distance between the pupil centers and the estimated IPD, the depth of convergence estimation module 728 can also calculate the depth of convergence using known, assumed, estimated, and / or determined geometry. As an example, the module 728 can combine the 3D distance between the pupil centers, the estimated IPD, and the 3D CoR position in a triangulation calculation to estimate (e.g., determine) the user's depth of convergence. In practice, calculating this determined 3D distance between the pupil centers for the estimated IPD can be used to indicate a measure of the user's current depth of convergence relative to optical infinity. In some examples, the depth of convergence estimation module 728 can simply receive or access data indicating the estimated 3D distance between the user's pupil centers in order to obtain this measure of depth of convergence. In some embodiments, the depth of convergence estimation module 728 can estimate the depth of convergence by comparing the user's left and right optical axes. Specifically, the depth of convergence estimation module 728 can estimate the depth of convergence by locating the distance relative to the user where the user's left and right optical axes intersect (or the projections of the user's left and right optical axes on a plane (e.g., a horizontal plane) intersect). By setting the zero depth to the depth where the user's left and right optical axes are separated by the user's IPD, the module 728 can utilize the user's IPD in this calculation.In at least some embodiments, the vergence depth estimation module 728 can determine the vergence depth by triangulating the eye tracking data with known or derived spatial relationships.

[0600] In some embodiments, the vergence depth estimation module 728 can estimate the user's vergence depth based on the intersection of the user's visual axis (rather than their optical axis), which can provide a more accurate indication of the distance the user is focused on. In at least some embodiments, the eye tracking module 614 can include an optical axis to visual axis mapping module 730. Figure 10 As discussed in more detail, a user's optical axis and visual axis are typically not aligned. The visual axis is the axis along which a person gazes, and the optical axis is defined by the center of a person's lens and pupil and may pass through the center of a person's retina. In particular, a user's visual axis is typically defined by the location of the user's fovea, which may be offset from the center of the user's retina, resulting in different optical and visual axes. In at least some of these embodiments, the eye tracking module 614 may include an optical axis to visual axis mapping module 730. The optical axis to visual axis mapping module 730 may correct for the difference between the user's optical axis and visual axis and provide information about the user's visual axis to other components in the wearable system (e.g., the convergence depth estimation module 728 and the light field rendering controller 618). In some examples, the module 730 may use an assumed eye size 704 that includes a typical offset between the optical axis and the visual axis of approximately 5.2° inward (nose-wise, toward the user's nose). In other words, module 730 may shift the user's left optical axis by 5.2° to the right (towards the nose) of the nose and the user's right optical axis by 5.2° to the left (towards the nose) of the nose in order to estimate the direction of the user's left and right optical axes. In other examples, module 730 may utilize per-user calibration data 706 when mapping the optical axis (e.g., as indicated by module 722 above) to the visual axis. As an additional example, module 730 may shift the user's optical axis by 4.0° to 6.5°, 4.5° to 6.0°, 5.0° to 5.4°, etc., toward the nose, or any range formed by any of these values. In some arrangements, module 730 may apply the shift based at least in part on characteristics of a particular user (e.g., their age, gender, vision prescription, or other relevant characteristics), and / or may apply the shift based at least in part on a calibration process for a particular user (e.g., determining the optical axis-visual axis offset for a particular user). In at least some embodiments, module 730 may also shift the origins of the left and right optical axes to correspond to the user's CoP (determined by module 732) rather than the user's CoR.

[0601] When the optional center of perspective (CoP) estimation module 732 is provided, the CoP estimation module 732 can estimate the locations of the user's left and right centers of perspective (CoP). The CoP can be a useful location for a wearable system, and in at least some embodiments, the CoP is a location directly in front of the pupil. In at least some embodiments, the CoP estimation module 732 can estimate the locations of the user's left and right centers of perspective based on the 3D location of the user's pupil center, the 3D location of the user's center of corneal curvature, or such suitable data, or any combination thereof. By way of example, the user's CoP can be approximately 5.01 mm in front of the center of corneal curvature (e.g., 5.01 mm from the center of the corneal sphere in a direction toward the cornea of ​​the eye and along the optical axis) and can be approximately 2.97 mm behind the outer surface of the user's cornea along the optical or visual axis. The user's center of perspective can be directly in front of the center of their pupil. For example, the user's CoP can be less than approximately 2.0 mm from the user's pupil, less than approximately 1.0 mm from the user's pupil, less than approximately 0.5 mm from the user's pupil, or any range between any of these values. As another example, the perspective center can correspond to a location within the anterior chamber of the eye. As other examples, the CoP can be between 1.0 mm and 2.0 mm, approximately 1.0 mm, between 0.25 mm and 1.0 mm, between 0.5 mm and 1.0 mm, or between 0.25 mm and 0.5 mm.

[0602] The perspective center described herein (which may be the desired location of the rendering camera pinhole and the anatomical location in the user's eye) can be a location for reducing and / or eliminating undesirable parallax shifts. Specifically, the optical system of the user's eye is closely equivalent to a theoretical system formed by a pinhole in front of a lens projected onto a screen, where the pinhole, lens, and screen roughly correspond to the user's pupil / iris, lens, and retina, respectively. Furthermore, when two point light sources (or objects) at different distances from the user's eye are rotated strictly around the pinhole opening (e.g., along a radius of curvature equal to the distance of each point light source relative to the pinhole opening), it is desirable to have little or no parallax shift. Therefore, it seems that the CoP should be located at the center of the eye's pupil (and such a CoP can be used in some embodiments). However, in addition to the lens and the pinhole in the pupil, the human eye also includes the cornea, which imparts additional optical power to light propagating toward the retina. Therefore, in the theoretical system described in this paragraph, the anatomical equivalent of the pinhole can be the area of ​​the user's eye located between the outer surface of the cornea of ​​the user's eye and the center of the pupil or iris of the user's eye. For example, the anatomical equivalent of the pinhole may correspond to an area within the anterior chamber of the user's eye. For various reasons discussed herein, it is desirable to set the CoP to such a location within the anterior chamber of the user's eye. Figures 22 to 24B The derivation and importance of CoP are described in more detail.

[0603] As described above, the eye tracking module 614 can provide data such as the estimated 3D position of the left and right eye centers of rotation (CoR), the convergence depth, the left and right eye optical axes, the 3D position of the user's eyes, the 3D position of the user's left and right centers of corneal curvature, the 3D position of the user's left and right pupil centers, the 3D position of the user's left and right perspective centers, the user's IPD, etc. to other components in the wearable system, such as the light field rendering controller 618 and the registration observer 620. The eye tracking module 614 can also include other submodules that detect and generate data associated with other aspects of the user's eyes. As an example, the eye tracking module 614 can include a blink detection module that provides a flag or other alert each time the user blinks and a glance detection module that provides a flag or other alert each time the user's eyes glance (e.g., quickly move the focus to another point).

[0604] Rendering Controller Example

[0605] A detailed block diagram of an example light field rendering controller 618 is provided in Figure 7B As shown in Figure 6 and 7B As shown, the rendering controller 618 can receive eye tracking information from the eye tracking module 614 and can provide output to the rendering engine 622, which can generate an image to be displayed for viewing by a user of the wearable system. As an example, the rendering controller 618 can receive the depth of convergence, the center of rotation of the left and right eyes (and / or the center of perspective), and other eye data, such as blink data, saccade data, etc.

[0606] The depth plane selection module 750 can receive the vergence depth information and other eye data and, based on this data, can cause the rendering engine 622 to deliver content to the user through a specific depth plane (e.g., at a specific adjustment or focal length). Figure 4As discussed, the wearable system can include multiple discrete depth planes formed by multiple waveguides, each depth plane transmitting image information at a varying level of wavefront curvature. In some embodiments, the wearable system can include one or more variable depth planes, such as optical elements that transmit image information at a time-varying level of wavefront curvature. In these and other embodiments, the depth plane selection module 750 can cause the rendering engine 622 to transmit content to the user at a selected depth based in part on the user's vergence depth (e.g., cause the rendering engine 622 to instruct the display 220 to switch depth planes). In at least some embodiments, the depth plane selection module 750 and the rendering engine 622 can render content at different depths and also generate depth plane selection data and / or provide the depth plane selection data to display hardware, such as the display 220. Display hardware, such as the display 220, can perform electrical depth plane switching in response to the depth plane selection data (which can be a control signal) generated and / or provided by modules such as the depth plane selection module 750 and the rendering engine 622.

[0607] Generally, it is desirable for the depth plane selection module 750 to select a depth plane that matches the user's current depth of convergence in order to provide the user with accurate accommodation cues. However, it is also desirable to switch depth planes in a discreet and unobtrusive manner. For example, it is desirable to avoid excessive switching between depth planes and / or to switch depth planes when the user is unlikely to notice the switch (e.g., during a blink or eye saccade).

[0608] The cross-hysteresis band detection module 752 can help avoid excessive switching between depth planes, particularly when the user's depth of convergence fluctuates at a midpoint or transition point between two depth planes. Specifically, the module 752 can cause the depth plane selection module 750 to exhibit hysteresis in its depth plane selection. As an example, the module 752 can cause the depth plane selection module 750 to switch from a first, farther depth plane to a second, closer depth plane only after the user's depth of convergence crosses a first threshold. Similarly, the module 752 can cause the depth plane selection module 750 (which in turn instructs a display, such as the display 220) to switch to the first, farther depth plane only after the user's depth of convergence crosses a second threshold that is farther from the user than the first threshold. In the overlapping region between the first and second thresholds, the module 750 can cause the depth plane selection module 750 to maintain the currently selected depth plane as the selected depth plane, thereby avoiding excessive switching between depth planes.

[0609] The eye event detection module 750 can detect Figure 7AThe eye tracking module 614 receives other eye data and can cause the depth plane selection module 750 to delay certain depth plane switches until an eye event occurs. As an example, the eye event detection module 750 can cause the depth plane selection module 750 to delay a planned depth plane switch until a user blink is detected; data indicating when the user is currently blinking can be received from the blink detection component in the eye tracking module 614; and in response, the depth plane selection module 750 can be caused to perform a planned depth plane switch during the blink event (e.g., by causing the module 750 to instruct the display 220 to perform a depth plane switch during the blink event). In at least some embodiments, the wearable system is able to shift content to a new depth plane during a blink event, making the shift less noticeable to the user. As another example, the eye event detection module 750 can delay a planned depth plane switch until an eye saccade is detected. As discussed in conjunction with blinking, this arrangement can facilitate discrete shifting of depth planes.

[0610] If desired, the depth plane selection module 750 can delay a planned depth plane switch for only a limited period of time before performing the depth plane switch, even in the absence of an eye event. Similarly, the depth plane selection module 750 can perform a depth plane switch when the user's convergence depth is substantially outside of the currently selected depth plane (e.g., when the user's convergence depth has exceeded the conventional threshold for depth plane switching by a predetermined threshold), even in the absence of an eye event. These arrangements can help ensure that the eye event detection module 754 does not delay a depth plane switch indefinitely and does not delay a depth plane switch in the presence of large accommodation errors. Further details on the operation of the depth plane selection module 750 and how the module times depth plane switches are provided herein in conjunction with Figure 12 supply.

[0611] The rendering camera controller 758 can provide information indicating the positions of the user's left and right eyes to the rendering engine 622. The rendering engine 622 can then generate content by simulating cameras at the positions of the user's left and right eyes and generating content based on the perspectives of the simulated cameras. As described above, a rendering camera is a simulated camera used to render virtual image content, which may be derived from a database of objects in the virtual world. Objects may have positions and orientations relative to the user or wearer, and possibly relative to real objects in the user's or wearer's surroundings. The rendering camera can be included in the rendering engine to render virtual images based on the database of virtual objects to be presented to the eyes. The virtual image can be rendered as if it were captured from the user's or wearer's perspective. For example, the virtual image can be rendered as if it were captured by a camera (corresponding to a "rendering camera") having an aperture, lens, and detector that views objects in the virtual world. The virtual image is captured from the perspective of this camera with the "rendering camera" position. For example, the virtual image can be rendered as if it were captured from the perspective of a camera with a specific position relative to the user's or wearer's eyes, thereby providing an image that appears to be from the user's or wearer's perspective. In some embodiments, the image is rendered as if captured from the perspective of a camera having an aperture at a particular position relative to the user or wearer's eyes (e.g., a perspective center or rotation center as discussed herein or elsewhere).

[0612] The rendering camera controller 758 can determine the positions of the left and right cameras based on the left and right eye centers of rotation (CoR) determined by the CoR estimation module 724 and / or based on the left and right eye centers of perspective (CoP) determined by the CoP estimation module 732. In some embodiments, the rendering camera controller 758 can switch between the CoR and CoP positions based on various factors. By way of example, the rendering camera controller 758 can, in various modes, always register the rendering camera to the CoR position, always register the rendering camera to the CoP position, switch or discretely switch between registering the rendering camera to the CoR position and registering the rendering camera to the CoP position over time based on various factors, or dynamically register the rendering camera to any one of a range of different positions along the optical axis (or visual axis) between the CoR and CoP positions over time based on various factors. The CoR and CoP positions may optionally be passed through a smoothing filter 756 (in any of the modes described above for rendering camera positioning) which may average the CoR and CoP positions over time to reduce noise in those positions and prevent jitter in the rendering camera rendering the simulation.

[0613] In at least some embodiments, the rendering camera can be modeled as a pinhole camera, where the pinhole is set at the location of the estimated CoR or CoP identified by the eye tracking module 614. In the case where the CoP deviates from the CoR, whenever the position of the rendering camera is based on the user's CoP, the rendering camera and its pinhole shift with the rotation of the user's eye (see, for example, Figure 16A and 16B ). In contrast, whenever the position of the rendering camera is based on the user's CoR, the position of the rendering camera pinhole does not move with eye rotation, although in some embodiments, the rendering camera (located behind the pinhole) may move with eye rotation. In other embodiments where the position of the rendering camera is based on the user's CoR, the rendering camera may not move (e.g., rotate) with the user's eye (see, e.g., Figure 17A and 17B , how the rendered camera does not move with eye rotation or linear translation).

[0614] Example of locating a user's cornea using an eye tracking system

[0615] Figure 8A Schematic diagram of an eye showing the cornea of ​​the eye. Figure 8A As shown, a user's eye 810 can have a cornea 812, a pupil 822, and a lens 820. The cornea 812 can have a nearly spherical shape, shown by a corneal sphere 814. The corneal sphere 814 can have a center point 816 (also referred to as a corneal center) and a radius 818. The hemispherical cornea of ​​the user's eye can be curved around the corneal center 816.

[0616] Figures 8B to 8E An example of using the 3D cornea center estimation module 716 and the eye tracking module 614 to locate the user's cornea center 816 is shown.

[0617] like Figure 8B As shown, the 3D cornea center estimation module 716 can receive an eye tracking image 852 including a corneal spot 854. The 3D cornea center estimation module 716 can then simulate the known 3D positions of the eye camera 324 and the light source 326 (which can be based on data in the eye tracking extrinsic and intrinsic features database 702, the assumed eye size database 704, and / or the per-user calibration data 706) in the eye camera coordinate system 850 to cast a ray 856 in the eye camera coordinate system. In at least some embodiments, the eye camera coordinate system 850 can have its origin at the 3D position of the eye tracking camera 324.

[0618] exist Figure 8C, the 3D cornea center estimation module 716 simulates a corneal sphere 814a (which may be based on an assumed eye size from the database 704) and a corneal curvature center 816a at a first location. The 3D cornea center estimation module 716 may then check to see if the corneal sphere 814a properly reflects light from the light source 326 to the bright spot location 854. Figure 8C As shown, when ray 860a does not intersect light source 326, the first position does not match.

[0619] Similarly, in Figure 8D , the 3D cornea center estimation module 716 simulates the cornea sphere 814b and the cornea curvature center 816b at the second position. Then, the 3D cornea center estimation module 716 checks whether the cornea sphere 814b properly reflects the light from the light source 326 to the bright spot position 854. Figure 8D As shown, the second position also does not match.

[0620] like Figure 8E As shown, the 3D cornea center estimation module 716 is ultimately able to determine that the correct position of the corneal sphere is the corneal sphere 814c and the corneal curvature center 816c. The 3D cornea center estimation module 716 confirms that the shown position is correct by checking that the light from the light source 326 is properly reflected from the corneal sphere at the correct position of the bright spot 854 on the image 852 and is imaged by the camera 324. With this arrangement, and with the known 3D positions of the light source 326 and the camera 324 and the optical properties of the camera (focal length, etc.), the 3D cornea center estimation module 716 can determine the 3D position of the corneal curvature center 816 (relative to the wearable system).

[0621] This paper combines at least Figures 8C to 8E The described process can be effectively implemented as an iterative, repetitive, or optimization process to identify the 3D location of the center of the user's cornea. In this way, any of a variety of techniques (e.g., iterative techniques, optimization techniques, etc.) can be used to effectively and quickly prune or reduce the search space of possible locations. In addition, in some embodiments, the system can include two, three, four, or more light sources (e.g., light source 326), and some of all of these light sources can be set at different positions, resulting in multiple bright spots (e.g., bright spot 584) located at different positions on image 852 and multiple rays (e.g., ray 856) having different origins and directions. Such embodiments can enhance the accuracy of the 3D cornea center estimation module 716 because the module 716 seeks to identify corneal locations that cause some or all of the bright spots and rays to be properly reflected between their corresponding light sources and their respective positions on image 852. In other words, in these embodiments, the positions of some or all of the light sources depend on Figures 8B to 8EIn some embodiments, the system may determine a vector or ray along which the corneal center lies (i.e., the 2D corneal center location) prior to performing the optimization process. In such embodiments, the 3D corneal center estimation module 716 may search for the corneal location only along such vectors, which may provide computational and / or time savings when performing the optimization process. In at least some of these embodiments, before determining such vectors, the system may first (i) define a first plane between the origin of the eye camera coordinate system 850, a first light source (e.g., light source 326a), and a first bright spot (e.g., bright spot 854a) generated by the first light source, and (ii) define a second plane between the origin of the eye camera coordinate system 850, a second light source (e.g., light source 326b), and a second bright spot (e.g., bright spot 854b) generated by the second light source. The system may then simply calculate the cross product of the first and second planes to determine the vector or ray along which the corneal center lies (i.e., the 2D corneal center location).

[0622] Example of normalizing the coordinate system of eye tracking images

[0623] Figures 9A to 9C Shown is the wearable system components (such as Figure 7A An example of normalizing the coordinate system of the eye-tracking image by the coordinate system normalization module 718 is shown. Normalizing the coordinate system of the eye-tracking image relative to the user's pupil position can compensate for slippage of the wearable system relative to the user's face (e.g., headset slippage), and such normalization can establish a consistent orientation and distance between the eye-tracking image and the user's eyes.

[0624] like Figure 9A As shown, coordinate system normalization module 718 may receive estimated 3D coordinates 900 of the user's corneal rotation center and may receive an unnormalized eye tracking image, such as image 852. As an example, eye tracking image 852 and coordinates 900 may be in unnormalized coordinate system 850 based on the position of eye tracking camera 324.

[0625] As a first normalization step, the coordinate system normalization module 718 can rotate the coordinate system 850 into a rotated coordinate system 902 so that the z-axis of the coordinate system (e.g., the convergence depth axis) can be aligned with the vector between the coordinate system origin and the center of corneal curvature coordinate 900, as shown in FIG. Figure 9B Specifically, the coordinate system normalization module 718 may rotate the eye tracking image 850 into a rotated eye tracking image 904 until the coordinates 900 of the center of corneal curvature of the user are perpendicular to the plane of the rotated image 904 .

[0626] As a second normalization step, the coordinate system normalization module 718 can translate the rotated coordinate system 902 into a normalized coordinate system 910 so that the center of corneal curvature coordinate 900 is a standard normalized distance 906 from the origin of the normalized coordinate system 910, as shown in FIG. Figure 9C Specifically, coordinate system normalization module 718 can translate rotated eye-tracking image 904 into normalized eye-tracking image 912. In at least some embodiments, standard normalization distance 906 can be approximately 30 mm. If desired, a second normalization step can be performed before the first normalization step.

[0627] Example of locating the centroid of a user's pupil using an eye tracking system

[0628] Figures 9D to 9G The use of the 3D pupil center locator module 720 and the eye tracking module 614 to locate the user's pupil center (e.g., as Figure 8A An example of the center of the user's pupil 822 shown).

[0629] like Figure 9D As shown, the 3D pupil center locator module 720 can receive a normalized eye tracking image 912 including a pupil centroid 913 (e.g., the center of the user's pupil identified by the pupil identification module 712). The 3D pupil center locator module 720 can then simulate a normalized 3D position 910 of the eye camera 324 to cast a ray 914 in the normalized coordinate system 910 through the pupil centroid 913.

[0630] exist Figure 9E , the 3D pupil center locator module 720 can simulate a corneal sphere, such as a corneal sphere 901 having a center of curvature 900 (e.g., in combination with the 3D pupil center locator module 720), based on data from the 3D corneal center estimation module 716. Figures 8B to 8E As an example, it is possible to combine Figure 8E The position of the identified center of curvature 816c and the Figures 9A to 9C The normalization process positions the corneal sphere 901 in the normalized coordinate system 910. Figure 9E As shown, the 3D pupil center locator module 720 can identify a first intersection point 916 between a ray 914 (eg, a ray between the origin of the normalized coordinate system 910 and the normalized position of the user's pupil) and the simulated cornea.

[0631] like Figure 9FAs shown, the 3D pupil center locator module 720 can determine the pupil sphere 918 based on the corneal sphere 901. The pupil sphere 918 can share a common curvature center with the corneal sphere 901, but have a smaller radius. The 3D pupil center locator module 720 can determine the distance between the corneal center 900 and the pupil sphere 918 (e.g., the radius of the pupil sphere 918) based on the distance between the corneal center and the pupil center. In some embodiments, the distance between the pupil center and the corneal curvature center can be determined based on the distance between the corneal center and the pupil center. Figure 7A The assumed eye size 704 is determined based on the eye tracking extrinsic and intrinsic feature database 702, and / or based on user calibration data 706. In other embodiments, the distance between the pupil center and the center of corneal curvature can be determined based on Figure 7A is determined by user calibration data 706.

[0632] like Figure 9G As shown, the 3D pupil center locator module 720 can locate the 3D coordinates of the user's pupil center based on various inputs. As an example, the 3D pupil center locator module 720 can determine the 3D coordinates of the user's pupil center using the 3D coordinates and radius of the pupil sphere 918, the 3D coordinates of the intersection 916 between the simulated corneal sphere 901 in the normalized eye tracking image 912 and the ray 914 associated with the pupil centroid 913, information about the corneal refractive index, and other relevant information such as the refractive index of air (which can be stored in the eye tracking extrinsic and intrinsic features database 702). Specifically, the 3D pupil center locator module 720 can bend the ray 916 into the refracted ray 922 based on the refractive index difference between air (having a first refractive index of approximately 1.00) and the corneal material (having a second refractive index of approximately 1.38) during simulation. After accounting for refraction caused by the cornea, the 3D pupil center locator module 720 can determine the 3D coordinates of a first intersection point 920 between the refracted ray 922 and the pupil sphere 918. The 3D pupil center locator module 720 can determine that the user's pupil center 920 is located approximately at the first intersection point 920 between the refracted ray 922 and the pupil sphere 918. With this arrangement, the 3D pupil center locator module 720 can determine the 3D position of the pupil center 920 (relative to the wearable system) in the normalized coordinate system 910. If desired, the wearable system can denormalize the coordinates of the pupil center 920 to the original eye camera coordinate system 850. The pupil center 920 can be used together with the corneal curvature center 900 to, among other things, determine the user's optical axis using the optical axis determination module 722 and determine the user's vergence depth by the vergence depth estimation module 728.

[0633] Accounting for corneal refraction may result in a determined pupil position that is more stable than a pupil position determined based on a first intersection 916 between ray 914 (i.e., a ray between the origin of the normalized coordinate system 910 and the normalized position of the user's pupil) and the simulated cornea, as shown in FIG. Figure 9E This is partially true because, while easier to calculate, first intersection point 916 may not correspond to a physical feature of the eye and therefore may not move with the eye as a physical entity. Conversely, calculating pupil center 920 by accounting for corneal refraction may better correspond to the physical pupil location of the eye, even though some variation may still exist due to viewing angle. Thus, in various embodiments, determining the optical axis of the eye thus involves calculating the true pupil center, rather than first intersection point 916, between ray 914 and the simulated cornea.

[0634] When the center of rotation (CoR) is estimated as a point at a fixed distance relative to the corneal center along the eye's optical axis, including corneal refraction can have significant benefits. Specifically, including corneal refraction when determining pupil position can significantly reduce variations in the calculation of the center of rotation for different eye orientations. For example, variations can occur when the eye as a whole moves in the camera coordinate system (e.g., during headset refitting) because the eye as a whole will be oriented differently relative to the refitted headset. Because pupil center 920 better corresponds to the eye's physical pupil location, variations in the CoR can be smaller when the eye as a whole moves in the camera coordinate system. Advantageously, including corneal surface refraction can result in a more stable and accurate CoR, which can be used to determine when to refit the headset on the user's head, can allow for more accurate placement of rendering cameras, can enable other novel gaze tracking algorithms, or any combination thereof. Additionally, the CoR, as a stable, slowly changing eye feature, can potentially be tracked via a multi-frame Kalman-type temporal filter to provide a geometric reference position for other applications. Figure 9H Example calculated positions of the pupil center resulting from including and excluding corneal refraction are shown. When the pupil center is calculated without including the effects of corneal refraction, the result can be an external pupil 960. External pupil center 960 can correspond to a first intersection point 916 between ray 914 (i.e., a ray between the origin of normalized coordinate system 910 and the normalized position of the user's pupil) and the simulated cornea, as shown in FIG. Figure 9E When the pupil center is calculated taking into account corneal refraction, the result may be a refracted pupil center 962. The refracted pupil center 962 may correspond to the pupil center 920, as shown in FIG. Figure 9G. Different locations of the external pupil center 960 and the refracted pupil center 962 can result in different calculations of the eye's center of rotation, determined as a fixed distance along the eye's optical axis relative to the corneal center 964. For example, the external pupil center 960 can result in an external center of rotation 968. The external center of rotation 968 can be significantly different from the center of rotation 966 calculated based on the refracted pupil center 962.

[0635] Figures 9I to 9L Example experimental variations of the calculated center of rotation of the eye based on different calculated pupil centers are shown using a collection of over 400 data sets. The data selected were only those frames with four bright spots and a valid pupil center (i.e., not all three x, y, z coordinate components were equal to zero) to exclude obvious pipeline failure cases. Different values ​​of the center of rotation (CoR) to the center of corneal curvature distance (R) were studied and it was found that R = 4.7 mm gave a near-optimal average result. However, specific user distance values ​​can be adjusted to provide smaller CoR coordinate variations.

[0636] Figure 9I Shown are the x, y, and z coordinates of different pupil centers, and a three-dimensional plot of the corresponding CoRs calculated using the different pupil centers for the above dataset. Figure 9J Using XY projection shows Figure 9I Cluster 970 corresponds to the coordinates of the external pupil center 960, cluster 972 corresponds to the coordinates of the correct refracted pupil center 962, cluster 974 corresponds to the corneal position (e.g., the three-dimensional corneal center position), cluster 978 corresponds to the CoR using the correct refracted pupil center 962, and cluster 980 corresponds to the CoR using the external pupil center 960. The size of cluster 978 (particularly the size in the x-direction) is smaller than that of cluster 980, indicating that the change in CoR when using the refracted pupil center 962 is smaller than the change in CoR when using the external pupil center 960.

[0637] Table 1

[0638]

[0639] As can be seen from Table 1, the standard deviation (or sigma) of the x-component of CoR is reduced by approximately half when the refracted pupil center 962 is used for the calculation compared to when the external pupil center 960 is used. By using the refracted pupil center 962, the overall three-dimensional standard deviation (sigma 3d ) also decreased significantly.

[0640] Figure 9K and 9LThe mean and median CoR standard deviations are shown, respectively, for a collection of over 400 data sets as a function of the distance from the CoR to the center of corneal curvature. Graphs 991L and 991R show the mean three-dimensional CoR standard deviation (sigma ) as a function of the distance from the CoR to the center of corneal curvature for the left eye and the right eye, respectively. 3d Graphs 992L and 992R show the median three-dimensional CoR standard deviation (sigma) of the CoR to the center of corneal curvature distance for the left eye and the right eye, respectively. 3d ). Curve 982A corresponds to the overall left eye average sigma 3d , curve 982B corresponds to the overall right eye average sigma 3d , curve 982C corresponds to the total left eye median sigma 3d , curve 982D corresponds to the total right eye median sigma 3d Curves 984A-D correspond to the sigma 3d The x-component of curves 990A-D corresponds to the sigma 3d The y component of curves 986A-D corresponds to the sigma 3d The z component of .

[0641] Figure 9M is shown as providing the minimum sigma 3d Example distribution of optimal radii calculated for each user dataset individually for the CoR-to-cornea distance values. The mean of the distribution was found to be at R = 4.9 mm with a standard deviation of 1.5 mm. Users with very small radii (R ~ 1 mm) were found to have very poor gaze tracking and therefore had to be excluded.

[0642] Example of the difference between the optical axis and the visual axis

[0643] If combined Figure 7A As discussed in the optical axis to visual axis mapping module 730, a user's optical axis and visual axis are typically not aligned, in part because the user's visual axis is defined by their fovea, and the fovea is typically not located at the center of a person's retina. Therefore, when a person wishes to focus their attention on a particular object, they align their visual axis with the object to ensure that light from the object falls on their fovea, while their optical axis (defined by the center of their pupil and the center of their corneal curvature) is actually slightly offset from the object. Figure 10 is an example of an eye 1000 showing the eye's optical axis 1002, the eye's visual axis 1004, and the offset between these axes. Figure 10The pupil center 1006 of the eye, the center of corneal curvature 1008 of the eye, and the mean center of rotation (CoR) 1010 of the eye are shown. In at least some people, the center of corneal curvature 1008 of the eye can be located approximately 4.7 mm in front of the mean center of rotation (CoR) 1010 of the eye, as shown by dimension 1012. Additionally, the perspective center 1014 of the eye can be located approximately 5.01 mm in front of the center of corneal curvature 1008 of the eye, approximately 2.97 mm behind the outer surface 1016 of the user's cornea, and / or just in front of the user's pupil center 1006 (e.g., corresponding to a location within the anterior chamber of the eye 1000). As additional examples, dimension 1012 can be between 3.0 mm and 7.0 mm, between 4.0 mm and 6.0 mm, between 4.5 mm and 5.0 mm, or between 4.6 mm and 4.8 mm, or any range between any values, as well as any value within any of these ranges. The center of perspective (CoP) 1014 of the eye can be a useful location for a wearable system because, in at least some embodiments, registering the rendering camera at the CoP helps reduce or eliminate parallax artifacts.

[0644] Figure 10 Also shown is a human eye 1000 that can be aligned with the pinhole of the rendering camera. Figure 1 As shown. Figure 10 As shown, the pinhole of the rendering camera can be aligned with a position 1014 that is along the optical axis 1002 or visual axis 1004 of the human eye 1000 and that is closer to the outer surface of the cornea than both (a) the center 1006 of the pupil or iris and (b) the center 1008 of the corneal curvature of the human eye 1000. For example, Figure 10 As shown, the pinhole of the rendering camera can be aligned with a position 1014 that is located approximately 2.97 mm behind the outer surface 1016 of the cornea and approximately 5.01 mm in front of the center of corneal curvature 1008 along the optical axis 1002 of the human eye 1000. The position 1014 of the pinhole of the rendering camera and / or the anatomical region of the human eye 1000 corresponding to the position 1014 can be considered to represent the perspective center of the human eye 1000. Figure 10 The optical axis 1002 of the illustrated human eye 1000 represents the most direct line through the center of corneal curvature 1008 and the center of the pupil or iris 1006. The visual axis 1004 of the human eye 1000 differs from the optical axis 1002 in that it represents a line extending from the fovea of ​​the human eye 1000 to the center of the pupil or iris 1006.

[0645] Example process of rendering content and checking registration based on eye tracking

[0646] Figure 111 is a process flow diagram of an example method 1100 for rendering content and providing registration feedback in a wearable device using eye tracking. Method 1100 can be performed by a wearable system as described herein. Embodiments of method 1100 can be used by a wearable system to render content and provide registration feedback (e.g., the fit of the wearable device to the user) based on data from an eye tracking system.

[0647] At block 1110, the wearable system may capture an image of one or more eyes of a user. The wearable system may use at least Figure 3 324 to capture eye images. If desired, the wearable system may also include one or more light sources 326 configured to shine IR light on the user's eyes and produce corresponding bright spots in the eye images captured by the eye cameras 324. As discussed herein, the eye tracking module 614 may use the bright spots to deduce various pieces of information about the user's eyes, including where the eyes are looking.

[0648] At block 1120, the wearable system may detect a bright spot and a pupil in the eye image captured in block 1110. As an example, block 1120 may include processing the eye image by the bright spot detection and marking module 714 to identify a two-dimensional location of the bright spot in the eye image, and processing the eye image by the pupil identification module 712 to identify a two-dimensional location of the pupil in the eye image.

[0649] At block 1130, the wearable system may estimate the three-dimensional position of the user's left and right corneas relative to the wearable system. As an example, the wearable system may estimate the position of the center of curvature of the user's left and right corneas and the distances between these centers of curvature and the user's left and right corneas. Block 1130 may include the 3D cornea center estimation module 716 identifying the position of the center of curvature, as described herein at least in conjunction with Figure 7A as well as Figures 8A to 8E As stated.

[0650] At block 1140, the wearable system may estimate the three-dimensional locations of the user's left and right pupil centers relative to the wearable system. As an example, the wearable system and the 3D pupil center locator module 720 may specifically estimate the locations of the user's left and right pupil centers, such as by combining at least Figure 7A as well as Figures 9D to 9G As described, as part of box 1140.

[0651] At block 1150, the wearable system may estimate the three-dimensional position of the user's left and right centers of rotation (CoR) relative to the wearable system. As an example, the wearable system and the CoR estimation module 724 may specifically estimate the positions of the CoRs of the user's left and right eyes, such as by combining at least Figure 7A and Figure 10 As a specific example, a wearable system can find the CoR of the eye by following the optical axis from the center of corneal curvature back to the retina.

[0652] At block 1160, the wearable system can estimate the user's IPD, vergence depth, center of perspective (CoP), optical axis, visual axis, and other desired properties based on the eye tracking data. As an example, the IPD estimation module 726 can estimate the user's IPD by comparing the 3D positions of the left and right CoRs, the vergence depth estimation module 728 can estimate the user's depth by finding the intersection (or near the intersection) of the left and right optical axes or the intersection of the left and right visual axes, the optical axis determination module 722 can identify the left and right optical axes over time, the optical axis to visual axis mapping module 730 can identify the left and right visual axes over time, and the CoP estimation module block 732 can identify the left and right centers of perspective as part of block 1160.

[0653] At block 1170, the wearable system may render content based in part on the eye tracking data identified in blocks 1120 through 1160 and may optionally provide registration feedback (e.g., the fit of the wearable system to the user's head). As an example, the wearable system may identify a suitable location for a rendering camera and then generate content for the user based on the location of the rendering camera, such as in conjunction with the light field rendering controller 618, Figure 7B and rendering camera 622. As another example, the wearable system can determine whether it is properly fitted to the user, or has slipped from its correct position relative to the user, and can provide optional feedback to the user indicating whether the fit of the device needs to be adjusted, as discussed in conjunction with registration viewer 620. In some embodiments, the wearable system can adjust the rendered content based on improper or less-than-ideal registration to attempt to reduce, minimize, or compensate for the effects of improper or incorrect registration.

[0654] Example image of rendering content in response to user eye movements

[0655] Figure 12 A set of example diagrams 1200a-1200j are included, which illustrate how a wearable system can switch depth planes in response to user eye movement. Figure 4As discussed in and 7, a wearable system can include multiple depth planes, where each depth plane is configured to present content to a user at a different simulated depth or through different accommodation cues (e.g., through various wavefront curvatures or light divergence levels). As an example, a wearable system can include a first depth plane configured to simulate a first depth range and a second depth plane configured to simulate a second depth range, although the two ranges can overlap as needed to facilitate switching hysteresis, but the second depth range can generally extend to a greater distance relative to the user. In such an embodiment, the wearable system can track the user's vergence depth, saccadic movements, and blinks to switch between the first and second depth planes in a manner that avoids excessive depth plane switching, excessive accommodation-vergence mismatch, excessive accommodation-vergence mismatch time periods, and seeks to reduce the visibility of depth plane switching (e.g., by switching depth planes during blinks and saccades).

[0656] Graph 1200a shows an example of a user's depth of convergence over time. Graph 1200b shows an example of a user's saccade signal or eye movement speed over time.

[0657] Graph 1200c may illustrate vergence depth data generated by the eye tracking module 614, and specifically, data generated by the vergence depth estimation module 728. As shown in graphs 1200c-1200h, the eye tracking data may be sampled at a rate of approximately 60 Hz within the eye tracking module 614. As shown between graphs 1200b and 1200c, the eye tracking data within the eye tracking module 614 may lag behind the user's actual eye movement by a delay 1202. As an example, at time t1, the user's vergence depth may cross the hysteresis threshold 1210a, but the cross-hysteresis band detection module 752 may not recognize this event until time t2 after the delay 1202.

[0658] Graph 1200c also illustrates various threshold values ​​1210a, 1210b, 1210c in the hysteresis band, which may be associated with the first and second depth planes (e.g., Figure 12 In some embodiments, the wearable system may attempt to display content in depth plane #1 whenever the user's vergence depth is greater than threshold 1210b; and may attempt to display content in depth plane #0 whenever the user's vergence depth is less than threshold 1210b. However, to avoid excessive switching, the wearable system may implement hysteresis such that the wearable system does not switch from depth plane #1 to depth plane #0 until the user's vergence depth crosses external threshold 1210c. Similarly, the wearable system does not switch from depth plane #0 to depth plane #1 until the user's vergence depth crosses external threshold 1210a.

[0659] Graph 1200d illustrates an internal flag that may be generated by the depth plane selection module 750 or the cross-hysteresis band detection module 752 that indicates whether the user's convergence depth is within the volume typically associated with depth plane #1 or within the volume typically associated with depth plane #2 (e.g., whether the user's convergence depth is greater than or less than threshold 1210b).

[0660] Graph 1200e illustrates an internal hysteresis band flag that may be generated by the depth plane selection module 750 or the cross-hysteresis band detection module 752 to indicate whether the user's vergence depth has crossed an external threshold, such as threshold 1210a or 1210c. Specifically, graph 1200e illustrates a flag that indicates whether the user's vergence depth has completely crossed the hysteresis band and entered a region outside the volume of the active depth plane (e.g., into a region associated with a depth plane other than the active depth plane), thereby potentially causing an undesirable accommodation-vergence mismatch (AVM).

[0661] Graph 1200f illustrates an internal AVM flag that may be generated by the depth plane selection module 750 or the cross-hysteresis band detection module 752 to indicate whether the user's vergence is outside the volume of the active depth plane for more than a predetermined time. Thus, the AVM flag can identify when a user is experiencing an undesirable accommodation-vergence mismatch for a near-excessive or excessive time period. Additionally or alternatively, the internal AVM flag can also indicate whether the user's vergence is beyond a predetermined distance from the volume of the active depth plane, thereby creating a potential excessive accommodation-vergence mismatch. In other words, the AVM flag can indicate when the user's vergence exceeds an additional threshold that is even further from threshold 1210b than thresholds 1210a and 1210c.

[0662] Graph 1200g illustrates an internal blink flag that can be generated by the eye event detection module 754 to determine when a user has blinked or is blinking. As described herein, it is desirable to switch depth planes when a user blinks to reduce the likelihood that the user will perceive the depth plane switch.

[0663] Graph 1200h shows an example of output from the depth plane selection module 750. Specifically, graph 1200h shows that the depth plane selection module 750 can send a Figure 6 ) outputs instructions that utilize a selected depth plane that varies over time.

[0664] Graphs 1200i and 1200j illustrate the latency that may exist in a wearable system, including the latency of the rendering engine 622 switching depth planes and the latency of the display 220 required to provide light associated with the new image frame in the new depth plane to effectuate the depth plane change.

[0665] Reference will now be made to graphs 1200a-1200j at different times (t0–t 10 ) shown in the event.

[0666] At some point around time t0, the user's depth of convergence may cross threshold 1210a, which may be an external hysteresis threshold. After delays associated with image capture and signal processing, the wearable system may generate a signal, as shown in graph 1200e, indicating that the user's depth of convergence is within the hysteresis band. In the example of graph 1200e, one or more modules of the light field rendering controller 618 may, in conjunction with the user's crossing of depth of convergence threshold 1210a, display a hysteresis band exceeded indicator at approximately time t1.

[0667] From time t0 to approximately time t4, the user's depth of convergence continues to decrease and then increases.

[0668] At time t1, the user's vergence depth crosses threshold 1210b, which can be the midpoint between two depth planes (e.g., depth planes #1 and #0). After processing delay 1202, eye tracking module 614 can change an internal flag indicating that the user's vergence depth has moved from the volume generally associated with depth plane #1 to the volume generally associated with depth plane #0 (as shown in graph 1200d).

[0669] At time t3, one or more modules of the light field rendering controller 618 may determine that the user's vergence depth (as shown in graph 1200a) has completely moved through the hysteresis band and crossed the outer threshold 1210c. Consequently, one or more modules of the light field rendering controller 618 may generate a signal indicating that the user's vergence depth is outside the hysteresis band (as shown in graph 1200e). In at least some embodiments, one or more modules of the light field rendering controller 618 (e.g., depth plane selection module 750) switch between the first and second depth planes only when the user's vergence depth is outside the hysteresis band between the two depth planes.

[0670] In at least some embodiments, one or more modules of the light field rendering controller 618 may be configured to switch depth planes at time t3. Specifically, one or more modules of the light field rendering controller 618 may be configured to switch depth planes based on determining that the convergence depth has moved from the volume of the currently selected depth plane (depth plane #1 shown in graph 1200h) into the volume of another depth plane (depth plane #0) and completely crossed the hysteresis band. In other words, whenever the hysteresis band is exceeded (graph 1200e is high) and an accommodation-convergence mismatch based on time or mismatch amount is detected (graph 1200f is high), one or more modules of the light field rendering controller 618 may implement a depth plane switch. In such an embodiment, one or more modules of the light field rendering controller 618 may provide a signal to the rendering engine 622 instructing the rendering engine 622 to switch to the other depth plane (depth plane #0). However, in Figure 12 In an example, one or more modules of the light field rendering controller 618 can be configured to delay depth plane switching until at least one additional condition is satisfied. These additional conditions can include, for example, an eye blink condition, an accommodation-convergence mismatch timeout condition, and an accommodation-convergence amount condition.

[0671] At time t4 and at Figure 12 In an example, one or more modules of the light field rendering controller 618 may be configured to switch depth planes. Specifically, one or more modules of the light field rendering controller 618 may determine that the user's vergence has been within the volume associated with depth plane #0 for longer than a predetermined time threshold (and, optionally, outside of a hysteresis band for that time period). Examples of predetermined time thresholds include 5 seconds, 10 seconds, 20 seconds, 30 seconds, 1 minute, and 90 seconds, as well as any ranges between any of these values. Based on such a determination, one or more modules of the light field rendering controller 618 may generate an AVM flag, as shown in graph 1200f, and instruct the rendering engine 622 to switch to depth plane #0, as shown in graph 1200h. In some embodiments, if the user's vergence depth is detected to be greater than a threshold distance from the currently selected depth volume, one or more modules of the light field rendering controller 618 may generate an AVM flag and instruct the rendering engine 622 to switch depth planes.

[0672] At time t5 and after delay 1204, rendering engine 622 may begin rendering content at the newly selected depth plane #0. After a delay 1206 associated with rendering and after light is transmitted to the user through display 220, display 220 may fully switch to the newly selected depth plane #0 before time t6.

[0673] Thus, graphs 1200a-j illustrate how the system responds to the user's changing vergence between times t0 and t6, and how it switches depth planes after the user's vergence moves away from the previous depth volume for more than a predetermined period of time. 10 The figure shows how the system responds to the user's changing convergence and how it switches depth planes when a user blink is detected (possibly before a predetermined time period).

[0674] At time t7, one or more modules of the light field rendering controller 618 may detect that the user's depth of convergence has entered a hysteresis region between depth planes #0 and #1 (e.g., the user's depth of convergence has crossed outer threshold 1210c). In response, one or more modules of the light field rendering controller 618 may change a hysteresis flag, as shown in graph 1200e.

[0675] At time t8, one or more modules of the light field rendering controller 618 may detect that the user's depth of convergence has crossed threshold 1210b and moved from the volume generally associated with depth plane #0 to the volume generally associated with depth plane #1. Accordingly, one or more modules of the light field rendering controller 618 may change the depth volume flag, as shown in graph 1200d.

[0676] At time t9, one or more modules of the light field rendering controller 618 may detect that the user's vergence depth has crossed threshold 1210a and has moved out of the hysteresis volume and into the volume typically associated only with depth plane # 1. In response, one or more modules of the light field rendering controller 618 may change the hysteresis flag, as shown in graph 1200e.

[0677] At time t 10 Around t, the user blinks, and one or more modules of the light field rendering controller 618 can detect the blink. As an example, the eye event detection module 754 can detect the user's blink. In response, one or more modules of the light field rendering controller 618 can generate a blink flag, as shown in graph 1200h. In at least some embodiments, whenever the hysteresis band is exceeded (graph 1200e is high) and a blink is detected (graph 1200g is high), one or more modules of the light field rendering controller 618 can implement depth plane switching. Therefore, one or more modules of the light field rendering controller 618 can instruct the rendering engine 622 to switch to the next depth plane at time t 10 Toggle depth plane.

[0678] Example rendering mode in a mixed reality system with multiple depth planes

[0679] In a mixed reality system, a computer-generated (rendered) scene can be transmitted to the human eye so that real objects and virtual objects are spatially aligned (from the user's perspective). To provide the user with a visual perception of spatial alignment between real objects and virtual objects, the perspective at which the computer-generated scene is rendered and presented can preferably correspond to the perspective (e.g., position and orientation) of the user's eyes. As an example, when a "real world" frame (a frame in which real objects are present) and a "rendered world" frame (a frame in which virtual objects are present) are accurately aligned with each other, the user can perceive that the real objects and virtual objects are spatially aligned in the desired manner.

[0680] Digital light field display devices (such as Figure 2 The wearable system 200 including the display 220 is an example of a mixed reality system in which one or more depth planes may be used to provide a light field representing 3D virtual content (virtual objects) to a user at various depths. The depth planes may be compared to one or more virtual screens located at different distances relative to the user, on which the virtual content may be projected or displayed, transformed into virtual pixels, and provided to the user. In this way, the mixed reality system may be optically equivalent to a system having one or more transparent floating screens located at different distances relative to the user. In this way, the digitized light field is projected through the user's iris onto the user's retina and forms an image of the 3D virtual content (e.g., the user perceives an image of the 3D virtual content).

[0681] Figure 13 A mixed reality system 1300 is shown in which a light field representing 3D virtual content including virtual objects is provided to a user's eyes using one or more depth planes, which may be optical structures on a wearable device that simulate a virtual screen at various distances from the user and the wearable device. The mixed reality system 1300 may include an eyepiece 1310, which may represent an eyepiece of a head-mounted digital light field display device (e.g., Figure 3 Such a system may be configured to project a light field representing a 3D virtual object 1330 onto the retina 1303 of the user's eye 1302 through the eyepiece 1310.

[0682] Figure 13 Also shown are depth planes 1321-1323 on which virtual objects 1330 and other virtual content may be projected or displayed and transformed into virtual pixels. Figure 13In the particular example shown, a virtual object 1330 is projected onto the depth plane 1322, thereby being transformed into virtual pixels 1332. As a result, light generated by the eyepiece 1310 (e.g., the display 220) can provide accommodation cues to the user's eye 1302, just as if the virtual object 1330 were provided on a physical display or projector screen at a distance from the user of the depth plane 1322. The head-mounted digital display device can generate a digitized light field representing the virtual pixels 1332 and can project such light field onto the retina 1303 of the user's eye 1302 through the eyepiece 1310.

[0683] As will be discussed in more detail below, it is possible to use mixed reality systems (e.g. Figure 13 Different rendering modes are employed in the mixed reality system 1300 in order to provide different eye accommodation cues across the user's field of view for different content and / or at different time periods. As an example, the mixed reality system may employ a discrete zoom mode (e.g., Figures 14A to 14B ), a hybrid zoom mode (as shown in FIG) may be employed in which a virtual object is displayed using two adjacent depth planes to generate an adjustment clue between the two depth planes. Figures 14C to 14D ), a multi-focus mode (such as Figures 14E to 14F In general, the mixed reality system can switch between these and other rendering modes during operation in response to various conditions. As an example, the mixed reality system can use a first rendering mode (e.g., a discrete zoom mode) when displaying a first type of content (e.g., text that can be provided at a single depth) and can use a second, different rendering mode (e.g., a multi-focus mode) when displaying a second type of content (e.g., content that can be provided at multiple depths simultaneously). In at least some embodiments, Figure 6 The light field rendering controller 618 may be configured to select which rendering mode to employ at any given time based on the various inputs and conditions discussed herein.

[0684] Single depth plane rendering mode (discrete zoom mode)

[0685] like Figure 14A and 14BAs shown, the wearable system described herein can use a single depth plane to render virtual reality objects at a time in what is referred to herein as a single depth plane rendering mode (also known as a discrete zoom mode). In discrete zoom mode, the wearable system can use a single depth plane across the entire field of view (FOV) of the display to display all currently rendered virtual objects (even if some of these objects are set at depths other than the depth associated with the depth plane). In other words, the wearable system can provide a single focus or accommodation cue across the entire FOV. Of course, the wearable system can switch the depth plane used to render content over time, thereby changing the accommodation cues of the rendered virtual content over time (in response to changes in the user's convergence depth, the depth of the virtual content, and other facts described in more detail herein). Typically, in discrete zoom mode, the mixed reality system only uses one depth plane at any one time.

[0686] Figures 14A to 14B Shown are top and side / isometric views, respectively, of a mixed reality system 1400A operating in discrete zoom modes to present content to a user's eye 1402 . Figures 14A to 14B Also shown are depth planes 1421, 1422, and 1423, onto which virtual content can be projected and transformed into virtual pixels. When operating in discrete zoom mode, virtual content can only be projected onto one of the depth planes 1421-1423 at a given time. Figures 14A to 14B In the example shown in FIG4 , mixed reality system 1400A has switched to a state in which virtual content is projected onto depth plane 1422, but not onto either depth plane 1421 or 1423. As shown, the virtual content projected onto depth plane 1422 is transformed into virtual pixels 1432A.

[0687] If combined Figure 4 、 6 As discussed in more detail in Figures 7A, 7B, and 12, the depth at which all pixels are placed (e.g., the depth plane on which the virtual content is projected) and the timing of switching between depth planes can be selected based on eye tracking and virtual content information. As an example, a wearable system operating in a discrete zoom mode can be based on the user's vergence depth, based on the user's vergence depth, but depth plane switching is delayed until a triggering event (e.g., when the user blinks or saccades, after a predetermined AVM mismatch timeout, as combined with Figure 12 etc.), based on the depth of the virtual content, based on the depth of the virtual content but the depth plane switching is delayed until a triggering event (e.g., when the user blinks or glances, after a predetermined AVM mismatch timeout, as in conjunction with Figure 12In at least some embodiments, one or more of the depth plane selection module 750, the cross-hysteresis band detection module 752, and the eye event detection module 754 can be configured, individually or in combination, to implement a desired depth plane switching scheme.

[0688] Hybrid Depth Planar Rendering Mode (Hybrid Zoom Mode)

[0689] like Figure 14C and 14D As shown, the wearable system described herein can use two adjacent depth planes to render virtual reality objects to generate accommodation or focus cues located between the depth planes. In some embodiments, this hybrid zoom mode can enable the wearable system to generate accommodation or focus cues at any distance (located between and including the depths provided by a set of depth planes in the wearable system). In other words, if the system includes three depth planes, the first depth plane provides accommodation cues at 1 foot, the second depth plane provides accommodation cues at 10 feet, and the third depth plane provides accommodation cues at optical infinity, then the wearable system is able to provide accommodation cues to the user at any position from a depth of 1 foot in the first plane to an optically infinite depth in the third depth plane.

[0690] Figures 14C to 14D A top view and a side view / isometric view of a mixed reality system 1400C operating in a hybrid zoom mode to present content to a user's eye 1402 are shown. The mixed reality system 1400C may, for example, have the same Figures 14A to 14B When operating in hybrid zoom mode, virtual content can be simultaneously projected onto two or more of the depth planes 1421-1423 at any given point in time to generate accommodation cues between the planes. Figures 14C to 14D In the example shown in FIG1 , mixed reality system 1400C has switched to a state in which virtual content is projected onto depth planes 1421 and 1422. As shown, the virtual content projected onto depth planes 1421 and 1422 is transformed into virtual pixel sets 1431C and 1432C, respectively. Virtual pixel sets 1431C and 1432C can be blended together and provide an accommodation cue to the user somewhere between depth planes 1421 and 1422 based on their relative intensities.

[0691] In hybrid zoom mode, the wearable system can provide the same focus or accommodation cues for all pixels across the display's FOV, and the accommodation cues can vary continuously between the depths of any pair of adjacent depth planes. By blending pixel intensities between the two depth planes, the wearable system can achieve continuously variable accommodation cues. As an example, the wearable system can display a virtual object with accommodation cues between depth planes 1422 and 1421 by rendering the virtual object in two depth planes 1421 and 1421. In another example where the virtual object is closer to the depth of depth plane 1421, the wearable system can render the virtual object with a greater light intensity (e.g., brightness) in depth plane 1421 than in depth plane 1422. In this arrangement, light from the two depth planes can blend so that the user perceives the virtual object as having accommodation cues located near depth plane 1421 (but still between planes 1421 and 1422).

[0692] In hybrid zoom mode, the wearable system is configured to select which adjacent depth planes to blend to provide the desired accommodation cues. However, because the accommodation cues can change continuously between planes by continuously changing the brightness, the timing of the depth plane switching may not be as important as the timing in discrete zoom mode. Therefore, the wearable system can be configured to switch the two depth planes that form a pair of adjacent depth planes without waiting for a triggering event such as a user blink, saccade, or AVM timeout. Instead, the wearable system can smoothly change the accommodation cues provided and the depth planes used over time as needed in response to the user's convergence depth, the depth of the virtual content, or a combination of these and other inputs.

[0693] Multi-depth plane rendering mode (multi-focus mode)

[0694] like Figure 14E and 14F As shown, the wearable system described herein can render virtual reality objects in a multi-focus mode using two or more depth planes, thereby generating two or more accommodation cues simultaneously. In other words, virtual content in a given frame can be presented across multiple depths simultaneously. As an example, the multi-focus mode can include using two or more depth planes to provide a first hybrid accommodation cue; using two or more depth planes to provide a second hybrid accommodation cue; using a single depth plane to provide a third accommodation cue; using a second single depth plane to provide a fourth accommodation cue; or using a combination of these and other focus modes to provide various accommodation cues in the manner described in conjunction with the hybrid zoom mode.

[0695] Figures 14E to 14FA top view and a side view / isometric view of a mixed reality system 1400E operating in a multi-focal mode to present content to a user's eye 1402 are shown. The mixed reality system 1400E may, for example, have the same Figures 14A to 14D The same architecture as the mixed reality systems 1400A and / or 1400C described above is used. When operating in multi-focus mode, virtual content can be simultaneously projected onto two or more depth planes 1421-1423 at any given point in time to generate two or more different focus cues. Figures 14E to 14F In the example of FIG, virtual content is simultaneously projected onto depth planes 1421, 1422, and 1423. As shown, the virtual content projected onto depth planes 1421-1423 is transformed into virtual pixel sets 1431E-1433E.

[0696] Typically, when operating in multifocal mode (or in hybrid zoom mode), a wearable system can provide mixed and non-mixed accommodation cues. Figure 14FAs shown, depth planes 1421 and 1422 can be configured to provide content 1450 using hybrid accommodation cues using pixels 1431E on depth plane 1421 and pixels 1432E on depth plane 1422, can be configured to provide content 1452 using different hybrid accommodation cues using pixels 1423F on depth plane 1422 and pixels 1433F on depth plane 1423, and can be configured to provide content 1454 using non-hybrid accommodation cues using pixels 1433G on depth plane 1423. Hybrid content 1452 can include content that extends within the depth range of the user's perspective and, therefore, includes a portion located at a depth associated with depth plane 1423 rendered as virtual pixel 1433F and another portion located at a different depth associated with depth plane 1422 rendered as virtual pixel 1432F. As an example, hybrid content 1452 can include non-overlapping content, such that virtual pixel 1432F does not overlap virtual pixel 1433F in a plane perpendicular to the depth axis. In other embodiments, blended content 1452 may include content located at a depth between planes 1422 and 1423, rendered by overlapping and blending virtual pixels 1433F and 1432F. Overlapping and blending pixels 1433F and 1432F may include changing the relative intensities of pixels 1433F and 1432F to shift the apparent depth toward or toward depth plane 1422. As an example, dimming pixel 1433F and amplifying or brightening pixel 1432F may have the effect of shifting the user's perceived depth toward depth plane 1422. Furthermore, overlapping pixels may have a different intensity (typically, a lower intensity) than non-overlapping pixels so that the overlapped and blended pixels have a desired intensity (to prevent the overlapped pixels from appearing overly bright due to the contributions of multiple depth planes to the light at that location in the display). These are merely illustrative examples, and in general, the wearable system may present any desired combination of blended and non-blended accommodation cues.

[0697] As discussed in conjunction with the hybrid zoom mode, the timing of depth plane switching may not be as important in modes with variable accommodation cues (e.g., hybrid zoom mode and multifocus mode) as it is in discrete zoom focus mode. Thus, the wearable system can be configured to switch the active depth plane in multifocus mode without waiting for a triggering event such as a user blink, saccade, or AVM timeout. Instead, the wearable system can smoothly change the accommodation cues provided and the depth plane used over time as needed in response to the user's convergence depth, the depth of the virtual content, or a combination of these and other inputs. However, in other embodiments, a triggering event, such as a user blink, saccade, or AVM timeout, can be utilized.

[0698] Effects of perspective center misalignment in various rendering modes

[0699] When projected onto one or more depth planes, it is desirable to render and view 3D virtual content from a particular center of perspective (CoP), which may be determined for the rendered world and the real world. When the content is rendered from the appropriate center of perspective in the rendered world, each pixel of the virtual screen may accurately appear to be the 3D virtual content when viewed from the appropriate center of perspective in the real world, which may include a specific position and orientation. However, if the same content is rendered from a different location in the rendered world or viewed from a different location in the real world, the 3D virtual content may not accurately resemble an image of the 3D virtual content. The rendering framework may be represented using a pinhole camera model, where the CoP is represented as a "virtual" or "rendering" pinhole camera that is positioned and oriented within the rendered world (e.g., 3D rendering space) in a manner that correctly captures the projection of the 3D virtual content. Additional details about the CoP are provided below. Figures 22 to 24B Describe, and Figures 15A to 15C and the corresponding description provided below further demonstrate the impact of CoP accuracy and / or precision on mixed reality system performance for each of various example rendering modes.

[0700] In operation, if the perspective of the virtual pinhole camera (in the rendered world frame) and the perspective of the user (in the real world frame) are misaligned, the digitized light field projected onto the retina of the user's eye may be used to form an image of the 3D virtual content that contains artifacts or exhibits other problematic characteristics. For simple scenes where one depth plane is employed, the misalignment between the virtual pinhole camera and the user's perspective may result in a digitized light field that forms an image of the 3D virtual content at an incorrect (unintended) location within the user's FOV. For scenes where two or more depth planes are employed, such as those where the depth planes described above are employed, the digitized light field may be used to form an image of the 3D virtual content that contains artifacts or exhibits other problematic characteristics. Figures 14A to 14F In scenarios with any of the described focus modes, misalignment between the virtual pinhole camera and the user's perspective can produce a digitized light field that forms an image of the 3D virtual content that appears to jump or pop between locations and / or includes visual artifacts (e.g., breaks and / or misalignments). More specifically, the occurrence of such misalignment in a mixed reality system operating in any of the described focus modes can result in the introduction of parallax shifts that cause 3D virtual content that has been planarized ...

Claims

1. A display system configured to project light into an eye of a user to display virtual image content in a field of view of the user, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, the pupil, and the cornea, the display system comprising: a frame configured to be supported on a head of the user; a head mounted display disposed on the frame, the display configured to project light into the eyes of the user to display virtual image content to the user's field of view with different amounts of at least one of divergence and collimation, whereby the displayed virtual image content appears to originate from different depths at different time periods; one or more eye-tracking cameras configured to image the user's eyes; as well as processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics being configured to obtain an estimate of a center of rotation of the eye based on images of the eye obtained by the one or more eye-tracking cameras, wherein to obtain the estimate of the center of rotation of the eye, the processing electronics is configured to: determining an array of positions based on a plurality of spatial positions on an image of the user's eyes obtained by the one or more eye tracking cameras, wherein the array of positions corresponds to at least a portion of an ellipse, and A plurality of linear paths are determined extending from locations on a first side of the array of locations through the array of locations to an opposite second side of the array of locations.

2. The display system according to claim 1 further comprises one or more light sources, wherein the one or more light sources are arranged on the frame relative to the user's eyes to illuminate the user's eyes, and the one or more eye tracking cameras use the light from the one or more light sources to form an image of the eyes.

3. The display system according to claim 2, wherein the one or more light sources include at least two light sources, and the at least two light sources are arranged on the frame relative to the user's eyes to illuminate the user's eyes. The display system of claim 3 , wherein the one or more light sources comprise infrared light emitters.

5. A display system according to claim 2, wherein one or more light sources form one or more bright spots on the eye, and the processing electronic device is configured to determine the position of the cornea based on the one or more bright spots.

6. A display system according to claim 5, wherein the cornea has a corneal sphere associated therewith, the corneal sphere having a center of curvature, and the processing electronic device is configured to determine the position of the center of curvature of the corneal sphere based on the one or more bright spots.

7. The display system of claim 1 wherein the cornea has a corneal sphere associated therewith, the corneal sphere having a center of curvature, and the processing electronics are configured to determine a location of the center of curvature of the corneal sphere.

8. The display system of claim 1, wherein the one or more eye tracking cameras are configured to image the pupil of the eye.

9. The display system of claim 1, wherein the processing electronics are configured to determine a location of a center of the pupil.

10. The display system of claim 1, wherein the processing electronics is configured to determine at least a portion of a boundary between the iris and the pupil.

11. The display system of claim 10, wherein the processing electronics is configured to determine a center of the boundary between the iris and the pupil.

12. The display system of claim 1, wherein the processing electronics are configured to determine a location of a center of the pupil in three-dimensional space relative to a center of curvature of the cornea.

13. The display system of claim 12, wherein the processing electronics are configured to determine the position and orientation of the optical axis based on the position of the center of the pupil in three-dimensional space.

14. The display system of claim 1, wherein the processing electronics are configured to determine a position and an orientation of the optical axis.

15. The display system of claim 14, wherein the processing electronics are configured to determine the position and orientation of the optical axis based on a position of a center of the pupil relative to a center of curvature of the cornea in three-dimensional space.

16. The display system of claim 1, wherein the processing electronics are configured to determine a location of the center of rotation of the eye based on a center of curvature of the cornea.

17. The display system of claim 1, wherein the processing electronics are configured to determine the position of the center of rotation of the eye based on the center of curvature of the cornea and the position and orientation of the optical axis.

18. The display system of claim 17, wherein the processing electronics are configured to determine the location of the center of rotation of the eye by translating a particular distance along the optical axis from the center of curvature of the cornea.

19. The display system of claim 18, wherein the specific distance from the center of curvature to the center of rotation is between 4.0 mm and 6.0 mm.

20. The display system of claim 18, wherein the specific distance from the center of curvature to the center of rotation is 4.7 mm.

21. The display system according to claim 18, wherein the specific distance is fixed.

22. The display system of claim 18, wherein the processing electronic device is configured to determine the specific distance based at least on one or more images of the eye previously obtained by the one or more eye tracking cameras.

23. The display system of claim 1, wherein the processing electronics are configured to determine a position and orientation of a viewing axis offset from the optical axis based on the position and orientation of the optical axis.

24. The display system of claim 1, wherein the processing electronics are configured to determine a position and orientation of a viewing axis based on an angular rotation relative to the optical axis.

25. The display system of claim 1, wherein the processing electronics are configured to determine the position and orientation of the viewing axis based on an angular rotation between 4.0° and 6.5° relative to the optical axis.

26. The display system of claim 1, wherein the processing electronics are configured to determine the position and orientation of the viewing axis based on an angular rotation of 5.2° relative to the optical axis.

27. A display system according to claim 1, wherein the processing electronic device is configured to determine the position and orientation of the visual axis based at least on one or more images of the eye previously obtained by the one or more eye tracking cameras.

28. A display system according to claim 1, wherein the processing electronic device is configured to determine the center of rotation of the eye based on multiple determinations of the position of the optical axis or visual axis over a time period during which the eye rotates.

29. A display system according to claim 1, wherein the processing electronic device is configured to determine the center of rotation of the eye by identifying multiple determined intersecting areas of the position of the optical axis or visual axis within a time period of rotation of the eye.

30. The display system of claim 1, wherein the processing electronic device is configured to determine a vergence distance of the user where the user's left and right eyes are looking based on a determination of a position and orientation of optical axes of the user's left and right eyes.

31. A display system according to claim 1, wherein the processing electronic device is configured to determine the user's convergence distance where the user's left eye and right eye are looking based on determining the position and orientation of the visual axes of the user's left eye and right eye.

32. The display system of claim 1, wherein the processing electronic device is configured to determine a convergence distance at which the user's left and right eyes are looking based on identifying an area where the visual axes of the user's left and right eyes intersect.

33. A display system according to claim 1, wherein the processing electronic device is configured to determine the convergence distance at which the left eye and the right eye of the user are looking by projecting the visual axes of the left eye and the right eye of the user onto a horizontal plane and identifying an area of ​​intersection of the projections of the visual axes of the left eye and the right eye on the horizontal plane.

34. The display system of claim 1, wherein the processing electronics is configured to determine a relative amount of at least one of divergence and collimation for projected image content based on a determination of a convergence distance of the user.

35. The display system of claim 1, wherein the processing electronics comprises electronics located on the frame.

36. The display system of claim 1, wherein the processing electronics include electronics located on the frame and electronics disposed at a location remote from the frame.

37. The display system of claim 1, wherein the processing electronics include electronics located on the frame and electronics located on a waist pack.

38. A display system according to claim 1, wherein at least a portion of the display is transparent and is positioned in front of the user's eyes when the user wears the head-mounted display, so that the transparent portion transmits light from a portion of the environment in front of the user and the head-mounted display to the user's eyes to provide a view of the portion of the environment in front of the user and the head-mounted display.

39. A display system according to claim 1, wherein the head-mounted display receives light from a portion of the environment in front of the user at a first divergence, and transmits light from the portion of the environment in front of the user to the user's eyes at a second divergence that is substantially the same as the first divergence.

40. A display system according to claim 1, wherein the processing electronic device is configured to obtain the estimate of the center of rotation by filtering, averaging, applying a Kalman filter, or performing any combination of the above items to multiple estimated center of rotation positions.

41. A display system according to claim 1, wherein the processing electronic device is configured to present the virtual image content to the user's eyes, and the virtual image content is rendered as if captured by a camera having an aperture located at a determined rotational center position of the user's eyes.

42. The display system of claim 1, wherein the processing electronics is configured to render a virtual image to be presented to the eye using a rendering camera located at the center of rotation.

43. A display system according to claim 1, wherein the processing electronic device is configured to use a rendering camera, the rendering camera being configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture closer to the center of rotation than the retina of the eye.

44. A display system according to claim 1, wherein the processing electronic device is configured to use a rendering camera, the rendering camera being configured to render a virtual image to be presented to the eye, the virtual image being rendered as if captured by a camera having an aperture located at the rotation center of the eye.

45. A display system according to claim 1, wherein the processing electronic device is configured to render a virtual image to be presented to the eye using a rendering camera located at the rotation center, the rendering camera being modeled as having an aperture located at the rotation center of the eye.

46. ​​A display system according to claim 1, wherein the processing electronic device is configured to obtain the estimate of the rotation center of the eye based on determining multiple gaze directions of the user's eye during a time period of rotation of the eye based on images of the eye obtained through the one or more eye tracking cameras.

47. A display system according to claim 46, wherein the processing electronic device is configured to determine the gaze direction based on changes in the shape of one or more of the pupil, iris, or limbus of the user's eye in the image obtained by the one or more eye tracking cameras over a time period during which the eye rotates.

48. A display system according to claim 1, wherein the processing electronic device is configured to determine the array of positions by fitting a curve to the multiple spatial positions on the image of the user's eye.

49. The display system of claim 48, wherein the curve comprises an ellipse.

50. A display system according to claim 1, wherein the multiple spatial locations on the image include spatial locations on the corneal limbus of the user's eye in the image.

51. A display system according to claim 1, wherein the multiple spatial locations on the image include spatial locations on a boundary between the iris and sclera of the user's eye in the image.

52. A display system according to claim 1, wherein the multiple spatial positions on the image include spatial positions on a boundary between the cornea and sclera of the user's eye in the image obtained by the one or more eye tracking cameras.

53. The display system of claim 1, wherein the processing electronics is configured to determine a circular area based on the plurality of linear paths, the circular area having a radius R.

54. A display system according to claim 53, wherein the radius R corresponds to the average radius of the corneal limbus.

55. A display system according to claim 53, wherein the radius R corresponds to a measured radius of the corneal limbus of the user's eye.

56. The display system of claim 53, wherein the radius R corresponds to an average radius of a pupil.

57. A display system according to claim 53, wherein the radius R corresponds to the measured radius of the pupil of the user's eye.

58. The display system of claim 53, wherein the processing electronics are configured to determine a position and direction of a normal through a central portion of the circular area.

59. A display system according to claim 53, wherein the processing electronic device is configured to determine the corresponding positions and directions of multiple normals passing through the central portion of each circular area based on multiple images of the eye previously obtained by the one or more eye tracking cameras.

60. The display system of claim 59, wherein the processing electronics are configured to determine locations where the plurality of normal lines converge or intersect.

61. A display system according to claim 59, wherein the processing electronic device is configured to obtain an estimate of the center of rotation of the user's eye by identifying an area of ​​intersection of multiple of the normals determined based on images of the user's eye obtained over a time period of rotation of the eye.

62. A display system according to claim 59, wherein the processing electronic device is configured to obtain an estimate of the center of rotation of the user's eye based on the position and direction of multiple of the multiple normals determined based on images of the eye obtained over a time period in which the user's eye rotates.

63. The display system of claim 1, wherein a position on the first side of the array positions corresponds to an origin of a coordinate system of one of the one or more eye tracking cameras.

64. A method of rendering virtual image content in a display system, the display system being configured to project light into an eye of a user to display the virtual image content in the user's field of view, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, the pupil, and the cornea, the method comprising: Determining, by one or more eye tracking cameras configured to image the eyes of the user to track movement of the eyes, a position of a center of rotation of the eyes, wherein determining the position of the center of rotation of the eyes comprises: determining an array of positions based on a plurality of spatial positions on an image of the user's eyes obtained by the one or more eye tracking cameras, wherein the array of positions corresponds to at least a portion of an ellipse, and determining a plurality of linear paths extending from locations on a first side of the array of locations through the array of locations to an opposite second side of the array of locations; rendering, by a rendering engine, virtual image content using a rendering camera located at the rotation center of the eye, the rendering camera being configured to render a virtual image to be presented to the eye; and Light is projected into the user's eyes through a head-mounted display to display rendered virtual image content to the user's field of view with different divergences, so that the virtual image content appears to originate from different depths at different time periods.

65. A display system configured to project light into an eye of a user to display virtual image content in a field of view of the user, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, the pupil, and the cornea, the display system comprising: a frame configured to be supported on a head of the user; a head mounted display disposed on the frame, the display configured to project light into the user's eyes to display virtual image content to the user's field of view with different divergences, whereby the displayed virtual image content appears to originate from different depths at different time periods, wherein the head mounted display is configured to project light having a first divergence into the user's eyes during a first time period, and is configured to project light having a second divergence into the user's eyes during a second time period, wherein the first divergence is different from the second divergence; one or more eye-tracking cameras configured to image the user's eyes; as well as processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics configured to obtain an estimate of a center of rotation of the eye based on images of the eye obtained by the one or more eye-tracking cameras, obtain an estimate of a vergence distance of the user based on images of the eye obtained by the one or more eye-tracking cameras, and transform from projecting light into the user's eye with the first divergence to projecting light into the user's eye with the second divergence based on the user's estimated vergence distance, wherein to obtain the estimate of the center of rotation of the eye, the processing electronics is configured to: determining an array of positions based on a plurality of spatial positions on an image of the user's eyes obtained by the one or more eye tracking cameras, wherein the array of positions corresponds to at least a portion of an ellipse, and A plurality of linear paths are determined extending from locations on a first side of the array of locations through the array of locations to an opposing second side of the array of locations.

66. A method of rendering virtual image content in a display system, the display system being configured to project light into a left eye and a right eye of a user to display the virtual image content in the user's field of view, each of the eyes having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, the pupil, and the cornea, the method comprising: Determining, by one or more eye tracking cameras configured to image the eyes of the user to track movement of the eyes, a position of a center of rotation of the left eye and a position of a center of rotation of the right eye, wherein determining the positions of the centers of rotation of the eyes comprises: determining an array of positions based on a plurality of spatial positions on an image of the user's eyes obtained by the one or more eye tracking cameras, wherein the array of positions corresponds to at least a portion of an ellipse, and determining a plurality of linear paths extending from locations on a first side of the array of locations through the array of locations to an opposite second side of the array of locations; estimating, by processing electronics in communication with the one or more eye tracking cameras, an interpupillary distance of the user based on the determined positions of the centers of rotation of the left eye and the right eye; determining, by the one or more eye tracking cameras, a current left eye pose and a current right eye pose; and The processing electronic device estimates a current vergence distance of the user by comparing the estimated pupil distance with the determined current left eye posture and the determined current right eye posture.

67. A display system configured to project light into an eye of a user to display virtual image content in a field of view of the user, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, the pupil, and the cornea, the display system comprising: a frame configured to be supported on a head of the user; a head mounted display disposed on the frame, the display configured to project light into the eyes of the user to display virtual image content to the user's field of view with different amounts of at least one of divergence and collimation, whereby the displayed virtual image content appears to originate from different depths at different time periods; one or more eye-tracking cameras configured to image the user's eyes; as well as processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics being configured to obtain a position estimate of a center of rotation of the eye based on images of the eye obtained by the one or more eye-tracking cameras, and to obtain a direction estimate of the optical axis of the eye based on the images, wherein to obtain the position estimate of the center of rotation of the eye, the processing electronics is configured to: determining an array of positions based on a plurality of spatial positions on an image of the user's eyes obtained by the one or more eye tracking cameras, wherein the array of positions corresponds to at least a portion of an ellipse, and determining a plurality of linear paths extending from locations on a first side of the array of locations through the array of locations to an opposite second side of the array of locations, Wherein the processing electronic device is configured to present the virtual image content to the user's eye, the virtual image content being rendered as if captured by a camera having an aperture, the aperture being disposed along the optical axis and spaced 6.0 mm to 13.0 mm from the position of the estimated center of rotation of the eye in a direction away from the retina.

68. A display system configured to project light into an eye of a user to display virtual image content in a field of view of the user, the eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, the pupil, and the cornea, the display system comprising: a frame configured to be supported on a head of the user; a head mounted display disposed on the frame, the display being configured to project light into the user's eyes to display virtual image content to the user's field of view, at least a portion of the display being transparent and disposed at a position in front of the user's eyes when the user wears the frame, such that the transparent portion transmits light from a portion of the user's front environment and the head mounted display to the user's eyes to provide a view of the portion of the user's front environment and the head mounted display; an inward-facing imaging system configured to image an eye of the user; as well as processing electronics in communication with the inward-facing imaging system, the processing electronics configured to obtain an estimate of a center of rotation of the eye based on a plurality of images of the eye obtained by the inward-facing imaging system, the processing electronics configured to determine a variation in a calculated value of the center of rotation and to select a statistically determined estimate of the center of rotation based on the variation, wherein to obtain the estimate of the center of rotation of the eye, the processing electronics is configured to: determining an array of positions based on a plurality of spatial positions on an image of the user's eye obtained by the inward-facing imaging system, wherein the array of positions corresponds to at least a portion of an ellipse, and A plurality of linear paths are determined extending from locations on a first side of the array of locations through the array of locations to an opposing second side of the array of locations.

Images