Eye-imaging apparatus using diffractive optical elements

Abstract

To provide a favorable eye-imaging apparatus using diffractive optical elements.SOLUTION: Examples of an eye-imaging apparatus using diffractive optical elements are provided. For example, an optical device comprises: a substrate having a proximal surface and a distal surface; a first coupling optical element disposed on one of the proximal and distal surfaces of the substrate; and a second coupling optical element disposed on one of the proximal and distal surfaces of the substrate and offset from the first coupling optical element. The first coupling optical element can be configured to deflect light at an angle for total internal reflection (TIR) of the light between the proximal and distal surfaces toward the second coupling optical element. The second coupling optical element can be configured to deflect light at an angle out of the substrate. The eye-imaging apparatus can be used in a head-mounted display such as an augmented or virtual reality display.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 474,419, filed March 21, 2017, entitled "EYE-IMAGING APPARATUS USING DIFFRACTIVE OPTICAL ELEMENTS," the contents of which are incorporated herein by reference in their entirety.

[0002] The present disclosure relates to virtual and augmented reality imaging and visualization systems, and more particularly to a compact imaging system for obtaining images of the eye that uses coupling optical elements to direct light to a camera assembly. [Background technology]

[0003] Modern computing and display technology has facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner that appears or can be perceived as real. Virtual reality or "VR" scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual input, while augmented reality or "AR" scenarios typically involve the presentation of digital or virtual image information as an augmentation to the user's visualization of the real world around them. Mixed reality or "MR" scenarios are a type of AR scenario that typically involve virtual objects integrated into and responsive to the natural world. For example, in MR scenarios, AR image content is perceived as appearing blocked by or otherwise interacting with objects in the real world.

[0004] Referring to Figure 1, an augmented reality scene 10 is depicted in which a user of the AR technology sees a real-world park-like setting 20 featuring people, trees, a building in the background, and a concrete platform 30. In addition to these items, the user of the AR technology also perceives that they are "seeing" "virtual content," such as a robotic figure 40 standing on the real-world platform 30 and a flying, cartoon-like avatar character 50 that appears to be an anthropomorphic bumblebee, although these elements 40, 50 do not exist in the real world. The human visual perception system is complex, making it difficult to create AR technology that facilitates a comfortable, natural-feeling, and rich presentation of virtual image elements among other virtual or real-world image elements.

[0005] The systems and methods disclosed herein address various challenges associated with AR and VR technologies. Summary of the Invention [Means for solving the problem]

[0006] Various implementations of methods and apparatus within the scope of the appended claims each have several aspects, no single one of which is responsible for the desirable attributes disclosed herein. Without limiting the scope of the appended claims, some prominent features will be described herein.

[0007] One aspect of the present disclosure provides for imaging an object with a camera assembly that does not directly view the object. Thus, according to embodiments described herein, an optical device is configured to direct light from the object to an off-axis camera assembly to capture an image of the object as if it were in direct view.

[0008] In some embodiments, systems, devices, and methods are disclosed for obtaining images of an object using an off-axis camera assembly. In one implementation, an optical device is disclosed that may include a substrate having a proximal surface and a distal surface, a first coupling optical element disposed on one of the proximal and distal surfaces of the substrate, and a second coupling optical element disposed on one of the proximal and distal surfaces of the substrate and offset from the first coupling optical element. The first coupling optical element may be configured to deflect light at an angle and cause total internal reflection (TIR) ​​of the light between the proximal and distal surfaces toward the second coupling optical element. The second coupling optical element may be configured to deflect light at an angle away from the substrate. In some embodiments, at least one of the first and second coupling optical elements includes a plurality of diffractive features.

[0009] In some embodiments, systems, devices, and methods are disclosed for obtaining images of an object using an off-axis camera assembly. In one implementation, a head-mounted display (HMD) configured to be worn on a user's head is disclosed, which may include a frame, a pair of optical elements supported by the frame such that each optical element of the pair can be positioned in front of the user's eyes, and an imaging system. The imaging system may include a camera assembly mounted on the frame and an optical device for directing light to the camera assembly. The optical device may include a substrate having a proximal surface and a distal surface, a first coupling optical element disposed on one of the proximal and distal surfaces of the substrate, and a second coupling optical element disposed on one of the proximal and distal surfaces of the substrate and offset from the first coupling optical element. The first coupling optical element may be configured to deflect light at an angle and TIR the light between the proximal and distal surfaces toward the second coupling optical element. The second coupling optical element may be configured to deflect the light out of the substrate at an angle.

[0010] In some embodiments, systems, devices, and methods are disclosed for obtaining images of an object using an off-axis camera assembly. In one implementation, an imaging system is disclosed that may include a substrate having a proximal surface and a distal surface. The substrate may include a first diffractive optical element disposed on one of the proximal and distal surfaces of the substrate and a second diffractive optical element disposed on one of the proximal and distal surfaces of the substrate and offset from the first coupling optical element. The first diffractive optical element may be configured to deflect light at an angle and TIR the light between the proximal and distal surfaces toward the second coupling optical element. The second diffractive optical element may be configured to deflect light incident thereon at an angle away from the substrate. The imaging system may also include a camera assembly and image the light deflected by the second coupling optical element. In some embodiments, the first and second diffractive optical elements comprise at least one of an off-axis diffractive optical element (DOE), an off-axis grating, an off-axis diffractive optical element (DOE), an off-axis holographic mirror (OAHM), or an off-axis volumetric diffractive optical element (OAVDOE), an off-axis cholesteric liquid crystal grating (OACLCG), a hot mirror, a prism, or a surface of a decorative lens.

[0011] In some embodiments, systems, devices, and methods are disclosed for obtaining an image of an object using an off-axis camera assembly. The method may include providing an imaging system in front of the object to be imaged. The imaging system may be a substrate, which may include a first coupling optical element and a second coupling optical element, each disposed on one of a proximal surface and a distal surface of the substrate and offset from each other. The first coupling optical element may be configured to deflect light at an angle and TIR the light between the proximal and distal surfaces toward the second coupling optical element. The second coupling optical element may be configured to deflect the light at an angle away from the substrate. The method may also include capturing light using a camera assembly oriented to receive light deflected by the second coupling optical element and generating an off-axis image of the object based on the captured light.

[0012] In any of the embodiments, the proximal and distal surfaces of the substrate can be, but need not be, parallel to one another. For example, the substrate may comprise a wedge.

[0013] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Neither this summary nor the following detailed description purports to define or limit the scope of the inventive subject matter. The present specification also provides, for example, the following items: (Item 1) 1. An optical device, comprising: a substrate having a proximal surface and a distal surface; a first coupling optical element disposed on one of the proximal surface and the distal surface; a second coupling optical element disposed on one of the proximal surface and the distal surface and laterally offset from the first coupling optical element along a direction parallel to the proximal surface or the distal surface; wherein the first coupling optical element is configured to deflect light at an angle and total internally reflect (TIR) ​​the light between the proximal and distal surfaces toward the second coupling optical element, and the second coupling optical element is configured to deflect light at an angle away from the substrate. (Item 2) Item 10. The optical device of item 1, wherein the substrate is transparent to visible light. (Item 3) Item 10. The optical device of item 1, wherein the substrate comprises a polymer. (Item 4) Item 10. The optical device of item 1, wherein the substrate comprises polycarbonate. (Item 5) Item 1, an optical device according to item 1, wherein the first and second coupling optical elements are external to and fixed to at least one of the proximal and distal surfaces of the substrate. (Item 6) Item 1, wherein the first and second coupling optical elements comprise a portion of the substrate. (Item 7) Item 10. The optical device of item 1, wherein at least one of the first and second coupling optical elements comprises a plurality of diffractive features. (Item 8) Item 8. The optical device of item 7, wherein the plurality of diffractive features have a relatively high diffraction efficiency for a range of wavelengths so as to diffract substantially all of the light in the range of wavelengths. (Item 9) Item 8. The optical device of item 7, wherein the plurality of diffractive features diffract light in at least one direction based in part on a period of the plurality of diffractive elements, the at least one direction being selected to TIR the light between the proximal and distal surfaces. (Item 10) Item 1, the optical device of item 1, wherein at least one of the first or second coupling optical elements comprises at least one of an off-axis diffractive optical element (DOE), an off-axis diffraction grating, an off-axis diffractive optical element (DOE), an off-axis holographic mirror (OAHM), an off-axis volumetric diffractive optical element (OAVDOE), or an off-axis cholesteric liquid crystal grating (OACLCG). (Item 11) Item 10. The optical device of item 1, wherein the first and second coupling optical elements are each configured to deflect light in a first wavelength range while transmitting light in a second wavelength range. (Item 12) Item 12. The optical device of item 11, wherein the first range of wavelengths includes light in at least one of the infrared (IR) or near-IR spectrum, and the second range of wavelengths includes light in the visible spectrum. (Item 13) Item 1, an optical device according to item 1, wherein the first and second coupling optical elements selectively reflect light within a range of wavelengths, the first coupling optical element being disposed on a distal surface of the substrate, and the second coupling optical element being disposed on a proximal surface of the substrate. (Item 14) Item 1, an optical device according to item 1, wherein the first and second coupling optical elements selectively transmit light within a range of wavelengths, the first coupling optical element being disposed on a proximal surface of the substrate, and the second coupling optical element being disposed on a distal surface of the substrate. (Item 15) Item 1, an optical device according to item 1, wherein the first coupling optical element selectively reflects light in a range of wavelengths and the second coupling optical element selectively transmits light in the range of wavelengths, and the first and second coupling optical elements are disposed on a distal surface of the substrate. (Item 16) Item 1, an optical device according to item 1, wherein the first coupling optical element selectively transmits light in a range of wavelengths, and the second coupling optical element selectively reflects light in the range of wavelengths, and the first and second coupling optical elements are disposed on a proximal surface of the substrate. (Item 17) 1. A head-mounted display (HMD) configured to be worn on a user's head, the HMD comprising: The frame and a pair of optical elements, the pair of optical elements being supported by the frame such that each optical element of the pair of optical elements can be positioned in front of an eye of the user; 1. An imaging system, comprising: a camera assembly mounted to the frame; The optical device according to item 1; an imaging system comprising: Equipped with an HMD. (Item 18) Item 18. The HMD of item 17, wherein at least one optical element of the pair of optical elements includes the substrate. (Item 19) Item 18. The HMD of item 17, wherein the substrate is positioned on a surface of at least one optical element of the pair of optical elements. (Item 20) Item 18. The HMD of item 17, wherein the frame includes a pair of ear hooks, and the camera assembly is mounted on one of the pair of ear hooks. (Item 21) Item 18. The HMD of item 17, wherein the camera assembly is a forward-facing camera assembly configured to image light received from the second combining optical element. (Item 22) Item 18. The HMD of item 17, wherein the camera assembly is a rear-facing camera assembly positioned in a direction facing toward the user, and the rear-facing camera assembly is configured to image light received from the second combining optical element. (Item 23) Item 18. The HMD of item 17, further comprising a light source that emits light in a first wavelength range toward at least one of the user's eye, a portion of the eye, or a portion of tissue surrounding the eye. (Item 24) Item 24. The HMD of item 23, wherein light in the first wavelength range is reflected by at least one of the user's eye, a portion of the eye, or a portion of tissue surrounding the eye toward the first coupling optical element. (Item 25) Item 18. The HMD of item 17, wherein each of the pair of optical elements is transparent to visible light. (Item 26) Item 18. The HMD of item 17, wherein each of the pair of optical elements is configured to display an image to a user. (Item 27) Item 18. The HMD of item 17, wherein the camera assembly is configured to image at least one of the user's eye, a portion of the eye, or a portion of tissue surrounding the eye based in part on light received from the second coupling optical element. (Item 28) 28. The HMD of item 27, wherein the HMD is configured to track the user's line of sight based on an image of at least one of the user's eye, a portion of the eye, or a portion of the tissue surrounding the eye. (Item 29) Item 28. The HMD of item 27, wherein the image formed by the camera assembly corresponds to an image formed by a camera positioned in front of the user's eye and directly viewing at least one of the user's eye, a portion of the eye, or a portion of the tissue surrounding the eye. (Item 30) Item 18. The HMD of item 17, wherein the optical device is arranged to reduce stray light received by the camera assembly. (Item 31) Item 18. The HMD of item 17, wherein the size of the first coupling optical element is less than a stride distance of light reflected between the distal and proximal surfaces of the substrate, the stride distance being based on a thickness of the substrate and an angle at which the first coupling optical element deflects the light. (Item 32) Item 32. The HMD of item 31, wherein the size of the first coupling optical element is based on the field of view of the user's eye. (Item 33) Item 18. An HMD as described in Item 17, wherein an image of the user's eye formed by the camera assembly is indistinguishable from an image of the user's eye formed by a camera installed in front of the user's eye. (Item 34) a non-transitory data storage device configured to store images acquired by the camera assembly; a hardware processor in communication with the non-transitory data storage device, the hardware processor being programmed with executable instructions to analyze the images and perform one or more of eye tracking, biometric identification, multi-view reconstruction of eye shape, estimation of an ocular accommodation state or imaging of the retina, iris, or other distinctive patterns of the eye, and assessment of a physiological state of the user; Item 18. The HMD of item 17, further comprising: (Item 35) 1. An imaging system comprising: A substrate having a proximal surface and a distal surface, the substrate comprising: a first diffractive optical element disposed on one of the proximal surface and the distal surface; a second diffractive optical element disposed on one of the proximal surface and the distal surface, the second diffractive optical element being offset from the first diffractive optical element along a direction parallel to the proximal surface or the distal surface; a substrate, wherein the first diffractive optical element is configured to deflect light at an angle and total internally reflect (TIR) ​​the light between the proximal and distal surfaces towards a second coupling optical element, and the second diffractive optical element is configured to deflect light incident thereon at an angle out of the substrate; a camera assembly for imaging the light deflected by the second diffractive optical element; and An imaging system comprising: (Item 36) Item 36. The imaging system of item 35, wherein the first and second diffractive optical elements comprise at least one of an off-axis diffractive optical element (DOE), an off-axis diffraction grating, an off-axis diffractive optical element (DOE), an off-axis holographic mirror (OAHM), or an off-axis volumetric diffractive optical element (OAVDOE), an off-axis cholesteric liquid crystal grating (OACLCG), a hot mirror, a prism, or a surface of a decorative lens. (Item 37) 1. A method of imaging an object using a virtual camera, the method comprising: providing an imaging system in front of an object to be imaged, said imaging system comprising: A substrate comprising a first coupling optical element and a second coupling optical element, each disposed on one of a proximal surface and a distal surface of the substrate and offset from one another, the first coupling optical element configured to deflect light at an angle and total internally reflect (TIR) ​​the light between the proximal and distal surfaces toward the second coupling optical element, and the second coupling optical element configured to deflect the light at an angle out of the substrate. and capturing the light deflected by the second coupling optical element with a camera assembly oriented to receive the light; generating an off-axis image of the object based on the captured light; A method comprising: (Item 38) Item 38. The method of item 37, wherein the first and second coupling optical elements each deflect light in a first wavelength range while transmitting light in a second wavelength range. (Item 39) Item 38. The method of item 37, further comprising illuminating the object with a first range of wavelengths emitted by a light source. (Item 40) analyzing the off-axis image; performing one or more of eye tracking, biometric identification, multi-view reconstruction of eye shape, estimation of the state of accommodation of the eye, or imaging of the retina, iris, or other distinctive patterns of the eye, and assessment of the physiological state of the user based in part on the analyzed off-axis images; Item 38. The method of item 37, further comprising: [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 illustrates a user's view of an augmented reality (AR) device.

[0015] [Figure 2] FIG. 2 illustrates an example of a wearable display system.

[0016] [Figure 3] FIG. 3 illustrates a conventional display system for simulating a three-dimensional image for a user.

[0017] [Figure 4] FIG. 4 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes.

[0018] [Figure 5] 5A-5C illustrate the relationship between the radius of curvature and the radius of focus.

[0019] [Figure 6] FIG. 6 illustrates an embodiment of a waveguide stack for outputting image information to a user.

[0020] [Figure 7] FIG. 7 illustrates an example of an output beam output by a waveguide.

[0021] [Figure 8]FIG. 8 illustrates an example of a stacked waveguide assembly where each depth plane contains an image formed using multiple different primary colors.

[0022] [Figure 9A] FIG. 9A illustrates a cross-sectional side view of an example of a set of stacked waveguides, each including an internal coupling optical element.

[0023] [Figure 9B] FIG. 9B illustrates a perspective view of the multiple stacked waveguide embodiment of FIG. 9A.

[0024] [Figure 9C] FIG. 9C illustrates a top-down plan view of the multiple stacked waveguide embodiment of FIGS. 9A and 9B.

[0025] [Figure 10A] 10A and 10B diagrammatically illustrate an exemplary imaging system comprising a combined optical element and camera assembly for tracking the eye. [Figure 10B] 10A and 10B diagrammatically illustrate an exemplary imaging system comprising a combined optical element and camera assembly for tracking the eye.

[0026] [Figure 11] FIG. 11 diagrammatically illustrates another exemplary imaging system comprising multiple combining optical elements for total internal reflection of light from an object through a substrate and imaging the object at a camera assembly.

[0027] [Figure 12A] FIG. 12A diagrammatically illustrates another exemplary imaging system comprising multiple combining optical elements for total internal reflection of light from an object through a substrate and imaging the object at a camera assembly.

[0028] [Figure 12B]FIG. 12B is an example image of an object using the imaging system of FIG. 12A.

[0029] [Figure 13] 13A and 13B diagrammatically illustrate another exemplary imaging system comprising multiple combining optical elements for totally internally reflecting light from an object through a substrate and imaging the object at a camera assembly.

[0030] [Figure 14] 14A-18 diagrammatically illustrate several example arrangements of imaging systems for imaging objects. [Figure 15] 14A-18 diagrammatically illustrate several example arrangements of imaging systems for imaging objects. [Figure 16] 14A-18 diagrammatically illustrate several example arrangements of imaging systems for imaging objects. [Figure 17] 14A-18 diagrammatically illustrate several example arrangements of imaging systems for imaging objects. [Figure 18] 14A-18 diagrammatically illustrate several example arrangements of imaging systems for imaging objects.

[0031] [Figure 19] FIG. 19 is a process flow diagram of an example method for imaging an object using an off-axis camera. DETAILED DESCRIPTION OF THE INVENTION

[0032] Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure. (overview)

[0033] Head-mounted displays (HMDs) can use information about the state of a wearer's eyes for various purposes. For example, this information can be used for estimating the wearer's gaze direction, biometric identification, vision studies, assessing the wearer's physiological state, and the like. However, imaging the eyes can be difficult. The distance between the HMD and the wearer's eyes is short. Furthermore, gaze tracking requires a wide field of view (FOV), while biometric identification requires a relatively high number of pixels on the iris target. For imaging systems that seek to accomplish both of these objectives, these requirements are largely in conflict. Furthermore, both issues can be further complicated by occlusions by the eyelids and eyelashes. Some current implementations for tracking eye movement use cameras mounted on the HMD, aimed directly toward the eyes and capturing direct images of the eyes. However, to achieve the desired FOV and pixel count, the cameras are mounted within the wearer's FOV and therefore tend to obstruct and interfere with the wearer's ability to see the surrounding world. Other implementations image the eye directly, moving the camera so as not to obstruct the wearer's view, but this results in imaging the eye from a high angle, causing image distortion and reducing the field of view available for imaging the eye.

[0034] Embodiments of the imaging system described herein address some or all of these issues. Various embodiments described herein provide devices and systems that allow the wearer to view the surrounding world while being able to image the eye. For example, the imaging system can include a substrate positioned along a line of sight between the eye and a camera assembly. The substrate includes one or more coupling optical elements configured to direct light from the eye into the substrate. The substrate can act as a light guide (sometimes also referred to as a waveguide), directing the light toward the camera assembly. The light can then exit the substrate and be directed to the camera assembly via one or more coupling optical elements. The camera assembly, which receives the light, can thus capture an image of the eye (sometimes also referred to hereinafter as a "direct-view image") as if it were in a direct-view position from a distance (sometimes also referred to herein as "off-axis").

[0035] Some embodiments of the imaging system described herein provide a substrate comprising first and second coupling optical elements laterally offset from each other. The substrate includes a surface closest to the eye (sometimes also referred to herein as the proximal surface) and a surface farthest from the eye (sometimes also referred to herein as the distal surface). The first and second coupling optical elements described herein can be disposed on or adjacent to the proximal surface, on or adjacent to the distal surface, or within the substrate. The first coupling optical element (sometimes also referred to herein as the internal coupling optical element) can be configured to deflect light from the eye into the substrate such that the light propagates through the substrate by total internal reflection (TIR). The light can be incident on a second coupling optical element configured to extract the light and deflect it toward the camera assembly. As used herein, deflection may refer to the change in direction of light after interacting with something, for example, an optical component that deflects light may refer to reflection, diffraction, refraction, change in direction while transmitting through the optical component, etc.

[0036] In some embodiments, the imaging system described herein may be part of the display optics of an HMD (or a lens in a pair of glasses). One or more coupling optical elements may be selected to deflect a first range of wavelengths while allowing unimpeded propagation of a second range of wavelengths (e.g., a range of wavelengths different from the first range) through the substrate. The first range of wavelengths may be in the infrared (IR) range, and the second range of wavelengths may be in the visible range. For example, the substrate may include a reflective coupling optical element that reflects IR light while transmitting visible light. In effect, the imaging system acts as if there were a virtual camera assembly pointed back toward the wearer's eyes. Thus, the virtual camera assembly can image virtual IR light propagating through the substrate from the wearer's eyes, while visible light from the outside world can be transmitted through the substrate and perceived by the wearer.

[0037] The camera assembly may be configured to view the wearer's eye and, for example, capture an image of the eye. The camera assembly can be mounted close to the wearer's eye so that it does not obstruct the wearer's view of the surrounding world or interfere with the operation of the HMD. In some embodiments, the camera assembly can be positioned on the frame of the wearable display system, for example, on the ear piece, or integrated into the HMD eyepiece or below the eye and above the cheek. In some embodiments, a second camera assembly can be used for the wearer's other eye so that each eye can be imaged separately. The camera assembly can include an IR digital camera sensitive to IR radiation.

[0038] The camera assembly can be mounted so that it faces forward (in the direction of the wearer's vision) or so that it faces backward and is pointed toward the eyes. In some embodiments, by placing the camera assembly closer to the wearer's ear, the weight of the camera assembly can also be closer to the ear, and the HMD can be easier to wear compared to HMDs in which the camera assembly is located closer to the front of the HMD or in a direct-view arrangement. In addition, by placing the camera assembly near the wearer's temple, the distance from the wearer's eyes to the camera assembly is approximately doubled compared to a camera assembly located near the front of the HMD. Because the depth of field of the image is approximately proportional to this distance, the depth of field for the camera assembly is approximately twice as large compared to a direct-view camera assembly. A larger depth of field for the camera assembly can be advantageous for imaging the eye region of wearers with large or prominent noses, brow ridges, etc. In some embodiments, the location of the camera assembly can be based on HMD packaging or design considerations. For example, in some configurations it may be advantageous to position the camera assembly in a rearward or forward facing position.

[0039] Without subscribe to any particular scientific theory, the embodiments described herein may include several non-limiting advantages. Some embodiments may increase the physical distance between the camera assembly and the eye, which may facilitate positioning the camera assembly outside the wearer's field of view and therefore enable capture of direct-view images of the eye while not obstructing the wearer's view. Some of the embodiments described herein may also be configured to enable eye tracking using a larger field of view than conventional systems and therefore enable eye tracking over a wide range of positions. The use of IR imaging may facilitate imaging the eye without interfering with the wearer's ability to see through the substrate and view the environment.

[0040] Reference is now made to the drawings, wherein like reference numbers refer to like parts throughout. Exemplary HMD Devices

[0041] FIG. 2 illustrates an example of a wearable display system 60. The display system 60 includes a display 70 and various mechanical and electronic modules and systems to support the functionality of the display 70. The display 70 may be coupled to a frame 80, which is wearable by a display system user or viewer 90 and configured to position the display 70 directly in front of the user's 90 eyes. The display 70 may, in some embodiments, be considered eyewear. In some embodiments, a speaker 100 is coupled to the frame 80 and configured to be positioned adjacent to the user's 90 ear canal (in some embodiments, another speaker, not shown, may be positioned adjacent the user's other ear canal to provide stereo / shapeable sound control). In some embodiments, the display system may also include one or more microphones 110 or other devices to detect sound. In some embodiments, the microphones may be configured to allow a user to provide input or commands to the system 60 (e.g., voice menu command selections, natural language queries, etc.) and / or enable audio communication with other persons (e.g., other users of similar display systems). The microphone may further be configured as an ambient sensor to collect audio data (e.g., sounds from the user and / or the environment). In some embodiments, the display system may also include an ambient sensor 120a, which may be separate from the frame 80 and attached to the body of the user 90 (e.g., on the head, torso, limbs, etc. of the user 90). The ambient sensor 120a, in some embodiments, may be configured to obtain data characterizing a physiological state of the user 90. For example, the sensor 120a may be an electrode.

[0042] 2, the display 70 is operably coupled by a communication link 130, such as wired or wireless connectivity, to a local data processing module 140, which may be mounted in a variety of configurations, such as fixedly attached to the frame 80, fixedly attached to a helmet or hat worn by the user, integrated into headphones, or otherwise removably attached to the user 90 (e.g., in a backpack-style configuration, in a belt-coupled configuration). Similarly, the sensor 120a may be operably coupled to the local processor and data module 140 by a communication link 120b, e.g., wired or wireless connectivity. The local processing and data module 140 may include a hardware processor and digital memory, such as non-volatile memory (e.g., flash memory or a hard disk drive), both of which may be utilized to aid in processing, caching, and storing data. The data may include a) data captured from sensors (e.g., which may be operatively coupled to frame 80 or otherwise attached to user 90), such as image capture devices (e.g., cameras, etc.), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, gyroscopes, and / or other sensors disclosed herein, and / or b) data obtained and / or processed using remote processing module 150 and / or remote data repository 160 (including data related to virtual content), possibly for passing to display 70 after processing or retrieval. Local processing and data module 140 may be operatively coupled to remote processing module 150 and remote data repository 160 by communication links 170, 180, such as via wired or wireless communication links, such that these remote modules 150, 160 are operatively coupled to each other and available as resources to local processing and data module 140.In some embodiments, local processing and data module 140 may include one or more of an image capture device, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope. In some other embodiments, one or more of these sensors may be mounted on frame 80 or may be a freestanding structure that communicates with local processing and data module 140 by a wired or wireless communication path.

[0043] 2 , in some embodiments, remote processing module 150 may comprise one or more processors configured to analyze and process data and / or image information. In some embodiments, remote data repository 160 may comprise a digital data storage facility, which may be available through the Internet or other networking configuration in a “cloud” resource configuration. In some embodiments, remote data repository 160 may include one or more remote servers that provide information, e.g., information for generating augmented reality content, to local processing and data module 140 and / or remote processing module 150. In some embodiments, all data is stored and all calculations are performed within the local processing and data module, allowing for fully autonomous use from the remote module.

[0044] The perception of an image as "three-dimensional" or "3-D" can be achieved by providing a slightly different presentation of the image to each eye of a viewer. FIG. 3 illustrates a conventional display system for simulating a three-dimensional image for a user. Two distinct images 190, 200, one for each eye 210, 220, are output to the user. The images 190, 200 are spaced from the eyes 210, 220 by a distance 230 along an optical axis, or z-axis, parallel to the viewer's line of sight. The images 190, 200 are flat, and the eyes 210, 220 can focus on the images by assuming a single accommodative state. Such a 3-D display system relies on the human visual system to combine the images 190, 200 and provide the perception of depth and / or scale of the combined image.

[0045] However, it should be appreciated that the human visual system is more complex and providing a realistic perception of depth is more difficult. For example, many viewers of conventional “3-D” display systems find such systems uncomfortable or may not perceive any sense of depth at all. Without being limited by theory, it is believed that a viewer of an object may perceive the object as “three-dimensional” due to a combination of vergence and accommodation. Vergence movement of two eyes relative to each other (e.g., rotation of the eyes such that the pupils move toward or away from each other, converging the lines of sight of the eyes, and fixating on an object) is closely linked to the focusing (or “accommodation”) of the eye lenses and pupils. Under normal conditions, a change in focus of the eye's lens, or accommodation of the eye, to change focus from one object to another at a different distance, will automatically cause a corresponding change in convergence and divergence to occur at the same distance, a relationship known as the "accommodation-vergence reflex" and pupil dilation or miosis. Similarly, a change in convergence and divergence will, under normal conditions, induce a corresponding change in accommodation of the lens shape and pupil size. As described herein, many stereoscopic or "3-D" display systems display a scene using slightly different presentations (and therefore slightly different images) to each eye so that a three-dimensional perspective is perceived by the human visual system. However, such systems, among other things, simply provide different presentations of a scene, but are uncomfortable for many viewers because they work against the "accommodation-vergence reflex" when the eyes view all image information in a single accommodated state. Display systems that offer better alignment between accommodation and convergence-divergence movements may create a more realistic and comfortable simulation of three-dimensional images, contributing to increased wear duration and, ultimately, compliance with diagnostic and therapy protocols.

[0046] FIG. 4 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes. With reference to FIG. 4 , objects at various distances from the eyes 210, 220 on the z-axis are accommodated by the eyes 210, 220 so that the objects are in focus. The eyes 210, 220 assume particular accommodated states, focusing objects at different distances along the z-axis. As a result, a particular accommodated state may be said to be associated with a particular one of the depth planes 240 having an associated focal length such that an object or portion of an object at a particular depth plane is in focus when the eye is in an accommodated state relative to that depth plane. In some embodiments, a three-dimensional image may be simulated by providing a different representation of an image for each eye 210, 220, and by providing a different representation of an image corresponding to each of the depth planes. While shown as separate for clarity of illustration, it should be understood that the fields of view of the eyes 210, 220 may overlap, for example, as the distance along the z-axis increases. Additionally, although shown as flat for ease of illustration, it should be understood that the contour of the depth plane may be curved in physical space so that all features within the depth plane are in focus with the eye in a particular accommodative state.

[0047] The distance between an object and the eye 210 or 220 can also change the amount of divergence of light from the object as viewed by that eye. Figures 5A-5C illustrate the relationship between distance and divergence of light rays. The distance between the object and the eye 210 is represented by decreasing distances R1, R2, and R3. As shown in Figures 5A-5C, light rays become more divergent as the distance to the object decreases. As the distance increases, the light rays become more collimated. In other words, the light field generated by a point (an object or part of an object) can be said to have a spherical wavefront curvature that is a function of the distance the point is from the user's eye. The curvature increases as the distance between the object and the eye 210 decreases. As a result, the divergence of light rays at different depth planes also differs, and the divergence increases as the distance between the depth plane and the viewer's eye 210 decreases. While only a single eye 210 is illustrated in Figures 5A-5C and other figures herein for clarity of illustration, it should be understood that the discussion regarding eye 210 may apply to both eyes 210 and 220 of a viewer.

[0048] Without being limited by theory, it is believed that the human eye is typically capable of interpreting a finite number of depth planes to provide depth perception. As a result, a highly realistic simulation of perceived depth can be achieved by providing the eye with different representations of images corresponding to each of these limited number of depth planes. The different representations may be focused separately by the viewer's eyes, thereby serving to provide depth cues to the user based on the ocular accommodation required to focus on different image features for a scene located on the different depth planes and / or based on the observation of different image features on the different depth planes that are out of focus. Example of a Waveguide Stack Assembly

[0049] FIG. 6 illustrates an example of a waveguide stack for outputting image information to a user. Display system 250 includes a stack of waveguides or stacked waveguide assembly 260 that can be utilized to provide a three-dimensional perception to the eye / brain using multiple waveguides 270, 280, 290, 300, 310. In some embodiments, display system 250 is system 60 of FIG. 2 , and FIG. 6 diagrammatically illustrates some portions of system 60 in greater detail. For example, waveguide assembly 260 may be part of display 70 of FIG. 2 . It should be understood that display system 250 may, in some embodiments, be considered a light field display.

[0050] Continuing with reference to FIG. 6 , the waveguide assembly 260 may also include multiple features 320, 330, 340, 350 between the waveguides. In some embodiments, the features 320, 330, 340, 350 may be one or more lenses. The waveguides 270, 280, 290, 300, 310 and / or multiple lenses 320, 330, 340, 350 may be configured to transmit image information to the eye using various levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a particular depth plane and configured to output image information corresponding to that depth plane. Image injection devices 360, 370, 380, 390, 400 may act as light sources for the waveguides and may be utilized to inject image information into waveguides 270, 280, 290, 300, 310, each configured to distribute incident light across each individual waveguide for output toward eye 210, as described herein. Light exits output surfaces 410, 420, 430, 440, 450 of image injection devices 360, 372, 380, 390, 400 and is injected into corresponding input surfaces 460, 470, 480, 490, 500 of waveguides 270, 280, 290, 300, 310. In some embodiments, each input surface 460, 470, 480, 490, 500 may be an edge of the corresponding waveguide or a portion of a major surface of the corresponding waveguide (i.e., one of the waveguide surfaces that directly faces the world 510 or the viewer's eye 210). In some embodiments, a single beam of light (e.g., a collimated beam) may be launched into each waveguide, outputting a total field of cloned collimated beams, directed toward the eye 210 at a particular angle (and divergence) corresponding to the depth plane associated with the particular waveguide. In some embodiments, a single one of the image launch devices 360, 370, 380, 390, 400 may be associated with and launch light into multiple (e.g., three) waveguides 270, 280, 290, 300, 310.

[0051] In some embodiments, each of the image input devices 360, 370, 384, 390, 400 is a discrete display that generates image information for input into a corresponding waveguide 270, 280, 290, 300, 310. In some other embodiments, the image input devices 360, 370, 380, 390, 400 are the output of a single multiplexed display that may, for example, send image information via one or more optical conduits (such as fiber optic cables) to each of the image input devices 360, 370, 380, 390, 400. It should be understood that the image information provided by the image input devices 360, 370, 380, 390, 400 may include light of different wavelengths or colors (e.g., different primary colors, as discussed herein).

[0052] In some embodiments, light injected into the waveguides 270, 280, 290, 300, 310 is provided by a light projector system 520, which includes a light module 530, which may include a light emitter such as a light emitting diode (LED). Light from the light module 530 may be directed and modified by a light modulator 540, e.g., a spatial light modulator, via a beam splitter 550. The light modulator 540 may be configured to vary the perceived intensity of the light injected into the waveguides 270, 280, 290, 300, 310. Examples of spatial light modulators include liquid crystal displays (LCDs), including liquid crystal on silicon (LCOS) displays.

[0053] In some embodiments, the display system 250 may be a scanning fiber display comprising one or more scanning fibers configured to project light in various patterns (e.g., raster scan, spiral scan, Lissajous pattern, etc.) into one or more waveguides 270, 280, 290, 300, 310 and ultimately to the viewer's eye 210. In some embodiments, the illustrated image injection devices 360, 370, 380, 390, 400 may diagrammatically represent a single scanning fiber or a bundle of scanning fibers configured to inject light into one or more waveguides 270, 280, 290, 300, 310. In some other embodiments, the illustrated image injection devices 360, 370, 380, 390, 400 may diagrammatically represent multiple scanning fibers or multiple bundles of scanning fibers, each configured to inject light into an associated one of the waveguides 270, 280, 290, 300, 310. It should be understood that one or more optical fibers may be configured to transmit light from the optical module 530 to one or more waveguides 270, 280, 290, 300, 310. It should be understood that one or more intervening optical structures may be provided between the scanning fiber or fibers and one or more waveguides 270, 280, 290, 300, 310, for example, to redirect light exiting the scanning fiber into one or more waveguides 270, 280, 290, 300, 310.

[0054] Controller 560 controls the operation of one or more of stacked waveguide assemblies 260, including the operation of image input devices 360, 370, 380, 390, 400, light source 530, and light module 540. In some embodiments, controller 560 is part of local data processing module 140. Controller 560 includes programming (e.g., instructions in a non-transitory medium) that coordinates the timing and provision of image information to waveguides 270, 280, 290, 300, 310, for example, according to any of the various schemes disclosed herein. In some embodiments, the controller may be a single integrated device or a distributed system connected by a wired or wireless communication channel. Controller 560 may, in some embodiments, be part of processing module 140 or 150 (FIG. 2).

[0055] Continuing with reference to FIG. 6 , waveguides 270, 280, 290, 300, 310 may be configured to propagate light within each individual waveguide via TIR. Each of waveguides 270, 280, 290, 300, 310 may be planar or have another shape (e.g., curved) with major top and bottom surfaces and edges extending between the major top and bottom surfaces. In the illustrated configuration, waveguides 270, 280, 290, 300, 310 may each include outcoupling optical elements 570, 580, 590, 600, 610 configured to extract light from the waveguide by redirecting the light to propagate within each individual waveguide and outputting image information from the waveguide to eye 210. The extracted light may also be referred to as outcoupled light, and the optical element that outcouples light may also be referred to as a light extraction optical element. The extracted beam of light may be output by the waveguide at a location where light propagating within the waveguide strikes the light extraction optical element. The outcoupling optical element 570, 580, 590, 600, 610 may be, for example, a grating including diffractive optical features as discussed further herein. While shown disposed on the bottom major surface of the waveguides 270, 280, 290, 300, 310 for ease of explanation and clarity of drawing, in some embodiments, the outcoupling optical element 570, 580, 590, 600, 610 may be disposed on the top and / or bottom major surfaces and / or directly within the volume of the waveguides 270, 280, 290, 300, 310, as discussed further herein. In some embodiments, the outcoupling optical elements 570, 580, 590, 600, 610 may be formed within a layer of material attached to a transparent substrate and forming the waveguides 270, 280, 290, 300, 310. In some other embodiments, the waveguides 270, 280, 290, 300, 310 may be monolithic material components, and the outcoupling optical elements 570, 580, 590, 600, 610 may be formed on and / or within the material components.

[0056] Continuing with reference to FIG. 6 , as discussed herein, each waveguide 270, 280, 290, 300, 310 is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 270 closest to the eye may be configured to deliver collimated light (injected into such waveguide 270) to the eye 210. The collimated light may represent an optical infinity focal plane. The next upper waveguide 280 may be configured to send collimated light that passes through a first lens 350 (e.g., a negative lens) before reaching the eye 210. Such first lens 350 may be configured to generate a slight convex wavefront curvature so that the eye / brain interprets light emerging from the next upper waveguide 280 as emerging from a first focal plane closer inward from optical infinity toward the eye 210. Similarly, the third upper waveguide 290 passes its output light through both the first lens 350 and the second lens 340 before reaching the eye 210. The combined refractive power of the first lens 350 and the second lens 340 may be configured to produce another, increasing amount of wavefront curvature such that the eye / brain interprets the light emerging from the third waveguide 290 as originating from a second focal plane that is closer inward from optical infinity towards the person than was the light from the next upper waveguide 280.

[0057] The other waveguide layers 300, 310 and lenses 330, 320 are similarly configured, with the highest waveguide 310 in the stack sending its output through all of the lenses between it and the eye for a collective focal power representing the focal plane closest to the person. To compensate for the stack of lenses 320, 330, 340, 350 when viewing / interpreting light originating from the world 510 on the other side of the stacked waveguide assembly 260, a compensating lens layer 620 may be placed on top of the stack to compensate for the collective power of the lower lens stacks 320, 330, 340, 350. Such a configuration provides as many perceived focal planes as there are available waveguide / lens pairs. Both the waveguide outcoupling optical elements and the focusing sides of the lenses may be static (i.e., not dynamic or electro-active). In some alternative embodiments, one or both may be dynamic using electro-active features.

[0058] In some embodiments, two or more of the waveguides 270, 280, 290, 300, 310 may have the same associated depth plane. For example, multiple waveguides 270, 280, 290, 300, 310 may be configured to output images set at the same depth plane, or multiple subsets of waveguides 270, 280, 290, 300, 310 may be configured to output images set at the same depth planes, with one set per depth plane. This may provide the advantage of forming tiled images to provide an extended field of view at those depth planes.

[0059] Continuing with reference to FIG. 6 , the outcoupling optical elements 570, 580, 590, 600, 610 may be configured to redirect light from its respective waveguide and output the light with an appropriate amount of divergence or collimation for a particular depth plane associated with that waveguide. As a result, waveguides with different associated depth planes may have different configurations of outcoupling optical elements 570, 580, 590, 600, 610, which output light with different amounts of divergence depending on the associated depth plane. In some embodiments, the light extraction optical elements 570, 580, 590, 600, 610 may be volume or surface features that may be configured to output light at specific angles. For example, the light extraction optical elements 570, 580, 590, 600, 610 may be volume holograms, surface holograms, and / or diffraction gratings. In some embodiments, the features 320, 330, 340, 350 may not be lenses. Rather, they may simply be spacers (eg, cladding layers and / or structures for forming air gaps).

[0060] In some embodiments, the outcoupling optical elements 570, 580, 590, 600, 610 are diffractive features or "diffractive optical elements" (also referred to herein as "DOEs") that form a diffraction pattern. Preferably, the DOEs have a sufficiently low diffraction efficiency so that only a portion of the light in the beam is deflected toward the eye 210 at each intersection of the DOE, while the remainder continues traveling through the waveguide via TIR. The light carrying the image information is thus split into several related output beams that exit the waveguide at multiple locations, resulting in a fairly uniform pattern of output emission toward the eye 210 for this particular collimated beam bouncing within the waveguide.

[0061] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer-dispersed liquid crystal in which microdroplets comprise a diffractive pattern in a host medium, and the refractive index of the microdroplets may be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets may be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0062] In some embodiments, a camera assembly 630 (e.g., a digital camera, including a visible light and IR light camera) may be provided to capture images of the eye 210, a portion of the eye 210, or at least a portion of the tissue surrounding the eye 210, for example, detecting user input, extracting biometric information from the eye, estimating and tracking the eye's gaze direction, monitoring a user's physiological condition, etc. As used herein, a camera may be any image capture device. In some embodiments, the camera assembly 630 may include an image capture device and a light source 632 that projects light (e.g., IR or near-IR light) onto the eye, which light may then be reflected by the eye and detected by the image capture device. In some embodiments, the light source 632 includes a light-emitting diode (“LED”) and emits IR or near-IR. While the light source 632 is illustrated as being attached to the camera assembly 630, it should be understood that the light source 632 may be positioned in other areas relative to the camera assembly such that light emitted by the light source (e.g., light source 530, described below) is directed toward the wearer's eye. In some embodiments, camera assembly 630 may be mounted on frame 80 (FIG. 2) and may be in electrical communication with processing module 140 or 150, which may process image information from camera assembly 630 and make various determinations regarding, for example, the user's physiological state, the wearer's gaze direction, iris identification, etc., as discussed herein. It should be understood that information regarding the user's physiological state may be used to determine the user's behavior or emotional state. Examples of such information include the user's movements or the user's facial expressions. The user's behavior or emotional state may then be triangulated with the collected environmental or virtual content data to determine a relationship between the behavior or emotional state, the physiological state, and the environmental or virtual content data. In some embodiments, one camera assembly 630 may be utilized per eye, monitoring each eye separately.

[0063] 7, an example of an output beam output by a waveguide is shown. While one waveguide is illustrated, it should be understood that other waveguides in waveguide assembly 260 (FIG. 6) may function similarly, and that waveguide assembly 260 includes multiple waveguides. Light 640 is launched into waveguide 270 at input surface 460 of waveguide 270 and propagates within waveguide 270 by TIR. At the point where light 640 impinges on DOE 570, a portion of the light exits the waveguide as output beam 650. Output beam 650 is illustrated as being approximately parallel, but may be redirected to propagate to eye 210 at an angle (e.g., divergent output beam formation), as discussed herein, and depending on the depth plane associated with waveguide 270. A nearly collimated exit beam may refer to a waveguide with outcoupling optics that outcouples light to form an image that appears to be set at a depth plane at a long distance (e.g., optical infinity) from the eye 210. Other waveguides or other sets of outcoupling optics may output a more divergent exit beam pattern, which would require the eye 210 to accommodate to a closer distance and focus on the retina, and would be interpreted by the brain as light from a distance closer to the eye 210 than optical infinity.

[0064] In some embodiments, a full-color image may be formed at each depth plane by overlaying an image in each of the primary colors, for example, three or more primary colors. FIG. 8 illustrates an example of a stacked waveguide assembly, with each depth plane including an image formed using multiple different primary colors. The illustrated embodiment shows depth planes 240a-240f, but more or fewer depths are also contemplated. Each depth plane may have three or more primary color images associated with it, including a first image of a first color G, a second image of a second color R, and a third image of a third color B. Different depth planes are indicated in the diagram by different numbers for diopters (dpt) following the letters G, R, and B. By way of example only, the number following each of these letters indicates the diopter (1 / m), i.e., the inverse distance of the depth plane from the viewer, and each box in the diagram represents an individual primary color image. In some embodiments, the exact locations of the depth planes for different primary colors may be varied to account for differences in the eye's focusing of light of different wavelengths. For example, different primary color images for a given depth plane may be placed on depth planes corresponding to different distances from the user. Such an arrangement may increase visual acuity and user comfort or reduce chromatic aberrations.

[0065] In some embodiments, light for each primary color may be output by a single dedicated waveguide, such that each depth plane may have multiple waveguides associated with it. In such embodiments, each box in the diagram containing the letter G, R, or B may be understood to represent an individual waveguide, and three waveguides may be provided per depth plane, resulting in three primary color images per depth plane. While the waveguides associated with each depth plane are shown adjacent to each other in this drawing for ease of illustration, it should be understood that in a physical device, the waveguides may all be arranged in a stack with one waveguide per level. In some other embodiments, multiple primary colors may be output by the same waveguide, such that, for example, only a single waveguide may be provided per depth plane.

[0066] 8, in some embodiments, G is green, R is red, and B is blue. In some other embodiments, other colors associated with other wavelengths of light, including magenta and cyan, may be used in addition to or may replace one or more of red, green, or blue. In some embodiments, features 320, 330, 340, and 350 may be active or passive optical filters configured to block or selectively transmit light from the surrounding environment to the viewer's eye.

[0067] It should be understood that references throughout this disclosure to a given color of light are understood to encompass light of one or more wavelengths within the range of wavelengths of light that are perceived by a viewer as that given color. For example, red light may include one or more wavelengths of light that are within the range of about 620-780 nm, green light may include one or more wavelengths of light that are within the range of about 492-577 nm, and blue light may include one or more wavelengths of light that are within the range of about 435-493 nm.

[0068] In some embodiments, the light source 530 (FIG. 6) may be configured to emit light at one or more wavelengths outside the viewer's visual perception range, such as IR or ultraviolet wavelengths. IR light can include light with wavelengths in the range of 700 nm to 10 μm. In some embodiments, IR light can include near-IR light with wavelengths in the range of 700 nm to 1.5 μm. Additionally, the waveguide in-coupling, out-coupling, and other light redirecting structures of the display 250 may be configured to direct and emit this light from the display toward the user's eye 210, for example, for imaging or user stimulation applications.

[0069] Referring now to FIG. 9A , in some embodiments, light impinging on a waveguide may need to be redirected to incoupling the light into the waveguide. An incoupling optical element may be used to redirect and incoupling the light into its corresponding waveguide. FIG. 9A illustrates a cross-sectional side view of an example of a plurality or set 660 of stacked waveguides, each including an incoupling optical element. The waveguides may each be configured to output light of one or more different wavelengths or one or more different wavelength ranges. Stack 660 may correspond to stack 260 ( FIG. 6 ), and the illustrated waveguides of stack 660 may correspond to a portion of multiple waveguides 270, 280, 290, 300, 310, although it should be understood that light from one or more of image injection devices 360, 370, 380, 390, 400 is injected into the waveguide from a location requiring the light to be redirected for incoupling.

[0070] The illustrated set 660 of stacked waveguides includes waveguides 670, 680, and 690. Each waveguide includes an associated internal coupling optical element (which may also be referred to as the light input area on the waveguide), for example, internal coupling optical element 700 is disposed on a major surface (e.g., the top major surface) of waveguide 670, internal coupling optical element 710 is disposed on a major surface (e.g., the top major surface) of waveguide 680, and internal coupling optical element 720 is disposed on a major surface (e.g., the top major surface) of waveguide 690. In some embodiments, one or more of internal coupling optical elements 700, 710, 720 may be disposed on the bottom major surface of an individual waveguide 670, 680, 690 (particularly, one or more internal coupling optical elements are reflective polarizing optical elements). As shown, the internal coupling optical elements 700, 710, 720 may be disposed on the upper major surface of the respective waveguide 670, 680, 690 (or on top of the next lower waveguide), and in particular, the internal coupling optical elements are transmissive turning optical elements. In some embodiments, the internal coupling optical elements 700, 710, 720 may be disposed within the body of the respective waveguide 670, 680, 690. In some embodiments, as discussed herein, the internal coupling optical elements 700, 710, 720 are wavelength selective, such that they selectively redirect one or more wavelengths of light while transmitting other wavelengths of light. While illustrated on one side or corner of the respective waveguide 670, 680, 690, it should be understood that the internal coupling optical elements 700, 710, 720 may be disposed within other areas of the respective waveguide 670, 680, 690 in some embodiments.

[0071] As shown, the in-coupling optical elements 700, 710, 720 may be laterally offset from one another. In some embodiments, each in-coupling optical element may be offset to receive light without that light passing through another in-coupling optical element. For example, each in-coupling optical element 700, 710, 720 may be configured to receive light from different image input devices 360, 370, 380, 390, and 400, as shown in FIG. 6 , and may be separated (e.g., laterally spaced) from the other in-coupling optical elements 700, 710, 720 so as to substantially not receive light from others of the in-coupling optical elements 700, 710, 720.

[0072] Each waveguide also includes an associated optically dispersive element, for example, optically dispersive element 730 is disposed on a major surface (e.g., the top major surface) of waveguide 670, optically dispersive element 740 is disposed on a major surface (e.g., the top major surface) of waveguide 680, and optically dispersive element 750 is disposed on a major surface (e.g., the top major surface) of waveguide 690. In some other embodiments, optically dispersive elements 730, 740, 750 may be disposed on the bottom major surfaces of associated waveguides 670, 680, 690, respectively. In some other embodiments, optically dispersive elements 730, 740, 750 may be disposed on both the top and bottom major surfaces of associated waveguides 670, 680, 690, respectively, or optically dispersive elements 730, 740, 750 may be disposed on different ones of the top and bottom major surfaces in different associated waveguides 670, 680, 690, respectively.

[0073] Waveguides 670, 680, 690 may be spaced apart and separated, for example, by gas, liquid, and / or solid layers of material. For example, as shown, layer 760a may separate waveguides 670 and 680, and layer 760b may separate waveguides 680 and 690. In some embodiments, layers 760a and 760b are formed from a low refractive index material (i.e., a material having a lower refractive index than the material forming the immediately adjacent waveguides 670, 680, 690). Preferably, the refractive index of the material forming layers 760a, 760b is 0.05 or more or 0.10 or less compared to the refractive index of the material forming waveguides 670, 680, 690. Advantageously, the lower refractive index layers 760a, 760b may function as cladding layers that promote TIR of light through the waveguides 670, 680, 690 (e.g., TIR between the top and bottom major surfaces of each waveguide). In some embodiments, the layers 760a, 760b are formed from air. Although not shown, it should be understood that the top and bottom of the illustrated set of waveguides 660 may include immediate cladding layers.

[0074] Preferably, for ease of manufacturing and other considerations, the materials forming waveguides 670, 680, 690 are similar or the same, and the materials forming layers 760a, 760b are similar or the same. In some embodiments, the materials forming waveguides 670, 680, 690 may differ between one or more waveguides, or the materials forming layers 760a, 760b may differ while still maintaining the various refractive index relationships discussed above.

[0075] 9A, light rays 770, 780, 790 enter the set of waveguides 660. It should be understood that light rays 770, 780, 790 may be injected into the waveguides 670, 680, 690 by one or more image injection devices 360, 370, 380, 390, 400 (FIG. 6).

[0076] In some embodiments, the light beams 770, 780, 790 have different properties, such as different wavelengths or different wavelength ranges, which may correspond to different colors. Each of the internal coupling optical elements 700, 710, 720 deflects the incident light so that the light propagates through a respective one of the waveguides 670, 680, 690 by TIR.

[0077] For example, in-coupling optical element 700 may be configured to deflect light ray 770 having a first wavelength or range of wavelengths. Similarly, transmitted light ray 780 impinges on and is deflected by in-coupling optical element 710, which is configured to deflect light of a second wavelength or range of wavelengths. Similarly, light ray 790 is deflected by in-coupling optical element 720, which is configured to selectively deflect light of a third wavelength or range of wavelengths.

[0078] 9A , the deflected light rays 770, 780, 790 are deflected to propagate through the corresponding waveguides 670, 680, 690. That is, the in-coupling optical element 700, 710, 720 of each waveguide deflects the light into its corresponding waveguide 670, 680, 690, in-coupling the light into the corresponding waveguide. The light rays 770, 780, 790 are deflected at an angle that causes the light to propagate through the respective waveguides 670, 680, 690 by TIR. The light rays 770, 780, 790 propagate through the respective waveguides 670, 680, 690 by TIR until they impinge on the waveguide's corresponding optical dispersive element 730, 740, 750.

[0079] 9B, a perspective view of the multiple stacked waveguide embodiment of FIG. 9A is illustrated. As previously described, in-coupled light rays 770, 780, 790 are deflected by in-coupling optical elements 700, 710, 720, respectively, and then propagate by TIR within waveguides 670, 680, 690, respectively. Light rays 770, 780, 790 then impinge on optically dispersive elements 730, 740, 750, respectively. Optically dispersive elements 730, 740, 750 deflect light rays 770, 780, 790 to propagate toward out-coupling optical elements 800, 810, 820, respectively.

[0080] In some embodiments, the optically dispersive elements 730, 740, 750 are orthogonal pupil expanders (OPEs). In some embodiments, the OPEs may both deflect or disperse light into the out-coupling optical elements 800, 810, 820 and also increase the beam or spot size of this light as it propagates into the out-coupling optical elements. In some embodiments, for example, if the beam size is already the desired size, the optically dispersive elements 730, 740, 750 may be omitted and the in-coupling optical elements 700, 710, 720 may be configured to deflect light directly into the out-coupling optical elements 800, 810, 820. For example, referring to FIG. 9A , the optically dispersive elements 730, 740, 750 may be replaced with the out-coupling optical elements 800, 810, 820, respectively, in some embodiments. In some embodiments, the outcoupling optical element 800, 810, 820 is an exit pupil (EP) or exit pupil expander (EPE) that directs light to the viewer's eye 210 ( FIG. 7 ). It should be understood that the OPE may be configured to increase the dimension of the eyebox in at least one axis, and the EPE may increase the eyebox in an axis that intersects (e.g., is perpendicular to) the axis of the OPE.

[0081] 9A and 9B, in some embodiments, a waveguide set 660 includes, for each primary color, waveguides 670, 680, 690, in-coupling optical elements 700, 710, 720, optically dispersive elements (e.g., OPEs) 730, 740, 750, and out-coupling optical elements (e.g., EPs) 800, 810, 820. The waveguides 670, 680, 690 may be stacked with an air gap / cladding layer between each one. The in-coupling optical elements 700, 710, 720 redirect or deflect incident light into that waveguide (with different in-coupling optical elements receiving light of different wavelengths). The light then propagates at an angle that will result in TIR within the individual waveguides 670, 680, 690. In the example shown, light ray 770 (e.g., blue light) is deflected by the first in-coupling optical element 700 in the manner described above, then continues bouncing down the waveguide, interacting with an optically dispersive element (e.g., OPE) 730 and then an out-coupling optical element (e.g., EP) 800. Light rays 780 and 790 (e.g., green and red light, respectively) pass through waveguide 670, with light ray 780 impinging on and being deflected by in-coupling optical element 710. Light ray 780 will then, via TIR, bounce down waveguide 680, to its optically dispersive element (e.g., OPE) 740 and then to the out-coupling optical element (e.g., EP) 810. Finally, light ray 790 (e.g., red light) passes through waveguide 690 and impinges on in-coupling optical element 720 of waveguide 690. The light in-coupling optical element 720 deflects the light ray 790 so that it propagates by TIR to the light dispersive element (e.g., OPE) 750 and then by TIR to the out-coupling optical element (e.g., EP) 820. The out-coupling optical element 820 then finally out-couples the light ray 790 to a viewer, who also receives the out-coupled light from the other waveguides 670, 680.

[0082] FIG. 9C illustrates a top-down plan view of an example of the multiple stacked waveguides of FIGS. 9A and 9B. As shown, waveguides 670, 680, 690 may be vertically aligned, along with each waveguide's associated optically dispersive elements 730, 740, 750 and associated out-coupling optical elements 800, 810, 820. However, as discussed herein, the in-coupling optical elements 700, 710, 720 are not vertically aligned. Rather, the in-coupling optical elements are preferably non-overlapping (e.g., laterally spaced apart, as seen in the top-down view). As discussed further herein, this non-overlapping spatial arrangement facilitates the injection of light from different sources into different waveguides on a one-to-one basis, thereby allowing a specific light source to be uniquely coupled to a specific waveguide. In some embodiments, arrays including non-overlapping, spatially separated in-coupling optical elements may be referred to as shifted-pupil systems, and the in-coupling optical elements in these arrays may correspond to sub-pupils. Exemplary Imaging System for Off-Axis Imaging

[0083] As described above, the eye or tissue surrounding the eye of a wearer of an HMD (e.g., wearable display system 200 shown in FIG. 2) can be imaged using multiple coupling optical elements that direct light from the eye through a substrate and into a camera assembly. The resulting images can be used to track one or both eyes, image the retina, reconstruct eye shape in three dimensions, extract biometric information from the eye (e.g., iris identification), etc.

[0084] As outlined above, there are various reasons why an HMD may use information about the state of the wearer's eyes. For example, this information can be used to estimate the wearer's gaze direction or for biometric identification. However, this problem is difficult due to the short distance between the HMD and the wearer's eyes. This is further complicated by the fact that gaze tracking requires a larger field of view, while biometric identification requires a relatively high number of pixels on the iris target. For an imaging system that would attempt to accomplish both of these objectives, the requirements of the two tasks are largely in conflict. Finally, both problems are further complicated by occlusion by the eyelids and eyelashes. Embodiments of the imaging system described herein may address at least some of these problems.

[0085] 10A and 10B schematically illustrate an example of an imaging system 1000a configured to image one or both eyes 210, 220 of a wearer 90. The imaging system 1000a comprises a substrate 1070 and a camera assembly 1030 arranged to view the eye 220. The embodiment of the imaging system 1000a described herein with reference to FIGS. 10A and 10B can be used in conjunction with an HMD, including a display device described herein (e.g., the wearable display system 200 shown in FIG. 2, the display system 250 shown in FIGS. 6 and 7, and the stack 660 of FIGS. 9A-9C). For example, in some implementations where imaging system 1000a is part of display system 250 of FIG. 6, substrate 1070 may replace one of waveguides 270, 280, 290, 300, or 310, may be positioned between waveguide stack 260 and eye 210, or may be positioned between waveguide stack 260 and world 510.

[0086] In some embodiments, camera assembly 1030 may be mounted proximate to the wearer's eye, for example, on frame 80 of wearable display system 60 of FIG. 2 (e.g., ear piece 82 near the wearer's temple), around the edge of display 70 of FIG. 2 (as shown in FIG. 10B), or integrated into display 70 of FIG. 2. Camera assembly 1030 may be substantially similar to camera assembly 630 of FIG. 6. In other embodiments, a second camera assembly may be used to separately image the wearer's other eye 210. Camera assembly 1030 may include an IR digital camera sensitive to IR radiation. Camera assembly 1030 may be mounted so as to face forward (e.g., toward the wearer's field of vision), as illustrated in FIG. 10A, or camera assembly 1030 may be mounted so as to face backward and be directed toward eye 220 (e.g., FIG. 10B).

[0087] In some embodiments, camera assembly 1030 may include an image capture device and a light source 1032 that projects light to eye 220, which may then be reflected by eye 220 and detected by camera assembly 1030. Although light source 1032 is illustrated as being attached to camera assembly 1030, light source 1032 may be positioned in other areas relative to the camera assembly such that light emitted by the light source is directed toward the wearer's eye and reflected to camera assembly 1030. For example, if imaging system 1000a is part of display system 250 (FIG. 6) and substrate 1070 replaces one of waveguides 270, 280, 290, 300, or 310, light source 1032 may be one of light emitters 360, 370, 380, 390, or light source 530.

[0088] In the embodiment illustrated in FIG. 10A , the camera assembly 1030 is positioned to view a proximal surface 1074 of a substrate 1070. The substrate 1070 can be, for example, part of the display 70 of FIG. 2 or a lens in a pair of eyeglasses. The substrate 1070 can be transparent to at least 10%, 20%, 30%, 40%, 50%, or more of the visible light incident on the substrate 1070. In other embodiments, the substrate 1070 need not be transparent (e.g., in a virtual reality display). The substrate 1070 can include one or more coupling optical elements 1078. In some embodiments, the coupling optical elements 1078 may be selected to reflect a first range of wavelengths while being substantially transparent to a second range of wavelengths different from the first range of wavelengths. In some embodiments, the first range of wavelengths can be IR wavelengths and the second range of wavelengths can be visible wavelengths. The substrate 1070 may comprise a polymer or plastic material, such as polycarbonate or other lightweight material having the desired optical properties. Without subscribe to any particular scientific theory, plastic materials may be less rigid and therefore less susceptible to breakage or defects during use. Plastic materials may also be lightweight, which, when combined with the rigidity of plastic materials, may allow for thinner substrates and facilitate the production of compact and lightweight imaging systems. While the substrate 1070 is described as comprising a polymer, such as polycarbonate or other plastic having the desired optical properties, other materials, such as glass, e.g., fused silica, having the desired optical properties are also possible.

[0089] The combining optical element 1078 may comprise a reflective optical element configured to reflect or redirect light in a first wavelength range (e.g., IR light) while transmitting light in a second wavelength range (e.g., visible light). In such embodiments, IR light 1010a, 1012a, and 1014a from the eye 220 propagates to and reflects from the combining optical element 1078, resulting in reflected IR light 1010b, 1012b, 1014b, which can be imaged by the camera assembly 1030. In some embodiments, the camera assembly 1030 may be sensitive to or capable of capturing at least a subset (such as a non-empty subset or a subset less than all) of the first wavelength range reflected by the combining optical element 1078. For example, if the combining optical element 1078 is a reflective element, the combining optical element 1078 may reflect IR light in the range of 700 nm to 1.5 μm, and the camera assembly 1030 may be sensitive to or capable of capturing near-IR light in wavelengths of 700 nm to 900 nm. As another example, the combining optical element 1078 may reflect IR light in the range of 700 nm to 1.5 μm, and the camera assembly 1030 may include a filter that filters out IR light in the range of 900 nm to 1.5 μm, such that the camera assembly 1030 can capture near-IR light in wavelengths of 700 nm to 900 nm.

[0090] Visible light from the outside world (e.g., world 510 in FIG. 6 ) can be transmitted through the substrate 1070 and perceived by the wearer. In effect, the imaging system 1000a can act as if there were a virtual camera assembly 1030c pointed back toward the wearer's eye 220 and capturing a direct-view image of the eye 220. The virtual camera assembly 1030c is labeled with a "c" because it can image virtual IR light 1010c, 1012c, and 1014c (shown as dotted lines) propagating from the wearer's eye 220 through the substrate 1070. The coupling optical element 1078 is illustrated as being disposed on the proximal surface 1074 of the substrate 1070, although other configurations are possible. For example, the coupling optical element 1078 could be disposed on the distal surface 1076 of the substrate 1060 or within the substrate 1070. In implementations in which the substrate 1070 is part of the display system 250 of FIG. 6, the coupling optical element 1078 may be an out-coupling optical element 570 , 580 , 590 , 600 , or 610 .

[0091] While an exemplary arrangement of the imaging system 1000a is shown in FIG. 10A, other arrangements are possible. For example, multiple coupling optical elements may be used and configured to in-couple light into the substrate 1070 and out-couple light to the camera assembly 1030 via TIR, e.g., as will be described in connection with FIGS. 11-18. While the coupling optical element 1078 is described as a reflective optical element, other configurations are possible. For example, the coupling optical element 1078 may be a transmissive coupling optical element that substantially transmits a first and second range of wavelengths. The transmissive coupling optical element may refract the first wavelength at an angle, e.g., inducing TIR within the substrate 1070, while allowing the second range of wavelengths to pass substantially unimpeded. Exemplary Imaging System for Off-Axis Imaging Using Multiple Combining Optical Elements

[0092] 11 schematically illustrates another exemplary imaging system 1000b comprising multiple combining optical elements for total internal reflection of light from an object through a substrate 1070 and imaging the object at a camera assembly 1030. FIG. 11 illustrates an embodiment of the imaging system 1000b comprising a substrate 1070 comprising at least two combining optical elements 1178a, 1188a disposed on one or more surfaces of the substrate 1070, and a camera assembly 1030 arranged to view an object positioned at an object plane 1120. While a specific arrangement is depicted in FIG. 11 , this is for illustrative purposes only and is not intended to be limiting. Other optical elements (e.g., lenses, waveguides, polarizers, prisms, etc.) may also be used to manipulate the light from the object, such as to focus, correct for aberrations, direct, etc., the light, as desired for a particular application.

[0093] In the embodiment of FIG. 11 , the substrate 1070 includes two coupling optical elements 1178 a, 1188 a, each disposed adjacent the distal and proximal surfaces 1076, 1074, respectively, of the substrate 1070. In some embodiments, the coupling optical elements 1178 a, 1188 a may be attached or fixed to the surface of the substrate 1070. In other embodiments, one or more of the coupling optical elements 1178 a, 1188 a may be built into or etched onto the surface of the substrate 1070. In still other embodiments, alone or in combination, the substrate 1070 may be fabricated to have regions that include the coupling optical elements 1178 a, 1188 a as part of the substrate 1070 itself. An example arrangement of the coupling optical elements 1178 a, 1188 a is shown in FIG. 11 , although other configurations are possible. For example, both coupling optical elements 1178a, 1188a may be positioned adjacent to the distal surface 1076 or the proximal surface 1074 (as shown in Figures 12A, 13A, 13B, and 14B), or coupling optical element 1178a may be positioned on the proximal surface 1074 while coupling optical element 1188a is positioned on the distal surface 1076 (as shown in Figure 14A).

[0094] The combining optical elements 1178a and 1188a may be similar to the combining optical element 1078 of Figures 10A and 10B. For example, Figure 11 illustrates an imaging system 1000b in which both combining optical elements 1178a, 1188a are reflective combining optical elements that are wavelength-selective so as to selectively redirect light of one or more wavelengths while transmitting light of other wavelengths, as described above in connection with Figure 10A. In some embodiments, the combining optical elements 1178a and 1188a deflect light of a first wavelength range (e.g., IR light, near-IR light, etc.) while transmitting a second wavelength range (e.g., visible light). As described below, the combining optical elements 1178a, 1188a may comprise diffractive features (e.g., DOEs) that form a diffraction pattern.

[0095] 11 , the camera assembly 1030 is mounted to face rearward toward the object plane 1120 and view the distal surface 1076. In various embodiments, the camera assembly 1030 may be mounted proximate to the wearer's eyes (e.g., on the frame 80 of FIG. 2 ) and may include a light source 1032 (not shown in FIG. 11 ). The camera assembly 1030 may include an IR digital camera sensitive to IR radiation. While the camera assembly 1030 of FIG. 11 is shown facing rearward, other arrangements are possible. For example, the camera assembly 1030 may be mounted to face forward.

[0096] In some embodiments, an object (e.g., eye 220 or a portion thereof) at object plane 1120 may be illuminated by light source 1032 (FIGS. 10A and 10B). For example, if the pupil is to be imaged, light source 1032 is directed thereto to illuminate the pupil of eye 220. In other embodiments, a first Purkinje image may be imaged, which is a virtual image formed by reflection of a point source from the anterior surface of the cornea. Any physical or optical object associated with the eye that can be uniquely identified and would indicate eye position, pupil position, or gaze direction may be imaged. Depending on the illumination, the object may reflect light toward substrate 1070 as light rays 1122a-e (collectively referred to hereafter as "1122"). For example, light rays 1122a-e may be illustrative of the diffusion of light reflected from the pupil, iris, eyelid, sclera, other tissues surrounding the eye, etc. In another example, light rays 1122a-e may be illustrative of specularly reflected light from a flash of light (e.g., a Purkinje image). Without subscribe to scientific theory, reflections from the eye, parts of the eye, or tissues surrounding the eye may rotate the polarization of the incident light depending on the orientation of the illumination. In some embodiments, light source 1032 (FIGS. 10A and 10B) may be an LED light source with no specific polarization unless a polarizer is implemented in the optical path, which may reduce the light intensity by, for example, 50%. While only light ray 1122 is shown in FIG. 11, this is for illustrative purposes only; any number of reflected light rays are possible. Each of light rays 1122 may be reflected from an object at the same or different angles. For example, FIG. 11 illustrates light ray 1122a being reflected at a first angle, which may be greater than the angle at which light ray 1122e is reflected from the object. Other configurations are possible.

[0097] While the above description refers to light rays 1122 reflected from the object, other configurations are possible. In some embodiments, light rays 1122 are emitted by a light source located at the object plane 1120, instead of reflecting light from source 1032 (FIGS. 10A and 10B). Thus, light rays 1122 may be directed toward the substrate 1070. It should be understood that light rays 1122 may be all or a portion of light reflected from or emitted by the object plane 1120.

[0098] 11 , upon emanating from the object plane 1120, light ray 1122 impinges on the substrate's proximal surface 1074 at an angle of incidence relative to an imaginary axis that is normal to the proximal surface 1074 at the point of incidence. Light ray 1122 then impinges on the substrate 1070 and is refracted, in part, based on the angle of incidence at the proximal surface 1074 and the ratio of the refractive indices of the media immediately adjacent the substrate 1070 and the proximal surface 1074.

[0099] Light ray 1122 travels and impinges on coupling optical element 1178a at an angle of incidence relative to an imaginary axis perpendicular to distal surface 1076 at the point of incidence. Light ray 1122 is deflected by coupling optical element 1178a so that it propagates through substrate 1070. That is, coupling optical element 1178a functions as a reflective internal coupling optical element, reflecting the light back into substrate 1070. Light ray 1122 is reflected at an angle such that internally coupled light ray 1122 propagates laterally through the substrate toward coupling optical element 1178a by total internal reflection. Without subscribe to any scientific theory, it is believed that the condition for total internal reflection occurs when the diffraction angle θ between the incident light and the perpendicular axis is greater than or equal to the critical angle θ of substrate 1070. C Under some circumstances, the total internal reflection condition can be expressed as: sin(θ C )=n o / n s [1] In the formula, n s is the refractive index of the substrate 1070, and n ois the refractive index of the medium adjacent to the surface substrate 1070. According to various embodiments, n s may be in the range of about 1 to about 2, about 1.4 to about 1.8, about 1.5 to about 1.7, or other suitable ranges. For example, the substrate 1070 may include a polymer such as polycarbonate or glass (e.g., fused silica, etc.). In some embodiments, the substrate 1070 may be 1 to 2 millimeters thick from the proximal surface 1074 to the distal surface 1076. For example, the substrate 1070 may be a 2-millimeter-thick portion of fused silica or a 1-millimeter-thick portion of polycarbonate. Other configurations for achieving the desired operation and image quality in the camera assembly 1030 are also possible as considerations.

[0100] In some embodiments, the substrate 1070 may be formed from a high refractive index material (e.g., a material having a higher refractive index than the medium directly adjacent to the substrate 1070). For example, the refractive index of the material directly adjacent to the substrate 1070 may be 0.05 or more, or 0.10 or more, lower than the substrate refractive index. Without subscribing to a particular scientific theory, a lower refractive index medium may function to promote TIR of light through the substrate 1070 (e.g., TIR between the proximal and distal surfaces 1074, 1076 of the substrate 1070). In some embodiments, the directly adjacent medium includes air with a refractive index n o of about 1. The critical angle can be in the range of 20 degrees to 50 degrees depending on the substrate material and the surrounding medium. In other embodiments, alone or in combination, the directly adjacent medium may include other structures and layers. For example, one or more of the layers described in connection with FIGS. 6 and 9A - 9C may be directly adjacent to either the proximal or distal surface 1074, 1076 of the substrate 1070.

[0101] The light then propagates through the substrate 1070 toward the coupling optical element 1188a in a direction generally parallel to the surface of the substrate 1070. Generally, "toward" can refer to a situation in which the light ray 1122 is reflected between the surfaces of the substrate 1070 and thus travels in a direction that may not be exactly parallel to the surface of the substrate 1070, but whose overall direction of travel is generally parallel to the surface of the substrate. The light ray 1122 propagates through the substrate 1070 by TIR until it impinges on the coupling optical element 1188a. In response to reaching the coupling optical element 1188a, the light ray 1122 is deflected to propagate out of the substrate 1070. That is, the coupling optical element 1188a functions as a reflective out-coupling optical element, reflecting the light out of the substrate 1070. The light ray 1120 propagates through the substrate 1070 when the TIR condition is no longer met (e.g., the diffraction angle θ is greater than the critical angle θ C The coupling optical element 1188a may also reflect the light ray 1122 at an angle toward the camera assembly 1030. For example, the light ray 1122 may be reflected at an angle to exit the substrate 1070, be refracted by an interface at the distal surface 1076, and propagate to the camera assembly 1030. The camera assembly 1030 then receives the light ray 1122 and images the object plane 1120 based thereon.

[0102] 11 illustrates a configuration in which light travels from combining optical element 1178a to combining optical element 1188a with two instances of total internal reflection, other configurations are possible. For example, light ray 1122 may be totally internally reflected any number of times (e.g., 1, 2, 3, 4, 5, 6, 7, etc.) such that light ray 1122 travels through substrate 1070 toward camera assembly 1030. Camera assembly 1030 may thus be positioned anywhere and configured to capture direct-view images at a distance from an object. While not subscribed to scientific theory, TIR can be highly efficient and involve substantially lossless reflection, and therefore the number of times light ray 1122 TIRs may be selected based on the desired position of the camera. However, in some embodiments, some leakage, although minimal, may occur with each reflection within substrate 1070. Thus, minimizing the number of reflections within the substrate 1070 may reduce light leakage and improve image capture performance. Additionally, without subscribe to scientific theory, reducing the number of reflections may improve image quality by reducing image blur or dimming caused by impurities or non-uniform surfaces of the substrate 1070 (e.g., fewer reflections may produce brighter, more intense images). Thus, the design of the described imaging system and its components may be optimized with these considerations in mind to minimize the number of TIR events and position the camera assembly 1030 as desired.

[0103] Efficient in- and out-coupling of light into the substrate 1070 can be a challenge when designing a waveguide-based see-through display, for example, for virtual / augmented / mixed reality display applications. For these and other applications, it may be desirable to include a diffraction grating whose structure is formed from a material that can be configured to optimize various optical properties, including diffractive properties. Desirable diffractive properties may include polarization selectivity, spectral selectivity, angular selectivity, high spectral bandwidth, and high diffraction efficiency, among other properties. To address these and other needs, in various embodiments disclosed herein, the coupling optical elements 1178a, 1188a may comprise diffractive features that form a diffraction pattern, such as a DOE or a diffraction grating.

[0104] Generally, a diffraction grating has a periodic structure that splits and diffracts light into several beams traveling in different directions. The direction of the beams depends, among other things, on the period of the periodic structure and the wavelength of the light. The period may be based, in part, on the grating spatial frequency of the diffractive features. For certain applications, such as in- and out-coupling light from the substrate 1070 to optimize diffraction efficiency and reduce potential rainbow effects, various material properties of the DOE can be optimized for a given wavelength. For example, if IR light is used, the spatial frequency of the DOEs 1178a, 1188a may be 600 to 2000 lines per millimeter. In one embodiment, the spatial frequency may be approximately 10 lines per millimeter (e.g., FIGS. 12A and 13A). In one embodiment, the exemplary DOE 1178a of FIG. 11 may have 10 lines per millimeter. In another embodiment, the spatial frequency is approximately 1400 lines per millimeter, as described in connection with FIG. 15. Thus, the spatial frequency of the combining optical elements 1178a, 1188a may be at least one consideration when optimizing the imaging systems described herein. For example, the spatial frequency may be selected to support TIR conditions. As another example, alone or in combination, the spatial frequency may be selected to maximize light throughput with minimal artifacts (e.g., residual or overlapping images as illustrated in FIG. 12B ), depending on the configuration and dimensions of the components of the imaging system. In some embodiments, the diffractive features may have any configuration. However, the first combining optical element 1178a may be optimized to have minimal or no visual artifacts (e.g., rainbow effects) because the first combining optical element 1178a may be positioned within a user's field of view.

[0105] In some implementations, the DOE may be an off-axis DOE, an off-axis holographic optical element (HOE), an off-axis holographic mirror (OAHM), or an off-axis volumetric diffractive optical element (OAVDOE). In some embodiments, the OAHM may also have optical power, in which case it may be an off-axis volumetric diffractive optical element (OAVDOE). In some embodiments, one or more of the coupling optical elements 1178a, 1188a may be an off-axis cholesteric liquid crystal grating (OACLCG), which may be configured to optimize polarization selectivity, bandwidth, phase profile, spatial variation of diffractive properties, spectral selectivity, and high diffraction efficiency, among others. For example, any of the CLCs or CLCGs described in U.S. Patent Application No. 15 / 835,108, filed December 7, 2017, and entitled "Diffractive Devices Based On Cholesteric Liquid Crystals" (which is incorporated herein by reference in its entirety for all that it discloses), may be implemented as a coupling optical element as described herein. In some embodiments, one or more of the coupling optical elements 1178a, 1188a may be switchable DOEs that can be switched between an actively diffracting, "on" state, and a non-significantly diffracting, "off" state.

[0106] In some embodiments, one or more of the coupling optical elements 1178a, 1188a may be any reflective or transmissive liquid crystal grating. The CLC or CLCG described above may be one example of a liquid crystal grating. Other liquid crystal gratings may also include liquid crystal features and / or patterns that have sizes less than the wavelength of visible light and may comprise what are referred to as Pancharatnam Berry Phase Effect (PBPE) structures, metasurfaces, or metamaterials. For example, any of the PBPE structures, metasurfaces, or metamaterials described in U.S. Patent Publication No. 2017 / 0010466, entitled "Display System With Optical Elements For In-Coupling Multiplexed Light Streams," U.S. Patent Application No. 15 / 879,005, filed January 24, 2018, entitled "Antireflection Coatings For Metasurfaces," or U.S. Patent Application No. 15 / 841,037, filed December 13, 2017, entitled "Patterning Of Liquid Crystals Using Soft-Imprint Replication Of Surface Alignment Patterns" (each of which is incorporated by reference in its entirety for all that it discloses) can be implemented as a coupling optical element, as described herein. Such structures may be configured to manipulate light, such as beam steering, wavefront shaping, wavelength separation, and / or polarization, and combining different wavelengths and / or polarizations, including liquid crystal gratings with metasurfaces, otherwise referred to as metamaterial liquid crystal gratings or liquid crystal gratings with PBPE structures. Liquid crystal gratings with PBPE structures can combine the high diffraction efficiency and low sensitivity to angle of incidence of liquid crystal gratings with the high wavelength sensitivity of PBPE structures.

[0107] In some embodiments, certain DOEs may offer non-limiting advantages when utilized as coupling optical elements, as described herein. For example, without subscribe to scientific theory, liquid crystal gratings, CLCs, CLCGs, volume phase gratings, and metasurface gratings may have optical properties configured to reduce or eliminate the appearance of visual artifacts, such as the rainbow effect described above and herein. In some embodiments, when employing these DOEs, it may be desirable to illuminate the DOE with polarized light (e.g., light beam 1122 may include a desired polarization) to maximize light throughput into substrate 1070. However, as described above, the eye may rotate the polarization of incident light depending on orientation, and thus, in some embodiments, light source 1030 may emit unpolarized light. The reflected light beam 1122 may also not be polarized, and thus, some light may not be able to be transmitted due to the polarizing properties of the DOE (e.g., up to 50% of light beam 1122 may be lost in coupling optical element 1178a). In some embodiments, to improve throughput, a dual-layer DOE may be used as the combining optical element 1178a, for example, where the first DOE layer is configured to operate in one polarization state and the second DOE layer is configured to operate in a second polarization state.

[0108] For some embodiments, it may be desirable to use a DOE with a sufficiently high diffraction efficiency so that many of the light rays 1122 are incoupled into the substrate 1070 and outcoupled toward the camera assembly. Without subscribe to scientific theory, a relatively high diffraction efficiency may allow substantially all of the light received at the coupling optical element 1178a to be directed toward the camera assembly 1030, thereby improving image quality and accuracy. In some embodiments, the diffraction efficiency may be based, in part, on the sensitivity of the camera assembly 1030 (e.g., higher sensitivity may allow for lower diffraction efficiency). In various embodiments, the DOE may be selected to have a high diffraction efficiency for a first wavelength range (e.g., IR light) and a low diffraction efficiency within a second wavelength range (e.g., visible light). Without subscribe to scientific theory, a low diffraction efficiency for visible light may reduce the rainbow effect in the user's viewing path.

[0109] In some applications, the DOE may produce a rainbow effect when a user views visible light through the diffractive features. Without subscribe to any particular scientific theory, the rainbow effect may be the result of certain wavelength ranges interacting with the diffractive features, thereby deflecting different wavelengths (e.g., colors) in different directions at different diffraction angles. In some embodiments described herein, the rainbow effect from the world interacting with the coupling optical elements 1178a, 1188b as viewed by a user may be reduced by modifying or controlling the diffractive features to reduce this effect. For example, because the diffraction angle of light on a DOE is based on the period or spatial frequency of the grating, the shape of the diffractive features may be selected (e.g., triangular cross-section or blazing) to concentrate a majority of the diffracted light in a specific location for a given range of wavelengths.

[0110] In some embodiments, the substrate 1070 may be one of the waveguides 270, 280, 290, 300, or 310 of FIG. 6. In this embodiment, the corresponding external coupling optical element 570, 580, 590, 600, or 610 may be replaced with an internal coupling optical element 1178a configured to induce TIR of light reflected by the eye. In some embodiments, a portion of the external coupling optical element 570, 580, 590, 600, or 610 may be replaced with an internal coupling optical element 1178a such that the corresponding waveguide 270, 280, 290, 300, or 310 may be used to direct reflected light to the camera assembly 630, as described in connection with FIG. 6.

[0111] In some embodiments, the substrate 1070 may be one of the waveguides 670, 680, or 690 in FIGS. 9A-9C. In these embodiments, the corresponding optically dispersive elements 800, 810, and 830, or portions thereof, may be replaced with the internal coupling optical element 1178a, while the internal coupling optical elements 700, 710, and 720, or portions thereof, may be replaced with the external coupling optical element 1188a. In some embodiments, the OPEs 730, 740, and 750 may remain in the optical path of light traveling from the internal coupling optical element 1178a to the external coupling optical element 1188a. However, the OPEs 730, 740, and 750 may be configured to disperse the light into the external coupling optical element 1188a and also reduce the beam spot size as it propagates.

[0112] In various embodiments, the field of view of the camera assembly 1030 is configured to be sufficient to image the entire object plane 1120 (e.g., the eye 220 in FIG. 10 , a portion thereof, or the tissue surrounding the eye) throughout various viewing positions. For example, in the example shown in FIG. 11 , the size of the imaged object plane 1120 may be 30 mm (horizontal) by 16 mm (vertical). In some embodiments, the combining optical elements 1178 a, 1188 a are designed to be large enough to match at least the size of the object to be imaged, i.e., the combining optical elements 1178 a, 1188 a are configured to receive light from the entire size of the object to be imaged. For example, the combining optical element 1178 a receives light originating from the eye 220. The combining optical element 1188 may be sized to reflect substantially all of the light rays 1122 propagating through the substrate 1070 toward the camera assembly 1030.

[0113] In various embodiments, other optical elements may be positioned along the path traveled by light rays 1122. For example, intervening optical elements may be included between substrate 1070 and object plane 1120 to direct light rays 1122 toward substrate 1070 at a desired angle. Intervening optical elements may be included between camera assembly 1030 and substrate 1070 to direct and focus light rays 1122 toward camera assembly 1030 so as to place camera assembly 1030 in any desired location. In some embodiments, intervening optical elements may be used to filter light rays 1122, change polarization, or correct aberrations. For example, corrective optical elements may be positioned along the optical path of light rays 1122, or if the imaging system is part of display system 250 of FIG. 6, other waveguides, or optical elements, arranged and configured to reduce or eliminate optical aberrations introduced by optical components of the imaging system. Alternative embodiment for off-axis imaging using multiple combining optical elements

[0114] FIG. 11 shows an example imaging system 1000b comprising a substrate 1070 having coupling optical elements 1178a, 1188a configured to TIR light from the object plane 1120 through the substrate 1070, although other configurations are possible. For example, FIG. 11 illustrates both coupling optical elements 1178a, 1188a as reflective coupling optical elements. However, one or both coupling optical elements may be transmissive coupling optical elements configured to refract a first range of wavelengths at an angle that satisfies the TIR condition while transmitting a second range of wavelengths substantially through the substrate 1070. FIGS. 12A-18 illustrate several embodiments of the substrate 1070, however, other configurations are possible.

[0115] Figure 12A schematically illustrates an example imaging system 1000c. The imaging system 1000c uses multiple coupling optical elements 1178a and 1188b to TIR light 1122 from an object plane 1220 through a substrate 1070. Similar to Figure 11, Figure 12A illustrates the coupling optical element 1178a as a reflective coupling optical element disposed on the distal surface 1076 of the substrate 1070 that in-couples the light rays 1122 into the substrate 1070. However, although the coupling optical element 1188b is substantially similar to the coupling optical element 1188a of Figure 11, Figure 12A illustrates a transmissive coupling optical element 1188b disposed on the distal surface 1076 of the substrate 1070. Thus, in response to propagating through the substrate 1070 via TIR, the light ray 1122 is reflected a third time on the proximal surface 1074 toward the transparent coupling optical element 1188b. The transparent coupling optical element 1188b refracts the light ray 1122 at an angle such that the TIR condition no longer holds and the light ray 1122 exits the substrate 1070. For example, if the transparent coupling optical element 1188b is a DOE, the light is refracted based on the spatial frequency of the DOE and is substantially deflected toward the camera assembly 1030.

[0116] FIG. 12A also illustrates stray light rays 1222 captured by camera assembly 1030. For example, stray light rays 1222 are reflected by object 1120, but instead of propagating through coupling optical elements 1178a, 1188b, some or all of stray light rays 1222 travel directly toward camera assembly 1030. Without subscribe to a particular theory, stray light rays 1222 are captured by camera assembly 1030, thereby generating a direct-view image, as described above. Thus, camera assembly 1030 may capture a direct-view image based on light rays 1222 (e.g., including the narrow FOV and defects described herein) along with a desired image based on light rays 1122 TIR through the substrate. Because camera assembly 1030 captures light rays that have traveled along different optical paths, the final image will include various imperfections. One such imperfection is illustrated in FIG. 12B, although others are possible.

[0117] FIG. 12B illustrates an example image 1210 of object 1120 captured using camera assembly 1030 of FIG. 12A. In example image 1210, camera assembly 1030 captured image 1210 of a front view of a laser diode used as the object and illuminated with an IR light source. While a laser diode is illustrated in this example, other objects, such as a user's eye 210, may be used for a similar effect. Image 1210 includes a direct-view image 1205 of the laser diode produced by light beam 1222 and a set of images 1240 produced by light beam 1122. Set of images 1240 includes a desired off-axis image (shown as image 1215 for illustrative purposes) and multiple overlapping images from different viewpoints (collectively shown as images 1212). Such overlapping images 1212 may, in some embodiments, require post-processing to synthesize a single-view image of the object, if desired. In other embodiments, the imaging system may be designed to capture a single viewpoint image 1215, reducing or eliminating unwanted overlapping images 1212 and direct view images 1205.

[0118] 13A and 13B schematically illustrate another view of imaging system 1000c. Figures 13A and 13B illustrate an example approach for reducing or eliminating overlapping images 1212. Without subscribe to a particular scientific theory, overlapping images 1212 may be reduced or substantially eliminated based on varying the thickness (t) of substrate 1070, the size (d1) of coupling optical element 1178a, and the stride distance (d2) of light ray 1122. The stride distance (d2) may refer to the distance parallel to substrate 1070 that a light ray travels as it reflects within the substrate, i.e., the distance between two adjacent points incident on distal surface 1076 of substrate 1070 due to, for example, a single instance of total internal reflection. In some embodiments, the direct view image 1205 may also be reduced or eliminated, for example, by including a coating (e.g., an IR coating configured to block or reduce IR light from the object 1220) on the proximal or distal surfaces 1074, 1076 proximate to the object 1220.

[0119] For example, residual images can be reduced or eliminated by reducing the size (d1) of the coupling optical element 1178a to a minimum size and varying the physical arrangement of the components of the imaging system 1000c so that the stride distance (d2) is greater than d1.

[0120] In some embodiments, it may be desirable to control the stride distance (d2) and minimize the size of the coupling optical element 1178a while achieving a large stride distance. Without subscribing to a particular scientific theory, a large stride distance may reduce the intensity of the residual image or allow placement of the camera assembly 1030 outside of the stray light rays 1030. Thus, under some circumstances, the stride distance may be expressed as: d2=2*t*tan(θ) [2] where θ is the diffraction angle of ray 1122 and t is the thickness of substrate 1070. Increasing the stride distance may be done by increasing the thickness (t) of the substrate or by increasing the diffraction angle (θ). As explained above, the diffraction angle (θ) may be based on the spatial frequency or period of the diffractive features. For example, lowest ray 1122e has the smallest diffraction angle (θ), and therefore, it may be preferable to increase this diffraction angle to increase the stride distance. Furthermore, increasing the thickness of substrate 1070 may also increase the stride distance. However, it may be desirable to balance the thickness of substrate 1070 with creating a lightweight and compact imaging system. In one embodiment, substrate 1070 is a 2.5 millimeter thick piece of polycarbonate (other materials are possible) and the grating spatial frequency is 720 lines per millimeter. Various embodiments may include different substrate thicknesses or grating spatial frequencies.

[0121] Figures 14A and 14B schematically illustrate examples of imaging systems with multiple coupling optical elements having different arrangements than imaging system 1000a of Figure 11. As illustrated in Figure 11, the coupling optical elements are configured as either internal or external coupling optical elements to induce TIR and direct light rays 1122 through substrate 1070 to camera assembly 1030. Figures 14A and 14B differ in the type and placement of the coupling optical elements.

[0122] For example, Figure 14A depicts imaging system 1000d that is substantially similar to imaging system 1000b of Figure 11. However, imaging system 1000d includes a transmissive coupling optical element 1178b disposed on the proximal surface 1074 of substrate 1070 and a transmissive coupling optical element 1188b disposed on the distal surface 1076 of substrate 1070. Transmissive coupling optical element 1178b may be configured as an internal coupling optical element that is transmissive but diffracts light 1122 of Figure 11 at a diffraction angle to induce TIR at distal surface 1046. Light 1122 may then be directed toward transmissive coupling optical element 1188b, configured as an external coupling optical element, as described above in connection with Figure 12A.

[0123] In the embodiment of Figure 14B, imaging system 1000e is substantially similar to imaging system 1000b of Figure 11. However, imaging system 1000e includes a transmissive coupling optical element 1178b and a reflective coupling optical element 1188a disposed on the proximal surface 1074 of substrate 1070. Transmissive coupling optical element 1178b may be configured as an internal coupling optical element that is transmissive but diffracts light 1122 of Figure 11 at a diffraction angle to induce TIR at distal surface 1046. Light 1122 may then be directed toward reflective coupling optical element 1188a, configured as an external coupling optical element, as described above in connection with Figure 11.

[0124] FIG. 15 schematically illustrates another exemplary imaging system 1000f substantially similar to the imaging system 1000c of FIGS. 12A-13B. Like the imaging systems described above, FIG. 15 illustrates imaging system 1000f comprising a reflective coupling optical element 1178a and a transmissive coupling optical element 1188b disposed on the distal surface 1076 of a substrate 1070. However, the coupling optical elements 1178a and 1188b have a spatial frequency of 1411.765 lines per millimeter and a pitch of 708.33 nanometers, and the substrate is a 1-millimeter-thick piece of polycarbonate. Thus, compared to the imaging system 1000c of FIGS. 12A-13B, the light 1122 may TIR multiple times within the substrate 1070, and the camera assembly may be shifted further away from the coupling optical element 1178a. Other configurations are also possible. Alternative Embodiments of the Imaging System for Off-Axis Imaging

[0125] FIG. 11 shows an example imaging system 1000b comprising a substrate 1070 having coupling optical elements 1178a, 1188a configured to TIR light from an object plane 1120 through the substrate 1070, although other configurations are possible.

[0126] For example, FIG. 16 illustrates imaging system 1000g including substrate 1070 positioned adjacent to optical component 1650. In some embodiments, optical component 1650 may be waveguide stack 260 of FIG. 6 or waveguide stack 660 of FIGS. 9A-9C. While substrate 1070 is illustrated as adjacent to and between object 1120 and optical component 1650, other configurations are possible. For example, optical component 1650 may be between substrate 1070 and object 1120, or substrate 1070 may be part of optical component 1650. Substrate 1070 may include multiple reflective elements 1678 and 1688. As illustrated in FIG. 16, light 1122 may travel from object 1120 toward substrate 1070 and interact with proximal surface 1074. The light 1122 may be refracted and directed to the reflective element 1678, which reflects the light 1122 at an angle such that the light TIRs on the proximal surface 1074. Thus, the light 1122 travels toward the reflective element 1688 via TIR. The light 1122 may be reflected by the reflective element 1688 toward the camera assembly 1030. Thus, the camera assembly 1030 may capture an off-axis image of the object 1120 as if the camera assembly 1030 were viewing the object 1120 directly (e.g., virtual camera assembly 1030c). In some embodiments, one or more of the reflective elements 1678, 1688 may be a "hot mirror," or comprise a reflective coating that is reflective in the IR but transparent in the visible spectrum.

[0127] In one embodiment of FIG. 16 , the substrate 1070 is a 2-millimeter thick piece of polycarbonate, with the proximal surface 1074 positioned 15.7 millimeters to the right (e.g., in the z-direction) of the object plane 1120. The object plane 1120 is 12 millimeters vertical (e.g., in the y-direction). In some embodiments, the reflective element 1678 is configured to capture substantially the entire FOV, with the central ray 1122c propagating at 25 degrees downward from the normal (e.g., in the negative y-direction). The camera assembly 1030 may be positioned 15.7 millimeters below and 18.79 millimeters to the right of the origin of the ray 1122c. In this arrangement, the imaging system 1000g captures images as if the view from the virtual camera 1030c were positioned 10.56 millimeters below and 22.65 millimeters to the right.

[0128] FIG. 17 illustrates an imaging system 1000h comprising a substrate 1770 disposed adjacent to an optical component 1650 (e.g., an optical cover glass or prescription glasses) and a reflective surface 1778 disposed adjacent to the substrate 1770. In some embodiments, the substrate 1770 may be substantially similar to the substrate 1070 described above. While a specific arrangement is shown in FIG. 17 , other configurations are possible. For example, the optical component 1650 may be between the substrate 1650 and the object 1120, or the substrate 1770 may be part of the optical component 1650. As illustrated in FIG. 17 , light 1122 may travel from the object 1120 toward and interact with the optical component 1650. The light 1122 may then be refracted or passed through the optical component 1650 as it travels toward the substrate 1770. After passing (refracting or passing) through substrate 1770, light 1122 is incident on reflective surface 1778. Reflective surface 1778 may have optical properties configured to reflect and direct light 1122 toward camera assembly 1030. Thus, camera assembly 1030 may capture an off-axis image of object 1120 as if camera assembly 1030 were viewing object 1120 directly. In one embodiment of Figure 17, imaging system 1000f is configured to capture a 16 millimeter by 24 millimeter object plane 1120, and central ray 1122c propagates at positive 17 degrees from the normal (shown as line 1790).

[0129] In some embodiments, the reflective surface 1778 may be the surface of a decorative or aesthetic lens or optical component. For example, the decorative lens may be a lens for use as sunglasses that filters out sunlight. In another embodiment, the decorative lens may be a color-filtering lens for use in goggles. In still other embodiments, the decorative lens may have a colored visual appearance visible to others not wearing the lens (e.g., the lens appears blue, red, etc. to others). The decorative lens may also include a color layer visible to others other than the user. The reflective surface 1778 may be a reflective coating on the inner surface of the decorative lens. The reflective coating may be reflective in the IR but transparent in the visible spectrum to allow the wearer to view the world. As shown in FIG. 17, the reflective surface 1778 may comprise a concave shape configured to direct light 1122 toward the camera assembly 1030. However, other configurations are possible.

[0130] FIG. 18 illustrates imaging system 1000i comprising a substrate 1770 disposed adjacent to optical component 1850 and a prism 1878 disposed adjacent to substrate 1770. In some embodiments, substrate 1770 may be substantially similar to substrate 1070 described above. Optical component 1850 may be substantially similar to optical component 1650, but may comprise one or more of exit pupil expanders 800, 810, 820 of FIGS. 9A-9C . While a specific arrangement is shown in FIG. 18 , other configurations are possible. For example, optical component 1850 may be between substrate 1770 and object 1120, or substrate 1770 may be part of optical component 1850. As illustrated in FIG. 18 , light 1122 may travel from object 1120 toward and interact with optical component 1850. Light 1122 may be refracted or passed as it travels toward optical component 1850. After passing (refracting or passing) through optical component 1850, light 1122 enters prism 1878 and is reflected by surface 1878a toward camera assembly 1030. Thus, camera assembly 1030 can capture an off-axis image of object 1120 as if camera assembly 1030 were viewing object 1120 directly. In some embodiments, the prism may be an IR prism "hot mirror," or surface 1878a may include a reflective coating that is reflective in the IR but transparent in the visible spectrum. In one embodiment of FIG. 18, imaging system 1000i includes camera assembly 1030 with a vertical FOV of 35 degrees and a focal length of 30.73 millimeters. Such an imaging system 1000i may be configured to capture a 16 mm by 24 mm object plane 1120, with the central ray 1122c propagating at negative 25 degrees from the normal (shown as line 1790). Example Routine for Imaging an Object

[0131] 19 is a process flow diagram of an illustrative routine for imaging an object (e.g., a user's eye) using an off-axis camera (e.g., camera assembly 630 of FIG. 6 or camera assembly 1030 of FIG. 10A). Routine 1900 describes how light from an object can be directed to a camera assembly positioned away from or off-axis with respect to the object to image the object as if the camera assembly were pointed directly toward the object.

[0132] In block 1910, an imaging system is provided configured to receive light from the object and direct the light to the camera assembly. The imaging system may be one or more of the imaging systems 1000a-i as described above in connection with FIGS. 10A-11, 12A, and 13A-18. For example, the imaging system may include a substrate (e.g., substrate 1070) including a first coupling optical element (e.g., first coupling optical element 1078, 1178a, or 1178b) and a second coupling optical element (e.g., second optical element 1188a or 1188b). The first and second optical elements may be disposed on a distal or proximal surface of the substrate, as described above and throughout this disclosure. The first and second optical elements may be laterally offset from each other along the substrate 1070. As described above and throughout this disclosure, the first coupling optical element may be configured to deflect light at an angle and TIR the light between the proximal and distal surfaces. The first optical element may be configured to deflect light at an angle generally toward the second coupling optical element. The second coupling optical element may be configured to receive light from the first coupling optical element and deflect the light at an angle away from the substrate.

[0133] In block 1920, light is captured with a camera assembly (e.g., camera assembly 630 of FIG. 6 or camera assembly 1030 of FIGS. 10A-11, 12A, and 13A-18). The camera assembly may be oriented toward the second combining optical element and receive light deflected by the second combining optical element. The camera assembly may be an off-axis camera in a forward-facing or rear-facing configuration. In block 1930, an off-axis image of the object may be generated based on the captured light, as described herein and throughout this disclosure.

[0134] In some embodiments, the routine 1900 may include an optional step (not shown) of illuminating the object with light from a light source (e.g., light source 632 of FIG. 6 or light source 1032 of FIGS. 10A-11, 12A, and 13A-18). In some embodiments, the light may include a range of wavelengths, including IR light.

[0135] In some embodiments, the off-axis image generated in block 1930 may be processed and analyzed, for example, using image processing techniques. The analyzed off-axis image may be used to perform one or more of eye tracking, biometric identification, multi-view reconstruction of eye shape, estimation of the ocular accommodation state or imaging of the retina, iris, or other distinctive patterns of the eye, and assessment of the user's physiological state based in part on the analyzed off-axis image, as described above and throughout this disclosure.

[0136] In various embodiments, the routine 1900 may be implemented by a hardware processor (e.g., local processing and data module 140 of FIG. 2) configured to execute instructions stored in memory. In other embodiments, a remote computing device (in network communication with the display device) with computer-executable instructions can cause the display device to implement aspects of the routine 1900. Additional Aspects 1. An optical device comprising: a substrate having a proximal surface and a distal surface; a first coupling optical element disposed on one of the proximal and distal surfaces; and a second coupling optical element disposed on one of the proximal and distal surfaces and laterally offset from the first coupling optical element along a direction parallel to the proximal or distal surfaces, wherein the first coupling optical element is configured to deflect light at an angle and cause total internal reflection (TIR) ​​of the light between the proximal and distal surfaces towards the second coupling optical element, and the second coupling optical element is configured to deflect light at an angle out of the substrate. 2. The optical device of aspect 1, wherein the substrate is transparent to visible light. 3. The optical device of aspect 1 or 2, wherein the substrate comprises a polymer. 4. The optical device according to any one of aspects 1-3, wherein the substrate comprises polycarbonate. 5. The optical device of any one of aspects 1-4, wherein the first and second coupling optical elements are external to and fixed to at least one of the proximal and distal surfaces of the substrate. 6. The optical device of any one of aspects 1-5, wherein the first and second coupling optical elements comprise a portion of the substrate. 7. The optical device of any one of aspects 1-6, wherein at least one of the first and second coupling optical elements comprises a plurality of diffractive features. 8. The optical device of aspect 7, wherein the plurality of diffractive features have a relatively high diffraction efficiency for a range of wavelengths so as to diffract substantially all of the light in the range of wavelengths. 9. The optical device of aspect 7 or 8, wherein the plurality of diffractive features diffract light in at least one direction based in part on the period of the plurality of diffractive elements, and the at least one direction is selected to TIR the light between the proximal surface and the distal surface. 10. The optical device of any one of aspects 1-7, wherein at least one of the first or second coupling optical elements comprises at least one of an off-axis diffractive optical element (DOE), an off-axis grating, an off-axis diffractive optical element (DOE), an off-axis holographic mirror (OAHM), or an off-axis volumetric diffractive optical element (OAVDOE), or an off-axis cholesteric liquid crystal grating (OACLCG). 11. The optical device of any one of aspects 1-7 and 10, wherein the first and second coupling optical elements are each configured to deflect light in a first wavelength range while transmitting light in a second wavelength range. 12. The optical device of aspect 11, wherein the first wavelength range includes light in at least one of the infrared (IR) or near-IR spectrum, and the second wavelength range includes light in the visible spectrum. 13. The optical device of any one of aspects 1, 7, and 11, wherein the first and second coupling optical elements selectively reflect light within a range of wavelengths, and the first coupling optical element is disposed on a distal surface of the substrate and the second coupling optical element is disposed on a proximal surface of the substrate. 14. The optical device of any one of aspects 1, 7, 10, and 11, wherein the first and second coupling optical elements selectively transmit light within a range of wavelengths, the first coupling optical element being disposed on a proximal surface of the substrate, and the second coupling optical element being disposed on a distal surface of the substrate. 15. The optical device of any one of aspects 1, 7, 10, and 11, wherein the first coupling optical element selectively reflects light in a range of wavelengths and the second coupling optical element selectively transmits light in the range of wavelengths, and the first and second coupling optical elements are disposed on a distal surface of the substrate. 16. The optical device of any one of aspects 1, 7, 10, and 11, wherein the first coupling optical element selectively transmits light in a range of wavelengths and the second coupling optical element selectively reflects light in the range of wavelengths, and the first and second coupling optical elements are disposed on a proximal surface of the substrate. 17. A head-mounted display (HMD) configured to be worn on a user's head, the HMD comprising: a frame; a pair of optical elements supported by the frame such that each optical element of the pair can be positioned in front of the user's eyes; and an imaging system comprising a camera assembly mounted to the frame and the optical device described in any one of aspects 1-16. 18. The HMD of aspect 17, wherein at least one optical element of the pair of optical elements includes a substrate. 19. The HMD of aspect 17 or 18, wherein the substrate is disposed on a surface of at least one optical element of the pair of optical elements. 20. The HMD of any one of aspects 17-19, wherein the frame includes a pair of ear hooks and the camera assembly is mounted on one of the pair of ear hooks. 21. The HMD of any one of aspects 17-20, wherein the camera assembly is a forward-facing camera assembly configured to image light received from the second coupling optical element. 22. An HMD as described in any one of aspects 17-20, wherein the camera assembly is a rear-facing camera assembly positioned in a direction facing toward the user, the rear-facing camera assembly being configured to image light received from the second coupling optical element. 23. The HMD of any one of aspects 17-22, further comprising a light source that emits light in a first wavelength range toward at least one of the user's eye, a portion of the eye, or a portion of tissue surrounding the eye. 24. The HMD of aspect 23, wherein light in the first wavelength range is reflected by at least one of the user's eye, a portion of the eye, or a portion of tissue surrounding the eye toward the first coupling optical element. 25. The HMD of any one of aspects 17-23, wherein each of the pair of optical elements is transparent to visible light. 26. The HMD of any one of aspects 17-23 and 25, wherein each of the pair of optical elements is configured to display an image to the user. 27. The HMD of any one of aspects 17-23, 25, and 26, wherein the camera assembly is configured to image at least one of the user's eye, a portion of the eye, or a portion of tissue surrounding the eye based in part on light received from the second coupling optical element. 28. The HMD of aspect 27, wherein the HMD is configured to track the user's gaze based on an image of at least one of the user's eye, a portion of the eye, or a portion of the tissue surrounding the eye. 29. The HMD of aspect 27, wherein the image formed by the camera assembly coincides with the image formed by a camera positioned in front of the user's eye and directly viewing at least one of the user's eye, a portion of the eye, or a portion of the tissue surrounding the eye. 30. The HMD of any one of aspects 17-23, 25, and 27, wherein the optical device is arranged to reduce stray light received by the camera assembly. 31. The HMD of any one of aspects 17-23, 25, 27, and 30, wherein the size of the first coupling optical element is less than a stride distance of light reflected between the distal and proximal surfaces of the substrate, the stride distance being based on the thickness of the substrate and the angle at which the first coupling optical element deflects the light. 32. The HMD of aspect 31, wherein the size of the first coupling optical element is based on the field of view of the user's eye. 33 An HMD described in any one of aspects 17-23, 25, 27, 30, and 31, wherein an image of the user's eye formed by the camera assembly and an image of the user's eye formed by a camera mounted in front of the user's eye are indistinguishable. 34. The HMD of any one of aspects 17-23, 25, 27, 30, 31, and 33, further comprising: a non-transitory data storage device configured to store images acquired by the camera assembly; and a hardware processor in communication with the non-transitory data storage device, the hardware processor being programmed with executable instructions to analyze the images and perform one or more of eye tracking, biometric identification, multi-view reconstruction of eye shape, estimation of the eye's accommodation state or imaging of the retina, iris, or other distinctive patterns of the eye, and assessment of the user's physiological state. 35. An imaging system comprising: a substrate having a proximal surface and a distal surface; a first diffractive optical element disposed on one of the proximal and distal surfaces; and a second diffractive optical element disposed on one of the proximal and distal surfaces, the second diffractive optical element being offset from the first diffractive optical element along a direction parallel to the proximal or distal surface, the first diffractive optical element configured to deflect light at an angle and total internally reflect (TIR) ​​the light between the proximal and distal surfaces towards a second coupling optical element, the second diffractive optical element configured to deflect light incident thereon at an angle out of the substrate; and a camera assembly for imaging light deflected by the second diffractive optical element. 36. The imaging system of aspect 35, wherein the first and second diffractive optical elements comprise at least one of an off-axis diffractive optical element (DOE), an off-axis grating, an off-axis diffractive optical element (DOE), an off-axis holographic mirror (OAHM), or an off-axis volumetric diffractive optical element (OAVDOE), an off-axis cholesteric liquid crystal grating (OACLCG), a hot mirror, a prism, or a surface of a decorative lens. 37. A method of imaging an object using a virtual camera, the method comprising: providing an imaging system in front of an object to be imaged, the imaging system comprising a substrate comprising a first coupling optical element and a second coupling optical element, each positioned on one of a proximal surface and a distal surface of the substrate and offset from one another, the first coupling optical element configured to deflect light at an angle and cause total internal reflection (TIR) ​​the light between the proximal and distal surfaces towards the second coupling optical element, and the second coupling optical element configured to deflect the light at an angle out of the substrate; capturing light with a camera assembly oriented to receive light deflected by the second coupling optical element; and generating an off-axis image of the object based on the captured light. 38. The method of aspect 37, wherein the first and second coupling optical elements each deflect light in a first wavelength range while transmitting light in a second wavelength range. 39. The method of aspect 37 or 38, further comprising illuminating the object with the first range of wavelengths emitted by the light source. 40. The method of any one of aspects 37-39, further comprising analyzing the off-axis image and performing one or more of eye tracking, biometric identification, multi-view reconstruction of eye shape, estimation of the eye's accommodation state, or imaging of the retina, iris, or other distinctive patterns of the eye, and assessment of the user's physiological state based in part on the analyzed off-axis image. 41. An imaging system comprising: a substrate having a proximal surface and a distal surface; a reflective optical element adjacent the distal surface, the reflective optical element configured to reflect light passing out of the substrate at an angle at the distal surface; and a camera assembly for imaging light reflected by the reflective optical element. 42. The imaging system of aspect 41, wherein the reflective optical element comprises a decorative lens surface. 43. The imaging system of aspect 41 or aspect 42, wherein the reflective optical element comprises a reflective coating on a surface of a decorative lens. 44. The imaging system of any one of aspects 41-43, wherein the reflective optical element comprises a reflective prism. 45. The imaging system of any one of aspects 41-44, wherein the reflective optical element is reflective to infrared light and transmissive to visible light. 46. ​​The imaging system of any one of aspects 41-45, further comprising a diffractive optical element adjacent the proximal surface. 47. The imaging system of any one of aspects 41-46, wherein the camera assembly is a forward-facing camera assembly configured to image light received from the reflective optical element. 48. A head-mounted display (HMD) configured to be worn on a user's head, the HMD comprising: a frame; a pair of optical elements supported by the frame such that each optical element of the pair can be positioned in front of an eye of the user; and an imaging system according to any one of claims 41-47. 49. The HMD of aspect 48, wherein at least one optical element of the pair of optical elements includes a substrate. 50. The HMD of aspect 48 or 49, wherein the substrate is disposed on a surface of at least one optical element of the pair of optical elements. 51. An HMD as described in any one of aspects 48-50, wherein the frame includes a pair of ear hooks and the camera assembly is mounted on one of the pair of ear hooks. 52. The HMD of any one of aspects 48-51, further comprising a light source that emits light in a first wavelength range toward at least one of the user's eye, a portion of the eye, or a portion of tissue surrounding the eye. 53. An HMD according to any one of aspects 48-52, wherein each of the pair of optical elements is transparent to visible light. 54. The HMD of any one of aspects 48-53, wherein each of the pair of optical elements is configured to display an image to the user. 55. The HMD of any one of aspects 48-54, wherein the camera assembly is configured to image at least one of the user's eye, a portion of the eye, or a portion of tissue surrounding the eye based in part on light received from the second coupling optical element. 56. An HMD according to any one of aspects 48-55, wherein the HMD is configured to track the user's gaze based on an image of at least one of the user's eye, a portion of the eye, or a portion of the tissue surrounding the eye. 57. An HMD as described in any one of aspects 48-56, wherein the image formed by the camera assembly coincides with the image formed by a camera positioned in front of the user's eye and directly viewing at least one of the user's eye, a portion of the eye, or a portion of the tissue surrounding the eye. 58. The HMD of any one of aspects 48-57, wherein the optical device is arranged to reduce stray light received by the camera assembly. 59. An HMD as described in any one of aspects 48-58, wherein an image of the user's eye formed by the camera assembly is indistinguishable from an image of the user's eye formed by a camera positioned in front of the user's eye. 60. The HMD of any one of aspects 48-59, further comprising: a non-transitory data storage device configured to store images acquired by the camera assembly; and a hardware processor in communication with the non-transitory data storage device, the hardware processor being programmed with executable instructions to analyze the images and perform one or more of eye tracking, biometric identification, multi-view reconstruction of eye shape, estimation of the eye's accommodation state or imaging of the retina, iris, or other distinctive patterns of the eye, and assessment of the user's physiological state. (Additional Considerations)

[0137] In the embodiments described above, the optical arrangement is described in relation to an eye-imaging display system, more specifically, an augmented reality display system. However, it will be understood that the principles and advantages of the optical arrangement can be used for head-mounted displays, optical systems, devices, or methods. In the foregoing, it should be understood that any feature of any one of the embodiments can be combined with and / or substituted for any other feature of any other one of the embodiments.

[0138] Unless the context clearly requires otherwise, throughout the description and claims, the words “comprise,” “comprising,” “include,” “including,” “have,” “having,” and the like, are to be construed in an inclusive sense, i.e., “including, but not limited to,” as opposed to an exclusive or inclusive sense. The word “coupled,” as generally used herein, refers to two or more elements that may be either directly connected or connected via one or more intermediate elements. Similarly, the word “connected,” as generally used herein, refers to two or more elements that may be either directly connected or connected via one or more intermediate elements. Depending on the context, “coupled” or “connected” can refer to an optical coupling or optical connection, such that light is coupled or connected from one optical element to another. Additionally, the words "herein," "above," "below," "infra," "supra," and words of similar import, when used in this application, shall refer to this application as a whole and not to any particular portions of this application. Where permitted by context, words in the above Detailed Description using the singular or plural form may also include the plural or singular form, respectively. The word "or," referring to a list of two or more items, is an inclusive (not exclusive) "or," and "or" covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of one or more items in the list, and does not exclude other items added to the list. Additionally, the articles "a," "an," and "the," as used in this application and the appended claims, shall be construed to mean "one or more" or "at least one," unless otherwise specified.

[0139] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single elements. As an example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Transitional phrases such as "at least one of X, Y, and Z" are generally understood differently in the context in which they are used to convey that an item, term, etc. may be at least one of X, Y, or Z, unless specifically stated otherwise. Thus, such transitional phrases generally are not intended to suggest that an embodiment requires that at least one of X, at least one of Y, and at least one of Z, respectively, be present.

[0140] Furthermore, it should be understood that conditional statements used herein, such as, among others, "can," "could," "might," "may," "e.g.," "for example," "such as," and the like, are generally intended to convey that certain embodiments include certain features, elements, and / or conditions, while other embodiments do not, unless specifically stated otherwise or understood otherwise within the context as used. Thus, such conditional statements generally are not intended to suggest that features, elements, and / or conditions are in any way required for one or more embodiments, or whether these features, elements, and / or conditions should be included or implemented in any particular embodiment.

[0141] While certain embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of the present disclosure. Indeed, the novel apparatus, methods, and systems described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions, and changes in the form of the methods and systems described herein may be made without departing from the spirit of the present disclosure. For example, while blocks are presented in a given arrangement, alternative embodiments may implement similar functionality using different components and / or circuit topologies, and some blocks may be deleted, moved, added, subdivided, combined, and / or modified. Each of these blocks may be implemented in a variety of different ways. Any suitable combination of elements and acts of the various embodiments described above can be combined to provide further embodiments. The various features and processes described above may be implemented independently of one another or may be combined in various ways. No element or combination of elements is necessary or essential for all embodiments. All suitable combinations and subcombinations of features of the present disclosure are intended to be within the scope of the present disclosure.

Claims

1. A head-mounted display system, comprising: a transparent eyepiece for projecting an image from a light projector system to an eye of a user during operation of the head mounted display, the transparent eyepiece comprising a substrate configured to direct light from the light projector system to the eye of the user; A camera and an imaging assembly defining an object plane and an axis orthogonal to the object plane at a center point of an object field of the imaging assembly, the imaging assembly being configured to image the user's eye onto the camera when the user's eye is positioned within the object field; Equipped with A head-mounted display system, wherein the imaging assembly comprises a reflective optical element arranged on the opposite side of the transparent eyepiece from the user's eye, the reflective optical element arranged to reflect light from the user's eye that is transmitted by the transparent eyepiece towards the camera, and the reflective optical element positioned away from the axis.

2. A head-mounted display system as described in claim 1, wherein the surface of the reflective optical element facing the substrate is concave.

3. A head-mounted display system as described in claim 1, further comprising an optical component having a refractive index different from the refractive index of the substrate, the optical component configured to refract light from the user's eye into the substrate.

4. A head-mounted display system as described in claim 3, wherein the optical components include optical cover glasses or prescription glasses.

5. A head-mounted display system as described in claim 3, wherein the optical component contacts the substrate.

6. A head-mounted display system as described in claim 1, wherein the axis of the reflective optical element is non-parallel to the axis of the imaging assembly.

7. A head-mounted display system as described in claim 1, wherein the reflective optical element includes a prism positioned on the world side of the substrate.

8. A head-mounted display system as described in claim 7, wherein the reflective optical element has zero refractive power.

9. A head-mounted display system as described in claim 8, wherein the reflective surface of the reflective optical element is angled at an acute angle relative to the axis.

10. A head-mounted display system as described in claim 7, further comprising an optical component having a refractive index different from the refractive index of the substrate, the optical component configured to refract light from the user's eye into the substrate.

11. A head-mounted display system as described in claim 10, wherein the optical component includes an exit pupil expander.

12. A head-mounted display system as described in claim 10, wherein the optical components include optical cover glasses or prescription glasses.

13. A head-mounted display system as described in claim 1, wherein the substrate is a transparent waveguide.

14. A head-mounted display system as described in claim 13, wherein the transparent waveguide comprises a polymer plastic material.

15. A head-mounted display system as described in claim 1, wherein the surface of the reflective optical element facing the substrate is reflective in the infrared spectrum and transparent in the visible spectrum.

16. A head-mounted display system as described in claim 1, further comprising a light source configured to illuminate the user's eyes.

17. A head-mounted display system as described in claim 16, wherein the light source emits infrared light.

Images