Eye imaging device using a diffractive optical element

Abstract

To provide a favorable eye-imaging apparatus using diffractive optical elements.SOLUTION: Examples of an eye-imaging apparatus using diffractive optical elements are provided. For example, an optical device comprises: a substrate having a proximal surface and a distal surface; a first coupling optical element disposed on one of the proximal and distal surfaces of the substrate; and a second coupling optical element disposed on one of the proximal and distal surfaces of the substrate and offset from the first coupling optical element. The first coupling optical element can be configured to deflect light at an angle for total internal reflection (TIR) of the light between the proximal and distal surfaces toward the second coupling optical element. The second coupling optical element can be configured to deflect light at an angle out of the substrate. The eye-imaging apparatus can be used in a head-mounted display such as an augmented or virtual reality display.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 474,419, filed on March 21, 2017, entitled "EYE - IMAGING APPARATUS USING DIFFRACTIVE OPTICAL ELEMENTS", the content of which is hereby incorporated by reference in its entirety into this specification.

[0002] This disclosure relates to virtual reality and augmented reality imaging and visualization systems, and more particularly to a compact imaging system for obtaining an image of an eye that uses a combined optical element to direct light to a camera assembly.

Background Art

[0003] Modern computing and display technologies have facilitated the development of systems for so - called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner that appears or is perceived to be real. A virtual reality or "VR" scenario typically involves the presentation of digital or virtual image information without transparency to other actual real - world visual inputs, and an augmented reality or "AR" scenario typically involves the presentation of digital or virtual image information as an augmentation to the visualization of the actual world around the user. A mixed reality or "MR" scenario is a type of AR scenario that typically involves virtual objects integrated into and responsive to the natural world. For example, in an MR scenario, AR image content is perceived to be blocked by or otherwise interact with objects within the real world.

[0004] Referring to FIG. 1, an augmented reality scene 10 is depicted, and to the user of the AR technology, a real-world park-like setting 20 is visible, featuring people, trees, buildings in the background, and a concrete platform 30. In addition to these items, the user of the AR technology also "sees" "virtual content" such as a robot image 40 standing on the real-world platform 30 and an avatar character 50 like a flying comic that appears to be an anthropomorphic honeybee, but these elements 40, 50 do not exist in the real world. The human visual perception system is complex, and the generation of AR technology that promotes a comfortable, natural, and rich presentation of virtual image elements among other virtual or real-world image elements is difficult.

[0005] The systems and methods disclosed herein address various issues related to AR and VR technologies.

SUMMARY OF THE INVENTION

MEANS FOR SOLVING THE PROBLEM

[0006] Various implementations of the methods and apparatuses within the scope of the appended claims each have several aspects, and no single one of them is involved in the desirable attributes disclosed herein. Without limiting the scope of the appended claims, several prominent features are described herein.

[0007] One aspect of the present disclosure provides a step of imaging an object using a camera assembly without directly visually recognizing the object. Thus, the optical device according to the embodiments described herein is configured to direct light from the object to an off-axis camera assembly so as to capture an image of the object as if it were in a direct visual recognition position.

[0008] In some embodiments, systems, devices, and methods for obtaining an image of an object using an off-axis camera assembly are disclosed. In one implementation, an optical device may include a substrate having a proximal surface and a distal surface, a first coupling optical element disposed on one of the proximal and distal surfaces of the substrate, and a second coupling optical element disposed on one of the proximal and distal surfaces of the substrate and offset from the first coupling optical element. The first coupling optical element may be configured to deflect light at an angle and to total internally reflect (TIR) the light between the proximal and distal surfaces toward the second coupling optical element. The second coupling optical element may be configured to deflect light outward from the substrate at an angle. In some embodiments, at least one of the first and second coupling optical elements includes a plurality of diffractive features.

[0009] In some embodiments, systems, devices, and methods for obtaining an image of an object using an off-axis camera assembly are disclosed. In one implementation, a head-mounted display (HMD) configured to be worn on a user's head may include a frame, a pair of optical elements supported by the frame such that each optical element of the pair of optical elements can be disposed in front of the user's eye, and an imaging system. The imaging system may include a camera assembly mounted to the frame and an optical device for directing light to the camera assembly. The optical device may include a substrate having a proximal surface and a distal surface, a first coupling optical element disposed on one of the proximal and distal surfaces of the substrate, and a second coupling optical element disposed on one of the proximal and distal surfaces of the substrate and offset from the first coupling optical element. The first coupling optical element may be configured to deflect light at an angle and to TIR the light between the proximal and distal surfaces toward the second coupling optical element. The second coupling optical element may be configured to deflect light outward from the substrate at an angle.

[0010] In some embodiments, systems, devices, and methods for obtaining an image of an object using an off-axis camera assembly are disclosed. In one implementation, an imaging system is disclosed that may include a substrate having a proximal surface and a distal surface. The substrate may include a first diffractive optical element disposed on one of the proximal and distal surfaces of the substrate, and a second diffractive optical element disposed on one of the proximal and distal surfaces of the substrate and offset from the first coupling optical element. The first diffractive optical element may be configured to deflect light at an angle and cause the light to undergo total internal reflection (TIR) between the proximal and distal surfaces toward a second coupling optical element. The second diffractive optical element may be configured to deflect light incident thereon out of the substrate at an angle. The imaging system may also include a camera assembly and may image the light deflected by the second coupling optical element. In some embodiments, the first and second diffractive optical elements include at least one of the surfaces of an off-axis diffractive optical element (DOE), an off-axis diffraction grating, an off-axis diffractive optical element (DOE), an off-axis holographic mirror (OAHM), or an off-axis volumetric diffractive optical element (OAVDOE), an off-axis cholesteric liquid crystal diffraction grating (OACLCG), a hot mirror, a prism, or a decorative lens.

[0011] In some embodiments, an off-axis camera assembly is used to obtain an image of an object, and systems, devices, and methods are disclosed. The method may include providing an imaging system in front of an object to be imaged. The imaging system may be a substrate on which a first coupling optical element and a second coupling optical element are disposed on one of the proximal surface and the distal surface of the substrate, respectively, and are offset from each other. The first coupling optical element may be configured to deflect light at an angle and cause the light to undergo total internal reflection (TIR) between the proximal surface and the distal surface toward the second coupling optical element. The second coupling optical element may be configured to deflect light out of the substrate at an angle. The method may also include capturing light using a camera assembly oriented to receive the light deflected by the second coupling optical element, and generating an off-axis image of the object based on the captured light.

[0012] In any of the embodiments, the proximal surface and the distal surface of the substrate can be parallel to each other, but this is not necessary. For example, the substrate may include a wedge.

[0013] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. Neither this summary nor the following detailed description purports to define or limit the scope of the subject matter of the invention. This specification also provides, for example, the following items. (Item 1) An optical device, comprising: a substrate having a proximal surface and a distal surface; a first coupling optical element disposed on one of the proximal surface and the distal surface; and a second coupling optical element disposed on one of the proximal surface and the distal surface and laterally offset from the first coupling optical element along a direction parallel to the proximal surface or the distal surface comprising, wherein the first coupling optical element is configured to deflect light at an angle and cause the light to undergo total internal reflection (TIR) between the proximal surface and the distal surface towards the second coupling optical element, and the second coupling optical element is configured to deflect light out of the substrate at an angle, an optical device. (Item 2) The optical device according to item 1, wherein the substrate is transparent to visible light. (Item 3) The optical device according to item 1, wherein the substrate includes a polymer. (Item 4) The optical device according to item 1, wherein the substrate includes polycarbonate. (Item 5) The optical device according to item 1, wherein the first and second coupling optical elements are external to and fixed to at least one of the proximal surface and the distal surface of the substrate. (Item 6) The optical device according to item 1, wherein the first and second coupling optical elements include a part of the substrate. (Item 7) The optical device according to item 1, wherein at least one of the first and second coupling optical elements includes a plurality of diffraction features. (Item 8) The optical device according to item 7, wherein the plurality of diffraction features have a relatively high diffraction efficiency for a certain wavelength range so as to diffract substantially all of the light in the certain wavelength range. (Item 9) The optical device according to item 7, wherein the plurality of diffraction features diffract light in at least one direction based at least in part on the period of the plurality of diffraction elements, and the at least one direction is selected such that the light undergoes TIR between the proximal surface and the distal surface. (Item 10) At least one of the first or second coupling optical elements includes at least one of an off-axis diffractive optical element (DOE), an off-axis diffraction grating, an off-axis diffractive optical element (DOE), an off-axis holographic mirror (OAHM), or an off-axis volumetric diffractive optical element (OAVDOE), or an off-axis cholesteric liquid crystal diffraction grating (OACLCG), the optical device according to item 1. (Item 11) The first and second coupling optical elements are each configured to deflect light in a first wavelength range while transmitting light in a second wavelength range, the optical device according to item 1. (Item 12) The first wavelength range includes light in at least one of the infrared (IR) or near-IR spectra, and the second wavelength range includes light in the visible spectrum, the optical device according to item 11. (Item 13) The first and second coupling optical elements selectively reflect light in a certain wavelength range, the first coupling optical element is disposed on the distal surface of the substrate, and the second coupling optical element is disposed on the proximal surface of the substrate, the optical device according to item 1. (Item 14) The first and second coupling optical elements selectively transmit light in a certain wavelength range, the first coupling optical element is disposed on the proximal surface of the substrate, and the second coupling optical element is disposed on the distal surface of the substrate, the optical device according to item 1. (Item 15) The first coupling optical element selectively reflects light in a certain wavelength range, the second coupling optical element selectively transmits light in the certain wavelength range, and the first and second coupling optical elements are disposed on the distal surface of the substrate, the optical device according to item 1. (Item 16) The first coupling optical element selectively transmits light in a certain wavelength range, the second coupling optical element selectively reflects light in the certain wavelength range, and the first and second coupling optical elements are disposed on the proximal surface of the substrate, the optical device according to item 1. (Item 17) A head-mounted display (HMD) configured to be worn on a user's head, the HMD comprising: a frame; a pair of optical elements supported by the frame such that each optical element of the pair of optical elements can be disposed in front of the user's eye; an imaging system, the imaging system comprising: a camera assembly mounted on the frame; the optical device according to item 1; and an imaging system; An HMD comprising. (Item 18) The HMD according to item 17, wherein at least one optical element of the pair of optical elements includes the substrate. (Item 19) The HMD according to item 17, wherein the substrate is disposed on a surface of at least one optical element of the pair of optical elements. (Item 20) The HMD according to item 17, wherein the frame includes a pair of earhooks, and the camera assembly is mounted on one of the pair of earhooks. (Item 21) The HMD according to item 17, wherein the camera assembly is a front-facing camera assembly configured to image light received from the second coupling optical element. (Item 22) The HMD according to item 17, wherein the camera assembly is a rear-facing camera assembly disposed in a direction facing the user, and the rear-facing camera assembly is configured to image light received from the second coupling optical element. (Item 23) The HMD according to item 17, further comprising a light source that emits light in a range of a first wavelength toward at least one of the user's eye, a part of the eye, or a part of the tissue surrounding the eye. (Item 24) Light in the range of the first wavelength is reflected toward the first coupling optical element by at least one of the user's eye, a part of the eye, or a part of the tissue surrounding the eye, in the HMD according to item 23. (Item 25) In the HMD according to item 17, each of the pair of optical elements is transparent to visible light. (Item 26) In the HMD according to item 17, each of the pair of optical elements is configured to display an image to the user. (Item 27) The camera assembly is configured to form an image of at least one of the user's eye, a part of the eye, or a part of the tissue surrounding the eye, based at least in part on the light received from the second coupling optical element, in the HMD according to item 17. (Item 28) In the HMD according to item 27, the HMD is configured to track the user's line of sight based on an image of at least one of the user's eye, a part of the eye, or a part of the tissue surrounding the eye. (Item 29) The image formed by the camera assembly is installed in front of the user's eye and coincides with the image formed by a camera that directly visualizes at least one of the user's eye, a part of the eye, or a part of the tissue surrounding the eye, in the HMD according to item 27. (Item 30) In the HMD according to item 17, the optical device is arranged to reduce stray light received by the camera assembly. (Item 31) In the HMD according to item 17, the size of the first coupling optical element is less than the stride distance of the light reflected between the distal surface and the proximal surface of the substrate, and the stride distance is based on the thickness of the substrate and the angle at which the first coupling optical element deflects the light. (Item 32) In the HMD according to item 31, the size of the first coupling optical element is based on the visual field of the user's eye. (Item 33) The HMD according to item 17, wherein an image of the user's eye formed by the camera assembly is indistinguishable from an image of the user's eye formed by a camera installed in front of the user's eye. (Item 34) A non-transitory data storage device configured to store an image obtained by the camera assembly, A hardware processor communicating with the non-transitory data storage device, the hardware processor being programmed with executable instructions for analyzing the image and performing one or more of eye tracking, biometric identification, multi-view reconstruction of the shape of the eye, estimation of the accommodation state of the eye, or imaging of the retina, iris, or other prominent patterns of the eye, and evaluation of the physiological state of the user. The HMD according to item 17, further comprising (Item 35) An imaging system, A substrate having a proximal surface and a distal surface, the substrate A first diffractive optical element disposed on one of the proximal surface and the distal surface, A second diffractive optical element disposed on one of the proximal surface and the distal surface, the second diffractive optical element being offset from the first diffractive optical element along a direction parallel to the proximal surface or the distal surface. Comprising, the first diffractive optical element is configured to deflect light at an angle and cause the light to undergo total internal reflection (TIR) between the proximal surface and the distal surface toward a second coupling optical element, and the second diffractive optical element is configured to deflect light incident thereon out of the substrate at an angle. A camera assembly for imaging the light deflected by the second diffractive optical element An imaging system comprising (Item 36) The first and second diffractive optical elements include at least one of the surfaces of an off-axis diffractive optical element (DOE), an off-axis diffraction grating, an off-axis diffractive optical element (DOE), an off-axis holographic mirror (OAHM), or an off-axis volumetric diffractive optical element (OAVDOE), an off-axis cholesteric liquid crystal diffraction grating (OACLCG), a hot mirror, a prism, or a decorative lens, the imaging system according to item 35. (Item 37) A method of imaging an object using a virtual camera, the method comprising: Providing in front of an object to be imaged by an imaging system, the imaging system comprising: A substrate having a first coupling optical element and a second coupling optical element, each disposed on one of a proximal surface and a distal surface of the substrate and offset from each other, the first coupling optical element being configured to deflect light at an angle and to totally internally reflect (TIR) the light between the proximal surface and the distal surface toward the second coupling optical element, the second coupling optical element being configured to deflect the light out of the substrate at an angle Comprising, Capturing the light using a camera assembly oriented to receive the light deflected by the second coupling optical element; Generating an off-axis image of the object based on the captured light; And including. (Item 38) The method according to item 37, wherein the first and second coupling optical elements each deflect light in a first wavelength range while transmitting light in a second wavelength range. (Item 39) The method according to item 37, further comprising illuminating the object using a first wavelength range emitted by a light source. (Item 40) Analyzing the off-axis image; Performing one or more of eye tracking, biometric identification, multi-view reconstruction of the shape of the eye, estimation of the accommodation state of the eye, or imaging of the retina, iris, or other prominent patterns of the eye, and evaluation of the physiological state of the user, based at least in part on the off-axis image being analyzed The method according to item 37, further comprising

Brief Description of the Drawings

[0014]

Figure 1

[0015]

Figure 2

[0016]

Figure 3

[0017]

Figure 4

[0018]

Figure 5

[0019]

Figure 6

[0020]

Figure 7

[0021]

Figure 8

[0022]

Figure 9A

[0023]

Figure 9B

[0024]

Figure 9C

[0025]

Figure 10A

Figure 10B

[0026]

Figure 11

[0027]

Figure 12A

[0028]

Figure 12B

[0029]

Figure 13

[0030]

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

[0031]

Figure 19

DETAILED DESCRIPTION OF THE INVENTION

[0032] Throughout the drawings, reference numerals may be reused to indicate corresponding between the elements being referenced. The drawings are provided to illustrate the exemplary embodiments described in this specification and are not intended to limit the scope of the present disclosure. (SUMMARY)

[0033] A head-mounted display (HMD) can use information about the wearer's eye condition for various purposes. For example, this information can be used for estimating the wearer's line of sight direction, biometric identification, vision research, evaluation of the wearer's physiological state, and the like. However, imaging the eye can be difficult. The distance between the HMD and the wearer's eye is short. Furthermore, while gaze tracking requires a wide field of view (FOV), biometric identification requires a relatively high number of pixels on a target on the iris. With respect to an imaging system seeking to accomplish both of these purposes, these requirements are mainly in conflict. Furthermore, both problems can be further complicated by occlusion by eyelids and eyelashes. Some current implementations for tracking eye movement use a camera mounted on the HMD that is directed directly at the eye to capture a direct image of the eye. However, in order to achieve the desired FOV and number of pixels, the camera is mounted within the wearer's FOV and thus tends to interfere with and obstruct the wearer's ability to see the surrounding world. Other implementations image the eye directly while moving the camera so as not to obstruct the wearer's view, but this results in imaging the eye from a high angle, causing image distortion and reducing the available field of view for imaging the eye.

[0034] Embodiments of the imaging system described herein address some or all of these problems. The various embodiments described herein provide devices and systems that enable the wearer to view the surrounding world while enabling the eye to be imaged. For example, the imaging system can include a substrate disposed along a line of sight between the eye and the camera assembly. The substrate can include one or more coupling optical elements configured to direct light from the eye into the substrate. The substrate can act as an optical waveguide (sometimes also referred to as a light pipe) that directs light towards the camera assembly. The light can then exit the substrate and be directed towards the camera assembly via one or more coupling optical elements. The camera assembly, which receives the light, can thus capture an image of the eye (sometimes hereinafter also referred to as a "direct view image") as if it were at a direct view position from a remote location (sometimes also referred to herein as "off-axis").

[0035] Some embodiments of the imaging system described herein provide a substrate that includes first and second coupling optical elements that are laterally offset from each other. The substrate includes a surface closest to the eye (sometimes also referred to herein as the proximal surface) and a surface farthest from the eye (sometimes also referred to as the distal surface). The first and second coupling optical elements described herein can be disposed on or adjacent to the proximal surface, on or adjacent to the distal surface, or within the substrate. The first coupling optical element (sometimes also referred to herein as an internal coupling optical element) can be configured to deflect light from the eye into the substrate such that the light propagates through the substrate by total internal reflection (TIR). The light can be incident on a second coupling optical element that is configured to extract the light and deflect it towards the camera assembly. As used herein, deflection can refer to a change in the direction of light after interacting with something, for example, an optical component that deflects light can refer to reflection, diffraction, refraction, a change in direction while transmitting through an optical component, etc.

[0036] In some embodiments, the imaging system described herein may be part of the display optics of the HMD (or the lenses within a pair of glasses). One or more coupling optical elements may be selected to deflect a first range of wavelengths while allowing unobstructed propagation of a second range of wavelengths (e.g., a range of wavelengths different from the first range) through the substrate. The first range of wavelengths can be within the infrared (IR), and the second range of wavelengths can be within the visible. For example, the substrate can comprise a reflective coupling optical element, which reflects IR light while transmitting visible light. In effect, the imaging system acts as if there were a virtual camera assembly directed backwards towards the wearer's eyes. Thus, the virtual camera assembly can image virtual IR light propagating from the wearer's eyes through the substrate, while visible light from the outside world can be transmitted through the substrate and perceived by the wearer.

[0037] The camera assembly may be configured to view the wearer's eyes and, for example, capture an image of the eyes. The camera assembly can be mounted in close proximity to the wearer's eyes such that the camera assembly does not interfere with the wearer's view of the surrounding world or disrupt the operation of the HMD. In some embodiments, the camera assembly can be positioned on a frame of the wearable display system, such as on an earpiece, or incorporated within the HMD's eyepiece or below the eye and above the cheek. In some embodiments, a second camera assembly can be used for the wearer's other eye so that each eye can be imaged separately. The camera assembly can include an IR digital camera sensitive to IR radiation.

[0038] The camera assembly can be mounted so as to face forward (in the direction of the wearer's vision) or face rearward and be directed towards the eyes. In some embodiments, by placing the camera assembly closer to the wearer's ear, the weight of the camera assembly can also be closer to the ear, and the HMD can be easier to wear compared to an HMD where the camera assembly is placed closer to the front of the HMD or in a direct viewing arrangement. Additionally, by placing the camera assembly near the wearer's temple, the distance from the wearer's eye to the camera assembly is approximately doubled compared to a camera assembly placed near the front of the HMD. Since the depth of field of the image is approximately proportional to this distance, the depth of field for the camera assembly is approximately twice as large as that of a direct viewing camera assembly. A larger depth of field for the camera assembly can be advantageous for imaging the eye region of a wearer having a large or protruding nose, supraorbital ridge, etc. In some embodiments, the position of the camera assembly may be based on the packaging or design considerations of the HMD. For example, in some configurations, it may be advantageous to place the camera assembly facing rearward or forward.

[0039] Without subscribing to any particular scientific theory, the embodiments described herein may include several non-limiting advantages. Some embodiments are capable of increasing the physical distance between the camera assembly and the eye, which can facilitate positioning the camera assembly outside the wearer's field of view and thus not interfere with the wearer's view while enabling the capture of a direct viewing image of the eye. Some of the embodiments described herein can also enable eye tracking using a larger field of view than conventional systems and can thus be configured to enable eye tracking over a wide range of positions. The use of IR imaging can facilitate imaging the eyes without interfering with the wearer's ability to view the environment through the substrate.

[0040] Reference is now made to the figures, where like reference numerals refer to like parts throughout. Exemplary HMD Device

[0041] Figure 2 illustrates an embodiment of a wearable display system 60. The display system 60 includes a display 70 and various mechanical and electronic modules and systems to support the functions of the display 70. The display 70 may be coupled to a frame 80, which is wearable by a display system user or viewer 90 and is configured to position the display 70 in front of the user's 90 eyes. The display 70 may be considered eyewear in some embodiments. In some embodiments, a speaker 100 is coupled to the frame 80 and configured to be positioned adjacent to the user's 90 outer ear canal (in some embodiments, another speaker, not shown, may be positioned adjacent to the user's other outer ear canal to provide stereo / formable sound control). In some embodiments, the display system may also include one or more microphones 110 or other devices to detect sound. In some embodiments, the microphone is configured to enable the user to provide an input or command to the system 60 (e.g., selection of a voice menu command, natural language question, etc.) and / or enable audio communication with other persons (e.g., other users of a similar display system). The microphone may further be configured as a peripheral sensor to collect audio data (e.g., sound from the user and / or environment). In some embodiments, the display system may also include a peripheral sensor 120a, which is separate from the frame 80 and may be attached to the user's 90 body (e.g., on the user's 90 head, torso, limbs, etc.). The peripheral sensor 120a may be configured to obtain data characterizing the physiological state of the user 90 in some embodiments. For example, the sensor 120a may be an electrode.

[0042] Continuing to refer to FIG. 2, the display 70 is operably coupled to a local data processing module 140 by a communication link 130, such as a wired conductor or wireless connectivity, which may be mounted in various configurations, such as fixedly attached to the frame 80, fixedly attached to a helmet or hat worn by the user, built into headphones, or removably attached to the user 90 in another manner (e.g., in a backpack configuration, in a belt attachment configuration). Similarly, the sensor 120a may be operably coupled to a local processor and data module 140 by a communication link 120b, such as a wired conductor or wireless connectivity. The local processing and data module 140 may comprise a digital memory such as a hardware processor and non-volatile memory (e.g., flash memory or hard disk drive), both of which may be utilized to assist in the processing, caching, and storage of data. The data may include a) data captured from sensors such as an image capture device (e.g., a camera, etc.), microphone, inertial measurement unit, accelerometer, compass, GPS unit, wireless device, gyroscope, and / or other sensors disclosed herein (e.g., operably coupled to the frame 80 or otherwise attachable to the user 90), and / or b) data obtained and / or processed using a remote processing module 150 and / or remote data repository 160 (including data related to virtual content), optionally for passage to the display 70 after processing or retrieval. The local processing and data module 140 may be operably coupled to the remote processing module 150 and remote data repository 160 by communication links 170, 180 via a wired or wireless communication link or the like, such that these remote modules 150, 160 are operably coupled to each other and available as resources to the local processing and data module 140.In some embodiments, the local processing and data module 140 may include one or more of an image capture device, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope. In some other embodiments, one or more of these sensors may be attached to the frame 80 or may be of an independent structure that communicates with the local processing and data module 140 via a wired or wireless communication path.

[0043] Continuing to refer to FIG. 2, in some embodiments, the remote processing module 150 may include one or more processors configured to analyze and process data and / or image information. In some embodiments, the remote data repository 160 may comprise a digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, the remote data repository 160 may include one or more remote servers that provide information, e.g., information for generating augmented reality content, to the local processing and data module 140 and / or the remote processing module 150. In some embodiments, all data is stored and all calculations are performed within the local processing and data module, enabling fully autonomous use from the remote module.

[0044] Perception of an image as "3D" or "3 - D" can be achieved by providing slightly different presentations of the image to each eye of the viewer. FIG. 3 illustrates a conventional display system for simulating a 3D image for a user. One of two distinct different images 190, 200 is output to the user for each eye 210, 220. Images 190, 200 are separated from eyes 210, 220 by a distance 230 along an optical axis or z - axis parallel to the viewer's line of sight. Images 190, 200 are flat, and eyes 210, 220 can be focused on the images by taking a single focused state. Such a 3 - D display system relies on the human visual system to combine images 190, 200 and provide perception of the depth and / or scale of the combined image.

[0045] However, it should be understood that the human visual system is more complex and it is more difficult to provide a realistic perception of depth. For example, many viewers of conventional "3-D" display systems find such systems uncomfortable or may not perceive a sense of depth at all. Although not limited by theory, an object viewer is thought to be able to perceive an object as "three-dimensional" due to the combination of vergence and accommodation. The movement of the vergence of two eyes relative to each other (e.g., the movement of the pupils towards or away from each other, the rotation of the eyes to converge the line of sight and fixate on an object) is closely associated with the focusing (or "accommodation") of the eye's lens and pupil. Under normal conditions, the change in the focus of the eye's lens or the accommodation of the eye to change the focus from one object to another object at a different distance will automatically cause a corresponding change in vergence under the relationship known as the "accommodation-vergence reflex" and dilation or constriction of the pupil. Similarly, a change in vergence will, under normal conditions, induce a corresponding change in accommodation of the lens shape and pupil size. As described herein, many stereoscopic or "3-D" display systems use slightly different presentations (and thus slightly different images) to each eye to display a scene so that a three-dimensional viewpoint is perceived by the human visual system. However, such systems are uncomfortable for many viewers, especially because they simply provide different presentations of the scene and, when the eyes view the entire image information in a single accommodated state, they function against the "accommodation-vergence reflex". A display system that provides better alignment between accommodation and vergence can form a more realistic and comfortable simulation of a three-dimensional image and contribute to increased wearing duration and thus compliance with diagnostic and therapy protocols.

[0046] FIG. 4 illustrates a side view of an approach for simulating a three-dimensional image using a plurality of depth planes. Referring to FIG. 4, objects at various distances from eyes 210, 220 on the z-axis are focused by eyes 210, 220 such that those objects are in focus. Eyes 210, 220 assume a particular focused state and focus objects at different distances along the z-axis. As a result, a particular focused state can be said to be associated with a particular one of depth planes 240 having an associated focal length such that an object or a portion of an object in a particular depth plane is in focus when the eye is in a focused state with respect to that depth plane. In some embodiments, the three-dimensional image may be simulated by providing different presentations of the image for each of eyes 210, 220 and also by providing different presentations of the image corresponding to each of the depth planes. For clarity of illustration, the fields of view of eyes 210, 220 are shown as being separate, but it should be understood that they may overlap, for example, as the distance along the z-axis increases. Additionally, for ease of illustration, the contours of the depth planes are shown as being flat, but it should be understood that they may be curved in physical space such that all features within a depth plane are in focus with the eye in a particular focused state.

[0047] The distance between the object and the eye 210 or 220 can also vary the amount of divergence of light from that object as viewed by that eye. FIGS. 5A - 5C illustrate the relationship between distance and light ray divergence. The distance between the object and the eye 210 is represented in the order of decreasing distances R1, R2, and R3. As shown in FIGS. 5A - 5C, the light rays diverge more as the distance to the object decreases. As the distance increases, the light rays become more collimated. In other words, it can be said that the light field generated by a point (object or part of an object) has a spherical wavefront curvature that is a function of the distance the point is away from the user's eye. The curvature increases with a decrease in the distance between the object and the eye 210. As a result, at different depth planes, the divergence of the light rays also differs, and the divergence increases with a decrease in the distance between the depth plane and the viewer's eye 210. Only the monocular 210 is illustrated in FIGS. 5A - 5C and other figures in this specification for clarity of illustration, but it should be understood that the discussion regarding the eye 210 can apply to both eyes 210 and 220 of the viewer.

[0048] Without being limited by theory, it is believed that the human eye can typically interpret a finite number of depth planes and provide depth perception. As a result, a highly realistic simulation of the perceived depth can be achieved by providing different presentations of images corresponding to each of these limited number of depth planes to the eye. The different presentations are separately focused by the viewer's eye, thereby based on the eye accommodation required to focus on different image features of a scene located on different depth planes and / or based on the observation of different out - of - focus image features on different depth planes, which may be used to provide depth cues to the user. Example of a waveguide stack assembly

[0049] FIG. 6 illustrates an example of a waveguide stack for outputting image information to a user. The display system 250 includes a stack of waveguides or a stacked waveguide assembly 260 that can be utilized to provide three-dimensional perception to the eye / brain using a plurality of waveguides 270, 280, 290, 300, 310. In some embodiments, the display system 250 is the system 60 of FIG. 2, and FIG. 6 schematically shows some parts of that system 60 in more detail. For example, the waveguide assembly 260 may be part of the display 70 of FIG. 2. It should be understood that the display system 250 may be regarded as a light field display in some embodiments.

[0050] Continuing to refer to FIG. 6, waveguide assembly 260 may also include a plurality of features 320, 330, 340, 350 between the waveguides. In some embodiments, features 320, 330, 340, 350 may be one or more lenses. Waveguides 270, 280, 290, 300, 310 and / or the plurality of lenses 320, 330, 340, 350 may be configured to transmit image information to the eye using various levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane. Image input devices 360, 370, 380, 390, 400 may function as light sources for the waveguides and may be utilized to input image information into waveguides 270, 280, 290, 300, 310, and each may be configured to disperse incident light across each individual waveguide for output toward the eye 210 as described herein. Light exits from output surfaces 410, 420, 430, 440, 450 of image input devices 360, 372, 380, 390, 400 and is input into corresponding input surfaces 460, 470, 480, 490, 500 of waveguides 270, 280, 290, 300, 310. In some embodiments, input surfaces 460, 470, 480, 490, 500 may each be an edge of the corresponding waveguide or a part of the major surface of the corresponding waveguide (i.e., one of the waveguide surfaces facing directly toward the world 510 or the viewer's eye 210). In some embodiments, a single beam of light (e.g., a collimated beam) may be input into each waveguide and output an entire field of cloned collimated beams, directed toward the eye 210 at a particular angle (and amount of divergence) corresponding to the depth plane associated with the particular waveguide. In some embodiments, a single one of image input devices 360, 370, 380, 390, 400 may be associated with and input light into a plurality (e.g., three) of waveguides 270, 280, 290, 300, 310.

[0051] In some embodiments, the image input devices 360, 370, 384, 390, 400 are each discrete displays that generate image information for input into their respective waveguides 270, 280, 290, 300, 310. In some other embodiments, the image input devices 360, 370, 380, 390, 400 are the output ends of a single multiplexed display that can send image information, for example, via one or more optical waveguides (such as optical fiber cables) to each of the image input devices 360, 370, 380, 390, 400. It should be understood that the image information provided by the image input devices 360, 370, 380, 390, 400 may include light of different wavelengths or colors (e.g., different primary colors as discussed herein).

[0052] In some embodiments, the light input into the waveguides 270, 280, 290, 300, 310 is provided by an optical projector system 520, which includes an optical module 530 that may include a light emitter such as a light emitting diode (LED). The light from the optical module 530 may be directed and modified by a beam splitter 550 and an optical modulator 540, such as a spatial light modulator. The optical modulator 540 may be configured to vary the perceived intensity of the light input into the waveguides 270, 280, 290, 300, 310. Examples of spatial light modulators include liquid crystal displays (LCDs) including liquid crystal on silicon (LCOS) displays.

[0053] In some embodiments, the display system 250 may be a scanning fiber display comprising one or more scanning fibers configured to project light in various patterns (e.g., raster scan, helical scan, Lissajous pattern, etc.) into one or more waveguides 270, 280, 290, 300, 310 and ultimately into the viewer's eye 210. In some embodiments, the illustrated image input devices 360, 370, 380, 390, 400 may schematically represent a single scanning fiber or a bundle of scanning fibers configured to input light into one or more waveguides 270, 280, 290, 300, 310. In some other embodiments, the illustrated image input devices 360, 370, 380, 390, 400 may schematically represent a plurality of scanning fibers or a plurality of bundles of scanning fibers, each configured to input light into an associated one of the waveguides 270, 280, 290, 300, 310. It should be understood that one or more optical fibers may be configured to transmit light from the optical module 530 to one or more waveguides 270, 280, 290, 300, 310. It should be understood that one or more intervening optical structures may be provided between the scanning fiber or fibers and one or more waveguides 270, 280, 290, 300, 310, and may, for example, redirect light exiting the scanning fiber into one or more of the waveguides 270, 280, 290, 300, 310.

[0054] Controller 560 controls the operation of one or more of the stacked waveguide assemblies 260, including the operation of the image input devices 360, 370, 380, 390, 400, the light source 530, and the optical module 540. In some embodiments, controller 560 is part of the local data processing module 140. Controller 560 includes programming (e.g., instructions in a non-transitory medium) that adjusts the timing and provision of image information to waveguides 270, 280, 290, 300, 310, for example, according to any of the various schemes disclosed herein. In some embodiments, the controller may be a single integrated device or a distributed system connected by wired or wireless communication channels. Controller 560 may, in some embodiments, also be part of processing module 140 or 150 (FIG. 2).

[0055] Continuing to refer to FIG. 6, the waveguides 270, 280, 290, 300, 310 may be configured to propagate light within each individual waveguide by TIR. The waveguides 270, 280, 290, 300, 310 may each be planar, or have another shape (e.g., curved), with a major top surface and a major bottom surface and an edge extending between their major top and bottom surfaces. In the illustrated configuration, the waveguides 270, 280, 290, 300, 310 each include external coupling optical elements 570, 580, 590, 600, 610 configured to extract light from the waveguides by redirecting the light to propagate within each individual waveguide and outputting image information from the waveguides to the eye 210. The extracted light may also be referred to as external coupled light, and the optical elements that externally couple the light may also be referred to as light extraction optical elements. The beam of extracted light may be output by the waveguide at the location where the light propagating within the waveguide impinges on the light extraction optical element. The external coupling optical elements 570, 580, 590, 600, 610 may be gratings, for example, that include diffractive optical features as further discussed herein. For ease of explanation and clarity of the drawings, the external coupling optical elements 570, 580, 590, 600, 610 are shown disposed on the bottom major surfaces of the waveguides 270, 280, 290, 300, 310, but in some embodiments, the external coupling optical elements 570, 580, 590, 600, 610 may be disposed on the top major surface and / or bottom major surface, and / or directly disposed within the volume of the waveguides 270, 280, 290, 300, 310, as further discussed herein. In some embodiments, the external coupling optical elements 570, 580, 590, 600, 610 may be attached to a transparent substrate and formed within a layer of the material forming the waveguides 270, 280, 290, 300, 310. In some other embodiments, the waveguides 270, 280, 290, 300, 310 may be monolithic material components, and the external coupling optical elements 570, 580, 590, 600, 610 may be formed on and / or within the surface of the material component.

[0056] Continuing to refer to FIG. 6, as discussed herein, each of the waveguides 270, 280, 290, 300, 310 is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 270 closest to the eye may be configured to deliver collimated light (input into such a waveguide 270) to the eye 210. The collimated light may represent an optically infinite focal plane. The next upper waveguide 280 may be configured to send out collimated light that passes through a first lens 350 (e.g., a negative lens) before reaching the eye 210. Such a first lens 350 may be configured to generate some convex wavefront curvature such that the eye / brain interprets the light originating from the next upper waveguide 280 as originating from a first focal plane that is closer in an inward direction from the optically infinite distance towards the eye 210. Similarly, the third upper waveguide 290 passes its output light through both the first lens 350 and the second lens 340 before reaching the eye 210. The combined refractive power of the first lens 350 and the second lens 340 may be configured to generate another incremental amount of wavefront curvature such that the eye / brain interprets the light originating from the third waveguide 290 as originating from a second focal plane that is closer in an inward direction from the optically infinite distance towards the person than the light from the next upper waveguide 280 was.

[0057] The other waveguide layers 300, 310 and lenses 330, 320 are similarly configured, and the top waveguide 310 in the stack sends its output through all of the lenses between it and the eye for the converging focusing power that represents the focal plane closest to the person. When viewing / interpreting light originating from the world 510 on the other side of the stacked waveguide assembly 260, a compensating lens layer 620 may be disposed on top of the stack to compensate for the stack of lenses 320, 330, 340, 350. Such a configuration provides the same number of perceived focal planes as there are available waveguide / lens pairs. Both the external coupling optical elements of the waveguides and the focusing sides of the lenses may be static (i.e., not dynamic or electroactive). In some alternative embodiments, one or both may be dynamic using electroactive features.

[0058] In some embodiments, two or more of the waveguides 270, 280, 290, 300, 310 may have the same associated depth plane. For example, a plurality of waveguides 270, 280, 290, 300, 310 may be configured to output images set to the same depth plane, or a plurality of subsets of the waveguides 270, 280, 290, 300, 310 may be configured to output images set to the same plurality of depth planes, with one set per depth plane. This can provide the advantage of forming tiled images to provide an extended field of view in those depth planes.

[0059] Continuing to refer to FIG. 6, the external coupling optical elements 570, 580, 590, 600, 610 may be configured to redirect light from their respective waveguides for a particular depth plane associated with the waveguide and output the light with an appropriate amount of divergence or collimation. As a result, waveguides having different associated depth planes may have different configurations of the external coupling optical elements 570, 580, 590, 600, 610, which output light with different amounts of divergence depending on the associated depth plane. In some embodiments, the light extraction optical elements 570, 580, 590, 600, 610 may be volume features or surface features configured to output light at a specific angle. For example, the light extraction optical elements 570, 580, 590, 600, 610 may be volume holograms, surface holograms, and / or diffraction gratings. In some embodiments, the features 320, 330, 340, 350 may not be lenses. Rather, they may simply be spacers (e.g., structures for forming cladding layers and / or voids).

[0060] In some embodiments, the external coupling optical elements 570, 580, 590, 600, 610 are diffraction features or “diffractive optical elements” (also referred to herein as “DOEs”) that form a diffraction pattern. Preferably, the DOE has a sufficiently low diffraction efficiency such that only a portion of the light of the beam is deflected toward the eye 210 at each intersection of the DOE, while the remainder continues to move through the waveguide via TIR. The light carrying the image information is thus split into several associated output beams that exit the waveguide at multiple locations, resulting in a very uniform pattern of output emission toward the eye 210 with respect to this particular collimated beam that bounces within the waveguide.

[0061] In some embodiments, one or more DOEs may be switchable between an “on” state where they actively diffract and an “off” state where they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer dispersed liquid crystal, in which microdroplets have a diffraction pattern in a host medium and the refractive index of the microdroplets may be switched to substantially match that of the host material (in which case the pattern does not significantly diffract the incident light), or the microdroplets may be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts the incident light).

[0062] In some embodiments, a camera assembly 630 (e.g., a digital camera including a visible light and IR light camera) may be provided to capture an image of the eye 210, a portion of the eye 210, or at least a portion of the tissue surrounding the eye 210, for example, to detect user input, extract biometric information from the eye, estimate and track the line of sight direction of the eye, monitor the user's physiological state, and the like. As used herein, the camera may be any image capture device. In some embodiments, the camera assembly 630 may include an image capture device and a light source 632 that projects light (e.g., IR or near IR light) onto the eye, and that light can then be reflected by the eye and detected by the image capture device. In some embodiments, the light source 632 includes a light emitting diode (「LED」) and emits IR or near IR. Although the light source 632 is shown as being attached to the camera assembly 630, it should be understood that the light source 632 may be located in other areas relative to the camera assembly such that the light emitted by the light source (e.g., the light source 530 described below) is directed at the wearer's eye. In some embodiments, the camera assembly 630 may be attached to the frame 80 (FIG. 2) and may communicate electrically with a processing module 140 or 150 that processes image information from the camera assembly 630 and can make various determinations regarding, for example, the user's physiological state, the wearer's line of sight direction, iris identification, and the like. It should be understood that information regarding the user's physiological state may be used to determine the user's behavior or emotional state. Examples of such information include the user's movement or the expression on the user's face. The user's behavior or emotional state may then be triangulated using the collected environmental or virtual content data to determine the relationship between the behavior or emotional state, the physiological state, and the environmental or virtual content data. In some embodiments, one camera assembly 630 may be utilized per eye to monitor each eye separately.

[0063] Referring now to FIG. 7, an embodiment of an output beam output by a waveguide is shown. Although one waveguide is illustrated, it should be understood that other waveguides within the waveguide assembly 260 (FIG. 6) may function in a similar manner, and the waveguide assembly 260 includes a plurality of waveguides. Light 640 is introduced into the waveguide 270 at the input surface 460 of the waveguide 270 and propagates within the waveguide 270 by TIR. At the point where the light 640 impinges on the DOE 570, a portion of the light exits the waveguide as the output beam 650. The output beam 650 is illustrated as being substantially parallel, but as discussed herein, it may also be redirected to propagate to the eye 210 at an angle (e.g., to form a diverging output beam) depending on the depth plane associated with the waveguide 270. The substantially parallel output beam may represent a waveguide with an external coupling optical element that externally couples the light to form an image that appears to be set in a depth plane at a long distance (e.g., optical infinity) from the eye 210. Other waveguides or other sets of external coupling optical elements may output a more divergent output beam pattern, which would require the eye 210 to focus at a closer distance and would be interpreted by the brain as light from a distance closer to the eye 210 than optical infinity.

[0064] In some embodiments, a full-color image may be formed in each depth plane by overlaying the image on each of the primary colors, e.g., three or more primary colors. FIG. 8 illustrates an example of a stacked waveguide assembly, and each depth plane includes an image formed using a plurality of different primary colors. The illustrated embodiment shows depth planes 240a - 240f, but more or fewer depths may also be considered. Each depth plane may have three or more primary color images, i.e., a first image of a first color G, a second image of a second color R, and a third image of a third color B, associated therewith. Different depth planes are shown in the figure by different numbers related to the diopter (dpt) following the letters G, R, and B. By way of example only, the numbers following each of these letters indicate the diopter (1 / m), i.e., the inverse distance of the depth plane from the viewer, and each box in the figure represents an individual primary color image. In some embodiments, to account for differences in the focusing of light of different wavelengths by the eye, the exact location of the depth planes for different primary colors may vary. For example, the different primary color images for a given depth plane may be placed on a depth plane corresponding to a different distance from the user. Such an arrangement may increase visual acuity and user comfort or reduce chromatic aberration.

[0065] In some embodiments, the light of each primary color may be output by a single dedicated waveguide, and as a result, each depth plane may have a plurality of waveguides associated therewith. In such embodiments, each box in the figure, including those containing the letters G, R, or B, may be understood to represent an individual waveguide, and three waveguides may be provided for each depth plane, with three primary color images being provided for each depth plane. The waveguides associated with each depth plane are shown adjacent to each other in this figure for ease of explanation, but it should be understood that in a physical device, all the waveguides may be arranged in a stack with one waveguide per level. In some other embodiments, a plurality of primary colors may be output by the same waveguide, e.g., such that only a single waveguide is provided for each depth plane.

[0066] Continuing to refer to FIG. 8, in some embodiments, G is green, R is red, and B is blue. In some other embodiments, other colors associated with other wavelengths of light, including magenta and cyan, may also be used in addition or may replace one or more of red, green, or blue. In some embodiments, features 320, 330, 340, and 350 may be active or passive optical filters configured to block or selectively transmit light from the surrounding environment to the viewer's eyes.

[0067] It is to be understood that references throughout this disclosure to the color of a given light include light of one or more wavelengths within the range of wavelengths of the light that is perceived by the viewer as that given color. For example, red light may include light of one or more wavelengths within the range of about 620 to 780 nm, green light may include light of one or more wavelengths within the range of about 492 to 577 nm, and blue light may include light of one or more wavelengths within the range of about 435 to 493 nm.

[0068] In some embodiments, light source 530 (FIG. 6) may be configured to emit light of one or more wavelengths outside the viewer's visual perception range, such as IR or ultraviolet wavelengths. IR light can include light with wavelengths in the range of 700 nm to 10 μm. In some embodiments, IR light can include near-IR light with wavelengths in the range of 700 nm to 1.5 μm. Additionally, the internal coupling, external coupling, and other light redirection structures of waveguide 250 of display 250 may be configured to direct and emit this light from the display towards user's eyes 210, for example, for imaging or user stimulation purposes.

[0069] Referring now to FIG. 9A, in some embodiments, light impinging on a waveguide may need to be redirected to internally couple the light into the waveguide. An internal coupling optical element may be used to redirect and internally couple the light into its corresponding waveguide. FIG. 9A illustrates a cross-sectional side view of an example of a plurality or set 660 of stacked waveguides, each including an internal coupling optical element. Each waveguide may be configured to output light of one or more different wavelengths or one or more different wavelength ranges. Stack 660 may correspond to stack 260 (FIG. 6), and the illustrated waveguides of stack 660 may correspond to a portion of the plurality of waveguides 270, 280, 290, 300, 310, but it should be understood that light from one or more of the image input devices 360, 370, 380, 390, 400 is input into the waveguide from a position where the light is required to be redirected for internal coupling.

[0070] The illustrated set 660 of stacked waveguides includes waveguides 670, 680, and 690. Each waveguide includes an associated internal coupling optical element (which may also be referred to as an optical input area on the waveguide). For example, internal coupling optical element 700 is disposed on a major surface (e.g., the upper major surface) of waveguide 670, internal coupling optical element 710 is disposed on a major surface (e.g., the upper major surface) of waveguide 680, and internal coupling optical element 720 is disposed on a major surface (e.g., the upper major surface) of waveguide 690. In some embodiments, one or more of the internal coupling optical elements 700, 710, 720 may be disposed on the bottom major surface of the respective waveguide 670, 680, 690 (in particular, one or more of the internal coupling optical elements are reflective deflecting optical elements). As illustrated, the internal coupling optical elements 700, 710, 720 may be disposed on the upper major surface of their respective waveguide 670, 680, 690 (or the upper portion of the next lower waveguide), and in particular, those internal coupling optical elements are transmissive deflecting optical elements. In some embodiments, the internal coupling optical elements 700, 710, 720 may be disposed within the body of the respective waveguide 670, 680, 690. In some embodiments, as discussed herein, the internal coupling optical elements 700, 710, 720 are wavelength selective such that they selectively redirect one or more wavelengths of light while transmitting other wavelengths of light. Although illustrated on one side or corner of their respective waveguide 670, 680, 690, it should be understood that the internal coupling optical elements 700, 710, 720 may be disposed within other areas of their respective waveguide 670, 680, 690 in some embodiments.

[0071] As shown, the internally coupled optical elements 700, 710, 720 may be laterally offset from each other. In some embodiments, each internally coupled optical element may be offset so that it receives light without the light passing through another internally coupled optical element. For example, each of the internally coupled optical elements 700, 710, 720 may be configured to receive light from different image input devices 360, 370, 380, 390, and 400, as shown in FIG. 6, and may be separated (e.g., laterally spaced) from the other internally coupled optical elements 700, 710, 720 so as to substantially not receive light from the other internally coupled optical elements 700, 710, 720.

[0072] Each waveguide also includes an associated light dispersing element. For example, the light dispersing element 730 is disposed on a major surface (e.g., the upper major surface) of the waveguide 670, the light dispersing element 740 is disposed on a major surface (e.g., the upper major surface) of the waveguide 680, and the light dispersing element 750 is disposed on a major surface (e.g., the upper major surface) of the waveguide 690. In some other embodiments, the light dispersing elements 730, 740, 750 may each be disposed on the bottom major surface of the associated waveguides 670, 680, 690. In some other embodiments, the light dispersing elements 730, 740, 750 may each be disposed on both the upper and bottom major surfaces of the associated waveguides 670, 680, 690, or the light dispersing elements 730, 740, 750 may each be disposed on different ones of the upper and bottom major surfaces within different associated waveguides 670, 680, 690.

[0073] Waveguides 670, 680, 690 may be separated and isolated, for example, by a gas, liquid, and / or solid layer of material. For example, as shown, layer 760a may separate waveguides 670 and 680, and layer 760b may separate waveguides 680 and 690. In some embodiments, layers 760a and 760b are formed from a low refractive index material (i.e., a material having a lower refractive index than the material forming the immediate waveguides 670, 680, 690). Preferably, the refractive index of the material forming layers 760a, 760b is 0.05 or more or 0.10 or less than the refractive index of the material forming waveguides 670, 680, 690. Advantageously, the lower refractive index layers 760a, 760b may function as cladding layers that promote TIR of light through waveguides 670, 680, 690 (e.g., TIR between the upper major surface and the bottom major surface of each waveguide). In some embodiments, layers 760a, 760b are formed from air. It should be understood that although not shown, the top and bottom of the illustrated set 660 of waveguides may include an immediate cladding layer.

[0074] Preferably, for ease of manufacture and other considerations, the materials forming waveguides 670, 680, 690 are similar or identical, and the materials forming layers 760a, 760b are similar or identical. In some embodiments, the materials forming waveguides 670, 680, 690 may be different between one or more waveguides, or the materials forming layers 760a, 760b may still be different while maintaining the various refractive index relationships described above.

[0075] Continuing to refer to FIG. 9A, light rays 770, 780, 790 are incident on the set 660 of waveguides. It should be understood that light rays 770, 780, 790 may be input into waveguides 670, 680, 690 by one or more image input devices 360, 370, 380, 390, 400 (FIG. 6).

[0076] In some embodiments, the light rays 770, 780, 790 may have different properties, such as different wavelengths or different wavelength ranges, corresponding to different colors. The internal coupling optical elements 700, 710, 720 each deflect the incident light so that the light propagates through an individual one of the waveguides 670, 680, 690 by TIR.

[0077] For example, the internal coupling optical element 700 may be configured to deflect the light ray 770 having a first wavelength or wavelength range. Similarly, the transmitted light ray 780 impinges on and is deflected by the internal coupling optical element 710 configured to deflect light of a second wavelength or wavelength range. Similarly, the light ray 790 is deflected by the internal coupling optical element 720 configured to selectively deflect light of a third wavelength or wavelength range.

[0078] Continuing to refer to FIG. 9A, the deflected light rays 770, 780, 790 are deflected to propagate through the corresponding waveguides 670, 680, 690. That is, the internal coupling optical elements 700, 710, 720 of each waveguide deflect the light into its corresponding waveguide 670, 680, 690 and internally couple the light into the corresponding waveguide. The light rays 770, 780, 790 are deflected at an angle that causes the light to propagate through the individual waveguides 670, 680, 690 by TIR. The light rays 770, 780, 790 propagate through the individual waveguides 670, 680, 690 by TIR until they impinge on the corresponding light dispersion elements 730, 740, 750 of the waveguide.

[0079] Referring now to FIG. 9B, a perspective view of an embodiment of the plurality of stacked waveguides of FIG. 9A is illustrated. As described above, the internally coupled light rays 770, 780, 790 are each deflected by the internally coupled optical elements 700, 710, 720 and then each propagate by TIR within waveguides 670, 680, 690, respectively. The light rays 770, 780, 790 then each impinge on light dispersing elements 730, 740, 750. The light dispersing elements 730, 740, 750 each deflect the light rays 770, 780, 790 so as to propagate towards external coupling optical elements 800, 810, 820, respectively.

[0080] In some embodiments, the light dispersing elements 730, 740, 750 are orthogonal pupil expanders (OPEs). In some embodiments, the OPE may both deflect or disperse light towards the external coupling optical elements 800, 810, 820 and increase the beam or spot size of the present light as it propagates towards the external coupling optical elements. In some embodiments, for example, if the beam size is already of a desired size, the light dispersing elements 730, 740, 750 may be omitted and the internally coupled optical elements 700, 710, 720 may be configured to deflect light directly towards the external coupling optical elements 800, 810, 820. For example, referring to FIG. 9A, in some embodiments, the light dispersing elements 730, 740, 750 may each be replaced by the external coupling optical elements 800, 810, 820. In some embodiments, the external coupling optical elements 800, 810, 820 are an exit pupil (EP) or an exit pupil expander (EPE) that direct light towards the viewer's eye 210 (FIG. 7). It should be understood that the OPE may be configured to increase the dimensions of the eyebox along at least one axis and the EPE may increase the eyebox along an axis that intersects (e.g., is orthogonal to) the axis of the OPE.

[0081] Thus, referring to FIGS. 9A and 9B, in some embodiments, a waveguide set 660 includes, for each primary color, waveguides 670, 680, 690, internal coupling optical elements 700, 710, 720, light dispersion elements (e.g., OPE) 730, 740, 750, and external coupling optical elements (e.g., EP) 800, 810, 820. The waveguides 670, 680, 690 may be stacked with a gap / cladding layer between each one. The internal coupling optical elements 700, 710, 720 redirect or deflect incident light into their respective waveguides (using different internal coupling optical elements that receive light of different wavelengths). The light then propagates at an angle that will result in TIR within the individual waveguides 670, 680, 690. In the example shown, a light ray 770 (e.g., blue light) is polarized by a first internal coupling optical element 700 in the manner described above and then bounces through the waveguide, interacting with a light dispersion element (e.g., OPE) 730 and then an external coupling optical element (e.g., EP) 800. Light rays 780 and 790 (e.g., green and red light, respectively) pass through waveguide 670. Light ray 780 strikes the internal coupling optical element 710 and is thereby deflected. Light ray 780 then bounces through waveguide 680 via TIR and will proceed to its light dispersion element (e.g., OPE) 740 and then an external coupling optical element (e.g., EP) 810. Finally, light ray 790 (e.g., red light) passes through waveguide 690 and strikes the internal coupling optical element 720 of waveguide 690. The internal coupling optical element 720 deflects light ray 790 such that the light ray propagates by TIR to a light dispersion element (e.g., OPE) 750 and then by TIR to an external coupling optical element (e.g., EP) 820. The external coupling optical element 820 then finally externally couples light ray 790 to the viewer, who also receives externally coupled light from the other waveguides 670, 680.

[0082] FIG. 9C illustrates top and bottom plan views of an example of a plurality of stacked waveguides of FIGS. 9A and 9B. As shown, waveguides 670, 680, 690 may be vertically aligned with associated optical dispersion elements 730, 740, 750 and associated external coupling optical elements 800, 810, 820 of each waveguide. However, as discussed herein, internal coupling optical elements 700, 710, 720 are not vertically aligned. Rather, the internal coupling optical elements are preferably non-overlapping (e.g., laterally spaced as seen in the top and bottom views). As further discussed herein, this non-overlapping spatial arrangement facilitates the injection of light from different resources into different waveguides on a one-to-one basis, thereby enabling a particular light source to be uniquely coupled to a particular waveguide. In some embodiments, an array that includes non-overlapping, spatially separated internal coupling optical elements may be referred to as an off-axis pupil system, and the internal coupling optical elements within these arrays may correspond to sub-pupils. Exemplary imaging system for off-axis imaging

[0083] As described above, the eye or the tissue around the eye of a wearer of an HMD (e.g., the wearable display system 200 shown in FIG. 2) can be imaged using a plurality of coupling optical elements that direct light from the eye through the substrate into the camera assembly. The resulting image can be used for, e.g., monocular or binocular eye tracking, retinal imaging, three-dimensional reconstruction of the eye shape, extraction of biometric information from the eye (e.g., iris identification), etc.

[0084] As outlined above, there are various reasons why an HMD may use information about the wearer's eye condition. For example, this information can be used for estimating the wearer's line of sight direction or biometric identification. However, this problem is difficult because the distance between the HMD and the wearer's eye is short. Gaze tracking is more complicated due to the fact that it requires a larger field of view, while biometric identification requires a relatively high pixel count on a target on the iris. For an imaging system that will attempt to perform both of these purposes, the requirements of the two tasks mainly conflict. Finally, both problems are further complicated by occlusion by eyelids and eyelashes. Embodiments of the imaging system described herein can address at least some of these problems.

[0085] FIGS. 10A and 10B schematically illustrate an example of an imaging system 1000a configured to image one or both eyes 210, 220 of a wearer 90. The imaging system 1000a includes a substrate 1070 and a camera assembly 1030 arranged to view the eye 220. The embodiments of the imaging system 1000a described herein with reference to FIGS. 10A and 10B can be used in combination with an HMD that includes a display device (e.g., the wearable display system 200 shown in FIG. 2, the display system 250 shown in FIGS. 6 and 7, and the stack 660 of FIGS. 9A-9C) described herein. For example, in some implementations where the imaging system 1000a is part of the display system 250 of FIG. 6, the substrate 1070 may replace one of the waveguides 270, 280, 290, 300, or 310, may be disposed between the waveguide stack 260 and the eye 210, or may be disposed between the waveguide stack 260 and the world 510.

[0086] In some embodiments, the camera assembly 1030 is mounted proximate to the wearer's eye, such as on the frame 80 of the wearable display system 60 of FIG. 2 (e.g., the earpiece 82 near the wearer's temple), around the edge of the display 70 of FIG. 2 (as shown in FIG. 10B), or incorporated within the display 70 of FIG. 2. The camera assembly 1030 may be substantially similar to the camera assembly 630 of FIG. 6. In other embodiments, a second camera assembly can be used to separately image the wearer's other eye 210. The camera assembly 1030 can include an IR digital camera sensitive to IR radiation. The camera assembly 1030 can be mounted facing forward (e.g., in the direction towards the wearer's vision) as shown in FIG. 10A, or the camera assembly 1030 can be mounted facing rearward and directed towards the eye 220 (e.g., FIG. 10B).

[0087] In some embodiments, the camera assembly 1030 may include an image capture device and a light source 1032 that projects light onto the eye 220 and that can then be reflected by the eye 220 and detected by the camera assembly 1030. Although the light source 1032 is shown as being attached to the camera assembly 1030, the light source 1032 may be disposed within another area relative to the camera assembly such that the light emitted by the light source is directed towards the wearer's eye and reflected by the camera assembly 1030. For example, if the imaging system 1000a is part of the display system 250 (FIG. 6) and the substrate 1070 replaces one of the waveguides 270, 280, 290, 300, or 310, the light source 1032 may be one of the light emitters 360, 370, 380, 390, or the light source 530.

[0088] In the embodiment illustrated in FIG. 10A, the camera assembly 1030 is positioned to view the proximal surface 1074 of the substrate 1070. The substrate 1070 can be, for example, part of the display 70 of FIG. 2 or a lens within a pair of glasses. The substrate 1070 can be transmissive to at least 10%, 20%, 30%, 40%, 50%, or more of the visible light incident on the substrate 1070. In other embodiments, the substrate 1070 need not be transparent (e.g., in a virtual reality display). The substrate 1070 can include one or more coupling optical elements 1078. In some embodiments, the coupling optical element 1078 can be selected to reflect a first range of wavelengths while being substantially transmissive to a second range of wavelengths different from the first range of wavelengths. In some embodiments, the first range of wavelengths can be an IR wavelength and the second range of wavelengths can be a visible wavelength. The substrate 1070 can include a polymer or plastic material such as polycarbonate or other lightweight materials having desired optical properties. Without subscribing to any particular scientific theory, plastic materials are less rigid and thus may be less likely to suffer breakage or defects during use. Plastic materials can also be lightweight and thus, when combined with the rigidity of the plastic material, can allow for thinner substrates and facilitate the manufacture of a compact and lightweight imaging system. Although the substrate 1070 is described as including a polymer such as polycarbonate or other plastics having desired optical properties, other materials such as glass having desired optical properties, e.g., fused silica, are also possible considerations.

[0089] The coupling optical element 1078 can comprise a reflective optical element configured to reflect or redirect light in a first wavelength range (e.g., IR light) while transmitting light in a second wavelength range (e.g., visible light). In such embodiments, IR light 1010a, 1012a, and 1014a from the eye 220 propagates to the coupling optical element 1078 and is reflected therefrom, resulting in reflected IR light 1010b, 1012b, 1014b, which can be imaged by the camera assembly 1030. In some embodiments, the camera assembly 1030 can be sensitive to, or capable of capturing, at least a subset (such as a non-empty subset or a subset less than all) of the first wavelength range reflected by the coupling optical element 1078. For example, if the coupling optical element 1078 is a reflective element, the coupling optical element 1078 can reflect IR light in the range of 700 nm to 1.5 μm, and the camera assembly 1030 can be sensitive to, or capable of capturing, near-IR light at wavelengths in the range of 700 nm to 900 nm. As another example, the coupling optical element 1078 can reflect IR light in the range of 700 nm to 1.5 μm, and the camera assembly 1030 can include a filter that filters out IR light in the range of 900 nm to 1.5 μm such that the camera assembly 1030 can capture near-IR light at wavelengths in the range of 700 nm to 900 nm.

[0090] Visible light from the outside world (e.g., the world 510 in FIG. 6) can be transmitted through the substrate 1070 and can be perceived by the wearer. In fact, the imaging system 1000a can act as if there is a virtual camera assembly 1030c that is retro-directed towards the wearer's eye 220 and captures a direct-view image of the eye 220. The virtual camera assembly 1030c is labeled with "c" because it can image virtual IR light 1010c, 1012c, and 1014c (shown as dotted lines) that propagates from the wearer's eye 220 through the substrate 1070. The coupling optical element 1078 is shown as being disposed on the proximal surface 1074 of the substrate 1070, but other configurations are also possible. For example, the coupling optical element 1078 can be disposed on the distal surface 1076 of the substrate 1060 or within the substrate 1070. In an implementation where the substrate 1070 is part of the display system 250 of FIG. 6, the coupling optical element 1078 can be an external coupling optical element 570, 580, 590, 600, or 610.

[0091] An exemplary arrangement of the imaging system 1000a is shown in FIG. 10A, but other arrangements are also possible. For example, a plurality of coupling optical elements can be used and configured to internally couple light into the substrate 1070 via TIR and externally couple the light to the camera assembly 1030, as will be described in connection with FIGS. 11 - 18. The coupling optical element 1078 is described as a reflective optical element, but other configurations are also possible. For example, the coupling optical element 1078 can be a transmissive coupling optical element that substantially transmits the first and second wavelength ranges. The transmissive coupling optical element can refract the first wavelength at an angle, for example, inducing TIR within the substrate 1070 while allowing the second wavelength range to pass through substantially unobstructed. Exemplary imaging system for off-axis imaging using a plurality of coupling optical elements

[0092] FIG. 11 schematically illustrates another exemplary imaging system 1000b that includes a plurality of coupling optical elements for total internally reflecting light from an object through a substrate 1070 and imaging the object at a camera assembly 1030. FIG. 11 illustrates an embodiment of an imaging system 1000b that includes a substrate 1070 having at least two coupling optical elements 1178a, 1188a disposed on one or more surfaces of the substrate 1070, and a camera assembly 1030 arranged to view an object positioned in an object plane 1120. Although a specific arrangement is depicted in FIG. 11, this is for illustrative purposes only and is not intended to be limiting. Other optical elements (e.g., lenses, waveguides, polarizers, prisms, etc.) may also be used to manipulate light from the object, such as focusing the light, correcting aberrations, directing the light, etc., as desired for a particular application.

[0093] In the embodiment of FIG. 11, the substrate 1070 includes two coupling optical elements 1178a, 1188a, each disposed adjacent to a distal surface and a proximal surface 1076, 1074 of the substrate 1070, respectively. In some embodiments, the coupling optical elements 1178a, 1188a may be attached or fixed to the surface of the substrate 1070. In other embodiments, one or more of the coupling optical elements 1178a, 1188a may be built into the substrate 1070 or etched on the surface of the substrate 1070. In still other embodiments, the substrate 1070 may be manufactured to have regions that include the coupling optical elements 1178a, 1188a as part of the substrate 1070 itself, either alone or in combination. An exemplary arrangement of the coupling optical elements 1178a, 1188a is shown in FIG. 11, but other configurations are also conceivable. For example, both coupling optical elements 1178a, 1188a may be positioned adjacent to the distal surface 1076 or the proximal surface 1074 (as shown in FIGS. 12A, 13A, 13B, and 14B), or the coupling optical element 1178a may be positioned on the proximal surface 1074 while the coupling optical element 1188a may be positioned on the distal surface 1076 (as shown in FIG. 14A).

[0094] The coupling optical elements 1178a and 1188a may be similar to the coupling optical element 1078 of FIGS. 10A and 10B. For example, FIG. 11 illustrates an imaging system 1000b, and both coupling optical elements 1178a, 1188a are reflective coupling optical elements that are wavelength selective such that they selectively redirect light of one or more wavelengths while transmitting light of other wavelengths, as described above in connection with FIG. 10A. In some embodiments, the coupling optical elements 1178a and 1188a deflect light in a first wavelength range (e.g., IR light, near-IR light, etc.) while transmitting light in a second wavelength range (e.g., visible light). As described below, the coupling optical elements 1178a, 1188a may comprise diffractive features (e.g., DOE) that form a diffraction pattern.

[0095] Referring to FIG. 11, the camera assembly 1030 is mounted facing rearwardly towards the object plane 1120 and viewing the distal surface 1076. In various embodiments, the camera assembly 1030 may be mounted proximate to the wearer's eye (e.g., on the frame 80 of FIG. 2) and may include a light source 1032 (not shown in FIG. 11). The camera assembly 1030 can include an IR digital camera sensitive to IR radiation. Although the camera assembly 1030 of FIG. 11 is shown facing rearwardly, other arrangements are also possible. For example, the camera assembly 1030 can be mounted facing forwardly.

[0096] In some embodiments, an object (e.g., the eye 220 or a part thereof) in the object plane 1120 may be illuminated by a light source 1032 (Figs. 10A and 10B). For example, when the pupil is to be imaged, the light source 1032 is directed thereto to illuminate the pupil of the eye 220. In other embodiments, a first Purkinje image, which is a virtual image formed by reflection of a point source from the anterior surface of the cornea, may be imaged. Any physical or optical object associated with the eye that can be uniquely identified and that would indicate the eye position, pupil position, or line of sight direction may be imaged. Depending on the illumination, the object may reflect light towards the substrate 1070 as light rays 1122a-e (collectively hereinafter referred to as "1122"). For example, the light rays 1122a-e may be illustrative of the diffusion of light reflected from the pupil, iris, eyelid, sclera, other tissues around the eye, etc. In another example, the light rays 1122a-e may be illustrative of specularly reflected light from a flash (e.g., a Purkinje image). Without subscribing to a scientific theory, reflection from the eye, a part of the eye, or tissues around the eye may rotate the polarization of the incident light depending on the orientation of the illumination. In some embodiments, the light source 1032 (Figs. 10A and 10B) may be an LED light source that does not have a specific polarization unless a polarizer is implemented in the optical path, which may reduce the light intensity by, for example, up to 50%. Only the light rays 1122 are shown in Fig. 11, which is for illustrative purposes only, and any number of reflected light rays are also possible. Each of the light rays 1122 may be reflected from the object at the same or different angles. For example, Fig. 11 illustrates that the light ray 1122a may be reflected at a first angle that may be greater than the angle at which the light ray 1122e is reflected from the object. Other configurations are also possible.

[0097] The above description refers to the light ray 1122 reflected from the object, but other configurations are also conceivable as possibilities. In some embodiments, the light ray 1122 is emitted by a light source located in the object plane 1120 instead of reflecting the light from the source 1032 (FIGS. 10A and 10B). Thus, the light ray 1122 can be directed towards the substrate 1070. It should be understood that the light ray 1122 may be all or part of the light reflected from or emitted thereby from the object plane 1120.

[0098] As shown in FIG. 11, in response to the emission from the object plane 1120, the light ray 1122 is incident on the proximal surface 1074 of the substrate at an incident angle with respect to an imaginary axis perpendicular to the proximal surface 1074 at the point of incidence. The light ray 1122 then enters the substrate 1070 and is refracted, at least in part, based on the ratio of the incident angle at the proximal surface 1074 and the refractive index of the medium immediately adjacent to the proximal surface 1074 of the substrate 1070.

[0099] The light ray 1122 travels and impinges on the coupling optical element 1178a at an incident angle with respect to an imaginary axis perpendicular to the distal surface 1076 at the point of incidence. The light ray 1122 is deflected by the coupling optical element 1178a so as to propagate through the substrate 1070. That is, the coupling optical element 1178a functions as a reflective internal coupling optical element that reflects the light into the substrate 1070. The light ray 1122 is reflected at an angle such that the internally coupled light ray 1122 propagates laterally through the substrate towards the coupling optical element 1188a by total internal reflection. Without subscribing to any scientific theory, the total internal reflection condition can be satisfied when the diffraction angle θ between the incident light and the perpendicular axis exceeds the critical angle θ C of the substrate 1070. In some situations, the total internal reflection condition can be expressed as follows. sin(θ C ) = n o / n s [1] where n s is the refractive index of the substrate 1070, and n ois the refractive index of the medium adjacent to the surface substrate 1070. According to various embodiments, n s may be in the range of about 1 to about 2, about 1.4 to about 1.8, about 1.5 to about 1.7, or other suitable ranges. For example, the substrate 1070 may include a polymer such as polycarbonate or glass (e.g., fused silica, etc.). In some embodiments, the substrate 1070 may be 1 to 2 millimeters thick from the proximal surface 1074 to the distal surface 1076. For example, the substrate 1070 may be a 2 - millimeter - thick portion of fused silica or a 1 - millimeter - thick portion of polycarbonate. Other configurations for achieving the desired operation and image quality in the camera assembly 1030 are also possible considerations.

[0100] In some embodiments, the substrate 1070 may be formed from a high - refractive - index material (e.g., a material having a higher refractive index than the medium directly adjacent to the substrate 1070). For example, the refractive index of the material directly adjacent to the substrate 1070 may be 0.05 or more, or 0.10 or more, lower than the substrate refractive index. Without subscribing to a particular scientific theory, the lower - refractive - index medium may function to promote TIR of light through the substrate 1070 (e.g., TIR between the proximal and distal surfaces 1074, 1076 of the substrate 1070). In some embodiments, the directly adjacent medium includes air with a refractive index n o of about 1. The critical angle can be in the range of 20 degrees to 50 degrees, depending on the substrate material and the surrounding medium. In other embodiments, alone or in combination, the directly adjacent medium may include other structures and layers. For example, one or more of the layers described in connection with FIGS. 6 and 9A - 9C may be directly adjacent to either the proximal surface or the distal surface 1074, 1076 of the substrate 1070.

[0101] The light then propagates through the substrate 1070 in a direction substantially parallel to the surface of the substrate 1070 towards the coupling optical element 1188a. Generally, "towards" can refer to a state where the light ray 1122 is reflected between the surfaces of the substrate 1070 and thus may not be exactly parallel to the substrate 1070, but the overall propagation direction is in a direction substantially parallel to the surface of the substrate. The light ray 1122 propagates through the substrate 1070 by TIR until it impinges on the coupling optical element 1188a. Upon reaching the coupling optical element 1188a, the light ray 1122 is deflected to propagate out of the substrate 1070. That is, the coupling optical element 1188a functions as a reflective external coupling optical element that reflects light out of the substrate 1070. The light ray 1120 is reflected at an angle such that the TIR condition is no longer satisfied (e.g., the diffraction angle θ is less than the critical angle θ C ). The coupling optical element 1188a may also reflect the light ray 1122 at an angle towards the camera assembly 1030. For example, the light ray 1122 may be reflected at an angle so as to exit the substrate 1070, refracted by the interface at the distal surface 1076, and propagate to the camera assembly 1030. The camera assembly 1030 then receives the light ray 1122 and forms an image of the object plane 1120 based thereon.

[0102] FIG. 11 illustrates a configuration in which light travels from coupling optical element 1178a to coupling optical element 1188a with two instances of total internal reflection, although other configurations are also conceivable. For example, light ray 1122 may be totally internally reflected any number of times (e.g., 1, 2, 3, 4, 5, 6, 7, etc.) such that light ray 1122 travels through substrate 1070 towards camera assembly 1030. Camera assembly 1030 may thus be positioned anywhere and configured to capture a direct view image at a distance from the object. Without subscribing to scientific theory, TIR can be highly efficient and include reflections that are substantially lossless, and thus the number of times light ray 1122 undergoes TIR may be selected based on the desired position of the camera. However, in some embodiments, some leakage may occur, albeit minimally, at each reflection within substrate 1070. Thus, minimizing the number of reflections within substrate 1070 can reduce light leakage and improve image capture performance. Further, without subscribing to scientific theory, reducing the number of reflections can improve image quality by reducing image blur or brightness degradation due to impurities or non-uniform surfaces in substrate 1070 (e.g., fewer reflections can produce a brighter, more intense image). Thus, the design of the imaging system and its components described can be optimized with these considerations in mind to minimize the number of TIR events and position camera assembly 1030 as desired.

[0103] Efficient internal and external coupling of light into the substrate 1070 can be an issue, for example, in designing waveguide-based see-through displays for virtual / augmented / composite reality display applications. For these and other applications, it may be desirable for the structure to include a diffraction grating formed from a material that can be configured to optimize various optical properties, including diffractive properties. Desirable diffractive properties may include, among other properties, polarization selectivity, spectral selectivity, angular selectivity, high spectral bandwidth, and high diffraction efficiency. To address these and other requirements, in various embodiments disclosed herein, the coupling optical elements 1178a, 1188a may comprise diffractive features that form a diffractive pattern, such as a DOE or a diffraction grating.

[0104] Generally, a diffraction grating has a periodic structure, which splits and diffracts light into several beams traveling in different directions. The direction of the beams depends, inter alia, on the period of the periodic structure and the wavelength of the light. The period can be based, in part, on the grating spatial frequency of the diffraction features. For certain applications, such as internally and externally coupling light from substrate 1070 to optimize certain optical properties, e.g., diffraction efficiency, and reduce potential rainbow effects, the various material properties of the DOE can be optimized for a given wavelength. For example, when IR light is used, the spatial frequencies of DOE1178a, 1188a can be 600 - 2000 lines per millimeter. In one embodiment, the spatial frequency can be about 1013 lines per millimeter (e.g., FIGS. 12A and 13A). In one embodiment, the exemplary DOE1178a of FIG. 11 can have 1013.95 lines per millimeter. In another embodiment, the spatial frequency is about 1400 lines per millimeter as described in connection with FIG. 15. Thus, the spatial frequencies of the coupling optical elements 1178a, 1188a can be at least one consideration when optimizing the imaging system described herein. For example, the spatial frequency can be selected to assist with TIR conditions. As another example, alone or in combination, the spatial frequency can be selected to maximize light throughput with minimal artifacts (e.g., ghosting or overlapping images as described in FIG. 12B) depending on the configuration and dimensions of the components of the imaging system. In some embodiments, the diffraction features can have any configuration. However, the first coupling optical element 1178a may be optimized to have minimal artifacts or no visual artifacts (e.g., rainbow effect) since the first coupling optical element 1178a can be positioned within the user's field of view.

[0105] In some implementations, the DOE may be an off-axis DOE, an off-axis holographic optical element (HOE), an off-axis holographic mirror (OAHM), or an off-axis volumetric diffractive optical element (OAVDOE). In some embodiments, the OAHM may also have refractive power and, in that case, can be an off-axis volumetric diffractive optical element (OAVDOE). In some embodiments, one or more of the coupling optical elements 1178a, 1188a may be an off-axis cholesteric liquid crystal diffraction grating (OACLCG), which can be configured, inter alia, to optimize polarization selectivity, bandwidth, phase profile, spatial variation of diffractive properties, spectral selectivity, and high diffraction efficiency. For example, either a CLC or a CLCG, as described in U.S. Patent Application No. 15 / 835,108, filed December 7, 2017, entitled "Diffractive Devices Based On Cholesteric Liquid Crystal" (which is incorporated herein by reference in its entirety for all that it discloses), can be implemented as a coupling optical element as described herein. In some embodiments, one or more of the coupling optical elements 1178a, 1188a may be a switchable DOE that can be switched between an "on" state where it actively diffracts and an "off" state where it does not significantly diffract.

[0106] In some embodiments, one or more of the coupling optical elements 1178a, 1188a may be any reflective or transmissive liquid crystal grating. The CLC or CLCG described above can be an example of a liquid crystal grating. Other liquid crystal gratings may also have dimensions less than the wavelength of visible light and may include liquid crystal features and / or patterns with a Pancharatnam-Berry phase effect (PBPE) structure, a metasurface, or what is referred to as a metamaterial. For example, "Display System With Optical Elements" U.S. Patent Publication No. 2017 / 0010466, filed on January 24, 2018 and entitled "For In-Coupling Multiplexed Light Streams", U.S. Patent Application No. 15 / 879,005, filed on December 13, 2017 and entitled "Antireflection Coatings For Metasurfaces", or U.S. Patent Application No. 15 / 841,037, filed on December 13, 2017 and entitled "Patterning Of Liquid Crystals Using Soft-Imprint Replication Of Surface Alignment Patterns" (each of which is hereby incorporated by reference in its entirety for all that it discloses) may be implemented as a coupling optical element as described herein. Such a structure may be configured to manipulate light, such as beam steering, wavefront shaping, wavelength separation, and / or polarization, and combining different wavelengths and / or polarizations may, alternatively, include a liquid crystal lattice with a metasurface, referred to as a liquid crystal lattice with a metasurface liquid crystal lattice or PBPE structure with a liquid crystal lattice. The liquid crystal lattice with a PBPE structure can combine high diffraction efficiency and low sensitivity to the incident angle of the liquid crystal lattice with high wavelength sensitivity of the PBPE structure.

[0107] In some embodiments, a certain DOE, when utilized as a coupling optical element as described herein, may provide non-limiting advantages. For example, without subscribing to a scientific theory, liquid crystal gratings, CLCs, CLCGs, volume phase gratings, and metasurface gratings may have optical properties configured to reduce or eliminate the manifestation of visual artifacts such as the rainbow effect described above and herein. In some embodiments, when employing these DOEs, it may be desirable to illuminate the DOE with polarized light (e.g., light ray 1122 may include a desired polarization) to maximize the throughput of light into substrate 1070. However, as described above, the eye can rotate the incident polarization depending on the orientation, and thus, in some embodiments, light source 1030 may emit non-polarized light. The reflected light ray 1122 may also not be polarized, and thus, a portion of the light may not be throughput due to the polarization properties of the DOE (e.g., up to 50% of light ray 1122 may be lost in coupling optical element 1178a). In some embodiments, a double-layer DOE may be used as coupling optical element 1178a to improve throughput. For example, a first DOE layer may be configured to operate in one polarization state and a second DOE layer may be configured to operate in a second polarization state.

[0108] Regarding some embodiments, it may be desirable to use a DOE such that many of the light rays 1122 are internally coupled within the substrate 1070 and externally coupled towards the camera assembly, having a sufficiently high diffraction efficiency. Without subscribing to scientific theory, a relatively high diffraction efficiency may direct substantially all of the light received at the coupling optical element 1178a towards the camera assembly 1030, thereby making it possible to improve image quality and accuracy. In some embodiments, the diffraction efficiency may be based, in part, on the sensitivity of the camera assembly 1030 (e.g., higher sensitivity may allow for lower diffraction efficiency). In various embodiments, the DOE may be selected to have a high diffraction efficiency for a first range of wavelengths (e.g., IR light) and a low diffraction efficiency within a second range of wavelengths (e.g., visible light). Without subscribing to scientific theory, a low diffraction efficiency for visible light may reduce the rainbow effect within the user's line of sight.

[0109] In some applications, the DOE may cause a rainbow effect when the user views visible light through the diffraction feature. Without subscribing to a particular scientific theory, the rainbow effect may be the result of a range of wavelengths interacting with the diffraction feature and thereby deflecting different wavelengths (e.g., colors) in different directions at different diffraction angles. In some embodiments described herein, the rainbow effect from the world interacting with the coupling optical elements 1178a, 1188b as viewed by the user may be reduced by modifying or controlling the diffraction feature to reduce this effect. For example, since the diffraction angle of light on the DOE is based on the period or spatial frequency of the grating, the shape of the diffraction feature may be selected to concentrate most of the diffracted light in a particular location for a given range of wavelengths (e.g., triangular cross-section or blazed).

[0110] In some embodiments, the substrate 1070 may be one of the waveguides 270, 280, 290, 300, or 310 of FIG. 6. In this embodiment, the corresponding external coupling optical element 570, 580, 590, 600, or 610 may be replaced by an internal coupling optical element 1178a configured to induce TIR of the light reflected by the eye. In some embodiments, a portion of the external coupling optical element 570, 580, 590, 600, or 610 may be replaced by an internal coupling optical element 1178 such that the corresponding waveguide 270, 280, 290, 300, or 310 can be used to direct the reflected light to the camera assembly 630 as described in connection with FIG. 6.

[0111] In some embodiments, the substrate 1070 may be one of the waveguides 670, 680, or 690 of FIGS. 9A-9C. In these embodiments, the corresponding light dispersion elements 800, 810, and 830, or a portion thereof, may be replaced by an internal coupling optical element 1178a, while the internal coupling optical elements 700, 710, and 720, or a portion thereof, may be replaced by an external coupling optical element 1188a. In some embodiments, the OPEs 730, 740, and 750 may remain within the optical path of the light traveling from the internal coupling optical element 1178a to the external coupling optical element 1188a. However, the OPEs 730, 740, and 750 may be configured to disperse the light to the external coupling optical element 1188a and reduce the beam spot size as the light propagates.

[0112] In various embodiments, the field of view of the camera assembly 1030 is configured to be sufficient to image the entire object plane 1120 (e.g., the eye 220 of FIG. 10, a portion thereof, or the tissue surrounding the eye) throughout various field of view positions. For example, in the embodiment shown in FIG. 11, the size of the object plane 1120 to be imaged may be 30 mm (horizontal) × 16 mm (vertical). In some embodiments, the coupling optical elements 1178a, 1188a are designed to be large enough to at least match the size of the object to be imaged, i.e., the coupling optical elements 1178a, 1188a are configured to receive light from the entire size of the object to be imaged. For example, the coupling optical element 1178a receives light originating from the eye 220. The coupling optical element 1188 may be sized to reflect substantially all of the light rays 1122 that propagate through the substrate 1070 towards the camera assembly 1030.

[0113] In various embodiments, other optical elements may be positioned along the path that the light rays 1122 travel. For example, an intervening optical element may be included between the substrate 1070 and the object plane 1120 to direct the light rays 1122 towards the substrate 1070 at a desired angle. The intervening optical element may be included between the camera assembly 1030 and the substrate 1070 to position the camera assembly 1030 at any desired location and direct and focus the light rays 1122 towards the camera assembly 1030. In some embodiments, the intervening optical element may be used to filter the light rays 1122, change the polarization, or correct for aberrations. For example, a correction optical element may be arranged and configured to reduce or eliminate optical aberrations introduced by the optical components of the imaging system, along the optical path of the light rays 1122, or positioned if the imaging system is part of the display system 250 of FIG. 6, other waveguides, or an optical element. Alternative embodiments for off-axis imaging using multiple coupling optical elements

[0114] FIG. 11 shows an exemplary imaging system 1000b comprising a substrate 1070 having coupling optical elements 1178a, 1188a configured to TIR light from an object plane 1120 through the substrate 1070, although other configurations are also conceivable. For example, FIG. 11 illustrates both coupling optical elements 1178a, 1188a as reflective coupling optical elements. However, one or both of the coupling optical elements may be transmissive coupling optical elements configured to refract a first wavelength range at an angle that satisfies TIR conditions while substantially transmitting a second wavelength range through the substrate 1070. FIGS. 12A - 18 illustrate some embodiments of the substrate 1070, although other configurations are also conceivable.

[0115] FIG. 12A schematically illustrates an exemplary imaging system 1000c. The imaging system 1000c uses a plurality of coupling optical elements 1178a and 1188b to TIR light 1122 from an object plane 1220 through the substrate 1070. Similar to FIG. 11, FIG. 12A illustrates the coupling optical element 1178a as a reflective coupling optical element disposed on the distal surface 1076 of the substrate 1070 that internally couples the light ray 1122 into the substrate 1070. However, the coupling optical element 1188b is substantially similar to the coupling optical element 1188a of FIG. 11, and FIG. 12A illustrates a transmissive coupling optical element 1188b disposed on the distal surface 1076 of the substrate 1070. Thus, in response to propagating through the substrate 1070 via TIR, the light ray 1122 is reflected for the third time towards the transmissive coupling optical element 1188b on the proximal surface 1074. The transmissive coupling optical element 1188b refracts the light ray 1122 at an angle such that the TIR condition no longer persists and the light ray 1122 exits the substrate 1070. For example, if the transmissive coupling optical element 1188b is a DOE, the light is refracted based on the spatial frequency of the DOE and is substantially deflected towards the camera assembly 1030.

[0116] FIG. 12A also illustrates stray light rays 1222 captured by camera assembly 1030. For example, although the stray light rays 1222 are reflected by object 1120, instead of propagating through coupling optical elements 1178a, 1188b, some or all of the stray light rays 1222 travel directly towards camera assembly 1030. Without subscribing to any particular theory, the stray light rays 1222 are captured by camera assembly 1030, thereby generating a direct view image as described above. Thus, camera assembly 1030 can capture a direct view image (e.g., including the narrow FOV and defects described herein) based on light rays 1222, along with a desired image based on light rays 1122 that undergo TIR through the substrate. Since camera assembly 1030 captures light rays that have traveled along different optical paths, the final image will include various imperfections. One such imperfection is illustrated in FIG. 12B, among others, which are also possible.

[0117] FIG. 12B illustrates an exemplary image 1210 of object 1120 captured using camera assembly 1030 of FIG. 12A. In the illustrative image 1210, camera assembly 1030 captured an image 1210 of the front of a laser diode that was used as an object, for example, and illuminated with an IR light source. Although the laser diode is illustrated in this embodiment, other objects, such as a user's eye 210, may also be used for similar effects. Image 1210 includes a direct view image 1205 of the laser diode generated by light rays 1222 and a set of images 1240 generated by light rays 1122. The set of images 1240 includes a desired off-axis image (shown as image 1215 for illustrative purposes) and a plurality of overlapping images from different viewpoints (collectively illustrated as image 1212). In some embodiments, such overlapping images 1212 may require post-processing to synthesize a single viewpoint image of the object, if desired. In other embodiments, the imaging system may be designed to reduce or eliminate unwanted overlapping images 1212 and direct view image 1205 so as to capture a single viewpoint image 1215.

[0118] For example, FIGS. 13A and B schematically illustrate another view of the imaging system 1000c. FIGS. 13A and 13B illustrate an exemplary approach for reducing or eliminating the overlapping image 1212. Without subscribing to a particular scientific theory, the overlapping image 1212 can be reduced or substantially eliminated based on varying the thickness (t) of the substrate 1070, the size (d1) of the coupling optical element 1178a, and the stride distance (d2) of the light ray 1122. The stride distance (d2) can refer to the distance parallel to the substrate 1070 that the light ray travels as it reflects within the substrate, i.e., the distance between two adjacent points that impinge on the distal surface 1076 of the substrate 1070, for example, due to a single instance of total internal reflection. In some embodiments, the direct-view image 1205 can also be reduced or removed, for example, by including a coating (e.g., an IR coating configured to block or reduce IR light from the object 1220) on the proximal or distal surface 1074, 1076 proximate to the object 1220.

[0119] For example, the ghost image can be reduced or eliminated by reducing the size (d1) of the coupling optical element 1178a to a minimum size such that the stride distance (d2) exceeds d1 and varying the physical arrangement of the components of the imaging system 1000c.

[0120] In some embodiments, it may be desirable to control the stride distance (d2) and minimize the size of the coupling optical element 1178a while achieving a large stride distance. Without subscribing to a particular scientific theory, a large stride distance may reduce the intensity of the ghost image or enable the placement of the camera assembly 1030 outside of the stray light ray 1030. Thus, in some situations, the stride distance can be expressed as follows. d2 = 2 * t * tan(θ) [2] Where θ is the diffraction angle of the light ray 1122 and t is the thickness of the substrate 1070. Increasing the stride distance may be done by increasing the thickness (t) of the substrate or by increasing the diffraction angle (θ). As explained above, the diffraction angle (θ) may be based on the spatial frequency or period of the diffraction feature. For example, the lowest light ray 1122e has the minimum diffraction angle (θ), and thus it may be preferable to increase this diffraction angle in order to increase the stride distance. Additionally, increasing the thickness of the substrate 1070 may also increase the stride distance. However, it may be desirable to balance the thickness of the substrate 1070 with the goal of producing a lightweight and compact imaging system. In one embodiment, the substrate 1070 is a 2.5 millimeter thick polycarbonate sheet (other materials are also possible candidates), and the grating spatial frequency is 720 lines per millimeter. Various embodiments may include different substrate thicknesses or grating spatial frequencies.

[0121] Figures 14A and 14B schematically illustrate examples of imaging systems with a plurality of coupled optical elements having an arrangement different from that of the imaging system 1000a of FIG. 11. As explained with respect to FIG. 11, the coupled optical element is configured as either an internal or external coupled optical element to induce TIR and direct the light ray 1122 through the substrate 1070 to the camera assembly 1030. FIGS. 14A and 14B differ in the type and placement of the coupled optical elements.

[0122] For example, FIG. 14A depicts an imaging system 1000d that is substantially similar to the imaging system 1000b of FIG. 11. However, the imaging system 1000d includes a transmissive coupling optical element 1178b disposed on the proximal surface 1074 of the substrate 1070 and a transmissive coupling optical element 1188b disposed on the distal surface 1076 of the substrate 1070. The transmissive coupling optical element 1178b may be configured as an internal coupling optical element that is transmissive but diffracts the light 1122 of FIG. 11 at a certain diffraction angle and induces TIR on the distal surface 1046. The light 1122 may then be directed towards the transmissive coupling optical element 1188b, which is configured as an external coupling optical element, as described above in connection with FIG. 12A.

[0123] In the embodiment of FIG. 14B, the imaging system 1000e is substantially similar to the imaging system 1000b of FIG. 11. However, the imaging system 1000e includes a transmissive coupling optical element 1178b and a reflective coupling optical element 1188a disposed on the proximal surface 1074 of the substrate 1070. The transmissive coupling optical element 1178b may be configured as an internal coupling optical element that is transmissive but diffracts the light 1122 of FIG. 11 at a certain diffraction angle and induces TIR on the distal surface 1046. The light 1122 may then be directed towards the reflective coupling optical element 1188a, which is configured as an external coupling optical element, as described above in connection with FIG. 11.

[0124] FIG. 15 schematically illustrates another exemplary imaging system 1000f that is substantially similar to the imaging system 1000c of FIGS. 12A - 13B. Similar to the above imaging system, FIG. 15 illustrates an imaging system 1000f that includes a reflective coupling optical element 1178a and a transmissive coupling optical element 1188b disposed on the distal surface 1076 of a substrate 1070. However, the coupling optical elements 1178a and 1188b have a spatial frequency of 1411.765 lines per millimeter and a pitch of 708.33 nanometers, and the substrate is a 1 - millimeter - thick polycarbonate sheet. Thus, compared to the imaging system 1000c of FIGS. 12A - 13B, light 1122 can undergo TIR multiple times within the substrate 1070, and the camera assembly can be offset further away from the coupling optical element 1178a. Other configurations are also possible. (Alternative Embodiments of Imaging Systems for Off - Axis Imaging)

[0125] FIG. 11 shows an exemplary imaging system 1000b that includes a substrate 1070 having coupling optical elements 1178a, 1188a configured to TIR light from an object plane 1120 through the substrate 1070, although other configurations are also possible.

[0126] For example, FIG. 16 illustrates an imaging system 1000g comprising a substrate 1070 disposed adjacent to an optical component 1650. In some embodiments, the optical component 1650 may be the waveguide stack 260 of FIG. 6 or the waveguide stack 660 of FIGS. 9A-9C. The substrate 1070 is illustrated as being adjacent to and between the object 1120 and the optical component 1650, although other configurations are also possible. For example, the optical component 1650 may be between the substrate 1070 and the object 1120, or the substrate 1070 may be part of the optical component 1650. The substrate 1070 may comprise a plurality of reflective elements 1678 and 1688. As illustrated in FIG. 16, light 1122 travels from the object 1120 towards the substrate 1070 and may interact with the proximal surface 1074. The light 1122 may be refracted and directed towards the reflective element 1678, which reflects the light 1122 at an angle such that the light undergoes TIR on the proximal surface 1074. Thus, the light 1122 travels towards the reflective element 1688 via TIR. The light 1122 may be reflected by the reflective element 1688 towards the camera assembly 1030. Accordingly, the camera assembly 1030 may capture an off-axis image of the object 1120 as if the camera assembly 1030 were viewing the object 1120 directly (e.g., virtual camera assembly 1030c). In some embodiments, one or more of the reflective elements 1678, 1688 may comprise a "hot mirror" or a reflective coating that is reflective in the IR but transmissive in the visible spectrum.

[0127] In one embodiment of FIG. 16, the substrate 1070 is a 2 millimeter thick polycarbonate sheet, and the proximal surface 1074 is positioned 15.7 millimeters to the right (e.g., in the z - direction) from the object plane 1120. The object plane 1120 is 12 millimeters in the vertical (e.g., y - direction). In some embodiments, the reflective element 1678 is configured to capture substantially the entire FOV, and the central ray 1122c propagates at 25 degrees downward (e.g., in the negative y - direction) from the normal. The camera assembly 1030 may be positioned 15.7 millimeters below and 18.79 millimeters to the right of the origin of the ray 1122c. In this arrangement, the imaging system 1000g captures an image as if the view from the virtual camera 1030c were positioned 10.56 millimeters below and 22.65 millimeters to the right.

[0128] FIG. 17 illustrates an imaging system 1000h comprising a substrate 1770 disposed adjacent to an optical component 1650 (e.g., an optical cover glass or a prescription glass), and a reflective surface 1778 disposed adjacent to the substrate 1770. In some embodiments, the substrate 1770 may be substantially similar to the substrate 1070 described above. A specific arrangement is shown in FIG. 17, but other configurations are also conceivable. For example, the optical component 1650 may be between the substrate 1650 and the object 1120, or the substrate 1770 may be part of the optical component 1650. As shown in FIG. 17, light 1122 travels from the object 1120 towards the optical component 1650 and can interact therewith. The light 1122 can then be refracted or passed through the optical component 1650 as it travels towards the substrate 1770. After passing (refracting or passing) through the substrate 1770, the light 1122 is incident on the reflective surface 1778. The reflective surface 1778 may have optical properties configured to reflect the light 1122 and direct it towards the camera assembly 1030. Thus, the camera assembly 1030 can capture an off-axis image of the object 1120 as if the camera assembly 1030 were viewing the object 1120 directly. In one embodiment of FIG. 17, the imaging system 1000f is configured to capture a 16 millimeter by 24 millimeter object plane 1120, and the central ray 1122c propagates at a positive 17 degrees from the normal (shown as line 1790).

[0129] In some embodiments, the reflective surface 1778 may be the surface of a decorative or aesthetic lens or optical component. For example, the decorative lens may be a lens for use as sunglasses that filter out sunlight. In another embodiment, the decorative lens may be a color filtering lens for use in goggles. In yet another embodiment, the decorative lens may have a colored visual appearance that is visible to other people not wearing the lens (e.g., the lens appears blue, red, etc. to other people). The decorative lens may also include a color layer that is visible to people other than the user. The reflective surface 1778 may be a reflective coating on the inner surface of the decorative lens. The reflective coating may be reflective within the IR while being transmissive within the visible spectrum so that the wearer can view the world. As shown in FIG. 17, the reflective surface 1778 may have a concave shape configured to direct light 1122 toward the camera assembly 1030. However, other configurations are also possible.

[0130] FIG. 18 illustrates an imaging system 1000i comprising a substrate 1770 disposed adjacent to an optical component 1850 and a prism 1878 disposed adjacent to the substrate 1770. In some embodiments, the substrate 1770 may be substantially similar to the substrate 1070 described above. The optical component 1850 may be substantially similar to the optical component 1650, but may comprise one or more of the exit pupil expanders 800, 810, 820 of FIGS. 9A-9C. Although a specific arrangement is shown in FIG. 18, other configurations are also possible. For example, the optical component 1850 may be between the substrate 1770 and the object 1120, or the substrate 1770 may be part of the optical component 1850. As shown in FIG. 18, light 1122 travels from the object 1120 towards the optical component 1850 and can interact therewith. As the light 1122 travels towards the optical component 1850, it can be refracted or passed through. After passing through (refracting or passing through) the optical component 1850, the light 1122 is incident on the prism 1878 and is reflected by the surface 1878a towards the camera assembly 1030. Thus, the camera assembly 1030 can capture an off-axis image of the object 1120 as if the camera assembly 1030 were viewing the object 1120 directly. In some embodiments, the prism may be an IR prism “hot mirror” or the surface 1878a may comprise a reflective coating that is reflective in the IR and transmissive in the visible spectrum. In one embodiment of FIG. 18, the imaging system 1000i comprises a camera assembly 1030 having a 35-degree vertical FOV and a 30.73-millimeter focal length. Such an imaging system 1000i may be configured to capture a 16-millimeter × 24-millimeter object plane 1120, and the central ray 1122c propagates at negative 25 degrees from the normal (shown as line 1790). (Exemplary Routine for Imaging an Object)

[0131] FIG. 19 is a process flow diagram of an illustrative routine for imaging an object (e.g., a user's eye) using an off-axis camera (e.g., camera assembly 630 of FIG. 6 or camera assembly 1030 of FIG. 10A). Routine 1900 illustrates a way in which light from the object can be directed to a camera assembly that is positioned off-axis from or away from the object as if the camera assembly were directed directly at the object to image the object.

[0132] In block 1910, an imaging system is provided that is configured to receive light from the object and direct the light to the camera assembly. The imaging system may be one or more of imaging systems 1000a-i as described above in connection with FIGS. 10A-11, 12A, and 13A-18. For example, the imaging system may include a substrate (e.g., substrate 1070) having a first coupling optical element (e.g., first coupling optical element 1078, 1178a, or 1178b) and a second coupling optical element (e.g., second optical element 1188a or 1188b). The first and second optical elements may be disposed on a distal or proximal surface of the substrate as described above and throughout the present disclosure. The first and second optical elements may be laterally offset from each other along the substrate 1070. As described above and throughout the present disclosure, the first coupling optical element may be configured to deflect light at an angle and TIR the light between the proximal and distal surfaces. The first optical element may be configured to deflect light at an angle generally toward the second coupling optical element. The second coupling optical element may be configured to receive light from the first coupling optical element and deflect the light out of the substrate at an angle.

[0133] In block 1920, light is captured by a camera assembly (e.g., camera assembly 630 of FIG. 6 or camera assembly 1030 of FIGS. 10A - 11, 12A, and 13A - 18). The camera assembly may be oriented towards a second coupling optical element and receive light deflected by the second coupling optical element. The camera assembly may be an off - axis camera in a front - facing or rear - facing configuration. In block 1930, an off - axis image of an object may be generated based on the captured light as described throughout this specification and disclosure.

[0134] In some embodiments, routine 1900 may include an optional step (not shown) of illuminating the object with light from a light source (e.g., light source 632 of FIG. 6 or light source 1032 of FIGS. 10A - 11, 12A, and 13A - 18). In some embodiments, the light may include a range of wavelengths, including IR light.

[0135] In some embodiments, the off - axis image generated in block 1930 may be processed and analyzed, for example, using image processing techniques. The analyzed off - axis image may be used to perform one or more of eye tracking, biometric identification, multi - viewpoint reconstruction of the shape of the eye, estimation of the accommodation state of the eye, or imaging of the retina, iris, or other prominent patterns of the eye, and assessment of the physiological state of the user, based in part on the analyzed off - axis image as described above and throughout the disclosure.

[0136] In various embodiments, routine 1900 may be implemented by a hardware processor (e.g., local processing and data module 140 of FIG. 2) configured to execute instructions stored in memory. In other embodiments, a remote computing device (communicating with a display device over a network) with computer - executable instructions may cause the display device to implement aspects of routine 1900. Additional aspects 1. An optical device comprising: a substrate having a proximal surface and a distal surface; a first coupling optical element disposed on one of the proximal surface and the distal surface; and a second coupling optical element disposed on one of the proximal surface and the distal surface and laterally offset from the first coupling optical element along a direction parallel to the proximal surface or the distal surface, wherein the first coupling optical element is configured to deflect light at an angle and cause the light to undergo total internal reflection (TIR) between the proximal surface and the distal surface towards the second coupling optical element, and the second coupling optical element is configured to deflect light out of the substrate at an angle. 2. The optical device according to aspect 1, wherein the substrate is transparent to visible light. 3. The optical device according to aspect 1 or 2, wherein the substrate comprises a polymer. 4. The optical device according to any one of aspects 1-3, wherein the substrate comprises polycarbonate. 5. The optical device according to any one of aspects 1-4, wherein the first and second coupling optical elements are external to and fixed to at least one of the proximal surface and the distal surface of the substrate. 6. The optical device according to any one of aspects 1-5, wherein the first and second coupling optical elements comprise a part of the substrate. 7. The optical device according to any one of aspects 1-6, wherein at least one of the first and second coupling optical elements comprises a plurality of diffraction features. 8. The optical device according to aspect 7, wherein the plurality of diffraction features have a relatively high diffraction efficiency for a certain wavelength range and diffract substantially all of the light in the certain wavelength range. 9. The optical device according to aspect 7 or 8, wherein the plurality of diffraction features diffract light in at least one direction, at least one direction being selected such that the light undergoes TIR between the proximal surface and the distal surface, based at least in part on the period of the plurality of diffraction elements. 10 At least one of the first or second coupling optical elements includes at least one of an off-axis diffractive optical element (DOE), an off-axis diffraction grating, an off-axis diffractive optical element (DOE), an off-axis holographic mirror (OAHM), or an off-axis volumetric diffractive optical element (OAVDOE), or an off-axis cholesteric liquid crystal diffraction grating (OACLCG), and is the optical device according to any one of aspects 1-7. 11 The first and second coupling optical elements are each configured to deflect light in a first wavelength range while transmitting light in a second wavelength range, and are the optical device according to any one of aspects 1-7 and 10. 12 The first wavelength range includes light in at least one of the infrared (IR) or near-IR spectra, and the second wavelength range includes light in the visible spectrum, and is the optical device according to aspect 11. 13 The first and second coupling optical elements selectively reflect light in a certain wavelength range, the first coupling optical element is disposed on the distal surface of the substrate, and the second coupling optical element is disposed on the proximal surface of the substrate, and is the optical device according to any one of aspects 1, 7, and 11. 14 The first and second coupling optical elements selectively transmit light in a certain wavelength range, the first coupling optical element is disposed on the proximal surface of the substrate, and the second coupling optical element is disposed on the distal surface of the substrate, and is the optical device according to any one of aspects 1, 7, 10, and 11. 15 The first coupling optical element selectively reflects light in a certain wavelength range, the second coupling optical element selectively transmits light in the certain wavelength range, and the first and second coupling optical elements are disposed on the distal surface of the substrate, and is the optical device according to any one of aspects 1, 7, 10, and 11. 16 The first coupling optical element selectively transmits light in a certain wavelength range, the second coupling optical element selectively reflects light in the certain wavelength range, and the first and second coupling optical elements are disposed on the proximal surface of the substrate, and is the optical device according to any one of aspects 1, 7, 10, and 11. A head-mounted display (HMD) configured to be worn on a user's head, comprising: a frame; a pair of optical elements supported by the frame such that each optical element of the pair of optical elements can be disposed in front of the user's eye; and an imaging system comprising a camera assembly mounted on the frame and the optical device according to any one of aspects 1-16 of the side. 18. The HMD according to aspect 17, wherein at least one optical element of the pair of optical elements comprises a substrate. 19. The HMD according to aspect 17 or 18, wherein the substrate is disposed on the surface of at least one optical element of the pair of optical elements. 20. The HMD according to any one of aspects 17-19, wherein the frame comprises a pair of ear hooks, and the camera assembly is mounted on one of the pair of ear hooks. 21. The HMD according to any one of aspects 17-20, wherein the camera assembly is a front-facing camera assembly configured to image light received from a second coupling optical element. 22. The HMD according to any one of aspects 17-20, wherein the camera assembly is a rear-facing camera assembly disposed in a direction facing the user, and the rear-facing camera assembly is configured to image light received from a second coupling optical element. 23. The HMD according to any one of aspects 17-22, further comprising a light source configured to emit light in a range of a first wavelength toward at least one of the user's eye, a part of the eye, or a part of the tissue surrounding the eye. 24. The HMD according to aspect 23, wherein the light in the range of the first wavelength is reflected by at least one of the user's eye, a part of the eye, or a part of the tissue surrounding the eye toward the first coupling optical element. 25. The HMD according to any one of aspects 17-23, wherein each of the pair of optical elements is transmissive to visible light. 26. The HMD according to any one of aspects 17-23 and 25, wherein each of the pair of optical elements is configured to display an image to the user. The camera assembly is configured to image at least one of a user's eye, a part of the eye, or a part of the tissue surrounding the eye, based at least in part on light received from the second coupling optical element, the HMD according to any one of aspects 17-23, 25, and 26. 28 The HMD is configured to track the user's line of sight based on an image of at least one of a user's eye, a part of the eye, or a part of the tissue surrounding the eye, the HMD according to aspect 27. 29 The image formed by the camera assembly is installed in front of the user's eye and coincides with the image formed by the camera that directly visualizes at least one of the user's eye, a part of the eye, or a part of the tissue surrounding the eye, the HMD according to aspect 27. 30 The optical device is arranged to reduce stray light received by the camera assembly, the HMD according to any one of aspects 17-23, 25, and 27. 31 The size of the first coupling optical element is less than the stride distance of the light reflected between the distal surface and the proximal surface of the substrate, the stride distance being based on the thickness of the substrate and the angle at which the first coupling optical element deflects light, the HMD according to any one of aspects 17-23, 25, 27, and 30. 32 The size of the first coupling optical element is based on the field of view of the user's eye, the HMD according to aspect 31. 33 The image of the user's eye formed by the camera assembly and the image of the user's eye formed by the camera installed in front of the user's eye are indistinguishable, the HMD according to any one of aspects 17-23, 25, 27, 30, and 31. A non-transitory data storage device configured to store an image obtained by a 34 camera assembly, and a hardware processor communicating with the non-transitory data storage device, the hardware processor being programmed with executable instructions for analyzing the image and performing one or more of eye tracking, biometric identification, multi-viewpoint reconstruction of the shape of the eye, estimation of the accommodation state of the eye, or imaging of the retina, iris, or other prominent pattern of the eye, and assessment of the physiological state of the user, the HMD according to any one of aspects 17-23, 25, 27, 30, 31, and 33. 35 An imaging system comprising a substrate having a proximal surface and a distal surface, a first diffractive optical element disposed on one of the proximal surface and the distal surface, and a second diffractive optical element disposed on one of the proximal surface and the distal surface, the second diffractive optical element being offset from the first diffractive optical element along a direction parallel to the proximal surface or the distal surface, the first diffractive optical element being configured to deflect light at an angle and to totally internally reflect (TIR) the light between the proximal surface and the distal surface towards a second coupling optical element, the second diffractive optical element being configured to deflect light incident thereon out of the substrate at an angle, the substrate, and a camera assembly for imaging the light deflected by the second diffractive optical element. 36 The first and second diffractive optical elements include at least one of the surfaces of an off-axis diffractive optical element (DOE), an off-axis diffractive grating, an off-axis diffractive optical element (DOE), an off-axis holographic mirror (OAHM), or an off-axis volumetric diffractive optical element (OAVDOE), an off-axis cholesteric liquid crystal diffractive grating (OACLCG), a hot mirror, a prism, or a decorative lens, the imaging system according to aspect 35. A method of imaging an object using a virtual camera, comprising providing an imaging system in front of an object to be imaged, the imaging system comprising a substrate having a first coupling optical element and a second coupling optical element disposed on one of a proximal surface and a distal surface of the substrate, respectively, and offset from each other, the first coupling optical element being configured to deflect light at an angle and to totally internally reflect (TIR) the light between the proximal surface and the distal surface towards the second coupling optical element, and the second coupling optical element being configured to deflect the light out of the substrate at an angle, and capturing light using a camera assembly oriented to receive the light deflected by the second coupling optical element, and generating an off-axis image of the object based on the captured light. 38 The method according to aspect 37, wherein the first and second coupling optical elements each deflect light in a range of a first wavelength while transmitting light in a range of a second wavelength. 39 The method according to aspect 37 or 38, further comprising illuminating the object using a range of a first wavelength emitted by a light source. 40 The method according to any one of aspects 37-39, further comprising analyzing the off-axis image and performing one or more of eye tracking, biometric identification, multi-viewpoint reconstruction of the shape of the eye, estimation of the accommodation state of the eye, or imaging of the retina, iris, or other prominent pattern of the eye, and evaluation of the physiological state of the user, based at least in part on the analyzed off-axis image. 41 An imaging system comprising a substrate having a proximal surface and a distal surface, a reflective optical element adjacent to the distal surface and configured to reflect light passing out of the substrate at an angle at the distal surface, and a camera assembly for imaging the light reflected by the reflective optical element. 42 The imaging system according to aspect 41, wherein the reflective optical element comprises a surface of a decorative lens. 43 The imaging system according to aspect 41 or 42, wherein the reflective optical element comprises a reflective coating on the surface of the decorative lens. 44 The reflective optical element is the imaging system according to any one of aspects 41 - 43, comprising a reflective prism. 45 The reflective optical element is the imaging system according to any one of aspects 41 - 44, being reflective to infrared light and transmissive to visible light. 46 The imaging system according to any one of aspects 41 - 45, further comprising a diffractive optical element adjacent to the proximal surface. 47 The imaging system according to any one of aspects 41 - 46, wherein the camera assembly is a front - facing camera assembly configured to image the light received from the reflective optical element. 48 A head - mounted display (HMD) configured to be worn on a user's head, comprising a frame, a pair of optical elements supported by the frame such that each optical element of the pair of optical elements can be disposed in front of the user's eye, and the imaging system according to any one of claims 41 - 47. 49 The HMD according to aspect 48, wherein at least one optical element of the pair of optical elements includes a substrate. 50 The HMD according to aspect 48 or 49, wherein the substrate is disposed on the surface of at least one optical element of the pair of optical elements. 51 The HMD according to any one of aspects 48 - 50, wherein the frame comprises a pair of earhooks, and the camera assembly is mounted on one of the pair of earhooks. 52 The HMD according to any one of aspects 48 - 51, further comprising a light source configured to emit light in a range of a first wavelength towards at least one of the user's eye, a part of the eye, or a part of the tissue surrounding the eye. 53 The HMD according to any one of aspects 48 - 52, wherein each of the pair of optical elements is transmissive to visible light. 54 The HMD according to any one of aspects 48 - 53, wherein each of the pair of optical elements is configured to display an image to the user. The 55 camera assembly is configured to image at least one of the user's eye, a part of the eye, or a part of the tissue surrounding the eye, based at least in part on the light received from the second coupling optical element, of the HMD according to any one of aspects 48 - 54. The 56 HMD is configured to track the user's line of sight based on an image of at least one of the user's eye, a part of the eye, or a part of the tissue surrounding the eye, of the HMD according to any one of aspects 48 - 55. The 57 image formed by the camera assembly is installed in front of the user's eye and coincides with the image formed by the camera that directly visualizes at least one of the user's eye, a part of the eye, or a part of the tissue surrounding the eye, of the HMD according to any one of aspects 48 - 56. The 58 optical device is arranged to reduce stray light received by the camera assembly, of the HMD according to any one of aspects 48 - 57. The 59 image of the user's eye formed by the camera assembly and the image of the user's eye formed by the camera installed in front of the user's eye are indistinguishable, of the HMD according to any one of aspects 48 - 58. A non - transitory data storage device configured to store the image obtained by the camera assembly, and a hardware processor communicating with the non - transitory data storage device, the hardware processor being programmed with executable instructions to analyze the image and perform one or more of eye tracking, biometric identification, multi - viewpoint reconstruction of the shape of the eye, estimation of the accommodation state of the eye, or imaging of the retina, iris, or other prominent patterns of the eye, and assessment of the user's physiological state, of the HMD according to any one of aspects 48 - 59. (Additional Considerations)

[0137] In the embodiments described above, the optical array is described in relation to an eye imaging display system, more specifically, an augmented reality display system. However, it will be understood that the principles and advantages of the optical array can be used for head-mounted displays, optical systems, devices, or methods. It should be understood that any one of the features of any of the embodiments can be combined with and / or substituted for any other one of the features of any of the other embodiments.

[0138] Unless the context clearly requires otherwise, throughout the description and claims, the words "comprise", "comprising", "include", "including", "have", "having", and equivalents are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in the sense of "including, but not limited to". The word "coupled", as generally used herein, refers to two or more elements that may be directly connected or connected through one or more intervening elements. Similarly, the word "connected", as generally used herein, refers to two or more elements that may be directly connected or connected through one or more intervening elements. Depending on the context, "coupled" or "connected" may refer to an optical coupling or optical connection such that light is coupled or connected from one optical element to another. Additionally, the words "herein", "above", "below", "infra", "supra", and words of similar import, when used in this application, refer to this application as a whole and not to any particular part of this application. Where the context permits, the words in the detailed description above using the singular or plural number may also include the plural or singular number, respectively. The word "or", referring to a list of two or more items, is inclusive (not exclusive), and "or" encompasses all of the following interpretations of the words, i.e., any of the items in the list, all of the items in the list, and any combination of one or more of the items in the list, without excluding other items added to the list. Additionally, the articles "a", "an", and "the", as used in this application and the appended claims, are to be construed to mean "one or more" or "at least one" unless otherwise specified. more)

[0139] As used herein, the phrase referring to a list of items "at least one of" refers to any combination of those items, including a single element. As an example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Connective phrases such as the phrase "at least one of X, Y, and Z" are generally understood in a context such that they are used to convey that an item, term, etc. can be at least one of X, Y, or Z, unless specifically stated otherwise. Thus, such connective phrases generally do not intend to suggest that an embodiment requires that at least one of X, at least one of Y, and at least one of Z each be present.

[0140] Furthermore, among other things, conditional statements used herein such as "can", "could", "might", "may", "e.g.", "for example", "such as", and equivalents are generally understood to convey that while one embodiment includes a certain feature, element, and / or state, other embodiments do not, unless specifically stated otherwise or understood otherwise within the context in which they are used. Thus, such conditional statements generally do not intend to suggest that a feature, element, and / or state is required in any way for one or more embodiments, or whether these features, elements, and / or states should be included or implemented in any particular embodiment.

[0141] While certain embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of the disclosure. Indeed, the novel apparatus, methods, and systems described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions, and changes in the form of the methods and systems described herein may be made without departing from the spirit of the disclosure. For example, while blocks are presented in a given arrangement, alternative embodiments may implement similar functionality using different components and / or circuit topologies, and some blocks may be deleted, moved, added, subdivided, combined, and / or modified. Each of these blocks may be implemented in a variety of different ways. Any suitable combination of the elements and acts of the various embodiments described above can be combined to provide further embodiments. The various features and processes described above may be implemented independently of one another or in various ways. Neither any element or combination of elements is necessary or essential with respect to all embodiments. All suitable combinations and sub-combinations of the features of the disclosure are intended to be within the scope of the disclosure.

Claims

1. An apparatus, wherein the apparatus comprises: a substrate extending in a plane, the substrate having a first surface and a second surface parallel to the first surface, the first and second surfaces being parallel to the plane, the substrate having a thickness t perpendicular to the plane; a first optical coupling element disposed on the first surface; a second optical coupling element disposed on the first surface or the second surface, the second optical coupling element being laterally offset from the first optical coupling element along a first direction within the plane; and comprising: the first optical coupling element is configured to deflect light incident on the first optical coupling element at a first angle θ with respect to a normal to the plane and to totally internally reflect (TIR) the light between the first surface and the second surface toward the second optical coupling element, and the second optical coupling element is configured to deflect light out of the substrate at a second angle; the first optical coupling element has a length along the first direction that is less than 2t×tan(θ); the first and second optical coupling elements are each configured to deflect light in a first wavelength range within the infrared (IR) or near-IR spectrum while transmitting light in a second wavelength range within the visible spectrum; the apparatus is an eyepiece for a head-mounted display.

2. The apparatus according to claim 1, wherein the second optical coupling element has a second length along the first direction that is less than 2t×tan(θ).

3. The apparatus according to claim 1, wherein at least one of the first and second angles depends on the wavelength of the light.

4. The apparatus according to claim 3, wherein the first optical coupling element is a diffractive optical element having a plurality of diffractive features, and θ depends at least in part on the period or spatial frequency of the diffractive features.

5. The apparatus according to claim 1, wherein the refractive index of the substrate is in the range of 1 to 2.

6. The apparatus according to claim 1, wherein t is in the range of 1 to 2 millimeters.

7. The apparatus according to claim 1, wherein the substrate is transparent to visible light.

8. The apparatus according to claim 1, wherein the substrate comprises a polymer.

9. The apparatus according to claim 8, wherein the polymer comprises polycarbonate.

10. The apparatus according to claim 1, wherein the first and second optical coupling elements are external to at least one of the first and second surfaces of the substrate and are fixed to the at least one of the first and second surfaces of the substrate.

11. The apparatus according to claim 1, wherein the first optical coupling element is configured to reflect the light before the light begins to undergo total internal reflection (TIR) towards the second optical coupling element.

12. A method of imaging an object using a camera, the method comprising: providing an imaging system in front of a user's eye to be imaged, the imaging system comprising a substrate extending in a plane, the substrate having a first surface and a second surface parallel to the first surface, the substrate having a thickness t perpendicular to the plane, and a first optical coupling element and a second optical coupling element, the first and second optical coupling elements being respectively disposed on one of the first and second surfaces of the substrate, offset from each other along a first direction, the first optical coupling element being configured to deflect light incident on the first optical coupling element at a first angle θ with respect to a normal to the plane, and to totally internally reflect (TIR) the light between the first surface and the second surface towards the second optical coupling element, the second optical coupling element being configured to deflect the light out of the substrate at a second angle, the first optical coupling element having a length along the first direction that is less than 2t×tan(θ), and the first and second optical coupling elements being configured to deflect light in a first wavelength range within the infrared (IR) or near-IR spectrum while transmitting light in a second wavelength range within the visible spectrum; capturing the light using a camera assembly oriented to receive the light deflected by the second optical coupling element; generating an off-axis image of the user's eye based on the captured light and including a method.

13. The method according to claim 12, wherein the first angle is selected such that the second optical coupling element has a second length along the first direction that is less than 2t×tan(θ).

14. The method according to claim 12, wherein at least one of the first and second angles depends on the wavelength of the light.

15. The method according to claim 12, wherein the first optical coupling element is configured to reflect the light before the light begins to undergo TIR towards the second optical coupling element. **Claim 16** The method according to claim 12, further comprising illuminating the object using the range of the first wavelength emitted by the light source. **Claim 17** analyzing the off-axis image; performing at least one of the functions selected from the group consisting of eye tracking, biometric identification, multi-viewpoint reconstruction of the shape of the eye, estimation of the accommodation state of the eye, and imaging of the retina, iris, and other prominent patterns of the eye, and evaluation of the physiological state of the user based on the analysis; The method according to claim 12, further comprising.

Images