Dual-wavelength eye imaging

Abstract

The eye is illuminated with a first invisible light wavelength and a second invisible light wavelength. A first ocular image is captured from the first reflected light having the first invisible light wavelength, and a second ocular image is captured from the second reflected light having the second invisible light wavelength.

Description

Technical Field

[0001] The present disclosure relates to stereoscopic imaging of one or more eyes with two different wavelengths. Background Art

[0002] Capturing images of the eye is useful in a variety of contexts. Images of the eye can be analyzed to determine the position of the eye, the size of the pupil, and / or where the eye is looking. Eye imaging systems vary in the speed and accuracy with which they image the eye. Some eye imaging systems require extensive calibration to operate accurately. It would be desirable to reduce or eliminate the calibration time or complexity of eye imaging. The speed and accuracy of eye imaging systems can also be improved. Summary of the invention

[0003] According to one aspect of the present invention, a near-eye imaging system is provided, comprising: a first illuminator configured to emit a first near-infrared light toward an eye area; a second illuminator configured to emit a second near-infrared light toward the eye area, the wavelength of the second near-infrared light being different from that of the first near-infrared light; a camera module; and a near-eye optical element comprising: a first combiner layer configured to allow visible scene light to pass through to reach the eye area and to guide the first near-infrared light reflected from the eye area to the camera module; and a second combiner layer configured to allow visible scene light to pass through to reach the eye area and to guide the second near-infrared light reflected from the eye area to the camera module.

[0004] In some embodiments, the camera module may be configured to capture a first ocular image of a first near infrared light and configured to capture a second ocular image of a second near infrared light, the camera module being configured to filter out light other than the first near infrared light and the second near infrared light.

[0005] In some embodiments, the camera module may include: a first filter covering a first pixel of the camera module, wherein the first filter allows a first near-infrared light to pass and rejects other light, and a first eye image is generated by the first pixel; and a second filter covering a second pixel of the camera module, wherein the second filter allows a second near-infrared light to pass and rejects other light, and a second eye image is generated by the second pixel.

[0006] In some embodiments, the near-eye imaging system may further include: processing logic configured to selectively activate a first illuminator during a first time period and configured to selectively activate a second illuminator during a second time period different from the first time period, wherein the camera module is configured to capture a first eye image during the first time period and configured to capture a second eye image during the second time period.

[0007] In some embodiments, at least one of the first combiner layer and the second combiner layer may include a holographic optical element (HOE).

[0008] In some embodiments, the first combiner layer can selectively direct first near infrared light incident at a first angle to the camera module, and wherein the second combiner layer selectively directs second near infrared light incident at a second angle to the camera module.

[0009] In some embodiments, the first angle may be at least 10 degrees greater than the second angle.

[0010] In some embodiments, the first illuminator and the second illuminator can be intra-field illuminators positioned on an illumination layer of a near-eye optical element that is within a field of view (FOV) of an eye of a user of the near-eye optical element.

[0011] In some embodiments, the first illuminator and the second illuminator may include at least one of a micro light emitting diode (micro-LED), an edge emitting LED, a vertical cavity surface emitting laser (VCSEL) diode, or a super luminescent diode (SLED).

[0012] In some embodiments, the first near infrared light may be centered around 850 nm and the second near infrared light may be centered around 940 nm.

[0013] According to another aspect of the present invention, a near-eye optical element is provided, comprising: a first illuminator configured to emit first light having a first invisible wavelength toward an eye area; a second illuminator configured to emit second light having a second invisible wavelength toward the eye area, the first invisible wavelength being different from the second invisible wavelength; a first combiner layer configured to allow visible scene light to pass to the eye area and to guide the first light reflected from the eye area to a camera module; and a second combiner layer configured to allow visible scene light to pass to the eye area and to guide the second light reflected from the eye area to the camera module.

[0014] In some embodiments, the first combiner layer may selectively direct first light incident at a first angle to the camera module, and wherein the second combiner layer selectively directs second light incident at a second angle to the camera module.

[0015] In some embodiments, at least one of the first combiner layer and the second combiner layer may include a holographic optical element (HOE).

[0016] In some embodiments, the first illuminator and the second illuminator can be intra-field illuminators positioned on an illumination layer of a near-eye optical element that is within a field of view (FOV) of an eye of a user of the near-eye optical element.

[0017] In some embodiments, the first illuminator and the second illuminator may include at least one of a micro light emitting diode (micro-LED), an edge emitting LED, a vertical cavity surface emitting laser (VCSEL) diode, or a super luminescent diode (SLED).

[0018] According to another aspect of the present invention, a method for near-eye stereoscopic imaging is provided, the method comprising: illuminating the eye with a first invisible light wavelength; illuminating the eye with a second invisible light wavelength different from the first invisible light wavelength; capturing a first eye image from a first reflected light having the first invisible light wavelength; and capturing a second eye image from a second reflected light having the second invisible light wavelength, wherein the first eye image and the second eye image are captured using a camera configured to filter out light other than the first invisible light wavelength and the second invisible light wavelength.

[0019] In some embodiments, the method may further include determining an eye tracking position based on the first eye image and the second eye image.

[0020] In some embodiments, a first intrafield illuminator may emit a first invisible light wavelength and a second intrafield illuminator may emit a second invisible light wavelength, the first intrafield illuminator and the second intrafield illuminator being included in a near-eye optical element located in a field of view (FOV) of a user.

[0021] In some embodiments, a first eye image may be captured by the camera when a first invisible light wavelength illuminates an eye area and a second invisible light wavelength does not illuminate the eye area, and wherein a second eye image is captured by the camera when the second invisible light wavelength illuminates the eye area and the first invisible light wavelength does not illuminate the eye area.

[0022] In some embodiments, a camera that captures a first ocular image and a camera that captures a second ocular image may include: a first filter covering a first pixel of the camera, wherein the first filter passes a first invisible light wavelength and rejects other light, and the first ocular image is generated by the first pixel; and a second filter covering a second pixel of the camera, wherein the second filter passes a second invisible light wavelength and rejects other light, and the second ocular image is generated by the second pixel.

[0023] It should be understood that any feature described herein as being suitable for incorporation into one or more aspects or embodiments of the present invention is intended to be universal in any and all aspects and embodiments of the present disclosure. Other aspects of the present disclosure can be understood by those skilled in the art based on the specification, claims and drawings of the present disclosure. The foregoing general description and the following detailed description are exemplary and explanatory only, and are not limitations of the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Non-limiting and non-exhaustive embodiments of the present invention are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.

[0025] Figure 1 An example head mounted display (HMD) is illustrated in accordance with aspects of the present disclosure.

[0026] Figure 2 is a top view of an example near-eye optic including an illumination layer according to an embodiment of the present disclosure.

[0027] Figure 3 Illustrated is a front view of an eye through an example lighting layer in accordance with aspects of the present disclosure.

[0028] Figure 4 Example light paths of the first invisible illumination light and the second invisible illumination light according to aspects of the present disclosure are illustrated.

[0029] Figure 5 An example image sensor including an array of image pixels in accordance with aspects of the present disclosure is shown.

[0030] Figure 6 An image sensor that may be used in a camera module according to aspects of the present disclosure is illustrated.

[0031] Figure 7 Illustrated is a first filter interspersed with a second filter that may be used in a camera module according to aspects of the present disclosure.

[0032] Figure 8 A stereoscopic eye imaging system is illustrated in accordance with aspects of the present disclosure, wherein a camera module captures a combined image including a first eye image at a first non-visible wavelength and a second eye image at a second non-visible wavelength.

[0033] Fig. 9A A stereoscopic eye imaging system is illustrated in which a camera module captures a first ocular image and a second ocular image in different image captures in accordance with aspects of the present disclosure.

[0034] Fig. 9B An example hybrid schematic timing diagram illustrating example operating techniques of a system according to aspects of the present disclosure is illustrated.

[0035] Fig.10 A flow chart illustrating an example process for near-eye stereoscopic imaging in accordance with aspects of the present disclosure is illustrated. DETAILED DESCRIPTION

[0036] Embodiments of dual wavelength eye imaging are described herein. In the following description, numerous specific details are set forth to provide a thorough understanding of the embodiments. However, a person skilled in the relevant art will recognize that the various techniques described herein can be practiced without one or more of the specific details or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring certain aspects.

[0037] References throughout the specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the phrases "in one embodiment" or "in an embodiment" appearing in various places throughout the specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0038] The present disclosure relates to stereoscopic imaging of one or more eyes with two different wavelengths. For example, embodiments of the present disclosure can be used in various environments including head-mounted devices for eye tracking purposes. In some aspects of the present disclosure, a first illuminator array emits a first near-infrared light (e.g., 850nm) toward the eye and a second illuminator array emits a second near-infrared light (e.g., 940nm) toward the eye. The optical combiner in the near-eye optical element can be configured to guide the first near-infrared light (reflected / scattered from the eye) and the second near-infrared light (also reflected / scattered from the eye) to a camera module, which generates a first eye image (based on the first near-infrared light) and a second eye image (based on the second near-infrared light). Analyzing the first eye image of the first near-infrared wavelength and the second eye image of the second near-infrared wavelength can allow greater eye imaging accuracy, especially in the z dimension (depth). This increased accuracy improves the tracking position of the eye and can reduce or eliminate the calibration process associated with eye position tracking. In addition, determining the eye's gaze interval (the position where the eye is looking) can be reduced by three times or more to more accurately define the position where the eye is looking. These and other embodiments will be combined Figure 1-Figure 10 Describe in more detail.

[0039] Figure 1 An example head mounted display (HMD) 100 is illustrated in accordance with aspects of the present disclosure. Figure 1 1, but embodiments of the present disclosure may be used in a variety of environments, including head mounted devices that do not include a display. The illustrated example of HMD 100 is shown as including frame 102, temples 104A and 104B, and near-eye optical elements 110A and 110B. Camera modules 108A and 108B are shown coupled to temples 104A and 104B, respectively. Figure 1Also illustrated is an exploded view of an example of a near-eye optical element 110A. The near-eye optical element 110A is shown to include an optional light-transmitting layer 120A, an illumination layer 130A, an optical combiner layer 140A, and a display layer 150A. The display layer 150A may include a waveguide 158 configured to direct a virtual image to an eye of a user of the HMD 100.

[0040] The lighting layer 130A is shown as including a plurality of illuminators 126 and 127. The illuminators 126 and 127 may be described as "in-field" because they are in the field of view (FOV) of the user of the HMD 100. In one embodiment, the in-field illuminators 126 and 127 may be in the same FOV in which the user views the display of the HMD. The in-field illuminators 126 and 127 may be in the same FOV in which the user views the external environment of the HMD 100 via scene light 191 propagating through the near-eye optical element 110. Although the in-field illuminators 126 and 127 may introduce minor obstructions into the near-eye optical element 110A, the in-field illuminators 126 and 127 and their corresponding electrical wiring may be so small as to be unnoticeable or unimportant to the wearer of the HMD 100. In addition, any occlusion from the field illuminators 126 and 127 will be placed very close to the eyes so that the human eye cannot focus, and thus help to be unnoticeable or unnoticeable in the field illuminators 126 and 127. In some embodiments, each field illuminator 126 or 127 has a footprint (or size) of less than about 200×200 microns. When the HMD 100 is worn by the user, the field illuminators 126 and 127 can be placed between 10mm and 30mm from the eyes. In some embodiments, the field illuminators 126 and 127 can be placed between 15mm and 25mm from the user's eyes. The field illuminators 126 and 127 can emit invisible light to the eye area. In some embodiments, the invisible light can be near infrared light (e.g., 750nm-1.5μm). Illuminator 126 emits a first invisible wavelength and illuminator 127 emits a second invisible wavelength having a wavelength different from the first invisible wavelength. Each of the intrafield illuminators 126 and 127 may be a micro light emitting diode (micro-LED), an edge emitting LED, a vertical cavity surface emitting laser (VCSEL) diode, or a super luminescent diode (SLED).

[0041] like Figure 1As shown in , frame 102 is coupled to temples 104A and 104B for fixing HMD 100 to the user's head. Example HMD 100 may also include support hardware incorporated into frame 102 and / or temples 104A and 104B. The hardware of HMD 100 may include any processing logic, a wired and / or wireless data interface for sending and receiving data, a graphics processor, and one or more memories for storing data and computer executable instructions. In one example, HMD 100 may be configured to receive a wired power source and / or may be configured to be powered by one or more batteries. In addition, HMD 100 may be configured to receive wired and / or wireless data including video data.

[0042] Figure 1 Near-eye optical elements 110A and 110B are illustrated as being configured to be mounted to frame 102. In some examples, near-eye optical elements 110A and 110B may appear transparent to a user to facilitate augmented reality or mixed reality, such that a user may view visible scene light 191 from the environment while also receiving display light directed to their eyes through display layer 150A. In further examples, some or all of near-eye optical elements 110A and 110B may be incorporated into a virtual reality headset, wherein the transparent nature of near-eye optical elements 110A and 110B allows a user to view an electronic display (e.g., a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a micro-LED display, etc.) incorporated into the virtual reality headset.

[0043] As mentioned above, the in-field illuminators 126 and 127 of the illumination layer 130A can be configured to emit invisible illumination light toward the eye side 109 of the near-eye optical element 110A to illuminate the user's eye. The near-eye optical element 110A is shown to include an optical combiner layer 140A, wherein the optical combiner layer 140A is disposed between the illumination layer 130A and the back surface 111 of the near-eye optical element 110A. In some aspects, the optical combiner 140A is configured to receive a first reflected light (having a wavelength emitted by the first illuminator 126) reflected / scattered by the user's eye and direct the first reflected light to the camera module 108A. The optical combiner 140A can also be configured to receive a second reflected light (having a wavelength emitted by the second illuminator 127) reflected / scattered by the user's eye and direct the second reflected light to the camera module 108A. The optical combiner 140A can have a first layer configured to direct the first reflected light and a second layer that reflects the second reflected light. The camera module 108A may be located in a different position than that illustrated. In some aspects, the optical combiner 140A may transmit visible light (e.g., approximately 400nm-700nm), such as scene light 191 incident on the back surface 111 of the near-eye optical element 110A. In some examples, the optical combiner 140A may be configured as a holographic optical element (HOE) or a volume hologram, which may include one or more Bragg gratings for directing the first reflected light and the second reflected light toward the camera module 108A. In some examples, the optical combiner 140A includes a polarization-selective volume hologram (aka polarization volume hologram) that diffracts (reflects) a specific polarization orientation of incident light having a specific wavelength toward the camera module 108A while passing other polarization orientations.

[0044] The display layer 150A may include one or more other optical elements depending on the design of the HMD 100. For example, the display layer 150A may include a waveguide 158 to direct display light generated by the electronic display to the user's eyes. In some implementations, at least a portion of the electronic display is included in the frame 102 of the HMD 100. The electronic display may include an LCD, an organic light emitting diode (OLED) display, a micro-LED display, a micro-projector, or a liquid crystal on silicon (LCOS) display for generating display light.

[0045] The light-transmitting layer 120A is shown as being disposed between the eyeward side 109 of the illumination layer 130A and the near-eye optical element 110A. The light-transmitting layer 120A can receive infrared illumination light emitted by the illumination layer 130A and pass the infrared illumination light to illuminate the user's eyes. As previously described, the light-transmitting layer 120A can also be transparent to visible light, such as scene light 191 received from the environment and / or display light received from the display layer 150A. In some examples, the light-transmitting layer 120A has a curvature for focusing light (e.g., display light and / or scene light) to the user's eyes. Therefore, in some examples, the light-transmitting layer 120A can be referred to as a lens. In some aspects, the light-transmitting layer 120A has a thickness and / or curvature corresponding to the user's specifications. In other words, the light-transmitting layer 120A can be a prescription lens. However, in other examples, the light-transmitting layer 120A can be a non-prescription lens.

[0046] Figure 2 2 is a top view of an example near-eye optical element 210, which includes an illumination layer 230, a combiner layer 240, and a display layer 250. In some embodiments, a transparent layer (not shown) may optionally be included between the illumination layer 230 and the eye 202. The first illuminator 236 emits a first invisible illumination light 238 to the eye region to illuminate the eye 202. Figure 2 In FIG. 2 , illuminators 236A, 236B, 236C, 236D, and 236E are illustrated, but there may be more or fewer illuminators in the illumination layer 230. The second illuminator 237 emits a second invisible illumination light 239 to the eye region to illuminate the eye 202. Figure 2 2, illuminators 237A, 237B, 237C, 237D, and 237E are illustrated, but there may be more or fewer illuminators in the illumination layer 230. The wavelength of the first invisible illumination light 238 is different from the wavelength of the second invisible illumination light 239. In some aspects, the first invisible illumination light 238 and the second invisible illumination light 239 may be near infrared light.

[0047] As described above, illuminators 236 and 237 may be VCSELs or SLEDs, and thus illumination light 238 / 239 may be narrowband infrared illumination light (e.g., 1-10 nm line width). As will be described in more detail below, reflected illumination light reflected by eye 202 may be received by combiner layer 240 and redirected to camera module 108A.

[0048] The camera module 108A generates first and second ocular images, wherein the first ocular image is of the first invisible illumination light 238 and the second ocular image is of the second invisible illumination light 239. For example, these first ocular images and the second ocular images may be used to determine the position of the eye 202 and / or the gaze direction of the eye 202. The camera module 108A may be configured to filter out light other than the first invisible illumination light and the second invisible illumination light, so that the camera module 108A images only the wavelengths of the first invisible illumination light 238 and the wavelengths of the second invisible illumination light 239.

[0049] Figure 2 It is shown that scene light 191 (visible light) from the external environment can propagate through the display layer 250, the combiner layer 240 and the illumination layer 230 to become light incident on the eye 202, so that the user can view the scene of the external environment. Figure 2 It is also shown that the display layer 250 can generate or redirect display light 293 to present a virtual image to the eye 202. The display light 293 is visible light and propagates through the combiner layer 240 and the illumination layer 230 to reach the eye 202.

[0050] The illumination layer 230 may include a transparent substrate on which the intrafield illuminator 237 is disposed. The intrafield illuminator 237 may also be encapsulated in a transparent material 232. The transparent material 232 is configured to transmit visible light (eg, 400nm-750nm) and near infrared light (eg, 750nm-1.5μm).

[0051] Figure 3 A front view of the eye 202 through an example lighting layer 330 is illustrated in accordance with aspects of the present disclosure. In the illustrated embodiment, the lighting layer 330 includes twenty-one first intra-field illuminators (336A-336U) and twenty-one second intra-field illuminators (337A-337U). Figure 3 In the example illustrated in , each first in-field illuminator 336 is arranged very close to the corresponding second in-field illuminator 337. However, other arrangements of the first in-field illuminator 336 and the second in-field illuminator 337 are contemplated. Although not specifically illustrated, transparent or translucent electrical traces can provide power to the first in-field illuminator 336 and the second in-field illuminator 337. In some embodiments, a transparent semiconductor material such as indium tin oxide (ITO) is used as the electrical trace. Corresponding electronic circuits (e.g., transistors and power supplies) for selectively activating the first in-field illuminator 336 and the second in-field illuminator 337 can be included in the frame 102 or the temple 104 of the head-mounted device. The first in-field illuminator 336 and the second in-field illuminator 337 can share a common ground or common power rail to reduce the need for electrical traces to pass through the lighting layer 330.

[0052] Figure 4 4 shows example light paths of a first invisible illumination light 438 and a second invisible illumination light 439 according to aspects of the present disclosure. Figure 4 In FIG. 4 , the array of first illuminators 436 is configured to emit first invisible illumination light 438 to illuminate the eye 202. Figure 4 4 shows only the illumination light from the illuminator 436A for illustrating and describing the optical path of the illumination light, but all the first illuminators 436 emit the first invisible illumination light 438. At least a portion of the first invisible illumination light 438 is reflected from the cornea 201 as the first reflected light 448 and propagates through the illumination layer 430 and encounters the combining optical element 440.

[0053] Similar to Figure 4 4, the array of second illuminators 437 is configured to emit second invisible illumination light 439 to illuminate the eye 202. Only the illumination light from illuminator 437B is shown for illustration and description of the optical path of the illumination light, but all second illuminators 437 emit second invisible illumination light 439. At least a portion of the second invisible illumination light 439 is reflected from the cornea 201 as second reflected light 449 and propagates through the illumination layer 430 and encounters the combiner optical element 440. The second reflected light 449 propagates through the first combiner layer 441 before encountering the second combiner layer 442.

[0054] Combiner optical element 440 receives first reflected light 448 and second reflected light 449 and redirects light 448 and 449 to a camera module (e.g., camera module 108). Figure 4 , combiner optical element 440 includes a first combiner layer 441 and a second combiner layer 442. First combiner layer 441 is configured to pass visible scene light (e.g., scene light 191) to reach eye 202 and direct first reflected light 448 (having the same wavelength as light 438) to camera module ( Figure 4 438 / 448. The first combiner layer 441 may include a wavelength selective diffractive structure that is specifically configured to redirect the wavelength of light 438 / 448. The second combiner layer 442 is configured to pass visible scene light (e.g., scene light 191) to reach the eye 202 and direct second reflected light 449 (having the same wavelength as light 439) to the camera module. The second combiner layer 442 may include a wavelength selective diffractive structure that is specifically configured to redirect the wavelength of light 439 / 449.

[0055] The first combiner layer 441 may be a holographic optical element (HOE). The first combiner layer 441 may include a polarization-selective volume hologram that reflects a first polarization orientation (e.g., right-handed circularly polarized light) of the first reflected light 448 and passes polarization orientations other than the first polarization orientation. The first combiner layer 441 may also include a folding mirror or a linear diffraction grating to redirect the first reflected light 448. The first combiner layer 441 may be a HOE. The second combiner layer 442 may include a polarization-selective volume hologram that reflects a first polarization orientation (e.g., right-handed circularly polarized light) of the second reflected light 449 and passes polarization orientations other than the first polarization orientation. The second combiner layer 442 may also include a folding mirror or a linear diffraction grating to redirect the second reflected light 449. Figure 4 In the example shown, the first combiner layer 441 is positioned between the lighting layer 430 and the second combiner layer 442 .

[0056] The first combiner layer 441 can selectively redirect the first reflected light 448 to the camera module at a first angle 471, and the second combiner layer 442 can selectively redirect the second reflected light 449 to the camera module at a second angle 472 that is different from the first angle 471. In some aspects, the first angle 471 can be greater than 10 degrees than the second angle 472. In some aspects, the first angle 471 can be greater than 30 degrees than the second angle 472. In some aspects, the first angle 471 can be greater than about 40 degrees than the second angle 472.

[0057] Figure 5 An example image sensor 500 is illustrated that includes an image pixel array 502. Image sensor 500 may include other electronics (e.g., readout circuitry) that are not specifically illustrated. Image sensor 500 may be a complementary metal oxide semiconductor (CMOS) image sensor. Image pixel array 502 includes an array of digital n image pixels P1, P2, P3 ... Pn. Image pixel array 502 is arranged in digital x columns and digital y rows.

[0058] Figure 6An image sensor 600 that can be used in a camera module according to aspects of the present disclosure is illustrated. Image sensor 600 includes an image pixel array 602 that can be configured similarly to image pixel array 502. A first filter 656 covers a first pixel of image pixel array 602 included in a camera module such as camera module 108. The first filter 656 passes (transmits) a wavelength of a first invisible illumination light (e.g., 438) and rejects light of other wavelengths. Therefore, the wavelength emitted by the first illuminator 436 that is ultimately incident on the camera module is passed by the first filter 656 as the first reflected light 448 to reach the first pixel of image pixel array 602. For example, the first filter 656 can be a bandpass filter with a transmission notch of less than 10 nm. A second filter 657 covers a second pixel of image pixel array 602 included in a camera module such as camera module 108. The second filter 657 passes (transmits) a wavelength of a second invisible illumination light (e.g., 439) and rejects light of other wavelengths. For example, the second filter 657 can be a bandpass filter with a transmission notch of less than 10 nm. Therefore, the wavelength emitted by second illuminator 437 that is ultimately incident on the camera module is passed by second filter 657 as second reflected light 449 to reach the second pixel of image pixel array 602 .

[0059] Figure 7 A first filter 756 interspersed with a second filter 757 is illustrated that may be used in a camera module according to aspects of the present disclosure. Figure 7 Image sensor 700 includes image pixel array 702, which may be configured similarly to image pixel array 502. Figure 7 In , first filter 756 covers a first pixel of image pixel array 702 and second filter 757 covers a second pixel of image pixel array 702. Thus, the first pixel of image pixel array 702 is interspersed with the second pixel of image pixel array 702. Figure 7, the first pixel and the first filter 756 are arranged in an every-other or nearest neighbor relationship with the second pixel and the second filter 757. The first filter 756 allows the wavelength of the first invisible illumination light (e.g., 438) to pass (transmit) and rejects (blocks) light of other wavelengths. Therefore, the wavelength emitted by the first illuminator 436 and finally incident on the camera module is passed by the first filter 756 as the first reflected light 448 to reach the first pixel of the image pixel array 702. For example, the first filter 756 can be a bandpass filter with a transmission notch of less than 10nm. The second filter 757 covers the second pixel of the image pixel array 702 included in the camera module such as the camera module 108. The second filter 757 allows the wavelength of the second invisible illumination light (e.g., 439) to pass (transmit) and rejects (blocks) light of other wavelengths. For example, the second filter 757 can be a bandpass filter with a transmission notch of less than 10nm. Thus, in some examples, the wavelength emitted by second illuminator 437 that is ultimately incident on the camera module is passed through second filter 757 to a second pixel of image pixel array 702 as second reflected light 449 .

[0060] exist Figure 6 and Figure 7 In the embodiment shown in , image pixel array 602 or image pixel array 702 can capture a combined image in a single image capture (that is, in a single global shutter integration period or a single rolling shutter integration period), wherein the combined image includes a first eye image generated by a first pixel and a second eye image generated by a second pixel. Figure 6 and Figure 7 An example arrangement is illustrated with respect to a first filter covering a first pixel and a second filter covering a second pixel, but other arrangements may also be used in accordance with aspects of the present disclosure.

[0061] Figure 8 A stereoscopic eye imaging system 800 is illustrated in which a camera module 850 captures a combined image 880 including a first eye image at a first non-visible wavelength and a second eye image at a second non-visible wavelength in accordance with aspects of the present disclosure. The system 800 includes the camera module 850, processing logic 870, a display layer 890, an illumination layer 430, and an optical combiner layer 440. The illumination layer 430 includes a first illuminator 436 and a second illuminator 437. The camera module 850 may include an image sensor 600 or 700.

[0062] In operation, the first illuminator 436 emits a first invisible light 438 toward the eye 202 and the second illuminator 437 emits a second invisible light 439 toward the eye 202. The logic 870 can drive the first illuminator 436 and the second illuminator 437 to illuminate the eye 202 with the first invisible light 438 and the second invisible light 439 simultaneously. The first invisible light 438 can be infrared light. The first invisible light 438 can be near infrared light. In some aspects, the first invisible light 438 can be near infrared light between 700nm and 1000nm. The first invisible light 438 can be narrow-band light with a line width of less than 10nm. The second invisible light 439 can be infrared light. The second invisible light 439 can be near infrared light. In some aspects, the second invisible light 439 can be near infrared light between 700nm and 1000nm. The second invisible light 439 can be narrow-band light with a line width of less than 10nm. A portion of the first invisible light 438 is reflected from the cornea of ​​the eye 202 as a first reflected light 448 that encounters the first combiner layer 441. The first combiner layer 441 redirects the first reflected light 448 to the camera module 850. By diffraction, the first combiner layer 441 can redirect the first reflected light 448 to the camera module 850. A portion of the second invisible light 439 is reflected from the cornea of ​​the eye 202 as a second reflected light 449 that encounters the second combiner layer 442. The second combiner layer 442 redirects the second reflected light 449 to the camera module 850. By diffraction, the second combiner layer 442 can redirect the second reflected light 449 to the camera module 850.

[0063] The camera module 850 receives the first reflected light 448 and the second reflected light 449 and the camera module 850 captures a combined image 880 including both the first reflected light 448 and the second reflected light 449. In some aspects, the first illuminator 436 and the second illuminator 437 simultaneously emit the first invisible light 438 and the second invisible light 439 so that the first reflected light 448 and the second reflected light 449 are incident on the image sensor of the camera module 850 at the same time. The first pixel of the image sensor of the camera module 850 captures the first reflected light 448 and the second pixel of the image sensor captures the second reflected light 449 in the same combined image 880. Since the first pixel is covered with a first filter (e.g., 656 or 756), only light with a wavelength of 438 / 448 is imaged by the first pixel. Similarly, since the second pixel is covered with a second filter (e.g., 657 or 757), only light with a wavelength of 439 / 449 is imaged by the second pixel. The combined image 880 includes a first eye image generated by the first pixel and a second eye image generated by the second pixel.

[0064] Figure 8It is shown that multiple combined images 880 can be captured sequentially as combined images 880A, 880B, 880C and 880D. The eye tracking position of eye 202 can be calculated based on the first eye image and the second eye image extracted from combined image 880. For example, the first eye tracking position of eye 202 can be determined based on the first eye image and the second eye image of combined image 880A. The second eye tracking position of eye 202 can be determined based on the first eye image and the second eye image of combined image 880B, the third eye tracking position of eye 202 can be determined based on the first eye image and the second eye image of combined image 880C, and the fourth eye tracking position of eye 202 can be determined based on the first eye image and the second eye image of combined image 880D. In this way, the position of eye 202 can be continuously tracked. Figure 8 In the illustration of , processing logic 870 is configured to receive combined image 880. Processing logic 870 can extract the first eye image and the second eye image from the combined image and determine the eye tracking position. In some embodiments, the gaze direction of eye 202 can be determined based on the eye tracking position determined from combined image 880. Based on the gaze direction, the virtual image presented to the user by display layer 890 can be changed. Figure 8 8, processing logic 870 can drive a virtual image 893 onto display layer 890 based on a gaze direction obtained from the first and second ocular images of combined image 880. Virtual image 893 is included in display light 899 presented to eye 202. In example system 800, display light 899 propagates through optical combiner layer 440 and illumination layer 430.

[0065] Fig. 9AA stereoscopic eye imaging system 900 is illustrated in accordance with aspects of the present disclosure, wherein a camera module 950 captures a first ocular image and a second ocular image in different image captures. The system 900 includes a camera module 950, processing logic 970, a display layer 890, an illumination layer 430, and an optical combiner layer 440. The illumination layer 430 includes a first illuminator 436 and a second illuminator 437. The illustrated camera module 950 includes a filter 951 configured to filter out light having wavelengths other than the wavelength of light 438 / 448, and to filter out light having wavelengths other than the wavelength of light 439 / 449. The filter 951 may include a plurality of dielectric layers. In an example embodiment where first illuminator 436 emits narrowband light centered at 850 nm and second illuminator 437 emits narrowband light centered at 940 nm, filter 951 may pass light having wavelengths from 849 nm to 851 nm and pass light having wavelengths from 939 nm to 941 nm and block all other wavelengths from being incident on the image sensor of camera module 950. In this example, an image sensor such as image sensor 500 may be used in camera module 950. In one example, filter 951 is positioned over a lens assembly of camera module 950. In other examples, a filter having the characteristics of filter 951 may be overlaid over image pixels of image pixel array 502 such that the filter is positioned between the image pixel array and a lens assembly configured to focus an image to an imaging plane of the image pixel array.

[0066] Fig. 9B An example hybrid schematic timing diagram 995 illustrating example operating techniques of the system 900 in accordance with aspects of the present disclosure is illustrated. At time period t1 , the processing logic 970 selectively drives the output X1 to cause the first illuminator to emit the first invisible light 438 toward the eye 202 . Fig. 9B An example first illuminator 936 is shown, which is illustrated as a diode that can be used as the first illuminator 436. Fig. 9B In the example schematic diagram of , when the output X1 is high, the transistor 963 is turned on, which allows current to flow through the first illuminator 936, thereby causing the emission of the first invisible light 438. In the illustrated example, the first illuminator 936 is arranged in parallel with the voltage source V DD 961 and ground V SS 969. However, in other arrangements, the first illuminator 936 can be arranged in one or more diode strings arranged in series. For example, Fig. 9B The diode in the figure can represent LED, SLED and laser diode (including VCSEL).

[0067] A portion of the first invisible light 438 emitted by the first illuminator 436 / 936 is reflected off the cornea of ​​the eye 202 as the first reflected light 448 that encounters the first combiner layer 441. The first combiner layer 441 redirects the first reflected light 448 to the camera module 950. When the first illuminator 436 / 936 illuminates the eye 202 with the first invisible light 438, the camera module 950 initiates image capture to capture a first ocular image 981. Thus, the first ocular image 981 is captured by the camera module 950 while the first invisible light 438 illuminates the ocular region of the eye 202 and the second invisible light 439 does not illuminate the ocular region. The processing logic 970 drives the output X1 low after the time period t1.

[0068] During time period t2 , processing logic 970 selectively drives output X2 to cause the second illuminator to emit second invisible light 439 toward the eye 202 . Fig. 9B An example second illuminator 937 is shown, which is illustrated as a diode that can be used as the second illuminator 437. Fig. 9B In the example schematic diagram of , when the output X2 is high, the transistor 964 is turned on, which allows current to flow through the second illuminator 937, thereby causing the emission of the second invisible light 439. In the illustrated example, the second illuminator 937 is arranged in parallel with the voltage source V DD 961 and ground V SS 969. However, in other arrangements, the second illuminator 937 may be arranged in one or more diode strings arranged in series.

[0069] A portion of the second invisible light 439 emitted by the second illuminator 437 / 937 is reflected off the cornea of ​​the eye 202 as second reflected light 449 that encounters the second combiner layer 442. The second combiner layer 442 redirects the second reflected light 449 to the camera module 950. When the second illuminator 437 / 937 illuminates the eye 202 with the second invisible light 439, the camera module 950 initiates image capture to capture a second ocular image 982. Thus, the second ocular image 982 is captured by the camera module 950 while the second invisible light 439 illuminates the ocular region of the eye 202 and the first invisible light 438 does not illuminate the ocular region. The processing logic 970 drives the output X2 low after the time period t2.

[0070] During time period t3, the processing logic 970 selectively drives the output X1 so that the first illuminator emits the first invisible light 438 toward the eye 202 and captures the first ocular image 983 in a manner similar to the first ocular image 981. During time period t4, the processing logic 970 selectively drives the output X2 so that the second illuminator emits the second invisible light 439 toward the eye 202 and captures the second ocular image 984 in a manner similar to the second ocular image 982. In this manner, the system 900 captures the first ocular image 438 of the first invisible light wavelength and the second ocular image 439 of the second invisible light wavelength using a time multiplexing technique.

[0071] Fig.10 A flowchart illustrating an example process 1000 for near-eye stereoscopic imaging according to aspects of the present disclosure is illustrated. The order in which some or all of the process blocks appear in process 1000 should not be considered limiting. Rather, one of ordinary skill in the art having the benefit of the present disclosure will appreciate that some of the process blocks may be performed in various orders not illustrated, or even in parallel.

[0072] In process block 1005, the eye is illuminated with a first invisible light wavelength. For example, the first invisible light wavelength may be a near infrared wavelength.

[0073] In process block 1010, the eye is illuminated with a second invisible light wavelength that is different from the first invisible light wavelength. The second invisible light wavelength may also be a near infrared wavelength. In some examples, the first invisible light wavelength may be separated from the first invisible light wavelength by 30 nm or more.

[0074] In process block 1015 , a first ocular image is captured from first reflected light (eg, light 448 ) having a first invisible wavelength.

[0075] In process block 1020, a second ocular image is captured from second reflected light having a second invisible wavelength (e.g., light 449). The first ocular image and the second ocular image are captured using a camera (e.g., camera module 850 or 950) configured to filter out light except for the first invisible wavelength and the second invisible wavelength.

[0076] Process 1000 may also include determining an eye tracking position based on the first ocular image and the second ocular image.

[0077] In an implementation of process 1000, a first intrafield illuminator emits a first invisible light wavelength and a second intrafield illuminator emits a second invisible light wavelength and the first intrafield illuminator and the second intrafield illuminator are included in a near-eye optical element (e.g., near-eye optical element 110 or 210) positioned in a field of view (FOV) of a user.

[0078] In an implementation of process 1000, a first eye image is captured by a camera when a first invisible light wavelength illuminates an eye region and a second invisible light wavelength does not illuminate the eye region, and a second eye image is captured by a camera when a second invisible light wavelength illuminates an eye region and the first invisible light wavelength does not illuminate the eye region. For example, system 900 can be used in this implementation.

[0079] In an implementation of process 1000, a camera that captures a first eye image and a second eye image includes a first filter covering a first pixel of the camera and a second filter covering a second pixel of the camera. The first filter passes a first invisible light wavelength and rejects other light, and the first eye image is generated by the first pixel. The second filter passes a second invisible light wavelength and rejects other light, and the second eye image is generated by the second pixel. For example, system 800 can be used in this implementation.

[0080] Embodiments of the present invention may include an artificial reality system or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), mixed reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of which may be presented in a single channel or multiple channels (such as stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may also be used, for example, to create content in artificial reality and / or otherwise be used in artificial reality (e.g., performing activities therein). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, an independent HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0081] The term "processing logic" (e.g., 870 or 970) in the present disclosure may include one or more processors, microprocessors, multi-core processors, application specific integrated circuits (ASICs), and / or field programmable gate arrays (FPGAs) to perform the operations disclosed herein. In some embodiments, a memory (not shown) is integrated into the processing logic to store instructions to perform operations and / or store data. The processing logic may also include analog or digital circuits to perform operations according to embodiments of the present disclosure.

[0082] One or more "memories" described in the present disclosure may include one or more volatile or non-volatile memory architectures. A "memory" or multiple "memories" may be removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Example memory technologies may include RAM, ROM, EEPROM, flash memory, CD-ROM, digital versatile disk (DVD), high-definition multimedia / data storage disk or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device.

[0083] The communication channel may include or be routed through one or more of the following wired or wireless communications: utilizing IEEE 802.11 protocol, SPI (Serial Peripheral Interface), I 2 C (inter-integrated circuit), USB (universal serial port), CAN (controller area network), cellular data protocol (e.g., 3G, 4G, LTE, 5G), optical communication network, Internet service provider (ISP), peer-to-peer network, local area network (LAN), wide area network (WAN), public network (e.g., the “Internet”), private network, satellite network, or other.

[0084] The computing devices may include desktop computers, laptop computers, tablet computers, phablets, smartphones, feature phones, server computers, etc. The server computers may be located remotely from a data center or stored locally.

[0085] The processes explained above are described in terms of computer software and hardware. The described techniques may constitute machine-executable instructions embodied in a tangible or non-transitory machine (e.g., computer) readable storage medium, which, when executed by a machine, will cause the machine to perform the described operations. In addition, these processes may be embodied in hardware, such as an application specific integrated circuit ("ASIC") or other.

[0086] Tangible, non-transitory machine-readable storage media include any machine that provides (i.e., stores) information in a form readable by a machine (e.g., a computer, a network device, a personal digital assistant, a manufacturing tool, any device having a collection of one or more processors, etc.). For example, machine-readable storage media include recordable / non-recordable media (e.g., read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.).

[0087] The above description of the illustrated embodiments of the present invention, including what is described in the Abstract, is not intended to be exhaustive or to limit the present invention to the precise forms disclosed. Although specific embodiments and examples of the present invention are described herein for illustrative purposes, various modifications may be made within the scope of the present invention as will be appreciated by those skilled in the relevant art.

[0088] These modifications may be made to the invention in light of the above detailed description. The terms used in the following claims should not be interpreted to limit the invention to the specific embodiments disclosed in the specification. Instead, the scope of the invention will be determined entirely by the following claims, which will be interpreted in accordance with established principles of claim interpretation.

Claims

1. A near-eye optical element, comprising: a first illuminator configured to emit a first light having a first invisible wavelength toward an eye region; a second illuminator configured to emit a second light having a second invisible wavelength toward the eye region; as well as a first combiner layer configured to pass visible scene light to the eye region and direct the first light reflected from the eye region to a camera module; Features: The first invisible wavelength is different from the second invisible wavelength; The near-eye optical element further includes a second combiner layer configured to pass the visible scene light to the eye region and direct the second light reflected from the eye region to the camera module; the first illuminator and the second illuminator being intra-field illuminators positioned on an illumination layer of the near-eye optical element, the near-eye optical element being within a field of view FOV of an eye of a user of the near-eye optical element; The lighting layer is positioned between the first combiner and the eye region, and the lighting layer is positioned between the second combiner and the eye region; and The illumination layer is configured to pass visible scene light through the eye region.

2. The near-eye optical element of claim 1 , wherein the first combiner layer selectively directs the first light incident at a first angle to the camera module, and wherein the second combiner layer selectively directs the second light incident at a second angle to the camera module.

3. The near-eye optical element of claim 1 or 2, wherein at least one of the first combiner layer and the second combiner layer comprises a holographic optical element (HOE).

4. A near-eye optical element according to claim 1 or 2, wherein the first illuminator and the second illuminator include at least one of a micro light emitting diode (micro-LED), an edge emitting LED, a vertical cavity surface emitting laser (VCSEL) diode or a superluminescent diode (SLED).

5. A near-eye imaging system, comprising: The near-eye optical element according to claim 1, wherein the light having the first invisible wavelength is a first near-infrared light, and the light having the second invisible wavelength is a second near-infrared light to the eye region; and Camera module.

6. The near-eye imaging system of claim 5, wherein the camera module is configured to capture a first eye image of the first near-infrared light and is configured to capture a second eye image of the second near-infrared light, and the camera module is configured to filter out light other than the first near-infrared light and the second near-infrared light.

7. The near-eye imaging system of claim 6, wherein the camera module comprises: a first filter covering a first pixel of the camera module, wherein the first filter passes the first near-infrared light and rejects other light, the first eye image being generated by the first pixel; as well as a second filter covering a second pixel of the camera module, wherein the second filter passes the second near-infrared light and rejects other light, and the second eye image is generated by the second pixel.

8. The near-eye imaging system of claim 6, further comprising: processing logic configured to selectively activate the first illuminator during a first time period and configured to selectively activate the second illuminator during a second time period different from the first time period, Wherein the camera module is configured to capture the first eye image during the first time period and is configured to capture the second eye image during the second time period.

9. The near-eye imaging system of any one of claims 5 to 8, wherein at least one of the first combiner layer and the second combiner layer comprises a holographic optical element (HOE).

10. The near-eye imaging system of any one of claims 5 to 8, wherein the first combiner layer selectively directs the first near-infrared light incident at a first angle to the camera module, and wherein the second combiner layer selectively directs the second near-infrared light incident at a second angle to the camera module.

11. The near-eye imaging system of claim 10, wherein the first angle is at least 10 degrees greater than the second angle.

12. The near-eye imaging system according to any one of claims 5 to 8, wherein the first illuminator and the second illuminator are in-field illuminators positioned on an illumination layer of the near-eye optical element, and the near-eye optical element is located within the field of view FOV of an eye of a user of the near-eye optical element.

13. A near-eye imaging system according to any one of claims 5 to 8, wherein the first illuminator and the second illuminator include at least one of a micro light emitting diode (micro-LED), an edge emitting LED, a vertical cavity surface emitting laser (VCSEL) diode, or a super luminescent diode (SLED).

14. The near-eye imaging system of any one of claims 5 to 8, wherein the first near infrared light is centered at 850 nm and the second near infrared light is centered at 940 nm.

15. A near-eye stereoscopic imaging method, the method comprising: illuminating an eye with a first invisible wavelength of light, the eye being positioned in an eye region; illuminating the eye with a second invisible wavelength of light; capturing a first ocular image from first reflected light having the first invisible wavelength; as well as capturing a second ocular image from second reflected light having the second invisible wavelength, wherein the first ocular image is captured using a camera configured to filter out light other than the first invisible wavelength using a first combiner layer; Features: The second ocular image is captured by a camera configured to filter out light other than the second invisible wavelength using a second combining layer; The second invisible light wavelength is different from the first invisible light wavelength; a first intrafield illuminator emitting the first invisible light wavelength and a second intrafield illuminator emitting the second invisible light wavelength, the first intrafield illuminator emitting and the second intrafield illuminator being included in an illumination layer of a near-eye optical element, the near-eye optical element being positioned in a field of view FOV of a user; The lighting layer is positioned between the first combiner layer and the eye region; and the lighting layer is positioned between the second combiner layer and the eye region; and The illumination layer is configured to pass visible scene light.

16. The method according to claim 15, further comprising: An eye tracking position is determined based on the first eye image and the second eye image.

17. The method of claim 15, wherein the first eye image is captured by the camera when the first invisible light wavelength illuminates the eye area and when the second invisible light wavelength does not illuminate the eye area, And wherein the second eye image is captured by the camera when the second invisible light wavelength illuminates the eye area and when the first invisible light wavelength does not illuminate the eye area.

18. The method of claim 17, wherein the camera capturing the first ocular image and capturing the second ocular image comprises: a first filter covering a first pixel of the camera, wherein the first filter passes the first invisible light wavelength and rejects other light, the first eye image being generated by the first pixel; as well as A second filter covers a second pixel of the camera, wherein the second filter passes the second invisible light wavelength and rejects other light, and the second eye image is generated by the second pixel.

Images