Apparatus for forming image of user's eye

Abstract

An apparatus for forming an image of a user's eye to track eye movement, comprising: a light guide (20) comprising an angled inner reflector (21); an image sensor (10); and a lens (11) associated with the light guide for focusing the light reflected from the user's eye and reflected by the inner reflector onto the image sensor. The width of the inner reflector, the effective aperture of the lens, and the deployment of the image sensor are such that light from different regions of the eye and reflected by the inner reflector reaches the lens via one of at least three different paths, resulting in discrete non-overlapping image regions on the image sensor. Sub-images derived from these regions of the sensor may be manipulated and combined to form an image of the eye.

Description

Technical Field

[0001] The present invention relates to eye tracking and, in particular, to an apparatus for forming an image of a user's eye via an optical waveguide in a face-to-face relationship with the user's eye. The present invention can be advantageously used in the context of waveguide-based near-eye displays. Background Art

[0002] Many near-eye display systems include a transparent optical waveguide or "waveguide" placed in front of a user's eye, which conveys an image by total internal reflection within the waveguide and then couples out the image towards the user's eye through a suitable output coupling mechanism. The output coupling mechanism can be based on embedded partially reflective surfaces or "facets", or can employ a diffraction pattern.

[0003] Some waveguide-based displays employ a waveguide device that implements an expansion of the optical aperture of an image projector in two dimensions to use a micro-projector to provide a much larger viewing area to the eye. The two-dimensional expansion can be achieved by employing an additional set of embedded partially reflective surfaces within the same waveguide (e.g., as disclosed in PCT Patent Application Publication No. WO2020 / 049542 A1), or by employing a separate rectangular waveguide (e.g., as disclosed in PCT Patent Application Publication No. WO 2018 / 065975 A1), and can include various combinations of reflective and diffractive optical elements (as disclosed in PCT Patent Application Publication No. WO 2018 / 154576 A1).

[0004] For many applications, it is desirable to track eye movements while a user views a near-eye display. For this purpose, it is desirable to obtain an image of the eye via an optical waveguide placed in a face-to-face relationship with the eye, which can be the waveguide of the display or an additional dedicated waveguide. Summary of the Invention

[0005] The present invention is an apparatus for forming an image of a user's eye via an optical waveguide in a face-to-face relationship with the user's eye.

[0006] In accordance with the teachings of embodiments of the present invention, there is provided an apparatus for forming an image of a user's eye to track eye movement. The apparatus includes: (a) an optical waveguide formed of a transparent material having a first major surface and a second major surface, the first major surface and the second major surface being planar and parallel to each other, the first major surface being deployed in a facing relationship with the user's eye such that the user views a scene through the optical waveguide; (b) an angled internal reflector deployed within the optical waveguide; (c) an image sensor including a two-dimensional array of pixel sensors; and (d) a lens associated with the optical waveguide for focusing light reflected from the user's eye and reflected by the internal reflector onto the image sensor, wherein the width of the internal reflector, the effective aperture of the lens, and the deployment of the image sensor are such that: (i) light reflected from the user's eye and reflected by the internal reflector and reaching the lens without being reflected from the first major surface or the second major surface is incident on a first region of the image sensor; (ii) light reflected from the user's eye and reflected by the internal reflector and reaching the lens after a single reflection from the first major surface and without being reflected from the second major surface is incident on a second region of the image sensor; and (iii) light reflected from the user's eye and reflected by the internal reflector and reaching the lens after a single reflection from the second major surface and without being reflected from the first major surface is incident on a third region of the image sensor, the first region, the second region, and the third region of the image sensor being non-overlapping.

[0007] In accordance with another feature of embodiments of the present invention, the internal reflector has a reflectivity of at least 20% for at least one wavelength of infrared light incident perpendicular to the major surface and a reflectivity of less than 10% for most of the spectrum of visible light incident perpendicular to the major surface.

[0008] In accordance with another feature of embodiments of the present invention, there is also provided an electronic device associated with the image sensor, the electronic device generating a digital image including a first sub-image corresponding to light sensed in the first region of the image sensor, a second sub-image corresponding to a reflected image of light sensed in the second region of the image sensor, and a third sub-image corresponding to a reflected image of light sensed in the third region of the image sensor.

[0009] According to another feature of an embodiment of the present invention, the width of the internal reflector, the effective aperture of the lens, and the deployment of the image sensor are also configured such that: (i) light from the user's eye that is reflected by the internal reflector and reaches the lens after a single reflection from the first main surface and then a single reflection from the second main surface is incident on a fourth region of the image sensor; and (ii) light from the user's eye that is reflected by the internal reflector and reaches the lens after a single reflection from the second main surface and then a single reflection from the first main surface is incident on a fifth region of the image sensor, where the fourth region and the fifth region do not overlap with each other and do not overlap with each of the first region, the second region, and the third region.

[0010] According to another feature of an embodiment of the present invention, an electronic device associated with the image sensor is also provided, which generates a digital image that includes a superposition of the following sub-images: a sub-image corresponding to the light sensed in the first region, the fourth region, and the fifth region of the image sensor, and a sub-image corresponding to the reflected image of the light sensed in the second region and the third region of the image sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present invention is described herein by way of example only with reference to the drawings, in which:

[0012] Figures 1A to 1C is a schematic side view of a device for forming an image of a user's eye via an optical waveguide in a face-to-face relationship with the user's eye, showing the optical paths from three different regions of an Eye Motion Box (EMB) respectively;

[0013] Figures 2A to 2C shows the optical paths from Figure 1A an optical device, showing the optical paths from three closely spaced points within the EMB around the blind spot of the detector respectively;

[0014] Figure 3A is a test image deployed at the EMB for the purpose of simulation shown in subsequent images;

[0015] Figure 3B is when Figure 3A the test image is located at the EMB, showing the output simulation of the image sensor of the device from Figure 1A ;

[0016] Figure 4A is Figure 3B the inversion of to correct the inversion of the central lobe of the image and the replicated version of the specular reflection;

[0017] Figure 4B is obtained by flipping, swapping, and shifting fromFigure 4A Reconstruction of the source image generated from the sidelobe sub-images;

[0018] Figure 5 is Figure 1A A variant implementation of the device;

[0019] Figure 6A is when Figure 3A The test image of is located at the EMB, and it is an analog of the output of the image sensor of the device from Figure 5 This image has a central lobe, first-order sidelobes, and second-order sidelobes;

[0020] Figure 6B is the reconstruction of the source image generated by flipping, swapping, and shifting the first-order sidelobe sub-images from Figure 6A plus the global inversion of the image;

[0021] Figure 7A and 7B are images similar to Figure 6A respectively showing the effects of shifting the internal reflector of the device by ±1.25 mm;

[0022] Figure 8 is Figure 1A Another variant implementation of the device;

[0023] Figure 9A is when Figure 3A The test image of is located at the EMB, and it is an analog of the output of the image sensor of the device from Figure 8 This image has partially redundant content;

[0024] Figure 9B is Figure 9A The inversion of, plus the flipping and swapping of the sidelobes, such that the image redundancy appears in the region adjacent to the blind line of the detector;

[0025] Figure 9C is the reconstruction of the source image generated by the weighted summation of the inward displacement of the sidelobes of Figure 9B and the pixel values in the overlapping region;

[0026] Figure 10 is a graph showing the variation of the overlap (correlation) parameter as a function of the inward pixel displacement of the sidelobes of Figure 9B for three different estimated exit pupil distances;

[0027] Figure 11 is a schematic side view of an optical waveguide showing a simplified geometric analysis for determining the dimensions of the internal reflector and lens aperture for certain implementations of the present invention; and

[0028] Figure 12A and Figure 12BYes Figure 1A Schematic side view of another variant implementation of the device, showing the use of a narrow folding mirror and a wide folding mirror respectively to fold the optical path to allow the imaging device to be deployed on the main surface of the light guide. Detailed implementation

[0029] The present invention is a device for forming an image of a user's eye via a light guide in a face-to-face relationship with the user's eye.

[0030] The principles and operations of the device according to the present invention can be better understood with reference to the accompanying drawings and the attached description.

[0031] Now referring to the accompanying drawings, Figures 1A to 1C There is shown a device for forming an image of a user's eye for tracking eye movement, the device being located within an eye movement box (EMB) 100. Generally, the device includes a light guide 20 formed of a transparent material, the light guide 20 having a first main surface 22 and a second main surface 23, the first main surface 22 and the second main surface 23 being planar and parallel to each other. The first main surface 22 is deployed in a face-to-face relationship with the user's eye (EMB 100) such that the user views the scene through the light guide. The device further includes an angled internal reflector 21 deployed within the light guide 20, an image sensor 10 having a two-dimensional array of pixel sensors, and a lens 11 associated with the light guide 20 for focusing light reflected from the user's eye and reflected by the internal reflector 21 onto the image sensor 10. The width and angle of the internal reflector 21, the effective aperture of the lens 11, and the deployment of the image sensor 10 are such that light reflected from the user's eye follows various discrete paths to reach discrete regions of the image sensor 10. Specifically, a first portion of the light reflected from the user's eye is reflected by the internal reflector 21 and reaches the lens 11 without being reflected from the first main surface 22 or the second main surface 23 (as Figure 1A shown) and is incident on a first region of the image sensor 10. A second portion of the light reflected from the user's eye is reflected by the internal reflector 21 and, after a single reflection from the first main surface 22 and without being reflected from the second main surface 23, is reflected to reach the lens 11 (as Figure 1B shown) and is incident on a second region of the image sensor 10. A third portion of the light reflected from the user's eye is reflected by the internal reflector 21 and, after a single reflection from the second main surface 23 and without being reflected from the first main surface 22, reaches the lens 11 (as Figure 1C shown) and is incident on a third region of the image sensor 10. The first region, the second region, and the third region of the image sensor are non-overlapping.

[0032] In this non - limiting example, the lens 11 is 40% of the width of the light guide. Thus, in the example shown, the width of the light guide is 2 mm and the width of the lens is 0.8 mm. As in any imaging device (although not shown), the imaging device should be sealed to allow only the light transmitted through the lens to reach the detector (image sensor 10). Imaging is achieved via the light guide 20 placed at approximately 20 mm from the EMB 100 (which is a typical exit pupil distance for a pair of glasses), but the device can be optimized for any other desired exit pupil distance depending on the intended application.

[0033] The internal reflector (mirror) 21 is preferably oriented such that the chief ray of the center of the EMB propagates parallel to the main surface of the light guide. In the case of an upright light guide perpendicular to the line of sight, the internal reflector will thus be at approximately 45 degrees to the main surface. In the case of a horizontally - extending reflector in a light guide deployed at a wide - angle tilt, or in the case of a vertically - extending internal reflector deployed at a face - curve angle (and imaging at the lateral side of the light guide), this preferred orientation of the internal reflector can vary from 45 degrees. In the preferred implementation shown here, the width of the internal reflector is slightly less than 40% of the maximum possible width inside the light guide, i.e., spanning slightly less than 40% of the thickness of the light guide. In the example shown here, the width of the mirror is approximately 1.1 mm and, when deployed at 45 degrees, the mirror spans 0.78 mm in the thickness direction of the light guide.

[0034] The internal reflector can be implemented as any suitable reflector, including but not limited to a metal coating or a multi - layer dielectric coating. In a particularly preferred case, the internal reflector is implemented as a multi - layer dielectric coating that is configured to be substantially transparent (reflectivity less than 10%, and preferably less than 5%) for visible light incident perpendicular to the main surface across most of the visible spectrum, and is more significantly reflective (reflectivity of at least about 20%, and preferably greater than 30%) for at least one wavelength of infrared light incident perpendicular to the main surface of the light guide. Such a reflector is sometimes referred to as a “heat mirror” which is reflective for IR and transparent for visible light.

[0035] Most preferably, the mirror should be transparent in the visible spectrum not only for the wearer himself but also for a person standing in front of the wearer and observing the glasses at a relatively wide angular span. Assuming an observation angle of ±30 degrees, this angle corresponds to approximately ±20 degrees inside the light guide material, and assuming that the internal reflector is at an angle of approximately 45 degrees to the light guide surface, this angle corresponds to an incident angle of 45 ± 20 degrees or 25 degrees to 65 degrees. Ideally, therefore, the mirror has a low reflectivity (preferably less than 10%, and more preferably less than 5%) across at least most of the visible spectrum in the incident angle range from 25 degrees to 65 degrees, and has a partial reflectivity at wavelengths in the NIR range (e.g., approximately 900 nm), with a reflectivity of at least approximately 20%, and preferably at least approximately 30% in the incident angle range between approximately 35 degrees and approximately 55 degrees.

[0036] The present invention preferably operates by actively illuminating the user's eyes at one or more wavelengths in the Near Infrared (NIR) region within the EMB, and the reflectivity of the internal reflector, the sensitivity of the image sensor, and possibly additional components (such as a selective passband or notch filter (not shown) deployed in front of the image sensor) are matched to one or more wavelengths. Details of the illumination device are not shown in the drawings, but may include illumination via LEDs at the periphery of the light guide or small LEDs embedded inside the light guide, or illumination via a dedicated light guide (not shown).

[0037] As already mentioned, Figures 1A to 1C Three different light paths from different regions of the EMB 100 to different regions of the image sensor 10 are shown. In Figure 1A , the light originating from the center of the EMB is incident on the folding mirror (internal reflector) 21 and reaches the imaging lens 11 without being incident on the main surface of the light guide 20. Figure 1B And Figure 1C show the same structure, which respectively show the paths of light from the edges 5 mm above and below the center of the EMB. The light from these regions of the EMB is reflected once from one of the main surfaces and then reaches the imaging lens. The narrowness of the folding mirror and the imaging lens ensures that the light rays propagating along each path form different "lobes" of an image that fall on a defined region of the image sensor and do not mix. This will be further explained with reference to Figures 2A to 2C below.

[0038] Figure 2A shows the light rays emerging from position 102 inside the EMB 100 reaching the mirror and entering the lens 11 via reflection from the first main surface 22 of the light guide. In Figure 2BIn [the figure], a light beam that exits from position 104 and is incident on the folding mirror 21 can be seen. It can be seen here that all the rays of the light beam do not fall within the aperture of the lens 11. At position 106( Figure 2C ), a ray that is incident on the edge of the mirror 21 and propagates directly towards the lens 21 is shown. Therefore, position 104 is a blind spot of the system, and the image flips between the two sides of this position.

[0039] Figure 3B Shows the resulting image sensed by the detector 10 when a test image provided at EMB 100 Figure 3A is present. As shown in the figure, the output image is divided into 3 parts ("lobes"), where the central part corresponds to the position in the EMB that reaches the lens directly without being reflected from the main surface, but is inverted (as if any single-lens imaging system is flipped), and the central part is also flipped by the specular reflection of the internal reflector 21. The two side lobes of the image correspond to such positions in the EMB: the light from these positions reaches the lens 11 after a single reflection at one of the main surfaces, which causes a further mirror image flip of the part of the image corresponding to these positions, making the letters correct. It is worth noting that although the input test image is continuous, there are discontinuities between the different lobes of the sensed image.

[0040] In many applications, it is desirable to process the detected image to reconstruct an approximate image that truthfully reproduces the EMB. Such an image can be used as a direct input to conventional eye-tracking software. For this purpose, the device of the present invention preferably further includes electronics (such as the processing system 30( Figures 1A to 1C )) associated with the image sensor 10, which generates a digital image that includes a first sub-image corresponding to the light sensed in the first region of the image sensor, a second sub-image corresponding to the reflected image of the light sensed in the second region of the image sensor, and a third sub-image corresponding to the reflected image of the light sensed in the third region of the image sensor.

[0041] This is intuitively shown in Figure 4A and Figure 4B . The discussion of image manipulation for reconstructing the entire image of the EMB herein ignores any global manipulation applied to the entire image. Therefore, Figure 4A shows Figure 3B the detected image after being rotated 180 degrees and horizontally flipped (which is geometrically equivalent to simply vertically flipping Figure 3B ), which causes the central lobe of the image to be upright. The two side lobes are identified by two dashed rectangles labeled 12 and 13. In Figure 4BThe corresponding unfolded image is shown, where two side lobes are flipped from left to right, the positions of the two side lobes are swapped, and the two side lobes are realigned with the central lobe. The resulting image is almost continuous, except for the small blind spots (dark vertical lines) between the lobes mentioned above, and the intensity drop towards the blind spots. The blind spots correspond to Figure 2B the position 104 shown.

[0042] The electronic device mentioned above can be implemented as a processing system 30, which generally includes one or more processors 32, a data storage component 34, and a driver 36 or other hardware, software, or firmware required to interface with the image sensor and any other components of the system. In many cases, the components of the processing system can also be components for performing eye tracking processing and / or for operating the display and / or other components of the system in which the present invention is deployed. In some cases, the electronic device can be simply implemented in the hardware or firmware of the image sensor by changing the read sequence of the columns of the sensor array to correspond to Figure 4B the transformation of.

[0043] It should also be noted that if a trained neural network is used for eye tracking processing, it may not be necessary to reconstruct the complete image, because the artificial intelligence can be trained to directly obtain the eye position from the original image (e.g., Figure 4A the original image of), without the need for the inversion and realignment required to generate the reconstructed image.

[0044] Therefore, the example shown results in an image formed by three image lobes, where the light reaching the image sensor is restricted to the optical path directly from the inner reflector 21 to the lens 11, or the optical path that has undergone exactly one reflection from one of the main surfaces of the light guide 20. However, the present invention is not limited to such an implementation, and in other cases, it can include optical paths that have undergone more reflections. As a non-limiting specific example, Figure 5 an implementation is shown in which the width of the inner reflector 21, the effective aperture of the lens 11, and the deployment of the image sensor 10 are such that, in addition to including the above-mentioned paths with zero and one reflection, it also includes additional optical paths that reach the sensor after two reflections from the EMB (one reflection from each of the main surfaces of the light guide). Therefore, as Figure 5As shown, the light from position 108 within the EMB 100 is reflected by the internal reflector 21 and reaches the lens 11 after a single reflection from the first main surface 22 and then a single reflection from the second main surface 24, and is focused by the lens 11 onto the fourth region of the image sensor 10. Similarly, the light from position 110 within the EMB 100 is reflected by the internal reflector 21 and reaches the lens 11 after a single reflection from the second main surface 23 and then a single reflection from the first main surface 22, and is focused onto the fifth region of the image sensor 10. To keep the drawing clear, the optical path from position 110 is not shown, but it is similar to the optical path from position 108. As previously described, the position 112 in the EMB is reflected by the internal reflector 21 to reach the lens 11 without any reflection from either main surface (forming the first region of the image), and the position 114 (and another position between 110 and 112) is reflected to only experience a single reflection from one of the main surfaces before reaching the lens 11 (forming the second and third regions of the image). The fourth region and the fifth region do not overlap with each other, and do not overlap with each of the above-mentioned first region, second region, and third region. The optical paths of the two reflections are non-reversed.

[0045] The geometric differences between the above-mentioned three-lobe image implementation and this five-lobe image implementation mainly lie in the thickness of the light guide, the length of the optical path from the internal reflector to the lens, and the size of the image sensor. These parameters are preferably selected such that the image will span the expected EMB size with the expected minimum exit pupil distance.

[0046] In Figure 6A the resulting image is shown, in which the image is divided into 5 lobes, where the flipped lobes are indicated by two dashed rectangles. The flipped lobes (i.e., the lobes flipped relative to the central lobe) correspond to the optical paths that exactly experience one reflection at one of the main surfaces of the light guide, while the other three lobes all have the same orientation. Figure 6B The reconstructed complete image is shown (after correctly orienting the central lobe by flipping it vertically), where the two flipped lobes are reversed from left to right, the positions of the two flipped lobes are interchanged, and the spaces between different lobes are removed. Such an image can be assembled by the above-mentioned electronic device, which generates a digital image that includes the superposition of the following sub-images: sub-images corresponding to the light sensed in the first region, fourth region, and fifth region of the image sensor, and sub-images corresponding to the reflected images of the light sensed in the second region and third region of the image sensor.

[0047] The blind spots of the imaging examples shown so far are not necessarily sufficient to affect the accuracy of eye tracking, but one or more additional features can preferably be employed to further reduce any impact that the blind spots may have. The first approach to this problem involves binocular tracking applications. For most people, the directions of gaze of the two eyes converge towards a single point in space (or are the same for distant objects). Thus, the tracking information from one eye can be used to supplement the missing tracking information of the second eye to estimate the overall direction of gaze. To this end, the blind spots of two devices deployed to image the user's two eyes can be offset relative to each other. This can be easily achieved by implementing two different distances between the folding mirror 21 and the imaging lens 11 and / or by slightly changing the tilt angle of the folding mirror 21, thereby generating images with different offsets. This is shown in Figure 7A and Figure 7B and two resulting images with shifts of +1.25 mm and -1.25 mm are shown in Figure 7A and Figure 7B As shown, the blind spots between the different images are shifted. For example, the letter R is cut in Figure 7A (shift -1.25) and appears completely in Figure 7B (shift 1.25). The letters U and L appear in Figure 7A and are cut in Figure 7B .

[0048] In the case of binocular eye tracking, the two images will be images of different eyes and may be misaligned relative to the corresponding EMBs, so the images will not be directly combined. However, it is assumed that any features that are not clear enough in the corresponding image of one eye to allow precise eye tracking will be clearly visible to the second eye in the second image, thereby facilitating continuous and highly accurate tracking of the overall eye movement.

[0049] In all the embodiments shown in Figures 1A to 2C and Figure 5 and in the figures further described below, the optical elements are shown in side view. In the width dimension (as shown entering the page), the internal reflector 21 preferably extends sufficiently to fill the field of view of the imaging device defined by the lens 11 and the image sensor 10, while the lens 11 (or some other aperture-defining element located near the lens) defines the optical aperture of the imaging system. For simplicity of construction, the internal reflector 21 can be implemented across the entire size of the light guide 10.

[0050] In the examples so far, the optical device is such that for each point in the EMB 100, all the light rays from that point can reach the imaging camera device via only a single type of optical path (with zero, one, or two reflections from the main surface of the light guide). However, as Figure 3Band Figure 6A As shown, such a device creates dark spaces between different lobes of the detected image on the sensor. Optionally, the size of the aperture defined by the folding mirror 21 and / or the imaging lens 11 can be increased such that light from some regions of the EMB can reach the image detector via more than one optical path, but there is still no overlap of any image regions on the detector. Such a configuration is shown in Figure 8 where both the mirror and the detector are increased in size. In this example, light rays from point 114 within the EMB 100 reach the detector along two different optical paths such that this region will appear in two different lobes of the image. In Figure 9A the resulting image on the detector is shown. It can be seen that there is no overlap between different lobes of the image on the detector, but some parts of the EMB appear twice on the detector. For example, a part of the letter E appears twice on the detector.

[0051] Figure 9B shows Figure 9A after a global vertical flip to correct the central lobe and invert and swap the two side lobes of Figure 9A . It can be seen that the middle lobe and the side lobes share some overlapping content. To generate a faithful reproduction of the image at the EMB, the side lobes and the central lobe should be properly aligned (e.g., by shifting the side lobes inwards), and the content of the side lobes and the central lobe should be added or otherwise combined. After such a shift and weighted summation, the resulting reconstructed image is shown in Figure 9C . It will be noted that the dark (blind) lines of Figure 4B and Figure 6B are eliminated here.

[0052] The exact amount of side lobe shift required to align the side lobes with the central lobe depends on several parameters of the system, such as focal length, light guide thickness, distance between the mirror and the lens, etc. Most of these parameters are characteristics of the system that do not change and can therefore be easily pre-calibrated. However, due to the parallax between the edges of the internal reflector, the shift also depends slightly on the eye-relief (ER) distance of the system. Although the ER usually does not change significantly during the use of glasses, the ER may vary between different users and may even vary for the same user wearing glasses on different occasions.

[0053] Thus, for some applications, a fixed pre-calibrated offset correction can be used and acceptable results can be provided. In cases where further optimization is needed, a simple image correlation algorithm can be used to determine the optimal displacement required. For the purposes of the completeness of the present disclosure, an algorithm that can be used for this purpose is presented herein by way of non-limiting example. However, it should be understood that various other known algorithms for image correlation can be used for this purpose, as will be apparent to those of ordinary skill in the art. Optionally, this algorithm can be used to pre-determine the optimal correction for various different values of the ER distance, and then, when the algorithm is executed for a given user (typically each time the system is turned on and / or whenever the system is re-positioned with respect to the face), the output of the algorithm indicates both the required offset for combining the image lobes and the current ER distance.

[0054] In the following example, the simulation of the algorithm generates an offset correction of 48 pixels for an ER of 24 mm, an offset correction of 52 pixels for an ER of 20 mm, and an offset correction of 58 pixels for an ER of 16 mm. A simple algorithm is used to determine the optimal position, where the overlap (OVL) is defined as

[0055] OVL(d) = ∑ ij (CL i，j-d + SL i，j ) 2

[0056] where i represents the row number, j represents the column number, d represents the displacement between the images, CL is the central lobe, SL is the side lobe, and OVL represents the overlap or correlation factor to be optimized.

[0057] Figure 10 Shows the parameter OVL calculated as a function of pixel displacement for three different ER distances of the system. As shown, the maximum value can be easily identified for each simulation, which indicates both the correct displacement and the current ER of the user.

[0058] Determining the ER distance of the current user can be useful for the operation of the eye tracking function itself and can also be a useful by-product for other aspects that contribute to the operation of the head-mounted display. For example, when the ER changes with the focusing distance, knowledge of the ER distance can facilitate determining the user's gaze distance.

[0059] Various considerations affect the choice of the size of the internal reflector 21 and the aperture of the lens 11. As described above, the use of a relatively narrow reflector and a small-aperture lens ensures that each image point appears at only one position in the detector plane, but results in low utilization efficiency of the detector area and is affected by blind spots. By increasing the width of the internal reflector and the lens aperture, Figures 9A to 9CIn the case where the EMB portion appears twice at the detector plane, but the overlap between different lobes of the image is still avoided. This enables better utilization of the detector plane and allows for the elimination of blind spots. However, it is important to keep the reflector and lens small enough to avoid overlap between the lobes on the detector plane, as this overlap would clutter the image content and cause a sharp deterioration in the output.

[0060] Figure 11 Geometric considerations for avoiding overlap of image lobes in the detector plane are schematically shown using the simplification that the assumed imaging distance (exit pupil distance) is long (i.e., the light rays from the image are assumed to be parallel).

[0061] The minimum distance between the specified mirror and the detector is designated as d. The width of the detector is designated as w1. The width of the light guide is w2 and the width of the light guide is w. Assuming that the detector and the mirror are centered within the light guide, the remaining distance between the edge of the mirror and the main surface of the light guide can be determined as And similarly, the detector aperture is

[0062] Using the simplification according to which, assuming that the camera device pupil is set to look at infinity, it is desired to prevent a single point on the detector from receiving light rays from two different positions in the EMB, which would occur if the direct light ray from the internal reflector to the lens aperture and the once-reflected light ray could enter the lens at the same angle. Therefore, observing the propagation of the light rays exiting from the upper corner of the mirror and incident on the two edges of the aperture, such that two different angled incident light rays will reach the same point on the detector, as Figure 11 shown.

[0063] For the left-side light ray, its propagation angle is given by For the right-side light ray 2, a similar equation can be obtained This is clearly shown in Figure 11 when reflecting the light rays incident on the light guide around the main surface. Assuming that, as before, the pupil of the lens and the mirror are centered, it can be written: Comparing the two equations gives

[0064]

[0065] w = w1 + w2

[0066] In other words, according to this simplification, if the sum of the thickness spanned by the internal reflector 21 and the width of the lens aperture is less than the thickness of the light guide, the overlap of the image on the detector plane should be avoided.

[0067] In fact, the imaging distance is not infinite. For a relatively short imaging distance, the sum of w1 + w2 can generally be greater than w without causing image lobe overlap. Therefore, suitable values for the width of the internal reflector and the lens aperture should preferably be determined by numerical and / or empirical methods based on simulation and / or laboratory experiments, as will be apparent to those of ordinary skill in the art.

[0068] The above examples have depicted the imaging device configuration as linear or "in-plane", where the lens and the image detector are deployed to collect image light exiting from the edge of the light guide. In some implementations, it may be more convenient to use another mirror to fold the optical axis of the imaging device. Two options for such an implementation are shown in Figure 12A and Figure 12B

[0069] In Figure 12A , a folding mirror 40 is deployed to fold the optical path in front of the lens 11 and the detector 10. The mirror 40 can be narrow and serve as the aperture stop of the lens 11 in at least one dimension, thereby defining a desired optical architecture similar to Figure 1A or similar to Figure 5 . Alternatively, as shown in Figure 12B , a wider mirror 222 can be provided, which spans the width of the light guide and folds all the light rays propagating towards the lens 11 within the light guide. In this case, the lens itself (or the associated aperture stop) serves as the stop. In all other respects, the implementations of Figure 12A and Figure 12B are similar in structure and function to the above Figure 1A and Figure 5

[0070] In all of the above examples, the internal reflector 21 preferably passes through the central portion of the light guide 20 to capture an image of the eye near the center of the field of view. Thus, the light guide is typically supported by a support structure (e.g., a glasses frame supported on the user's nose and ears, or a head-mounted goggle), which supports the light guide in a face-to-face relationship with each of the user's eyes, and the internal reflector preferably passes through within about 20 degrees, and more preferably within about 10 degrees, of the center of the user's field of view, which corresponds to the axis along which the user views the external world when looking straight ahead. This central positioning provides a frontal view of the eye, which simplifies the eye tracking process and / or improves the eye tracking accuracy. The device of the present invention is preferably combined with a display that delivers a visual image to the user's eye via reflective coupling output or diffractive coupling output of the image through the same light guide as the device of the present invention or through another parallel light guide, all of which are known in the art.

[0071] It should be understood that the above description is only intended to be exemplary, and it should be understood that many other embodiments are possible within the scope of the invention as defined in the appended claims.

Claims

1. An apparatus for forming an image of a user's eye to track eye movement, the apparatus comprising: (a) An optical waveguide formed of a transparent material, the optical waveguide having a first major surface and a second major surface, the first major surface and the second major surface being planar and parallel to each other, the first major surface being deployed in a facing relationship with the user's eye such that the user views a scene through the optical waveguide; (b) An angled internal reflector deployed within the optical waveguide; (c) An image sensor including a two-dimensional array of pixel sensors; And (d) A lens associated with the optical waveguide, the lens for focusing light reflected from the user's eye and reflected by the internal reflector onto the image sensor, wherein the width of the internal reflector, the effective aperture of the lens, and the deployment of the image sensor are such that: (i) Light reflected from the user's eye by the internal reflector and reaching the lens without being reflected from the first major surface or the second major surface is incident on a first region of the image sensor; (ii) Light reflected from the user's eye by the internal reflector and reaching the lens after a single reflection from the first major surface and without being reflected from the second major surface is incident on a second region of the image sensor; and (iii) Light reflected from the user's eye by the internal reflector and reaching the lens after a single reflection from the second major surface and without being reflected from the first major surface is incident on a third region of the image sensor, the first region, the second region, and the third region of the image sensor being non-overlapping.

2. The device according to claim 1, wherein, The internal reflector has a reflectivity of at least 20% for at least one wavelength of infrared light incident perpendicular to the major surface and a reflectivity of less than 10% for most of the spectrum of visible light incident perpendicular to the major surface.

3. The apparatus according to claim 1, further comprising an electronic device associated with the image sensor, the electronic device generating a digital image, the digital image including a first sub-image corresponding to light sensed in the first region of the image sensor, a second sub-image corresponding to a reflected image of light sensed in the second region of the image sensor, and a third sub-image corresponding to a reflected image of light sensed in the third region of the image sensor.

4. The device according to claim 1, wherein, The width of the internal reflector, the effective aperture of the lens, and the deployment of the image sensor are further configured such that: (i) Light reflected from the user's eye by the internal reflector and reaching the lens after a single reflection from the first major surface followed by a single reflection from the second major surface is incident on a fourth region of the image sensor; And (ii) Light reflected from the user's eye by the internal reflector and reaching the lens after a single reflection from the second major surface followed by a single reflection from the first major surface is incident on a fifth region of the image sensor, The fourth region and the fifth region do not overlap with each other and do not overlap with each of the first region, the second region, and the third region.

5. The apparatus according to claim 4, further comprising an electronic device associated with the image sensor, the electronic device generating a digital image, the digital image including a superposition of the following sub-images: a sub-image corresponding to light sensed in the first region, the fourth region, and the fifth region of the image sensor, and a sub-image corresponding to a reflected image of light sensed in the second region and the third region of the image sensor.

Images