Eye tracking system and virtual image display device

Abstract

The present invention addresses the problem of providing: a line-of-sight tracking system which can easily detect line-of-sight; and a virtual image display device using the line-of-sight tracking system. The problem is solved by having an infrared light source array, a virtual image generation optical system, and an infrared light detector, wherein infrared light sources of the infrared light source array are sequentially turned on, the infrared light is collimated by the virtual image generation optical system, the collimated infrared light is caused to be incident on the eyes of a user at different angles, and among the infrared light incident on the eyes, infrared light, incident on the retinas from the pupils and reflected by the retinas, is detected by the infrared light detector.

Description

[Technical Field]

[0001] The present invention relates to an eye-tracking system used in a head-mounted display or the like, and a virtual image display device equipped with this eye-tracking system. [Background technology]

[0002] BACKGROUND ART Head-mounted displays (HMDs), AR glasses, and the like have been put to practical use as means for providing users with virtual reality (VR) and augmented reality (AR).

[0003] VR systems that provide VR and AR systems that provide AR are expected to be equipped with a function to detect the user's line of sight. By detecting the user's line of sight, it is possible to know what the user is observing, and various processes can be performed accordingly, such as displaying what the user is observing in detail, emphasizing what the user is observing, focusing on what the user is observing, displaying what the user is observing in high resolution, and using the user's line of sight as a pointing device. This will improve the functionality of HMDs, AR glasses, etc., and make it possible to realize higher performance HMDs, AR glasses, etc.

[0004] In gaze detection in HMDs and AR glasses, invisible light such as infrared light is shone on the user's eyes, the reflected light is photographed, and the resulting image is analyzed to detect the user's gaze. For example, Patent Document 1 describes an HMD with a user's gaze detection function, which includes a convex lens that is positioned opposite the user's cornea when the HMD is worn by the user, multiple infrared light sources that are positioned around the convex lens and irradiate infrared light toward the user's cornea, a camera that captures images including the user's cornea, and a housing to house these components, and when the periphery of the convex lens is equally divided into a first region that is the area on the outer corner of the user's eye, a second region that is the area on the inner corner of the eye, a third region that is the area on the top of the head, and a fourth region that is the area on the chin, the infrared light source is positioned in the first region or the second region. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2016-157485 Summary of the Invention [Problem to be solved by the invention]

[0006] The HMD described in Patent Document 1 detects the line of sight (direction of line of sight) of the user and uses this as a pointing device, thereby improving the convenience of the HMD.

[0007] Here, gaze detection, including that described in Patent Document 1, which has conventionally been installed in HMDs and AR glasses, detects the gaze by irradiating the user's eye (eyeball) with invisible light such as infrared light and analyzing the reflected image of the light reflected by the eyeball. For example, conventional gaze detection involves irradiating the eye with infrared light and analyzing the invisible light reflected by the anterior cornea, the anterior and posterior surfaces of the lens, and the posterior surface of the cornea. These reflected images are called Purkinje images. However, gaze detection using Purkinje images or the like has the problem of requiring complex calculations and imposing a heavy load.

[0008] The gaze of a user of an HMD or other device moves at an extremely fast speed. Therefore, when complex calculations are being performed, the gaze detection system may not be able to keep up with the gaze movement. If the gaze detection cannot keep up with the movement of the gaze, the above-mentioned processes such as highlighting what the user is observing and the function as a pointing device will not be performed properly.

[0009] The object of the present invention is to solve the problems of the conventional technology by providing an eye-tracking system that can easily detect the user's gaze in HMDs, AR glasses, etc. without performing complex calculations, and a virtual image display device that uses this eye-tracking system. [Means for solving the problem]

[0010] To achieve this object, the present invention has the following configuration. [1] An infrared light source array, a virtual image generating optical system, and an infrared light detector; An eye tracking system characterized by sequentially turning on infrared light sources of an infrared light source array, collimating the infrared light using a virtual image generating optical system, directing the collimated infrared light into the user's eye at different angles, and detecting the infrared light that enters the eye through the pupil onto the retina and is reflected by the retina using an infrared light detector. [2] An eye-tracking system according to [1] and an image display device, A virtual image display device in which an infrared light source array is incorporated into an image display device. [3] An eye-tracking system according to [1] and an image display device, the image display device has an area that transmits infrared light, A virtual image display device in which an infrared light source array is disposed on the opposite side of the image display device from the viewing side. [4] The virtual image display device according to [2] or [3], wherein the infrared light detector is incorporated into the image display device. [5] The image display device has an area that transmits infrared light, The virtual image display device according to [2] or [3], wherein the infrared light detector is disposed on the side opposite to the viewing side of the image display device. [6] The virtual image display device according to any one of [2] to [5], wherein the virtual image generating optical system has at least one of a convex lens and a Fresnel lens. [7] The virtual image display device according to any one of [2] to [5], wherein the virtual image generating optical system includes a folding optical system having a reflective polarizer and a half mirror. [8] The virtual image display device according to any one of [2] to [5], wherein the virtual image generating optical system includes a light guide plate having a light entrance portion and a light exit portion. [9] The virtual image display device according to [8], wherein at least one of the light entrance section and the light exit section has a diffraction element.

[10] The virtual image display device according to [9], wherein the diffraction element is a liquid crystal diffraction element. [Effects of the Invention]

[0011] According to the present invention, in an HMD, AR glasses, or the like, it is possible to easily detect the line of sight of a user without performing complex calculations. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a conceptual diagram for explaining the gaze tracking system of the present invention. [Figure 2] FIG. 2 is a diagram conceptually illustrating an example of the gaze tracking system of the present invention. [Figure 3] FIG. 3 is a diagram conceptually illustrating another example of the eye-tracking system of the present invention. [Figure 4] FIG. 4 is a conceptual diagram illustrating an example of a virtual image generating optical system. [Figure 5] FIG. 5 is a conceptual diagram illustrating an example of a virtual image generating optical system. [Figure 6] FIG. 6 is a diagram conceptually showing an example of a virtual image display device of the present invention. [Figure 7] FIG. 7 is a diagram conceptually showing another example of the virtual image display device of the present invention. [Figure 8]FIG. 8 is a diagram conceptually showing another example of the virtual image display device of the present invention. [Figure 9] FIG. 9 is a diagram conceptually showing another example of the virtual image display device of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The eye-tracking system and virtual image display device of the present invention will be described in detail below based on preferred embodiments shown in the drawings.

[0014] In this specification, a numerical range expressed using "to" means a range that includes the numerical values ​​before and after it as the lower and upper limits. In the present invention, visible light refers to light with a wavelength of 380 nm or more and less than 700 nm, and infrared light refers to light with a wavelength of 700 nm to 1 mm.

[0015] First, the basic concept of the eye-tracking system of the present invention will be explained with reference to FIG.

[0016] In conventional VR and AR systems, such as HMDs and AR glasses, gaze detection involves shining invisible light such as infrared light onto the eyeball and then processing the reflected image, known as the Purkinje image, to detect the user's gaze. However, as mentioned above, conventional gaze detection, such as gaze detection using Purkinje images, requires complex calculations and places a heavy load on the user.

[0017] In contrast, the gaze tracking system of the present invention detects the user's gaze by irradiating collimated infrared light onto the user's eye E (eyeball) and detecting the infrared light that passes through the pupil P and is reflected by the retina R, as conceptually shown in Figure 1, which illustrates a VR system. When collimated infrared light is irradiated onto the eye E, most of the infrared light is usually reflected by or near the surface of the eye E, such as the cornea and lens, and only a very small amount of infrared light passes through the pupil P and reaches the retina R. In contrast, as will be described in detail later, when the line of sight is directed in the direction of incidence of collimated infrared light, the infrared light enters the eye E through the pupil P, reaches the retina R, is retroreflected by the retina R, and exits from the pupil P, as shown in Figure 1. In other words, if collimated infrared light is incident on the eye E from various directions and the infrared light retroreflected by the retina R can be detected, it is considered that the gaze is directed in the direction of the incident collimated light.

[0018] The eye-tracking system of the present invention utilizes this. Fig. 2 conceptually shows an example of application of the gaze tracking system of the present invention to a VR system such as an HMD. Fig. 3 conceptually shows an example of application of the gaze tracking system of the present invention to an AR system such as AR glasses. Note that a VR system typically has an image display device for displaying virtual reality, and an AR system typically has an image display device for displaying augmented reality. The gaze tracking system of the present invention uses an infrared light source array 14 in which infrared light sources 14a are arranged one-dimensionally, preferably two-dimensionally, and the infrared light sources 14a are sequentially turned on and collimated by a virtual image generating optical system 12 to enter the eye E (eye, eyeball), and the infrared light retroreflected by the retina R is detected by an infrared light detector 16, thereby detecting and tracking the gaze.

[0019] That is, as described above, even if collimated infrared light is incident on the eye E, most of the infrared light is usually reflected by the surface of the eye E or its vicinity, and does not reach the retina R. However, when the user's line of sight is directed in the direction of the lit infrared light source 14a of the infrared light source array 14, the collimated infrared light passes through the pupil P and enters the eye E, is retroreflected by the retina R, and exits from the pupil P, as shown in Figure 1 above. Therefore, if the infrared light reflected by the retina R can be detected by the infrared light detector 16, it can be detected that the user's line of sight is directed in the direction of the infrared light emitted by the infrared light source 14a that was turned on at that time. As a result, the gaze tracking system of the present invention can easily detect the gaze of a user in a VR system, an AR system, etc., without performing complex calculations.

[0020] Specifically, in an example corresponding to the VR system shown in FIG. 2, the infrared light sources 14a of the infrared light source array 14 are turned on in sequence. When the infrared light source 14a is turned on, the infrared light is collimated by the virtual image generating optical system 12 and enters the eye E, similar to an image on an image display device in a VR system, that is, a virtual reality image. Here, if the user's line of sight is not directed in the incident direction of the infrared light emitted by the turned-on infrared light source 14a, even if collimated infrared light is incident on the eye E, as described above, most of the infrared light is reflected by (or near) the surface of the eye E and does not reach the retina R. Therefore, in this case, the infrared light detector 16 does not measure the reflected light from the retina R. On the other hand, when the user's line of sight is directed in the incident direction of the infrared light emitted by the infrared light source 14a, as shown in Fig. 1 above, the collimated infrared light passes through the pupil P, is retroreflected by the retina R, and is emitted from the pupil P, and can be detected by the infrared light detector 16. Therefore, the incident direction of the infrared light emitted by the infrared light source 14a, which is turned on at this time, can be detected as the user's line of sight.

[0021] On the other hand, an example corresponding to the AR system shown in FIG. 3 includes an infrared light source array 14, a light guide plate 50 acting as a virtual image generating optical system, a light entrance section 52, and a light exit section . Similarly, in the case of an AR system, the infrared light sources 14a of the infrared light source array 14 are sequentially turned on. When the infrared light source 14a is turned on, the infrared light is refracted by the light incident unit 52 and enters the light guide plate 50 at an angle at which it propagates after undergoing total reflection, similar to an image displayed on an image display device in an AR system, i.e., an augmented reality image. The infrared light that enters the light guide plate 50 propagates while repeatedly undergoing total reflection within the light guide plate 50, and then enters the light exit unit 54. The infrared light is collimated by the action of the light guide plate 50, which is a virtual image generating optical system, and the light incident unit 52. The light incident unit 52 may include a lens or the like for collimating the light, as necessary. The infrared light that enters the light exit unit 54 is refracted by the light exit unit 54, exits the light guide plate 50, and enters the eye E. Here, if the user's line of sight is not directed in the incident direction of the infrared light emitted by the turned-on infrared light source 14a, even if collimated infrared light is incident on the eye E, as described above, most of the infrared light is reflected by (or near) the surface of the eye E and does not reach the retina R. Therefore, in this case, the infrared light detector 16 does not measure the reflected light from the retina R. On the other hand, when the user's line of sight is directed in the incident direction of the infrared light emitted by the infrared light source 14a, as shown in Fig. 1 above, the collimated infrared light passes through the pupil P, is retroreflected by the retina R, and is emitted from the pupil P, and can be detected by the infrared light detector 16. Therefore, the incident direction of the infrared light emitted by the infrared light source 14a, which is turned on at this time, can be detected as the user's line of sight.

[0022] In this way, the gaze tracking system of the present invention detects the user's gaze by irradiating collimated infrared light onto the user's eye E, allowing it to enter through the pupil P, and detecting the light retroreflected by the retina R. As a method for detecting (photographing) infrared light reflected by the retina R, the bright pupil method is known as an example.

[0023] The bright pupil method is a method in which collimated light, that is, highly parallel light, is incident on the eye E and the light reflected from the retina R is detected. When collimated light is incident on eye E, if the center line of pupil P of eye E (a straight line passing through the center of pupil P and perpendicular to the surface of the cornea) and the optical axis connecting the light source and the center of pupil P are coincident or close to each other, the light incident from pupil P is reflected by retina R, passes through pupil P again, and is retroreflected along the above-mentioned optical axis. Therefore, in this case, reflected light is emitted from pupil P with a relatively strong intensity, and pupil P is detected as brighter than the peripheral area. If the incident light is visible light, the reflected light appears red due to the blood flowing in the capillaries of the retina. This is the phenomenon known as red eye. On the other hand, if the center line of the pupil P does not coincide with the optical axis connecting the light source and the center of the pupil P, as described above, the light incident from the pupil P will not reach the retina R, or even if it reaches the retina R and is reflected, it will not reach the pupil P again and will not be retroreflected. As a result, the area around the pupil P will be dark. Because the iris around the pupil P is colored, the reflected light from the peripheral area of ​​the pupil P will be stronger than the reflected light from the pupil P area, and the pupil P area will be detected as darker than the peripheral area.

[0024] In this way, when the bright pupil method is used and the pupil P is photographed brighter than the peripheral area, it can be determined that reflected light from the retina R has been detected. In addition, it can be seen that the center line of the pupil P and the optical axis connecting the light source and the center of the pupil P are coincident or form a close angle. Because the center line of the pupil P roughly coincides with the user's line of sight vector, the direction of the user's line of sight can be determined. That is, as described above, depending on the position of the infrared light source 14a in the infrared light source array 14, the incident direction of infrared light from the infrared light source 14a to the eye E can be detected as the direction of the user's line of sight. It should be noted that the accuracy of detecting the direction of the gaze can be further improved by measuring in advance the deviation between the center line of the user's pupil P and the gaze vector and correcting it using this data.

[0025] This bright pupil method is preferably used when the optical axes of the infrared light source 14a and the infrared light detector 16 are aligned or close to each other, as shown in Figures 8 and 9 (described later). In other words, the bright pupil method is preferably used when the optical path of light retroreflected from the retina R by the infrared light source 14a and the position of the infrared light detector 16 are close to each other.

[0026] In the present invention, two or more types of infrared light having different wavelengths may be used, and the infrared light reflected by the retina R may be detected (photographed). This method utilizes the fact that the intensity of light reflected by the retina R differs depending on the wavelength, and distinguishes between light reflected from the retina R and light reflected from areas other than the retina R. For example, when infrared light A with a wavelength of 800 nm and infrared light B with a wavelength of 1000 nm are used, infrared light A easily reaches the retina R and is detected as retroreflected light. In contrast, the amount of infrared light B that reaches the retina R is small due to absorption in the eye E, and therefore the reflection intensity from the retina R is also small. Therefore, if the detected intensity of infrared light A is greater than the detected intensity of infrared light B, it can be determined that infrared light A is reflected light from the retina R. On the other hand, if the difference in detected intensity between infrared light A and infrared light B is small, it can be estimated that these lights are lights reflected from the surface of the cornea and the surface of the eye E, etc., and are not reflected light from the retina. Therefore, when the detected intensity of infrared light A is greater than the detected intensity of infrared light B, the incident direction of infrared light from the infrared light source 14a that emitted infrared light A in the infrared light source array 14 can be detected as the direction of the user's line of sight, depending on the position of the infrared light source 14a that emitted infrared light A in the infrared light source array 14.

[0027] This method of using infrared light of multiple wavelengths is preferably used when the optical axes of the infrared light source 14a and the infrared light detector 16 are neither aligned nor close to each other, as shown in Figures 2, 3, 6, and 7. In other words, this method of using infrared light of multiple wavelengths is preferably used when the optical path of light retroreflected from the retina R by the infrared light source 14a is far from the position of the infrared light detector 16. In this example, it is preferable that the infrared light sources 14a that emit infrared light of different wavelengths are provided close to each other.

[0028] In the gaze tracking system of the present invention, there is no limitation on the infrared light detector 16, and various types of light detectors capable of detecting infrared light can be used. Therefore, the infrared light detector 16 may be a light detection element consisting of a single pixel and not having the function of capturing an image. In this case, it is preferable that the optical axis connecting the infrared light source 14a and the center of the pupil P and the optical axis connecting the infrared light detector 16 and the center of the pupil P coincide with or are close to each other. In other words, it is preferable that the optical axes of the infrared light source 14a and the infrared light detector 16 coincide with each other or are close to each other. When the infrared light source 14a and the infrared light detector 16 are arranged so that their optical axes coincide, the infrared light retroreflected by the retina R is detected with high intensity. Therefore, with this arrangement, it is possible to distinguish whether the reflected light is retroreflected light from the retina R or light reflected from the periphery based on the detected intensity of the reflected light. In other words, the infrared light detector 16, which is made up of a single pixel, is suitable for use when performing gaze detection using the bright pupil method described above.

[0029] The infrared light detector 16 may also be an imaging device capable of taking an image of the eye E (the user's eye). In this case, by using the captured image to distinguish between the pupil P area and the peripheral area and comparing the brightness of each, it is possible to distinguish between reflected light from the retina R and reflected light from areas other than the retina R. Therefore, by using an imaging device as the infrared light detector 16, it is possible to distinguish between light reflected by the retina R and light reflected from other sources, even if the optical axes of the infrared light source 14a and the infrared light detector 16 are not aligned. That is, in an embodiment using an imaging device as the infrared light detector 16, gaze detection can be performed using the bright pupil method by determining from an image whether the pupil P area is brighter or darker than the surrounding area. Furthermore, if the infrared light detector 16 has pixels that detect infrared light of different wavelengths, such as the above-mentioned infrared light A and infrared light B, gaze detection can be performed using the above-mentioned method using two or more types of infrared light of different wavelengths by comparing the intensities of infrared light A and infrared light B in the light reflected from the pupil P area.

[0030] The gaze tracking system of the present invention uses infrared light as detection light for gaze detection. There is no limitation on the wavelength of the infrared light, and it may be infrared light within the above-mentioned wavelength range. Here, the wavelength of the infrared light is preferably 700 nm or more, and more preferably 800 nm or more, in order to prevent the detection light for gaze detection from being visible to the user and to increase the reflectance at the retina R. Furthermore, in order to increase the transmittance at the eye E, the wavelength of the infrared light is preferably 1000 nm or less, and more preferably 900 nm or less.

[0031] In the eye-tracking system of the present invention, collimated infrared light is incident on the eye E by a virtual image generating optical system 12 . The eye tracking system of the present invention is basically used in VR systems such as HMDs and AR systems such as AR glasses. VR systems display virtual reality using an image display device, allowing the user to observe it. On the other hand, AR systems display augmented reality using an image display device, allowing the user to observe it. In both VR and AR systems, in order to allow the user to observe an appropriate image, the image displayed by the image display device, which is actually located a few centimeters from the user's eyes, must appear to be located several meters away. Accordingly, VR and AR systems are designed to use virtual image generation optical systems to enable the user to see a virtual image several meters away. Therefore, in the virtual image generation optical systems of VR and AR systems, the image displayed by the image display device is collimated at the position where it enters the user's eyes, making it nearly parallel light. In other words, VR and AR systems generate virtual images that appear to be far away from the user by collimating the image displayed by an image display device located a few centimeters in front of the user's eyes, i.e., the light emitted by the device, using a virtual image generation optical system. In other words, VR and AR systems are inherently equipped with the function of collimating the light emitted from an image display device located a few centimeters in front of the user's eyes using a virtual image generation optical system, making the image appear as if it were far away.

[0032] The present invention takes advantage of this. That is, the gaze tracking system of the present invention uses a virtual image generating optical system 12 provided in the VR system and the AR system to collimate the infrared light emitted by each infrared light source 14a of the infrared light source array 14 and make it incident on the user's eye E.

[0033] There are no limitations on the virtual image generating optical system 12, and various known virtual image generating optical systems used in VR systems and AR systems can be used. As an example of a virtual image generating optical system used in a VR system, as conceptually shown in Fig. 4, a virtual image generating optical system using a Fresnel lens 24 that collimates the image (illumination light) displayed by the image display device 20 is exemplified. In this virtual image generating optical system, instead of the Fresnel lens 24, a convex lens that collimates the image displayed by the image display device 20 may be used.

[0034] A pancake lens, which includes a folding optical system with a half mirror and a reflective polarizer, is also suitable for use as a virtual image generating optical system in a VR system. Figure 5 conceptually shows an example of this virtual image generating optical system. 5 has, from the image display device 20 side, a ¼λ wavelength plate 30, a half mirror 32, and a reflective polarizer 34. The reflective polarizer 34 is a reflective circular polarizer that reflects circularly polarized light with one rotation direction and transmits circularly polarized light with the opposite rotation direction. The pancake lens is not limited to the configuration shown in FIG. 5, and various pancake lenses used as virtual image generating optical systems in VR systems can be used.

[0035] 5, image display device 20 emits linearly polarized light, such as a liquid crystal display device or an organic electroluminescence display having an anti-reflection film. When image display device 20 emits unpolarized light, a linear polarizer may be provided between ¼λ wave plate 30 and image display device 20. The linearly polarized image displayed by the image display device 20 is converted by the ¼λ wave plate 30 into circularly polarized light in the rotation direction that is reflected by the reflective polarizer 34. In this example, as an example, the ¼λ wave plate 30 converts the linearly polarized image displayed by the image display device 20 into right-handed circularly polarized light that is reflected by the reflective polarizer 34. Approximately half of the right-handed circularly polarized image is transmitted through the half mirror and enters the reflective polarizer 34. The reflective polarizer 34 selectively reflects right-handed circularly polarized light. Therefore, the right-handed circularly polarized image is reflected by the reflective polarizer 34 and enters the half mirror 32 again. Approximately half of the right-handed circularly polarized image that is incident on the half mirror 32 is reflected by the half mirror 32. During this reflection, the right-handed circularly polarized image is converted into left-handed circularly polarized image. The left-handed circularly polarized image reflected by the half mirror 32 then enters the reflective polarizer 34. As described above, the reflective polarizer 34 selectively reflects right-handed circularly polarized light. Therefore, the left-handed circularly polarized image passes through the reflective polarizer 34 and is observed by the user as virtual reality. In this way, the pancake lens makes the light go back and forth between the half mirror 32 and the reflective polarizer 34, thereby lengthening the optical path length and allowing the user to view the virtual image as if it were located far away.

[0036] On the other hand, AR systems such as AR glasses allow the user to observe an image displayed by an image display device as an augmented reality image using a light guide plate 50 having a light entrance section 52 and a light exit section 54, similar to the infrared light emitted by the infrared light source 14a of the infrared light source array 14 described above. That is, as described above, the image (emitted light) displayed by the image display device is refracted at the light entrance section 52, enters the light guide plate 50, and propagates by repeatedly undergoing total reflection within the light guide plate 50. The image propagated within the light guide plate 50 eventually enters the light exit section 54, is refracted at the light exit section 54, and exits from the light guide plate 50, and is observed by the user as augmented reality. In the AR system, the light that becomes the virtual image is collimated by the action of the light guide plate 50 and the light entrance section 52 that form the virtual image generating optical system.

[0037] In the gaze tracking system and virtual image display device of the present invention, there are no limitations on the light entrance unit 52 and light exit unit 54 used in the AR system, and various known units used in AR systems can be used. Diffraction elements are preferably used for the light entrance portion 52 and the light exit portion 54. There are no limitations on the diffraction element, and various known diffraction elements such as a liquid crystal diffraction element, a volume hologram diffraction element, and a surface relief diffraction element can be used. In the example shown in FIG. 3, when a diffraction element is used for the light entrance portion 52 and the light exit portion 54, a transmissive diffraction element is used; however, the present invention is not limited to this, and light may be incident on and / or emitted from the light guide plate 50 by a reflective diffraction element.

[0038] As the diffraction element, a liquid crystal diffraction element is preferably used. There is no limitation on the liquid crystal diffraction element, and various known liquid crystal diffraction elements can be used. An example of a transmissive liquid crystal diffraction element is a liquid crystal diffraction element described in International Publication No. 2019 / 131918, etc., which includes an optically anisotropic layer formed using a composition containing a liquid crystal compound and having a liquid crystal orientation pattern in which the direction of the optical axis derived from the liquid crystal compound changes while continuously rotating along at least one direction in the plane. Further, an example of a reflective liquid crystal diffraction element is a liquid crystal diffraction element including a cholesteric liquid crystal layer having a liquid crystal orientation pattern in which the orientation of the optical axis derived from the liquid crystal compound changes while continuously rotating along at least one direction in the plane, as described in International Publication No. 2019 / 163944, etc.

[0039] FIG. 6 conceptually shows an example of a virtual image display device of the present invention that uses the eye-tracking system of the present invention. Note that the examples shown in the following Figures 6 to 9 are all examples in which the virtual image display device of the present invention is used in a VR system such as an HMD, but an AR system such as AR glasses can also be made into an AR system that uses the virtual image display device of the present invention by similarly arranging an image display device and an infrared light detector 16 corresponding to the infrared light source array 14 shown in Figure 3.

[0040] Each of the virtual image display devices of the present invention has the eye-tracking system of the present invention, and includes an image display device and a virtual image generating optical system 12. As described above, the gaze tracking system of the present invention collimates the infrared light emitted from the infrared light source of the infrared light source array using the virtual image generating optical system 12 of the VR system. That is, the virtual image display device of the present invention is a known VR system (AR system) incorporating the above-mentioned infrared light source array and infrared light detector.

[0041] In the virtual image display device of the present invention, there are no limitations on the image display device, and various known image display devices used in VR systems (AR systems) can be used. Examples include a liquid crystal display, an organic electroluminescence display, and a micro LED (Light Emitting Diode) display.

[0042] A virtual image display device 60 shown in FIG. 6 uses an image display device 62 incorporating an infrared light source array. That is, the image display device 62 has pixels that serve as infrared light sources 14a that emit infrared light in addition to pixels that display images in red, green, and blue, which are shown in white, thereby incorporating an infrared light source array into the image display device. In this virtual image display device 60, while a virtual reality is displayed by an image display device 62, the infrared light sources 14a built into the image display device 62 are sequentially turned on and collimated by a virtual image generating optical system 12, and the infrared light reflected by the retina R is detected by an infrared light detector 16, thereby detecting and tracking the user's line of sight.

[0043] On the other hand, a virtual image display device 64 shown in FIG. 7 uses an image display device 68 having an area through which infrared light can pass. An infrared light source array 14, in which infrared light sources 14a are arranged, is disposed on the side opposite to the viewing side (display surface) of this image display device 68. Therefore, the image display device 68 does not have pixels for image display at positions corresponding to the infrared light sources 14a in the infrared light source array 14, and these positions become regions through which infrared light can pass. In this virtual image display device 64, while a virtual reality is displayed by the image display device 68, the infrared light sources 14a of the infrared light source array 14 are sequentially turned on and collimated by the virtual image generating optical system 12, and the infrared light reflected by the retina R is detected by the infrared light detector 16, thereby detecting and tracking the user's line of sight.

[0044] In the image display device 68, the area through which infrared light can pass may be provided by various known methods, such as providing a through hole or using a substrate through which infrared light can pass as the substrate of the image display device 68.

[0045] As described above, in the virtual image display device shown in Figures 6 and 7, the infrared light source array preferably has infrared light sources 14a that emit two or more types of infrared light with different wavelengths. In this case, as described above, it is preferable that the infrared light sources 14a with different wavelengths are provided close to each other.

[0046] The virtual image display device 70 shown in FIG. 8 uses an image display device 72 incorporating an infrared light source array and an infrared photodetector 16. That is, the image display device 72 has pixels that serve as infrared light sources 14a that emit infrared light in addition to pixels that display images in red, green, and blue, which are shown in white, thereby incorporating an infrared light source array into the image display device. Furthermore, an infrared light detector 16, indicated by an oval, is incorporated into the image display device 72 in correspondence with the infrared light source 14a. The infrared light detector 16 may be incorporated into the image display device 72 by a known method. In this virtual image display device 70, while a virtual reality is displayed by an image display device 72, the infrared light sources 14a built into the image display device 72 are sequentially turned on and collimated by a virtual image generating optical system 12, and the infrared light retroreflected by the retina R is detected by an infrared light detector 16 built into the image display device 72, thereby detecting and tracking the user's line of sight.

[0047] On the other hand, a virtual image display device 74 shown in FIG. 9 uses an image display device 68 having an area through which infrared light can pass, similar to the virtual image display device 64 shown in FIG. An infrared light source array 14 having an array of infrared light sources 14a is disposed on the side opposite to the viewing side (display surface) of this image display device 68. Furthermore, a detector array 76 having an array of infrared light detectors 16 corresponding to the arrangement of the infrared light sources 14a in the infrared light source array 14 is disposed on the side opposite to the image display device 68 of the infrared light source array 14. Therefore, in this example, the infrared light source array 14 also has an area through which infrared light can pass, similar to the image display device 68, in accordance with the infrared light detectors 16 of the detector array 76. In this virtual image display device 74, while virtual reality is displayed by the image display device 68, the infrared light sources 14a of the infrared light source array 14 are sequentially turned on and collimated by the virtual image generating optical system 12, and the infrared light reflected by the retina R is detected by the infrared light detector 16 of the detector array 76, thereby detecting and tracking the user's line of sight.

[0048] In the virtual image display device of the present invention, there is no limitation on the formation density of the infrared light sources 14a, and it may be set appropriately depending on the accuracy and spatial resolution required for line of sight detection. Preferably, one side of the screen of the image display device provided in the virtual image display device is divided into 10 or more equal parts, more preferably 100 or more equal parts, and even more preferably 1000 or more equal parts, and one infrared light source 14a is provided in each section.

[0049] Furthermore, there is no limit to the speed at which the infrared light sources 14a are sequentially turned on, and it may be set appropriately depending on the accuracy and temporal resolution required for line-of-sight detection. Preferably, all the infrared light sources 14a are sequentially turned on in a time shorter than the time it takes for the image display device to display one frame, according to the refresh rate of the image display device provided in the virtual image display device.

[0050] The above describes the gaze tracking system and virtual image display device of the present invention, but the present invention is not limited to the above, and it goes without saying that various improvements and modifications may be made within the scope that does not deviate from the gist of the present invention. [Industrial Applicability]

[0051] The present invention can be suitably used for gaze detection in VR systems and AR systems such as HMDs and AR glasses. [Explanation of symbols]

[0052] 12 Virtual image generation optical system 14 Infrared light source array 14a infrared light source 16 Infrared photodetector 20,62,68,72 Image display devices 24 Fresnel lens 30 1 / 4λ wave plate 32 Half Mirror 34 Reflective polarizer 50 Light guide plate 52 Light incidence part 54 Light emitting part 60, 64, 70, 74 Virtual image display device E-eye P pupil R retina

Claims

1. an infrared light source array, a virtual image generating optical system, and an infrared light detector; the infrared light sources of the infrared light source array are sequentially turned on, the infrared light is collimated by the virtual image generating optical system, the collimated infrared light is made incident on the user's eye at different angles, and the infrared light that has entered the eye is incident on the retina through the pupil and is reflected by the retina, and is detected by the infrared light detector; an eye tracking system that uses two or more types of infrared light having different wavelengths and detects the two or more types of infrared light reflected from a retina; an image display device; A virtual image display device that sequentially lights up all of the infrared light sources in a time shorter than the time it takes for the image display device to display one frame.

2. The virtual image display device of claim 1 , wherein the infrared light detector is located outside of an optical path from the infrared light source array to the user's eye.

3. The virtual image display device according to claim 1 , wherein the infrared light source array is incorporated into the image display device.

4. the image display device has an area that transmits infrared light, The virtual image display device according to claim 1 , wherein the infrared light source array is disposed on a side opposite to a viewing side of the image display device.

5. The virtual image display device according to claim 1 , wherein the infrared light detector is incorporated into the image display device.

6. the image display device has an area that transmits infrared light, The virtual image display device according to claim 1 , wherein the infrared light detector is disposed on a side opposite to a viewing side of the image display device.

7. 7. The virtual image display device according to claim 1, wherein the virtual image generating optical system has at least one of a convex lens and a Fresnel lens.

8. 7. The virtual image display device according to claim 1, wherein the virtual image generating optical system includes a folding optical system having a reflective polarizer and a half mirror.

9. 7. The virtual image display device according to claim 1, wherein the virtual image generating optical system includes a light guide plate having a light entrance portion and a light exit portion.

10. The virtual image display device according to claim 9 , wherein at least one of the light entrance portion and the light exit portion has a diffraction element.

11. The virtual image display device according to claim 10 , wherein the diffraction element is a liquid crystal diffraction element.

Images