Image display device

Abstract

A problem to be addressed by the present invention is to provide an image display device capable of detecting a line of sight in a simple manner. The present invention solves the problem by having: an image display optical system; an infrared light source which irradiates infrared light on eyeballs of a user; a hologram element on which the infrared light source irradiates and which allows infrared light reflected by the eyeballs of the user to pass; and an infrared light image sensor which measures infrared light passing through the hologram element, wherein the hologram element does not act on visible light but acts on infrared light, and emits reproduced infrared light having an intensity distribution in the planar direction in accordance with the light of sight of the user.

Description

[Technical Field]

[0001] The present invention relates to an image display device such as a head-mounted display. [Background technology]

[0002] BACKGROUND ART Head-mounted displays (HMDs), AR glasses, and the like have been put to practical use as means for providing users with virtual reality (VR) and augmented reality (AR).

[0003] VR systems that provide VR and AR systems that provide AR are expected to be equipped with a function to detect the user's line of sight. By detecting the user's line of sight, it is possible to know what the user is observing, and various processes can be performed accordingly, such as displaying what the user is observing in detail, emphasizing what the user is observing, focusing on what the user is observing, displaying what the user is observing in high resolution, and using the user's line of sight as a pointing device. This will improve the functionality of HMDs, AR glasses, etc., and make it possible to realize higher performance HMDs, AR glasses, etc.

[0004] In gaze detection in HMDs and AR glasses, invisible light such as infrared light is shone onto the user's eyes, the reflected light is captured, and the resulting image is analyzed to detect the user's gaze. For example, Patent Document 1 describes an HMD with a user's gaze detection function, which includes a convex lens that is positioned opposite the user's cornea when the HMD is worn by the user, multiple infrared light sources that are positioned around the convex lens and irradiate infrared light toward the user's cornea, a camera that captures images including the user's cornea, and a housing to house these components, and when the periphery of the convex lens is equally divided into a first region that is the area on the outer corner of the user's eye, a second region that is the area on the inner corner of the eye, a third region that is the area on the top of the head, and a fourth region that is the area on the chin, the infrared light source is positioned in the first region or the second region. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2016-157485 Summary of the Invention [Problem to be solved by the invention]

[0006] The HMD described in Patent Document 1 detects the line of sight (direction of line of sight) of the user and uses this as a pointing device, thereby improving the convenience of the HMD.

[0007] Here, gaze detection, including that described in Patent Document 1, which has conventionally been installed in HMDs and AR glasses, detects the gaze by irradiating the user's eye (eyeball) with invisible light such as infrared light and analyzing the reflected image of the light reflected by the eyeball. For example, conventional gaze detection involves irradiating the eye with infrared light and analyzing the invisible light reflected by the anterior cornea, the anterior and posterior surfaces of the lens, and the posterior surface of the cornea. These reflected images are called Purkinje images. However, gaze detection using Purkinje images or the like has the problem of complex calculation processing and a large calculation load.

[0008] The gaze of a user of an HMD or other device moves at an extremely fast speed. Therefore, when complex calculations are being performed, the gaze detection system may not be able to keep up with the gaze movement. If the gaze detection cannot keep up with the movement of the gaze, the above-mentioned processes such as highlighting what the user is observing and the function as a pointing device will not be performed properly.

[0009] Furthermore, in order to detect the line of sight at a sufficient speed, it is necessary to take measures such as increasing the size of the power source (battery) and installing a high-performance arithmetic processing means. These measures would result in the HMD and AR glasses becoming larger and heavier, and would also increase the amount of heat dissipated by the processing unit, making the HMD less comfortable to wear.

[0010] The object of the present invention is to solve the problems of the conventional technology and to provide an image display device in an HMD, AR glasses, etc., equipped with a gaze detection function that can easily and quickly detect the user's gaze without performing complex calculations. [Means for solving the problem]

[0011] To achieve this object, the present invention has the following configuration. [1] an image display optical system; an infrared light source that irradiates the user's eyeball with infrared light; a hologram element that is irradiated by an infrared light source and through which infrared light reflected by the user's eyeball passes; an infrared light image sensor that captures an image of infrared light that has passed through the hologram element; The hologram element does not act on visible light but acts on infrared light, and is an image display device that emits reproduced infrared light having a surface intensity distribution that corresponds to the user's line of sight. [2] The image display device according to [1], wherein the image display optical system has a lens optical system and an image display element. [3] The image display device according to [2], wherein the infrared image sensor is mounted on a substrate constituting an image display element together with pixels for displaying an image. [4] The image display device according to [2], wherein the image display element has an area that transmits infrared light, and the infrared light image sensor is disposed on the opposite side of the image display element from the viewing side. [5] The image display device according to [1], wherein the image display optical system has an image display element and a light guide plate through which an image displayed by the image display element is incident and propagates. [6] The light guide plate is infrared light transmissive; [5] An image display device according to [5], which has an infrared mirror that transmits visible light and reflects infrared light, arranged on the opposite side of the light guide plate from the user's eyeball, and the infrared image sensor receives the infrared light reflected by the infrared mirror. [7] The image display device according to any one of [1] to [6], wherein the hologram element is disposed between the image display optical system and the eyeball of the user. [Effects of the Invention]

[0012] According to the present invention, in an image display device such as an HMD or AR glasses, the line of sight of a user can be easily detected without performing complex calculations. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram conceptually showing an example of an image display device of the present invention. [Figure 2] FIG. 2 is a conceptual diagram showing an example of a pancake lens. [Figure 3] FIG. 3 is a conceptual diagram for explaining the operation of the image display device of the present invention. [Figure 4] FIG. 4 is a conceptual diagram for explaining the operation of the image display device of the present invention. [Figure 5] FIG. 5 is a conceptual diagram for explaining the operation of the image display device of the present invention. [Figure 6] FIG. 6 is a diagram conceptually showing another example of the image display device of the present invention. [Figure 7] FIG. 7 is a diagram conceptually showing another example of the image display device of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The image display device of the present invention will be described in detail below based on preferred embodiments shown in the drawings.

[0015] In this specification, a numerical range expressed using "to" means a range that includes the numerical values ​​before and after it as the lower and upper limits. In the present invention, visible light refers to light with a wavelength of 380 to 780 nm, and infrared light (infrared rays) refers to light with a wavelength of more than 780 nm and not more than 1600 nm.

[0016] FIG. 1 conceptually shows an example in which the image display device of the present invention is used in an HMD (Head Mounted Display), which is a VR system. The HMD 10 shown in FIG. 1 includes an image display optical system 12, a hologram element 14, and an infrared light source 16. The image display optical system 12 is an optical system for displaying a VR image (virtual reality image) in an HMD or the like, and includes an image display element 20 and a lens optical system 24. Like known HMDs, the HMD 10 shown in FIG. 1 allows a user to view a VR image by viewing an image displayed by an image display element 20 as a virtual image through a lens optical system 24. Here, the image display element 20 has an infrared light image sensor 28 in addition to pixels 26 for displaying normal VR images. Also, an image processing unit 32 is connected to the infrared light image sensor 28.

[0017] In the present invention, the hologram element 14 is a hologram element that does not act on visible light but acts on infrared light. As described above, the HMD 10 allows the user to view VR images in the same way as a normal HMD, and also irradiates the user's eyeball E with infrared light (dashed line) from an infrared light source 16. The infrared light reflected by the eyeball E is incident on a hologram element 14, and the infrared light diffracted by the hologram element 14 is captured by an infrared light image sensor 28. The HMD 10 analyzes this infrared light image with an image processing unit 32, thereby detecting the line of sight of the user viewing the VR image. This point will be discussed in more detail later.

[0018] In the HMD 10, the image display optical system 12 is basically a known optical system for displaying VR images in an HMD, etc., except that the image display element 20 has an infrared image sensor 28 in addition to pixels 26 for displaying red, green, and blue images. Therefore, the image display element 20 can be a known image display element used in VR systems such as HMDs, except that the infrared light image sensor 28 is mounted on a substrate that constitutes the image display element 20. Examples of the image display element 20 include a liquid crystal display, an organic electroluminescence display, and a micro LED (Light Emitting Diode) display.

[0019] There are no restrictions on the infrared light image sensor 28 mounted on the image display element 20, and various known sensors can be used as long as they are sensitive to the infrared light emitted by the infrared light source 16 and can capture infrared light images. Examples include an area CCD (Charge Coupled Device) sensor, a CMOS sensor, an InGaAs sensor, an organic CMOS sensor, a quantum dot sensor, and a black silicon photodiode. The infrared image sensor 28 may be mounted on the substrate of the image display element 20 by a known method. As described above, the infrared image sensor 28 is connected to the image processing section 32. The image processing section 32 will be described in detail later.

[0020] The image display optical system 12 includes an image display element 20 and a lens optical system 24 . The lens optical system 24 is also a known optical system used in a VR system that displays virtual reality, such as an HMD. Examples of the lens optical system 24 include a so-called pancake lens, a single lens, a lens array, a Fresnel lens, a liquid crystal lens, a metasurface lens, and a GRIN lens, which include a folded optical system having a half mirror and a reflective polarizer.

[0021] FIG. 2 conceptually shows an example of a pancake lens. 2 has, from the image display element 20 side, a ¼λ wavelength plate 40, a half mirror 42, and a reflective polarizer 46. The reflective polarizer 46 is a reflective circular polarizer that reflects circularly polarized light with one rotation direction and transmits circularly polarized light with the opposite rotation direction. The pancake lens is not limited to the configuration shown in FIG. 2, and various known pancake lenses used in VR systems can be used.

[0022] 2, the image display element 20 emits linearly polarized light, such as a liquid crystal display device or an organic electroluminescence display having an anti-reflection film. If the image display element 20 emits unpolarized light, a linear polarizer may be provided between the ¼λ wave plate 40 and the image display element 20. The linearly polarized image displayed by the image display element 20 is converted by the ¼λ wavelength plate 40 into circularly polarized light in the rotation direction that is reflected by the reflective polarizer 46. In this example, as an example, the ¼λ wavelength plate 40 converts the linearly polarized image displayed by the image display element 20 into right-handed circularly polarized light that is reflected by the reflective polarizer 46. Approximately half of the right-handed circularly polarized image is transmitted through the half mirror and enters the reflective polarizer 46. The reflective polarizer 46 selectively reflects right-handed circularly polarized light. Therefore, the right-handed circularly polarized image is reflected by the reflective polarizer 46 and enters the half mirror 42 again. Approximately half of the right-handed circularly polarized image that is incident on the half mirror 42 is reflected by the half mirror 42. During this reflection, the right-handed circularly polarized image is converted into left-handed circularly polarized image. The left-handed circularly polarized image reflected by the half mirror 42 then enters the reflective polarizer 46. As described above, the reflective polarizer 46 selectively reflects right-handed circularly polarized light. Therefore, the left-handed circularly polarized image passes through the reflective polarizer 46 and is observed by the user as virtual reality. In this way, the pancake lens transmits light back and forth between the half mirror 42 and the reflective polarizer 46, lengthening the optical path length and allowing the user to view the VR image as if the image (virtual image) were located far away.

[0023] As described above, the HMD 10 includes an infrared light source 16 . In the HMD 10, infrared light (dashed line) is irradiated from the infrared light source 16 onto the user's eyeball E, the infrared light reflected by the eyeball E is incident on and passes through the hologram element 14, and the infrared light diffracted by the hologram element 14 is captured by the infrared light image sensor 28. In the HMD 10, the infrared light image captured in this manner is analyzed by the image processing unit 32 to detect the user's line of sight. Here, the hologram element 14 does not act on visible light but acts on infrared light, and emits reconstructed light having a surface intensity distribution that corresponds to the user's line of sight. The image display device of the present invention has such a configuration, and thus makes it possible to easily detect the line of sight of a user in a VR system, an AR system, or the like, without performing complex calculations.

[0024] When infrared light for gaze detection (eye tracking) is irradiated toward the eyeball E, the infrared light is reflected on the surface of the eyeball E and / or the surface of the iris after passing through the cornea. If the position of the infrared light source 16 and the position of the user's eyeball E are roughly fixed, the emitted infrared light is reflected with a distribution that corresponds one-to-one to the direction of the user's line of sight. The infrared light irradiated onto the eyeball E is patterned so that the reflected light from the eyeball E has a discrete distribution. The discrete reflection pattern from the eyeball E obtained when the patterned infrared light is incident in this way can be considered to contain the direction of the user's line of sight converted into information on the "positions and angles at which individual light beams are incident on the hologram element 14."

[0025] Here, the hologram element 14 can be formed so as to selectively diffract light, the incident position and incident angle at that position, and the wavelength of which are known in advance, in a specific direction using a technique called computer-generated holograms. In this case, hologram element 14 can be thought of as a logical operator that converts information on the incident position and incident angle of reflected light from eyeball E into a vector called the emission direction and outputs it. Therefore, if the positional relationship between infrared light image sensor 28 and hologram element 14 is fixed, the collection of vector information output from hologram element 14 will be recognized as encoded signal light on infrared light image sensor 28.

[0026] As described above, there is a one-to-one relationship between the discrete light beams of reflected light from eyeball E that are generated depending on the direction of the user's line of sight and the position and angle of incidence of this reflected light on hologram element 14. There is also a one-to-one relationship between the position and angle of incidence of the light beam that has entered hologram element 14 and the vector of the light beam that enters infrared light image sensor 28 from hologram element 14. Therefore, a one-to-one relationship can be established in advance between the set of vectors of the light beams incident on the infrared light image sensor 28 from the hologram element 14, i.e., the encoded signal light, and the direction of the user's line of sight. Therefore, by back-calculating this predetermined one-to-one relationship, it is possible to accurately identify the direction of the line of sight from the obtained signal light, i.e., the captured infrared light image, with a low computational load.

[0027] A specific description will be given below with reference to the schematic diagrams of Figures 3 to 5. Note that Figures 3 to 5 are two-dimensional schematic diagrams for clarity of explanation. 3 is a schematic diagram showing the state in which the eyeball E faces the hologram element 14. That is, in this example, the user's line of sight is directed forward. A beam of infrared light discretely emitted from infrared light source 16 is reflected by eyeball E and enters hologram element 14. The reflected beam that enters hologram element 14 is diffracted by hologram element 14, and predetermined encoded signal light is generated. This signal light enters infrared light image sensor 28 and is imaged.

[0028] 4 schematically shows the state in which the eyeball E has rotated upward. That is, in this example, the user's line of sight is directed upward. In Fig. 4, the rotation of eyeball E causes a change in the position of the cornea, which in turn causes a change in the position and angle of incidence of the reflected light onto hologram element 14. Hologram element 14 converts this change into vector information to generate encoded signal light, thereby generating signal light with a code specific to this line of sight direction and different from that in the state where the line of sight is directed straight ahead as shown in Fig. 3. That is, infrared light image sensor 28 captures an infrared light image specific to the state where the line of sight is directed upward, in accordance with this signal light.

[0029] 5 also shows a schematic representation of the eyeball E rotated downward, i.e., the user's line of sight is directed downward in this example. As in Fig. 4, as the position of the cornea changes, a reflected light beam different from that shown in Fig. 3 or 4 is incident on hologram element 14. Hologram element 14 converts this change into vector information to generate encoded signal light, thereby generating signal light with a code specific to this gaze direction that differs from that shown in Fig. 3 when the gaze is directed forward and that shown in Fig. 4 when the gaze is directed upward. That is, infrared light image sensor 28 captures an infrared light image specific to the downward gaze state in accordance with this signal light. In this case, it is possible that the location where one of the light beams enters hologram element 14 happens to be the same as one of the reflected light beams generated when the line of sight is directed upward, as shown in FIG. 4. Even in this case, if hologram element 14 is an angle-multiplexed hologram element, the angle of incidence will be different, and the light beam diffracted by hologram element 14 will be converted into a different vector and can enter the imaging element. Therefore, there will be no loss of information in the encoded signal light shown in FIGS. 3 and 4. In this way, the use of an angle-multiplexed hologram can maintain detection accuracy.

[0030] In other words, the hologram element 14 emits encoded signal light specific to the direction of the user's line of sight as reconstructed light obtained by reproducing reflected light from the eyeball E, which has a planar intensity distribution specific to the direction of the line of sight. In other words, the hologram element 14 generates an infrared light image having a planar intensity distribution specific to the direction of the line of sight, by reproducing the light reflected from the eyeball E according to the direction of the user's line of sight.

[0031] In the HMD 10, the image processing unit 32 stores, as a reference image, an infrared light image generated by the hologram element 14 in correspondence with the user's line of sight (the rotation direction of the eyeball E) using coded signal light that is specific to each line of sight direction. That is, the image processing unit 32 stores a look-up table (LUT) that indicates the relationship between the infrared light image generated by the hologram element 14 in correspondence with each line of sight and the line of sight direction. Examples of the infrared light image (reference image) stored by the image processing unit 32 according to the direction of the line of sight include a binarized black and white pattern image and a monochrome image according to the amount of infrared light.

[0032] As described above, in the HMD 10, in addition to displaying a VR image, the infrared light source 16 emits infrared light, which is reflected by the eyeball E and diffracted by the hologram element 14 to form encoded signal light (infrared light), which is then captured by the infrared light image sensor 28. An image signal (image data) of the infrared image captured by the infrared image sensor 28 is supplied to the image processing unit 32. The image processing unit 32 performs the following processing, for example.

[0033] When the image processing unit 32 receives the image signal of the captured image from the infrared image sensor 28, it first processes the image signal to generate an image in the same format as the reference image stored in itself. In the following description, for convenience, the image in the same format as the reference image, obtained by processing the image signal of the image captured by the infrared image sensor 28, will be referred to as the "processed captured image." For example, the image processing unit 32 binarizes the supplied image signal to generate a black and white pattern image as the processed captured image, or processes the supplied image signal to generate a monochrome image according to the amount of infrared light as the processed captured image.

[0034] After processing the supplied image signal to generate a processed captured image, the image processing unit 32 selects an image that is the same as or most similar to the generated processed captured image from the reference images stored in itself according to each line of sight direction. In other words, after generating a processed captured image, the image processing unit matches the reference images stored in itself with the processed captured image. Image matching may be performed using a known method that is used for identifying, recognizing, and authenticating various types of images.

[0035] After selecting the same reference image as the processed captured image or the most similar reference image, the image processing unit 32 detects the direction of the user's gaze corresponding to that reference image and supplies the gaze direction detection result to, for example, a control unit (not shown) of the image display element 20. When the control unit of the image display element 20 receives information about the direction of the user's line of sight, it performs processing according to the user's line of sight, such as highlighting the object the user is observing or displaying the object the user is observing in high resolution, as described above.

[0036] The function of such gaze detection (eye tracking) is similar to that of AR systems such as AR glasses, which will be described later.

[0037] As described above, gaze detection in conventional HMDs and AR glasses has the problem of complex arithmetic processing and a heavy computational load. In contrast to this, as described above, the image display device of the present invention can detect the gaze by, for example, image matching, without the need for complex calculations. Therefore, according to the present invention, in a VR system such as an HMD and an AR system such as AR glasses, the gaze of a user can be detected simply and quickly without performing complex calculations.

[0038] The hologram element 14 may be produced by any of various known methods. As an example, a method is exemplified in which the hologram element 14 is created using the above-mentioned computer-generated hologram, etc., in combination with the patterned infrared light emitted by the infrared light source 16, and binarized to obtain a black and white pattern image (binarized image) specific to the direction of the line of sight. Alternatively, hologram element 14 having a pattern such as stripes, checks, or a grid pattern may be created in advance. In this case, the gaze direction is first input to, for example, image processing unit 32 of HMD 10. Next, in HMD 10, using the created hologram element 14, infrared light source 16 irradiates infrared light while the user faces the input gaze direction, and infrared light image sensor 28 captures an image of the infrared light reflected by eyeball E. Then, image processing unit 32 associates the captured image with the previously input gaze direction and stores it. This process is performed for various gaze directions, and a lookup table showing the relationship between the infrared light image and the gaze direction is created.

[0039] As described above, the hologram element 14 used in the present invention does not act on visible light but acts on infrared light. More specifically, it is transparent to the display light that is displayed by the image display element 20 and directed toward the eye E. In other words, the hologram element 14 is transparent to visible light. Such a hologram can be realized, for example, in the case of a phase hologram, by increasing the film thickness or increasing the refractive index difference to be formed, so that diffraction in the visible range exceeds the diffraction limit, while keeping the infrared light used for gaze detection (eye tracking) within the diffraction range. Furthermore, an amplitude hologram can be realized by making the absorbent or reflective material have substantially no absorption in the visible range and have absorption or reflectivity only for the infrared light used for line of sight detection.

[0040] The holographic element 14 used in the present invention is preferably an angle-multiplexed hologram. As described above, with an angle-multiplexed hologram, even if reflected light is incident at different angles depending on the line of sight and at the same point on hologram element 14, the diffraction directions can be treated as different and used for signal detection. Therefore, by using an angle-multiplexed hologram as hologram element 14, a more accurate line-of-sight detection system can be constructed.

[0041] As materials for forming the hologram element 14 used in the present invention, various known materials that can be used to create holograms that do not react to visible light but react to infrared light, as described above, can be used. Examples of materials that can be used to form the hologram element 14 used in the present invention include wet hologram materials such as dichromated gelatin materials and silver halide holography photosensitive materials, dry hologram materials containing photopolymer materials, photorefractive materials, photochromic materials, etc. in binders or matrix materials, imprint materials for forming surface patterns, and color resist materials. From the viewpoint of durability, it is preferable to use a dry hologram material, an imprint material, a color resist material, etc. Also, from the viewpoint of wavelength selectivity, it is preferable to use a dry hologram material and a color resist material.

[0042] As the dry hologram material, a material containing a photopolymer material in a binder or matrix material can be preferably used. Examples of these materials include those described in Japanese Patent Nos. 2849021, 3075082, and 4142396. Commercially available materials such as Bayfol HX (trade name) available from Bayer Material Sanense Co., Ltd. may also be used. These dry hologram materials are so-called phase holograms, which use the refractive index difference formed in the layers to create a hologram. Therefore, by using computer-generated hologram techniques, it is possible to create a hologram element 14 that works only in the infrared wavelength range and is substantially transparent in the visible range.

[0043] Color resist materials that can be used include photo-curable materials (negative resist materials) that contain dyes, pigments, and other coloring matter in a binder or photo-curable material, as well as positive resist materials that become soluble in exposed areas. Examples of these materials include color resist materials for infrared filters described in WO 2017 / 130825 and WO 2019 / 058882, etc. The color resist material contains a dye or pigment that is infrared-absorbing and substantially transparent in the visible range, thereby forming a hologram material that acts as an amplitude hologram in the infrared range and is substantially transparent in the visible range.

[0044] There are no restrictions on the infrared light source 16, and various known infrared light sources can be used as long as they can irradiate the user's entire cornea, or the entire cornea and conjunctiva, with patterned infrared light (structured infrared light). Examples of the infrared light source 16 include an infrared LED, an infrared laser, an infrared OLED (Organic Light Emitting Diode), and a light source that combines a visible light source with a wavelength conversion film. In a light source that combines a visible light source with a wavelength conversion film, examples of the visible light source include an LED and an OLED that emit visible light. Examples of the wavelength conversion film include a wavelength conversion film (wavelength conversion member) that uses quantum dots (QDs) and phosphors.

[0045] There are also no limitations on the pattern of infrared light emitted by the infrared light source 16. In other words, various patterns can be used for the pattern of infrared light emitted by the infrared light source 16 as long as they can, in combination with the hologram element 14, produce unique encoded signal light according to the user's line of sight. Examples include point patterns such as lattice point patterns and random point patterns, lattice line patterns, line patterns, and checkered patterns (checkered patterns).

[0046] There is no limitation on the wavelength of the infrared light emitted by the infrared light source 16, as long as it is infrared light. The wavelength of the infrared light emitted by the infrared light source 16 is preferably 800 to 1550 nm, more preferably 800 to 1000 nm, and even more preferably 800 to 950 nm. Setting the wavelength of the infrared light emitted by the infrared light source 16 within the above range is preferable in that it prevents the infrared light used for gaze detection from being visible to the user and enables CCD sensors, CMOS sensors, etc. to detect the infrared light with high sensitivity.

[0047] In the HMD 10 shown in FIG. 1, the infrared light image sensor 28 is mounted on the substrate of the image display element 20, but the present invention is not limited to this. As an example, as conceptually shown in FIG. 6, an image display element 50 having an area through which infrared light can pass may be used, and the infrared light image sensor 28 may be separate from the image display element 50. 6, the infrared light image sensor 28 is disposed on the side opposite to the viewing side (image display surface) of the image display element 50. Accordingly, the image display element 50 does not have pixels 26 for image display at a position corresponding to the infrared light image sensor 28, and this becomes an area through which infrared light can pass.

[0048] In the image display element 50, the region through which infrared light can pass may be provided by various known methods, such as a method of providing a through hole or a method of using a substrate through which infrared light can pass as the substrate of the image display element 50.

[0049] The HMD 10 shown in FIG. 1 is an example in which the image display device of the present invention is used in a VR system such as an HMD, but the present invention is not limited to this. That is, the image display device of the present invention can be suitably used in an AR system such as AR glasses. An example in which the image display device of the present invention is used in AR glasses is conceptually shown in Fig. 7. Note that the AR glasses shown in Fig. 7 use many of the same components as the HMD 10 described above, so the same components are given the same reference numerals, and the following description will mainly focus on the different parts.

[0050] The AR glasses 60 shown in FIG. 7 include an image display element 62, a light guide plate 64, an infrared mirror 68, the hologram element 14, the infrared light source 16, the infrared image sensor 28, and the image processing unit 32 described above. In the AR glasses 60, the image display element 62 and the light guide plate 64 form an image display optical system.

[0051] The image display element 62 is an image display element for displaying AR images, which is used in known AR systems such as AR glasses. Therefore, various image display elements used in AR systems, such as liquid crystal displays, organic electroluminescence displays, micro LED displays, LCOS (Liquid Crystal on Silicon) displays, and MEMS (Micro Electro Mechanical Systems) laser displays, can be used as the image display element 62.

[0052] As with normal AR glasses, the image displayed by the image display element 62 is incident on the light guide plate 64, propagates within the light guide plate 64 while repeatedly undergoing total reflection, and is then emitted from the light guide plate 64. The light emitted from the light guide plate 64 is observed by the user as an AR image. A known light guide plate used in AR glasses can also be used for the light guide plate 64. It is preferable that the light guide plate 64 is infrared light transmissive.

[0053] There are no limitations on the method for making the image displayed by the image display element 62 enter the light guide plate 64 and for making the image that has propagated through the light guide plate 64 exit, and any known method can be used. Light (image) is preferably made to enter and exit the light guide plate 64 using a diffraction element. There are no limitations on the diffraction element, and various known diffraction elements such as a liquid crystal diffraction element, a volume hologram diffraction element, and a surface relief diffraction element can be used. In the example shown in FIG. 6, when a diffraction element is used to input and output light to and from the light guide plate 64, a transmissive diffraction element is used. However, the present invention is not limited to this, and light may be input and / or output to and from the light guide plate 64 using a reflective diffraction element.

[0054] As the diffraction element, a liquid crystal diffraction element is preferably used. There is no limitation on the liquid crystal diffraction element, and various known liquid crystal diffraction elements can be used. An example of a transmissive liquid crystal diffraction element is a liquid crystal diffraction element described in International Publication No. 2019 / 131918, etc., which includes an optically anisotropic layer formed using a composition containing a liquid crystal compound and having a liquid crystal orientation pattern in which the direction of the optical axis derived from the liquid crystal compound changes while continuously rotating along at least one direction in the plane. Further, an example of a reflective liquid crystal diffraction element is a liquid crystal diffraction element including a cholesteric liquid crystal layer having a liquid crystal orientation pattern in which the orientation of the optical axis derived from the liquid crystal compound changes while continuously rotating along at least one direction in the plane, as described in International Publication No. 2019 / 163944, etc.

[0055] As with the HMD 10 shown in FIG. 1, the AR glasses 60 shown in FIG. 7 also allow the user to view an AR image and detect the user's line of sight using infrared light. In the AR glasses 60, as in the HMD 10, an infrared light source 16 irradiates an eye E of a user with infrared light (indicated by a chain line). The infrared light reflected by the user's eyeball E enters and passes through the hologram element 14. As described above, the infrared light reflected by the eyeball E is diffracted by the hologram element 14, thereby becoming a unique encoded signal light that corresponds to the orientation of the eyeball E, i.e., the direction of the user's line of sight.

[0056] The infrared light transmitted through the hologram element 14 passes through the light guide plate 64, is reflected by the infrared mirror 68, and enters the infrared image sensor 28, where an image is captured. The infrared mirror 68 is a mirror that transmits visible light and reflects infrared light. By reflecting the infrared light that has transmitted through the light guide plate 64 by the infrared mirror 68 and making it incident on the infrared image sensor 28, it is possible to prevent the infrared image sensor 28 from being located within the user's line of sight. The infrared light mirror 68 may be a known mirror, such as a dichroic mirror, that has wavelength selectivity such that it transmits visible light and reflects infrared light.

[0057] The infrared image captured by the infrared image sensor 28 is sent to the image processing unit 32 in the same manner as above. As before, the image processing unit 32 processes the supplied infrared image, matches it with an infrared image stored in itself that corresponds to the direction of the user's gaze, and detects the direction of the user's gaze using the matching result. The image processing unit 32 sends the detection result of the user's gaze to, for example, the control unit of the image display element 62.

[0058] In both the HMD 10 shown in Fig. 1 and the AR glasses 60 shown in Fig. 7, the hologram element 14 is disposed between the image display optical system and the user's eyeball E. Note that in the HMD 10 shown in Fig. 1, the image display optical system is designated by the reference numeral 12, and in the AR glasses 60 shown in Fig. 7, the image display optical system is designated by the image display element 62 and the light guide plate 64. However, when the image display device of the present invention is used in AR glasses, the position of hologram element 14 is not limited to this. That is, when the image display device of the present invention is used in AR glasses, hologram element 14 may be disposed between light guide plate 64 and infrared mirror 68, or between infrared mirror 68 and infrared image sensor 28.

[0059] The image display device of the present invention has been described above, but the present invention is not limited to the above, and various improvements and modifications may be made without departing from the spirit of the present invention. [Industrial Applicability]

[0060] The present invention can be suitably used in VR systems such as HMDs and AR systems such as AR glasses. [Explanation of symbols]

[0061] 10 HMD 12 Image display optical system 14 Hologram element 16 Infrared light source 20,62 Image display element 26 pixels 28 Infrared image sensor 32 Image processing section 40 1 / 4λ wave plate 42 Half Mirror 46 Reflective polarizer 60 AR Glasses 64 Light guide plate 68 Infrared mirror E Eyeball

Claims

1. an image display optical system; an infrared light source that irradiates the user's eyeball with infrared light; a hologram element through which the infrared light emitted by the infrared light source and reflected by the user's eyeball passes; an infrared light image sensor that captures an image of infrared light that has passed through the hologram element, The hologram element does not act on visible light but acts on infrared light, and emits reproduced infrared light having a surface intensity distribution according to the user's line of sight.

2. The image display device according to claim 1 , wherein the image display optical system comprises a lens optical system and an image display element.

3. 3. The image display device according to claim 2, wherein the infrared image sensor is mounted on a substrate constituting the image display element together with pixels for displaying an image.

4. 3. The image display device according to claim 2, wherein the image display element has a region that transmits infrared light, and the infrared light image sensor is disposed on the opposite side of the image display element from the viewing side.

5. 2. The image display device according to claim 1, wherein the image display optical system comprises an image display element and a light guide plate that receives and propagates the image displayed by the image display element.

6. the light guide plate is infrared light transmissive, 6. The image display device according to claim 5, further comprising an infrared mirror that transmits visible light and reflects infrared light, the infrared mirror being arranged on an opposite side of the light guide plate from the eyeball of the user, and the infrared image sensor receives the infrared light reflected by the infrared mirror.

7. 7. The image display device according to claim 1, wherein the hologram element is disposed between the image display optical system and the eyeball of the user.

Images