Eyeball information detection device, display device, eyeball information detection method and display method

Abstract

Provided is an eyeball information detection device capable of improving detection accuracy.　The present technique provides an eyeball information detection device comprising: a support member to be mounted on the head of a user; a substrate provided on the support member so as to face an eyeball of the user; an illumination system provided on the support member and including at least one light source that emits invisible light toward the substrate; at least one optical element provided on the substrate so as to be irradiated with the invisible light and generating a virtual light source of the invisible light; a light-receiving system including a light-receiving element that receives the invisible light reflected by the eyeball through the optical element; and a detection system that detects eyeball information, which is information about the eyeball, on the basis of the output of the light-receiving system.

Description

[Technical Field]

[0001] The technology according to the present disclosure (hereinafter also referred to as "the technology") relates to an eyeball information detection device, a display device, an eyeball information detection method, and a display method. [Background technology]

[0002] BACKGROUND ART Conventionally, eyeball information detecting devices (for example, gaze detecting devices) that detect information about a user's eyeball (for example, gaze) by irradiating the eyeball with invisible light are known (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-225207 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional eyeball information detection devices have room for improvement in terms of detection accuracy.

[0005] Therefore, an object of the present technology is to provide an eyeball information detection device that can improve detection accuracy. [Means for solving the problem]

[0006] The present technology includes a support member attached to a user's head; a substrate provided on the support member so as to face the eyeball of the user; an illumination system provided on the support member and including at least one light source that emits invisible light toward the substrate; at least one optical element, other than a plane mirror, that is provided on the substrate so as to be irradiated with the invisible light and that generates a virtual light source of the invisible light; a light receiving system including a light receiving element that receives the invisible light reflected by the eyeball via the optical element; a detection system that detects eyeball information based on an output of the light receiving system; An eyeball information detection device is provided, comprising: The at least one optical element may be a plurality of optical elements. The plurality of optical elements may include at least three optical elements arranged two-dimensionally along an in-plane direction of the substrate. The at least one light source may be a plurality of light sources. The plurality of light sources may include at least two light sources corresponding to different optical elements among the plurality of optical elements. The plurality of light sources may correspond to the plurality of optical elements, respectively. The at least one light source may include light sources corresponding to at least two optical elements of the plurality of optical elements. The at least one light source may be a single light source corresponding to the plurality of optical elements. The number of the at least one optical element may be greater than the number of the at least one light source. Each of the plurality of light sources may correspond to at least two of the plurality of optical elements, and the invisible light from each of the plurality of light sources may propagate through the substrate and be irradiated onto the corresponding at least two optical elements. The substrate may be provided with an optical member that guides the invisible light from the light source to the plurality of optical elements. The invisible light from the light source may propagate within the substrate and be irradiated onto the plurality of optical elements via the optical member. The light receiving system may be provided on the substrate. The support member may have temples including ear hooks, the light receiving system may be provided on the temples, and another optical element may be provided on the substrate to guide the invisible light reflected by the eyeball to the light receiving system. The plurality of optical elements may be provided on the substrate around the other optical element. The support member may have a temple including an ear hook portion, and the light source may be provided on the temple. The support member may have a temple including an ear hook portion and an extension portion extending on the opposite side of the base plate from the ear hook portion side, and the light source may be provided on the extension portion. The optical element may be of the reflective type. The optical element may be a diffractive element. The diffractive elements may have a non-uniform pitch. The diffraction element may be of a wavefront reconstruction type. The at least one optical element may be a plurality of diffraction elements, and the wavefront shapes recorded on at least two of the plurality of diffraction elements may be different, and the detection system may have a fitting unit that fits a plurality of reflected images of the invisible light at the eyeball to the wavefront shapes recorded on the plurality of diffraction elements. The diffractive element may be a diffusing element. The at least one optical element may be a plurality of diffusion elements, and the detection system may have a fitting unit that fits a plurality of reflection images of the invisible light on the eyeball to the plurality of diffusion elements. At least two of the plurality of diffusing elements may have different shapes. The diffractive element may have curved mirror properties. The at least one diffraction element may be a plurality of diffraction elements having different diffraction powers, at least two of the plurality of virtual light sources generated by the plurality of diffraction elements may be positioned at different positions in the thickness direction of the substrate, and the detection system may have a depth estimation unit that estimates the depth of the eyeball from the output of the light receiving system. The illumination system may be capable of irradiating at least two of the plurality of optical elements with the invisible light at different times. The plurality of optical elements may include first and second optical elements corresponding to different light sources, and the illumination system may have a light source driving unit that selectively drives the light source corresponding to the first optical element and the light source corresponding to the second optical element. The optical element may be a diffraction element having wavelength selectivity, the plurality of light sources may include at least two light sources having different emission wavelengths and corresponding to the diffraction element, and the illumination system may have a light source driving unit that selectively drives the at least two light sources. The optical element may be a diffraction element having polarization dependency, and the illumination system may be capable of varying the polarization direction of the invisible light. The optical element may be a diffraction element having incidence angle dependency, and the illumination system may be capable of varying the incidence angle of the invisible light onto the diffraction element. The optical element may be a curved mirror. The eyeball information may include at least one of the orientation of the eyeball, the position and size of the pupil, and the position of the iris. The present technology provides the image light generation device, an optical system provided on the substrate and guiding image light from the image light generating device to the eyeball; Equipped with The image light generating device also provides a display device that generates the image light based on the detection result of the eyeball information detecting device. The present technology includes a step of irradiating invisible light onto at least one optical element provided on a substrate facing the user's eyeball to generate a virtual light source of the invisible light; receiving the invisible light reflected by the eyeball via the optical element; detecting eyeball information based on a light reception result in the light receiving step; Also provided is a method for detecting eye information, including: The present technology includes a step of irradiating invisible light onto at least one optical element provided on a substrate facing the user's eyeball to generate a virtual light source of the invisible light; receiving the invisible light reflected by the eyeball via the optical element; detecting eyeball information based on a light reception result in the light receiving step; generating image light based on the detection result in the detecting step and guiding the image light to the eyeball; Also provided is a display method, including: [Brief explanation of the drawings]

[0007] [Figure 1] 1A and 1B are diagrams for explaining an eyeball information detecting device of Comparative Example 1. FIG. [Figure 2] 2A to 2C are diagrams for explaining the eyeball information detecting device of Comparative Example 2. FIG. [Figure 3] 3A and 3B are diagrams for explaining the eyeball information detecting device of Comparative Example 3. FIG. [Figure 4] 4A to 4C are diagrams for explaining a first configuration example of the eyeball information detecting device of the present technology. [Figure 5] 5A and 5B are diagrams for explaining a second configuration example of the eyeball information detecting device of the present technology. [Figure 6] 6A and 6B are diagrams for explaining a third configuration example of the eyeball information detecting device of the present technology. [Figure 7] 7A and 7B are diagrams illustrating a configuration of an eyeball information detecting device according to Example 1 of an embodiment of the present technology. [Figure 8] 1 is a block diagram showing functions of an eyeball information detecting device according to a first example of an embodiment of the present technology. [Figure 9] 10 is a flowchart for explaining an operation of the eyeball information detecting device according to Example 1 of an embodiment of the present technology. [Figure 10] 10A and 10B are diagrams illustrating a configuration of an eyeball information detecting device according to Example 2 of an embodiment of the present technology. [Figure 11] 11A and 11B are diagrams illustrating a configuration of an eyeball information detecting device according to a third example of an embodiment of the present technology. [Figure 12] 12A and 12B are diagrams illustrating a configuration of an eyeball information detecting device according to Example 4 of an embodiment of the present technology. [Figure 13] FIG. 10 is a block diagram showing functions of an eyeball information detecting device according to Example 4 of an embodiment of the present technology. [Figure 14]14A and 14B are diagrams illustrating a configuration of an eyeball information detecting device according to a fifth example of an embodiment of the present technology. [Figure 15] FIG. 10 is a block diagram showing functions of an eyeball information detecting device according to a fifth example of an embodiment of the present technology. [Figure 16] 16A and 16B are diagrams illustrating a configuration of an eyeball information detecting device according to Example 6 of an embodiment of the present technology. [Figure 17] FIG. 13 is a block diagram showing functions of an eyeball information detecting device according to a sixth example of an embodiment of the present technology. [Figure 18] 18A and 18B are diagrams illustrating a configuration of an eyeball information detecting device according to Example 7 of an embodiment of the present technology. [Figure 19] FIG. 13 is a diagram illustrating a configuration of an eyeball information detecting device according to an eighth example of an embodiment of the present technology. [Figure 20] FIG. 13 is a block diagram showing the functions of an eyeball information detecting device according to an eighth example of an embodiment of the present technology. [Figure 21] 13 is a flowchart illustrating an operation of an eyeball information detecting device according to Example 8 of an embodiment of the present technology. [Figure 22] 22A and 22B are diagrams illustrating a configuration of an eyeball information detecting device according to Example 9 of an embodiment of the present technology. [Figure 23] 23A and 23B are diagrams illustrating a configuration of an eyeball information detecting device according to a tenth example of an embodiment of the present technology. [Figure 24] FIG. 16 is a diagram illustrating a configuration of an eyeball information detecting device according to an eleventh example of an embodiment of the present technology. [Figure 25] FIG. 23 is a block diagram showing the functions of an eyeball information detection device according to an eleventh example of an embodiment of the present technology. [Figure 26] 16 is a flowchart for explaining the operation of an eyeball information detecting device according to an eleventh example of an embodiment of the present technology. [Figure 27] 27A and 27B are diagrams illustrating a configuration of an eyeball information detecting device according to a twelfth example of an embodiment of the present technology. [Figure 28] FIG. 23 is a block diagram showing the functions of an eyeball information detection device according to a twelfth example of an embodiment of the present technology. [Figure 29] 23 is a flowchart for explaining the operation of the eyeball information detecting device according to Example 12 of an embodiment of the present technology. [Figure 30] 30A and 30B are diagrams illustrating a configuration of an eyeball information detecting device according to a thirteenth example of an embodiment of the present technology. [Figure 31] FIG. 23 is a block diagram showing the functions of an eyeball information detection device according to a thirteenth example of an embodiment of the present technology. [Figure 32] 23 is a flowchart illustrating an operation of an eyeball information detecting device according to a thirteenth example of an embodiment of the present technology. [Figure 33] 33A and 33B are diagrams illustrating a configuration of an eyeball information detecting device according to Example 14 of an embodiment of the present technology. [Figure 34] FIG. 23 is a block diagram showing the functions of an eyeball information detection device according to a fourteenth example of an embodiment of the present technology. [Figure 35] 23 is a flowchart for explaining the operation of an eyeball information detecting device according to Example 14 of an embodiment of the present technology. [Figure 36] 36A and 36B are diagrams illustrating a configuration of an eyeball information detecting device according to a fifteenth example of an embodiment of the present technology. [Figure 37] FIG. 23 is a block diagram showing the functions of an eyeball information detection device according to a fifteenth example of an embodiment of the present technology. [Figure 38] 23 is a flowchart for explaining the operation of an eyeball information detecting device according to Example 15 of an embodiment of the present technology. [Figure 39] 39A and 39B are diagrams illustrating a configuration of a display device including an eyeball information detecting device according to Example 7 of an embodiment of the present technology. [Figure 40] FIG. 13 is a block diagram showing functions of a display device including an eyeball information detecting device according to Example 7 of an embodiment of the present technology. [Figure 41] 13 is a flowchart for explaining an operation of a display device including an eyeball information detecting device according to Example 7 of an embodiment of the present technology. [Figure 42]42A to 42D are diagrams showing a diffusion type HOE, a concave mirror type HOE, a convex mirror type HOE, and a wavefront reproduction type HOE, respectively. DETAILED DESCRIPTION OF THE INVENTION

[0008] Preferred embodiments of the present technology will be described in detail below with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted. The embodiments described below illustrate typical embodiments of the present technology, and the scope of the present technology should not be interpreted as being narrow. Even when it is described in this specification that an eyeball information detection device, a display device, an eyeball information detection method, and a display method according to the present technology achieve multiple effects, it is sufficient that the eyeball information detection device, the display device, the eyeball information detection method, and the display method according to the present technology achieve at least one effect. The effects described in this specification are merely examples and are not limiting, and other effects may also be achieved.

[0009] The explanation will be given in the following order: 1. Introduction 2. Eyeball information detection devices of comparative examples 1 to 3. 3. Eyeball information detection devices according to configuration examples 1 to 3 of the present technology 4. Eyeball Information Detection Device of Example 1 of an Embodiment of the Present Technology 5. Eyeball Information Detection Device of Example 2 of an Embodiment of the Present Technology 6. Eyeball information detection device according to Example 3 of an embodiment of the present technology 7. Eyeball information detection device according to Example 4 of an embodiment of the present technology 8. Eyeball Information Detection Device of Example 5 of an Embodiment of the Present Technology 9. Eyeball Information Detection Device of Example 6 of an Embodiment of the Present Technology 10. Eyeball information detection device according to Example 7 of an embodiment of the present technology 11. Eyeball information detection device according to Example 8 of an embodiment of the present technology 12. Eyeball information detection device according to Example 9 of an embodiment of the present technology 13. Eyeball information detection device according to Example 10 of an embodiment of the present technology 14. Eyeball information detection device according to Example 11 of an embodiment of the present technology 15. Eyeball information detection device according to Example 12 of an embodiment of the present technology 16. Eyeball information detection device according to Example 13 of an embodiment of the present technology 17. Eyeball information detection device according to Example 14 of an embodiment of the present technology 18. Eyeball information detection device according to Example 15 of an embodiment of the present technology 19. Display device including eyeball information detection device according to Example 7 of an embodiment of the present technology 20. Modifications of this technology

[0010] <1. Introduction> Eye sensing (sensing eye information) is expected to have a variety of applications. Eye-sensing is expected to be used in research fields such as neuroscience, bioengineering, and medicine, in the industrial sector for technology transfer through eye-tracking and gaze-based UI (user interaction), and in security through iris authentication. Eye-sensing is also used for foveated rendering and expanding the viewing area (eyebox) in head-mounted displays (for AR / VR), where development competition has accelerated in recent years. Due to the expansion of these applications, there has been a growing need for near-eye eye sensing in recent years. For example, near-eye eye tracking involves sensing using an invisible light source and an image sensor that captures the eye mounted on a head-mounted device (e.g., eyeglass frames). As mentioned above, camera-based methods have generally been used for eye sensing for many years. Among these, pupil-cornea detection is a common gaze detection method. In this gaze detection method, it is important to distinguish and appropriately label (ID) the corneal reflections of light emitted from multiple light sources in order to improve detection accuracy. Here, we consider a method for miniaturizing the head-mounted jig while still ensuring the identification of corneal reflection light. Miniaturizing the head-mounted jig shortens the eye relief. For this reason, if a light source is placed on the rim that holds the base of an eyeglass frame (corresponding to the eyeglass lens), the light from the light source cannot be directed to the eye, making it impossible to capture an image of corneal reflection light. To address this issue, a design has been proposed in which the light source is mounted on a board attached to the rim of the eyeglass frame, but this design detracts from the appearance and makes it difficult to provide a clear view due to the light source being in the field of view. Patent Document 1 (Japanese Patent Laid-Open Publication No. 2003-225207) attempts to address this problem. In Patent Document 1, light from a light source is irradiated onto the eye via a reflective film (plane mirror) on a substrate held on the rim of an eyeglass frame. However, in Patent Document 1, the virtual image of the light source is located at a position significantly farther from the light source than the frame, causing the corneal reflected light to appear closer in the captured image. This is disadvantageous for labeling the corneal reflected light. In other words, Patent Document 1 leaves room for improvement in detection accuracy. Therefore, the inventor developed an eyeball information detection device according to the present technology as an eyeball information detection device capable of improving detection accuracy.

[0011] <2. Eyeball Information Detecting Devices of Comparative Examples 1 to 3>

[0012] (Eyeball information detection device of comparative example 1) Fig. 1A is a diagram showing the configuration and operation of an eyeball information detection device C1 (near-eye type) of Comparative Example 1. Fig. 1B is a diagram showing an image captured by the eyeball information detection device C1 (with a large-rimmed eyeglass frame GF). In the eyeball information detection device C1 of Comparative Example 2, as shown in FIG. 1A, an imaging unit IP (e.g., a camera) is provided on the temple of the eyeglass frame GF, and multiple (e.g., four) invisible light sources LS1 and LS2 (only two invisible light sources are shown in FIG. 1A) are provided on the rim (holding part that holds the substrate S) of the eyeglass frame GF. If the rim of the spectacle frame GF is enlarged to ensure a sufficient field of view, the invisible light beams L1 and L2 emitted from the invisible light sources LS1 and LS2, respectively, cannot be incident on the pupil of the eyeball EB, and a Purkinje image cannot be captured (see Figure 1B).On the other hand, if the rim of the spectacle frame GF is reduced, the invisible light beams L1 and L2 can be incident on the pupil of the eyeball EB, and a Purkinje image can be captured, but the field of view is limited. That is, in the eyeball information detecting device C1 of Comparative Example 1, there is a trade-off between ensuring the visual field and capturing the Purkinje image. For example, if each invisible light source is placed on the substrate S instead of on a rim, it is possible to capture a Purkinje image while ensuring the field of view. However, since the invisible light sources, wiring, etc. are within the field of view, unnecessary images will be visible and the appearance will be marred.

[0013] (Eyeball information detection device of comparative example 2) Fig. 2A is a diagram showing the configuration and operation of eyeball information detection device C2 (near-eye type) of Comparative Example 2. Fig. 2B is an optically equivalent diagram of Fig. 2A. Fig. 2C is a diagram showing a captured image captured by eyeball information detection device C2 of Fig. 2A. In the eyeball information detection device C2 of Comparative Example 2, as shown in FIG. 2A, multiple (e.g., four) invisible light sources LS1, LS2 (only two invisible light sources are shown in FIG. 2A) and an imaging unit IP are provided on the temples of the eyeglass frame GF, and a plane mirror PM (semi-transparent mirror) is provided on a substrate S held by the eyeglass frame GF. In the eye information detecting device C2, invisible light L1 emitted from invisible light source LS1 is reflected by the plane mirror PM and incident on the pupil of the eye EB. At this time, a virtual image VI1 of invisible light source LS1 is formed at a position symmetrical to invisible light source LS1 with respect to the plane mirror PM. Invisible light L2 emitted from invisible light source LS2 is reflected by the plane mirror PM and incident on the pupil of the eye EB. At this time, a virtual image VI2 of invisible light source LS2 is formed at a position symmetrical to invisible light source LS2 with respect to the plane mirror PM. In Comparative Example 2, the distance between the reflection position of the invisible light L1 on the plane mirror PM and the virtual image VI1 and the distance between the reflection position of the invisible light L2 on the plane mirror PM and the virtual image VI2 are relatively long. Therefore, as shown in Figures 2B and 2C, the Purkinje images PI, which are corneal reflection images of each invisible light, are positioned close to each other on the image captured by the imaging unit IP, making it difficult to separate them. Furthermore, in Comparative Example 2, aberration can be suppressed because invisible light is reflected by the plane mirror PM, but the plane mirror PM is relatively thick, making it difficult to reduce the size.

[0014] (Eyeball information detection device of comparative example 3) Fig. 3A is a diagram showing the configuration and operation of an eyeball information detecting device C3 (near-eye type) of Comparative Example 3. Fig. 3B is a diagram showing an image captured by the eyeball information detecting device C3. In the eyeball information detection device C3 of Comparative Example 3, as shown in FIG. 3A, multiple (e.g., four) invisible light sources LS1, LS2 (only two invisible light sources are shown in FIG. 3A) and an imaging unit IP are provided on the temples of the eyeglass frame GF, and multiple (e.g., four) small plane mirrors PM1, PM2 (all semi-transparent mirrors; only two plane mirrors are shown in FIG. 3A) are provided on a substrate S held on the rim of the eyeglass frame GF. In the eye information detecting device C3, invisible light L1 emitted from invisible light source LS1 is reflected by plane mirror PM1 and enters the pupil of eye EB. At this time, a virtual image VI1 of invisible light source LS1 is formed at a position symmetrical to invisible light source LS1 with respect to the plane mirror PM1. Invisible light L2 emitted from invisible light source LS2 is reflected by plane mirror PM2 and enters the pupil of eye EB. At this time, a virtual image VI2 of invisible light source LS2 is formed at a position symmetrical to invisible light source LS2 with respect to the plane mirror PM2. In Comparative Example 3, the distance between the plane mirror PM1 and the virtual image VI1 and the distance between the plane mirror PM and the virtual image VI2 are relatively long, so that the Purkinje images PI, which are corneal reflection images of each invisible light, are positioned close to each other on the image captured by the imaging unit IP, as shown in Fig. 3B, making it difficult to separate them.

[0015] 3. Eyeball Information Detection Devices of Configuration Examples 1 to 3 of the Present Technology (Eyeball information detection device according to configuration example 1 of the present technology) Fig. 4A is a diagram showing the configuration and operation of an eyeball information detection device A1 (near-eye type) according to Configuration Example 1 of the present technology. Fig. 4B is an optically equivalent diagram of Fig. 4A. Fig. 4C is a diagram showing an image captured by the eyeball information detection device A1 of Fig. 4A. In the eyeball information detection device A1 of configuration example 1, as shown in FIG. 4A, a plurality (e.g., four) of invisible light sources LS1, LS2 (only two invisible light sources are shown in FIG. 4A) and an imaging unit IP are provided on the temples of the eyeglass frame GF, and a plurality (e.g., four) of small curved mirrors CM1, CM2 (e.g., concave mirrors, convex mirrors, etc.; only two curved mirrors are shown in FIG. 4A) are provided on a substrate S held on the rim of the eyeglass frame GF. In the eye information detecting device A1, invisible light L1 emitted from an invisible light source LS1 is reflected by a curved mirror CM1 and enters the pupil of the eye EB. When the invisible light L1 is emitted, the concave mirror CM1 generates a virtual light source VLS1 for the invisible light L1. When the invisible light L2 is emitted from an invisible light source LS2, the curved mirror CM2 reflects the invisible light L2 and enters the pupil of the eye EB. When the invisible light L2 is emitted, the concave mirror CM2 generates a virtual light source VLS2 for the invisible light L2. In configuration example 1, the distance between curved mirror CM1 and virtual light source VLS1 and the distance between curved mirror CM2 and virtual light source VLS2 are relatively short, so that the Purkinje images PI, which are corneal reflection images of each invisible light, are positioned far apart on the image captured by imaging unit IP, as shown in Figures 4B and 4C, making it easy to separate them.

[0016] (Eyeball information detection device according to configuration example 2 of the present technology) Fig. 5A is a diagram showing the configuration and operation of an eyeball information detecting device A2 (near-eye type) according to Configuration Example 2 of the present technology, Fig. 5B is a diagram showing a captured image captured by the eyeball information detecting device A2 of Fig. 5A. In the eyeball information detection device A2 of configuration example 2, as shown in Figure 5A, multiple (e.g., four) invisible light sources LS1 and LS2 (only two invisible light sources are shown in Figure 5A) and an imaging unit IP are provided on the temples of the eyeglass frame GF, and multiple (e.g., four) small diffusion elements DE1 and DE2 are provided on a substrate S held on the rim of the eyeglass frame GF. In the eye information detecting device A2, invisible light L1 emitted from an invisible light source LS1 is diffusely reflected by a diffusing element DE1 and enters the pupil of the eye EB. When the invisible light L1 is emitted, the diffusing element DE1 generates a virtual light source VLS1 for the invisible light LS1. When the invisible light L2 is emitted from an invisible light source LS2, the diffusing element DE2 diffusely reflects the invisible light L2 and enters the pupil of the eye EB. When the invisible light L2 is emitted, the diffusing element DE2 generates a virtual light source VLS2 for the invisible light L2. In configuration example 2, the distance between diffusing element DE1 and virtual light source VLS1 and the distance between diffusing element DE2 and virtual light source VLS2 are relatively short, so that the Purkinje images PI, which are corneal reflection images of each invisible light, are positioned far apart on the image captured by imaging unit IP, as shown in Fig. 5B, making it easy to separate them.

[0017] (Eyeball information detection device according to configuration example 3 of the present technology) Fig. 6A is a diagram showing the configuration and operation of an eyeball information detecting device A3 (near-eye type) according to Configuration Example 3 of the present technology, Fig. 6B is a diagram showing a captured image captured by the eyeball information detecting device A3 of Fig. 6A. In the eyeball information detection device A3 of configuration example 3, as shown in FIG. 6A, a plurality (for example, four) of invisible light sources LS1 and LS2 (only two invisible light sources are shown in FIG. 6A) and an imaging unit IP are provided on the temples of the eyeglass frame GF, and a plurality (for example, four) of small hologram elements HOE1 and HOE2 are provided on a substrate S held on the rim of the eyeglass frame GF. In the eye information detecting device A3, invisible light L1 emitted from an invisible light source LS1 is diffusely reflected by a hologram element HOE1 and enters the pupil of the eye EB. When the hologram element HOE1 is irradiated with the invisible light L1, it generates a virtual light source VLS1 for the invisible light L1. When the hologram element HOE1 is irradiated with the invisible light L2, it generates a virtual light source VLS2 for the invisible light L2. When the hologram element HOE2 is irradiated with the invisible light L2, it generates a virtual light source VLS2 for the invisible light L2. In configuration example 3, the distance between hologram element HOE1 and virtual light source VLS1 and the distance between hologram element HOE2 and virtual light source VLS2 are relatively short, so that the Purkinje images PI, which are corneal reflection images of each invisible light, are positioned far apart on the image captured by imaging unit IP, as shown in Fig. 6B, making it easy to separate them.

[0018] <4. Eyeball Information Detection Device of Example 1 of One Embodiment of the Present Technology> An eyeball information detecting device according to Example 1 of an embodiment of the present technology will be described below with reference to Figs. 7A to 9. Fig. 7A is a diagram of the eyeball information detecting device 10-1 of Example 1 as seen from above. Fig. 7B is a diagram of the eyeball information detecting device 10-1 of Example 1 as seen from the eyeball EB side. Fig. 8 is a block diagram showing the functions of the eyeball information detecting device 10-1 of Example 1. Fig. 9 is a flowchart for explaining the operation of the eyeball information detecting device 10-1 of Example 1.

[0019] [Configuration of eyeball information detection device] The eyeball information detecting device 10-1 is a device (eye sensing device) that detects eyeball information, which is information about the user's eyes. The eyeball information detecting device 10-1 is mounted, for example, on an HMD (Head Mounted Display) and used for eye tracking, etc. Here, the "eyeball information" may include, for example, at least one of the orientation of the eyeball, the position and size of the pupil, and the position of the iris.

[0020] As shown in Figures 7A and 7B, the eyeball information detection device 10-1 includes a support member 50, a substrate 150, an illumination system 100-1 (see Figure 8), multiple (e.g., four) optical elements 201, a camera 300 (light receiving system), and a detection system 400-1 (see Figure 8).

[0021] (support member) The support member 50 is, for example, a spectacle frame worn on the user's head. The support member 50 has, for example, temples including ear hooks, and rims connected to the temples and holding the substrate 150. Note that the support member 50 may be a rimless frame in which the temples directly hold the substrate 150.

[0022] (substrate) As an example, the substrate 150 is provided on the support member 50 so as to face the user's eyeball EB. That is, the substrate 150 faces the user's eyeball EB when the support member 50 is attached to the user's head. The substrate 150 is a resin plate or a glass plate, and may be transparent, translucent, or opaque.

[0023] (irradiation system) The illumination system 100-1 is provided at the temple of the support member 50, for example. The illumination system 100-1 includes a plurality of (for example, four) light sources 101 (for example, first to fourth light sources 101-1, 101-2, 101-3, and 101-4) and a light source driving unit 110 (light source driver) that drives each light source 101 (see FIG. 8). The plurality of light sources 101 emit invisible light toward the substrate 150. Each light source 101 emits, for example, infrared light. Each light source 101 may be, for example, an LED (Light Emitting Diode), an OLED (Organic Light Emitting Diode), an LD (Laser diode), or the like.

[0024] (Optical elements) A plurality of (for example, four) optical elements 201 (for example, first to fourth optical elements 201-1 to 201-4)) are provided on substrate 150 so as to be irradiated with invisible light from illumination system 100-1, and generate a virtual light source of the invisible light.

[0025] The first optical element 201-1 is disposed on the substrate 150 at a position on the optical path of the invisible light L1 emitted from the first light source 101-1. The second optical element 201-2 is disposed on the substrate 150 at a position on the optical path of the invisible light L2 emitted from the second light source 101-2. The third optical element 201-3 is disposed on the substrate 150 at a position on the optical path of the invisible light L3 emitted from the third light source 101-3. The fourth optical element 201-4 is disposed on the substrate 150 at a position on the optical path of the invisible light L4 emitted from the fourth light source 101-4.

[0026] As an example, each optical element 201 is provided on the surface of the substrate 150 facing the eyeball EB.

[0027] For example, a small concave mirror is used as each optical element 201. Note that at least one optical element 201 may be, for example, a small convex mirror.

[0028] A plurality of (for example, four) optical elements 201 are arranged two-dimensionally along the in-plane direction of the substrate 150 (for example, arranged at the four corners of a rectangle).

[0029] When invisible light L1 is irradiated from first light source 101-1, first optical element 201-1 generates a virtual light source 101-1V of the invisible light L1 on the side opposite to the eyeball EB side of substrate 150. The invisible light L1 that passes through first optical element 201-1 is irradiated onto eyeball EB while diverging at a predetermined divergence angle as if emitted from virtual light source 101-1V.

[0030] When invisible light L2 is irradiated from second light source 101-2, second optical element 201-2 generates a virtual light source 101-2V of invisible light L2 on the side opposite to eyeball EB of substrate 150. The invisible light L2 passing through second optical element 201-2 is irradiated onto eyeball EB while diverging at a predetermined divergence angle as if emitted from virtual light source 101-2V.

[0031] The third optical element 201-3 is irradiated with invisible light L3 from the third light source 101-3, and when the invisible light L3 is irradiated, generates a virtual light source of the invisible light L3 on the side opposite to the eyeball EB side of the substrate 150. The invisible light L3 that passes through the third optical element 201-3 is irradiated onto the eyeball EB while diverging at a predetermined divergence angle as if emitted from the virtual light source.

[0032] The fourth optical element 201-4 is irradiated with invisible light L4 from the fourth light source 101-4, and when the invisible light L4 is irradiated, generates a virtual light source of the invisible light L4 on the side opposite to the eyeball EB side of the substrate 150. The invisible light L4 that passes through the fourth optical element 201-4 is irradiated onto the eyeball EB while diverging at a predetermined divergence angle as if emitted from the virtual light source.

[0033] That is, the plurality of light sources 101 correspond to different optical elements among the plurality of optical elements 201. More specifically, the plurality of light sources 101 correspond to the plurality of optical elements 201 individually.

[0034] (Light receiving system) The camera 300 as a light receiving system receives (images) invisible light (corneal reflected light) reflected by the eyeball EB via each optical element 201, and outputs the light receiving result (image capturing result) to the detection system 400-1. As an example, the camera 300 is provided at a position slightly shifted from the position facing the eyeball EB on the substrate 150 so that the eyeball EB is within the angle of view. The camera 300 has an image sensor 300a (e.g., an image sensor) as a light receiving element (see FIG. 8). The camera 300 may further have a light receiving optical system having a light receiving lens that forms an image of invisible light reflected by the eye EB on the image sensor 300a. The image pickup element 300a receives invisible light reflected by the eyeball EB at a plurality of pixels, performs photoelectric conversion for each pixel, and outputs captured image data made up of the resulting electrical signals to the detection system 400-1.

[0035] (Detection system) The detection system 400-1 detects eyeball information (for example, the direction of the eyeball EB) that is information about the eyeball EB based on the imaging result (captured image data) of the camera 300.

[0036] The detection system 400-1 includes a corneal reflection detection unit 400a and a gaze estimation unit 400b (see FIG. 8). The detection system 400-1 is realized by hardware including, for example, a CPU, a chipset, and the like.

[0037] The corneal reflection image detection unit 400a detects a Purkinje image, which is an image (corneal reflection image) of invisible light (corneal reflected light) reflected by the eyeball EB, from the captured image data from the imaging element 300a, and outputs the detection result to the gaze estimation unit 400b.

[0038] The gaze estimation unit 400b estimates the gaze, which is the direction of the eyeball EB, based on the detection result of the corneal reflection detection unit 400a, and outputs the estimation result.

[0039] [Operation of eyeball information detection device] The operation of the eyeball information detecting device 10-1 of the first embodiment (the eyeball information detecting method using the eyeball information detecting device 10-1) will be described below with reference to the flowchart of FIG. In the first step S1, the illumination system 100-1 illuminates invisible light onto a plurality of (for example, four) optical elements 201. Specifically, the light source driving unit 110 drives (turns on) the first to fourth light sources 101-1 to 101-4. As a result, the invisible light emitted from each light source 101 is illuminated onto the corresponding optical element 201, and the optical element 201 generates a virtual light source. The invisible light that passes through the optical element 201 is illuminated onto the eyeball EB and reflected by the eyeball EB.

[0040] In the next step S2, the camera 300 captures an image. Specifically, the image sensor 300a captures an image of the reflected light (corneal reflected light) of invisible light from the eye EB via each optical element 201, and outputs captured image data, which is the result of the imaging, to the corneal reflection image detection unit 400a.

[0041] In the next step S3, the detection system 400-1 detects a corneal reflection image (Purkinje image). Specifically, the corneal reflection image detection unit 400a detects a high-brightness image in the captured image data from the image sensor 300a as a corneal reflection image. At this time, the multiple corneal reflection images corresponding to the multiple optical elements 201 in the captured image data are detected spaced apart from each other (see FIG. 4C), allowing them to be separated with high accuracy.

[0042] In the next step S4, the detection system 400-1 estimates the gaze. Specifically, the gaze estimation unit 400b calculates the gaze, which is the direction of the eyeball EB, from the position of the corneal reflection detected by the corneal reflection detection unit 400a by a predetermined calculation, and outputs the calculation result.

[0043] [Effects of the eyeball information detection device and eyeball information detection method] An eyeball information detection device 10-1 of Example 1 of one embodiment of the present technology includes a support member 50 attached to a user's head, a substrate 150 arranged on the support member 50 facing the user's eyeball EB, an irradiation system 100-1 arranged on the support member 50 and including at least one light source 101 that emits invisible light toward the substrate 150, a plurality of optical elements 201 that are not plane mirrors and are arranged on the substrate 150 so that the invisible light is irradiated and that generate a virtual light source of the invisible light, a camera 300 (light receiving system) including an imaging element 300a (light receiving element) that receives the invisible light reflected by the eyeball EB via the plurality of optical elements 201, and a detection system 400-1 that detects eyeball information, which is information about the eyeball EB, based on the imaging result of the camera 300 (output of the light receiving system). In the eyeball information detecting device 10-1, a plurality of corneal reflection images corresponding to a plurality of optical elements 201 can be separated with high accuracy in the imaging result (captured image) of the camera 300, so that eyeball information (for example, line of sight) can be detected with high accuracy. As a result, the eyeball information detecting device 10-1 can provide an eyeball information detecting device that can improve detection accuracy.

[0044] In the eyeball information detecting device 10-1, the corneal reflection image can be separated with high accuracy even if the distance (eye relief) between the substrate 150 and the eyeball EB is short, so that a small-sized eyeball information detecting device with high detection accuracy can be realized.

[0045] The plurality of optical elements 201 includes at least three (for example, four) optical elements 201 arranged two-dimensionally along the in-plane direction of the substrate 150. This allows eyeball information to be detected with high accuracy with a relatively small amount of calculation and calculation time.

[0046] The at least one light source 101 is a plurality of light sources 101. This makes it possible to irradiate each of the plurality of optical elements 201 with invisible light relatively easily.

[0047] The plurality of light sources 101 correspond to different optical elements among the plurality of optical elements 201. This makes it possible to reliably irradiate each of the plurality of optical elements 201 with invisible light.

[0048] The plurality of light sources 101 correspond to the plurality of optical elements 201, respectively. This makes it possible to more reliably irradiate each of the plurality of optical elements 201 with invisible light.

[0049] The camera 300 serving as a light receiving system is provided on the substrate 150. This allows the invisible light reflected by the eyeball EB (corneal reflected light) to be directly and accurately captured.

[0050] The support member 50 has temples including ear hooks, and the plurality of light sources 101 are provided on the temples, which prevents the light sources 101 and their wiring from entering the user's field of vision.

[0051] Each optical element 201 is of a reflective type, so that invisible light irradiated onto the corresponding optical element 201 from each light source 101 provided on the temple of the support member 50 can be reflected and guided to the eyeball EB.

[0052] Each optical element 201 is a concave mirror, which allows each optical element 201 to generate a virtual light source of the irradiated invisible light at a position relatively close to the optical element 201. Each optical element 201 may be a convex mirror or a diffuser plate instead of a concave mirror. The plurality of optical elements 201 may be configured to include at least two of a concave mirror, a convex mirror, and a diffuser plate.

[0053] The eyeball information detection method using the eyeball information detection device 10-1 of Example 1 includes the steps of irradiating invisible light onto multiple (e.g., four) optical elements 201 provided on a substrate 150 facing the user's eyeball EB to generate a virtual light source of the invisible light, receiving (imaging) the invisible light reflected by the eyeball EB via the optical elements 201, and detecting eyeball information, which is information about the eyeball EB, based on the light reception result (imaging result) in the light reception (imaging) step. In this eyeball information detection method, a plurality of corneal reflection images corresponding to a plurality of optical elements 201 can be separated with high accuracy in the imaging result (captured image), so that eyeball information (for example, line of sight) can be detected with high accuracy. As a result, according to the eyeball information detection method, eyeball information can be detected with high accuracy.

[0054] <5. Eyeball Information Detection Device of Example 2 of One Embodiment of the Present Technology> An eyeball information detecting device according to Example 2 of an embodiment of the present technology will be described below with reference to Fig. 10A and Fig. 10B. Fig. 10A is a diagram of the eyeball information detecting device 10-2 according to Example 2 as seen from above. Fig. 10B is a diagram of the eyeball information detecting device 10-2 according to Example 2 as seen from the eyeball EB side.

[0055] In the eyeball information detection device 10-2 of Example 2, as shown in Figures 10A and 10B, the support member 55 (e.g., a spectacle frame) has temples including an ear hook portion 55a and an extension portion 55b extending to the side of the substrate 150 opposite the ear hook portion 55a side, and has a configuration generally similar to that of the eyeball information detection device 10-1 of Example 1, except that multiple light sources 101 (e.g., first to fourth light sources 101-1 to 101-4) are provided on the extension portion 55b. As an example, the substrate 150 is transparent to at least the emission wavelength of each light source 101 and transmits the invisible light from the light source 101 .

[0056] In the eyeball information detecting device 10-2, invisible light emitted from each light source 101 is transmitted through the substrate 150 and irradiated onto the corresponding optical element 201. When irradiated with invisible light, the optical element 201 generates a virtual light source of the invisible light. The invisible light that passes through the optical element 201 is irradiated onto the eyeball EB while diverging at a predetermined divergence angle.

[0057] According to the eyeball information detecting device 10-2 of the second embodiment, the same effects as those of the eyeball information detecting device 10-1 of the first embodiment are achieved.

[0058] <6. Eyeball Information Detection Device of Example 3 of One Embodiment of the Present Technology> An eyeball information detecting device according to Example 3 of an embodiment of the present technology will be described below with reference to Figs. 11A, 11B, and 42A to 42C. Fig. 11A is a diagram of the eyeball information detecting device 10-3 according to Example 3 as seen from above. Fig. 11B is a diagram of the eyeball information detecting device 10-3 according to Example 3 as seen from the eyeball EB side.

[0059] The eyeball information detection device 10-3 of Example 3 has a configuration generally similar to that of the eyeball information detection device 10-1 of Example 1, except that each optical element 202 is a diffraction element with a non-uniform pitch (e.g., a diffraction grating with a non-uniform grating pitch). Each optical element 202 as a diffractive element is broadly included in the category of DOE (Diffracted Optical Element). Each optical element 202 is, for example, a holographic optical element (HOE).

[0060] Optical element 202 as an HOE includes, for example, diffuser plate type HOE 202a (broadly speaking, a diffuser plate; see FIG. 42A), concave mirror type HOE 202b (broadly speaking, a concave mirror; see FIG. 42B), convex mirror type HOE 202c (broadly speaking, a convex mirror; see FIG. 42C), and wavefront reconstruction type HOE 202d (see FIG. 42D). In FIGS. 42A to 42D, the symbol L indicates invisible light irradiated onto the HOE. A diffused HOE is realized by using a one-dimensional HOE to illuminate a two-dimensional pattern, as shown in the right diagram of Figure 42A. A diffused HOE is effective in that it can provide uniform illumination while increasing the light-emitting area (area of ​​the virtual light source). The concave mirror type HOE 202b and the convex mirror type HOE 202c are effective in that they can narrow down the light emitting point (virtual light source) and can detect the position of the Purkinje image with high accuracy. The wavefront reconstruction type HOE 202d is advantageous in separating Purkinje images by distinguishing light emitting points (virtual light sources) by their shapes.

[0061] According to the eyeball information detection device 10-3 of Example 3, the optical element 202 can be made thinner, and therefore the substrate 150 can also be made thinner (resulting in, for example, improved transparency), and the virtual light source can be optically controlled, thereby further improving detection accuracy. Although a diffraction element having a single pitch can be used as each optical element 202, in this case there is a concern that the influence of astigmatism will be large and it will be difficult to separate the Purkinje images.

[0062] 7. Eyeball Information Detection Device of Example 4 of an Embodiment of the Present Technology An eyeball information detecting device according to a fourth example embodiment of the present technology will be described below with reference to Figs. 12A, 12B, and 13. Fig. 12A is a diagram of the eyeball information detecting device 10-4 according to the fourth example embodiment as seen from above. Fig. 12B is a diagram of the eyeball information detecting device 10-4 according to the fourth example embodiment as seen from the eyeball EB side. Fig. 13 is a block diagram showing the functions of the eyeball information detecting device 10-4 according to the fourth example embodiment.

[0063] As shown in Figures 12A, 12B and 13, the eyeball information detection device 10-4 of Example 4 has a configuration generally similar to that of the eyeball information detection device 10-3 of Example 3, except that the multiple (e.g., two) light sources 101 of the illumination system 100-4 include light sources 101 corresponding to at least two (e.g., two) optical elements 202 out of the multiple (e.g., four) optical elements 202. That is, in the eyeball information detecting device 10-4, the number of optical elements 202 is greater than the number of light sources 101. More specifically, in the eyeball information detecting device 10-4, the first light source 101-13 irradiates invisible light L13 onto first and third optical elements 202-1 and 202-3, which are spaced apart from each other above and below. When irradiated with the invisible light L13, each of the first and third optical elements 202-1 and 202-3 generates a virtual light source 101-13V for the invisible light L13. The invisible light L13 that passes through each of the first and third optical elements 202-1 and 202-3 is irradiated onto the eyeball EB while diverging at a predetermined divergence angle. In the eyeball information detection device 10-4, the second light source 101-24 irradiates invisible light L24 onto second and fourth optical elements 202-2 and 202-4, which are spaced apart from each other above and below. When irradiated with the invisible light L24, each of the second and fourth optical elements 202-2 and 202-4 generates a virtual light source 101-24V for the invisible light L24. The invisible light L24 that passes through each of the second and fourth optical elements 202-2 and 202-4 is irradiated onto the eyeball EB while diverging at a predetermined divergence angle.

[0064] According to the eyeball information detecting device 10-4 of the fourth embodiment, it is possible to achieve the same effects as the eyeball information detecting device 10-3 of the third embodiment, and also to reduce the number of parts (number of light sources).

[0065] The eyeball information detecting device 10-4 may have a single light source corresponding to a plurality of (for example, four) optical elements, in which case the number of parts can be further reduced.

[0066] <8. Eyeball Information Detection Device of Example 5 of an Embodiment of the Present Technology> An eyeball information detecting device according to Example 5 of an embodiment of the present technology will be described below with reference to Figs. 14A, 14B, and 15. Fig. 14A is a diagram of the eyeball information detecting device 10-5 according to Example 5 as seen from above. Fig. 14B is a diagram of the eyeball information detecting device 10-5 according to Example 5 as seen from the eyeball EB side. Fig. 15 is a block diagram showing the functions of the eyeball information detecting device 10-5 according to Example 5.

[0067] As shown in Figures 14A to 15, the eyeball information detection device 10-5 of Example 5 has a configuration generally similar to that of the eyeball information detection device 10-4 of Example 4, except that each of the multiple (e.g., two) light sources 101 of the irradiation system 100-5 corresponds to at least two (e.g., two) optical elements 202 of the multiple (e.g., four) optical elements 202, and invisible light from each of the multiple light sources 101 propagates within the substrate 150 and is irradiated onto the corresponding at least two (e.g., two) optical elements 202.

[0068] In the eyeball information detecting device 10-5, the first light source 101-12 corresponds to the first and second optical elements 202-1 and 202-2, and the second light source 101-34 corresponds to the third and fourth optical elements 202-3 and 202-4. The first and second light sources 101-12 and 101-34 are arranged one above the other. More specifically, the first light source 101-12 is arranged on the upper side, and the second light source 101-34 is arranged on the lower side. As an example, a transmissive diffractive element is used as the first and second optical elements 202-1 and 202-2. As an example, each optical element 202 is provided on the surface of the substrate 150 on the side of the eyeball EB. Note that reflective diffractive elements may be used as the first and second optical elements 202-1 and 202-2. In this case, however, each optical element 202 (each diffractive element) needs to be provided on the surface of the substrate 150 opposite to the eyeball EB side.

[0069] A prism 160 is provided on the surface of the substrate 150 opposite to the eyeball EB side, which allows the invisible light emitted from each light source 101 and transmitted through the substrate 150 to enter the substrate 150 so as to satisfy the total reflection conditions within the substrate 150 (at an incident angle that results in total reflection within the substrate 150). That is, in the eyeball information detecting device 10-5, the substrate 150 is used as a light guide plate.

[0070] Invisible light L12 emitted from first light source 101-12 and incident on substrate 150 via substrate 150 and prism 160 in this order propagates within substrate 150 while undergoing total reflection, and is sequentially irradiated onto first and second optical elements 202-1 and 202-2. When irradiated with invisible light L12, first optical element 202-1 generates virtual light source 101-12V1. The invisible light L12 irradiated onto first optical element 202-1 is transmitted and diffracted toward eyeball EB. When irradiated with invisible light L12, second optical element 202-2 generates virtual light source 101-12V2. The invisible light L12 irradiated onto second optical element 202-2 is transmitted and diffracted toward eyeball EB.

[0071] The invisible light L34 emitted from the second light source 101-34 and incident on the substrate 150 via the substrate 150 and the prism 160 in this order propagates within the substrate 150 while being totally reflected, and is sequentially irradiated onto the third and fourth optical elements 202-3 and 202-4. The third optical element 202-3 generates a virtual light source when irradiated with the invisible light L34. The invisible light L34 irradiated onto the third optical element 202-3 is transmitted and diffracted toward the eye EB. The fourth optical element 202-4 generates a virtual light source when irradiated with the invisible light L34. The invisible light L34 irradiated onto the fourth optical element 202-4 is transmitted and diffracted toward the eye EB.

[0072] The eyeball information detection device 10-5 of Example 5 has the same effect as the eyeball information detection device 10-4 of Example 4, and also transmits invisible light from each light source 101 through the substrate 150 and is incident on the eyeball EB via the corresponding optical element 202, thereby making it possible to shorten the eye relief, further miniaturization can be achieved, and light utilization efficiency can be improved.

[0073] 9. Eyeball Information Detection Device of Example 6 of an Embodiment of the Present Technology An eyeball information detecting device according to Example 6 of an embodiment of the present technology will be described below with reference to Figs. 16A, 16B, and 17. Fig. 16A is a diagram of the eyeball information detecting device 10-6 according to Example 6 as seen from above. Fig. 16B is a diagram of the eyeball information detecting device 10-6 according to Example 6 as seen from the eyeball EB side. Fig. 17 is a block diagram showing the functions of the eyeball information detecting device 10-6 according to Example 6.

[0074] The eyeball information detection device 10-6 of Example 6 has a configuration generally similar to that of the eyeball information detection device 10-5 of Example 5, except that the light source 101-1234 is a single light source corresponding to multiple (e.g., four) optical elements 202, and an optical element 250 is provided on the substrate 150 to guide the invisible light L1234 from the light source 101-1234 to multiple (e.g., four) optical elements 202. That is, in the eyeball information detecting device 10-6, the number of optical elements 202 is greater than the number of light sources 101. In the eyeball information detecting device 10-6, invisible light L1234 from the light source 101-1234 propagates through the substrate 150 and is irradiated onto a plurality of (for example, four) optical elements 202 via the optical member 250. The optical member 250 is, for example, a reflective diffraction element having a uniform grating pitch in two dimensions. More specifically, invisible light L1234 emitted from light source 101-1234 is transmitted through substrate 150 and incident on prism 160 into substrate 150 so as to satisfy the condition for total reflection. The invisible light L1234 propagating through substrate 150 while being totally reflected is reflected and diffracted in all directions by optical member 250, and separated into four invisible light beams L1 to L4, which are then irradiated onto the corresponding optical elements 202. Each optical element 202 generates a virtual light source when the corresponding invisible light is irradiated. The invisible light passing through each optical element 202 is irradiated onto the eye EB while diverging at a predetermined divergence angle.

[0075] The eyeball information detection device 10-6 of Example 6 has the same effect as the eyeball information detection device 10-5 of Example 5, and can efficiently and reliably irradiate invisible light onto multiple (e.g., four) optical elements 202 using a single light source 101-1234, thereby efficiently and reliably generating a virtual light source of the invisible light.

[0076] 10. Eyeball Information Detection Device of Example 7 of an Embodiment of the Present Technology An eyeball information detecting device according to Example 7 of an embodiment of the present technology will be described below with reference to Fig. 18A and Fig. 18B. Fig. 18A is a diagram of the eyeball information detecting device 10-7 according to Example 7 as seen from above. Fig. 18B is a diagram of the eyeball information detecting device 10-7 according to Example 7 as seen from the eyeball EB side.

[0077] The eyeball information detection device 10-7 of Example 7 has a configuration generally similar to that of the eyeball information detection device 10-1 of Example 1, except that a camera 300 as a light receiving system is provided on the temple of the eyeglass frame as the support member 50, and another optical element 270 (light receiving optical element) that guides invisible light reflected by the eyeball EB to the camera 300 is provided on the substrate 150.

[0078] In the eyeball information detecting device 10-7, a plurality of (for example, four) optical elements 202 are provided on the substrate 150 around another optical element 270. The other optical element 270 is, for example, a reflective diffraction element. The reflective diffraction element as the other optical element 270 is provided, for example, on the surface of the substrate 150 facing the eyeball EB at a position directly facing the eyeball EB. In addition, when the substrate 150 is transparent to the emission wavelength of each light source 101, for example, the reflective diffraction element may be provided on the surface of the substrate 150 opposite to the eyeball EB side.

[0079] The invisible light emitted from each light source 101 and passing through the corresponding optical element 201 is reflected by the eye EB (corneal reflected light) and is reflected and diffracted by another optical element 270 toward the camera 300 .

[0080] The eyeball information detection device 10-7 of Example 7 has the same effect as the eyeball information detection device 10-1 of Example 1, and the camera 300 as the light receiving system is provided on the temple of the support member 50, preventing it from entering the user's field of view, thereby preventing unnecessary images from being seen, and for example, improving see-through properties.

[0081] <11. Eyeball Information Detection Device of Example 8 of an Embodiment of the Present Technology> An eyeball information detecting device according to Example 8 of an embodiment of the present technology will be described below with reference to Figs. 19 to 21. Fig. 19 is a diagram showing the eyeball information detecting device 10-8 according to Example 8 as viewed from above. Fig. 20 is a block diagram showing the functions of the eyeball information detecting device 10-8 according to Example 8. Fig. 21 is a flowchart for explaining the operation of the eyeball information detecting device 10-8 according to Example 8.

[0082] As shown in Figures 19 and 20, the eyeball information detection device 10-8 of Example 8 has a configuration that is generally similar to that of the eyeball information detection device 10-1 of Example 1, except that multiple (e.g., three) optical elements 203 (e.g., first to third optical elements 203-1 to 203-3) are diffractive elements with non-uniform pitches and are wavefront reproduction type diffractive elements (wavefront reproduction type HOEs), and that the multiple optical elements 203 correspond to a single light source 101-123 of the illumination system 100-8.

[0083] The wavefront reproduction type HOE as each optical element 203 has a recorded wavefront, and when irradiated with invisible light, generates a virtual light source in the recorded wavefront shape. For example, each optical element 203 has a different wavefront shape recorded thereon. Light source 101-123 simultaneously irradiates first to third optical elements 203-1, 203-2, and 203-3 with invisible light (diffused light).

[0084] The first to third optical elements 203-1 to 203-3 are arranged in a horizontal row on the surface of the substrate 150 facing the eyeball EB, for example. Each optical element 203 is, for example, a reflective type. The first optical element 203-1 records a first wavefront shape (e.g., a star shape) and, when non-visible light L1 from the light source 101-123 is irradiated, generates a virtual light source 101-V1 of the non-visible light L1 in the first wavefront shape (e.g., a star shape) by wavefront reconstruction. The second optical element 203-2 has a wavefront recorded as a second wavefront shape (e.g., circular), and when invisible light L2 from the light source 101-123 is irradiated, generates a virtual light source 101-V2 of the invisible light L2 in the second wavefront shape (e.g., circular) by wavefront reconstruction. The third optical element 203-3 records a third wavefront shape (e.g., rectangular), and when invisible light L3 from the light source 101-123 is irradiated, generates a virtual light source 101-V3 of the invisible light L3 in the third wavefront shape (e.g., rectangular) by wavefront reconstruction.

[0085] The detection system 400-8 of the eyeball information detection device 10-8 has a fitting unit 400c in addition to a corneal reflection detection unit 400a and a line of sight estimation unit 400b. Based on the detection results of the corneal reflection detection unit 400a, the fitting unit 400c performs fitting between the shapes of multiple (e.g., four) corneal reflection images and the wavefront shapes (stored in an internal memory) recorded in multiple (e.g., four) optical elements 203, and outputs the results to the gaze estimation unit 400b. The gaze estimation unit 400b performs a predetermined calculation according to the fitting results and calculates the direction (gaze) of the eyeball EB.

[0086] The operation of the eyeball information detecting device 10-8 (the eyeball information detecting method using the eyeball information detecting device 10-8) will be described below with reference to the flowchart of FIG.

[0087] In the first step S11, the illumination system 100-8 illuminates invisible light onto a plurality of (e.g., three) optical elements 203. Specifically, the light source driving unit 110 drives (turns on) the light sources 101-123. As a result, the invisible light emitted from the light sources 101-123 is illuminated onto the corresponding optical elements 203, and the optical elements 203 generate a virtual light source. The invisible light that passes through the optical elements 203 is illuminated onto the eyeball EB and reflected by the eyeball EB.

[0088] In the next step S12, the camera 300 captures an image. Specifically, the image sensor 300a captures an image of invisible light that passes through each optical element 203 and is reflected by the eyeball EB (corneal reflected light).

[0089] In the next step S13, the detection system 400-8 detects the corneal reflection image (Purkinje image). Specifically, the corneal reflection image detection unit 400a detects a high-brightness image in the captured image data from the image sensor 300a as the corneal reflection image. At this time, the multiple corneal reflection images corresponding to the multiple optical elements in the captured image data are detected separated from each other (see FIG. 4C), allowing them to be separated with high accuracy.

[0090] In the next step S14, the detection system 400-8 fits the corneal reflection images to the stored wavefront shapes. Specifically, the fitting unit 400c fits the shapes of the multiple corneal reflection images to the wavefront shapes recorded in the multiple optical elements 203 based on the detection results of the corneal reflection detection unit 400a.

[0091] In the final step S15, detection system 400-8 estimates the gaze. Specifically, gaze estimation unit 400b performs a predetermined calculation according to the fitting result of fitting unit 400c, calculates the gaze, which is the direction of eyeball EB, and outputs the calculation result.

[0092] According to the eyeball information detection device 10-8 of Example 8, since the wavefront shapes recorded on the multiple optical elements 203 are different, the accuracy of fitting between the shapes of the multiple corneal reflection images and the wavefront shapes recorded on the multiple optical elements 203 can be improved, and the accuracy of gaze detection can be improved.

[0093] In the eighth embodiment, the plurality of optical elements 203 may include at least two optical elements 203 having the same wavefront shape recorded thereon. Even if the wavefront shapes recorded thereon are the same, for example, if the wavefront shape is a special shape, the fitting accuracy can be improved.

[0094] <12. Eyeball Information Detection Device of Example 9 of an Embodiment of the Present Technology> An eyeball information detecting device according to Example 9 of an embodiment of the present technology will be described below with reference to Fig. 22A and Fig. 22B. Fig. 22A is a diagram of the eyeball information detecting device 10-9 according to Example 9 as seen from above. Fig. 22B is a diagram of the eyeball information detecting device 10-9 according to Example 9 as seen from the eyeball EB side.

[0095] As shown in Figures 22A and 22B, the eyeball information detection device 10-9 of Example 9 has a configuration generally similar to that of the eyeball information detection device 10-1 of Example 1, except that the multiple optical elements 204 (e.g., first to fourth optical elements 204-1 to 204-4) are multiple diffusion elements, and the detection system fits multiple reflected images of invisible light from the eyeball EB to the multiple optical elements 204 (diffusion elements).

[0096] In the eyeball information detecting device 10-9, the optical elements 204 have the same shape (for example, a star shape). Note that the shape of each optical element 204 may be, for example, a circle, an ellipse, a polygon, or the like.

[0097] According to the eyeball information detecting device 10-9 of the ninth embodiment, it is possible to generate a plurality of virtual light sources having shapes corresponding to the shapes of the plurality of optical elements 204, thereby improving the fitting accuracy. In this case, by making the shapes of the plurality of optical elements 204 special shapes (for example, star shapes), it is possible to further improve the fitting accuracy.

[0098] <13. Eyeball Information Detection Device of Example 10 of an Embodiment of the Present Technology> An eyeball information detecting device according to Example 10 of an embodiment of the present technology will be described below with reference to Fig. 23A and Fig. 23B. Fig. 23A is a diagram of the eyeball information detecting device 10-10 according to Example 10 as seen from above. Fig. 23B is a diagram of the eyeball information detecting device 10-10 according to Example 10 as seen from the eyeball EB side.

[0099] The eyeball information detection device 10-10 of Example 10 has a configuration generally similar to that of the eyeball information detection device 10-9 of Example 9, except that the multiple optical elements 204 (e.g., the first to fourth optical elements 204-1 to 204-4), which are diffusion elements, have different shapes, as shown in Figures 23A and 23B.

[0100] According to the eyeball information detecting device 10-10 of the tenth embodiment, the shapes of the optical elements 204 (diffusion elements) are different, so that the fitting accuracy can be further improved.

[0101] In the eyeball information detecting device 10-10 of the tenth embodiment, the plurality of optical elements 204 may include optical elements 204 of the same shape.

[0102] <14. Eyeball Information Detection Device of Example 11 of an Embodiment of the Present Technology> An eyeball information detecting device according to an eleventh example of an embodiment of the present technology will be described below with reference to Figs. 24 to 26. Fig. 24 is a diagram showing the eyeball information detecting device 10-11 according to the eleventh example as viewed from above. Fig. 25 is a block diagram showing the functions of the eyeball information detecting device 10-11 according to the eleventh example. Fig. 26 is a flowchart for explaining the operation of the eyeball information detecting device 10-11 according to the eleventh example.

[0103] The eyeball information detecting device 10-11 of the embodiment 11 has a configuration generally similar to that of the eyeball information detecting device 10-1 of the embodiment 1, except that the diffraction powers of a plurality of (e.g., three) optical elements 205 (e.g., first to third optical elements 205-1 to 205-3) which are diffraction elements are different, and the positions of a plurality of virtual light sources generated by the plurality of optical elements 205 in the thickness direction of the substrate 150 are different. The diffraction elements have curved mirror characteristics (convex mirror characteristics with negative power or concave mirror characteristics with positive power).

[0104] The illumination system 100-8 of the eyeball information detecting device 10-11 has a single light source 101-123 corresponding to a plurality of optical elements 205.

[0105] The detection system 400-11 of the eyeball information detecting device 10-11 has a depth estimation unit 400d that estimates the depth of the eyeball EB from the imaging result of the camera 300, in addition to a corneal reflection detection unit 400a and a line of sight estimation unit 400b.

[0106] The operation of the eyeball information detecting device 10-11 (the eyeball information detecting method using the eyeball information detecting device 10-11) will be described below with reference to the flowchart of FIG.

[0107] In the first step S21, the illumination system 100-8 illuminates invisible light onto a plurality of (for example, three) optical elements 205. Specifically, the light source driving unit 110 drives (turns on) the light sources 101-123. As a result, the invisible light emitted from the light sources 101-123 is illuminated onto each optical element 205, and the optical elements 205 generate virtual light sources at different positions in the thickness direction of the substrate 150. The invisible light that passes through the optical elements 205 is illuminated onto the eyeball EB and reflected by the eyeball EB.

[0108] In the next step S22, the camera 300 captures an image. Specifically, the image sensor 300a captures an image of invisible light that passes through each optical element 205 and is reflected by the eyeball EB (corneal reflected light).

[0109] In the next step S23, the detection system 400-11 detects corneal reflection images (Purkinje images). Specifically, the corneal reflection detection unit 400a detects high-brightness images in the captured image data from the image sensor 300a as corneal reflection images. At this time, multiple (e.g., three) corneal reflection images corresponding to multiple (e.g., three) optical elements 205 in the captured image data are detected spaced apart from each other (see FIG. 4C ), allowing for accurate separation.

[0110] In the next step S24, the detection system 400-11 estimates the depth of the eyeball EB. Specifically, the depth estimation unit 400d estimates the depth of the eyeball EB based on the positions of the multiple corneal reflections detected by the corneal reflection detection unit 400a, and outputs the estimation result to the gaze estimation unit 400b. Additionally, since the positions of the corneal reflections differ depending on the depth of the eyeball EB, it is possible to estimate the depth of the eyeball EB from the positions of the corneal reflections.

[0111] In the final step S25, detection system 400-11 estimates the gaze. Specifically, gaze estimation unit 400b performs a predetermined calculation using the estimation result from depth estimation unit 400d to calculate the gaze, which is the direction of eyeball EB, and outputs the calculation result.

[0112] According to the eyeball information detection device 10-11 of Example 11, the direction (gaze) of the eyeball EB can be detected taking into account the depth of the eyeball EB, so that the direction (gaze) of the eyeball EB can be detected with high accuracy regardless of the depth of the eyeball EB.

[0113] <15. Eyeball Information Detection Device of Example 12 of Embodiment of the Present Technology> An eyeball information detecting device according to a twelfth example of an embodiment of the present technology will be described below with reference to Figs. 27A to 29. Fig. 27A is a diagram (part 1) of the eyeball information detecting device 10-12 according to the twelfth example seen from above. Fig. 27B is a diagram (part 2) of the eyeball information detecting device 10-12 according to the twelfth example seen from above. Fig. 28 is a block diagram showing the functions of the eyeball information detecting device 10-12 according to the twelfth example. Fig. 29 is a flowchart for explaining the operation of the eyeball information detecting device 10-12 according to the twelfth example.

[0114] As shown in Figures 27A to 28, the eyeball information detection device 10-12 of Example 12 has a configuration generally similar to that of the eyeball information detection device 10-5 of Example 5, except that multiple (e.g., eight) optical elements 202 are positioned at different thickness-wise positions in the substrate 150 and include first and second optical element groups corresponding to different light source groups, and the illumination system 100-12 has a light source driving unit 110 that selectively drives a first light source group 101-12G corresponding to the first optical element group and a second light source group 101-34G corresponding to the second optical element group.

[0115] The first light source group 101-12G includes two first light sources 101-12 arranged one above the other. In Figures 27A and 27B, only the upper first light source 101-12 is shown. The second light source group 101-34G includes two second light sources 101-34 arranged one above the other. In Figures 27A and 27B, only the upper second light source 101-34 is shown.

[0116] The first optical element group includes two pairs of first and second optical elements 202-1 and 202-2 provided on the surface of the substrate 150 opposite the eyeball EB side. The two pairs of first and second optical elements 202-1 and 202-2 are arranged one above the other. Only the upper pair of first and second optical elements 202-1 and 202-2 is shown in Figures 27A and 27B. The upper set of first and second optical elements 202-1, 202-2 corresponds to the upper first light source 101-12, and the lower set of first and second optical elements 202-1, 202-2 corresponds to the lower first light source 101-12. Reflective diffractive elements are used as the first and second optical elements 202-1 and 202-2 of each pair.

[0117] The second optical element group includes two pairs of third and fourth optical elements 202-3 and 202-4 provided on the surface of the substrate 150 facing the eyeball EB. The two pairs of third and fourth optical elements 202-3 and 202-4 are arranged one above the other. Only the upper pair of third and fourth optical elements 202-3 and 202-4 is shown in Figures 27A and 27B. The upper set of third and fourth optical elements 202-3, 202-4 corresponds to the upper second light source 101-34, and the lower set of third and fourth optical elements 202-3, 202-4 corresponds to the lower second light source 101-34. The third and fourth optical elements 202-3 and 202-4 of each pair are transmissive diffractive elements.

[0118] First and second prisms 160-12 and 160-34 are provided on the surface of the substrate 150 opposite to the eyeball EB side. The first prism 160-12 causes the invisible light L12 emitted from each of the first light sources 101-12 in the first light source group 101-12G to enter the substrate 150 so as to satisfy the total reflection condition. The second prism 160-34 causes the invisible light L34 emitted from each second light source 101-34 of the second light source group 101-34G to enter the substrate 150 so as to satisfy the total reflection condition.

[0119] 27A , invisible light L12 emitted from each first light source 101-12 in first light source group 101-12G is incident on first prism 160-12 via substrate 150. The invisible light L12 transmitted through first prism 160-12 propagates within substrate 150 while being totally reflected, and is sequentially irradiated onto corresponding first and second optical elements 202-1 and 202-2. When irradiated with invisible light L12, first optical element 202-1 generates a virtual light source 101-1V for the invisible light L12. When irradiated with invisible light L12, second optical element 202-2 generates a virtual light source 101-2V for the invisible light L12.

[0120] 27B, invisible light L34 emitted from each second light source 101-34 in the second light source group 101-34G is incident on the second prism 160-34 via the substrate 150. The invisible light L34 transmitted through the second prism 160-34 propagates within the substrate 150 while being totally reflected, and is irradiated onto the corresponding third and fourth optical elements 202-3 and 202-4. When irradiated with the invisible light L34, the third optical element 202-3 generates a virtual light source 101-3V for the invisible light L34. When irradiated with the invisible light L34, the fourth optical element 202-4 generates a virtual light source 101-4V for the invisible light L34.

[0121] In the illumination system 100-12 of the eyeball information detecting device 10-12, the light source driving section 110 has a switch section 110a that switches the driving target between the first and second light source groups 101-12G and 101-34G (see FIG. 28).

[0122] The detection system 400-12 of the eyeball information detecting device 10-12 has a corneal reflection detecting section 400a, a gaze estimating section 400b, and a correcting section 400e that corrects the positional deviation of the corneal reflection due to the depth of the eyeball EB (see FIG. 28).

[0123] The operation of the eyeball information detecting device 10-12 (the eyeball information detecting method using the eyeball information detecting device 10-12) will be described below with reference to the flowchart of FIG.

[0124] In the first step S31, the illumination system 100-12 illuminates the first optical element group with invisible light. Specifically, the light source driving unit 110 drives the upper and lower first light sources 101-12. As a result, the invisible light L12 emitted from each first light source 101-12 is sequentially illuminated onto the corresponding first and second optical elements 202-1 and 202-2, which generate virtual light sources 101-1V and 101-2V, respectively. The invisible light L12 that passes through each of the first and second optical elements 202-1 and 202-2 is illuminated onto the eyeball EB and reflected by the eyeball EB.

[0125] In the next step S32, the camera 300 captures an image. Specifically, the image sensor 300a captures an image of the invisible light L12 reflected by the eye EB (corneal reflection light) through each pair of the first and second optical elements 202-1 and 202-2, and outputs captured image data, which is the result of the imaging, to the corneal reflection image detection unit 400a.

[0126] In the next step S33, the detection system 400-12 detects corneal reflection images (Purkinje images). Specifically, the corneal reflection detection unit 400a detects high-brightness images in the captured image data output from the image sensor 300a in step S32 as corneal reflection images. At this time, multiple (e.g., four) corneal reflection images corresponding to multiple (e.g., four) optical elements 202 in the captured image data are detected spaced apart from each other (see FIG. 4C), allowing for accurate separation.

[0127] In the next step S34, the illumination system 100-12 illuminates the second optical element group with invisible light. Specifically, the light source driving unit 110 drives the upper and lower second light sources 101-34. As a result, the invisible light L34 emitted from each second light source 101-34 is sequentially illuminated onto the corresponding third and fourth optical elements 202-3 and 202-4, which generate virtual light sources 101-3V and 101-4V, respectively. The invisible light L34 that passes through each of the third and fourth optical elements 202-3 and 202-4 is illuminated onto the eyeball EB and reflected by the eyeball EB.

[0128] In the next step S35, the camera 300 captures an image. Specifically, the image sensor 300a captures an image of the invisible light L34 reflected by the eye EB (corneal reflection light) through each pair of the third and fourth optical elements 202-3 and 202-4, and outputs captured image data, which is the imaging result, to the corneal reflection image detection unit 400a.

[0129] In the next step S36, the detection system 400-12 detects corneal reflection images (Purkinje images). Specifically, the corneal reflection detection unit 400a detects high-brightness images in the captured image data output from the image sensor 300a in step S35 as corneal reflection images. At this time, multiple (e.g., four) corneal reflection images corresponding to multiple (e.g., four) optical elements 202 in the captured image data are detected spaced apart from each other (see FIG. 4C), allowing for accurate separation.

[0130] In the next step S37, the detection system 400-12 corrects the positional deviation of the corneal reflection image due to the depth of the eyeball EB. Specifically, the correction unit 400e obtains the positional deviation of the corneal reflection image from a reference position due to the depth of the eyeball EB from the multiple (e.g., four) corneal reflection images of the invisible light L12 detected in step S33 and the multiple (e.g., four) corneal reflection images of the invisible light L34 detected in step S36, calculates the position of the corneal reflection image with the positional deviation corrected, and outputs the calculation result to the gaze estimation unit 400b.

[0131] In the final step S38, detection system 400-12 estimates the gaze. Specifically, gaze estimation unit 400b performs a predetermined calculation based on the position of the corrected corneal reflection image from correction unit 400e, calculates the gaze, which is the direction of eyeball EB, and outputs the calculation result.

[0132] According to the eyeball information detection device 10-12 of the twelfth embodiment, the direction (gaze) of the eyeball EB is calculated taking into account the depth of the eyeball EB, so that the direction (gaze) of the eyeball EB can be detected with high accuracy regardless of the depth of the eyeball EB.

[0133] <16. Eyeball Information Detection Device of Example 13 of Embodiment of the Present Technology> An eyeball information detecting device according to a thirteenth example of an embodiment of the present technology will be described below with reference to Figs. 30A to 32. Fig. 30A is a diagram (part 1) of the eyeball information detecting device 10-13 according to the thirteenth example seen from above. Fig. 30B is a diagram (part 2) of the eyeball information detecting device 10-13 according to the thirteenth example seen from above. Fig. 31 is a block diagram showing the functions of the eyeball information detecting device 10-13 according to the thirteenth example. Fig. 32 is a flowchart for explaining the operation of the eyeball information detecting device 10-13 according to the thirteenth example.

[0134] As shown in Figures 30A to 31, the eyeball information detection device 10-13 of Example 13 has a configuration generally similar to that of the eyeball information detection device 10-5 of Example 5, except that multiple (e.g., four) optical elements 215 (e.g., two first optical elements 215-1 and two second optical elements 215-2) are diffraction elements having polarization dependency (the property of having different diffraction power depending on the polarization direction), and the illumination system 100-13 is capable of changing the polarization direction of invisible light.

[0135] The illumination system 100-13 of the eyeball information detection device 10-13 is configured such that the light source driving unit 110 drives the first light source group 101-12. PD1 G and the second light source group 101-12 PD2 G (see FIG. 31). 1st light source group 101-12 PD1 G is a first linearly polarized light L12 having a predetermined polarization direction. PD1 A plurality of (for example, two) first light sources 101-12 that emit (invisible light) PD1 (See FIGS. 30A and 30B.) A plurality of (for example, two) first light sources 101-12 PD1 are arranged one above the other. 2nd light source group 101-12 PD2 G is the first linearly polarized light L12 PD1 The second linearly polarized light L12 is perpendicular to the PD2 A plurality of (for example, two) second light sources 101-12 that emit (invisible light) PD2(See FIGS. 30A and 30B.) PD2 are arranged one above the other.

[0136] The two first optical elements 215-1 are arranged one above the other, and the two second optical elements 215-2 are arranged one above the other. The upper first and second optical elements 215-1 and 215-2 are connected to the upper first and second light sources 101-12. PD1 , 101-12 PD2 Corresponds to. The lower first and second optical elements 215-1 and 215-2 are connected to the lower first and second light sources 101-12. PD1 , 101-12 PD2 Corresponds to. Each optical element 215 is, for example, a reflective diffraction element provided on the surface of the substrate 150 opposite to the eyeball EB side, and is configured to reflect the first linearly polarized light L12 PD1 The diffraction power of the second linearly polarized light L12 PD2 is smaller than the diffraction power that diffracts the Each optical element 215 may be a transmissive diffractive element provided on the surface of the substrate 150 on the side of the eyeball EB.

[0137] As shown in FIG. 30A, the first light source group 101-12 PD1 Each first light source 101-12 in G PD1 The first linearly polarized light L12 emitted from PD1 is incident on the substrate 150 through the substrate 150 and the prism 160, propagates through the substrate 150 while being totally reflected, and is sequentially irradiated onto the corresponding first and second optical elements 215-1 and 215-2. PD1 are irradiated, the first linearly polarized light L12 PD1 Virtual Light Source 101-1V PD1 , 101-2V PD1 is generated at a position relatively far from the substrate 150.

[0138] As shown in FIG. 30B, the second light source group 101-12 PD2 G's second light sources 101-12PD2 The first linearly polarized light L12 emitted from PD2 is incident on the substrate 150 through the substrate 150 and the prism 160, propagates through the substrate 150 while being totally reflected, and is sequentially irradiated onto the corresponding first and second optical elements 215-1 and 215-2. PD2 When the second linearly polarized light L12 is irradiated, PD2 Virtual Light Source 101-1V PD2 , 101-2V PD2 is generated at a position relatively close to the substrate 150.

[0139] The operation of the eyeball information detecting device 10-13 (the eyeball information detecting method using the eyeball information detecting device 10-13) will be described below with reference to the flowchart of FIG.

[0140] In the first step S41, the illumination system 100-13 irradiates the first linearly polarized light L12 onto the two first optical elements 215-1 and the two second optical elements 215-2. PD1 Specifically, the light source driving unit 110 drives the two first light sources 101-12 PD1 As a result, each of the first light sources 101-12 PD1 The first linearly polarized light L12 emitted from PD1 are sequentially irradiated onto the corresponding first and second optical elements 215-1 and 215-2, and the first and second optical elements 215-1 and 215-2 respectively become virtual light sources 101-1V PD1 , 101-2V PD1 The first linearly polarized light L12 passes through each of the first and second optical elements 202-1 and 202-2. PD1 is irradiated onto the eyeball EB and reflected by the eyeball EB.

[0141] In the next step S42, the camera 300 captures an image. Specifically, the image sensor 300a captures the first linearly polarized light L12 through the two first optical elements 215-1 and the two second optical elements 215-2. PD1 The reflected light (corneal reflected light) from the eyeball EB is captured, and the captured image data, which is the result of the capturing, is output to the corneal reflection image detection unit 400a.

[0142] In the next step S43, the detection system 400-12 detects corneal reflection images (Purkinje images). Specifically, the corneal reflection detection unit 400a detects high-brightness images in the captured image data output from the image sensor 300a in step S42 as corneal reflection images. At this time, multiple (e.g., four) corneal reflection images corresponding to multiple (e.g., four) optical elements 215 in the captured image data are detected spaced apart from each other (see FIG. 4C), allowing them to be separated with high accuracy.

[0143] In the next step S44, the illumination system 100-13 irradiates the two first optical elements 215-1 and the two second optical elements 215-2 with the second linearly polarized light L12. PD2 Specifically, the light source driving unit 110 drives the two second light sources 101-12 PD2 As a result, each of the second light sources 101-12 PD2 The second linearly polarized light L12 emitted from PD2 are sequentially irradiated onto the corresponding first and second optical elements 215-1 and 215-2, and the first and second optical elements 215-1 and 215-2 respectively become virtual light sources 101-1V PD2 , 101-2V PD2 The second linearly polarized light L12 passes through each of the first and second optical elements 215-1 and 215-2. PD2 is irradiated onto the eyeball EB and reflected by the eyeball EB.

[0144] In the next step S45, the camera 300 captures an image. Specifically, the image sensor 300a captures the second linearly polarized light L12 through the two first optical elements 215-1 and the two second optical elements 215-2. PD2 The reflected light (corneal reflected light) from the eyeball EB is captured, and the captured image data, which is the result of the capturing, is output to the corneal reflection image detection unit 400a.

[0145] In the next step S46, the detection system 400-12 detects corneal reflection images (Purkinje images). Specifically, the corneal reflection detection unit 400a detects high-brightness images in the captured image data output from the image sensor 300a in step S45 as corneal reflection images. At this time, multiple (e.g., four) corneal reflection images corresponding to multiple (e.g., four) optical elements 215 in the captured image data are detected spaced apart from each other (see FIG. 4C), allowing for accurate separation.

[0146] In the next step S47, the detection system 400-12 corrects the positional shift of the corneal reflection image due to the depth of the eyeball EB. Specifically, the correction unit 400e corrects the first linearly polarized light L12 detected in step S43. PD1 and the second linearly polarized light L12 detected in step S46. PD2 The positional deviation of the corneal reflection image from the reference position due to the depth of the eyeball EB is calculated from the multiple (for example, four) corneal reflection images, the position of the corneal reflection image corrected for the positional deviation is calculated, and the calculation result is output to the gaze estimation unit 400b.

[0147] In the final step S48, detection system 400-12 estimates the gaze. Specifically, gaze estimation unit 400b performs a predetermined calculation based on the position of the corrected corneal reflection image from correction unit 400e, calculates the gaze, which is the direction of eyeball EB, and outputs the calculation result.

[0148] In the eyeball information detecting device 10-13 of the thirteenth embodiment, the first and second light sources having orthogonal polarization directions are switched. For example, the invisible light from the first light source is converted into a first linearly polarized light by a first polarizer, and the invisible light from the second light source is converted into a second linearly polarized light (the polarization direction of which is the first Alternatively, the light may be converted into a polarized light (orthogonal to the linearly polarized light).

[0149] According to the eyeball information detecting device 10-13 of the thirteenth embodiment, the same effects as those of the eyeball information detecting device 10-12 of the twelfth embodiment can be achieved.

[0150] <17. Eyeball Information Detection Device of Example 14 of an Embodiment of the Present Technology> An eyeball information detecting device according to a fourteenth example of an embodiment of the present technology will be described below with reference to Figs. 33A to 35. Fig. 33A is a diagram (part 1) of the eyeball information detecting device 10-14 according to the fourteenth example seen from above. Fig. 33B is a diagram (part 2) of the eyeball information detecting device 10-14 according to the fourteenth example seen from above. Fig. 34 is a block diagram showing the functions of the eyeball information detecting device 10-14 according to the fourteenth example. Fig. 35 is a flowchart for explaining the operation of the eyeball information detecting device 10-14 according to the fourteenth example.

[0151] As shown in FIGS. 33A to 34, the eyeball information detecting device 10-14 of the fourteenth embodiment has a plurality of optical elements 206 (for example, two first optical elements 206-1 and two second optical elements 206-2) which are diffractive elements having wavelength selectivity (diffractive elements whose diffractive power varies depending on the emission wavelength), and a first light source group 101-2V λ1 G is the first light source 101-12 corresponding to each pair of the first and second optical elements 206-1 and 206-2. λ1 and the second light source group 101-2V λ2 G is the second light source 101-12 corresponding to each pair of the first and second optical elements 206-1 and 206-2. λ2 and the first and second light sources 101-12 λ1 , 101-12 λ2 The configuration is generally similar to that of the eyeball information detecting device 10-5 of the fifth embodiment, except that the emission wavelength is different.

[0152] The illumination system 100-14 of the eyeball information detection device 10-14 includes first and second light source groups 101-12. λ1 G, 101-12 λ2 The light source driving unit 110 selectively drives the first light source group 101-12. λ1 G and the second light source group 101-12 λ2 G (see FIG. 34). 1st light source group 101-12 λ1 G is a plurality (for example, two) of first light sources 101-12 having an emission wavelength λ1 (emitting invisible light having a first wavelength λ1). λ1(See FIGS. 33A and 33B.) λ1 are arranged one above the other. 2nd light source group 101-12 λ2 G is a plurality (for example, two) of second light sources 101-12 having an emission wavelength λ2 (emitting invisible light having a second wavelength λ2). λ2 (See FIGS. 33A and 33B.) λ2 are arranged one above the other.

[0153] The two first optical elements 206-1 are arranged one above the other, and the two second optical elements 206-2 are arranged one above the other. The upper first and second optical elements 206-1 and 206-2 are connected to the upper first and second light sources 101-12. λ1 , 101-12 λ2 Corresponds to. The lower first and second optical elements 206-1 and 206-2 are connected to the lower first and second light sources 101-12. λ1 , 101-12 λ2 Corresponds to. As an example, each optical element 206 is a reflective diffraction element provided on the surface of the substrate 150 opposite the eyeball EB side, and its diffraction power for diffracting light of wavelength λ1 is smaller than its diffraction power for diffracting light of wavelength λ2. Each optical element 206 may be a transmissive diffractive element provided on the surface of the substrate 150 on the side of the eyeball EB.

[0154] As shown in FIG. 33A, the first light source group 101-12 λ1 Each first light source 101-12 in G λ1 Invisible light L12 emitted from λ1 is incident on the substrate 150 through the substrate 150 and the prism 160, propagates through the substrate 150 while being totally reflected, and is sequentially irradiated onto the corresponding first and second optical elements 206-1 and 206-2. λ1 When the invisible light L12 is irradiated, λ1 Virtual Light Source 101-1V λ1 , 101-2Vλ1 is generated at a position relatively far from the substrate 150.

[0155] As shown in FIG. 33B, the second light source group 101-12 λ2 G's second light sources 101-12 λ2 Invisible light L12 emitted from λ2 is incident on the substrate 150 through the substrate 150 and the prism 160, propagates through the substrate 150 while being totally reflected, and is sequentially irradiated onto the corresponding first and second optical elements 206-1 and 206-2. λ2 When the invisible light L12 is irradiated, λ2 Virtual Light Source 101-1V λ2 , 101-2V λ2 is generated at a position relatively close to the substrate 150.

[0156] The operation of the eyeball information detecting device 10-14 (the eyeball information detecting method using the eyeball information detecting device 10-14) will be described below with reference to the flowchart of FIG.

[0157] In the first step S51, the illumination system 100-14 irradiates the two first optical elements 206-1 and the two second optical elements 206-2 with the invisible light L12 having the first wavelength λ1. λ1 Specifically, the light source driving unit 110 drives the two first light sources 101-12 λ1 As a result, each of the first light sources 101-12 λ1 Invisible light L12 emitted from λ1 are sequentially irradiated onto the corresponding first and second optical elements 206-1 and 206-2, and the first and second optical elements 206-1 and 206-2 respectively become virtual light sources 101-1V λ1 , 101-2V λ1 The invisible light L12 is emitted through the first and second optical elements 206-1 and 206-6. λ1 is irradiated onto the eyeball EB and reflected by the eyeball EB.

[0158] In the next step S52, the camera 300 captures an image. Specifically, the image sensor 300a captures invisible light L12 through the two first optical elements 206-1 and the two second optical elements 206-2. λ1 The reflected light (corneal reflected light) from the eyeball EB is captured, and the captured image data, which is the result of the capturing, is output to the corneal reflection image detection unit 400a.

[0159] In the next step S53, the detection system 400-12 detects corneal reflection images (Purkinje images). Specifically, the corneal reflection detection unit 400a detects high-brightness images in the captured image data output from the image sensor 300a in step S52 as corneal reflection images. At this time, multiple (e.g., four) corneal reflection images corresponding to multiple (e.g., four) optical elements 206 in the captured image data are detected spaced apart from each other (see FIG. 4C), allowing for accurate separation.

[0160] In the next step S54, the illumination system 100-14 irradiates the two first optical elements 206-1 and the two second optical elements 206-2 with the invisible light L12 having the second wavelength λ2. λ2 Specifically, the light source driving unit 110 drives the two second light sources 101-12 λ2 As a result, each of the second light sources 101-12 λ2 Invisible light L12 emitted from λ2 are sequentially irradiated onto the corresponding first and second optical elements 206-1 and 206-2, and the first and second optical elements 206-1 and 206-2 respectively become virtual light sources 101-1V λ2 , 101-2V λ2 The invisible light L12 passes through each of the first and second optical elements 206-1 and 206-2. λ2 is irradiated onto the eyeball EB and reflected by the eyeball EB.

[0161] In the next step S55, the camera 300 captures an image. Specifically, the image sensor 300a captures invisible light L12 through the two first optical elements 206-1 and the two second optical elements 206-2. λ2The reflected light (corneal reflected light) from the eyeball EB is captured, and the captured image data, which is the result of the capturing, is output to the corneal reflection image detection unit 400a.

[0162] In the next step S56, the detection system 400-12 detects corneal reflection images (Purkinje images). Specifically, the corneal reflection detection unit 400a detects high-brightness images in the captured image data output from the image sensor 300a in step S55 as corneal reflection images. At this time, multiple (e.g., four) corneal reflection images corresponding to multiple (e.g., four) optical elements 206 in the captured image data are detected spaced apart from each other (see FIG. 4C), allowing for accurate separation.

[0163] In the next step S57, the detection system 400-12 corrects the positional deviation of the corneal reflection image due to the depth of the eyeball EB. Specifically, the correction unit 400e corrects the non-visible light L12 detected in step S53. λ1 and the non-visible light L12 detected in step S56. λ2 The positional deviation of the corneal reflection image from the reference position due to the depth of the eyeball EB is calculated from the multiple (for example, four) corneal reflection images, the position of the corneal reflection image corrected for the positional deviation is calculated, and the calculation result is output to the gaze estimation unit 400b.

[0164] In the final step S58, detection system 400-12 estimates the gaze. Specifically, gaze estimation unit 400b performs a predetermined calculation based on the position of the corrected corneal reflection image from correction unit 400e, calculates the gaze, which is the direction of eyeball EB, and outputs the calculation result.

[0165] According to the eyeball information detecting device 10-14 of the fourteenth embodiment, the same effects as those of the eyeball information detecting device 10-12 of the twelfth embodiment can be achieved.

[0166] <18. Eyeball Information Detection Device of Example 15 of an Embodiment of the Present Technology> An eyeball information detecting device according to a fifteenth example of an embodiment of the present technology will be described below with reference to Figs. 36A to 38. Fig. 36A is a diagram (part 1) of the eyeball information detecting device 10-15 according to the fifteenth example seen from above. Fig. 36B is a diagram (part 2) of the eyeball information detecting device 10-15 according to the fifteenth example seen from above. Fig. 37 is a block diagram showing the functions of the eyeball information detecting device 10-15 according to the fifteenth example. Fig. 38 is a flowchart for explaining the operation of the eyeball information detecting device 10-15 according to the fifteenth example.

[0167] As shown in Figures 36A to 37, the eyeball information detection device 10-15 of Example 15 has a configuration generally similar to that of the eyeball information detection device 10-5 of Example 5, except that the multiple optical elements 207 (for example, two first optical elements 207-1 and two second optical elements 207-2) are diffraction elements having incident angle dependency (diffraction elements whose diffraction power varies depending on the incident angle), and the irradiation system 100-15 can change the incident angle of non-visible light to each diffraction element.

[0168] The illumination system 100-15 of the eyeball information detecting device 10-15 includes a light source driving unit 110 that drives the light source group 101-12G, and an actuator 120 that moves the light source group 101-12G (see FIG. 37). The light source group 101-12G includes a plurality of (for example, two) light sources 101-12 (see FIGS. 36A and 36B). The plurality of (for example, two) light sources 101-12 are arranged one above the other. The actuator 120 moves a plurality of (for example, two) light sources 101-12 of the light source group 101-12G in an integrated manner to the left and right.

[0169] The two first optical elements 207-1 are arranged one above the other, and the two second optical elements 207-2 are arranged one above the other. The upper first and second optical elements 207-1 and 207-2 correspond to the upper light source 101-12. The lower first and second optical elements 207-1 and 206-2 correspond to the lower light source 101-12. As an example, each optical element 207 is a reflective diffraction element provided on the surface of the substrate 150 opposite the eyeball EB side, and the diffraction power for diffracting light incident at a first incident angle θ1 is smaller than the diffraction power for diffracting light incident at a second incident angle θ2. Each optical element 207 may be a transmissive diffractive element provided on the surface of the substrate 150 on the side of the eyeball EB.

[0170] As shown in FIG. 36A, invisible light L12 emitted from each light source 101-12 of the light source group 101-12G located on the left side θ1 is incident on the substrate 150 through the substrate 150 and the prism 160 at a first incident angle θ1 (an incident angle equal to or greater than the critical angle), propagates through the substrate 150 while being totally reflected, and is incident on the corresponding first and second optical elements 207-1 and 207-2 at the first incident angle θ1. The first and second optical elements 207-1 and 207-2 convert the invisible light L12 θ1 When the invisible light L12 is irradiated, θ1 Virtual Light Source 101-1V θ1 , 101-2V θ1 is generated at a position relatively far from the substrate 150.

[0171] As shown in FIG. 36B, invisible light L12 emitted from each light source 101-12 of the light source group 101-12G located on the right side θ2 is incident on the substrate 150 through the substrate 150 and the prism 160 at a second incident angle θ2 (an incident angle equal to or greater than the critical angle), propagates through the substrate 150 while being totally reflected, and is incident on the corresponding first and second optical elements 207-1 and 207-2 at the second incident angle θ2. The first and second optical elements 207-1 and 207-2 convert the invisible light L12 θ2 When the invisible light L12 is irradiated, θ2 Virtual Light Source 101-1V θ2 , 101-2V θ2 is generated at a position relatively close to the substrate 150.

[0172] The operation of the eyeball information detecting device 10-15 (the eyeball information detecting method using the eyeball information detecting device 10-15) will be described below with reference to the flowchart of FIG.

[0173] In the first step S61, the illumination system 100-15 irradiates the invisible light L12 onto the two first optical elements 207-1 and the two second optical elements 207-2 at a first incident angle θ1. θ1 Specifically, the light source driving unit 110 drives the two light sources 101-12 located on the left side by the actuator 120. As a result, the invisible light L12 emitted from each light source 101-12 is irradiated onto the corresponding first and second optical elements 207-1 and 207-2 at the first incident angle θ1, and the first and second optical elements 207-1 and 207-2 respectively become virtual light sources 101-1V. θ1 , 101-2V θ1 (See FIG. 36A). The invisible light L12 is generated through each of the first and second optical elements 207-1 and 207-2. θ1 is irradiated onto the eyeball EB and reflected by the eyeball EB.

[0174] In the next step S62, the camera 300 captures an image. Specifically, the image sensor 300a captures invisible light L12 through the two first optical elements 207-1 and the two second optical elements 207-2. θ1 The reflected light (corneal reflected light) from the eyeball EB is captured, and the captured image data, which is the result of the capturing, is output to the corneal reflection image detection unit 400a.

[0175] In the next step S63, the detection system 400-12 detects corneal reflection images (Purkinje images). Specifically, the corneal reflection detection unit 400a detects high-brightness images in the captured image data output from the image sensor 300a in step S62 as corneal reflection images. At this time, multiple (e.g., four) corneal reflection images corresponding to multiple (e.g., four) optical elements 207 in the captured image data are detected spaced apart from each other (see FIG. 4C ), allowing for accurate separation.

[0176] In the next step S64, the illumination system 100-15 irradiates the invisible light L12 onto the two first optical elements 207-1 and the two second optical elements 207-2 at a second incident angle θ2. θ2 Specifically, the light source driving unit 110 drives the two second light sources 101-12 located on the right side by the actuator 120. As a result, the invisible light L12 emitted from each light source 101-12 is θ2 are sequentially irradiated onto the corresponding first and second optical elements 207-1 and 207-2, and the first and second optical elements 207-1 and 207-2 respectively become virtual light sources 101-1V θ2 , 101-2V θ2 (See FIG. 36B). θ2 is irradiated onto the eyeball EB and reflected by the eyeball EB.

[0177] In the next step S65, the camera 300 captures an image. Specifically, the image sensor 300a captures invisible light L12 through the two first optical elements 207-1 and the two second optical elements 207-2. θ2 The reflected light (corneal reflected light) from the eyeball EB is captured, and the captured image data, which is the result of the capturing, is output to the corneal reflection image detection unit 400a.

[0178] In the next step S66, the detection system 400-12 detects corneal reflection images (Purkinje images). Specifically, the corneal reflection detection unit 400a detects high-brightness images in the captured image data output from the image sensor 300a in step S65 as corneal reflection images. At this time, multiple (e.g., four) corneal reflection images corresponding to multiple (e.g., four) optical elements 207 in the captured image data are detected spaced apart from each other (see FIG. 4C), allowing for accurate separation.

[0179] In the next step S67, the detection system 400-12 corrects the positional deviation of the corneal reflection image due to the depth of the eyeball EB. Specifically, the correction unit 400e corrects the non-visible light L12 detected in step S63. θ1 and the non-visible light L12 detected in step S66.θ2 The positional deviation of the corneal reflection image from the reference position due to the depth of the eyeball EB is calculated from the multiple (for example, four) corneal reflection images, the position of the corneal reflection image corrected for the positional deviation is calculated, and the calculation result is output to the gaze estimation unit 400b.

[0180] In the final step S68, detection system 400-12 estimates the gaze. Specifically, gaze estimation unit 400b performs a predetermined calculation based on the position of the corrected corneal reflection image from correction unit 400e, calculates the gaze, which is the direction of eyeball EB, and outputs the calculation result.

[0181] According to the eyeball information detecting device 10-15 of the fifteenth embodiment, the same effects as those of the eyeball information detecting device 10-12 of the twelfth embodiment can be achieved.

[0182] 19. Display device including eyeball information detection device according to Example 7 of an embodiment of the present technology Hereinafter, a display device 1 including an eyeball information detecting device 10-7 according to Example 7 of an embodiment of the present technology will be described with reference to Figs. 39A to 41. Fig. 39A is a diagram of the display device 1 as seen from above. Fig. 39B is a diagram of the display device 1 as seen from the eyeball EB side. Fig. 40 is a block diagram showing functions of the display device 1. Fig. 41 is a flowchart for explaining the operation of the display device 1.

[0183] The display device 1 can be applied to both AR display and VR display.

[0184] As shown in Figures 39A to 40, the display device 1 includes, in addition to the eyeball information detection device 10-7, an image light generation device 500 and an image display optical element 430 (optical system) that is provided on the substrate 150 and guides the image light IL from the image light generation device 500 to the eyeball EB, and the image light generation device 500 generates the image light IL based on the detection results of the eyeball information detection device 10-7.

[0185] The image light generating device 500 is provided, for example, on a temple of an eyeglass frame serving as the support member 50.

[0186] The image light generating device 500 includes an LD 510 (edge-emitting laser) that emits visible light (laser light), an MEMS mirror 520 that serves as a deflector that deflects the visible light from the LD 510, and a control unit 501 that controls the LD 510 and the MEMS mirror 520. It should be noted that, for example, a VCSEL (vertical cavity surface emitting laser) may be used instead of the LD. The control unit 501 generates the image light IL by controlling the LD 510 and the MEMS mirror 520 based on the image data. The control unit 501 is realized by hardware including, for example, a CPU and a chipset.

[0187] The image display optical element 430 is, for example, a reflective diffraction element.

[0188] In the image light generation device 500, image light IL generated by deflecting visible light from the LD 510 by the MEMS mirror 520 is irradiated onto the image display optical element 430. The image light IL irradiated onto the image display optical element 430 is reflected and diffracted by the image display optical element 430 toward the eyeball EB, and is irradiated onto the eyeball EB. This allows the user to view the image.

[0189] The operation of the display device 1 (a display method using the display device 1) will be described below with reference to the flowchart of FIG. In the first step S71, the illumination system 100-1 illuminates invisible light onto a plurality of (for example, four) optical elements 201. Specifically, the light source driving unit 110 drives (lights up) the first to fourth light sources 101-1 to 101-4, which correspond to the first to fourth optical elements 201-1 to 201-4, respectively. As a result, the invisible light emitted from each light source 101 is illuminated onto the corresponding optical element 202, and the optical element 202 generates a virtual light source. The invisible light that passes through the optical element 202 is illuminated onto the eyeball EB and reflected by the eyeball EB.

[0190] In the next step S72, the camera 300 captures an image. Specifically, the image sensor 300a captures an image of the reflected light (corneal reflected light) of invisible light from the eye EB via each optical element 201, and outputs captured image data, which is the imaging result, to the corneal reflection image detection unit 400a.

[0191] In the next step S73, the detection system 400-1 detects a corneal reflection image (Purkinje image). Specifically, the corneal reflection image detection unit 400a detects a high-brightness image in the captured image data from the image sensor 300a as a corneal reflection image. At this time, the multiple corneal reflection images corresponding to the multiple optical elements 202 in the captured image data are detected spaced apart from each other (see FIG. 4C), allowing them to be separated with high accuracy.

[0192] In the next step S74, the detection system 400-1 estimates the gaze. Specifically, the gaze estimation unit 400b calculates the gaze, which is the direction of the eyeball EB, from the position of the corneal reflection detected by the corneal reflection detection unit 400a by a predetermined calculation, and outputs the calculation result.

[0193] In the final step S75, the image light generation device 500 generates image light based on the line of sight direction. Specifically, the control unit 501 controls the LD 510 and the MEMS mirror 520 so that an image is displayed in a field of view centered on the line of sight direction from the detection system 400-1.

[0194] According to the display device 1, image light is generated based on the line of sight accurately detected by the eyeball information detection device 10-7, and therefore an image with good visibility can be displayed regardless of the direction of the user's line of sight.

[0195] A display method using the display device 1 includes the steps of irradiating invisible light onto a plurality of optical elements 202 provided on a substrate 150 facing the user's eyeball EB to generate a virtual light source of the invisible light, receiving the invisible light reflected by the eyeball EB through each of the plurality of optical elements 202, detecting eye information (e.g., the orientation of the eyeball EB) that is information about the eyeball EB based on the light reception results from the light reception step, and generating image light IL based on the detection results from the detection step and guiding the image light IL to the eyeball EB.

[0196] According to this display method, an image with good visibility can be displayed to the user at all times.

[0197] 20. Modifications of this technology The configurations of the eyeball information detecting device and the display device including the eyeball information detecting device according to the embodiments of the present technology described above can be modified as appropriate.

[0198] For example, the eyeball information detection device in each of the above embodiments has four or three optical elements that generate virtual light sources, but it may also have only one or two, or five or more.

[0199] For example, in the display device 1 including the eyeball information detecting device 10-7 of the seventh embodiment, the image light generating device 500 is of a scanning type, but it may be of a non-scanning type including a display.

[0200] For example, in the eyeball information detection device and display device of each of the above embodiments, instead of an image sensor, a multi-segment PD (photodiode) having multiple light receiving areas may be used as the light receiving element, or a PD (photodiode) having a single light receiving area may be used.

[0201] The operation flow of the eyeball information detecting device (the flow of the eyeball information detecting method) and the operation flow of the display device (the flow of the display method) may be repeated multiple times for each frame.

[0202] In the eyeball information detecting device and display device of each of the above-mentioned embodiments, the type, number, shape, arrangement, etc. of the light source and optical elements can be changed as appropriate.

[0203] At least a part of the configuration of the eyeball information detecting device of each of the above embodiments may be combined with each other within a range that does not contradict.

[0204] The present technology can also be configured as follows. (1) a support member attached to the user's head; a substrate provided on the support member so as to face the eyeball of the user; an illumination system provided on the support member and including at least one light source that emits invisible light toward the substrate; at least one optical element, other than a plane mirror, that is provided on the substrate so as to be irradiated with the invisible light and that generates a virtual light source of the invisible light; a light receiving system including a light receiving element that receives the invisible light reflected by the eyeball via the optical element; a detection system that detects eyeball information based on an output of the light receiving system; An eyeball information detection device comprising: (2) The eyeball information detection device according to (1), wherein the at least one optical element is a plurality of optical elements. (3) The eyeball information detection device according to (2), wherein the plurality of optical elements include at least three optical elements arranged two-dimensionally along the in-plane direction of the substrate. (4) The eyeball information detection device according to (2) or (3), wherein the at least one light source is a plurality of light sources. (5) The eyeball information detection device according to (4), wherein the plurality of light sources include at least two light sources corresponding to different optical elements among the plurality of optical elements. (6) The eyeball information detection device according to (4) or (5), wherein the plurality of light sources correspond to the plurality of optical elements individually. (7) The eyeball information detecting device according to any one of (2) to (6), wherein the at least one light source includes a light source corresponding to at least two optical elements among the plurality of optical elements. (8) The eyeball information detection device according to (2) or (3), wherein the at least one light source is a single light source corresponding to the plurality of optical elements. (9) The eyeball information detecting device according to any one of (1) to (8), wherein the number of the at least one optical element is greater than the number of the at least one light source. (10) An eyeball information detection device described in any one of (4) to (9), wherein each of the plurality of light sources corresponds to at least two of the plurality of optical elements, and the invisible light from each of the plurality of light sources propagates within the substrate and is irradiated onto the corresponding at least two optical elements. (11) The eyeball information detecting device according to any one of (2) to (10), wherein an optical member that guides the invisible light from the light source to the plurality of optical elements is provided on the substrate. (12) The eyeball information detection device according to (11), wherein the invisible light from the light source propagates within the substrate and is irradiated onto the plurality of optical elements via the optical member. (13) The eyeball information detecting device according to any one of (1) to (12), wherein the light receiving system is provided on the substrate. (14) An eyeball information detection device described in any one of (1) to (13), wherein the support member has a temple including an ear hook portion, the light receiving system is provided on the temple, and another optical element that guides the invisible light reflected by the eyeball to the light receiving system is provided on the substrate. (15) The eyeball information detection device according to (14), wherein the plurality of optical elements are provided on the substrate around the other optical element. (16) The eyeball information detecting device according to any one of (1) to (15), wherein the support member has a temple including an ear hook portion, and the light source is provided on the temple. (17) An eyeball information detection device described in any one of (1) to (15), wherein the support member has a temple including an ear hook portion and an extension portion extending to the opposite side of the base from the ear hook portion, and the light source is provided in the extension portion. (18) The eyeball information detecting device according to any one of (1) to (17), wherein the optical element is of a reflective type. (19) The eyeball information detecting device according to any one of (1) to (18), wherein the optical element is a diffraction element. (20) The eyeball information detection device according to (19), wherein the diffraction element has a non-uniform pitch. (21) The eyeball information detection device according to (19) or (20), wherein the diffraction element is a wavefront reproduction type. (22) The eyeball information detection device described in (21), wherein the at least one optical element is a plurality of diffraction elements, the wavefront shapes recorded on at least two of the plurality of diffraction elements are different, and the detection system has a fitting unit that fits a plurality of reflected images of the invisible light at the eyeball to the wavefront shapes recorded on the plurality of diffraction elements. (23) The eyeball information detection device according to (19) or (20), wherein the diffraction element is a diffusion element. (24) The eyeball information detection device described in (23), wherein the at least one optical element is a plurality of diffusion elements, and the detection system has a fitting unit that fits a plurality of reflection images of the invisible light at the eyeball to the plurality of diffusion elements. (25) The eyeball information detection device according to (24), wherein at least two of the plurality of diffusion elements have different shapes. (26) The eyeball information detection device according to (19) or (20), wherein the diffraction element has curved mirror characteristics. (27) An eyeball information detection device described in any one of (19) to (26), wherein the at least one diffraction element is a plurality of diffraction elements having different diffraction powers, at least two of the plurality of virtual light sources generated by the plurality of diffraction elements are positioned at different positions in the thickness direction of the substrate, and the detection system has a depth estimation unit that estimates the depth of the eyeball from the output of the light receiving system. (28) The eyeball information detection device according to any one of (2) to (27), wherein the irradiation system is capable of irradiating the invisible light onto at least two optical elements among the plurality of optical elements at different timings. (29) An eyeball information detection device described in any one of (2) to (28), wherein the plurality of optical elements include first and second optical elements corresponding to different light sources, and the illumination system has a light source driving unit that selectively drives the light source corresponding to the first optical element and the light source corresponding to the second optical element. (30) An eyeball information detection device described in any one of (4) to (29), wherein the optical element is a diffraction element having wavelength selectivity, the plurality of light sources include at least two light sources having different emission wavelengths and corresponding to the diffraction element, and the illumination system has a light source driving unit that selectively drives the at least two light sources. (31) An eyeball information detection device described in any one of (1) to (30), wherein the optical element is a diffraction element having polarization dependency, and the illumination system is capable of varying the polarization direction of the invisible light. (32) An eyeball information detection device described in (1) to (30), wherein the optical element is a diffraction element having incidence angle dependency, and the illumination system is capable of varying the incidence angle of the non-visible light to the diffraction element. (33) The eyeball information detecting device according to any one of (1) to (32), wherein the optical element is a curved mirror. (34) The eyeball information detection device according to any one of (1) to (33), wherein the eyeball information includes at least one of the orientation of the eyeball, the position and size of the pupil, and the position of the iris. (35) An eyeball information detection device according to any one of (1) to (34), an image light generating device; an optical system provided on the substrate and guiding image light from the image light generating device to the eyeball; Equipped with The image light generating device generates the image light based on the detection result of the eyeball information detecting device. (36) A step of irradiating invisible light onto at least one optical element provided on a substrate facing the user's eyeball to generate a virtual light source of the invisible light; receiving the invisible light reflected by the eyeball via the optical element; detecting eyeball information based on a light reception result in the light receiving step; An eyeball information detection method, comprising: (37) A step of irradiating invisible light onto at least one optical element provided on a substrate facing the user's eyeball to generate a virtual light source of the invisible light; receiving the invisible light reflected by the eyeball via the optical element; detecting eyeball information based on a light reception result in the light receiving step; generating image light based on the detection result in the detecting step and guiding the image light to the eyeball; including, how it is displayed. [Explanation of symbols]

[0205] 1: display device, 10-1 to 10-15: eyeball information detection device, 50: support member, 100-1, 100-4, 100-5, 100-6, 100-8, 100-12, 100-13, 100-14, 100-15: illumination system, 101-1V, 101-2V, 101-1V PD1 , 101-2V PD2 , 101-1V λ1 , 101-2V λ2 , 101-1V θ1 , 101-2V θ2 : Virtual light source, 150: Substrate, 101-1 to 101-4, 101-13, 101-24, 101-12, 101-34, 101-1234, 101-123: Light source, 201-1 to 201-4, 202-1 to 202-4, 202a to 202d, 203-1 to 203-3, 204-1 to 204-4, 205-1 to 205-3, 215-1, 215-2, 20 6-1, 206-2, 207-1, 207-2: optical elements, 250: optical member, 270: another optical element, 300: camera (light receiving system), 400-1, 400-8, 400-11, 400-12: detection system, 430: optical element for image display (optical system), 500: image light generating device, EB: eyeball, L1 to L4, L13, L24, L12, L34, L1234, L12 PD1 , L12 PD2 , L12 λ1 , L12λ2 L12 θ1 L12 θ2 :Non-visible light, IL:image light.

Claims

1. a support member to be worn on the user's head; a substrate provided on the support member so as to face the eyeball of the user; an illumination system provided on the support member and including at least one light source that emits invisible light toward the substrate; at least one optical element, other than a plane mirror, provided on the substrate so as to be irradiated with the invisible light and which generates a virtual light source of the invisible light; a light receiving system including a light receiving element that receives the invisible light reflected by the eyeball via the optical element; a detection system that detects eyeball information based on an output of the light receiving system; Equipped with the at least one optical element is a plurality of optical elements; The support member has a temple including an ear hook portion, the light receiving system is provided in the temple, An eyeball information detection device, wherein another optical element is provided on the substrate to guide the invisible light reflected by the eyeball to the light receiving system.

2. The eyeball information detecting device according to claim 1 , wherein the plurality of optical elements are provided on the substrate around the other optical element.

3. The eyeball information detecting device according to claim 1 , wherein the plurality of optical elements include at least three optical elements arranged two-dimensionally along an in-plane direction of the substrate.

4. The eyeball information detecting device according to claim 1 , wherein the at least one light source is a plurality of light sources.

5. The eyeball information detecting device according to claim 4 , wherein the plurality of light sources include at least two light sources corresponding to different optical elements among the plurality of optical elements.

6. The eyeball information detecting device according to claim 4 , wherein the plurality of light sources correspond to the plurality of optical elements, respectively.

7. The eyeball information detection device according to claim 1 , wherein the at least one light source includes light sources corresponding to at least two optical elements among the plurality of optical elements.

8. The eyeball information detection device according to claim 1 , wherein the at least one light source is a single light source corresponding to the plurality of optical elements.

9. The eyeball information detecting device according to claim 1 , wherein the number of the at least one optical element is greater than the number of the at least one light source.

10. each of the plurality of light sources corresponds to at least two optical elements of the plurality of optical elements; The eyeball information detecting device according to claim 4 , wherein the invisible light from each of the plurality of light sources propagates through the substrate and is irradiated onto the corresponding one of the at least two optical elements.

11. The eyeball information detecting device according to claim 8 , wherein an optical member that guides the invisible light from the light source to the plurality of optical elements is provided on the substrate.

12. The eyeball information detecting device according to claim 11 , wherein the invisible light from the light source propagates within the substrate and is irradiated onto the plurality of optical elements via the optical member.

13. The eyeball information detecting device according to claim 1 , wherein the light receiving system is provided on the substrate.

14. The support member has a temple including an ear hook portion, The eyeball information detecting device according to claim 1 , wherein the light source is provided in the temple.

15. the support member has a temple including an ear hook portion and an extension portion extending to a side of the base plate opposite to the ear hook portion side, The eyeball information detecting device according to claim 1 , wherein the light source is provided on the extension.

16. The eyeball information detecting device according to claim 1 , wherein the optical element is a reflective type.

17. The eyeball information detecting device according to claim 1 , wherein the optical element is a diffractive element.

18. a support member to be worn on the user's head; a substrate provided on the support member so as to face the eyeball of the user; an illumination system provided on the support member and including at least one light source that emits invisible light toward the substrate; at least one optical element, other than a plane mirror, provided on the substrate so as to be irradiated with the invisible light and which generates a virtual light source of the invisible light; a light receiving system including a light receiving element that receives the invisible light reflected by the eyeball via the optical element; a detection system that detects eyeball information based on an output of the light receiving system; Equipped with the optical element is a diffractive element, The diffractive element has a non-uniform pitch.

19. The eyeball information detecting device according to claim 18, wherein the diffraction element is of a wavefront reproducing type.

20. the at least one optical element is a plurality of diffractive elements; wavefront shapes recorded on at least two of the plurality of diffraction elements are different; 20. The eyeball information detection device according to claim 19, wherein the detection system has a fitting unit that fits a plurality of reflected images of the invisible light from the eyeball to wavefront shapes recorded on the plurality of diffraction elements.

21. The eyeball information detecting device according to claim 18 , wherein the diffraction element is a diffusion element.

22. the at least one optical element is a plurality of diffusing elements; The eyeball information detection device according to claim 21 , wherein the detection system has a fitting unit that fits a plurality of reflection images of the invisible light on the eyeball to the plurality of diffusion elements.

23. The eyeball information detection device according to claim 22 , wherein at least two of the plurality of diffusion elements have different shapes.

24. The eyeball information detection device according to claim 18 , wherein the diffraction element has a curved mirror characteristic.

25. A plurality of the diffraction elements having different diffraction powers are provided, At least two of the virtual light sources generated by the diffraction elements are located at different positions in a thickness direction of the substrate, 25. The eyeball information detecting device according to claim 24, wherein the detection system includes a depth estimation unit that estimates a depth of the eyeball from an output of the light receiving system.

26. The eyeball information detecting device according to claim 1 , wherein the illumination system is capable of irradiating the invisible light onto at least two optical elements of the plurality of optical elements at different timings.

27. the plurality of optical elements include first and second optical elements corresponding to different light sources; The eyeball information detecting device according to claim 4 , wherein the illumination system includes a light source driving unit that selectively drives the light source corresponding to the first optical element and the light source corresponding to the second optical element.

28. a support member to be worn on the user's head; a substrate provided on the support member so as to face the eyeball of the user; an illumination system provided on the support member and including at least one light source that emits invisible light toward the substrate; at least one optical element, other than a plane mirror, provided on the substrate so as to be irradiated with the invisible light and which generates a virtual light source of the invisible light; a light receiving system including a light receiving element that receives the invisible light reflected by the eyeball via the optical element; a detection system that detects eyeball information based on an output of the light receiving system; Equipped with the at least one optical element is a plurality of optical elements; the at least one light source is a plurality of light sources; the optical element is a diffractive element having wavelength selectivity, the plurality of light sources include at least two light sources having different emission wavelengths and corresponding to the diffraction element; The illumination system includes a light source driving unit that selectively drives the at least two light sources.

29. the optical element is a diffraction element having polarization dependency, The eyeball information detecting device according to claim 1 , wherein the illumination system is capable of changing the polarization direction of the invisible light.

30. a support member to be worn on the user's head; a substrate provided on the support member so as to face the eyeball of the user; an illumination system provided on the support member and including at least one light source that emits invisible light toward the substrate; at least one optical element, other than a plane mirror, provided on the substrate so as to be irradiated with the invisible light and which generates a virtual light source of the invisible light; a light receiving system including a light receiving element that receives the invisible light reflected by the eyeball via the optical element; a detection system that detects eyeball information based on an output of the light receiving system; Equipped with the optical element is a diffraction element having incidence angle dependency, The illumination system is capable of varying the angle of incidence of the invisible light on the diffraction element.

31. The eyeball information detecting device according to claim 1 , wherein the optical element is a curved mirror.

32. The eyeball information detection device according to claim 1 , wherein the eyeball information includes at least one of a direction of the eyeball, a position and size of a pupil, and a position of an iris.

33. The eyeball information detection device according to claim 1; an image light generating device; an optical system provided on the substrate and guiding image light from the image light generating device to the eyeball; Equipped with The image light generating device generates the image light based on the detection result of the eyeball information detecting device.

Images