Image display device and image display method

Abstract

Provided is an image display device wherein tracking precision can be improved while also reducing the size of the image display device.　An image display device 100 is provided with: a light source unit 111 that emits image display light OL for displaying an image and emits conjugated light CL of the image display light OL; an optical system that projects the image display light OL emitted by the light source unit 111 to the pupils of the eyes of the user and emits the conjugated light CL emitted by the light source unit 111 in the vicinity of the pupils of the eyes of the user; a detection unit 103 that detects reflected light, which is the conjugated light projected by the optical system and reflected in the vicinity of the pupil; and a control unit 104 that, on the basis of the reflected light detected by the detection unit 103, controls the position of the image to be displayed.

Description

[Technical field]

[0001] The present technology relates to an image display device and an image display method, and more particularly to an image display device that displays a holographic image and an image display method using the same. [Background technology]

[0002] 2. Description of the Related Art In recent years, image display devices (eyewear) such as head mounted displays (HMDs) have become known as devices that are worn on the body of a user, such as the face, to display images to the user.

[0003] When performing eye tracking with such eyewear, a lighting system different from the image light is generally required, and lighting using infrared light is usually installed. Furthermore, these are often placed around the frame, making the eyewear itself larger.

[0004] Furthermore, in the case of see-through eyewear, the eyewear frame must not obstruct the field of view because it can reduce the sense of immersion when images are superimposed. However, the frame width tends to be inevitably increased in order to accommodate the infrared light illumination, which can reduce the sense of immersion.

[0005] Furthermore, if you want to generate a targeted pattern and project it onto the eyeball, additional optical systems are required, which tends to increase the overall size of the eyewear.

[0006] In view of the above problems, techniques for miniaturizing eyewear have been proposed.

[0007] For example, Patent Document 1 proposes an image projection device comprising: a light source that emits an image light beam and a test light beam to form an image; an optical system that projects the image light beam emitted from the light source onto a first surface area of ​​a user's eye and projects it onto the user's retina, and projects the test light beam emitted from the light source onto a second surface area of ​​the user's eye away from the first surface area; a photodetector that detects reflected light of the test light beam reflected by the user's eye; and a control unit that controls at least one of the light source and the optical system based on the detection result of the reflected light by the photodetector. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2017-009986 A Summary of the Invention [Problem to be solved by the invention]

[0009] However, the technique of Patent Document 1 does not propose a solution for reducing the size of the image display device and improving the tracking accuracy, and further development of an image display device that satisfies both of these requirements at the same time is desired.

[0010] Therefore, a main object of the present technology is to provide an image display device that is capable of improving tracking accuracy while reducing the size of the image display device. [Means for solving the problem]

[0011] The present technology provides an image display device including a light source unit that emits image display light and conjugate light of the image display light to display an image, an optical system that projects the image display light emitted from the light source unit onto the pupil of a user's eye and projects the conjugate light emitted from the light source unit onto the periphery of the pupil of the user's eye, a detection unit that detects reflected light of the conjugate light projected from the optical system that is reflected off the periphery of the pupil, and a control unit that controls the position of an image to be displayed based on the reflected light detected by the detection unit.

[0012] In addition, the present technology provides an image display method including the steps of emitting image display light and conjugate light of the image display light for displaying an image, projecting the emitted image display light and conjugate light onto the pupil of a user's eye and the periphery of the pupil of the user's eye, respectively, detecting reflected light of the projected conjugate light reflected around the pupil, and controlling the position of the image to be displayed based on the detected reflected light. Effect of the Invention

[0013] According to the present technology, it is possible to provide an image display device that can improve tracking accuracy while reducing the size of the image display device. Note that the above effects are not necessarily limiting, and any of the effects shown in this specification or other effects that can be understood from this specification may be achieved together with or instead of the above effects. [Brief description of the drawings]

[0014] [Figure 1] 1 is a plan view of an image display device according to a first embodiment of the present technology, as viewed from above. [Diagram 2] 1 is a schematic diagram of an example of a captured image captured using the image display device according to the first embodiment of the present technology. [Diagram 3] 1 is a plan view of an image display device according to a first embodiment of the present technology, as viewed from above. [Figure 4] 1 is a schematic diagram of an example of a captured image captured using the image display device according to the first embodiment of the present technology. [Diagram 5]3 is a block diagram showing an example configuration of a control unit according to the first embodiment of the present technology. FIG. [Figure 6] 4 is a flowchart showing an example of an image display method according to the first embodiment of the present technology. [Figure 7] 10A to 10D are schematic diagrams illustrating an example of a captured image captured using a modified example of the image display device according to the first embodiment of the present technology. [Figure 8] 11 is a plan view of an image display device according to a second embodiment of the present technology, as viewed from above. FIG. [Figure 9] 11 is a plan view of an image display device according to a third embodiment of the present technology, as viewed from above. FIG. [Figure 10] 13 is a schematic diagram of an example of a captured image captured using an image display device according to a third embodiment of the present technology. FIG. [Figure 11] 13 is a schematic diagram of another example of a captured image captured using the image display device according to the third embodiment of the present technology. FIG. [Figure 12] 13 is a schematic diagram of another example of a captured image captured using the image display device according to the third embodiment of the present technology. FIG. [Figure 13] 13 is a schematic diagram of another example of a captured image captured using the image display device according to the third embodiment of the present technology. FIG. [Figure 14] 13 is a plan view of an image display device according to a fourth embodiment of the present technology, as viewed from above. FIG. [Figure 15] 13 is a plan view of an image display device according to a fifth embodiment of the present technology, as viewed from above. FIG. [Figure 16] 13 is a schematic diagram illustrating an example of a captured image captured using an image display device according to a sixth embodiment of the present technology. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, preferred embodiments for carrying out the present technology will be described with reference to the drawings. The embodiments described below are examples of typical embodiments of the present technology, and any of the embodiments can be combined. In addition, the scope of the present technology is not narrowly interpreted by these. The description will be given in the following order. 1. First embodiment (1) Example of the configuration of an image display device (2) Example of control unit configuration (3) Example of image display method (tracking method) (4) Variations 2. Second embodiment 3. Third embodiment (1) Example of the configuration of an image display device (2) Example of captured image 4. Fourth embodiment 5. Fifth embodiment 6. Sixth embodiment 7.Other applications

[0016] 1. First embodiment (1) Example of the configuration of an image display device First, a configuration example of an image display device according to a first embodiment of the present technology will be described with reference to Fig. 1 and Fig. 2. Fig. 1 is a plan view seen from above showing a configuration example of an image display device 100 according to the present embodiment. Fig. 2 is a schematic diagram showing an example of a captured image obtained by capturing an image of the front of an eyeball 105 using the image display device 100.

[0017] The image display device 100 can be used as a holographic wavefront reconstruction eyewear display that, for example, applies light to a hologram and uses diffraction caused by interference fringes recorded on the hologram to create a wavefront identical to the object light, which is the original signal wave. The image display device 100 can be applied to an off-axis optical system that is mainly intended for augmented reality (AR).

[0018] As shown in FIG. 1, the image display device 100, as an example, includes an optical unit 101 which is an optical engine that performs wavefront reconstruction, a combiner 102 which is a focusing unit that changes the direction of light beams such as image display light OL and conjugate light CL to focus the light on the user's eye and guide it to the pupil, an imaging unit 103 which is a detection unit that acquires positional relationship data between the eyeball and the light beams, and a control unit 104 which generates an image, adjusts the position of the image, etc.

[0019] The image display device 100 forms an optical system with an optical unit 101 and a combiner 102. The optical system can also refer to the optical unit 101 and the combiner 102 excluding a light source unit described later, and has a role of projecting conjugate light CL emitted from the light source unit onto the periphery of the pupil of the user's eye. The combiner 102 has an optical element 113 that reflects the conjugate light CL, and a diffractive optical element (DOE) is used for the optical element 113.

[0020] The imaging unit 103 is included in a detection unit that detects reflected light of the conjugate light CL projected from the optical system and reflected around the pupil. The imaging unit 103 captures reflected light of an image formed on the retina when the image display light OL enters the pupil, and a reflected image of the conjugate light CL reflected around the pupil. The imaging unit 103 can obtain information on the image display light OL and the conjugate light CL not only from one eye, but also from both eyes.

[0021] The optical unit 101 has a light source unit 111 that generates a (partially) coherent light source using a semiconductor laser (LD), a superluminescent diode (SLD), a light emitting diode (LED), or the like, and a spatial light phase modulator (SLM) 112 that is a modulation unit that spatially modulates the amplitude or phase of an image display light OL and a conjugate light CL. The optical unit 101 generates an image display light (object light) OL that displays an image of an object, and a conjugate light CL of the image display light OL. Here, the conjugate light refers to light that is guaranteed to form an angle with the corresponding image display light. Specific examples of the optical unit 101 include optical components such as a lens or a holographic optical element (HOE) having lens performance, a diffractive optical element (DOE), a metasurface, and a metamaterial.

[0022] The image display light OL and its conjugate light CL emitted from the light source unit 111 of the optical unit 101 are displayed on a computer generated hologram (CGH) on the SLM 112. At this time, the image display light OL and its conjugate light CL are conjugate to each other, so that the angular relationship between them is guaranteed. The image display light OL and its conjugate light CL enter the combiner 102, and a wavefront reflected from each optical element on the combiner is generated, and the reconstructed image display light OL enters the user's eyeball 105. The image display light OL entered the eyeball 105 enters the pupil and forms an image on the retina.

[0023] In FIG. 1, the depth direction, which is the line of sight of the user's eyeball 105, is the Z direction (the up-down direction on the paper), and in a plane perpendicular to the Z direction, the directions perpendicular to each other are the X direction (the left-right direction on the paper) and the Y direction (the front-to-back direction on the paper).

[0024] 2 is a schematic diagram showing an example of a captured image captured using the image display device 100. As shown in FIG. 2, the image display light OL is incident on the retina from the pupil 121 of the eyeball 105. On the other hand, the conjugate light CL is incident on the periphery of the pupil 121 of the eyeball 105, for example, on the boundary between the iris 122 and the sclera 123. Here, the periphery of the pupil 121 refers to the iris 122, the sclera 123, the boundary between the pupil 121 and the iris 122, and the boundary between the iris 122 and the sclera 123. Note that by projecting the conjugate light CL on the boundary between the iris 122 and the sclera 123, the position of the image display light OL can be more easily determined.

[0025] In FIG. 2, the depth direction, which is the line of sight of the user's eyeball 105, is the Z direction (the front-to-back direction on the paper), and in a plane perpendicular to the Z direction, the directions perpendicular to each other are the X direction (the left-to-right direction on the paper) and the Y direction (the up-to-down direction on the paper).

[0026] Next, an overview of eye tracking using the image display device 100 will be described with reference to Fig. 1 to Fig. 4. Fig. 3 is a plan view seen from above showing an example of the configuration of the image display device 100 when a displaced image is incident on the pupil. Fig. 4 is a schematic diagram showing an example of a captured image when a displaced image is incident on the pupil.

[0027] 3, when the incident image display light OL is displaced from the center (initial reference position) of the pupil 121, the conjugate light CL also displaces by the same amount and is incident on the periphery of the pupil 121. Specifically, for example, as shown in Fig. 4, when the image display light OL is displaced rightward on the paper surface of Fig. 4 by a displacement amount d1 from the center of the pupil 121, the conjugate light CL also displaces rightward on the paper surface of Fig. 4 by the displacement amount d1 and is incident on the sclera 123.

[0028] At this time, the image display light OL incident on the eyeball 105 is incident on the pupil shifted from the center of the pupil 121 by the shift amount d1 and forms an image on the retina. On the other hand, the conjugate light CL is reflected on the sclera 123 at a position shifted by the shift amount d1, and the reflected image is captured by the imaging unit 103. The captured image captured by the imaging unit 103 is acquired by the control unit 104, which calculates the shift amount d1 and the correction amount, and provides feedback to the control unit 104 again.

[0029] The correction method may involve changing the phase pattern superimposed on the SLM 112 in the optical unit 101 when generating the wavefront, or may involve inserting an additional steering element such as a dynamic optical element (MEMS mirror, liquid prism, etc.).

[0030] The deviation from the initial reference position in the up / down / left / right directions (XY directions) on the same plane as the screen can be obtained by calculating the deviation from the initial reference position by observing the position of the Fourier image of the conjugate light CL on the eyeball 105. In addition, if there is a deviation in the depth direction (Z direction), which is the line of sight of the eyeball 105, it can be determined from the defocus amount, which is the amount by which the Fourier image of the conjugate light CL is shifted from the focus position in the optical axis direction.

[0031] (2) Example of control unit configuration Next, an example of the configuration of the control unit according to this embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing an example of the configuration of the control unit 104 of the image display device 100. The control unit 104 generates images and adjusts the positions of the images, and also stores data of initial reference positions. The control unit 104 also performs calculations from the image data to derive the positions of the images and predict the positions.

[0032] Here, for example, when using a head mounted display (HMD), the initial reference position is set to the position where the image can be seen based on the position of the user's head when wearing it. Methods for setting this initial reference position include mechanical methods like VR headsets, and electrical methods using electrically driven mirrors, etc. Note that an SLM is not used from the beginning because we want to secure a range for adjusting deviations after setting the initial position.

[0033] 5, the control unit 104 includes, for example, a central processing unit (CPU) 131, a calculation unit 132, a determination unit 133, and a storage unit 134. The CPU 131 is electrically connected to each of the calculation unit 132, the determination unit 133, and the storage unit 134, and controls commands to each unit.

[0034] The calculation unit 132 calculates the amount of deviation and the amount of correction of the conjugate light CL. The determination unit 133 determines whether the amount of deviation of the conjugate light CL is within an allowable range. The storage unit 134 stores the amount of deviation and the amount of correction calculated by the calculation unit 132. The CPU 131 reads out the amount of deviation or the amount of correction stored in the storage unit 134 at an appropriate timing.

[0035] The control unit 104 can calculate the user's current line of sight direction and optical axis direction by calculating the shift amount and correction amount of the image to be displayed. The control unit 104 can also calculate the shift in the XY direction from the initial reference position of the image to be displayed based on the position of the Fourier image of the conjugate light CL around the pupil 121. The control unit 104 can also calculate the shift in the Z direction from the initial reference position of the image to be displayed based on the defocus amount of the Fourier image of the conjugate light CL. Furthermore, the control unit 104 can change the size and separation width of the Fourier image of the image to be displayed by adjusting the initial phase of the image to be displayed.

[0036] (3) Example of image display method (tracking method) Next, an example of an image display method (tracking method) according to this embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of an image display method using the image display device 100.

[0037] First, in step S1, a user wears eyewear, that is, the image display device 100. After wearing the image display device 100, the power supply of the image display device 100 is turned on.

[0038] Next, in step S2, the control unit 104 aligns the initial reference position of the image to be displayed with, for example, the center of the pupil 121. After aligning the initial reference position, the initial reference position is stored. After that, the optical unit 101 emits image display light OL for displaying the image and conjugate light CL of the image display light OL.

[0039] In step S3, the optical unit 101 projects the emitted image display light OL and conjugate light CL onto the pupil 121 of the user's eye and the periphery of the pupil 121 of the user's eye, respectively. Then, the user views the image or video displayed on the image display device 100.

[0040] In step S4, the detection unit detects the reflected light of the projected conjugate light CL reflected around the pupil 121. That is, the imaging unit 103, which is the detection unit, captures and obtains an image or video due to the conjugate light CL of the image display light OL.

[0041] In step S5, the control unit 104 controls the position of the image to be displayed based on the detected reflected light. That is, the calculation unit 132 of the control unit 104 calculates the amount of deviation for each set frame.

[0042] In step S6, the determination unit 133 determines whether or not the deviation amount d1 is within the allowable range. If it is within the allowable range (YES), the process returns to step S3, and the user continues to view the image or video displayed on the image display device 100. If it is not within the allowable range (NO), the process proceeds to step S7.

[0043] In step S7, the calculation unit 132 calculates a correction amount (shift value) based on the deviation amount d1, and corrects the incident position of the image display light OL.

[0044] In step S8, storage unit (memory) 134 stores the correction amount calculated by calculation unit 132, and the process proceeds to step S9.

[0045] In step S9, CPU 131 causes image display device 100 to display the image corrected using the correction amount, and the process returns to step S3, where the user is allowed to view the corrected image or video.

[0046] According to the image display device 100 of this embodiment, the relationship between the image display light OL and the conjugate light CL is determined, so that the image display light OL can be always irradiated at a position a certain distance away from the image display light OL. Therefore, the relative position of the irradiated light with respect to the image display light OL is completely guaranteed by the physical diffraction condition, and there is no need to worry about deviation of their relative positions.

[0047] In this way, the image display device 100 uses the conjugate light CL that is generated simultaneously with the image display light OL and is emitted at a physically determined angle for the eyewear display of the wavefront reconstruction type using the CGH, so that precise and accurate eye tracking can be performed without requiring an additional light source or pattern generator for eye tracking. Note that the image display device 100 can also dynamically change the irradiation position and the number of emissions of the image display light OL.

[0048] Furthermore, the image display device 100 can change the distance of the conjugate light CL from the image display light OL and the pattern of the conjugate light CL by changing the initial phase of the image (reproduced image) to be displayed. This makes it possible to create a dense pattern of the conjugate light CL, so that while an image is displayed by the image display light OL, the eye direction can be detected by the conjugate light CL that appears nearby, thereby improving tracking accuracy.

[0049] Moreover, the image display device 100 emits a plurality of two-dimensional image display lights OL, and can detect translational deviation in the XY plane from changes in the reflection pattern on the curved surface of the eyeball 105. Furthermore, the image display device 100 can sense the depth in the Z direction and detect the amount of deviation by utilizing the defocus amount of the conjugate light CL. The amount of deviation can be sensed and updated at regular intervals.

[0050] As described above, in an image display device 100 such as a wavefront reproduction display using a CGH, the conjugate light CL is emitted in addition to the image display light OL. This conjugate light CL has conventionally been removed by a filter or the like as unnecessary light or has not been used, but by converting the conjugate light CL into eye tracking as an index, it can be used as a very useful reference in eye tracking, gaze tracking, and the like.

[0051] (4) Variations Next, a modified example of the image display device 100 will be described with reference to Fig. 7. Fig. 7 is a schematic diagram of an example of an image captured using a modified example of the image display device 100 according to the present embodiment. This modified example uses a multiple order diffractive optical element (DOE) in the combiner 102.

[0052] By using, for example, a thin DOE or the like that produces multiple orders of diffraction as an optical element that reflects the conjugate light CL on the combiner 102, it is possible to obtain a plurality of diffraction patterns of the conjugate light CL as shown in Fig. 7. This makes it possible to derive the line of sight and the position of the pupil 121 with higher accuracy, thereby improving the accuracy of eye tracking.

[0053] 2. Second embodiment Next, a configuration example of an image display device according to a second embodiment of the present technology will be described with reference to Fig. 8. Fig. 8 is a plan view seen from above showing a configuration example of an image display device 200 according to the present embodiment. The image display device 200 can be applied to a coaxial optical system such as virtual reality (VR).

[0054] 8, image display device 200 includes optical section 101, imaging section 103, and control section 104, similar to image display device 100 according to the first embodiment. Image display device 200 further includes eyepiece 201. Optical section 101, control section 104, and eyepiece 201 of image display device 200 are arranged such that the centers of the respective sections are aligned on the same line, and the optical axes are linear.

[0055] According to the image display device 200 of this embodiment, even though it is a coaxial optical system, precise and accurate eye tracking can be performed without requiring an additional light source or pattern generator for eye tracking, as with the image display device 100 of the first embodiment.

[0056] 3. Third embodiment (1) Example of the configuration of an image display device Next, a configuration example of an image display device according to a third embodiment of the present technology will be described with reference to Fig. 9. Fig. 9A is a plan view seen from above showing a configuration example of an image display device 300 according to the present embodiment. The image display device 300 can divide a pattern on the pupil by adjusting an initial phase.

[0057] 9A, the image display device 300 includes, as an example, a light source unit 111, an SLM 112, a lens L1, a lens L2, and an eyepiece lens L3. The components of the image display device 300 are arranged in a straight line.

[0058] Image display light OL emitted from the light source unit 111 of the image display device 300 with its initial phase adjusted passes through the SLM 112 and the lens L1, and generates a first Fourier plane FP1 between the lens L1 and the lens L2. After that, the image display light OL passes through the lens L2 to form an image plane 301 between the lens L2 and the eyepiece lens L3. After that, the image display light OL passes through the eyepiece lens L3 to generate a Fourier plane FP2 in front of the pupil 121 of the eyeball 105.

[0059] Fig. 9B shows a case where a periodic pattern is added to the image to be displayed on the second Fourier plane FP2. When a periodic pattern is added to the image to be displayed as shown in Fig. 9B, it is possible to increase the number of points on the second Fourier plane FP2 in front of the pupil, and the field of view of the eyeball 105 is expanded.

[0060] (2) Example of captured image 10 to 13, examples of images captured using the image display device 300 will be described. Fig. 10 is a schematic diagram of an image captured by capturing an image of the front of the eyeball 105 using the image display device 300.

[0061] As shown in Fig. 10, when a periodic pattern is added to an image to be displayed, the number of points on the second Fourier plane FP2 in front of the pupil is increased, so that the image display light OL irradiated to the pupil 121 is increased from one point to multiple points in a predetermined narrow range. Then, the conjugate light CL irradiated to the iris 122 and sclera 123 around the pupil 121 is also increased to multiple points like the image display light OL, so that it becomes easier to determine the amount of tracking deviation. Note that the more the number of multiple points is increased and the narrower the interval (pitch) between them, the more accurate the tracking can be.

[0062] Therefore, when it is desired to detect the amount of deviation precisely, the image display device 300 can change the image to be displayed on the pupil 121, that is, the size and separation width of the Fourier image to be generated in front of the pupil 121, by adjusting the phase of the original image at the stage of calculating the CGH. For example, it is possible to generate a Fourier image with a larger number of divisions, and by inserting this into an arbitrary frame, it is possible to increase the detection accuracy, which is useful not only for estimating the position of the pupil 121 but also for estimating the amount of rotation of the eyeball 105 in more detail when performing eye tracking.

[0063] Next, with reference to Figs. 11 to 13, the detection of a shift in a captured image obtained by capturing the front of the eyeball 105 using the image display device 300 will be described. Fig. 11 is a schematic diagram for explaining the detection of a translation shift in the XY plane of a captured image obtained by capturing the front of the eyeball 105 using the image display device 300. Fig. 11A is a schematic diagram showing a reference state in which there is no shift and the image display light OL is irradiated to the center of the pupil 121. Fig. 11B is a schematic diagram showing a state in which the image display light OL is irradiated to the pupil 121 while being shifted to the right side as viewed on the paper. Fig. 11C is a schematic diagram showing a state in which the image display light OL is irradiated to the pupil 121 while being shifted to the left side as viewed on the paper.

[0064] As shown in Figure 11B or Figure 11C, when the conjugate light CL at multiple points in the captured image is shifted in the XY direction from the initial reference position beyond the allowable range, the amount of shift can be found by using the image display device 300 to observe the position of the Fourier image of the conjugate light CL on the eyeball 105 and calculating the shift from the initial reference position.

[0065] Fig. 12 is a schematic diagram for explaining detection of depth shift in the Z direction of an image captured by capturing an image of the front of the eyeball 105 using the image display device 300. Fig. 12A is a schematic diagram showing a reference state in which the image display light OL is irradiated to the center of the pupil 121 without any focus shift. Fig. 12B is a schematic diagram showing a state in which the focus of the image display light OL is irradiated to a position farther away from the eyeball 105 than the initial reference position. Fig. 12C is a schematic diagram showing a state in which the focus of the image display light OL is irradiated to a position closer to the eyeball 105 than the initial reference position.

[0066] As shown in Figure 12B or Figure 12C, when the focus of the conjugate light CL at multiple points in the captured image is shifted in the Z direction from the initial reference position beyond the allowable range, the amount of shift can be determined by the defocus amount of the Fourier image of the conjugate light CL.

[0067] Fig. 13 is a schematic diagram for explaining the detection of rotational displacement when the orientation of the pupil 121 rotates to the left or right by a rotation angle θ from an initial reference position in a plane in which the eyeball 105 is viewed from above. Fig. 13A is a schematic diagram showing a reference state in which there is no rotational displacement and the center of the pupil 121 is irradiated with the image display light OL. Fig. 13B is a schematic diagram showing a state in which the pupil 121 is irradiated with the image display light OL when the pupil 121 rotates leftward toward the incident image display light OL. Fig. 13C is a schematic diagram showing a state in which the pupil 121 is irradiated with the image display light OL when the pupil 121 rotates rightward toward the incident image display light OL.

[0068] 13B or 13C, when the conjugate light CL at multiple points in the captured image is deviated from the initial reference position in the rotation direction beyond an allowable range, the amount of deviation can be obtained by observing the position of the Fourier image of the conjugate light CL on the eyeball 105 using the image display device 300 and calculating the deviation from the initial reference position. In addition, when the deviation is in the rotation direction, it is also possible to determine the deviation from the initial reference position by observing the asymmetry of the left and right widths between the iris 122 and the sclera 123 as shown in FIG. 13B or 13C.

[0069] The image display device 300 according to this embodiment changes the distance of the conjugate light CL from the image display light OL and its pattern by changing the initial phase of the image to be displayed, and observes the conjugate light CL at a plurality of points in a predetermined narrow range, so that the tracking accuracy can be further improved compared to the image display device 100 according to the first embodiment. Also, the image display device 300 can derive the line of sight and the position of the pupil 121 with higher accuracy.

[0070] Furthermore, the image display device 300 can sense not only the translational shift in the XY plane but also the Z direction (depth) and detect the amount of misalignment by utilizing the defocus amount of the conjugate light CL. The amount of misalignment can be sensed and updated at regular intervals.

[0071] 4. Fourth embodiment Next, a configuration example of an image display device according to a fourth embodiment of the present technology will be described with reference to Fig. 14. Fig. 14 is a plan view seen from above showing a configuration example of an image display device 400 according to the present embodiment. The image display device 400 includes a device for shifting or splitting a light flux in an optical section.

[0072] 14, like the image display device 100 according to the first embodiment, the image display device 400 includes an optical unit 401, a combiner 102, an imaging unit 103, and a control unit 104. In the image display device 400, the optical unit 401 and the combiner 102 form an optical system.

[0073] Although not shown, the optical section 401 includes a light source section 111 and an SLM, similar to the image display device 100. The optical section 401 further includes a beam shifting and / or splitting device 411, for example, on the Fourier plane.

[0074] The image display device 400 has a light flux shifting and / or splitting device 411 in the optical unit 401, and can, for example, shift the image display light emitted from the optical unit 401 to image display light OL1, OL2, and OL3, and the conjugate light of these image display lights to conjugate light CL1, CL2, and CL3.

[0075] The image display device 400 according to this embodiment amplifies the Fourier image by splitting the conjugate light CL emitted from the optical unit 401, and observes the conjugate light CL at a plurality of points, so that the tracking accuracy can be further improved compared to the image display device 100 according to the first embodiment. Furthermore, the image display device 400 can track the pupil 121 even when the pupil 121 is dilated.

[0076] 5. Fifth embodiment Next, a configuration example of an image display device according to a fifth embodiment of the present technology will be described with reference to Fig. 15. Fig. 15 is a plan view seen from above showing a configuration example of an image display device 500 according to this embodiment. The image display device 500 can perform eye tracking using a light source unit with multiple light sources.

[0077] As shown in FIG. 15, as an example, image display device 500 includes an optical unit 501 having a light source unit that emits light beam 511 from light source 1 and light beam 512 from light source 2, a reflector 502, a prism 503, an SLM 112, a lens L1, a lens L2, and an eyepiece lens L3.

[0078] The light beam 511 from the light source 1 and the light beam 512 from the light source 2, which are emitted after adjusting the initial phase in the light source section of the optical section 501, are reflected by the reflecting mirror 502 toward the prism 503. The reflected light beam 511 from the light source 1 and the light beam 512 from the light source 2 are reflected in the prism 503 toward the SLM 112, and from the SLM 112 pass through the prism 503 again toward the lens. The light beam 511 from the light source 1 and the light beam 512 from the light source 2, which have passed through the lens L1, generate a first Fourier plane FP1 between the lens L1 and the lens L2. The image display light OL, which has then passed through the lens L2, forms an image plane 301 between the lens L2 and the eyepiece lens L3. The image display light OL, which has then passed through the eyepiece lens L3, generates a Fourier plane FP2 in front of the pupil 121 of the eyeball 105.

[0079] The image display device 500 according to this embodiment uses a plurality of light sources to shift image display light, for example, to a light ray 511 from the light source 1 and a light ray 512 from the light source 2, and also shifts their conjugate light, thereby amplifying the Fourier image and observing the conjugate light CL at a plurality of points. Therefore, according to the image display device 500, the tracking accuracy is further improved compared to the image display device 100 according to the first embodiment, and the pupil 121 can be tracked even when the pupil 121 is enlarged.

[0080] 6. Sixth embodiment Next, an example of eye tracking using an image display device according to a sixth embodiment of the present technology will be described with reference to Fig. 16. Fig. 16 is a schematic diagram of an example of a captured image captured using the image display device according to the present technology. In this embodiment, the optical system inserts an emission pattern of an initial reference position into a part of the frame, measures the current amount of deviation of the image to be displayed, and periodically performs calibration.

[0081] Fig. 16A is a schematic diagram showing a reference state in which there is no misalignment of the image display device and the image display light OL is irradiated to the center (initial reference position) of the pupil 121. Fig. 16B is a schematic diagram showing a state in which misalignment of the image display device occurs and the image display light OL is irradiated to the pupil 121 while shifted to the right side as viewed from the paper surface. Fig. 16C is a schematic diagram showing a state in which the amount of misalignment caused by the misalignment is corrected and the shifted image display light SOL is irradiated to the center of the pupil 121.

[0082] 16A and 16B, the emission pattern of the initial reference position is emitted for a moment and inserted into a part of the frame, and the current deviation amount d2 of the image to be displayed is measured and calibration is performed periodically. Based on the measured deviation amount d2, the correction amount is calculated, and the shifted image display light SOL is moved to irradiate the center of the pupil 121.

[0083] As described above, in this embodiment, since the accumulation error is small, the accuracy of eye tracking can be improved. In this embodiment, it is preferable to perform eye tracking using a combination of the image display light OL that is not irradiated to the pupil 121 and its conjugate light CL.

[0084] 7.Other applications In the above embodiment, the image display device is described as being applied to one eye, but the image display device according to the present technology is not limited to being applied to one eye, and can also be applied to both eyes. When the image display device according to the present technology is used for both eyes, it is possible to derive a more accurate line of sight and pupil position from data of both eyes, and therefore it is possible to estimate an amount of eyeball rotation, a gaze point, and therefore an amount of convergence and adjustment with even higher accuracy.

[0085] In addition, the present technology can have the following configurations. (1) a light source unit that emits image display light for displaying an image and conjugate light of the image display light; an optical system that projects the image display light emitted from the light source unit onto a pupil of a user's eye and projects the conjugate light emitted from the light source unit onto a periphery of the pupil of the user's eye; a detection unit that detects reflected light of the conjugate light projected from the optical system and reflected around the pupil; and a control unit that controls a position of an image to be displayed based on the reflected light detected by the detection unit. (2) The image display device described in (1), wherein the optical system has a focusing unit that changes the direction of the image display light and the conjugate light to focus them on the user's eye, and a modulation unit that spatially modulates the amplitude or phase of the image display light and the conjugate light. (3) the detection unit has an imaging unit configured to capture an image formed on a retina when the image display light is incident on the pupil and a reflected image formed by the conjugate light reflected around the pupil, The image display device according to (1) or (2), wherein the control unit calculates an amount of shift and an amount of correction for an image to be displayed based on the formed image and the reflected image captured by the imaging unit. (4) The image display device according to (3), wherein the control unit calculates the amount of shift and the amount of correction of the image to be displayed, thereby calculating the current line of sight direction and the optical axis direction of the user. (5) The image display device according to any one of (1) to (4), wherein the pupil area is one of the iris, the sclera, the boundary between the pupil and the iris, and the boundary between the iris and the sclera. (6) The image display device according to any one of (1) to (5), wherein the control unit calculates a deviation in a plane perpendicular to the line of sight of the eye from an initial position of the image to be displayed based on a position of the Fourier image of the conjugate light around the pupil. (7) The image display device according to any one of (1) to (6), wherein the control unit calculates a deviation of the gaze direction of the eye from an initial position of the image to be displayed based on a defocus amount of a Fourier image of the conjugate light. (8) The image display device according to any one of (1) to (7), wherein the control unit changes a size or a separation width of a Fourier image of the image to be displayed by adjusting an initial phase of the image to be displayed. (9) The image display device according to any one of (1) to (8), wherein the optical system is a coaxial optical system. (10) The image display device according to any one of (1) to (9), wherein the detection unit acquires information on the image display light and the conjugate light from both eyes. (11) The image display device according to (2), wherein the light-collecting section uses a diffractive optical element as an optical element that reflects the conjugate light. (12) The image display device according to any one of (1) to (11), wherein the optical system has a beam shifting and / or splitting device in a Fourier plane. (13) The image display device according to any one of (1) to (12), wherein the light source section uses a plurality of light sources. (14) The image display device according to any one of (1) to (13), wherein the optical system includes an emission pattern at an initial position in a portion of the frame, measures the amount of deviation of the image to be displayed, and periodically performs calibration. (15) emitting image display light for displaying an image and conjugate light of the image display light; projecting the emitted image display light and the emitted conjugate light onto a pupil of a user's eye and a peri-pupil of the user's eye, respectively; detecting light reflected from the projected conjugate light around the pupil; controlling a position of an image to be displayed based on the detected reflected light; An image display method comprising: [Explanation of symbols]

[0086] 100, 200, 300, 400, 500 Image display device 101, 401, 501 Optical Department 102 Combiner (light collecting part) 103 Imaging unit 104 Control section 105 Eyeball (right eye) 111 Light source section 112 SLM 121 Pupil 122 Iris 123 Sclera 131 CPU 132 Arithmetic section 133 Judgment section 134 Storage section 201 Eyepiece 301 Image surface 411 Light beam shifting and / or splitting devices 502 Reflector 503 Prism 511 Light from light source 1 512 Light from light source 2 OL, OL1~OL3, SOL Image display light (object light) CL, CL1-CL4, SCL conjugate light d1, d2 deviation amount L1~L3 Lenses FP1, FP2 Fourier plane

Claims

1. a light source unit that emits image display light for displaying an image and conjugate light of the image display light; an optical system that projects the image display light emitted from the light source unit onto a pupil of a user's eye and projects the conjugate light emitted from the light source unit onto a periphery of the pupil of the user's eye; a detection unit that detects reflected light of the conjugate light projected from the optical system and reflected around the pupil; and a control unit that controls a position of an image to be displayed based on the reflected light detected by the detection unit.

2. 2. The image display device according to claim 1, wherein the optical system includes a focusing unit that changes directions of the image display light and the conjugate light to focus the light on the user's eye, and a modulation unit that spatially modulates the amplitude or phase of the image display light and the conjugate light.

3. the detection unit has an imaging unit configured to capture an image formed on a retina when the image display light is incident on the pupil and a reflected image formed by the conjugate light reflected around the pupil, The image display device according to claim 1 , wherein the control unit calculates an amount of shift and an amount of correction of the image to be displayed based on the formed image and the reflected image captured by the imaging unit.

4. The image display device according to claim 3 , wherein the control unit calculates a shift amount and a correction amount of the image to be displayed, to thereby calculate a current line of sight direction and an optical axis direction of the user.

5. The image display device according to claim 1 , wherein the pupil periphery is any one of an iris, a sclera, a boundary between the pupil and the iris, and a boundary between the iris and the sclera.

6. The image display device according to claim 1 , wherein the control unit calculates a deviation in a plane perpendicular to a line of sight of the eye from an initial position of the image to be displayed, based on a position of the Fourier image of the conjugate light in the periphery of the pupil.

7. The image display device according to claim 1 , wherein the control unit calculates a deviation of a line of sight of the eye from an initial position of the image to be displayed based on a defocus amount of the Fourier image of the conjugate light.

8. The image display device according to claim 1 , wherein the control unit changes a size and a separation width of a Fourier image of the image to be displayed by adjusting an initial phase of the image to be displayed.

9. The image display device according to claim 1 , wherein the optical system is a coaxial optical system.

10. The image display device according to claim 1 , wherein the detection section acquires information on the image display light and the conjugate light from both eyes.

11. The image display device according to claim 2 , wherein the light collecting section uses a diffractive optical element as an optical element for reflecting the conjugate light.

12. 2. The image display apparatus according to claim 1, wherein the optical system comprises a beam shifting and / or splitting device in the Fourier plane.

13. The image display device according to claim 1 , wherein the light source section uses a plurality of light sources.

14. 2 . The image display device according to claim 1 , wherein the optical system is periodically calibrated by inserting an emission pattern at an initial position into a portion of the frame and measuring an amount of deviation of the image to be displayed.

15. emitting image display light for displaying an image and conjugate light of the image display light; projecting the emitted image display light and the emitted conjugate light onto a pupil of a user's eye and a peri-pupil of the user's eye, respectively; detecting light reflected from the projected conjugate light around the pupil; controlling a position of an image to be displayed based on the detected reflected light; An image display method comprising:

Images