Image projection device

Abstract

To suppress supply of an image causing a feeling of incongruity.SOLUTION: An image projection device 100 includes: a projection section 10 including a light source 11, a scanning section 13 for performing scanning of rays 50 emitted from the light source 11, and a projection optical system 20 for converging the multiple rays 50 scanned by the scanning section 13 and emitted in different directions from the scanning section 13 to a convergence point 70 in a user's eyes 60 and then projecting them to a retina 61 for projecting an image; a drive section 41 for moving components included in the projection section 10 capable of moving the position of the convergence point 70; an imaging section 40 for imaging the user's eyes 60; an acquisition section 34 for acquiring a value regarding a pupil 62 projected to a captured image 80 captured by the imaging section 40; and a drive control section 33 for controlling a drive section 41 on the basis of the position of the pupil 62 projected to the captured image 80 such that the multiple rays 50 are projected to the retina 61, and preventing control of the drive section 41 based on the position of the pupil 62 when a value regarding the pupil 62 acquired by the acquisition section 34 satisfies a predetermined condition.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image projection device. [Background technology]

[0002] There is known an image projection device that uses light rays emitted from a light source to directly project an image onto a user's retina. Such an image projection device uses a method called Maxwellian vision. In Maxwellian vision, an image is projected onto the retina by converging a plurality of light rays that form an image inside the user's eye and then projecting the image onto the retina.

[0003] In an image projection device using Maxwellian vision, when a user moves their line of sight, the light rays may deviate from the user's pupil, and the image may not be projected onto the retina. Therefore, a method has been proposed for allowing the image to continue to be projected onto the retina even when the user moves their line of sight (e.g., Patent Document 1). Also, a method has been proposed for providing an image that makes it easy to gaze at the peripheral area when the user moves their line of sight to gaze at the peripheral area of ​​the image (e.g., Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2009-122550 A [Patent Document 2] JP 2023-078881 A Summary of the Invention [Problem to be solved by the invention]

[0005] It is conceivable to control the projection position of the light beam onto the user's eye based on the position of the pupil captured in the captured image of the user's eye so that the image continues to be projected onto the retina even if the user moves the direction of his or her line of sight. In this case, it is necessary to prevent the image presented to the user from feeling unnatural.

[0006] The present invention has been made in consideration of the above-mentioned problems, and has an object to prevent the provision of images that give a sense of discomfort. [Means for solving the problem]

[0007] The present invention is an image projection device comprising: a projection unit including a light source, a scanning unit that scans light rays emitted from the light source, and a projection optical system that causes a plurality of light rays emitted from the scanning unit in different directions to converge to a first convergence point in a user's eye by being scanned by the scanning unit, and then projects the light rays onto a retina to project an image onto the retina; a drive unit that moves a part provided on the projection unit that can move the position of the first convergence point; an imaging unit that images the user's eye; an acquisition unit that acquires a value related to the pupil captured in an image captured by the imaging unit; and a drive control unit that controls the drive unit based on the position of the pupil captured in the image so that the plurality of light rays are projected onto the retina, and does not control the drive unit based on the position of the pupil if the value related to the pupil acquired by the acquisition unit satisfies a predetermined condition.

[0008] In the above configuration, the acquisition unit acquires a value related to the change in size of the pupil captured in the captured image, and the drive control unit can be configured not to control the drive unit based on the position of the pupil if the absolute value of the value related to the change is greater than a threshold value.

[0009] In the above configuration, the acquisition unit acquires a value related to the movement of the pupil position captured in the captured image, and the drive control unit can be configured not to control the drive unit based on the pupil position if the value related to the movement is less than a threshold value.

[0010] In the above configuration, the acquisition unit acquires a value related to the variation in the position of the pupil captured in the captured image, and the drive control unit can be configured not to control the drive unit based on the position of the pupil if the value related to the variation is less than a threshold value.

[0011] In the above configuration, the drive control unit can be configured to change the drive current applied to the drive unit to a desired drive current over a period of 80% or more of a frame time among multiple frames captured by the imaging unit.

[0012] In the above configuration, the drive control unit can be configured to control the drive unit based on control information that associates the position of the pupil captured in the captured image, a plurality of points in the image captured by the imaging unit, and a drive current for positioning the first convergence point at the plurality of points.

[0013] The above configuration may further include a storage unit that stores the control information.

[0014] In the above configuration, the projection optical system includes a projection unit that is positioned in front of the user's eye and converges the multiple light beams to the first convergence point, a reflection unit that converges the multiple light beams emitted from the scanning unit to a second convergence point in front of the projection unit, and an optical member that is provided between the projection unit and the reflection unit and converts each of the multiple light beams into converging light, converges the light beams at a converging point in front of the projection unit, and then causes the light beams to be incident on the projection unit as diffused light, and the component can be configured to be the reflection unit.

[0015] In the above configuration, the image projection device can be configured to include a warning control unit that outputs a warning if the position of the pupil captured in the captured image is outside a specified range when the user wears the image projection device, and changes the warning depending on the amount of deviation from the specified range. Effect of the Invention

[0016] According to the present invention, it is possible to prevent images that give a sense of incongruity from being provided. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram illustrating an image projection device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram mainly showing an optical system in the image projection device according to the first embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of a captured image of a user's eye captured by the imaging section. [Figure 4] FIG. 4(a) to FIG. 4(d) are diagrams showing an example of a method for identifying a pupil by the identifying unit. [Diagram 5] 5(a) and 5(b) are diagrams showing changes in the trajectory of a light ray when a reflecting mirror moves. [Figure 6] 6(a) to 6(c) are diagrams explaining the movement of the pupil and the light rays projected onto the retina. [Figure 7] 7(a) to 7(c) are diagrams for explaining an example of values ​​related to the pupil acquired by the acquisition unit. [Figure 8] 8(a) and 8(b) show the measurement results of the rate of change of pupil size and the rate of movement of the pupil center position. [Figure 9] FIG. 9 shows the measurement results of the variation rate of the pupil center position. [Figure 10] FIG. 10(a) and FIG. 10(b) are diagrams (part 1) showing changes in the center position of the pupil captured in a captured image. [Figure 11] FIG. 11(a) and FIG. 11(b) are diagrams (part 2) showing changes in the center position of the pupil captured in a captured image. [Figure 12] FIG. 12 is a flowchart illustrating a method for manufacturing the image projection device according to the first embodiment. [Figure 13] 13(a) to 13(c) are diagrams showing an example of capturing an image of light rays at a convergence point by the imaging section. [Figure 14] FIG. 14 is a flowchart (part 1) illustrating an example of a control method of the control unit in the first embodiment. [Figure 15] FIG. 15 is a flowchart (part 2) illustrating an example of a control method of the control unit in the first embodiment. [Figure 16] FIG. 16 is a diagram showing application of a drive current by the drive control section. [Figure 17] FIG. 17 is a diagram illustrating an image projection device according to the second embodiment. [Figure 18] FIG. 18 is a flowchart illustrating an example of a method for controlling a warning by the control unit in the second embodiment. [Figure 19] 19(a) to 19(c) are diagrams showing how to align the eye position with the image projection device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, embodiments of the present invention will be described with reference to the drawings. EXAMPLES

[0019] Fig. 1 is a diagram showing an image projection device 100 according to a first embodiment. The dashed lines in Fig. 1 indicate electrical connections. The image projection device 100 according to the first embodiment is a retinal projection type head mounted display that utilizes Maxwellian vision, in which light rays (laser light) for allowing a user to visually recognize an image are directly projected onto the user's retina.

[0020] 1, an image projection device 100 according to the first embodiment includes a projection unit 10, a control unit 30, an imaging unit 40, a driving unit 41, and a storage unit 42. The projection unit 10 includes a light source 11, a lens 12, a scanning unit 13, and a projection optical system 20. The projection optical system 20 includes a reflecting mirror 21, a projection mirror 22, and a lens 23. The control unit 30 includes a projection control unit 31, an imaging control unit 32, a driving control unit 33, an acquisition unit 34, and an identification unit 35.

[0021] The light source 11 emits a light beam 50 (laser light) based on an instruction from the projection control unit 31. The light source 11 emits the light beam 50, which is visible light, for example, red laser light (wavelength: about 610 nm to 660 nm), green laser light (wavelength: about 515 nm to 540 nm), and blue laser light (wavelength: about 440 nm to 480 nm). An example of the light source 11 that emits red, green, and blue laser light is a light source in which laser diode chips for each of RGB (red, green, and blue) and a three-color synthesis device are integrated. The light source 11 may emit the light beam 50 of a single wavelength.

[0022] The projection control unit 31 receives image data input from a camera and / or a recording device (not shown). The projection control unit 31 controls the emission of light beams 50 from the light source 11 based on the input image data. The projection control unit 31 also controls the driving of the scanning unit 13. The scanning unit 13 scans (e.g., raster scans) the light beams 50 at high speed from the upper left to the lower right of the image based on an instruction from the projection control unit 31. The projection control unit 31 controls the light source 11 and the scanning unit 13 to control the projection of the light beams 50 onto the user's retina 61. If a camera is installed at an appropriate position facing the user's line of sight, an image captured by the camera in the line of sight direction can be projected onto the retina 61. In addition, it is also possible to project an image input from a recording device or the like, or to superimpose a camera image and an image from the recording device, to project a so-called augmented reality (AR) image.

[0023] A light beam 50 emitted by the light source 11 passes through the lens 12. The lens 12 is a condenser lens that converts the light beam 50 from diffuse light to focused light. The light beam 50 that passes through the lens 12 enters the scanning unit 13. The scanning unit 13 (scanner) scans the light beam 50 in two-dimensional directions, the horizontal direction and the vertical direction. The scanning unit 13 is, for example, a MEMS (Micro Electric Mechanical System) mirror. Note that the scanning unit 13 may be a mirror other than the MEMS mirror, and may be, for example, potassium tantalate niobate (KTN) or the like.

[0024] The scanning unit 13 scans in two-dimensional directions, and the multiple light beams 50 emitted from the scanning unit 13 in different directions at different times are incident on the projection optical system 20. The projection optical system 20 converges the multiple light beams 50 emitted from the scanning unit 13 to a convergence point 70 inside the user's eye 60, and then projects the multiple light beams 50 on the retina 61 to project an image on the retina 61. For example, the multiple light beams 50 emitted from the scanning unit 13 are incident on a reflecting mirror 21. The reflecting mirror 21 is a concave mirror having a reflective surface made of a curved surface such as a free-form surface, and has a positive light-collecting power. The multiple light beams 50 reflected by the reflecting mirror 21 are converged at a convergence point 71 in front of the projection mirror 22. A lens 23 is provided at the convergence point 71 between the projection mirror 22 and the reflecting mirror 21. The lens 23 is, for example, a biconvex lens. The multiple light beams 50 pass through the lens 23 and are incident on the projection mirror 22. The projection mirror 22 is disposed in front of the user's eye 60. The projection mirror 22 reflects a plurality of light rays 50 toward the user's eye 60. The projection mirror 22 is a concave mirror having a reflective surface formed of a curved surface such as a free-form surface, and has a positive light-collecting power. The plurality of light rays 50 reflected by the projection mirror 22 pass through the user's pupil 62, converge at the lens 63 or a convergence point 70 near the lens 63, and are then projected onto the retina 61. This allows the user to view an image formed by the light rays 50. The projection mirror 22 may be a half mirror. In this case, the user can view an external image in a see-through manner.

[0025] Here, a process in which the light ray 50 emitted from the light source 11 is projected onto the retina 61 will be described. FIG. 2 is a diagram mainly showing an optical system in the image projection device 100 according to the first embodiment. As shown in FIG. 2, the light ray 50 emitted from the light source 11 passes through the lens 12. The lens 12 is a condensing lens that converts the light ray 50 from diffuse light to focused light. The light ray 50 that has passed through the lens 12 enters the scanning unit 13 in the form of focused light. The lens 12 is provided between the light source 11 and the scanning unit 13 in order to convert the light ray 50 reflected by the reflecting mirror 21 into a substantially parallel light.

[0026] A plurality of light beams 50 scanned in two-dimensional directions by the scanning unit 13 and emitted in different directions at different times from the scanning unit 13 are incident on the reflecting mirror 21. Each of the plurality of light beams 50 is condensed in front of the reflecting mirror 21, and then becomes diffused light and is incident on the reflecting mirror 21. Since the reflecting mirror 21 has a positive collecting power, each of the plurality of light beams 50 is reflected by the reflecting mirror 21 and converted from diffused light to approximately parallel light.

[0027] The multiple light rays 50 reflected by the reflection mirror 21 converge at a convergence point 71 in front of the projection mirror 22. A lens 23 is provided at the convergence point 71. The lens 23 is a condensing lens that converts each of the multiple light rays 50 from approximately parallel light to convergent light. The lens 23 is provided at the convergence point 71 to convert each of the multiple light rays 50 reflected by the projection mirror 22 into approximately parallel light. Each of the multiple light rays 50 transmitted through the lens 23 is condensed at a condensing point 72 in front of the projection mirror 22, and then becomes diffuse light and enters the projection mirror 22.

[0028] Since the projection mirror 22 has a positive focusing power, each of the multiple light beams 50 is converted from diffuse light to approximately parallel light by being reflected by the projection mirror 22. The multiple light beams 50 converge at a convergence point 70 in the user's eye 60. The convergence point 70 is located, for example, at the crystalline lens 63 or in the vicinity of the crystalline lens 63. The light beams 50 are converted from approximately parallel light to convergent light by the crystalline lens 63 and are focused in the vicinity of the retina 61. The light beams 50 are projected onto the retina 61, allowing the user to view an image.

[0029] The optical path length of the light ray 50a corresponding to the center of the image projected on the retina 61 will be described. If the optical path length of the light ray 50a between the scanning unit 13 and the reflecting mirror 21 is L1, and the optical path length of the light ray 50a between the projection mirror 22 and the convergence point 70 is L4, the optical path length L1 and the optical path length L4 are different. If the optical path length of the light ray 50a between the reflecting mirror 21 and the convergence point 71 is L2, and the optical path length of the light ray 50a between the convergence point 71 and the projection mirror 22 is L3, the optical path length L2 and the optical path length L3 are different. On the other hand, the ratio (L2 / L1) of the optical path length L2 to the optical path length L1 and the ratio (L3 / L4) of the optical path length L3 to the optical path length L4 are the same. Therefore, the layout is a similar shape at the same magnification, and the scanning angle of the light ray 50 by the scanning unit 13 and the convergence angle at which the multiple light rays 50 converge to the convergence point 70 are the same. The proportion of similarity may be determined by the distance between the projection mirror 22 and the user's eye 60, the shape of the user's face, and / or the space on the sides of the user's face, etc.

[0030] As shown in FIG. 1, the imaging unit 40 is disposed in front of the user's eye 60, and captures the user's eye 60 based on an instruction from the imaging control unit 32. The imaging unit 40 is, for example, an infrared camera, and captures the eye 60 by irradiating the eye 60 with infrared light. Considering that the user sees the outside world image in a see-through manner, the imaging unit 40 may be disposed at a position slightly shifted from the front of the eye 60 within a range in which the eye 60 can be captured. For example, when the projection optical system 20 is mounted on a glasses-type frame, the imaging unit 40 may be provided near the temples or nose pads of the glasses-type frame. FIG. 3 is a diagram showing an example of an image 80 of the user's eye 60 captured by the imaging unit 40. As shown in FIG. 3, at least the pupil 62 and the iris 64 are captured in the image 80 of the eye 60 captured by the imaging unit 40.

[0031] 1, the identification unit 35 acquires a captured image 80 of the eye 60 captured by the imaging unit 40 from the imaging unit 40, and identifies the pupil 62 captured in the captured image 80. The identification unit 35 identifies the pupil 62 captured in the captured image 80 by performing image analysis on the captured image 80. Here, an example of a method for identifying the pupil 62 by the identification unit 35 will be described.

[0032] 4(a) to 4(d) are diagrams showing an example of a method for identifying the pupil 62 by the identification unit 35. As shown in FIG. 4(a), a template image 82 including the pupil 62 and the iris 64 is stored in advance in the storage unit 42. As shown in FIG. 4(b), the identification unit 35 uses the template image 82 to identify an area 83 in the captured image 80 that has a high similarity to the template image 82. A commonly known method is used for the template matching here.

[0033] As shown in FIG. 4(c), the identification unit 35 performs a smoothing process on the extracted image of the region 83 to remove noise. In addition, since the image of the region 83 is a portion of the captured image 80, it is considered that the gradation values ​​are biased. For this reason, the identification unit 35 performs a process of greatly expanding the gradation values ​​of the image of the region 83. This makes it easier to obtain contours in the image of the region 83.

[0034] As shown in Fig. 4(d), the identification unit 35 identifies the range of the pupil 62 by detecting the contour of the image of the region 83 that has been subjected to the gradation value processing. At this time, the identification unit 35 may determine whether the identified range of the pupil 62 falls within the range of 2 mm to 6 mm, which is a general size (diameter) of a pupil, and if it does not fall within that range, it may be determined that the pupil 62 cannot be identified. Next, the identification unit 35 identifies the center position 65 of the pupil 62. The center position 65 may be the geometric center of the identified range of the pupil 62.

[0035] 1, the driving unit 41 is, for example, a biaxial actuator, and moves the reflecting mirror 21 based on instructions from the drive control unit 33. The reflecting mirror 21 is a concave mirror as described above. When the apex of the concave curved surface is located approximately in the center, the reflecting mirror 21 moves so as to oscillate around the apex. Of the multiple light rays 50, a light ray 50a corresponding to the center of the image projected onto the retina 61 is incident on the oscillation center of the reflecting mirror 21.

[0036] Figures 5(a) and 5(b) are diagrams showing changes in the trajectory of light ray 50 when the reflecting mirror 21 moves. In Figures 5(a) and 5(b), the reflecting mirror 21 in Figure 2 is shown by a dotted line. Figure 2 shows a case where light ray 50a, which corresponds to the center of the image projected onto the retina 61, enters the eye 60 almost head-on.

[0037] As shown in Figure 2, when a light ray 50a corresponding to the center of the image projected onto the retina 61 enters the eye 60 almost from the front, the convergence point 71 where the multiple light rays 50 reflected by the reflecting mirror 21 converge is near the center of the lens 23.

[0038] As shown in FIG. 5(a), when the user turns his / her gaze to the right to see the right side of the image, the pupil 62 moves to the right with respect to the user's face. In this case, the drive control unit 33 instructs the drive unit 41 to move the reflecting mirror 21 so that the light rays 50 pass through the left region of the lens 23 in the traveling direction of the light rays 50. As a result, the convergence point 71 where the light rays 50 converge is located in the left region of the lens 23 in the traveling direction of the light rays 50. For example, the drive unit 41 moves the reflecting mirror 21 counterclockwise around the swing center point 44 near the apex of the concave curved surface of the reflecting mirror 21, which is a concave mirror. The projection control unit 31 controls the light source 11 and the scanning unit 13 so that the light ray 50a corresponding to the center of the image projected on the retina 61 is incident on the vicinity of the swing center point 44. The multiple light rays 50 pass through a left region of the lens 23 with respect to the traveling direction of the light rays 50, and are refracted by the lens 23 to the right in the traveling direction, and then enter the projection mirror 22. As the multiple light rays 50 are refracted to the right in the traveling direction by the lens 23, the position and angle of incidence of the multiple light rays 50 on the projection mirror 22 change from the state in Fig. 2. As a result, the position of the convergence point 70 of the multiple light rays 50 moves to the right, and the multiple light rays 50 will enter the pupil 62 even if the pupil 62 moves to the right.

[0039] At least light ray 50a of the multiple light rays 50 enters the pupil 62 from the left side opposite to the direction (rightward) in which the pupil 62 has moved based on the direction T in which the pupil 62 faces after the pupil 62 has moved (i.e., the line of sight direction), and is projected onto a right-side region of the retina 61 in the direction in which the pupil 62 has moved. The point at which light ray 50a enters the reflecting mirror 21 substantially coincides with, for example, the oscillation center point 44 of the reflecting mirror 21 (for example, the vertex of a free-form surface).

[0040] As shown in FIG. 5B, when the user turns his / her gaze to the left to see the left side of the image, the pupil 62 moves to the left with respect to the user's face. In this case, the drive control unit 33 instructs the drive unit 41 to move the reflecting mirror 21 so that the light rays 50 pass through the right region of the lens 23 with respect to the traveling direction of the light rays 50. As a result, the convergence point 71 where the light rays 50 converge is located in the right region of the lens 23 with respect to the traveling direction of the light rays 50. For example, the drive unit 41 moves the reflecting mirror 21 clockwise around the swing center point 44 of the reflecting mirror 21. The projection control unit 31 controls the light source 11 and the scanning unit 13 so that the light rays 50a are incident near the swing center point 44. The light rays 50 pass through the right region of the lens 23 with respect to the traveling direction of the light rays 50, and are refracted by the lens 23 to the left in the traveling direction, and then enter the projection mirror 22. The multiple light rays 50 are refracted to the left in the traveling direction by the lens 23, so that the position and angle of incidence at which they are incident on the projection mirror 22 change from the state in Fig. 2. As a result, the position of the convergence point 70 of the multiple light rays 50 moves to the left, and the multiple light rays 50 become incident on the pupil 62 even if the pupil 62 moves to the left.

[0041] At least light ray 50a of the multiple light rays 50 enters the pupil 62 from the right side opposite to the direction (leftward) in which the pupil 62 has moved based on the direction T in which the pupil 62 faces after the pupil 62 has moved (i.e., the line of sight direction), and is projected onto the left side region of the retina 61, which is the direction in which the pupil 62 has moved. The point at which light ray 50a enters the reflecting mirror 21 substantially coincides with, for example, the oscillation center point 44 of the reflecting mirror 21 (for example, the vertex of a free-form surface).

[0042] As shown in FIG. 5(a) and FIG. 5(b), even if the reflecting mirror 21 moves in a swinging manner, the distance L between the lens 23 and the condensing point 72 of the light beam 50a is set to be almost the same as the distance L in FIG. 2. This is because, when the position of the condensing point 72 changes due to the swinging of the reflecting mirror 21 following the movement of the pupil 62, the light beam 50a after the condensing point 72 is diffused light, so that the spot diameter when reflected by the projection mirror 22 changes. As a result, the spot diameter of the light beam 50a projected on the retina 61 changes, and the image becomes out of focus depending on the viewing position. In this way, even if the position at which the multiple light beams 50 enter the lens 23 changes, the distance L between the lens 23 and the condensing point 72 can be made to remain unchanged by using a lens 23 having image height characteristics within an applicable range. In addition, the lens 23 must have an effective diameter sufficient to cover the projection range of the light beam 50 caused by the swinging of the reflecting mirror 21.

[0043] Moreover, even when the reflecting mirror 21 moves in a swinging manner, the ratio (L2 / L1) of the optical path length L2 of the light ray 50a between the reflecting mirror 21 and the convergence point 71 to the optical path length L1 of the light ray 50a between the scanning unit 13 and the reflecting mirror 21, and the ratio (L3 / L4) of the optical path length L3 of the light ray 50a between the convergence point 71 and the projection mirror 22 to the optical path length L4 of the light ray 50a between the projection mirror 22 and the convergence point 70 are almost the same. Moreover, even when the reflecting mirror 21 moves in a swinging manner, the convergence angle at which the multiple light rays 50 converge to the convergence point 70 remains almost the same.

[0044] 5(a) and 5(b) show an example in which the pupil 62 moves in the left-right direction (i.e., the eye 60 rotates in the left-right direction), but the same applies when the pupil 62 moves in other directions such as up-down. At least the light ray 50a of the multiple light rays 50 enters the pupil 62 from the opposite side to the direction in which the pupil 62 has moved, based on the direction T in which the pupil 62 faces, and is projected onto a region of the retina 61 in the direction in which the pupil 62 has moved.

[0045] 6(a) to 6(c) are diagrams for explaining the movement of the pupil 62 and the light ray 50 projected onto the retina 61. As shown in FIG. 6(a), when the pupil 62 faces the front with respect to the user's face, the light ray 50a among the multiple light rays 50a, 50b, and 50c enters the pupil 62 from almost the front. The light rays 50a, 50b, and 50c pass almost the center of the pupil 62 and are projected onto the retina 61. The light ray 50a corresponding to the pixel P1 at the center of the image projected onto the retina 61 is projected onto the position Q of the retina 61. The light ray 50b corresponding to the pixel P2 is projected onto the left side of the position Q, and the light ray 50c corresponding to the pixel P3 is projected onto the right side of the position Q. Note that the image viewed by the user is an inverted image of the image projected onto the retina 61. Thus, the user views an image in which the pixel P1 is located at the center, the pixel P2 is located on the right side, and the pixel P3 is located on the left side. In this way, the coordinate system as perceived by the user corresponds to the coordinate system in which the user views the projection mirror 22 .

[0046] As shown in FIG. 6(b), when the user moves the pupil 62 to the right relative to the user's face in order to view pixel P2 of the image projected onto the retina 61, by moving the reflecting mirror 21 as described above, at least the light rays 50a and 50c enter the pupil 62 from the left side opposite to the direction in which the pupil 62 faces. Even in this case, the light rays 50a, 50b, and 50c pass through approximately the center of the pupil 62. The light rays 50a and 50c are projected onto the right side region of the retina 61. The light ray 50b corresponding to pixel P2 of the image projected onto the retina 61 is projected onto position Q of the retina 61. Therefore, the position of the user's gaze with respect to the image projected onto the retina 61 becomes pixel P2, and the user can view an image with the pixel P2 he or she wanted to view positioned at the center.

[0047] As shown in FIG. 6(c), when the user moves the pupil 62 to the left relative to the user's face in order to view pixel P3 of the image projected onto the retina 61, by moving the reflecting mirror 21 as described above, at least the light rays 50a and 50b enter the pupil 62 from the right side, opposite to the direction in which the pupil 62 faces. Even in this case, the light rays 50a, 50b, and 50c pass through approximately the center of the pupil 62. The light rays 50a and 50b are projected onto the left region of the retina 61. The light ray 50c corresponding to pixel P3 of the image projected onto the retina 61 is projected onto position Q of the retina 61. Therefore, the position of the user's gaze with respect to the image projected onto the retina 61 becomes pixel P3, and the user can view an image with the pixel P3 he or she wanted to view positioned at the center.

[0048] As shown in FIG. 2, FIG. 5(a), and FIG. 5(b), by moving the reflecting mirror 21, the position of the convergence point 70 where the multiple light rays 50 converge moves. As a result, even if the user changes the line of sight, the multiple light rays 50 are incident on the pupil 62. Therefore, if the position of the pupil 62 is specified and the reflecting mirror 21 is moved so that the convergence point 70 is located at the specified position of the pupil 62, the multiple light rays 50 can be continuously incident on the pupil 62. In order to realize this, control information that associates multiple points in the captured image captured by the imaging unit 40 with a drive current that drives the driving unit 41 to position the convergence point 70 at these multiple points is acquired in advance. Then, the driving unit 41 is controlled based on the position of the pupil 62 shown in the captured image 80 captured by the imaging unit 40 and the control information to move the reflecting mirror 21. As a result, the convergence point 70 can be located at the pupil 62. The control information is stored in advance in the storage unit 42, for example.

[0049] Table 1 is an example of the control information stored in the storage unit 42. Table 1 is an example in which the horizontal viewing angle of the image projected on the retina 61 is 28° and the vertical viewing angle is 14°. As shown in Table 1, the control information describes a plurality of points in the captured image captured by the imaging unit 40 in association with the drive current value for positioning the convergence point 70 at the plurality of points. Here, the plurality of points are defined by a horizontal angle and a vertical angle. Since the driving unit 41 that moves the reflecting mirror 21 is, for example, a biaxial actuator, the drive current value for the yaw rotation (the numerical value on the left side of Table 1) and the drive current value for the pitch rotation (the numerical value on the right side of Table 1) are described, assuming that the normal direction at the apex of the concave curved surface of the reflecting mirror 21 is the roll axis, the horizontal direction of the reflecting mirror 21 is the pitch axis, and the vertical direction is the yaw axis. For example, to position the convergence point 70 at a point where the horizontal and vertical angles are defined as (10.5°, 3.5°), the driver 41 is driven with a drive current of (36.44 mA, 9.75 mA) to move the reflecting mirror 21. [Table 1]

[0050] The control information stored in the storage unit 42 may be a table other than that shown in Table 1, so long as it is information that associates multiple points in an image captured by the imaging unit 40 with drive current values ​​for moving the convergence point 70 to the multiple points. For example, an approximation such as a linear function or a quadratic function may be stored in the storage unit 42 as the control information.

[0051] 1, the drive control unit 33 controls the drive unit 41 to move the reflecting mirror 21 based on the center position 65 of the pupil 62 captured in the captured image 80 and the control information stored in the storage unit 42, so that the convergence point 70 is located at the pupil 62. For example, when the center position 65 of the pupil 62 captured in the captured image 80 is a point defined as (-7.0°, -3.5°) based on the horizontal angle and the vertical angle, the drive control unit 33 refers to Table 1 and drives the drive unit 41 with a drive current of (-11.86 mA, -7.63 mA) to move the reflecting mirror 21. This allows a plurality of light rays 50 to pass through the pupil 62, and an image can be projected onto the retina 61.

[0052] The acquisition unit 34 acquires a value related to the pupil 62 captured in the captured image 80. When the value related to the pupil 62 acquired by the acquisition unit 34 satisfies a predetermined condition, the drive control unit 33 does not control the drive unit 41 based on the center position 65 of the pupil 62 captured in the captured image 80.

[0053] 7(a) to 7(c) are diagrams for explaining an example of values ​​related to the pupil 62 acquired by the acquisition unit 34. As shown in Fig. 7(a), for example, the acquisition unit 34 acquires a size (e.g., radius) r0 of the pupil 62 captured in the captured image 80 of the latest frame captured by the imaging unit 40 and a size (e.g., radius) r -1 and obtains a value related to the change in size of the pupil 62. For example, the obtaining unit 34 may obtain the rate of change dr of the size of the pupil 62 using the following formula (1).

number

[0054] 7(b), for example, the acquisition unit 34 compares the center position 65a of the pupil 62 captured in the captured image 80 of the latest frame captured by the imaging unit 40 with the center position 65b of the pupil 62 captured in the captured image 80 of the previous frame to acquire a value related to the movement of the center position 65 of the pupil 62. For example, the acquisition unit 34 may acquire the movement rate dLc from the center position 65b to the center position 65a by using the following equations (2a) to (2c). X0 and Y0 in equations (2a) and (2b) are the X and Y coordinates of the center position 65a, and X -1 , Y -1 are the X and Y coordinates of the center position 65b. W is the number of pixels in the X direction (width direction) of the captured image 80, and H is the number of pixels in the Y direction (length direction).

number

[0055] 7(c), for example, the acquisition unit 34 acquires values ​​related to the variation of the center positions 65a, 65b, and 65c of the pupil 62 captured in the captured images 80 of the most recent frames (for example, the most recent three frames) captured by the imaging unit 40. For example, the acquisition unit 34 may acquire the variation rate dSc of the center positions 65a, 65b, and 65c of the pupil 62 by using the following formulas (3a) to (3c). X in formula (3a) ave is the average value of the X coordinates of the central positions 65a, 65b, and 65c. ave is the average value of the Y coordinates of the central positions 65a, 65b, and 65c. W is the number of pixels in the X direction (width direction) of the captured image 80, and H is the number of pixels in the Y direction (length direction).

number

[0056] The control unit 30 is a processor such as a CPU (Central Processing Unit). The projection control unit 31, the imaging control unit 32, the drive control unit 33, the acquisition unit 34, and the specification unit 35 may function by the same processor or different processors. The storage unit 42 is a non-volatile semiconductor memory such as a flash memory.

[0057] [experiment] The eyes 60 of three users A, B, and C wearing the image projection device 100 according to the first embodiment were captured by the imaging unit 40. When the users A, B, and C freely viewed the image projected by the image projection device 100, 1) the change rate dr of the size of the pupil 62, 2) the movement rate dLc of the center position 65 of the pupil 62, and 3) the variation rate dSc of the center position 65 of the pupil 62 were obtained from the pupil 62 captured in the captured image 80 of multiple frames captured by the imaging unit 40. Figures 8(a), 8(b), and 9 show the results of the change rate dr of the size of the pupil 62, the movement rate dLc of the center position 65 of the pupil 62, and the variation rate dSc of the center position 65 of the pupil 62.

[0058] As shown in Fig. 8(a), the change rate dr of the size of the pupil 62 sometimes suddenly became a large value. This is thought to be due to erroneous detection of the pupil 62. As shown in Fig. 8(b) and Fig. 9, the movement rate dLc and the variation rate dSc of the center position 65 of the pupil 62 sometimes fluctuated by small values. This is thought to be due to fixational eye movement of the eye 60 or the like, and is not due to the user intentionally moving his or her gaze.

[0059] Next, the following cases 1), 2), 3) and 4) will be described regarding the fluctuation of the center position 65 of the pupil 62 captured in the captured images 80 of a plurality of frames. 1) Raw data of the fluctuation of the center position 65 of the pupil 62 2) Data obtained by removing the cases where the movement rate dLc of the center position 65 of the pupil 62 is dLc<0.75% from the raw data of 1) above. 3) In addition to 2) above, data in which the variation rate dSc of the center position 65 of the pupil 62 is dSc<0.75% is excluded. 4) In addition to 3) above, data in which the rate of change dr of pupil 62 size was greater than 3.0% and less than -3.0% was excluded.

[0060] 10(a) to 11(b) are diagrams showing the fluctuation of the center position 65 of the pupil 62 captured in the captured image 80. FIG. 10(a) is raw data of the X and Y coordinates of the center position 65 in 1) above. FIG. 10(b) is data of the X and Y coordinates of the center position 65 obtained by removing the cases where dLc<0.75% from the raw data in 2) above. FIG. 11(a) is data of the X and Y coordinates of the center position 65 obtained by removing the cases where dSc<0.75% as well as the cases where dLc<0.75% from the raw data in 3) above. FIG. 11(b) is data of the X and Y coordinates of the center position 65 obtained by removing the cases where dr>3.0% and dr<-3.0% as well as the cases where dLc<0.75% and dSc<0.75% from the raw data in 4) above.

[0061] As shown in Fig. 10(a) and Fig. 10(b), by removing the cases where dLc<0.75% from the raw data, minute fluctuations in the X and Y coordinates are removed. In other words, fixation eye movements and measurement errors are removed. As shown in Fig. 11(a), by removing the cases where dSc<0.75% in addition to dLC<0.75%, minute fluctuations in the X and Y coordinates are further removed. As shown in Fig. 11(b), by removing the cases where dr>3.0% and dr<-3.0%, sudden large fluctuations in the X and Y coordinates are removed. In other words, cases where the pupil 62 is erroneously detected are removed.

[0062] [Manufacturing method] Fig. 12 is a flowchart showing a manufacturing method of the image projection device 100 according to the first embodiment. As shown in Fig. 12, first, the image projection device 100 is prepared (step S10). As shown in Fig. 1, the image projection device 100 includes a projection unit 10 including a light source 11, a scanning unit 13, and a projection optical system 20, an imaging unit 40, and a driving unit 41. Next, the light source 11 is driven to emit a light beam 50 from the image projection device 100 to the outside (step S12). The light beam 50 emitted from the image projection device 100 converges at a convergence point 70 as shown in Fig. 2.

[0063] Next, the driving unit 41 is controlled to move the reflecting mirror 21, the convergence point 70 is moved to a plurality of positions, and the light beam 50 at each convergence point 70 is imaged by the imaging unit 40 (step S14). FIG. 13(a) to FIG. 13(c) are diagrams showing an example of imaging the light beam 50 at the convergence point 70 by the imaging unit 40. In FIG. 13(a) to FIG. 13(c), the position of the convergence point 70 is different because the driving unit 41 has moved the reflecting mirror 21. As shown in FIG. 13(a) to FIG. 13(c), a screen 85 is disposed at the position of the convergence point 70, and the light beam 50 irradiated to the screen 85 is imaged by the imaging unit 40. Even if the position of the convergence point 70 is moved, the screen 85 is not moved. Even if the position of the convergence point 70 is moved by moving the reflecting mirror 21, the position of the convergence point 70 in the traveling direction of the light beam 50 hardly changes.

[0064] 12, from the captured image captured in step S14, control information (for example, Table 1 above) is acquired that associates a plurality of points in the captured image captured by the imaging unit 40 with the drive current value that controls the driver 41 to move the reflecting mirror 21 when the convergence point 70 is located at each of the plurality of points (step S16). Since the convergence point 70 is to be located near the pupil 62 as shown in FIG. 2, the convergence point 70 is moved to a plurality of positions as in steps S14 and S16, and control information such as Table 1 is acquired from the captured image captured by the imaging unit 40 of the light beam 50 at each convergence point 70, so that the convergence point 70 can be aligned with the position of the pupil 62 using the position of the pupil 62 depicted in the captured image 80 captured by the imaging unit 40 and this control information.

[0065] Next, the control information acquired in step S16 is stored in the storage unit 42 (step S18). This control information can be configured to be acquired and stored individually for each manufactured device when the image projection device 100 is manufactured. When the image projection device 100 is manufactured, when the reflecting mirror 21, the projection mirror 22, etc. are installed, there is a possibility that the installation positions and directions of the mirrors cannot be completely identical for each device. In that case, the light beam 50 will be slightly shifted from the center of the pupil 62. If the control information is acquired and stored for each device, it is possible to absorb the difference in the installation state of the parts used in the manufacture of each device and to standardize the performance between the devices.

[0066] [Control method] 14 and 15 are flowcharts showing an example of a control method of the control unit 30 in the first embodiment. As shown in Fig. 14 and 15, when image data is input, the projection control unit 31 controls the light source 11 and the scanning unit 13 to start projecting an image onto the user's retina 61 (step S20). In synchronization with the start of the image projection, the imaging control unit 32 controls the imaging unit 40 to start capturing an image of the user's eye 60 (step S22). Note that the imaging control unit 32 may cause the imaging unit 40 to start capturing an image in response to an instruction from the user.

[0067] Next, the identification unit 35 acquires a captured image 80 of the user's eye 60 from the imaging unit 40 (step S24). For example, the captured image 80 shown in FIG. 3 is acquired. Next, the identification unit 35 uses a template image 82 stored in advance in the storage unit 42 to extract a region 83 with high similarity from the captured image 80 (step S26). For example, the template image 82 shown in FIG. 4(a) is used to extract a region 83 with high similarity as shown in FIG. 4(b). Template matching uses a generally known method. Note that the template image 82 is not limited to being used in a single image, and two or more images may be used. In this case, a region 83 with high similarity may be identified in each of the multiple template images 82, and the region 83 with the highest similarity may be selected from among them. There are various possible cases of the size of the user's pupil 62 and the color of the iris 64, and a case where matching is not possible when only one template image 82 is used may occur. However, using multiple images suppresses the inability to match.

[0068] Next, the identification unit 35 determines whether the similarity between the template image 82 and the image of the extracted region 83 is equal to or greater than a threshold (step S28). If it is less than the threshold (No), it is possible that the image of the region 83 is something other than the eye 60, so steps S30 to S48 are skipped and the process proceeds to step S50. On the other hand, if it is equal to or greater than the threshold (Yes), the process proceeds to step S30.

[0069] In step S30, as shown in Figures 4(c) and 4(d), the identification unit 35 performs a smoothing process and adjusts the gradation value on the image of the extracted region 83 to make it easier to obtain the contour, and then identifies the pupil 62 by detecting the contour of the image of the region 83 (step S30). Next, the identification unit 35 detects the position coordinates of the center position 65 of the identified pupil 62 and the size of the pupil 62, and stores them in the storage unit 42 (step S32). For example, the identification unit 35 stores the X and Y coordinates of the center position 65 and the diameter or radius of the pupil 62 in the storage unit 42.

[0070] Next, the acquisition unit 34 determines whether the pupil 62 identified by the identification unit 35 in step S30 is that of the captured image 80 of the first frame since the start of image projection (step S34). If it is that of the captured image 80 of the first frame (Yes), the process proceeds to step S50. On the other hand, if it is that of the captured image 80 of the second or subsequent frame (No), the process proceeds to step S36.

[0071] In step S36, the acquisition unit 34 acquires a value related to the movement of the center position 65 of the pupil 62, which indicates how much the center position 65 of the pupil 62 detected in step S32 has moved from the captured image 80 of the previous frame (step S36). For example, the acquisition unit 34 acquires the movement rate dLc of the center position 65 of the pupil 62 as the value related to the movement of the center position 65 of the pupil 62, using the above-mentioned formulas (2a) to (2c).

[0072] Next, the acquisition unit 34 acquires values ​​related to the variation of the center position 65 of the pupil 62 in the captured images 80 of the most recent frames (e.g., the most recent three frames) including the center position 65 of the pupil 62 detected in step S32 (step S38). For example, the acquisition unit 34 acquires the variation rate dSc of the center position 65 of the pupil 62 using the above-mentioned formulas (3a) to (3c) as the value related to the variation of the center position 65 of the pupil 62.

[0073] Next, the acquisition unit 34 acquires a value related to the change in size of the pupil 62, which indicates how much the size of the pupil 62 detected in step S32 has changed from the captured image 80 of the previous frame (step S40). For example, the acquisition unit 34 acquires the rate of change dr of the size of the pupil 62 using the above-mentioned formula (1) as the value related to the change in size of the pupil 62.

[0074] Next, the drive control unit 33 judges whether or not the value (e.g., the movement rate dLc) related to the movement of the center position 65 of the pupil 62 acquired in step S36 is equal to or greater than a threshold (step S42). If it is less than the threshold (No), it is considered that the user has not intentionally moved his / her line of sight as described in FIG. 7(b), but that the movement is due to fixational micromovement of the eye. In this case, if the position of the convergence point 70 is adjusted based on the center position 65 of the pupil 62 detected in step S32, the image projected on the retina 61 may move even though the user has not intended to move his / her line of sight, which may cause discomfort. Therefore, if it is less than the threshold (No), the position of the convergence point 70 is not adjusted and the process proceeds to step S50. On the other hand, if it is equal to or greater than the threshold (Yes), it is considered that the user has intentionally moved his / her line of sight, so the process proceeds to step S44.

[0075] In step S44, the drive control unit 33 judges whether or not the value (e.g., the variation rate dSc) relating to the variation of the center position 65 of the pupil 62 acquired in step S38 is equal to or greater than a threshold (step S44). If it is less than the threshold (No), it is considered that the user has not intentionally moved his / her line of sight as described in FIG. 7(c), but that the movement is due to fixational micromovement of the eye. In this case, as described above, if the position of the convergence point 70 is adjusted based on the center position 65 of the pupil 62 detected in step S32, the image projected on the retina 61 moves, which may cause the user to feel uncomfortable. Therefore, if it is less than the threshold (No), the position of the convergence point 70 is not adjusted and the process proceeds to step S50. On the other hand, if it is equal to or greater than the threshold (Yes), it is considered that the user has intentionally moved his / her line of sight, and the process proceeds to step S46.

[0076] Here, the reason why it is preferable to perform both the judgments in step S42 and step S44, that is, judgments on both the value related to the movement of the center position 65 of the pupil 62 and the value related to the variation, will be described with reference to Fig. 7(c). In Fig. 7(c), the captured image 80 on the left shows the captured image of the frame two frames before, the captured image 80 in the middle shows the captured image of the frame one frame before, and the captured image 80 on the right shows the captured image of the latest frame. As shown in Fig. 7(c), in the captured image 80 of the frame one frame before (center figure), the pupil 62 has moved to the left compared to the captured image 80 of the frame two frames before (left figure), but the value related to the movement of the center position 65 of the pupil 62 is less than a threshold value, and it is assumed that this is due to fixational micromovement of the eye 60. In the captured image 80 (right) of the latest frame, the pupil 62 has moved to the right compared to the captured image 80 (left) of the frame two frames before, and the value related to the movement of the center position 65 of the pupil 62 from the captured image 80 (center) of the frame one frame before is equal to or greater than the threshold. In this case, the determination of the value related to the movement of the center position 65 of the pupil 62 in step S42 is determined as Yes, and the user intentionally moves his / her gaze. However, when compared to the captured image 80 (left) of the frame two frames before, the value related to the movement of the center position 65 of the pupil 62 may be less than the threshold. In other words, it may be within the range of fixational micro-movement of the eye 60. Therefore, in such a case, it is preferable to determine both the value related to the movement of the center position 65 of the pupil 62 and the value related to the variation so as not to adjust the position of the convergence point 70.

[0077] Returning to FIG. 14 and FIG. 15, in step S46, the drive control unit 33 judges whether the absolute value of the value (e.g., the rate of change dr) regarding the change in the size of the pupil 62 is equal to or less than a threshold (step S46). If it is greater than the threshold (No), erroneous detection of the pupil 62 is possible as described in FIG. 7(a). In this case, if the position of the convergence point 70 is adjusted based on the center position 65 of the pupil 62 detected in step S32, the convergence point 70 is adjusted to an incorrect position, and it may occur that an image that gives the user a sense of discomfort is provided or an image cannot be provided to the user. Therefore, if it is greater than the threshold (No), the position of the convergence point 70 is not adjusted and the process proceeds to step S50. On the other hand, if it is equal to or less than the threshold (Yes), the possibility of erroneous detection of the pupil 62 is low, so the process proceeds to step S48.

[0078] In step S48, the drive control unit 33 applies a drive current to the drive unit 41 based on the center position 65 of the pupil 62 detected in step S32 and the control information (for example, Table 1) stored in the storage unit 42 to move the reflecting mirror 21 and adjust the convergence point 70 to be located at the pupil 62 (step S48). For example, when the control information shown in Table 1 is stored in the storage unit 42, the drive control unit 33 specifies a drive current corresponding to the center position 65 of the pupil 62 detected in step S32 from the control information in Table 1, and applies the specified drive current to the drive unit 41 to move the reflecting mirror 21. This makes it possible to keep the multiple light rays 50 incident on the pupil 62 and keep projecting an image on the retina 61 even if the user changes the line of sight. Note that when the reflecting mirror 21 is a concave mirror and has a vertex of the concave curved surface at the approximate center, the reflecting mirror 21 is moved so as to swing around the vertex. The projection control unit 31 controls the light source 11 and the scanning unit 13 so that the light ray 50 a corresponding to the center of the image projected onto the retina 61 is incident on the reflection mirror 21 near its swing center.

[0079] Next, the drive control unit 33 waits for a certain period of time (step S50). The certain period of time may be shorter than and equal to or greater than half the period of one frame among the multiple frames captured by the imaging unit 40. The reason for waiting for the certain period of time is to reduce the influence of high-frequency components such as fixational eye movement.

[0080] Next, the projection control unit 31 judges whether or not to continue projecting the image onto the retina 61 (step S52). If the image is to be continued to be projected (No), the process returns to step S24 and steps S24 to S50 are repeated. On the other hand, if the image projection is to be ended (Yes), the process ends.

[0081] Here, the application of the drive current to the drive unit 41 in step S48 will be described. FIG. 16 is a diagram showing the application of the drive current by the drive control unit 33. The solid line in FIG. 16 is the trajectory of the drive current application when the current control mode is ON, and the dotted line is the trajectory when it is OFF. As shown in FIG. 16, when the current control mode is OFF (dotted line), the drive current increases to a desired value as quickly as possible, but when the current control mode is ON (solid line), the drive current is increased over 80% or more of the time of one frame (for example, about 0.08 s) in multiple frames captured by the imaging unit 40. For example, when the drive control unit 33 specifies the drive current to be applied from the captured image 80 of frame A as 30 mA and increases the drive current from 0 mA to 30 mA, when the current control mode is ON (solid line), the drive current is increased to 30 mA over 80% or more (for example, 100%) of the time of one frame. Then, after waiting for a certain period of time (step S50), if the drive current to be applied from the captured image 80 of the next frame B is determined to be 50 mA, the drive current is increased from 30 mA to 50 mA over a period of 80% or more (e.g., 100%) of one frame time.

[0082] By turning on the current control mode and slowly changing the drive current over 80% or more of the time for one frame, the image can be moved smoothly in response to the movement of the user's line of sight.

[0083] As described above, according to the first embodiment, the control unit 30 includes the drive control unit 33 and the acquisition unit 34 as shown in FIG. 1. The acquisition unit 34 acquires values ​​related to the pupil 62 captured in the captured image 80 captured by the imaging unit 40 as shown in steps S36 to S40 of FIG. 14 and FIG. 15. The drive control unit 33 controls the drive unit 41 based on the position of the pupil 62 captured in the captured image 80 so that the multiple light rays 50 are projected onto the retina 61 as shown in steps S42 to S48 of FIG. 15, but does not control the drive unit 41 based on the position of the pupil 62 when the value related to the pupil 62 acquired by the acquisition unit 34 satisfies a predetermined condition. This allows the image to continue to be projected onto the retina 61 even when the user changes the line of sight, and the drive unit 41 is not controlled when the user does not intentionally move the line of sight or when there is an error in the detection of the pupil 62 captured in the captured image 80, so that it is possible to prevent an image that gives the user a sense of discomfort from being provided.

[0084] In addition, in the first embodiment, as in step S40 in FIG. 15, the acquisition unit 34 acquires a value related to the change in size of the pupil 62 captured in the captured image 80. As in step S46 in FIG. 15, when the absolute value of the value related to the change in size of the pupil 62 is greater than the threshold, the drive control unit 33 does not control the drive unit 41 based on the center position 65 of the pupil 62. When the absolute value of the value related to the change in size of the pupil 62 is greater than the threshold, it is considered that the pupil 62 has been erroneously detected. For this reason, if the drive unit 41 is controlled based on the position of the pupil 62, an image that gives the user a sense of discomfort may be provided. Therefore, in such a case, by not controlling the drive unit 41 based on the center position 65 of the pupil 62, it is possible to prevent an image that gives the user a sense of discomfort from being provided.

[0085] In the first embodiment, the case where the rate of change dr of the size of the pupil 62 is used as the value related to the change in the size of the pupil 62 is shown as an example. When the rate of change dr is used, the threshold value may be 8.0%, 5.0%, or 3.0%, taking into consideration the results of Fig. 8(a) and Fig. 11(b). Also, a value other than the rate of change dr may be used as the value related to the change in the size of the pupil 62.

[0086] In addition, in the first embodiment, as in step S36 of FIG. 14, the acquisition unit 34 acquires a value related to the movement of the center position 65 of the pupil 62 captured in the captured image 80. As in step S42 of FIG. 15, when the value related to the movement of the center position 65 of the pupil 62 is less than the threshold, the drive control unit 33 does not control the drive unit 41 based on the center position 65 of the pupil 62. When the value related to the movement of the center position 65 of the pupil 62 is less than the threshold, it is considered that the user has not intentionally moved his / her line of sight. For this reason, if the drive unit 41 is controlled based on the center position 65 of the pupil 62, an image that gives the user a sense of discomfort may be provided. Therefore, in such a case, by not controlling the drive unit 41 based on the center position 65 of the pupil 62, it is possible to prevent an image that gives the user a sense of discomfort from being provided.

[0087] In the first embodiment, the movement rate dLc of the center position 65 is used as a value related to the movement of the center position 65 of the pupil 62. When the movement rate dLc is used, the threshold value may be 3.0%, 1.5%, or 0.75%, taking into consideration the results of Fig. 8(b) and Fig. 10(b). Furthermore, a value other than the movement rate dLc may be used as a value related to the movement of the center position 65 of the pupil 62. A value related to movement at a position other than the center position 65 of the pupil 62 may be used.

[0088] In addition, in the first embodiment, as in step S38 of FIG. 15, the acquisition unit 34 acquires a value related to the variation of the center position 65 of the pupil 62 captured in the captured image 80. As in step S44 of FIG. 15, when the value related to the variation of the center position 65 of the pupil 62 is less than the threshold, the drive control unit 33 does not control the drive unit 41 based on the center position 65 of the pupil 62. When the value related to the variation of the center position 65 of the pupil 62 is less than the threshold, it is considered that the user does not intentionally move his / her gaze. For this reason, if the drive unit 41 is controlled based on the center position 65 of the pupil 62, an image that gives the user a sense of discomfort may be provided. Therefore, in such a case, by not controlling the drive unit 41 based on the center position 65 of the pupil 62, it is possible to prevent the user from receiving an image that gives the user a sense of discomfort.

[0089] In the first embodiment, the case where the variation rate dSc of the central position 65 is used as the value related to the variation of the central position 65 of the pupil 62 is shown as an example. When the variation rate dSc is used, the threshold value may be 3.0%, 1.5%, or 0.75%, taking into consideration the results of Fig. 9 and Fig. 11(a). Furthermore, a value other than the variation rate dSc may be used as the value related to the variation of the central position 65 of the pupil 62. A value related to the variation at a position other than the central position 65 of the pupil 62 may be used.

[0090] In the first embodiment, an example is shown in which all of the values ​​relating to the change in size of the pupil 62, the value relating to the movement of the central position 65 of the pupil 62, and the value relating to the variation in the central position 65 of the pupil 62 are obtained and compared with a threshold value, but it is also possible to obtain at least one value and compare it with a threshold value.

[0091] In the first embodiment, as shown in Fig. 16, the drive control unit 33 changes the drive current applied to the drive unit 41 to a desired drive current over 80% or more of the time of one frame among multiple frames captured by the imaging unit 40. In this way, by gradually changing the drive current applied to the drive unit 41, the image can be moved smoothly in response to the movement of the user's line of sight, and an image that gives the user little discomfort can be provided. From the viewpoint of providing an image that gives little discomfort, it is preferable to change the drive current over 90% or more of the time of one frame, more preferably over 95% or more, and even more preferably over 98% or more.

[0092] In the first embodiment, the drive control unit 33 controls the drive unit 41 based on control information that associates the position of the pupil 62 captured in the captured image 80, a plurality of points in the captured image captured by the imaging unit 40, and the drive current when the convergence point 70 is positioned at the plurality of points. As a result, even if the user changes the line of sight, a simple method using the captured image 80 of the eye 60 captured by the imaging unit 40 can cause the plurality of light rays 50 to continue to be incident on the pupil 62 and an image can continue to be projected onto the retina 61.

[0093] In the first embodiment, the control information shown in Table 1 is stored in the storage unit 42, but it may be stored in an external device other than the image projection device 100. In this case, the drive control unit 33 acquires the control information from the external device.

[0094] In the first embodiment, the projection optical system 20 includes a projection mirror 22 (projection unit) that converges the multiple light rays 50 to a convergence point 70, a reflection mirror 21 (reflection unit) that converges the multiple light rays 50 to a convergence point 71 in front of the projection mirror 22, and a lens 23 (optical member) that converts each of the multiple light rays 50 into convergent light, focuses the light at a convergence point 72 in front of the projection mirror 22, and then causes the light rays to enter the projection mirror 22 as diffused light. The drive unit 41 moves the reflection mirror 21 to move the position of the convergence point 70. This makes it possible to move the position of the convergence point 70 while suppressing the influence on the light rays 50, as compared to, for example, moving the scanning unit 13 to move the position of the convergence point 70. Note that, as long as it is possible to move the position of the convergence point 70, a component of the projection unit 10 other than the component constituting the projection optical system 20 may be moved.

[0095] In the first embodiment, the swing center point 44 of the reflecting mirror 21 is located near the apex of the concave surface at the center of the reflecting mirror 21. However, depending on the swing mechanism of the reflecting mirror 21 by the driving unit 41, the swing center point 44 may not necessarily be located on the reflecting mirror 21 but may be located at an offset position. For example, depending on the driving mechanism of the driving unit 41, the swing center point 44 may be located at an offset position in the direction of the driving unit 41 (the opposite direction to the projection mirror 22), and the reflecting mirror 21 may swing around the swing center point 44. The swing center point 44 in the offset state is preferably 5 mm or less away from the swing center point 44 on the reflecting mirror 21. This distance is a distance that may occur when the components used in the first embodiment are arranged inside the eyeglass-type frame of the head-mounted display. If this distance is large, it is visually recognized by the user as a distortion of the image, but if it is 5 mm or less, there is almost no effect. The swing center point 44 is located near the center of the reflecting mirror 21 within this distance range.

[0096] In the first embodiment, the reflection mirror 21 and the projection mirror 22 are not limited to being concave mirrors, and may be other optical components such as a combination of lenses and mirrors, a diffraction element, or the like, other than curved mirrors, as long as they have positive light-collecting power. The lens 23 is not limited to being a convex lens, and may be other optical members such as a mirror or a diffraction element, as long as it is possible to cause the light beam 50 to be incident on the projection mirror 22 as diffused light. The lens 23 may have a function of suppressing chromatic aberration.

[0097] In the first embodiment, the convergence angle at which the multiple light beams 50 converge to the convergence point 70 is preferably equal to or larger than the scanning angle of the scanning unit 13, both before and after the reflection mirror 21 moves. This makes it possible to keep the viewing angle of the image projected onto the retina 61 large. EXAMPLES

[0098] In the second embodiment, an example of outputting a warning when the user's eye is displaced from a predetermined position with respect to the image projection device when the user wears the image projection device will be described. FIG. 17 is a diagram showing an image projection device 200 according to the second embodiment. As shown in FIG. 17, in the second embodiment, the control unit 30 includes a warning control unit 36 ​​in addition to the projection control unit 31, the image capture control unit 32, the drive control unit 33, the acquisition unit 34, and the identification unit 35. When the user wears the image projection device 200, the warning control unit 36 ​​acquires a captured image 80 of the eye 60 from the image capture unit 40, and when the center position 65 of the pupil 62 is out of a predetermined range, causes the warning output unit 43 included in the image projection device 200 to output a warning. The warning output unit 43 outputs, for example, a warning sound. The warning control unit 36 ​​may change the warning output from the warning output unit 43 according to the amount of displacement of the center position 65 of the pupil 62 from the predetermined range, and may change the volume and / or interval of the warning sound according to the amount of displacement.

[0099] [Control method] Fig. 18 is a flowchart showing an example of a method for controlling a warning by the control unit 30 in the second embodiment. As shown in Fig. 18, when a user wears the image projection device 200, the imaging control unit 32 controls the imaging unit 40 to start imaging the user's eye 60 (step S60). The imaging control unit 32 may control the imaging unit 40 to start imaging in response to an instruction from the user.

[0100] Next, the warning control unit 36 ​​acquires a captured image 80 of the user's eye 60 from the imaging unit 40 (step S62). Next, the warning control unit 36 ​​identifies the center position 65 of the pupil 62 depicted in the captured image 80 (step S64). The center position 65 of the pupil 62 may be identified using a method similar to the method described in steps S26 to S30 of FIG.

[0101] Next, the warning control unit 36 ​​determines whether or not the center position 65 of the pupil 62 identified in step S64 is located within a predetermined range in the captured image 80 (step S66). If it is located within the predetermined range (Yes), the user's eye 60 is in an appropriate position relative to the image projection device 200, and therefore this process ends.

[0102] On the other hand, if it is out of the predetermined range (No), the warning control unit 36 ​​calculates the amount of deviation of the center position 65 of the pupil 62 from the predetermined range (step S68). Next, the warning control unit 36 ​​causes the warning output unit 43 to output a warning according to the amount of deviation of the center position 65 of the pupil 62 (step S70). For example, when the amount of deviation is large, the warning control unit 36 ​​increases the volume of the warning sound output by the warning output unit 43 or shortens the interval of the warning sound compared to when the amount of deviation is small. Thereafter, steps S62 to S70 are repeated until it is determined Yes in step S66. Thereby, as shown in Figs. 19(a) to 19(c), the user can align the center position 65 of the pupil 62 with the predetermined range 81 in the captured image 80 using the warning sound output by the warning output unit 43 as a guide. Therefore, the eye 60 can be brought to an appropriate position.

[0103] As described above, according to the second embodiment, when the user wears the image projection device 200, the warning control unit 36 ​​outputs a warning if the center position 65 of the pupil 62 captured in the captured image 80 is out of the predetermined range 81, and changes the warning according to the amount of deviation from the predetermined range 81. It is difficult for a user wearing the image projection device 200 to determine whether or not their eye 60 is deviated from the initial position, but the provision of such a warning control unit 36 ​​allows the user to easily align the eye 60 when it is deviated from the initial position.

[0104] In the first and second embodiments, the image projection device may be attached to a glasses-type frame, or may be attached to a goggle-type, eye patch-type, ear hook-type, helmet-mounted type, or other frame.

[0105] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0106] 10...projection unit, 11...light source, 12...lens, 13...scanning unit, 20...projection optical system, 21...reflection mirror, 22...projection mirror, 23...lens, 30...control unit, 31...projection control unit, 32...imaging control unit, 33...drive control unit, 34...acquisition unit, 35...identification unit, 36...warning control unit, 40...imaging unit, 41...drive unit, 42...storage unit, 43...warning output unit, 44...oscillation center point, 5 0, 50a, 50b, 50c...light ray, 60...eye, 61...retina, 62...pupil, 63...crystalline lens, 64...iris, 65, 65a, 65b, 65c...center position, 70...convergence point (first convergence point), 71...convergence point (second convergence point), 72...light collection point, 80...captured image, 81...predetermined range, 82...template image, 83...area, 85...screen, 100, 200...image projection device

Claims

1. a projection unit including a light source, a scanning unit that scans a light beam emitted from the light source, and a projection optical system that causes a plurality of light beams emitted in different directions from the scanning unit to converge at a first convergence point in a user's eye and then projects the light beams onto a retina to project an image onto the retina; a drive unit that moves a component included in the projection unit capable of moving a position of the first convergence point; An imaging unit that images an eye of the user; an acquisition unit that acquires a value related to a pupil captured in an image captured by the imaging unit; an image projection device comprising: a drive control unit that controls the drive unit based on the position of the pupil captured in the captured image so that the multiple light rays are projected onto the retina, and does not control the drive unit based on the position of the pupil if a value related to the pupil acquired by the acquisition unit satisfies a predetermined condition.

2. The acquisition unit acquires a value related to a change in size of the pupil captured in the captured image, The image projection device according to claim 1 , wherein the drive control unit does not control the drive unit based on the position of the pupil when an absolute value of the value relating to the change is greater than a threshold value.

3. The acquisition unit acquires a value related to a movement of a position of the pupil captured in the captured image, The image projection device according to claim 1 , wherein the drive control unit does not control the drive unit based on the position of the pupil when the value related to the movement is less than a threshold value.

4. The acquisition unit acquires a value related to a variation in the position of the pupil captured in the captured image, The image projection device according to claim 1 , wherein the drive control unit does not control the drive unit based on the position of the pupil when the value relating to the variation is less than a threshold value.

5. 5. The image projection device according to claim 1, wherein the drive control unit changes the drive current applied to the drive unit to a desired drive current over a period of 80% or more of a frame time in a plurality of frames captured by the imaging unit.

6. 5. The image projection device of claim 1, wherein the drive control unit controls the drive unit based on control information that associates the position of the pupil captured in the captured image, a plurality of points in the image captured by the imaging unit, and a drive current for positioning the first convergence point at the plurality of points.

7. The image projection device according to claim 6 , further comprising a storage unit that stores the control information.

8. the projection optical system includes a projection unit disposed in front of an eye of the user and converging the plurality of light beams to the first convergence point, a reflection unit disposed in front of the scanning unit and converging the plurality of light beams to a second convergence point in front of the projection unit, and an optical member disposed between the projection unit and the reflection unit and converting each of the plurality of light beams into converging light, converging the light beams at a converging point in front of the projection unit, and then causing the light beams to enter the projection unit as diffused light; The image projection device according to claim 1 , wherein the component is the reflecting portion.

9. 5. The image projection device according to claim 1, further comprising a warning control unit that outputs a warning if the position of the pupil captured in the captured image is outside a predetermined range when the user wears the image projection device, and changes the warning depending on the amount of deviation from the predetermined range.

Images