Image projection device

Abstract

This image projection device 100 is provided with: a projection unit 10 including a light source 11, a scanning unit 13 that scans with a light beam 50 emitted from the light source 11, and a projection optical system 20 that projects an image onto a retina 61 by converging a plurality of light beams 50 emitted in different directions from the scanning unit 13 by the scanning of the scanning unit 13 to a convergent point 70 in an eye 60 of a user and thereafter projecting the converged plurality of light beams onto the retina 61; an imaging unit 40 that captures an image of the eye 60 of the user; a generation unit 34 that generates a superimposed image 82 obtained by superimposing a captured image 80 of the eye 60 of the user captured by the imaging unit 40 and a reference image 81 in which an initial position 83 to be aligned with a pupil 62 of the user is displayed; and a projection control unit 31 that controls the projection unit 10 to project the superimposed image 82 onto the retina 61.

Description

Image Projection Device

[0001] The present invention relates to an image projection device.

[0002] Image projection devices utilizing Maxwellian vision are known, which project an image directly onto a user's retina using light rays emitted from a light source. In Maxwellian vision, a plurality of light rays forming an image are converged inside the user's eye and then irradiated onto the retina, thereby projecting the image onto the retina. In image projection devices using Maxwellian vision, an image cannot be projected onto the retina unless the light rays pass through the pupil. Therefore, a method has been proposed in which the light rays continue to pass through the pupil even when the pupil moves (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2023-76137

[0004] When a user wears an image projection device, it is desirable for the pupil to be in a predetermined position corresponding to the convergence point where multiple light rays converge inside the eye. However, the pupil may deviate from this predetermined position. To align the pupil to the predetermined position, one possible method is to play a sound whose volume corresponds to the amount of deviation from the predetermined position. However, adjusting the position is difficult because the direction of deviation cannot be determined from the volume of the sound alone. Another possible method is to use an audio guide to make announcements, but this method can cause delays in the audio guide as the pupil position changes during the announcement.

[0005] The present invention has been made in consideration of the above-mentioned problems, and has an object to make it possible to easily align the position of the pupil with a predetermined position.

[0006] The present invention is an image projection device comprising: a projection unit including a scanning unit that scans light rays emitted from a light source; and a projection optical system that, as a result of being scanned by the scanning unit, causes a plurality of light rays emitted in different directions from the scanning unit to converge at a convergence point within the user's eye and then projects the light rays onto the retina, thereby projecting an image onto the retina; an imaging unit that images the user's eye; a generation unit that generates a superimposed image by superimposing the image of the user's eye captured by the imaging unit and a reference image that displays the position at which the user's pupil should be aligned; and a projection control unit that controls the projection unit to project the superimposed image onto the retina.

[0007] In the above configuration, the generation unit generates a first guidance sign that guides the user so that the center position of the user's pupil coincides with the position in the reference image, based on the center position of the user's pupil captured in the captured image and the position in the reference image, and the projection control unit can be configured to control the projection unit to project the first guidance sign onto the retina.

[0008] In the above configuration, the first guidance marker may be an arrow.

[0009] In the above configuration, the projection unit and the image capture unit can be attached to a frame that is worn by the user.

[0010] In the above configuration, the projection unit and the imaging unit are attached to a frame worn by the user, and the projection unit and the imaging unit are attached to an illumination light source that irradiates a plurality of illumination lights onto the user's eyes, and the generation unit generates a second guidance sign that guides the user so that the intervals between the plurality of illumination lights captured in the captured image become a predetermined value, and the projection control unit can be configured to control the projection unit to project the second guidance sign onto the retina.

[0011] In the above configuration, the second guidance sign may have a color that changes depending on the intervals between the plurality of illumination lights.

[0012] In the above configuration, the illumination light source can be configured to irradiate the user's eyes with at least four illumination lights that are opposed to each other in a first direction and a second direction perpendicular to the first direction as the plurality of illumination lights, and the generation unit can be configured to generate the second guidance sign that guides the user so that the distance between the two illumination lights that are opposed to each other in the first direction and the distance between the two illumination lights that are opposed to each other in the second direction become the predetermined value.

[0013] In the above configuration, an illumination light source is attached to a frame worn by the user together with the projection unit and the imaging unit, and irradiates the user's eyes with a plurality of illumination lights, and the generation unit can be configured to generate the superimposed image that displays the position based on the plurality of illumination lights captured in the captured image and the center position of the pupil captured in the captured image.

[0014] In the above configuration, the illumination light source can irradiate the user's eyes with at least four illumination lights that are opposed to each other in a first direction and a second direction perpendicular to the first direction as the plurality of illumination lights, and the generation unit can be configured to determine the position based on the plurality of illumination lights as an intersection point between a line passing through the centers of two illumination lights that are opposed to each other in the first direction and a line passing through the centers of two illumination lights that are opposed to each other in the second direction.

[0015] According to the present invention, it is possible to easily align the pupil position to a predetermined position.

[0016] FIG. 1 is a diagram illustrating an image projection device according to a first embodiment. FIG. 2 is a perspective view illustrating a configuration in which a housing incorporating a projection unit and an imaging unit according to the first embodiment is attached to an eyeglass-type frame. FIGS. 3A to 3C are diagrams illustrating a captured image, a reference image, and a superimposed image according to the first embodiment. FIGS. 4A to 4C are diagrams illustrating a process in which light rays are irradiated onto a retina according to the first embodiment. FIG. 5 is a diagram illustrating an example of a method for acquiring a reference image according to the first embodiment. FIG. 6 is a flowchart illustrating an example of control by a control unit according to the first embodiment. FIGS. 7A and 7B are diagrams illustrating an example of a projected image according to the first embodiment. FIG. 8A is a diagram illustrating an image projection device according to a second embodiment, and FIG. 8B is a diagram illustrating irradiation of the cornea with illumination light emitted from an illumination light source. FIGS. 9A to 9E are diagrams illustrating the relationship between the distance between the illumination light source and the cornea and the illumination light reflected in an image captured by the imaging unit according to the second embodiment. 10(a) and 10(b) are diagrams illustrating a case where the line of sight of the eye is tilted with respect to the optical axis of the imaging unit in Example 2. Fig. 11(a) to Fig. 11(c) are diagrams illustrating examples of a superimposed image and a second guidance sign generated by the generating unit in Example 2. Fig. 12 is a flowchart illustrating an example of control by the control unit in Example 2.

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0018] Fig. 1 is a diagram showing an image projection device 100 according to Example 1. Broken lines in Fig. 1 indicate electrical connections. The image projection device 100 according to Example 1 is a retinal projection head-mounted display that utilizes Maxwellian vision, in which light rays (laser light) for allowing a user to view an image are directly projected onto the user's retina.

[0019] As shown in FIG. 1 , the image projection device 100 according to the first embodiment includes a projection unit 10, a control unit 30, an imaging unit 40, a driving unit 41, and a storage unit 42. The projection unit 10 includes a light source 11, a lens 12, a scanning unit 13, and a projection optical system 20. The projection optical system 20 includes a reflecting mirror 21, a projection mirror 22, and a lens 23. The control unit 30 includes a projection control unit 31, a position control unit 32, an imaging control unit 33, and a generation unit 34. The control unit 30 is a processor such as a CPU (Central Processing Unit). The projection control unit 31, the position control unit 32, the imaging control unit 33, and the generation unit 34 may function using the same processor or different processors. The storage unit 42 is a non-volatile semiconductor memory such as a flash memory.

[0020] The light source 11 emits a light beam 50 (laser light) based on instructions from the projection control unit 31. The light source 11 emits the light beam 50, which is visible light, for example, red laser light (wavelength: approximately 610 nm to 660 nm), green laser light (wavelength: approximately 515 nm to 540 nm), and blue laser light (wavelength: approximately 440 nm to 480 nm). An example of the light source 11 that emits red, green, and blue laser light is a light source in which RGB (red, green, and blue) laser diode chips and a three-color combining device are integrated. Note that the light source 11 may also emit a light beam 50 of a single wavelength.

[0021] The projection control unit 31 receives image data from a camera and / or a recording device (not shown). The projection control unit 31 also receives image data generated by the generation unit 34 from the generation unit 34. The projection control unit 31 controls the emission of light beams 50 from the light source 11 based on the input image data. The projection control unit 31 also controls the driving of the scanning unit 13. The scanning unit 13 rapidly scans (e.g., raster scans) the light beams 50 from the upper left to the lower right of the image based on instructions from the projection control unit 31. The projection control unit 31 controls the light source 11 and the scanning unit 13 to control the projection of the light beams 50 onto the user's retina 61. By installing a camera at an appropriate position facing the user's line of sight, an image captured by the camera in the line of sight direction can be projected onto the retina 61. Furthermore, it is possible to project an image input from a recording device or superimpose a camera image on an image from the recording device, thereby projecting a so-called augmented reality (AR) image.

[0022] A light beam 50 emitted from the light source 11 passes through the lens 12. The lens 12 is a condenser lens that converts the light beam 50 from diffused light into focused light. The light beam 50 that passes through the lens 12 is incident on the scanning unit 13. The scanning unit 13 (scanner) scans the light beam 50 in two-dimensional directions, that is, the horizontal and vertical directions. The scanning unit 13 is, for example, a MEMS (Micro Electric Mechanical System) mirror. Note that the scanning unit 13 may be a mirror other than a MEMS mirror, and may be, for example, a scanner using potassium tantalate niobate (KTN) or the like.

[0023] The plurality of light beams 50 scanned in two dimensions by the scanning unit 13 and emitted from the scanning unit 13 in different directions at different times are incident on the projection optical system 20. The projection optical system 20 converges the plurality of light beams 50 emitted from the scanning unit 13 to a convergence point 70 inside the user's eye 60 and then projects the light beams 50 onto the retina 61, thereby projecting an image onto the retina 61. For example, the plurality of light beams 50 emitted from the scanning unit 13 are incident on a reflecting mirror 21. The reflecting mirror 21 is a concave mirror having a reflective surface formed of a curved surface such as a free-form surface, and has positive focusing power. The plurality of light beams 50 reflected by the reflecting mirror 21 converge at a convergence point 71 in front of the projection mirror 22. A lens 23 is provided at the convergence point 71 between the projection mirror 22 and the reflecting mirror 21. The lens 23 is, for example, a biconvex lens. The plurality of light beams 50 pass through the lens 23 and are incident on the projection mirror 22. The projection mirror 22 is disposed in front of the user's eye 60. The projection mirror 22 reflects a plurality of light rays 50 toward the user's eye 60. The projection mirror 22 is a concave mirror having a reflective surface formed of a curved surface such as a free-form surface, and has positive focusing power. The plurality of light rays 50 reflected by the projection mirror 22 pass through the user's pupil 62, converge at a convergence point 70 on or near the crystalline lens 63, and are then projected onto the retina 61. This allows the user to view an image formed by the light rays 50. The projection mirror 22 may be a half mirror. In this case, the user can view an external world image in a see-through manner.

[0024] 2 is a perspective view showing a configuration in which a housing 91 incorporating the projection unit 10 and the imaging unit 40 in Example 1 is attached to an eyeglass-type frame 90. As shown in Fig. 2, the housing 91 incorporating the projection unit 10 and the imaging unit 40 is attached to the eyeglass-type frame 90 that is worn on the user's face. The eyeglass-type frame 90 is provided with an adjustment unit 92 (e.g., an adjustment knob) that can adjust the position of the housing 91.

[0025] 1 , the imaging unit 40 is disposed in front of the user's eye 60 and captures an image of the user's eye 60 based on instructions from the imaging control unit 33. The imaging unit 40 is, for example, an infrared camera. Considering that the user will be able to see through the external world, the imaging unit 40 may be disposed at a position slightly offset from the front of the eye 60 within a range where it can capture an image of the eye 60. For example, in the case where the projection unit 10 is attached to the eyeglass-type frame 90, the imaging unit 40 may be provided near the temples or nose pads of the eyeglass-type frame 90.

[0026] The generation unit 34 acquires from the imaging unit 40 an image of the eye 60 captured by the imaging unit 40, and generates a superimposed image by superimposing the acquired image on a reference image stored in the storage unit 42. The reference image is an image that displays an initial position to which the center position of the pupil 62 should be aligned when the user wears the eyeglass-type frame 90. Furthermore, the generation unit 34 generates a first guidance sign that guides the user to align the center position of the pupil 62 with the initial position, based on the center position of the pupil 62 captured in the image of the eye 60 and the initial position in the reference image.

[0027] 3A to 3C are diagrams illustrating a captured image 80, a reference image 81, and a superimposed image 82 in Example 1. As shown in FIG. 3A, the captured image 80 captured by the imaging unit 40 at least includes the pupil 62 and iris 64 of the user's eye 60. As shown in FIG. 3B, the reference image 81 stored in the storage unit 42 displays an initial position 83 to which the center position of the pupil 62 should be aligned. Straight lines passing through the initial position 83 and extending in the horizontal and vertical directions may be displayed. A superimposed image 82 obtained by superimposing the captured image 80 and the reference image 81 is as shown in FIG. 3C. In the superimposed image 82, a center position 65 of the pupil 62 and straight lines passing through the center position 65 and extending in the horizontal and vertical directions may be displayed. The center position 65 of the pupil 62 may be determined by, for example, identifying the area of ​​the pupil by performing image analysis on the captured image 80 or the superimposed image 82, and identifying the center part of the identified area as the center position 65 of the pupil 62, or by other methods.

[0028] 1 , the driving unit 41 is, for example, a biaxial actuator, and moves the reflecting mirror 21 based on instructions from the position control unit 32. As described above, the reflecting mirror 21 is a concave mirror. If the apex of the concave curved surface is located approximately in the center, the reflecting mirror 21 moves so as to oscillate around the apex. Of the multiple light rays 50, light ray 50a corresponding to the center of the projected image projected onto the retina 61 is incident on the oscillation center of the reflecting mirror 21 even when the reflecting mirror 21 moves.

[0029] 4(a) to 4(c) are diagrams showing the process of irradiating the retina 61 with the light ray 50 in Example 1. Fig. 4(b) and Fig. 4(c) show the case where the reflecting mirror 21 has moved relative to Fig. 4(a).

[0030] As shown in FIGS. 4A to 4C , a light ray 50 emitted from the light source 11 passes through the lens 12. The lens 12 is a condensing lens that converts the light ray 50 from diffused light to converged light. The light ray 50 that passes through the lens 12 enters the scanning unit 13 in a converged state. The lens 12 is provided between the light source 11 and the scanning unit 13 to convert the light ray 50 reflected by the reflecting mirror 21 into substantially parallel light. The scanning unit 13 scans the light ray 50 in two dimensions, and multiple light rays 50 that are emitted from the scanning unit 13 in different directions at different times enter the reflecting mirror 21. Each of the multiple light rays 50 is condensed in front of the reflecting mirror 21, then becomes diffused light, and enters the reflecting mirror 21. Because the reflecting mirror 21 has a positive focusing power, each of the multiple light rays 50 is converted from diffused light to substantially parallel light by being reflected by the reflecting mirror 21.

[0031] The multiple light rays 50 reflected by the reflection mirror 21 converge at a convergence point 71 in front of the projection mirror 22. A lens 23 is provided at the convergence point 71. The lens 23 is a condensing lens that converts each of the multiple light rays 50 from approximately parallel light into convergent light. The lens 23 is provided at the convergence point 71 to convert each of the multiple light rays 50 reflected by the projection mirror 22 into approximately parallel light. Each of the multiple light rays 50 transmitted through the lens 23 is condensed at a condensing point 72 in front of the projection mirror 22, and then becomes diffused light before entering the projection mirror 22. Because the projection mirror 22 has a positive condensing power, each of the multiple light rays 50 is converted from diffused light into approximately parallel light by being reflected by the projection mirror 22. The multiple light rays 50 converge at a convergence point 70 in the user's eye 60. The convergence point 70 is located, for example, on the crystalline lens 63 or near the crystalline lens 63. The light beam 50 is converted from approximately parallel light into converging light by the crystalline lens 63 and is focused near the retina 61. This allows the user to view the image.

[0032] As shown in Figure 4(a), when a light ray 50a corresponding to the center of the projected image projected onto the retina 61 enters the eye 60 almost from the front, the convergence point 71 where the multiple light rays 50 reflected by the reflecting mirror 21 converge is near the center of the lens 23.

[0033] As shown in FIG. 4B , when the user turns their gaze to the right to view the right side of the image, the pupil 62 moves to the right relative to the user's face. In this case, the position control unit 32 instructs the drive unit 41 to move the reflecting mirror 21 so that the plurality of light rays 50 pass through a region of the lens 23 to the left of the traveling direction of the light rays 50. As a result, the convergence point 71 where the plurality of light rays 50 converge is located in a region of the lens 23 to the left of the traveling direction of the light rays 50. By passing through a region of the lens 23 to the left of the traveling direction of the light rays 50, the plurality of light rays 50 are refracted by the lens 23 to the right of the traveling direction of the light rays 50, and then enter the projection mirror 22. By refracting the plurality of light rays 50 to the right of the traveling direction of the light rays 50 by the lens 23, the position and angle of incidence of the plurality of light rays 50 on the projection mirror 22 change from the state shown in FIG. 4A . As a result, the position of the convergence point 70 of the plurality of light rays 50 moves to the right, and the plurality of light rays 50 are incident on the pupil 62 even if the pupil 62 moves to the right.

[0034] As shown in Figure 4(c), when the user turns their gaze leftward to view the left side of the image, the pupil 62 moves leftward relative to the user's face. In this case, the position control unit 32 instructs the drive unit 41 to move the reflecting mirror 21 so that the multiple light rays 50 pass through a region of the lens 23 that is to the right of the traveling direction of the light rays 50. As a result, the convergence point 71 where the multiple light rays 50 converge is located in a region of the lens 23 that is to the right of the traveling direction of the light rays 50. By passing through a region of the lens 23 that is to the right of the traveling direction of the light rays 50, the multiple light rays 50 are refracted by the lens 23 to the left of the traveling direction of the light rays 50, and then enter the projection mirror 22. By refracting the multiple light rays 50 to the left of the traveling direction of the light rays 50 by the lens 23, the position and angle of incidence of the multiple light rays 50 on the projection mirror 22 change from the state shown in Figure 4(a). As a result, the position of the convergence point 70 of the plurality of light rays 50 moves to the left, and the plurality of light rays 50 are incident on the pupil 62 even if the pupil 62 moves leftward.

[0035] 4(b) and 4(c) illustrate an example in which the pupil 62 moves in the left-right direction, but the same applies when the pupil 62 moves in other directions, such as up-down. In this way, the driver 41 can move the reflecting mirror 21 to move the position of the convergence point 70, thereby allowing the light rays 50 to be incident on the pupil 62 even when the line of sight of the eye 60 moves. Therefore, by identifying the position of the pupil 62 from the captured image 80 captured by the imaging unit 40 and moving the reflecting mirror 21 so that the convergence point 70 is located at the identified position of the pupil 62, it is possible to allow multiple light rays 50 to be continuously incident on the pupil 62. To achieve this, control information is acquired in advance, which associates multiple points in the captured image 80 captured by the imaging unit 40 with control values ​​for driving the driver 41 to position the convergence point 70 at these multiple points. The position control unit 32 then identifies the position of the pupil 62 shown in the captured image 80 captured by the imaging unit 40, and controls the drive unit 41 to move the reflecting mirror 21 based on the identified position of the pupil 62 and the control information, thereby positioning the convergence point 70 at the pupil 62. The control information is stored in advance in the storage unit 42, for example.

[0036] Table 1 is an example of the control information stored in the memory unit 42. Table 1 shows an example in which the horizontal field of view of the image projected onto the retina 61 is 28° and the vertical field of view is 14°. As shown in Table 1, the control information associates multiple points within the captured image 80 captured by the imaging unit 40 with current values ​​(control values) for positioning the convergence point 70 at these multiple points. Here, the multiple points are defined by horizontal and vertical angles. Since the drive unit 41 that moves the reflecting mirror 21 is, for example, a biaxial actuator, if the normal direction at the vertex of the concave curved surface of the reflecting mirror 21 is the roll axis, the horizontal direction of the reflecting mirror 21 is the pitch axis, and the vertical direction is the yaw axis, the current values ​​for yaw rotation (the values ​​on the left side of Table 1) and the current values ​​for pitch rotation (the values ​​on the right side of Table 1) are listed. For example, to position the convergence point 70 at a point where the horizontal and vertical angles are defined as (10.5°, 3.5°), the driver 41 is driven with a current value of (36.44 mA, 9.75 mA) to move the reflecting mirror 21.

[0037] Here, an example of a method for acquiring the reference image 81 shown in FIG. 3B will be described. FIG. 5 is a diagram showing an example of a method for acquiring the reference image 81 in Example 1. As shown in FIG. 5, the image projection device 100 is prepared, and the light source 11 is driven to emit light rays 50 from the image projection device 100 to the outside. At this time, the driving unit 41 may or may not be driven, and the reflecting mirror 21 is in its initial position. The light rays 50 emitted from the image projection device 100 converge at a convergence point 70. A screen 84 is placed at the convergence point 70. The light rays 50 irradiated onto the screen 84 are captured by the imaging unit 40. The image captured by the imaging unit 40 becomes the reference image 81 shown in FIG. 3B, and the position of the convergence point 70 of the light rays 50 when the reflecting mirror 21 is in its initial position becomes the initial position 83.

[0038] Therefore, when the user wears the eyeglass-type frame 90 (image projection device 100), the center position 65 of the pupil 62 shown in the captured image 80 can be aligned with the initial position 83 displayed in the reference image 81, thereby aligning the center position 65 of the pupil 62 with the position of the convergence point 70 where the light rays 50 converge. Therefore, by moving the reflecting mirror 21 using the control information shown in Table 1, the light rays 50 can continue to be incident on the pupil 62 even if the line of sight of the eye 60 moves.

[0039] [Control Method] Fig. 6 is a flowchart showing an example of control by the control unit 30 in Example 1. As shown in Fig. 6, when the eyeglass-type frame 90 (image projection device 100) is worn on the face of a user, the imaging control unit 33 controls the imaging unit 40 to start imaging the user's eye 60 (step S10). For example, when an instruction to start imaging is input from the user wearing the eyeglass-type frame 90, the imaging control unit 33 controls the imaging unit 40 to start imaging the user's eye 60.

[0040] Next, the generation unit 34 acquires a captured image 80 of the user's eye 60 from the imaging unit 40 (step S12). For example, the generation unit 34 acquires the captured image 80 as shown in FIG. 3A. Next, the generation unit 34 generates a superimposed image 82 by superimposing the acquired captured image 80 on a reference image 81 indicating an initial position 83 stored in the storage unit 42 (step S14). For example, the generation unit 34 superimposes the captured image 80 as shown in FIG. 3A on the reference image 81 as shown in FIG. 3B to generate the superimposed image 82 as shown in FIG. 3C.

[0041] Furthermore, the generation unit 34 generates a first guidance sign that guides the user to align the center position 65 of the pupil 62 with the initial position 83 based on the center position 65 of the pupil 62 captured in the captured image 80 and the initial position 83 in the reference image 81 (step S16). For example, the generation unit 34 generates, as the first guidance sign, an arrow that indicates a direction to move from the center position 65 of the pupil 62 and the initial position 83 to align the center position 65 of the pupil 62 with the initial position 83.

[0042] Next, the projection control unit 31 controls the light source 11 and the scanning unit 13 to project the superimposed image 82 and the first guidance marker generated by the generation unit 34 onto the user's retina 61 (step S18). FIGS. 7A and 7B are diagrams illustrating an example of a projection image 86 in the first embodiment. As illustrated in FIG. 7A , the projection image 86 including the superimposed image 82 and a first guidance marker 87, e.g., an arrow, is projected onto the retina 61. The projection image 86 may include an image based on image data input from a camera and / or a recording device (e.g., a mountain, a river, or a forest in FIGS. 7A and 7B ). Projection of the projection image 86 including the superimposed image 82 and the first guidance marker 87 allows the user to visually grasp the deviation between the position of the pupil 62 and the initial position 83. Therefore, for example, the user can adjust the adjustment unit 92 (see FIG. 2 ) that moves the position of the projection unit 10 or the position of the nose pads of the eyeglass-type frame 90 so that the center position 65 of the pupil 62 coincides with the initial position 83. Fig. 7(b) shows an example of a superimposed image 82 after alignment.

[0043] 6 , the generation unit 34 determines whether or not the position adjustment for adjusting the center position 65 of the pupil 62 to align with the initial position 83 has been completed (step S20), and if not completed (No), the process returns to step S12, and if completed (Yes), the process ends. Whether or not the position adjustment has been completed may be determined, for example, based on whether or not an instruction indicating completion has been input by the user, or based on whether or not the amount of deviation between the center position 65 of the pupil 62 and the initial position 83 has become equal to or less than a predetermined value.

[0044] According to the first embodiment, as shown in FIGS. 3A to 3C , the generation unit 34 generates a superimposed image 82 by superimposing an image 80 of the user's eye 60 captured by the imaging unit 40 on a reference image 81 displaying an initial position 83 to which the user's pupil 62 should be aligned. The projection control unit 31 controls the projection unit 10 to project the superimposed image 82 generated by the generation unit 34 onto the user's retina 61. This allows the user to visually grasp the positional relationship between the position of their own pupil 62 and the initial position 83 to which the pupil 62 should be aligned. Therefore, the user can adjust the position of the pupil 62 and the initial position 83 while viewing the superimposed image 82, thereby facilitating alignment.

[0045] 7A , in the first embodiment, the generation unit 34 generates a first guidance sign 87 that guides the user to align the center position 65 of the pupil 62 with the initial position 83, based on the center position 65 of the pupil 62 captured in the captured image 80 and the initial position 83 in the reference image 81. The projection control unit 31 projects the first guidance sign 87 onto the retina 61. This allows the user to visually grasp the direction to be adjusted in order to align the center position 65 of the pupil 62 with the initial position 83, thereby facilitating alignment.

[0046] In Example 1, the first guidance sign 87 is an arrow, but a sign other than an arrow may be used as long as it is possible to guide the center position 65 of the pupil 62 to coincide with the initial position 83.

[0047] FIG. 8A illustrates an image projection device 200 according to a second embodiment, and FIG. 8B illustrates irradiation of the cornea 66 with illumination light 17 emitted by the illumination light source 15. As illustrated in FIG. 8A, the image projection device 200 according to the second embodiment includes a plurality of illumination light sources 15 in front of the user's eye 60. The illumination light sources 15 are incorporated into the housing 91 in FIG. 2 together with the projection unit 10 and the imaging unit 40 and are attached to the eyeglass-type frame 90. The illumination light sources 15 are, for example, infrared light-emitting diodes (LEDs), and emit infrared illumination light 17 based on instructions from the imaging control unit 33. At least four illumination light sources 15 are provided so as to face the pupil 62 in the vertical and horizontal directions. Therefore, as illustrated in FIG. 8B, four illumination light sources 17 facing the pupil 62 in the vertical and horizontal directions are irradiated onto the user's cornea 66. The other configurations are the same as those of the first embodiment, and therefore description thereof will be omitted.

[0048] 9( a ) to 9 ( e ) are diagrams illustrating the relationship between the distance d between the illumination light source 15 and the cornea 66 in Example 2 and the illumination light 17 captured in the captured image 80 captured by the imaging unit 40. As shown in FIG. 9( a ), the distance from the multiple illumination light sources 15 to the vertex 67 of the cornea 66 is d. Because the illumination light source 15 emits the illumination light 17 in an oblique direction inclined toward the vertex 67 of the cornea 66, as shown in FIGS. 9( b ) to 9 ( e ), the distances D1 and D2 of the illumination light 17 captured in the captured image 80 vary depending on the distance d from the illumination light source 15 to the vertex 67 of the cornea 66. FIG. 9( b ) shows an example where the distance d is 3 mm, FIG. 9( c ) shows an example where the distance d is 5 mm, FIG. 9( d ) shows an example where the distance d is 7 mm, and FIG. 9( e ) shows an example where the distance d is 9 mm. As the distance d increases, the distances D1 and D2 decrease. The distance D1 is the distance between the centers of two illumination lights 17 that face each other in the vertical direction with respect to the pupil 62. The distance D2 is the distance between the centers of two illumination lights 17 that face each other in the horizontal direction with respect to the pupil 62.

[0049] As shown in FIG. 8A , it is desirable that the convergence point 70 at which the multiple light rays 50 emitted from the image projection device 100 converge is located at an appropriate position on or near the crystalline lens 63. However, when a user wears the eyeglass-type frame 90 (image projection device 100), the position of the convergence point 70 may deviate from the appropriate position in the direction of travel of the light rays 50 depending on the position of the eyeglass-type frame 90 relative to the user's face. As shown in FIGS. 9A to 9E , there is a correlation between the distance d and the distances D1 and D2, so the distance d can be determined by calculating the distances D1 and D2. Because the illumination light source 15, together with the projection unit 10 and the imaging unit 40, is built into the housing 91 and attached to the eyeglass-type frame 90, the value of the distance d when the convergence point 70 is located at the appropriate position can be determined. Therefore, the convergence point 70 can be located at the appropriate position by adjusting the distances D1 and D2 to appropriate values.

[0050] Therefore, in the second embodiment, the generation unit 34 detects the intervals D1 and D2 between the illumination lights 17 captured in the captured image 80, and generates second guidance signs that guide the user so that the intervals D1 and D2 become predetermined values. The projection control unit 31 projects the second guidance signs generated by the generation unit 34 onto the retina 61.

[0051] 10( a) and 10(b) are diagrams illustrating a case where the line of sight of the eye 60 is tilted with respect to the optical axis 44 of the image capturing unit 40 in Example 2. In FIGS. 10(a) and 10(b), the upper diagram is a cross-sectional view of the eye 60, and the lower diagram is a front view of the eye 60. As shown in the upper diagram of FIG. 10(a), when there is no tilt between the optical axis 44 of the image capturing unit 40 and the line of sight of the eye 60, the apex 67 of the cornea 66 and the center position 65 of the pupil 62 are located on the optical axis 44 of the image capturing unit 40. In this case, as shown in the lower diagram, the irradiation position of the illumination light 17 on the cornea 66 is such that an intersection 18 between a line passing through the center of the illumination light 17 that faces the pupil 62 in the vertical direction and a line passing through the center of the illumination light 17 that faces the pupil 62 in the horizontal direction is located on the optical axis 44.

[0052] As shown in the upper diagram of Figure 10(b), when a user wears the eyeglass-type frame 90 (image projection device 100), the optical axis 44 of the image capture unit 40 and the line of sight of the eye 60 may be tilted at an angle θ. In this case, as shown in the upper and lower diagrams, the center position 65 of the pupil 62 is shifted by an amount δ from the optical axis 44 of the image capture unit 40. In other words, the center position 65 of the pupil 62 is shifted by an amount δ from the intersection 18 of the lines passing through the centers of the multiple illumination lights 17. By eliminating this amount δ, the tilt shift can be suppressed.

[0053] Therefore, in Example 2, in order to guide the user to reduce the deviation amount δ, the generation unit 34 generates a superimposed image 82 that displays the intersection 18 of the lines passing through the centers of the multiple illumination lights 17 captured in the captured image 80 and the center position 65 of the pupil 62 captured in the captured image 80.

[0054] 11( a) to 11(c) are diagrams illustrating examples of a superimposed image 82 and a second guidance sign 88 generated by the generation unit 34 in the second embodiment. As shown in FIG. 11(a), the generation unit 34 may generate a superimposed image 82 in which a second guidance sign 88 (hatched portion) whose color changes depending on the deviation of the intervals D1 and D2 from the appropriate size is displayed in a portion overlapping with the illumination light 17. As shown in FIG. 11(b), the generation unit 34 may generate a superimposed image 82 in which a second guidance sign 88 (hatched portion) whose color changes depending on the deviation of the intervals D1 and D2 from the appropriate size is displayed in a region not overlapping with the eye 60. As shown in FIG. 11(c), the generation unit 34 may generate a second guidance sign 88 (hatched portion) whose color changes depending on the deviation of the intervals D1 and D2 from the appropriate size in a region other than the superimposed image 82 in the projection image 86. For example, the second guidance sign 88 may be red when the amount of deviation is large, and may change color to orange, yellow, green, or blue as the amount of deviation becomes smaller.

[0055] Furthermore, as shown in Figures 11(a) to 11(c), the generation unit 34 generates a superimposed image 82 that displays the intersection 18 of the straight lines passing through the centers of the multiple illumination lights 17 captured in the captured image 80 and the center position 65 of the pupil 62 captured in the captured image 80.

[0056] [Control Method] Fig. 12 is a flowchart showing an example of control by the control unit 30 in Example 2. As shown in Fig. 12 , when the eyeglass-type frame 90 (image projection device 100) is worn on the user's face, the imaging control unit 33 controls the illumination light source 15 to irradiate the user's eye 60 with illumination light 17 (step S30). For example, when an instruction to irradiate the illumination light 17 is input from the user wearing the eyeglass-type frame 90, the imaging control unit 33 controls the illumination light source 15 to irradiate the user's eye 60 with illumination light 17. Next, the imaging control unit 33 controls the imaging unit 40 to start capturing an image of the user's eye 60 (step S32).

[0057] Next, the generation unit 34 acquires a captured image 80 of the user's eye 60 from the imaging unit 40 (step S34). Next, the generation unit 34 detects the intervals D1 and D2 of the illumination light 17 captured in the captured image 80, and generates second guidance signs 88 that guide the user so that the intervals D1 and D2 become predetermined values, based on the detection result (step S36). For example, the generation unit 34 generates second guidance signs 88 to be displayed in the superimposed image 82 or second guidance signs 88 to be displayed in an area of ​​the projected image 86 other than the superimposed image 82, as shown in FIGS. 11( a) to 11(c).

[0058] Next, the generation unit 34 superimposes the captured image 80 on a reference image 81 indicating the initial position 83, and generates a superimposed image 82 that displays the intersection 18 of the lines passing through the centers of the illumination lights 17 captured in the captured image 80 and the center position 65 of the pupil 62 captured in the captured image 80 (step S38). For example, the generation unit 34 generates a superimposed image 82 that displays the intersection 18 and the center position 65 as shown in Figures 11(a) to 11(c).

[0059] In addition, the generation unit 34 generates a first guidance sign 87 that guides the user to align the center position 65 of the pupil 62 with the initial position 83 based on the center position 65 of the pupil 62 captured in the captured image 80 and the initial position 83 in the reference image 81 (step S40).

[0060] Next, the projection control unit 31 controls the light source 11 and the scanning unit 13 to project the superimposed image 82, the first guidance marker 87, and the second guidance marker 88 generated by the generation unit 34 onto the user's retina 61 (step S42). As a result, similar to the first embodiment, the user can visually grasp the deviation between the position of his / her pupil 62 and the initial position 83 by using the superimposed image 82. Therefore, the user can adjust the position of the pupil 62 to match the initial position 83 by referring to the first guidance marker 87 (see FIG. 7A). The second guidance marker 88 also allows the user to adjust the position of the convergence point 70 where the multiple light rays 50 converge to an appropriate position. Furthermore, the intersection 18 of the lines passing through the centers of the multiple illumination lights 17 and the center position 65 of the pupil 62 are displayed, allowing the user to adjust the tilt deviation between the optical axis 44 of the imaging unit 40 and the line of sight of the eye 60. For example, the user can make the above-mentioned various adjustments by adjusting the adjustment unit 92 (see FIG. 2) that moves the position of the projection unit 10 or the position of the nose pads of the eyeglass-type frame 90.

[0061] Next, the generation unit 34 determines whether the position adjustment using the superimposed image 82 has been completed (step S44), and if it has not been completed (No), the process returns to step S34, and if it has been completed (Yes), the process ends. Whether the position adjustment has been completed may be determined, for example, based on whether the user has input an instruction that the adjustment has been completed, or by other methods.

[0062] According to the second embodiment, as shown in FIG. 8A , the illumination light source 15 is attached to the eyeglass-type frame 90 by being built into a housing 91 together with the projection unit 10 and the imaging unit 40, and irradiates a plurality of illumination lights 17 onto the user's eye 60. As shown in FIGS. 9A to 9E and 11A to 11C , the generation unit 34 generates second guidance signs 88 that guide the user so that the intervals D1 and D2 of the plurality of illumination lights 17 shown in the captured image 80 become predetermined values, and the projection control unit 31 controls the projection unit 10 to project the second guidance signs 88 onto the retina 61. This allows the user to adjust the intervals D1 and D2 of the plurality of illumination lights 17 to appropriate sizes based on the second guidance signs 88, and as a result, the position of the convergence point 70 can be adjusted to an appropriate position.

[0063] In the second embodiment, as shown in Fig. 8(b), the illumination light source 15 irradiates the user's eye 60 with four illumination lights 17 that face each other in the vertical direction (first direction) and the horizontal direction (second direction). As shown in Figs. 9(b) to 9(e), the generation unit 34 generates a second guidance sign 88 that guides the user so that the interval D1 between the two illumination lights 17 facing each other in the vertical direction and the interval D2 between the two illumination lights 17 facing each other in the horizontal direction become predetermined values. By adjusting the intervals D1 and D2 to appropriate values, the position of the convergence point 70 can be adjusted to an appropriate position.

[0064] In Example 2, the second guidance sign 88 has a color that changes depending on the spacing D1 and D2 between the multiple illumination lights 17, but it may be something other than a color as long as it can guide the user so that the spacing D1 and D2 become predetermined values.

[0065] 11( a) to 11(c), the generation unit 34 generates a superimposed image 82 that displays the intersection 18, which is a position based on the plurality of illumination lights 17 captured in the captured image 80, and the center position 65 of the pupil 62 captured in the captured image 80. This allows the user to adjust the position so that the intersection 18 and the center position 65 of the pupil 62 are aligned, and as a result, it is possible to reduce the tilt deviation between the optical axis 44 of the image capture unit 40 and the line of sight of the eye 60.

[0066] 10( a) and 10(b), the generation unit 34 determines an intersection 18, where a line passing through the centers of two opposing illumination lights 17 in the vertical direction intersects with a line passing through the centers of two opposing illumination lights 17 in the horizontal direction, as a position based on the plurality of illumination lights 17. By displaying such intersection 18 in the superimposed image 82 and adjusting the position so that the intersection 18 and the center position 65 of the pupil 62 coincide with each other, it is possible to reduce the tilt deviation between the optical axis 44 of the imaging unit 40 and the line of sight of the eye 60.

[0067] In the first and second embodiments, the reflecting mirror 21 and the projection mirror 22 are not limited to concave mirrors, and may be other optical components such as a combination of lenses and mirrors, or a diffraction element, other than curved mirrors, as long as they have positive light-collecting power. The lens 23 is not limited to a convex lens, and may be other optical components such as a mirror or a diffraction element, as long as it is possible to make the light beam 50 incident on the projection mirror 22 as diffused light. The lens 23 may have a function of suppressing chromatic aberration.

[0068] In the first and second embodiments, the image projection device is attached to the eyeglass-type frame 90, but it may also be attached to a goggle-type, eye patch-type, ear hook-type, helmet-mounted type, or other frame.

[0069] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

Claims

1. An image projection device comprising: a projection unit including a scanning unit that scans light rays emitted from a light source; and a projection optical system that causes a plurality of light rays emitted in different directions from the scanning unit to converge to a convergence point inside the user's eye by being scanned by the scanning unit, and then projects the light rays onto the retina to project an image onto the retina; an imaging unit that images the user's eye; a generation unit that generates a superimposed image by superimposing the image of the user's eye captured by the imaging unit and a reference image indicating the position at which the user's pupil should be aligned; and a projection control unit that controls the projection unit to project the superimposed image onto the retina.

2. The image projection device described in claim 1, wherein the generation unit generates a first guiding sign that guides the user so that the center position of the user's pupil coincides with the position based on the center position of the user's pupil captured in the captured image and the position in the reference image, and the projection control unit controls the projection unit to project the first guiding sign onto the retina.

3. The image projection device according to claim 2, wherein the first guidance sign is an arrow.

4. The image projection device according to claim 1 or 2, wherein the projection unit and the imaging unit are attached to a frame worn by the user.

5. The image projection device of claim 1, further comprising an illumination light source attached together with the projection unit and the imaging unit to a frame worn by the user, and irradiating the user's eye with a plurality of illumination lights, wherein the generation unit generates a second guidance sign that guides the user so that the intervals between the plurality of illumination lights captured in the captured image become a predetermined value, and the projection control unit controls the projection unit to project the second guidance sign onto the retina.

6. The image projection device according to claim 5, wherein the second guidance sign has a color that changes according to the intervals between the plurality of illumination lights.

7. The image projection device described in claim 5 or 6, wherein the illumination light source irradiates the user's eyes with at least four illumination lights that are opposed in a first direction and a second direction perpendicular to the first direction as the multiple illumination lights, and the generation unit generates the second guidance sign that guides the user so that the distance between the two illumination lights that are opposed in the first direction and the distance between the two illumination lights that are opposed in the second direction become the predetermined value.

8. The image projection device according to claim 1, further comprising an illumination light source attached together with the projection unit and the imaging unit to a frame worn by the user, and irradiating the user's eyes with a plurality of illumination lights, wherein the generation unit generates the superimposed image displaying a position based on the plurality of illumination lights captured in the captured image and a center position of the pupil captured in the captured image.

9. The image projection device described in claim 8, wherein the illumination light source irradiates the user's eye with at least four illumination lights that are opposed in a first direction and a second direction perpendicular to the first direction as the multiple illumination lights, and the generation unit determines that the position based on the multiple illumination lights is an intersection point between a line passing through the centers of two illumination lights that are opposed in the first direction and a line passing through the centers of two illumination lights that are opposed in the second direction.

Images