Image projection device

Abstract

An image projection device 100 comprises: a first imaging unit 40 that images the surrounding of a user; an image projection unit 10 that projects, onto eyes 60 of the user, a first image 71 of the surrounding of the user imaged by the first imaging unit 40; a second imaging unit 41 that images the eyes 60 of the user; an identification unit 34 that identifies, on the basis of a second image 72 of the eyes 60 imaged by the second imaging unit 41, a gaze point 70 on which the user is gazing in the first image 71; and a focus control unit 35 that controls the first imaging unit 40 such that the focus of the first imaging unit 40 matches the gaze point 70.

Description

Image projection device

[0001] The present invention relates to an image projection device.

[0002] Image projection devices that project images onto a user's eyes are known (for example, Patent Documents 1 and 2). In addition, control methods related to focus have been proposed in imaging devices such as cameras (for example, Patent Documents 3 and 4).

[0003] Japanese Unexamined Patent Application Publication No. 2023-76137, Japanese Unexamined Patent Application Publication No. 2024-90193, Japanese Unexamined Patent Application Publication No. 2023-34489, Japanese Unexamined Patent Application Publication No. 2022-66266

[0004] Image projection devices equipped with an imaging unit that captures an image in front of the user and projects the image captured by the imaging unit onto the user's eyes are known. In such an image projection device, the user may gaze at various locations such as not only the central part of the image but also the peripheral part. When the user changes the gaze point, it is desirable that an image focused on the changed gaze point be projected.

[0005] The present invention has been made in view of the above problems, and an object thereof is to project an image focused on the gaze point.

[0006] The present invention includes a first imaging unit that captures the surroundings of the user, an image projection unit that projects a first image of the surroundings of the user captured by the first imaging unit onto the user's eyes, a second imaging unit that captures the user's eyes, a specifying unit that specifies a gaze point on which the user is gazing in the first image based on a second image of the eyes captured by the second imaging unit, and a focus control unit that controls the first imaging unit so that the focus of the first imaging unit matches the gaze point.

[0007] In the above configuration, it is possible to provide a storage unit that stores first information indicating the relationship between the pupil position and the line-of-sight angle, and the specifying unit specifies the line-of-sight angle of the eyes from the position of the pupil shown in the second image and the first information, and specifies the gaze point on which the user is gazing in the first image using the specified line-of-sight angle of the eyes.

[0008] In the above configuration, the image projection unit includes a scanning unit that scans a light ray emitted from a light source, and a projection unit that, after the light ray is scanned by the scanning unit, concentrates a plurality of light rays emitted in different directions from the scanning unit to a convergence point inside the eye, and then projects the first image onto the retina of the eye. The storage unit stores second information indicating the relationship between the line of sight angle and the scanning angle of the scanning unit, and the identification unit identifies the scanning angle of the scanning unit corresponding to the identified line of sight angle of the eye from the second information, and identifies the point of fixation in the first image that the user is fixated on from the identified scanning angle of the scanning unit.

[0009] In the above configuration, the image projection unit comprises a display unit that displays the first image and a focusing lens positioned between the display unit and the eye, the storage unit stores third information indicating the relationship between the line of sight angle and the position on the display unit, and the identification unit identifies the position on the display unit using the identified line of sight angle of the eye and the third information, and identifies the point of fixation in the first image that the user is fixated on from the identified position on the display unit.

[0010] In the above configuration, when the user wears the image projection device, a plurality of target images having targets at different positions are projected onto the eye, a plurality of eye images are captured by the second imaging unit while the user is looking at the targets in the plurality of target images, and the first information is created and stored in the storage unit by identifying the positions of a plurality of pupils captured in the plurality of eye images and a plurality of line-of-sight angles corresponding to the positions of the plurality of pupils.

[0011] In the above configuration, the image projection unit includes a scanning unit that scans a light ray emitted from a light source, and a projection unit that, after the light ray is scanned by the scanning unit, converges a plurality of light rays emitted in different directions from the scanning unit to a convergence point inside the eye, and then projects the plurality of target images onto the retina of the eye. The creation unit can be configured to create the first information by specifying the plurality of line-of-sight angles corresponding to the positions of the plurality of pupils depicted in the plurality of eye images from the position of the target in the plurality of target images and fourth information indicating the relationship between the position on the plurality of target images and the line-of-sight angle.

[0012] In the above configuration, the image projection unit comprises a display unit that displays the plurality of target images and a focusing lens positioned between the display unit and the eye, and the creation unit can be configured to create the first information by specifying the plurality of line-of-sight angles corresponding to the positions of the plurality of pupils projected onto the plurality of eye images from the position of the target on the display unit and fifth information indicating the relationship between the position on the display unit and the line-of-sight angle.

[0013] According to the present invention, an image focused on the point of gaze can be projected.

[0014] Figure 1 is a block diagram of an image projection device according to Embodiment 1. Figure 2 is a diagram showing an example of the arrangement of the image projection unit, the first imaging unit, and the second imaging unit in Embodiment 1. Figures 3(a) and 3(b) are diagrams showing an example of a second image acquired by the identification unit in Embodiment 1. Figure 4 is a diagram showing the relationship between the position on the display unit and the line of sight angle in Embodiment 1. Figure 5 is a flowchart showing an example of the creation of information showing the relationship between pupil position and line of sight angle by the creation unit in Embodiment 1. Figures 6(a) and 6(b) are diagrams showing the creation of information showing the relationship between pupil position and line of sight angle by the creation unit in Embodiment 1. Figures 7(a) to 7(d) are diagrams showing the identification of the point of fixation by the identification unit in Embodiment 1. Figure 8 is a flowchart showing an example of the control of the control unit in Embodiment 1. Figure 9 is a block diagram of an image projection device according to Embodiment 2. Figure 10 is a diagram showing the optical system of the image projection device according to Embodiment 2. Figure 11 is a diagram showing an example of the arrangement of the scanning unit, reflection unit, projection unit, optical components, first imaging unit, and second imaging unit in Embodiment 2. Figure 12 is a flowchart showing an example of control of the control unit in Embodiment 2.

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0016] Figure 1 is a block diagram of an image projection device 100 according to Embodiment 1. The image projection device 100 is a head-mounted display that is worn, for example, on the user's face or head and projects an image onto the user's eyes. As shown in Figure 1, the image projection device 100 comprises an image projection unit 10, a control unit 30, a first imaging unit 40, a second imaging unit 41, and a storage unit 42. The image projection unit 10 includes a display unit 11. The control unit 30 includes an image control unit 31, a creation unit 32, an imaging control unit 33, a identification unit 34, and a focus control unit 35.

[0017] The first imaging unit 40 captures video footage of the area in front of the user wearing the image projection device 100, for example. In the following, the image of the area in front of the user captured by the first imaging unit 40 may be referred to as the first image.

[0018] The image control unit 31 acquires image data of the first image from the first imaging unit 40 and displays the first image on the display unit 11. The display unit 11 is a display device such as a liquid crystal display. A user wearing the image projection device 100 can view the first image captured by the first imaging unit 40 by looking at the display unit 11 placed in front of their eyes.

[0019] The imaging control unit 33 instructs the second imaging unit 41 to start taking images based on a control signal from the image control unit 31. The second imaging unit 41 is positioned in front of the user's eyes and takes images of the user's eyes. In the following, the image of the user's eyes taken by the second imaging unit 41 while the user is viewing the first image may be referred to as the second image.

[0020] The identification unit 34 acquires image data of the second image from the second imaging unit 41. Based on the acquired second image, the identification unit 34 identifies the point of focus that the user is fixated on in the first image displayed on the display unit 11. For example, the storage unit 42 stores information indicating the relationship between pupil position and line of sight angle, and information indicating the relationship between line of sight angle and position on the display unit 11. The information indicating the relationship between pupil position and line of sight angle is created by the creation unit 32 when the user puts on the image projection device 100 and stored in the storage unit 42. The information indicating the relationship between line of sight angle and position on the display unit 11 is created in advance from design information of the optical system provided in the image projection unit 10 and stored in the storage unit 42. The identification unit 34 identifies the position of the pupil of the eye captured in the second image, and identifies the line of sight angle of the eye captured in the second image from the identified pupil position and the information indicating the relationship between pupil position and line of sight stored in the storage unit 42. Then, the identification unit 34 identifies a position on the display unit 11 based on the identified viewing angle and the information stored in the memory unit 42 that shows the relationship between the viewing angle and the position on the display unit 11, and identifies the position of the first image corresponding to that position as the point of focus that the user is looking at.

[0021] The focus control unit 35 acquires positional information of the point of focus that the user is fixated on in the first image from the identification unit 34, and controls the first imaging unit 40 so that the focus of the first imaging unit 40 matches the point of focus.

[0022] The image control unit 31, creation unit 32, imaging control unit 33, identification unit 34, and focus control unit 35 are processed by a processor such as a CPU (Central Processing Unit) in cooperation with a program. The image control unit 31, creation unit 32, imaging control unit 33, identification unit 34, and focus control unit 35 may be specially designed circuits. The image control unit 31, creation unit 32, imaging control unit 33, identification unit 34, and focus control unit 35 may be one circuit or different circuits. The storage unit 42 is a storage device, such as a non-volatile semiconductor memory such as flash memory.

[0023] Figure 2 shows an example of the arrangement of the image projection unit 10, the first imaging unit 40, and the second imaging unit 41 in Embodiment 1. As shown in Figure 2, the display unit 11, the first imaging unit 40, and the second imaging unit 41 are mounted, for example, on a spectacle-type frame 50 worn on the user's face. The display unit 11 is positioned in front of the user's eye 60. That is, the display unit 11 is mounted inside the rim 51 of the spectacle-type frame 50. A condensing lens 12 is positioned between the display unit 11 and the eye 60. The condensing lens 12 is also mounted inside the rim 51 of the spectacle-type frame 50. The image projection unit 10 includes the display unit 11 and the condensing lens 12. The user views the first image displayed on the display unit 11 through the condensing lens 12.

[0024] The first imaging unit 40 is attached, for example, to the bridge 52 of an eyeglass-type frame 50. As a result, the first imaging unit 40 is positioned in the center of both of the user's eyes 60, and by displaying the first image captured by the first imaging unit 40 on the display unit 11 and projecting it onto the eyes 60, the user can get the sensation of seeing directly with their eyes 60.

[0025] The second imaging unit 41 is positioned in front of the eye 60 so as not to obstruct the eye 60 from viewing the display unit 11, and is mounted on the spectacle-type frame 50. The second imaging unit 41 is mounted, for example, near the temple 53 of the spectacle-type frame 50.

[0026] [Method for identifying the point of focus by the identification unit] When the image control unit 31 starts displaying the first image on the display unit 11, the imaging control unit 33 causes the second imaging unit 41 to start taking pictures based on the control signal from the image control unit 31. Therefore, when a user wearing the image projection device 100 is looking at the first image displayed on the display unit 11, the second imaging unit 41 is taking pictures of the user's eye 60. In this case, first, the identification unit 34 acquires the captured image (second image) of the eye 60 taken by the second imaging unit 41 and identifies the position of the pupil 61 of the eye 60 as captured in the second image. Figures 3(a) and 3(b) show examples of the second image 72 acquired by the identification unit 34 in Embodiment 1. Figure 3(a) shows the case where the eye 60 is facing forward, and Figure 3(b) shows the case where the eye 60 is facing to the upper right. As shown in Figures 3(a) and 3(b), the second image 72 of the eye 60 captured by the second imaging unit 41 shows at least the pupil 61 and iris 62. The identification unit 34 identifies the area of ​​the pupil by performing image analysis on the second image 72, and identifies the center 65 of the identified area as the position of the pupil 61. For example, the identification unit 34 represents the second image 72 in a Cartesian coordinate system and identifies the position of the pupil 61 in XY coordinates. The position of the pupil 61 may also be identified by other methods.

[0027] Next, the identification unit 34 determines the line of sight angle of the eye 60 from the position of the identified pupil 61. The method for determining the line of sight angle of the eye 60 is described below.

[0028] Figure 4 shows the relationship between the position on the display unit 11 and the viewing angle in Embodiment 1. As shown in Figure 4, when the surface of the display unit 11 is represented in a Cartesian coordinate system, the central position O on the display unit 11 is the reference position (origin position) because it is the position where the center of the first image is displayed. The viewing direction of the eye 60 when looking at position O is defined as the reference direction. The viewing angle θ is the angle between the viewing direction of the eye 60 when looking at position A, which is a distance r from position O, and the reference direction. In this case, the relationship r = k1 × θ holds. k1 is a constant determined by the optical system of the image projection unit 10, including the display unit 11 and the condensing lens 12, and can be determined in advance from the design information of the optical system, etc. Therefore, the relationship r = k1 × θ is determined in advance and stored in the storage unit 42 as information showing the relationship between the position on the display unit 11 and the viewing angle when looking at that position. The value of k1 is 0.285 [mm / °] as an example.

[0029] Figure 5 is a flowchart showing an example of the creation of information indicating the relationship between pupil position and line of sight angle by the creation unit 32 in Embodiment 1. Figures 6(a) and 6(b) are diagrams showing the creation of information indicating the relationship between pupil position and line of sight angle by the creation unit 32 in Embodiment 1. As shown in Figure 5, when the user puts on the image projection device 100, the creation unit 32 instructs the image control unit 31 to display a target image having a target at the central position on the display unit 11 (step S10). That is, as shown in the upper part of Figure 6(a), a target image 73 in which the target 74 is located at the central position O on the display unit 11 is displayed on the display unit 11. The target image 73 is not limited to cases where the target 74 is a dark color such as black on a white background, but may also be, for example, a case where the target 74 is a bright spot on a dark color such as black. Position O is the origin position (0,0) when the surface of the display unit 11 is represented in a Cartesian coordinate system, as described above.

[0030] Next, the creation unit 32 acquires an eye image from the second imaging unit 41 of the eye 60 as it is looking at the target 74 displayed at position O, and identifies the position of the pupil 61 captured in the eye image (step S12). That is, as shown in the lower part of Figure 6(a), the creation unit 32 acquires an eye image 75 of the eye 60 as it is looking at the target 74 displayed at position O, and identifies the position T0 of the pupil 61 captured in the eye image 75. The method described in Figures 3(a) and 3(b) is used to identify the position T0. In order to use the position T0 of the pupil 61 as the reference position when it is looking at the target 74 displayed at position O, which is the origin position on the display unit 11, when the eye image 75 is represented in a Cartesian coordinate system, position T0 is set as the origin position (0,0). In addition, the creation unit 32 sets the line of sight angle θ0 when the pupil 61 is at position T0 as 0° in order to use the line of sight direction when it is looking at the target 74 displayed at position O as the reference direction.

[0031] Next, the creation unit 32 instructs the image control unit 31 to display a target image 73 having the target 74 at a position other than the center on the display unit 11 (step S14). For example, as shown in the upper diagram of Figure 6(b), the display unit 11 displays a target image 73 in which the target 74 is located at the upper right position B1 on the display unit 11. Position B1 is, for example, (x1, y1) on the display unit 11.

[0032] Next, the creation unit 32 acquires an eye image 75 from the second imaging unit 41, which is a photograph of the eye 60 looking at the target 74 displayed at position B1, and identifies the position of the pupil 61 captured in the eye image 75 (step S16). That is, as shown in the lower diagram of Figure 6(b), the creation unit 32 acquires an eye image 75, which is a photograph of the eye 60 looking at the target 74 displayed at position B1, and identifies the position T1 of the pupil 61 captured in the eye image 75. The method described in Figures 3(a) and 3(b) is used to identify the position T1. Position T1 is, for example, (X1, Y1).

[0033] Next, the creation unit 32 identifies the line of sight angle θ1 corresponding to the position T1 of the pupil 61 (step S18). For example, the line of sight angle θ1 is identified using the information r = k1 × θ stored in the memory unit 42. That is, since the line of sight angle θ1 is the line of sight angle when looking at the target 74 displayed at position B1 (x1, y1), √(x1２ +y1 ２ ) = k1 × θ1, and the line of sight angle θ1 is θ1 = √(x1 ２ +y1 ２ ) / k1 can be identified.

[0034] Next, the creation unit 32 determines whether or not all target images 73 have been displayed on the display unit 11 (step S20). If the display of all target images 73 has not been completed (step S20: No), the creation unit 32 returns to step S14. On the other hand, if the display of all target images 73 has been completed (step S20: Yes), the creation unit 32 proceeds to step S22.

[0035] The creation unit 32 creates information showing the relationship between pupil positions and line of sight angles from the positions of multiple pupils 61 and their corresponding line of sight angles obtained by repeatedly executing steps S14 to S20, and stores it in the storage unit 42 (step S22). For example, the creation unit 32 creates information showing the relationship between pupil positions T0 to Tn and line of sight angles θ0 to θn, as shown in Table 1, and stores it in the storage unit 42.

[0036] The identification unit 34 uses the position of the pupil 61 identified in the second image 72 captured by the second imaging unit 41 while the first image is displayed on the display unit 11, and information like that shown in Table 1 stored in the storage unit 42, to determine the line of sight angle of the eye 60 as captured in the second image 72. For example, if the identified position of the pupil 61 is T2 (X2, Y2), the identification unit 34 refers to the information like that shown in Table 1 stored in the storage unit 42 to determine that the line of sight angle of the eye 60 is θ2.

[0037] Next, the identification unit 34 uses the identified line of sight angle of the eye 60 and the information stored in the memory unit 42 that shows the relationship between the line of sight angle and the position on the display unit 11 to identify the point of fixation that the user is looking at in the first image. Figures 7(a) to 7(d) show the identification of the point of fixation by the identification unit 34 in Embodiment 1. For example, suppose the position of the pupil 61 identified by the identification unit 34 is T2 (X2, Y2) and the line of sight angle of the eye 60 is θ2. In this case, first, the identification unit 34 calculates r2 = k1 × θ2 from the information stored in the memory unit 42 that shows the relationship between the line of sight angle and the position on the display unit 11. As a result, as shown in Figure 7(a), it can be seen that the point of fixation is located on the circumference of a circle at a distance of r2 = k1 × θ2 from the central origin position on the display unit 11. Next, the identification unit 34 determines the angle α of position T2 (X2, Y2) of the pupil 61 identified in the second image 72 from the X-axis in the second image 72, as shown in Figure 7(b). That is, √(X2 ２ +Y2 ２ ) × cosα = X², or √(X²) ２ +Y2 ２ The angle α is determined from ) × sinα = Y². Next, the identification unit 34 identifies a position U (x², y²) in the display unit 11 where the distance from the origin is r² and the angle from the X-axis is α, as shown in Figure 7(c). Then, as shown in Figure 7(d), the identification unit 34 identifies the position of the first image 71 corresponding to position U as the gaze point 70 that the user is fixated on.

[0038] [Focus adjustment of the first imaging unit] The focus control unit 35 acquires position information of the gaze point 70 that the user is fixated on in the first image 71 from the identification unit 34. The focus control unit 35 controls the first imaging unit 40 so that the focus of the first imaging unit 40 matches the gaze point 70.

[0039] [Method of Controlling Focus Adjustment] FIG. 8 is a flowchart showing an example of the control of the control unit 30 in the first embodiment. The flowchart of FIG. 8 is executed after the user wears the image projection device 100, information indicating the relationship between the pupil position and the line-of-sight angle is created by the creation unit 32, stored in the storage unit 42, and the first imaging unit 40 starts shooting in front of the user. As shown in FIG. 8, the image control unit 31 acquires the first image 71 obtained by shooting in front of the user from the first imaging unit 40 and causes the display unit 11 to display the first image 71 (step S30).

[0040] Next, the specifying unit 34 acquires, from the second imaging unit 41, a second image 72 of the eye 60 when looking at the first image 71 displayed on the display unit 11 (step S32). For example, the second image 72 as shown in FIGS. 3(a) and 3(b) is acquired. As shown in FIG. 2, since the second imaging unit 41 is provided for each of the left and right eyes 60, the specifying unit 34 acquires the second image 72 of each of the left and right eyes 60 from the second imaging unit 41.

[0041] Next, the specifying unit 34 specifies the line-of-sight angle of the eye 60 shown in the second image 72 acquired in step S32 (step S34). As described above, the specifying unit 34 specifies the position of the pupil 61 of the eye 60 shown in the second image 72, and uses the specified position of the pupil 61 and information such as Table N-1 stored in the storage unit 42 to specify the line-of-sight angle of the eye 60. Since the specifying unit 34 has acquired the second image 72 of each of the left and right eyes 60, the line-of-sight angle is specified for each of the left and right eyes ⑥.

[0042] Next, the specifying unit 34 specifies the fixation point 70 that the user is fixating on in the first image 71 by using the line-of-sight angle of the eye 60 specified in step S34 (step S36). As shown in FIGS. 7(a) to 7(d), the specifying unit 34 uses the specified line-of-sight angle of the eye 60, the information indicating the relationship between the line-of-sight angle stored in the storage unit 42 and the position on the display unit 11 (r = k1 × θ), to specify the position U on the display unit 11. Since the specifying unit 34 specifies the line-of-sight angle for each of the left and right eyes 60, the position on the display unit 11 is specified for each of the left and right eyes 60, and the average position is specified as the position U on the display unit 11. Then, the specifying unit 34 specifies the position of the first image 71 corresponding to the specified position U on the display unit 11 as the fixation point 70 that the user is fixating on. Note that the specifying unit 34 may obtain the average value of the line-of-sight angles of each of the left and right eyes 60 and specify the position U on the display unit 11 from this average value of the line-of-sight angles.

[0043] Next, the focus control unit 35 acquires the position information of the fixation point 70 of the first image 71 from the specifying unit 34, and controls the first imaging unit 40 so that the focus of the first imaging unit 40 matches the fixation point 70 (step S38).

[0044] Next, the image control unit 31 determines whether to continue displaying the first image on the display unit 11 (step S40). If the display of the first image is to be continued (step S40: Yes), the process returns to step S30. On the other hand, if the display of the first image is not continued and ends (step S40: No), this process ends.

[0045] According to Embodiment 1, as shown in Figures 1 and 2, the system includes a first imaging unit 40 that photographs the area around the user and a second imaging unit 41 that photographs the user's eyes 60. As shown in Figure 8, the identification unit 34 displays a first image 71 of the area around the user, captured by the first imaging unit 40, on the display unit 11, and acquires a second image 72 from the second imaging unit 41, which is a photograph of the user's eyes 60 when they are looking at the first image 71 displayed on the display unit 11 (step S32). Then, based on the acquired second image 72, the identification unit 34 identifies the point of focus 70 that the user is fixated on in the first image 71 (steps S34, S36). The focus control unit 35 controls the first imaging unit 40 so that it focuses on the point of focus 70 identified by the identification unit 34 (step S38). This allows the display unit 11 to display the first image 71 in focus on the point of fixation 70, even if the user changes the position of the point of fixation 70 in the first image 71. Therefore, since the first image 71 is projected in focus on the part the user wants to see, the user can view the first image 71 in a way that is close to how it would actually appear to the eye 60.

[0046] In the first embodiment, the memory unit 42 stores information such as Table 1 (first information in the claims) that shows the relationship between pupil position and gaze angle. As shown in Figure 8, the identification unit 34 identifies the position of the pupil 61 captured in the second image 72 acquired from the second imaging unit 41, and identifies the gaze angle of the eye 60 from the identified pupil 61 position and the information such as Table 1 (step S34). Then, the identification unit 34 uses the identified gaze angle to identify the point of fixation 70 that the user is fixating on in the first image 71 (step S36). This makes it possible to identify the point of fixation 70 that the user is fixating on in the first image 71 with high accuracy and ease.

[0047] In the first embodiment, the memory unit 42 stores information such as r = k1 × θ (third information in the claims) which shows the relationship between the viewing angle and the position on the display unit 11. As shown in Figures 7(a) to 7(d), the identification unit 34 uses the identified viewing angle and information such as r = k1 × θ to identify the position U on the display unit 11, and identifies the point of focus 70 that the user is fixated on in the first image 71 from the identified position U on the display unit 11. This makes it possible to identify the point of focus 70 that the user is fixated on in the first image 71 with high accuracy and ease.

[0048] Furthermore, in Embodiment 1, as shown in Figures 5 and 6(a) and 6(b), the creation unit 32 projects multiple target images 73, each having a target 74 at a different position, onto the eye 60 when the user wears the image projection device 100, and acquires multiple eye images 75 captured by the second imaging unit 41 while the user is looking at the targets 74 in the multiple target images 73. The creation unit 32 then identifies multiple pupils 61 captured in the multiple eye images 75 and multiple line-of-sight angles corresponding to the multiple pupils 61, thereby creating information showing the relationship between pupil position and line-of-sight angle (first information in the claims) as shown in Table 1 and storing it in the storage unit 42. This makes it possible to obtain information showing the appropriate relationship between pupil position and line-of-sight angle for each user wearing the image projection device 100. Therefore, by using this information to identify the point of fixation 70 of the first image 71, the point of fixation 70 of the first image 71 can be identified with high accuracy regardless of individual differences in the structure of the user's eye 60.

[0049] Furthermore, in Embodiment 1, the creation unit 32 identifies multiple line-of-sight angles corresponding to the positions of multiple pupils 61 projected onto multiple eye images 75, using the position of the visual target 74 on the display unit 11 and information such as r = k1 × θ (the fifth piece of information in the claims) that shows the relationship between the line-of-sight angle and the position on the display unit 11, thereby creating information showing the relationship between the pupil position and the line-of-sight angle (the first piece of information in the claims). As a result, for each user wearing the image projection device 100, information showing the relationship between the pupil position and the line-of-sight angle is created by adjusting the line-of-sight angle and the position on the display unit 11. By using this information to identify the point of fixation 70 in the first image 71, the point of fixation 70 in the first image 71 can be identified with high accuracy.

[0050] Figure 9 is a block diagram of the image projection device 200 according to Embodiment 2. As shown in Figure 9, in the image projection device 200, the image projection unit 10a includes a light source 13, a scanning unit 14, a reflection unit 15, and a projection unit 16 instead of a display unit 11. The other configurations are the same as in Embodiment 1, so their explanation is omitted.

[0051] Based on instructions from the image control unit 31, the light source 13 emits a ray 80 (laser light) which is monochromatic light such as red laser light (wavelength: approximately 610 nm to 660 nm), green laser light (wavelength: approximately 515 nm to 540 nm), or blue laser light (wavelength: approximately 440 nm to 480 nm), or combined light obtained by combining these lights. As an example of a light source 13 that emits red, green, and blue laser light, an example of a light source that integrates RGB (red, green, and blue) laser diode chips and a three-color combining device can be cited.

[0052] The scanning unit 14 (scanner) is, for example, a MEMS (Micro Electro Mechanical System) mirror, and scans the light ray 80 in a two-dimensional direction based on instructions from the image control unit 31. The light ray 80 scanned by the scanning unit 14 is reflected by the reflecting unit 15 toward the projection unit 16, and then projected onto the user's eye 60 by the projection unit 16.

[0053] Figure 10 shows the optical system of the image projection device 200 according to Embodiment 2. As shown in Figure 10, the image projection device 200 is a retinal projection type head-mounted display that utilizes Maxwell's vision, in which light rays 80 for viewing images are directly projected onto the user's retina 63.

[0054] The light source 13 emits a light ray 80 based on instructions from the image control unit 31 (see Figure 9). The light ray 80 passes through the lens 17. The lens 17 is a focusing lens that converts the light ray 80 from diffuse light to focused light. The light ray 80 that has passed through the lens 17 enters the scanning unit 14 in a focused state. The scanning unit 14 scans the light ray 80 in two dimensions, horizontally and vertically, based on instructions from the image control unit 31 (see Figure 9).

[0055] Multiple light rays 80, scanned in a two-dimensional direction by the scanning unit 14 and emitted from the scanning unit 14 in different directions at different times, are incident on the reflecting unit 15. Each of the multiple light rays 80 is focused before reaching the reflecting unit 15 and then becomes diffused light before being incident on the reflecting unit 15. The reflecting unit 15 is a concave mirror having a reflective surface made of a curved surface such as a freeform surface, and has positive focusing power. Therefore, each of the multiple light rays 80 is converted from diffused light to approximately parallel light by being reflected by the reflecting unit 15. The lens 17 is provided between the light source 13 and the scanning unit 14 so that the light rays 80 reflected by the reflecting unit 15 become approximately parallel light.

[0056] Since the reflective section 15 has positive focusing power, the multiple light rays 80 reflected by the reflective section 15 converge at the convergence point 20 in front of the projection section 16. An optical component 18 is provided at the convergence point 20. The optical component 18 is a focusing lens that converts each of the multiple light rays 80 from substantially parallel light to focused light. Each of the multiple light rays 80 that have passed through the optical component 18 converges in front of the projection section 16 and then becomes diffused light before entering the projection section 16.

[0057] The projection unit 16 is positioned in front of the user's eye 60. The projection unit 16 is a concave mirror having a reflective surface made of a curved surface such as a freeform surface, and has positive focusing power. Therefore, each of the multiple light rays 80 is reflected by the projection unit 16, converted from diffuse light to approximately parallel light, and incident on the user's eye 60. The multiple light rays 80 pass through the pupil 61 of the user's eye 60 and converge at a convergence point 21 inside the eye 60. The convergence point 21 is located, for example, in or near the lens 64. Each of the multiple light rays 80 is converted from approximately parallel light to focused light by the lens 64, etc., and focuses near the retina 63. As the multiple light rays 80 are projected onto the retina 63, the user can see the first image 71.

[0058] When the pupil 61 is facing directly toward the user's face, one of the multiple light rays 80, ray 80a, enters the pupil 61 almost directly in front of it. That is, the direction in which ray 80a enters the pupil 61 corresponds to the line of sight of the eye 60. At the central position Q on the retina 63, ray 80a corresponding to the central pixel P1 of the first image 71 projected onto the retina 63 is projected. To the left of position Q, ray 80b corresponding to pixel P2 is projected, and to the right of position Q, ray 80c corresponding to pixel P3 is projected. The image viewed by the user is an inverted version of the image projected onto the retina 63. Therefore, the user views the first image in which pixel P1 is in the center, pixel P2 is on the right, and pixel P3 is on the left. Thus, the coordinate system viewed by the user corresponds to the coordinate system in which the user views the projection unit 16.

[0059] When the user attempts to view pixel P2 of the first image 71 projected onto the retina 63, they move their pupil 61 to the right relative to their face. In this case, ray 80b, one of the multiple light rays 80, enters the pupil 61 directly from almost directly in front of it. That is, the direction in which ray 80b enters the pupil 61 corresponds to the line of sight of the eye 60. Also, when the user attempts to view pixel P3 of the first image 71 projected onto the retina 63, they move their pupil 61 to the left relative to their face. In this case, ray 80c, one of the multiple light rays 80, enters the pupil 61 directly from almost directly in front of it. That is, the direction in which ray 80c enters the pupil 61 corresponds to the line of sight of the eye 60.

[0060] The reference direction is defined as the line of sight of the eye 60 when the pupil 61 is facing forward and viewing the pixel P1 onto which the light ray 80a is projected. That is, the direction in which the light ray 80a enters the pupil 61 is the reference direction. The line of sight angle φ is defined as the angle between the line of sight of the eye 60 when viewing a pixel onto which a light ray 80 other than light ray 80a is projected, and the above-mentioned reference direction. Here, as an example, we will explain the case when viewing the pixel P2 onto which the light ray 80b is projected. In this case, the line of sight angle φ is the angle between the direction in which the light ray 80b enters the pupil 61 (the line of sight of the eye 60) and the above-mentioned reference direction. In the scanning unit 14, the scanning angle δ is defined as the angle between the light ray 80a and the light ray 80b. In this case, the relationship δ = k² × φ holds true. k2 is a constant determined by the optical system of the image projection unit 10a, which includes the scanning unit 14, the reflecting unit 15, the projection unit 16, the lens 17, and the optical components 18, and can be determined in advance from the design information of the optical system. The memory unit 42 has the relationship δ = k2 × φ stored in advance as information showing the relationship between the line of sight angle and the scanning angle. The value of k2 is 1.1 as an example.

[0061] Figure 11 shows an example of the arrangement of the scanning unit 14, reflective unit 15, projection unit 16, optical component 18, first imaging unit 40, and second imaging unit 41 in Embodiment 2. As shown in Figure 11, the scanning unit 14, reflective unit 15, projection unit 16, and optical component 18 are attached to the spectacle-type frame 50. The scanning unit 14, reflective unit 15, and optical component 18 are attached, for example, to the temples 54 of the spectacle-type frame 50. The projection unit 16 is attached, for example, to the inside of the rim 51 of the spectacle-type frame 50 and is positioned in front of the user's eye 60. Although not shown, the light source 13 and lens 17 may also be attached to the temples 54 of the spectacle-type frame 50. The second imaging unit 41 is attached, for example, near the bridge 52 of the spectacle-type frame 50. The other configurations are the same as in Figure 2 of Embodiment 1, so their description is omitted.

[0062] [Method for identifying the point of focus by the identification unit] Similar to Embodiment 1, when a user wearing the image projection device 200 is looking at the first image 71 projected onto the retina 63, the second imaging unit 41 is photographing the user's eye 60. Therefore, first, the identification unit 34 acquires the second image 72 of the eye 60 taken by the second imaging unit 41, similar to Embodiment 1, and identifies the position of the pupil 61 of the eye 60 as captured in the second image 72.

[0063] Next, the identification unit 34 identifies the line of sight angle of the eye 60 from the identified pupil 61 position. Here, in Embodiment 2 as well, the creation unit 32 projects multiple target images 73 having targets 74 at different positions onto the retina 63, similar to the case in Figure 5, to create information showing the relationship between pupil position and line of sight angle as shown in Table 1, and stores it in the storage unit 42. That is, the creation unit 32 projects multiple target images 73 having targets 74 at different positions onto the retina 63 and acquires multiple eye images 75 of the eye 60 when the user is looking at the targets 74 of the multiple target images 73. Then, the creation unit 32 identifies the position of the pupil 61 in the acquired multiple eye images 75. Here, as described above, the storage unit 42 has information showing the relationship between the line of sight angle φ and the scanning angle δ (δ = k² × φ) stored in advance. Also, since the scanning unit 14 projects an image onto the retina 63 by two-dimensional scanning, it has information showing the relationship between the scanning angle δ of the scanning unit 14 and the position on the image corresponding to the scanning angle δ in advance. From this information, information showing the relationship between the line of sight angle φ and the position on the image is obtained in advance. Therefore, from this information and the position of the target 74 in the target image 73, the line of sight angle φ corresponding to the position of the target 74 in multiple target images 73 can be determined. Thus, the creation unit 32 can determine the line of sight angle φ corresponding to the position of the pupil 61 in multiple eye images 75, create information like Table 1 showing the relationship between pupil position and line of sight angle, and store it in the storage unit 42.

[0064] The identification unit 34 uses the position of the pupil 61 identified in the second image 72 captured by the second imaging unit 41 when the first image 71 is projected onto the retina 63, and the information shown in Table 1 stored in the memory unit 42, to determine the line of sight angle of the eye 60 as captured in the second image 72.

[0065] Next, the identification unit 34 identifies the scanning angle of the scanning unit 14 that corresponds to the line of sight angle of the identified eye 60. Since the storage unit 42 stores information such as δ = k² × φ, which shows the relationship between the line of sight angle φ and the scanning angle δ, this information is used to identify the scanning angle that corresponds to the identified line of sight angle.

[0066] Next, the identification unit 34 identifies the point of fixation 70 that the user is fixated on in the first image 71 projected onto the retina 63 based on the identified scanning angle. As described above, since the first image 71 is projected onto the retina 63 by scanning the scanning unit 14 in two dimensions, the identification unit 34 has in advance information showing the relationship between the scanning angle of the scanning unit 14 and the position on the first image 71 that corresponds to that scanning angle. Therefore, the identification unit 34 can identify the point of fixation 70 that the user is fixated on in the first image 71 based on the identified scanning angle of the scanning unit 14.

[0067] [Focus Adjustment Control Method] Figure 12 is a flowchart showing an example of the control of the control unit 30 in Embodiment 2. The flowchart in Figure 12 is executed after the user puts on the image projection device 200, the creation unit 32 creates information showing the relationship between the pupil position and the line of sight angle and stores it in the storage unit 42, and the first imaging unit 40 starts capturing the area in front of the user, similar to Embodiment 1. As shown in Figure 12, the image control unit 31 controls the light source 13 and the scanning unit 14 to project a plurality of light rays 80 onto the retina 63, thereby projecting the first image 71, which is a capture of the area in front of the user acquired from the first imaging unit 40, onto the retina 63 (step S50).

[0068] Next, the identification unit 34 acquires a second image 72 of the eye 60, which was captured by the second imaging unit 41 when the first image 71 is projected onto the retina 63 (step S52). As shown in Figure 11, since a second imaging unit 41 is provided for each of the left and right eyes 60, the identification unit 34 acquires a second image 72 of each of the left and right eyes 60 from the second imaging unit 41.

[0069] Next, the identification unit 34 identifies the line of sight angle of the eye 60 as captured in the second image 72 acquired in step S52 (step S54). As described above, in Embodiment 2 as well, the memory unit 42 stores information as shown in Table 1, so the identification unit 34 identifies the position of the pupil 61 of the eye 60 as captured in the second image 72, and uses the identified position of the pupil 61 and the information as shown in Table 1 stored in the memory unit 42 to identify the line of sight angle of the eye 60. Since the identification unit 34 has acquired the second image 72 for both the left and right eyes 60, it identifies the line of sight angle for each of the left and right eyes 60.

[0070] Next, the identification unit 34 identifies the scanning angle of the scanning unit 14 from the viewing angle identified in step S54 (step S56). As described above, the storage unit 42 stores information such as δ = k² × φ, which shows the relationship between the viewing angle φ and the scanning angle δ. Therefore, the identification unit 34 uses the above information stored in the storage unit 42 to identify the scanning angle corresponding to the viewing angle identified in step S54. Since the identification unit 34 identifies the viewing angle for each of the left and right eyes 60, it may also determine the average value of the viewing angles for each of the left and right eyes 60 and identify the scanning angle corresponding to the average value of the viewing angles. Alternatively, the identification unit 34 may determine the first and second scanning angles corresponding to the viewing angles of each of the left and right eyes 60, and identify the average value of the first and second scanning angles as the scanning angle.

[0071] Next, the identification unit 34 identifies the point of fixation 70 that the user is fixated on in the first image 71 projected onto the retina 63, based on the scanning angle of the scanning unit 14 identified in step S56 (step S58). As described above, since the scanning unit 14 performs a two-dimensional scan and projects the first image 71 onto the retina 63, it has prior information indicating the relationship between the scanning angle of the scanning unit 14 and the position on the first image 71 corresponding to that scanning angle. Therefore, the identification unit 34 can identify the point of fixation 70 that the user is fixated on in the first image 71, based on the scanning angle of the scanning unit 14 identified in step S56.

[0072] Next, the focus control unit 35 acquires positional information of the fixation point 70 of the first image 71 from the identification unit 34 and controls the first imaging unit 40 so that the focus of the first imaging unit 40 matches the fixation point 70 (step S60).

[0073] Next, the image control unit 31 determines whether or not to continue projecting the first image 71 onto the retina 63 (step S62). If it decides to continue projecting the first image 71 (step S62: Yes), it returns to step S50. On the other hand, if it decides to terminate without continuing the projection of the first image 71 (step S62: No), it terminates this process.

[0074] In Embodiment 2, as shown in Figure 12, the identification unit 34 acquires a second image 72 from the second imaging unit 41 (step S52). Then, based on the acquired second image 72, the identification unit 34 identifies the point of focus 70 that the user is fixated on in the first image 71 (steps S54 to S58). The focus control unit 35 controls the first imaging unit 40 so that the first imaging unit 40 is in focus on the point of focus 70 identified by the identification unit 34 (step S60). As a result, similar to Embodiment 1, even if the user changes the position of the point of focus 70 that they are fixated on in the first image 71, the first image 71 with the point of focus 70 can be displayed on the display unit 11.

[0075] Furthermore, in Embodiment 2, similar to Embodiment 1, the memory unit 42 stores information such as Table 1 (first information in the claims) that shows the relationship between the pupil position and the line of sight angle. As shown in Figure 12, the identification unit 34 identifies the position of the pupil 61 captured in the second image 72 acquired from the second imaging unit 41, and identifies the line of sight angle of the eye 60 from the identified pupil 61 position and the above information (step S54). Then, the identification unit 34 uses the identified line of sight angle to identify the point of fixation 70 that the user is fixating on in the first image 71 (steps S56, S58). This makes it possible to identify the point of fixation 70 that the user is fixating on in the first image 71 with high accuracy and ease.

[0076] In addition, in the second embodiment, information such as δ = k² × φ (second information in the claims), which shows the relationship between the line of sight angle and the scanning angle, is also stored. As shown in Figure 12, the identification unit 34 identifies the line of sight angle from the position of the pupil 61 captured in the second image 72 and the information shown in Table 1 (step S54), and identifies the scanning angle corresponding to the identified line of sight angle from the information such as δ = k² × φ (step S56). Then, the identification unit 34 identifies the point of fixation 70 that the user is fixating on in the first image 71 from the identified scanning angle (step S58). This makes it possible to identify the point of fixation 70 that the user is fixating on in the first image 71 with high accuracy and ease.

[0077] Furthermore, in Embodiment 2, the creation unit 32 identifies multiple line-of-view angles corresponding to the positions of multiple pupils 61 projected onto multiple eye images 75 from the positions of targets 74 in multiple target images 73 and information indicating the relationship between the position on the multiple target images 73 and the line-of-view angle (the fourth piece of information in the claims), thereby creating information indicating the relationship between pupil position and line-of-view angle (the first piece of information in the claims). As a result, information indicating the relationship between pupil position and line-of-view angle is created for each user wearing the image projection device 100, and by using this information to identify the point of fixation 70 in the first image 71, the point of fixation 70 in the first image 71 can be identified with high accuracy.

[0078] In the second embodiment, as described in Patent Document 1 (Japanese Patent Application Publication No. 2023-76137), a drive unit for moving the reflecting part 15 may be provided, and the drive unit may be controlled to move the reflecting part so that the position in which multiple light rays 80 incident on the optical component 18 changes according to the direction in which the pupil 61 moves. Such control may be performed in parallel with steps S54 to S60 in Figure 12.

[0079] While Examples 1 and 2 show examples of glasses-type image projection devices worn on the user's face, other configurations such as head-mounted image projection devices worn on the user's head are also acceptable.

[0080] Although Examples 1 and 2 show examples of applying the present invention to an image projection device, it can also be applied to imaging devices such as cameras. That is, an imaging device such as a camera may be equipped with an imaging unit that photographs the user's eyes as they look through the viewfinder, and the focal position of the captured image may be changed according to the line of sight angle of the eyes photographed by the imaging unit. Alternatively, in the case of an imaging device, the image projected onto the user's eyes may be converted into an image in which the point of focus is located in the center, and the converted image may be projected onto the user's eyes.

[0081] Although embodiments of the present invention have been described in detail above, the present invention is not limited to these specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention as described in the claims.

Claims

1. An image projection device comprising: a first imaging unit for capturing images of the user's surroundings; an image projection unit for projecting a first image of the user's surroundings captured by the first imaging unit onto the user's eyes; a second imaging unit for capturing images of the user's eyes; a identification unit for identifying a point of focus in the first image that the user is fixated on, based on a second image of the eyes captured by the second imaging unit; and a focus control unit for controlling the first imaging unit so that its focus aligns with the point of focus.

2. The image projection device according to claim 1, comprising a storage unit that stores first information indicating the relationship between pupil position and line of sight angle, wherein the identification unit identifies the line of sight angle of the eye from the position of the pupil projected in the second image and the first information, and identifies the point of fixation in the first image that the user is fixating on using the identified line of sight angle of the eye.

3. The image projection device according to claim 2, wherein the image projection unit comprises a scanning unit that scans a light ray emitted from a light source, and a projection unit that, after the light ray is scanned by the scanning unit, converges a plurality of light rays emitted in different directions from the scanning unit to a convergence point inside the eye, and then projects the first image onto the retina of the eye, the storage unit stores second information indicating the relationship between the line of sight angle and the scanning angle of the scanning unit, and the identification unit identifies the scanning angle of the scanning unit corresponding to the identified line of sight angle of the eye from the second information, and identifies the point of fixation in the first image that the user is fixated on from the identified scanning angle of the scanning unit.

4. The image projection device according to claim 2, comprising: an image projection unit comprising: a display unit for displaying the first image; and a focusing lens disposed between the display unit and the eye; the storage unit for storing third information indicating the relationship between the line of sight angle and the position on the display unit; and the identification unit for identifying the position on the display unit using the identified line of sight angle of the eye and the third information, and for identifying the point of fixation in the first image that the user is fixating on from the identified position on the display unit.

5. The image projection device according to claim 2, further comprising a creation unit that, when the user wears the image projection device, projects a plurality of target images having targets at different positions onto the eye, acquires a plurality of eye images captured by the second imaging unit while the user is looking at the targets in the plurality of target images, and creates the first information by identifying the positions of a plurality of pupils captured in the plurality of eye images and a plurality of line-of-sight angles corresponding to the positions of the plurality of pupils, and stores it in the storage unit.

6. The image projection device according to claim 5, wherein the image projection unit comprises a scanning unit that scans a light ray emitted from a light source, and a projection unit that, after the light ray is scanned by the scanning unit, converges a plurality of light rays emitted in different directions from the scanning unit to a convergence point inside the eye, and projects the plurality of target images onto the retina of the eye, and the creation unit creates the first information by specifying the plurality of line-of-view angles corresponding to the positions of the plurality of pupils captured in the plurality of eye images from the position of the target in the plurality of target images and fourth information indicating the relationship between the position on the plurality of target images and the line-of-view angle.

7. The image projection device according to claim 5, wherein the image projection unit comprises a display unit for displaying the plurality of target images and a focusing lens disposed between the display unit and the eye, and the creation unit creates the first information by specifying the plurality of line-of-view angles corresponding to the positions of the plurality of pupils projected onto the plurality of eye images from the position of the target on the display unit and fifth information indicating the relationship between the position on the display unit and the line-of-view angle.

Images