Eye-line estimation device and eye-line estimation method

Abstract

To improve estimation accuracy of a sight line direction of a user.SOLUTION: A sight line estimation device 10 includes: a video acquisition unit 22 which acquires a video obtained by capturing a user; an eye direction detection unit 24 which detects directions of a right eye and a left eye of the user from the video acquired by the video acquisition unit 22; a memory unit 34 which stores dominant eye information showing which of the right eye or the left eye a dominant eye of the user is; an eye direction correlation unit 26 which produces correlation information showing correlation between the direction of the dominant eye and the direction of a non-dominant eye of the user using the direction of each of the right eye and the left eye of the user detected by the eye direction detection unit 24 from the video including both eyes of the user and the dominant eye information stored in the memory unit 34; and a sight line direction estimation unit 28 which estimates a sight line direction of the user using the directions of the eyes of the user detected by the eye direction detection unit 24 and the correlation information produced by the eye direction correlation unit 26.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a gaze direction estimation device and a gaze direction estimation method.

Background Art

[0002] Techniques for detecting a user's line of sight and estimating the user's fixation point are known. In cases where the user has a squint, etc., a technique has been proposed for determining the user's dominant eye and estimating the user's gaze direction based on the line-of-sight direction of the dominant eye and the line-of-sight direction of the non-dominant eye (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a situation where the user is looking away, etc., it may be possible to identify only the line-of-sight direction of one eye of the user. The technique of Patent Document 1 is premised on using the line-of-sight directions of both eyes of the user, and does not touch on the case where only the line-of-sight direction of one eye can be identified.

[0005] The present invention has been made in view of the above circumstances, and an object thereof is to provide a technique for improving the estimation accuracy of the user's line-of-sight direction.

Means for Solving the Problems

[0006] A gaze estimation device according to one aspect of the present invention includes: an image acquisition unit that acquires images captured by an imaging device that images a user; an eye direction detection unit that detects the orientation of the user's right and left eyes from the images acquired by the image acquisition unit; a storage unit that stores dominant eye information indicating whether the user's dominant eye is the right or left eye; an eye direction correlation unit that generates correlation information showing the correlation between the orientation of the user's dominant eye and the orientation of the non-dominant eye using the orientations of the user's right and left eyes detected by the eye direction detection unit from images including both of the user's eyes, and the dominant eye information stored in the storage unit; and a gaze direction estimation unit that estimates the user's gaze direction using the orientation of the user's eyes detected by the eye direction detection unit and the correlation information generated by the eye direction correlation unit.

[0007] Another aspect of the present invention is a gaze estimation method. This method comprises the steps of: acquiring an image captured by an imaging device that images a user; detecting the orientation of the user's right and left eyes from the acquired image; generating correlation information showing the correlation between the orientation of the user's dominant and non-dominant eyes, using the orientations of the user's right and left eyes detected from the image including both of the user's eyes, and dominant eye information indicating whether the user's dominant eye is the right or left eye; and estimating the user's gaze direction using the orientation of the user's eyes detected from the user's image and the generated correlation information.

[0008] Furthermore, any combination of the above components, or any substitution of components or expressions of the present invention between methods, apparatus, systems, etc., is also valid as an embodiment of the present invention. [Effects of the Invention]

[0009] According to the present invention, the accuracy of estimating the user's gaze direction can be improved. [Brief explanation of the drawing]

[0010] [Figure 1] This diagram schematically illustrates how the gaze estimation device according to the embodiment is used. [Figure 2]This diagram schematically shows the gaze direction and eye orientation of a user without strabismus. [Figure 3] This diagram schematically shows the gaze direction and eye orientation of a user with strabismus. [Figure 4] This is a schematic block diagram showing the functional configuration of the gaze estimation device according to the embodiment. [Figure 5] This diagram schematically illustrates one example of a method for detecting the direction of the user's eyes. [Figure 6] Figures 6(a) and 6(b) schematically show examples of images displayed as virtual images. [Figure 7] This diagram schematically illustrates one example of how a user's dominant eye is determined. [Figure 8] This flowchart shows an example of a gaze estimation method according to an embodiment. [Figure 9] This flowchart shows an example of a method for determining the dominant eye according to an embodiment. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described below with reference to the drawings. The specific numerical values ​​and other details shown in these embodiments are merely examples to facilitate understanding of the invention and do not limit the present invention unless otherwise specified. Elements not directly related to the present invention are omitted from the drawings.

[0012] Figure 1 is a schematic diagram showing how the gaze estimation device 10 according to an embodiment is used. The gaze estimation device 10 can be implemented, for example, as an in-vehicle device such as a drive recorder or driver monitor mounted on a vehicle 70. The gaze estimation device 10 is installed, for example, on the dashboard 76 of the vehicle 70. The gaze estimation device 10 may also be installed on the steering column of the vehicle 70.

[0013] The line-of-sight estimation device 10 acquires the video of the user 40 imaged by the imaging device 12, and estimates the line-of-sight direction of the user 40 from the acquired video. The line-of-sight estimation device 10 guides the line-of-sight direction of the user 40 by controlling the operation of the display device 14.

[0014] The imaging device 12 is a camera that images the user 40. The imaging device 12 is attached to, for example, the inner mirror 72 of the vehicle 70, and the angle of view is set so as to image the rear of the passenger compartment from the inner mirror 72. The imaging device 12 is set with an angle of view such that when the user 40 faces forward in the vehicle 70, both the right eye and the left eye of the user 40 (that is, both eyes) can be imaged simultaneously.

[0015] The display device 14 is a so-called head-up display, and displays a virtual image 16 in front of the user 40. The display device 14 projects display light 18 toward, for example, the windshield 74 of the vehicle 70, and displays the virtual image 16 in front of the vehicle 70 in the traveling direction. The user 40 can visually recognize the virtual image 16 superimposed on the real scenery through the windshield 74. The virtual image 16 appears, for example, at a position about 2 m to 10 m away from the user 40 in the forward direction. The display device 14 is provided, for example, inside the dashboard 76 of the vehicle 70.

[0016] The purpose of this embodiment is to improve the estimation accuracy of the line-of-sight direction of the user 40 when the user 40 has a perspective or when only one eye of the user 40 is included in the imaged video because the user 40 is looking sideways or the like. Hereinafter, the line-of-sight direction of the user 40 when there is no perspective and when there is a perspective on the user 40 will be described.

[0017] FIG. 2 schematically shows the line-of-sight direction 50 of user 40 without squint and the orientations 52L, 52R of the eyes, showing the case where user 40 is gazing at the fixation point 60 of the front object 20 in a front view. In the present embodiment, the line-of-sight direction 50 of user 40 can be defined as the direction from the midpoint 44 between the right eye 42R and the left eye 42L of user 40 to the fixation point 60. When there is no squint in user 40, the orientation 52R of the right eye 42R of user 40 is the direction from the right eye 42R to the fixation point 60, and the orientation 52L of the left eye 42L of user 40 is the direction from the left eye 42L to the fixation point 60.

[0018] There is a convergence angle α between the orientation 52R of the right eye and the orientation 52L of the left eye. The convergence angle α depends on the interpupillary distance D between the two eyes of user 40 and the distance L from user 40 to the object 20, and α ≒ 2tan -1 (D / 2L). The interpupillary distance D is generally 60 mm to 70 mm for adult males, and the average value is 64 mm. The interpupillary distance D is generally 60 mm to 65 mm for adult females, and the average value is 62 mm. When the interpupillary distance D is 64 mm and the distance L is 2 m, the convergence angle α is about 1.83 degrees. When user 40 is driving the vehicle 70, the distance L to the object 20 that is the fixation target is often 2 m or more, and the magnitude of the convergence angle α is very small. Therefore, in the case of user 40 without squint, the line-of-sight direction 50 of user 40 can be regarded as substantially the same as the orientations 52L, 52R of the eyes of user 40.

[0019] FIG. 3 schematically shows the line-of-sight direction 50 of user 40 with squint and the orientations 52L, 52R of the eyes. When there is squint in user 40, the orientation of the dominant eye is the direction toward the fixation point 60, but the orientation of the non-dominant eye is a direction deviated from the direction toward the fixation point 60. In the example of FIG. 3, the right eye 42R of user 40 is the dominant eye, and the orientation 52R of the right eye is the direction from the right eye 42R to the fixation point 60. The line-of-sight direction 50 of user 40 with squint can be regarded as substantially the same as the orientation of the dominant eye of user 40 (the orientation 52R of the right eye in the example of FIG. 3).

[0020] On the other hand, the direction 52L of the user 40's non-dominant left eye is deviated from the direction from the left eye 42L towards the point of fixation 60. For example, if user 40 has strabismus and user 40 is looking away, for example by turning their head to the right, causing user 40's dominant eye (right eye 42R in the example of Figure 3) to be outside the field of view of the imaging device 12, the imaging device 12 can only image user 40's non-dominant eye. In such a situation, if the direction of user 40's non-dominant eye (the direction 52L of the left eye in the example of Figure 3) is considered the direction of gaze, the error with the actual direction of gaze 50 will be large, and the accuracy of estimating the direction of gaze will decrease.

[0021] In this embodiment, the accuracy of estimating the direction of gaze is improved when the video only contains the user's non-dominant eye, by generating correlation information that shows the correlation between the direction of the user's dominant eye and the direction of their non-dominant eye. Specifically, the direction of the user's dominant eye is estimated from the direction of the user's non-dominant eye using the correlation information, and the direction of the user's gaze is estimated from the estimated direction of the user's dominant eye.

[0022] Figure 4 is a schematic block diagram showing the functional configuration of the gaze estimation device 10 according to the embodiment. Each functional block shown can be realized in hardware terms by elements and mechanical devices such as the CPU (Central Processing Unit) and memory of a computer, and in software terms by computer programs, etc., but here it is depicted as a functional block realized through the cooperation of these. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways by combinations of hardware and software.

[0023] The gaze estimation device 10 comprises an image acquisition unit 22, an eye direction detection unit 24, an eye direction correlation unit 26, a gaze direction estimation unit 28, and a storage unit 34. The gaze estimation device 10 may also include a display control unit 30 and a dominant eye determination unit 32.

[0024] The video acquisition unit 22 acquires the video captured by the imaging device 12. The video acquisition unit 22 acquires the video of the user 40's face. The video acquisition unit 22 acquires the video that includes at least one of the user 40's right eye 42R and left eye 42L. For example, if the user 40 is facing forward of the vehicle 70, the video acquisition unit 22 acquires the video that includes the user 40's right eye 42R and left eye 42L (i.e., both eyes). For example, if the user 40 is facing to the right, the video acquisition unit 22 may acquire the video that includes only the left eye 42L and does not include the right eye 42R. For example, if the user 40 is facing to the left, the video acquisition unit 22 may acquire the video that includes only the right eye 42R and does not include the left eye 42L.

[0025] The eye direction detection unit 24 detects the user 40's right eye 42R and left eye 42L from the video acquired by the video acquisition unit 22, and detects the directions 52R and 52L of the detected right eye 42R and left eye 42L, respectively. If the video acquired by the video acquisition unit 22 contains only one of the user 40's right eye 42R or left eye 42L, the eye direction detection unit 24 uses image recognition technology to detect which it is, and detects the direction of only one of the detected right eye 42R or left eye 42L. The direction of the user 40's eyes can be detected based on the difference in position between a moving point that moves when the eyeball moves and a fixed point that does not move even if the eyeball moves. The iris or pupil of the eyeball can be used as the moving point, for example, the center of the iris or pupil can be used as the viewpoint to detect the direction of the eyes. In addition, the inner corner or outer corner of the eye, or the corneal reflection point created when a point light source for illuminating the driver is reflected on the eyeball, can be used as the fixed point.

[0026] Figure 5 is a schematic diagram illustrating an example of a method for detecting the direction of user 40's eyes. Figure 5 corresponds to an image acquired by the image acquisition unit 22 that includes user 40's right eye 42R and left eye 42L. The eye direction detection unit 24 uses image recognition technology to detect user 40's right eye 42R and left eye 42L, respectively. For the detected right eye 42R, the eye direction detection unit 24 detects the positions of the inner corner 44R, outer corner 46R, and iris 48R, and detects the direction 52R of the right eye 42R based on the positional relationship of the inner corner 44R, outer corner 46R, and iris 48R. For the detected left eye 42L, the eye direction detection unit 24 detects the positions of the inner corner 44L, outer corner 46L, and iris 48L, and detects the direction 52L of the left eye 42L based on the positional relationship of the inner corner 44L, outer corner 46L, and iris 48L. The eye orientation detection unit 24 detects parameters that indicate the eye orientation 52R, 52L, such as the angle in the x-direction (yaw angle) with respect to the z-axis, which is the imaging axis of the imaging device 12, and the angle in the y-direction (pitch angle) with respect to the z-axis.

[0027] The eye orientation correlation unit 26 generates correlation information showing the correlation between the orientation of the user 40's dominant eye and the orientation of the non-dominant eye, using the orientations 52R and 52L of the user 40's eyes detected by the eye orientation detection unit 24 and the user 40's dominant eye information stored in the storage unit 34. The dominant eye information stored in the storage unit 34 is information indicating whether the user 40's dominant eye is the right eye or the left eye. The dominant eye information is determined, for example, by the dominant eye determination unit 32, which will be described later. The dominant eye information may also be set by an input operation by the user 40.

[0028] The eye orientation correlation unit 26 generates correlation information for, for example, the yaw angle and pitch angle, which are parameters indicating the orientation of the non-dominant eye relative to the orientation of the user 40's dominant eye. The correlation information can also be described as parameters indicating the difference (e.g., angular difference) between the orientation of the user 40's dominant eye and the orientation of the non-dominant eye. The eye orientation correlation unit 26 may also generate correlation information that associates a first parameter indicating the orientation of the dominant eye with a second parameter indicating the orientation of the non-dominant eye.

[0029] The gaze direction estimation unit 28 estimates the gaze direction of user 40 using the direction of user 40's eyes detected by the eye direction detection unit 24 and the correlation information generated by the eye direction correlation unit 26. If the direction of user 40's dominant eye is detected by the eye direction detection unit 24, the gaze direction estimation unit 28 estimates the gaze direction of user 40 from the direction of user 40's dominant eye. For example, the gaze direction estimation unit 28 uses the direction of user 40's dominant eye as the gaze direction of user 40. If the direction of user 40's dominant eye is not detected by the eye direction detection unit 24, and only the direction of user 40's non-dominant eye is detected, the gaze direction estimation unit 28 estimates the direction of user 40's dominant eye using the direction of user 40's non-dominant eye and the correlation information. The gaze direction estimation unit 28 estimates the gaze direction of user 40 from the estimated direction of user 40's dominant eye. The gaze direction estimation unit 28, for example, sets the direction of the user 40's dominant eye, which it has estimated, as the user 40's gaze direction.

[0030] The display control unit 30 controls the operation of the display device 14 and controls the display content of the virtual image 16 that is visible in front of the user 40. The display control unit 30 displays an image on the display device 14 that is difficult for the user 40's eyes to focus on, so that the user 40's eyes can relax while viewing the virtual image 16. Here, an image that is difficult to focus on is an image in which the difference in brightness values ​​between adjacent pixels of the image is less than a predetermined threshold throughout the entire image. An image that is difficult to focus on may also be an image in which the difference in color between adjacent pixels of the image is less than a predetermined threshold throughout the entire image. Here, the predetermined threshold is, for example, 10% or less or 5% or less of the maximum brightness value of the image. An image that is difficult to focus on can be generated, for example, by applying an averaging filter to blur an image to any image.

[0031] Figures 6(a) and 6(b) schematically show examples of images displayed as virtual images 16. Figure 6(a) is an image in which the brightness value of the entire image is uniform, for example, an image that is a solid surface filled with a single color such as white. Figure 6(b) is an image in which a gradient is set so that the brightness value gradually decreases from the center to the periphery, and the image appears blurred because there are no edges where the brightness value of the image changes abruptly.

[0032] The dominant eye determination unit 32 determines the dominant eye information using a reference direction from the user 40 toward the virtual image 16 and the direction of the right eye 52R and the direction of the left eye 52L detected by the eye direction detection unit 24. The reference direction from the user 40 toward the virtual image 16 is determined with reference to the z direction, which is the imaging axis of the imaging device 12, and is set in advance, for example, based on the geometric arrangement relationship of the imaging device 12 and the display device 14. The direction from the user 40 toward the virtual image 16 corresponds to the line of sight 50 of the user 40 viewing the virtual image 16.

[0033] Figure 7 is a schematic diagram illustrating an example of how to determine the dominant eye of user 40. The dominant eye determination unit 32 acquires the direction of the right eye 52R and the direction of the left eye 52L detected by the eye direction detection unit 24 when user 40 is looking at the virtual image 16. The dominant eye determination unit 32 also acquires the direction of the right eye 52R and the direction of the left eye 52L detected by the eye direction detection unit 24 when, for example, the display device 14 displays an image that is difficult to focus on, as shown in Figures 6(a) and (b). The dominant eye determination unit 32 calculates the right eye angle θR, which is the direction of the right eye 52R with respect to a reference direction 56 from user 40 toward the virtual image 16, and the left eye angle θL, which is the direction of the left eye 52L with respect to the reference direction 56, and determines the dominant eye as the one with the smaller angle between the right eye angle θR and the left eye angle θL. The dominant eye determination unit 32 determines the dominant eye to be the right eye if, for example, the right eye angle θR is smaller than the left eye angle θL (i.e., θL > θR), and the dominant eye to be the left eye if this is not the case (i.e., θL ≤ θR).

[0034] The dominant eye determination unit 32 may determine the dominant eye based on the respective reference directions 58R and 58L of the right and left eyes. The respective reference directions 58R and 58L of the right and left eyes correspond to the orientations of the right and left eyes in the absence of strabismus (for example, Figure 2), and correspond to the direction obtained by changing the orientation by half the convergence angle α (α / 2) from the reference direction 56 from the user 40 toward the virtual image 16. The dominant eye determination unit 32 may compare the right eye angle, which is the orientation of the right eye 52R relative to the reference direction 58R of the right eye, and the left eye angle, which is the orientation of the left eye 52L relative to the reference direction 58L of the left eye, and select the eye with the smaller angle as the dominant eye.

[0035] The eye orientation correlation unit 26 may generate correlation information showing the correlation between the direction of the right eye 52R and the direction of the left eye 52L from the direction of the right eye 52R and the direction of the left eye 52L used to determine the dominant eye information by the dominant eye determination unit 32, and store it in the storage unit 34. For example, when the display device 14 displays an image that is difficult to focus, such as those shown in Figures 6(a) and (b), the eye orientation detection unit 24 may generate correlation information showing the correlation between the direction of the right eye 52R and the direction of the left eye 52L and store it in the storage unit 34. This makes it possible to use correlation information between the direction of the dominant and non-dominant eye when the user's eyes are relaxed, and improves the accuracy of estimating the direction of the dominant eye from the direction of the non-dominant eye.

[0036] The eye orientation correlation unit 26 may generate correlation information showing the correlation between multiple right eye orientations 52R and left eye orientations 52L corresponding to multiple situations from the right eye orientation 52R and left eye orientation 52L detected by the eye orientation detection unit 24 in multiple situations, and store it in the storage unit 34. Here, in each of the multiple situations, the displayed content of the virtual image 16 seen by the user 40 may be different from each other, and the user 40's gaze direction 50 may be different from each other. For example, the eye orientation correlation unit 26 may store in the storage unit 34 correlation information using the orientations of both eyes detected in a situation where an image that is difficult to focus is displayed, as shown in Figures 6(a) and (b), and correlation information using the orientations of both eyes detected in a situation where clear characters or symbols that are easy to focus are displayed as a virtual image 16. The eye orientation correlation unit 26 may store correlation information using the orientation of both eyes detected when the virtual image 16 is not displayed in the storage unit 34, or it may store correlation information using the orientation of both eyes detected when the user 40 is looking at the imaging device 12 in the storage unit 34.

[0037] The gaze direction estimation unit 28 may estimate multiple candidates for the user 40's gaze direction 50 using multiple correlation information stored in the memory unit 34 and the direction of the non-dominant eye detected by the eye direction detection unit 24. For example, the gaze direction estimation unit 28 may estimate a first candidate for the gaze direction 50 using first correlation information and estimate a second candidate for the gaze direction 50 using second correlation information. The gaze direction estimation unit 28 may calculate an estimated range for the gaze direction 50 based on the multiple estimated candidates for the gaze direction 50. By estimating the gaze direction 50 using multiple candidates or numerical ranges when only the non-dominant eye can be detected, it is possible to reduce the possibility of, for example, the user 40 looking away but the distraction not being detected.

[0038] Figure 8 is a flowchart illustrating an example of a gaze estimation method according to an embodiment. The gaze estimation device 10 acquires an image of the user 40 captured by the imaging device 12 (step S10) and detects the direction of the user 40's eyes included in the acquired image (step S12). If the gaze estimation device 10 detects the direction of the user 40's dominant eye (Yes in step S14), it estimates the user 40's gaze direction 50 using the detected dominant eye direction (step S16). If the gaze estimation device 10 has not detected the direction of the user 40's dominant eye (No in step S14) but has detected the direction of the non-dominant eye (Yes in step S18), it estimates the user 40's gaze direction 50 using the detected non-dominant eye direction and correlation information between the dominant eye direction and the non-dominant eye direction (step S20). If the non-dominant eye direction has not been detected in step S18 (No in step S18), the process in step S18 is skipped.

[0039] Figure 9 is a flowchart illustrating an example of a dominant eye determination method according to an embodiment. The gaze estimation device 10 displays a virtual image 16 on the display device 14 (step S30) and acquires an image of the user 40 viewing the virtual image 16 using the imaging device 12 (step S32). The gaze estimation device 10 detects the orientation of both of the user 40's eyes included in the acquired image (step S34) and determines the dominant eye of the user 40 using the reference direction 56 from the user 40 toward the virtual image 16 and the detected orientation of both of the user 40's eyes (step S36). The gaze estimation device 10 generates correlation information showing the correlation between the orientation of the user 40's dominant and non-dominant eyes using the detected orientation of both of the user 40's eyes (step S38). The gaze estimation device 10 stores the dominant eye information showing the determined dominant eye and the generated correlation information showing the correlation between the orientation of the dominant and non-dominant eyes (step S40).

[0040] Although the present invention has been described above with reference to the embodiments described above, the present invention is not limited to the embodiments described above, and the present invention is also included in combinations or substitutions of the configurations shown in each example.

[0041] This embodiment may be provided as a program. This program may be configured to cause a computer to implement the following functions: acquiring video footage captured by an imaging device that images the user; detecting the orientation of the user's right and left eyes from the acquired video footage; generating correlation information showing the correlation between the orientation of the user's dominant and non-dominant eyes, using the orientations of the user's right and left eyes detected from the video footage including both of the user's eyes, and dominant eye information indicating whether the user's dominant eye is the right or left eye; and estimating the user's gaze direction, using the orientation of the user's eyes detected from the video footage and the generated correlation information. [Explanation of Symbols]

[0042] 10...Eye gaze estimation device, 12...Imagine device, 14...Display device, 16...Virtual image, 22...Image acquisition unit, 24...Eye direction detection unit, 26...Eye direction correlation unit, 28...Eye gaze direction estimation unit, 30...Display control unit, 32...Dominant eye determination unit, 34...Memory unit, 40...User, 42L...Left eye, 42R...Right eye, 50...Eye gaze direction, 52L...Direction of the left eye, 52R...Direction of the right eye.

Claims

1. A video acquisition unit that acquires video captured by an imaging device that captures images of the user, An eye direction detection unit detects the direction of the user's right and left eyes from the video acquired by the video acquisition unit, A storage unit that stores dominant eye information indicating whether the user's dominant eye is the right eye or the left eye, An eye orientation correlation unit generates correlation information showing the correlation between the orientation of the user's dominant eye and the orientation of their non-dominant eye, using the orientations of the user's right and left eyes detected by the eye orientation detection unit from an image containing both of the user's eyes, and the dominant eye information stored in the memory unit. A gaze direction estimation unit estimates the user's gaze direction using the direction of the user's eyes detected by the eye direction detection unit and the correlation information generated by the eye direction correlation unit. The system includes a display control unit that controls the operation of a display device that displays a virtual image visible to the user, The eye orientation correlation unit is a gaze estimation device that generates correlation information using the orientations of the user's right eye and left eye, respectively, detected by the eye orientation detection unit from an image that includes both of the user's eyes viewing the virtual image.

2. The gaze direction estimation device according to claim 1, wherein the gaze direction estimation unit estimates the gaze direction of the user using the direction of the user's non-dominant eye detected by the eye direction detection unit from an image that includes the user's non-dominant eye but does not include the user's dominant eye, and correlation information generated by the eye direction correlation unit.

3. The gaze estimation device according to claim 1, wherein the display control unit causes the display device to display either an image in which the amount of change in brightness value between adjacent pixels is less than a predetermined threshold, or an image in which the amount of change in color between adjacent pixels is less than a predetermined threshold.

4. The steps include acquiring images captured by an imaging device that captures images of the user, The steps include detecting the orientation of the user's right and left eyes from the acquired video, The steps include controlling the operation of a display device that shows a virtual image visible to the user, A step of generating correlation information showing the correlation between the orientation of the user's dominant and non-dominant eye, using the orientation of the user's right and left eyes detected from an image that includes both of the user's eyes viewing the virtual image, and dominant eye information indicating whether the user's dominant eye is the right or left eye. The steps include: estimating the user's gaze direction using the direction of the user's eyes detected from the user's video and the generated correlation information; A gaze estimation method comprising the following features.

Images