Gaze Detection Device, Gaze Detection Method, and Computer-Readable Storage Medium

Abstract

The present invention provides a gaze detection device, a gaze detection method, and a computer-readable storage medium. The gaze detection device according to the present invention includes: a detection unit configured to detect a gaze of the eye based on an eye image obtained by photographing the eye viewing a display surface; and a determination unit configured to determine whether the detection of the gaze is incorrect based on a current eye image and a past eye image.

Description

Technical Field

[0001] The present invention relates to a line-of-sight (gaze) detection device and an imaging device equipped with the line-of-sight detection device. Background Art

[0002] In recent years, cameras have become more and more automated and intelligent. Japanese Unexamined Patent Application Publication No. 2004-8323 discloses a technique that allows a user to identify a target subject and perform focus control based on information about the viewpoint (line-of-sight (gaze) position) of the user observing a viewfinder, without manually inputting the subject position. In addition, Japanese Patent Application Laid-Open No. 07-289517 discloses a technique in which a captured eye image is used to determine the reliability of a line-of-sight detection result, and based on this reliability, it is determined whether line-of-sight detection can be performed. Then, if it is determined that line-of-sight detection cannot be performed, the reason for the inability to perform line-of-sight detection is estimated.

[0003] In the technique disclosed in Japanese Patent Application Laid-Open No. 07-289517, a single eye image is used for error determination of line-of-sight detection (determination as to whether there is an error (failure) in line-of-sight detection; determination as to whether the line of sight is accurately detected). However, in this method, it is impossible to accurately determine an erroneous detection of the line of sight as an error. If the accuracy of error determination is low, for example, improper processing is frequently performed based on an erroneously detected line of sight. In the case of displaying an item at the detected viewpoint, the item is frequently displayed at a position different from the actual viewpoint (erroneously detected viewpoint), and the display position of the item is scattered (the item moves unstably) and the usability of the device deteriorates. Summary of the Invention

[0004] The present invention provides a technique that allows for highly accurate error determination of line-of-sight (gaze) detection.

[0005] In a first aspect of the present invention, there is provided a gaze detection device including: a detection unit configured to detect a gaze of an eye based on an eye image obtained by photographing the eye viewing a display surface; and a determination unit configured to determine whether there is an error in the detection of the gaze based on a current eye image and a past eye image.

[0006] In a second aspect of the present invention, there is provided a gaze detection method including: detecting a gaze of an eye based on an eye image obtained by photographing the eye viewing a display surface; and determining whether there is an error in the detection of the gaze based on a current eye image and a past eye image.

[0007] In a third aspect of the present invention, there is provided a computer-readable storage medium storing a program for causing a computer to execute the steps of the above-described gaze detection method.

[0008] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. Description of the Drawings

[0009] Figure 1A and Figure 1B is an external view of a camera according to Embodiment 1;

[0010] Figure 2 is a cross-sectional view of a camera according to Embodiment 1;

[0011] Figure 3 is a block diagram of a camera according to Embodiment 1;

[0012] Figures 4A to 4C is a diagram depicting the field of view inside the viewfinder according to Embodiment 1;

[0013] Figure 5 is a diagram for describing the principle of a line-of-sight (gaze) detection method according to Embodiment 1;

[0014] Figure 6A is a diagram depicting an eye image according to Embodiment 1;

[0015] Figure 6B is a diagram indicating the brightness distribution of an eye image according to Embodiment 1;

[0016] Figure 7 is a flowchart of a line-of-sight detection operation according to Embodiment 1;

[0017] Figure 8A is a diagram depicting an eye image according to Embodiment 1;

[0018] Figure 8B is a diagram indicating the brightness distribution of an eye image according to Embodiment 1;

[0019] Figure 9A is a diagram depicting an eye image according to Embodiment 1;

[0020] Figure 9B is a diagram indicating the brightness distribution of an eye image according to Embodiment 1;

[0021] Figure 10 is a flowchart of an error determination operation (first method) according to Embodiment 1;

[0022] Figure 11 is a diagram depicting an eye image according to Embodiment 1;

[0023] Figure 12 is a flowchart of an error determination operation (second method) according to Embodiment 1;

[0024] Figure 13is a flowchart of camera operations according to Embodiment 1; and

[0025] Figure 14 is a flowchart of calibration operations according to Embodiment 2. DETAILED DESCRIPTION

[0026] Embodiment 1

[0027] Embodiment 1 of the present invention will be described with reference to the accompanying drawings.

[0028] Description of the Configuration

[0029] Figure 1A and Figure 1B is an external view of the camera 1 (digital still camera; interchangeable-lens camera) according to Embodiment 1. Figure 1A is a front perspective view, and Figure 1B is a rear perspective view. As Figure 1A shown, the camera 1 has a photographic lens unit 1A and a camera body 1B. A release button 5, which is an operation member for receiving a photographing operation from a user (photographer), is provided on the camera body 1B. As Figure 1B shown, an eyepiece 12 (eyepiece optical system) for a user to observe a display device 10 (display panel), which will be described later, included in the camera body 1B is provided on the back surface of the camera body 1B. The eyepiece optical system may include a plurality of lenses. Operation members 41 to 43 for receiving various operations from the user are also provided on the back surface of the camera body 1B. For example, the operation member 41 is a touch panel that receives a touch operation, the operation member 42 is an operation lever that can be pressed in various directions, and the operation member 43 is a four-way key that can be pressed in each of four directions. The operation member 41 (touch panel) includes a display panel (e.g., a liquid crystal panel) and has a function of displaying an image on the display panel.

[0030] Figure 2 is a cross-sectional view when the camera 1 is cut along the YZ plane formed by the Figure 1A shown Y-axis and Z-axis, and depicts a general internal configuration of the camera 1.

[0031] The photographic lens unit 1A includes two lenses 101 and 102, an aperture 111, an aperture drive unit 112, a lens drive motor 113, a lens drive member 114, a photoelectric coupler 115, a pulse board 116, mounting contacts 117, a focus adjustment circuit 118, etc. The lens drive member 114 is composed of a drive gear, etc., and the photoelectric coupler 115 detects the rotation of the pulse board 116 linked to the lens drive member 114 and transmits this rotation to the focus adjustment circuit 118. The focus adjustment circuit 118 drives the lens drive motor 113 based on the information from the photoelectric coupler 115 and the information from the camera body 1B (information related to the lens drive amount), and changes the focus position by moving the lens 101. The mounting contacts 117 are the interface between the photographic lens unit 1A and the camera body 1B. For simplicity, two lenses 101 and 102 are shown, but the photographic lens unit 1A actually includes more than two lenses.

[0032] The camera body 1B includes an imaging element 2, a CPU 3, a storage unit 4, a display device 10, a display device drive circuit 11, etc. The imaging element 2 is provided on a predetermined imaging surface of the photographic lens unit 1A. The CPU 3 is the control processing unit of a microcomputer and generally controls the camera 1. The storage unit 4 stores images taken by the imaging element 2, etc. The display device 10 is composed of liquid crystal, etc., and displays the taken images (subject images), etc. on the display surface of the display device 10. The display device drive circuit 11 drives the display device 10. The user can view the display surface of the display device 10 through the eyepiece 12.

[0033] The camera body 1B also includes light sources 13a and 13b, a beam splitter 15, a light receiving lens 16, an eye imaging element 17, etc. The light sources 13a and 13b are light sources for a single-lens reflex camera to detect the line-of-sight (gaze) direction based on the relationship between a reflected image (corneal reflection image; Purkinje image) formed by the corneal reflection of light and the pupil, and are light sources for illuminating the user's eyeball 14. Specifically, the light sources 13a and 13b are, for example, infrared light-emitting diodes that emit infrared light not perceptible to the user, and are provided around the eyepiece 12. The optical image (eyeball image: an image formed by the reflected light emitted from the light sources 13a and 13b and reflected by the eyeball 14) of the illuminated eyeball 14 passes through the eyepiece 12 and is reflected by the beam splitter 15. Then, by using the light receiving lens 16, the eyeball image is formed on the eye imaging element 17 having a two-dimensional array of photoelectric elements such as a CMOS. The light receiving lens 16 is positioned such that the pupil of the eyeball 14 and the eye imaging element 17 are in a conjugate image formation relationship. By a predetermined algorithm described later, the line-of-sight direction (viewpoint (line-of-sight position) on the display surface of the display device 10) of the eyeball 14 is detected based on the position of the corneal reflection image in the eyeball image formed on the eye imaging element 17.

[0034] Figure 3 is a block diagram depicting the electrical configuration inside the camera 1. A line-of-sight detection circuit 201, a photometry circuit 202, an autofocus detection circuit 203, a signal input circuit 204, a display device drive circuit 11, a light source drive circuit 205, etc. are connected to the CPU 3. In addition, the CPU 3 transmits a signal to a focus adjustment circuit 118 provided in the photographic lens unit 1A via a mounting contact 117, and transmits it to an aperture control circuit 206 included in an aperture drive unit 112 inside the photographic lens unit 1A. The storage unit 4 attached to the CPU 3 has a function of storing the imaging signals from the imaging element 2 and the eye imaging element 17, and a function of storing line-of-sight correction parameters (described later) for correcting individual differences in the line of sight.

[0035] In a state where the eyeball image is formed on the eye imaging element 17, the line-of-sight detection circuit 201 performs A / D conversion on the output of the eye imaging element 17 (the eye image capturing the eye) and sends the result to the CPU 3. The CPU 3 extracts feature points required for line-of-sight detection from the eye image according to a predetermined algorithm (described later), and calculates the user's line of sight (viewpoint on the display surface of the display device 10) based on the positions of the feature points.

[0036] The photometry circuit 202 amplifies, logarithmically compresses, and A / D-converts the signal (specifically, the luminance signal corresponding to the luminance of the field) obtained from the imaging element 2 which also serves as a photometry sensor, and sends the result as field luminance information to the CPU 3.

[0037] The autofocus detection circuit 203 A / D-converts the signal voltages from a plurality of detection elements (a plurality of pixels) for phase difference detection included in the imaging element 2, and sends the result to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to each focus detection point based on the signals of the plurality of detection elements. This is a technique called "imaging plane phase difference AF". In the first embodiment, for example, it is assumed that the focus detection points are located at each of 180 points corresponding to 180 points indicated in the viewfinder image (display surface of the display device 10) inside the viewfinder as shown in Figure 4A the figure.

[0038] The switch SW1 that is turned on in the first stroke of the release button 5 to start the photometry, distance measurement, line-of-sight detection operation, etc. of the camera 1, and the switch SW2 that is turned on in the second stroke of the release button 5 to start the photographing operation are connected to the signal input circuit 204. The ON signal from the switch SW1 or SW2 is input to the signal input circuit 204 and sent to the CPU 3.

[0039] Figure 4A is a diagram depicting the viewfinder image inside the viewfinder, and indicates the state in which the display device 10 is operating (the state of displaying an image). As shown in Figure 4A the figure, the viewfinder image inside the viewfinder includes a focus detection area 400, 180 distance measurement point indicators 401, a viewfinder mask 402, etc. Each of the 180 distance measurement point indicators 401 is set on the imaging plane (the imaging plane is superimposed on the through image (live view image) displayed on the display device 10) to be displayed at a position corresponding to the focus detection point. The distance measurement point indicator 401 corresponding to the current viewing point A (estimated position) among the 180 distance measurement point indicators 401 is highlighted by a frame or the like.

[0040] Description of the line-of-sight detection operation

[0041] will be described with reference to Figure 5 , Figure 6A , Figure 6B and Figure 7 the line-of-sight detection method. Figure 5 is a diagram for describing the principle of the line-of-sight detection method, and is a schematic diagram of the optical system for line-of-sight detection. As shown in Figure 5As shown, light sources 13a and 13b are arranged approximately symmetrically with respect to the optical axis of light receiving lens 16 and illuminate the user's eyeball 14. A part of the light emitted from light sources 13a and 13b and reflected by eyeball 14 is collected by light receiving lens 16 at eye imaging element 17. Figure 6A is a schematic diagram depicting an eye image (eyeball image projected onto eye imaging element 17) captured by eye imaging element 17, and Figure 6B is a diagram indicating the output intensity of the CMOS in eye imaging element 17. Figure 7 is a general flowchart of the line-of-sight detection operation.

[0042] When the line-of-sight detection operation starts, in Figure 7 step S701, light sources 13a and 13b emit infrared light towards the user's eyeball 14. An image of the user's eyeball illuminated by the infrared light is formed on eye imaging element 17 via light receiving lens 16 and is photoelectrically converted by eye imaging element 17. Thus, an electrical signal of a processable eye image can be obtained.

[0043] In step S702, the line-of-sight detection circuit 201 sends the eye image (eye image signal: electrical signal of the eye image) obtained from eye imaging element 17 to CPU 3.

[0044] In step S703, CPU 3 determines the coordinates of the points corresponding to the corneal reflection images Pd and Pe of light sources 13a and 13b and the point corresponding to the pupil center c from the eye image obtained in step S702.

[0045] The infrared light emitted from light sources 13a and 13b illuminates the cornea 142 of the user's eyeball 14. At this time, the corneal reflection images Pd and Pe formed by the part of the infrared light reflected on the surface of cornea 142 are collected by light receiving lens 16, form images on eye imaging element 17, and become the corneal reflection images Pd' and Pe' on the eye image. In the same way, the light beams from the edges a and b of pupil 141 also form images on eye imaging element 17 and become the pupil edge images a' and b' on the eye image.

[0046] Figure 6B Indicates Figure 6A the luminance information (luminance distribution) of region α in the eye image in Figure 6B In, the luminance distribution of the eye image in the X-axis direction is indicated (the horizontal direction of the eye image is the X-axis direction, and the vertical direction of the eye image is the Y-axis direction). In Embodiment 1, it is assumed that the coordinates of the corneal reflection images Pd' and Pe' in the X-axis direction (horizontal direction) are Xd and Xe, and the coordinates of the pupil edge images a' and b' in the X-axis direction are Xa and Xb. As Figure 6BAs shown, extremely high levels of brightness are obtained at the X coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. In the region from the X coordinate Xa to the X coordinate Xb corresponding to the region of the pupil 141 (the region of the pupil image obtained by imaging the light beam from the pupil 141 on the eye imaging element 17), extremely low levels of brightness are obtained except at the X coordinates Xd and Xe. In the region of the iris 143 outside the pupil 141 (the region of the iris outside the pupil image obtained by imaging the light beam from the iris 143), intermediate brightness between the above two levels of brightness is obtained. Specifically, in the region where the X coordinate (coordinate in the X-axis direction) is less than the X coordinate Xa, and in the region where the X coordinate is greater than the X coordinate Xb, intermediate brightness between the above two levels of brightness is obtained.

[0047] It is possible to obtain Figure 6B the X coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X coordinates Xa and Xb of the pupil edge images a' and b' from the brightness distribution shown. Specifically, the coordinates with extremely high brightness can be obtained as the coordinates of the corneal reflection images Pd' and Pe', and the coordinates with extremely low brightness can be obtained as the coordinates of the pupil edge images a' and b'. When the rotation angle θx of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16 is small, the X coordinate Xc of the pupil center image c' (the center of the pupil image) obtained by imaging the light beam from the pupil center c on the eye imaging element 17 can be expressed as Xc≈(Xa + Xb) / 2. In other words, the X coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa and Xb of the pupil edge images a' and b'. In this way, the coordinates of the corneal reflection images Pd' and Pe' and the coordinates of the pupil center image c' can be estimated.

[0048] In step S704, the CPU 3 calculates the image formation magnification β of the eyeball image. The image formation magnification β is the magnification determined by the position of the eyeball 14 relative to the light receiving lens 16, and can be determined using a function of the interval (Xd - Xe) between the corneal reflection images Pd' and Pe'.

[0049] In step S705, the CPU 3 calculates the rotation angle of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16. The X coordinate of the midpoint between the corneal reflection image Pd and the corneal reflection image Pe is approximately the same as the X coordinate of the curvature center O of the cornea 142. Therefore, if the standard distance from the curvature center O of the cornea 142 to the center c of the pupil 141 is Oc, the rotation angle θx of the eyeball 14 in the Z-X plane (the plane perpendicular to the Y axis) can be calculated by the following expression 1. The rotation angle θy of the eyeball 14 in the Z-Y plane (the plane perpendicular to the X axis) can also be calculated by the same method as the method for calculating the rotation angle θx.

[0050] β × Oc × SINθx ≈ {(Xd + Xe) / 2} - Xc...(Expression 1)

[0051] In step S706, the CPU 3 uses the rotation angles θx and θy calculated in step S705 to determine (estimate) the user's viewing point (the position where the line of sight points: the position the user is observing) on the display surface of the display device 10. If the coordinates (Hx, Hy) of the viewing point are the coordinates corresponding to the pupil center c, the coordinates (Hx, Hy) of the viewing point can be calculated by using the following Expressions 2 and 3.

[0052] Hx = m × (Ax × θx + Bx)...(Expression 2)

[0053] Hy = m × (Ay × θy + By)...(Expression 3)

[0054] The parameter "m" in Expressions 2 and 3 is a constant determined by the structure of the viewfinder optical system (such as the light receiving lens 16) of the camera 1, and is a conversion coefficient for converting the rotation angles θx and θy into the coordinates corresponding to the pupil center c on the display surface of the display device 10. It is assumed that the parameter "m" is predetermined and stored in the storage unit 4. The parameters Ax, Bx, Ay, and By are line-of-sight correction parameters for correcting individual differences in the line of sight, and can be obtained by performing a calibration operation (described later). It is assumed that the parameters Ax, Bx, Ay, and By are stored in the storage unit 4 before the start of the line-of-sight detection operation.

[0055] In step S707, the CPU 3 stores the coordinates (Hx, Hy) of the viewing point in the storage unit 4 and ends the line-of-sight detection operation.

[0056] Description of the calibration operation

[0057] Although the viewing point can be estimated through the above line-of-sight detection operation, it may be difficult to accurately estimate the viewing point due to individual differences in the shape of the eyeball and other reasons under the same circumstances. Specifically, unless the line-of-sight correction parameters Ax, Ay, Bx, and By are adjusted to values suitable for the user, as Figure 4B shown, there will be a deviation between the actual viewing point B and the estimated viewing point C. In Figure 4B , the user is looking at a person, but the camera 1 erroneously estimates that the user is looking at the background, and accurate focus detection and adjustment cannot be performed in this state.

[0058] Therefore, it is necessary to perform a calibration operation before the camera 1 starts photographing to obtain the line-of-sight correction parameters and store the parameters in the camera 1.

[0059] In the conventionally known technology, before imaging, a calibration operation is performed by emphasizing a plurality of indicators (as shown in Figure 4C ) displayed at different positions on the display surface of the display device 10 and having the user observe these indicators. Then, when the user is gazing at each indicator, a line-of-sight detection operation is performed, and line-of-sight correction parameters suitable for the user are determined based on the calculated plurality of viewpoints (estimated positions) and the actual coordinates of each indicator. However, there is no particular limitation on the method of displaying the indicators, as long as the position for the user to observe is indicated, and the indicators may be displayed in the form of graphics, or the indicators may be displayed by changing the brightness or color of an image (e.g., a captured image).

[0060] Description of the state of generating a line-of-sight detection error

[0061] As described above, in the line-of-sight detection operation, a corneal reflection image and a pupil image (specifically, the coordinates of the corneal reflection image and the coordinates of the pupil image) are detected from the eye image, and the rotation angles θx and θy of the eyeball 14 are obtained based on these detected images. Then, the viewpoints are estimated by transforming the rotation angles θx and θy into the coordinates corresponding to the pupil center c on the display surface of the display device 10.

[0062] Therefore, if there are false detection errors in the corneal reflection image and the pupil image (if the detection errors of the corneal reflection image and the pupil image are large), the rotation angles θx and θy cannot be correctly calculated, and a line-of-sight detection error (failure of line-of-sight detection; increase in viewpoint detection error; false detection of the viewpoint) is generated.

[0063] Examples of the state of generating a line-of-sight detection error will be described with reference to Figure 8A and Figure 8B . When the user wears glasses, light from at least one of the light sources 13a and 13b (the light source illuminating the eyeball 14) may be reflected by the surface of the glasses and enter the eye imaging element 17 through the eyepiece 12, and is captured as a ghost in the eye image. Figure 8A is a schematic diagram of the eye image (the eyeball image projected onto the eye imaging element 17) captured by the eye imaging element 17, and Figure 8B is a diagram indicating the output intensity of the CMOS in the eye imaging element 17. In Figure 8A , the ghost 801 formed by the glasses is captured. As shown in Figure 8B , the output intensity of the ghost 801 is similar to the output intensity of the corneal reflection image.

[0064] In Figure 8A and Figure 8BIn the state shown in [Figure], since the output intensity of the ghost image 801 is similar to the output intensity of the corneal reflection image, the ghost image 801 may be erroneously detected as the corneal reflection image. Specifically, the X coordinate Xf of the ghost image 801 may be erroneously detected as the X coordinate Xd of the corneal reflection image Pd' or the X coordinate Xe of the corneal reflection image Pe'. If the corneal reflection image is erroneously detected, the image formation magnification β of the eye image is not accurately calculated, and the calculation errors of the rotation angles θx and θy increase. In this case, if the estimated viewing point (the viewing point obtained by converting the calculated rotation angles θx and θy) is outside the display surface of the display device 10, it can be easily determined that a line-of-sight detection error has occurred (the viewing point estimation accuracy of the viewing point is low). However, if the estimated viewing point is inside the display surface of the display device 10, the viewing point may be correctly or erroneously estimated (that is, it is difficult to correctly determine the estimation accuracy of the viewing point), and it is difficult to correctly determine whether a line-of-sight detection error has occurred.

[0065] will be described with reference to Figure 9A and Figure 9B examples of states in which a line-of-sight detection error is generated. Figure 9A is a schematic diagram of an eye image (the eye image projected onto the eye imaging element 17) captured by the eye imaging element 17, and Figure 9B is a diagram indicating the output intensity of the CMOS in the eye imaging element 17. In Figure 9A , the ghost image 901 formed by the glasses is captured. As shown in Figure 9B , the output intensity of the ghost image 901 is similar to the output intensity of the iris image.

[0066] In Figure 9A and Figure 9B 's state, since the output intensity of the iris image is similar to the output intensity of the ghost image 901, the boundary between the pupil image and the iris image (the X coordinates Xa and Xb of the pupil edge images a' and b') may not be correctly detected. If the X coordinates Xa and Xb are not correctly detected, the center coordinates of the pupil image (the X coordinate Xc of the pupil center image c') are not correctly calculated, and the calculation errors of the rotation angles θx and θy increase. In this case, if the estimated viewing point is outside the display surface of the display device 10, or if the pupil diameter estimated based on the X coordinates Xa and Xb is outside the expected range (for example, outside the pupil diameter range of ordinary adults), it can be easily determined that a line-of-sight detection error has occurred. However, if the estimated viewing point is inside the display surface of the display device 10 and the estimated pupil diameter is within the expected range, the viewing point can be correctly estimated, and it is difficult to correctly determine whether a line-of-sight detection error has occurred.

[0067] As described above, the estimation accuracy of the viewing point is greatly affected by the detection error of the corneal reflection image and the detection error of the pupil image. Due to the influence of noise in the corneal reflection image and the eye imaging element 17, etc., compared with the detection error of the corneal reflection image, the detection error of the pupil image tends to be particularly large (the detection results of the pupil image tend to be scattered). Therefore, when the detection error of the pupil image is large, the viewing point estimation accuracy tends to easily decrease (when the detection results of the pupil image are scattered, the estimated viewing points tend to be scattered).

[0068] Specifically, the calculation error of the interval "Xd - Xe" of multiple corneal reflection images, the calculation error of the center position "(Xd + Xe) / 2" of multiple corneal reflection images, the calculation error of the center position "Xc" of the pupil image, etc. have a great influence on the estimation accuracy of the viewing point. Therefore, it is possible to determine whether a line-of-sight detection error has occurred based on the calculation error of the interval "Xd - Xe" of multiple corneal reflection images, the calculation error of the center position "(Xd + Xe) / 2" of multiple corneal reflection images, the calculation error of the center position "Xc" of the pupil image, etc.

[0069] The case where the line-of-sight detection error is generated by double imaging has been described, but the line-of-sight detection error may be generated for various other reasons. For example, when the pupil cannot be correctly detected due to the eyelid or eyelashes, a line-of-sight detection error may be generated. In addition, when the corneal reflection image is reflected on the tears and is misdetected, a line-of-sight detection error may be generated. In addition, due to the influence of the ambient light around the user, a line-of-sight detection error may be generated.

[0070] Description of the first error determination

[0071] A first method for error determination for line-of-sight detection (determining whether the line-of-sight detection of the line-of-sight detection circuit 201 is incorrect; determining whether the line of sight is correctly detected) will be described. As described above, if error determination is performed based on a single eye image, it is difficult to perform error determination with high accuracy unless the detection error is very large. Therefore, in the first method, error determination is performed based on the current eye image and the past eye image. Specifically, error determination is performed based on multiple corneal reflection images detected from the current eye image and multiple corneal reflection images detected from the past eye image. Thereby, error determination can be performed with high accuracy. The past eye image may or may not be the eye image captured immediately before the current eye image. For the past eye image, a single eye image may be used, or multiple eye images may be used.

[0072] The interval "Xd - Xe" between multiple corneal reflection images is closely related to the distance from the viewfinder (e.g., the light receiving lens 16) to the cornea. When using the camera 1, an operation of moving the eyeball 14 of the user observing the viewfinder back and forth in the Z direction (the direction parallel to the optical axis of the light receiving lens 16) is not usually performed. This means that during the period when the user observes the viewfinder, the interval Xd - Xe of the multiple corneal reflection images is approximately constant.

[0073] Therefore, in the first method, the interval between multiple corneal reflection images detected from the current eye image is compared with the interval between multiple corneal reflection images detected from the past eye image to perform error determination. Thus, error determination can be performed with high precision. This error determination can be regarded as error determination for line-of-sight detection, or can be regarded as error determination for the detection (calculation) of the interval "Xd - Xe" of multiple corneal reflection images.

[0074] Reference will be made to Figure 10 the flowchart in to describe the error determination operation (first method) of line-of-sight detection.

[0075] In step S1001, the CPU 3 determines whether multiple corneal reflection images are detected from the current eye image. If multiple corneal reflection images are detected, the process proceeds to step S1002, or if multiple corneal reflection images are not detected, the process proceeds to step S1007.

[0076] In step S1007, the CPU 3 determines that the current line-of-sight detection is incorrect.

[0077] In step S1002, the CPU 3 calculates the interval between multiple corneal reflection images (reflection image interval) detected from the current eye image and stores the reflection image interval in the storage unit 4. When the number of light sources illuminating the eyeball 14 is three or more and three or more corneal reflection images are detected, a representative value (e.g., average value, maximum value, minimum value, mode value, median value) of multiple reflection image intervals obtained from these three or more detected corneal reflection images can be calculated. All reflection image intervals obtained from three or more corneal reflection images can be calculated, or partial reflection image intervals can be calculated. For example, in the case of four light sources (two rows × two columns), the interval between only two corneal reflection images set in the row direction can be calculated, or the interval between only two corneal reflection images set in the column direction can be calculated, or the interval between only two corneal reflection images set in the diagonal direction can be calculated. Among these three types of reflection image intervals, only two types of reflection image intervals can be calculated.

[0078] In step S1003, the CPU 3 determines whether the reflection image interval has been calculated at least a predetermined number of times (e.g., 5 times, 10 times), in other words, whether the process in step S1002 has been performed at least a predetermined number of times. Then, if the reflection image interval has been calculated at least a predetermined number of times, the CPU 3 causes the process to proceed to step S1004, or if the number of times the reflection image interval is calculated does not reach the predetermined number, the error determination operation ends without performing error determination.

[0079] In step S1004, the CPU 3 obtains from the storage unit 4 the reflection image interval calculated based on the past eye images. One reflection image interval calculated based on a single past eye image can be obtained, but in Embodiment 1, it is assumed that the CPU 3 obtains one or more reflection image intervals calculated in a past predetermined period, and calculates the average value of the one or more obtained reflection image intervals. Instead of the average value, different representative values (e.g., maximum value, minimum value, mode value, median value) can be calculated. The past predetermined period is, for example, a period of 5 frames up to now, or a period from the timing 5 frames before now to the timing 2 frames before now.

[0080] In step S1005, the CPU 3 calculates the difference between the reflection image interval (past reflection image interval: average value in Embodiment 1) obtained in step S1004 and the reflection image interval (current reflection image interval) calculated in step S1002.

[0081] In step S1006, the CPU 3 compares the difference calculated in step S1005 with a predetermined threshold value, and determines whether the difference is equal to or greater than the threshold value. If the difference is equal to or greater than the predetermined threshold value, the CPU 3 causes the process to proceed to step S1007, and determines that the current line-of-sight detection is incorrect. If the difference is less than the predetermined threshold value, the CPU 3 determines that the current line-of-sight detection is successful, and ends the error determination operation.

[0082] Description of the second error determination

[0083] A second method for error determination in line-of-sight detection will be described. Also in the second method, error determination is performed based on the current eye image and the past eye image. Specifically, in the second method, error determination is performed based on a plurality of corneal reflection images and pupil images detected from the current eye image, and a plurality of corneal reflection images and pupil images detected from the past eye image. Thus, error determination can be performed with high accuracy.

[0084] The central position of multiple corneal reflection images, "(Xd + Xe) / 2", has a strong correlation with the central position of the pupil image, "Xc". Unless the head of the user observing the viewfinder is translated, if the central position of the pupil image, "Xc", moves, the central position of the multiple corneal reflection images, "(Xd + Xe) / 2", also moves in the same direction as the central position of the pupil image, "Xc". Similarly, in the case of using camera 1, under normal circumstances, the head of the user observing the viewfinder does not perform continuous translation. Therefore, under normal circumstances, there will be no movement of only one of the central position of the multiple corneal reflection images, "(Xd + Xe) / 2", and the central position of the pupil image, "Xc".

[0085] Therefore, in the second method, error detection is performed by comparing the following two types of distances. Thus, error determination can be performed with high precision. This error determination can be regarded as the error determination for line-of-sight detection, or can be regarded as the error determination for the detection (calculation) of the central position of the multiple corneal reflection images, "(Xd + Xe) / 2", or the error determination for the detection (calculation) of the central position of the pupil image, "Xc".

[0086] - The distance from the position (representative position: e.g., central position) of the multiple corneal reflection images detected from the current eye image to the position (e.g., central position) of the pupil image detected from the current eye image

[0087] - The distance from the position (representative position: e.g., central position) of the multiple corneal reflection images detected from the past eye image to the position (e.g., central position) of the pupil image detected from the past eye image

[0088] However, even when there are three or more light sources for illuminating the eyeball 14, the same number of corneal reflection images as the light sources are not always detected, and the representative position (e.g., central position) of the multiple corneal reflection images changes according to the number and arrangement of the actually detected corneal reflection images.

[0089] Figure 11 is a schematic diagram depicting an eye image (an eyeball image projected onto the eye imaging element 17) captured by the eye imaging element 17. Figure 11The eye image in [the reference] is an eye image under the condition that there are four light sources illuminating the eyeball 14, and includes four corneal reflection images P1 to P4. When all the corneal reflection images P1 to P4 are detected, the central positions Pc1 of the corneal reflection images P1 to P4 are calculated as the central positions of the plurality of corneal reflection images. On the other hand, when the corneal reflection images P1 and P2 are not detected and only the corneal reflection images P3 and P4 are detected, the central positions Pc1 of the corneal reflection images P1 to P4 are not calculated as the central positions of the plurality of corneal reflection images, but the central positions Pc2 of the corneal reflection images P3 and P4 are calculated.

[0090] In this way, the representative position of the plurality of corneal reflection images may change according to the number and arrangement of the detected corneal reflection images. Therefore, in some cases, only the representative position of the plurality of corneal reflection images may change greatly. If an eye image in which the number of detected corneal reflection images is different from the number of the current eye image is used as the past eye image, the above comparison cannot be performed appropriately. Therefore, for the past eye image, it is preferable to use an eye image that at least satisfies the condition that the number of detected corneal reflection images is approximately the same as the number of the current image. In addition, for the past eye image, it is more preferable to use an eye image that also satisfies the condition that the arrangement of the detected corneal reflection images is approximately the same as the arrangement of the current image. Similarly, in the case of using the first method for error determination, for the past eye image, it is preferable to use an eye image that satisfies these conditions.

[0091] Reference will be made to Figure 12 the flowchart in [the reference] to describe the error determination operation (second method) of gaze detection.

[0092] In step S1201, the CPU 3 determines whether the reflection center (the central position of the plurality of corneal reflection images) and the pupil center (the central position of the pupil image) are calculated based on the current eye image. Then, if both the reflection center and the pupil center are calculated, the CPU 3 causes the process to proceed to step S1202, or if at least one of the reflection center and the pupil center is not calculated, the CPU 3 causes the process to proceed to step S1208.

[0093] In step S1208, the CPU 3 determines that the current gaze detection is incorrect.

[0094] In step S1202, the CPU 3 determines the current detection state of the corneal reflection images (the number and arrangement of the multiple corneal reflection images detected from the current eye image), and determines which light source the detected corneal reflection image is from. It is possible to misdetect a ghost image as a corneal reflection image, but this type of detection error is determined as the error in the above first method. The arrangement, size, etc. of the multiple images that are candidates for being detected as corneal detection images can be considered to detect the corneal reflection images so that a ghost image is less likely to be misdetected as a corneal reflection image. In the case where more images than the number of light sources are detected as candidates for corneal reflection images, multiple viewpoints can be estimated based on multiple combinations of the same number of candidates as the number of light sources. Then, each candidate whose estimated viewpoint is most likely to be the actual viewpoint (such as the viewpoint closest to the previous viewpoint) can be detected as a corneal reflection image.

[0095] In step S1203, the CPU 3 calculates the distance between the reflection center and the pupil center of the current eye image, and stores this distance in the storage unit 4.

[0096] In step S1204, the CPU 3 determines whether the center distance (the distance between the reflection center and the pupil center) has been calculated at least a predetermined number of times (for example, 5 times, 10 times) in a state where the detection state of the corneal reflection images (the number and arrangement of the detected multiple corneal reflection images) is approximately the same as the current state. In other words, the CPU 3 determines whether the process in step S1203 has been performed at least a predetermined number of times in a state where the detection state of the corneal reflection images is approximately the same as the current state. Then, if the center distance has been calculated at least a predetermined number of times in a state where the detection state of the corneal reflection images is approximately the same as the current state, the CPU 3 causes the process to proceed to step S1205. If the number of times the center distance is calculated in a state where the detection state of the corneal reflection images is approximately the same as the current state does not reach the predetermined number, the CPU 3 ends the error determination operation without performing error determination.

[0097] In step S1205, the CPU 3 obtains the center distance calculated based on a past eye image in which the detection state of the corneal reflection images is approximately the same as the current eye image. In Embodiment 1, the CPU 3 obtains at least one center distance calculated in a past predetermined period (predetermined period length) in which the detection state of the corneal reflection images is approximately the same as the current state, and calculates the average value of the at least one center distance. Instead of the average value, different representative values (such as the maximum value, minimum value, mode value, median value) can be calculated. One center distance calculated based on a single past eye image can be obtained.

[0098] In step S1206, the CPU 3 calculates the difference between the center distance obtained in step S1205 (past center distance: average value in Embodiment 1) and the center distance calculated in step S1203 (current center distance).

[0099] In step S1207, the CPU 3 compares the difference calculated in step S1206 with a predetermined threshold value and determines whether the difference is equal to or greater than the threshold value. If the difference is equal to or greater than the predetermined threshold value, the CPU 3 causes the process to proceed to step S1208 and determines that the current line-of-sight detection is incorrect. If the difference is less than the predetermined threshold value, the CPU 3 determines that the current line-of-sight detection is successful and ends the error determination operation.

[0100] Description of Camera Operation

[0101] The operation of camera 1 will be described with reference to Figure 13 the flowchart in

[0102] When the power of camera 1 is turned on, the imaging element 2 starts acquiring a through-image in step S1301, sends the image signal of the through-image to the CPU 3, and the CPU 3 displays the acquired through-image on the display device 10. The user checks the subject by viewing the through-image displayed on the display device 10. The power of camera 1 is turned on / off according to the user operation on camera 1.

[0103] In step S1302, the CPU 3 determines whether the power of camera 1 is turned off, and if the power is turned off, ends Figure 13 the camera operation in

[0104] or if the power is not turned off, causes the process to proceed to step S1303. Figure 7 In step S1303, the CPU 3 starts acquiring the eye image of the user who started viewing the through-image in step S1301 and performs

[0105] the line-of-sight detection operation in Figure 10 and Figure 12 in

[0106] In step S1304, the CPU 3 performs

[0107] In step S1306, the CPU 3 estimates the current viewpoint based on a plurality of viewpoints calculated in a past predetermined time period. When performing the process in step S1306, instead of the viewpoint calculated in step S1303, the viewpoint estimated in step S1306 is used as the current viewpoint. The method for estimating the viewpoint is not particularly limited. For example, the current viewpoint can be estimated based on the amount and direction of movement of past viewpoints, etc. When the viewpoint moves, the current viewpoint can be estimated by smoothly extending the trajectory of the viewpoint. When the viewpoint approximately stops at a point (e.g., swings around a point), the center position or average position of the plurality of viewpoints, etc., can be estimated as the current viewpoint.

[0108] In step S1307, the CPU 3 superimposes the viewpoint frame on the through-image at the position corresponding to the current viewpoint (estimated position) such that the line-of-sight frame (the frame indicating the viewpoint) is displayed at the current viewpoint (estimated position) on the display surface of the display device 10. Thereby, the display as shown in Figure 4A is performed (display of superimposing the line-of-sight frame on the through-image), and the current viewpoint A (estimated position) can be notified to the user. Instead of the viewpoint frame, for example, a point indicating the viewpoint can be displayed.

[0109] In step S1308, the CPU 3 stands by for a predetermined time.

[0110] In step S1309, the CPU 3 determines whether the user presses (half-presses) the release button 5 to turn on the switch SW1. For example, when the user agrees to focus on the position of the viewpoint frame (the frame indicating the estimated viewpoint) displayed superimposed on the through-image, the user half-presses the release button 5 to turn on the switch SW1. If the switch SW1 is turned on, the CPU 3 causes the process to proceed to step S1310, or if the switch SW1 is not turned on, the process returns to step S1303 and the viewpoint is estimated again.

[0111] In step S1310, the CPU 3 performs a distance measurement operation at the position of the current line-of-sight frame and notifies the user that the distance measurement operation has been performed by highlighting (such as changing the display color of the line-of-sight frame).

[0112] In step S1311, the CPU 3 drives the lens 101 in the photographic lens unit 1A based on the distance measurement result obtained in step S1310. Thereby, focusing on the position of the viewpoint frame displayed superimposed on the through-image is achieved.

[0113] In step S1312, the CPU 3 determines whether the user fully presses (depresses completely) the release button 5 to turn on the switch SW2. For example, in the case where the user agrees to take a photograph at the current focus position, the user fully presses the release button 5 to turn on SW2. If the switch SW2 is turned on, the CPU 3 advances the process to step S1313, or if the switch SW2 is not turned on, the process returns to step S1309.

[0114] In step S1313, the CPU 3 performs a photographing operation, thereby storing the image signal acquired by the imaging element 2 in the storage unit 4.

[0115] In step S1314, the CPU 3 displays the image (photographed image) stored in the storage unit 4 in step S1313 on the display device 10 for a predetermined time, and returns the process to step S1302.

[0116] Conclusion

[0117] As described above, according to the first embodiment, for the error determination of the gaze detection, not only the current eye image but also the past eye image is used. Specifically, the error determination is performed based on the change over time of the features of the eye image. Thereby, the error determination of the gaze detection can be performed with high accuracy. In addition, the gaze (viewpoint) with a large detection error can be appropriately eliminated, and the user can appropriately improve the visibility state, so that focusing can be achieved at a position based on the accurate gaze.

[0118] Second Embodiment

[0119] Next, a second embodiment of the present invention will be described. Hereinafter, the description of the same aspects (such as configurations and processes) as those in the first embodiment will be omitted, and the aspects different from those in the first embodiment will be described. In the first embodiment, an example of performing error determination during the camera operation was described. In the second embodiment, an example of performing error determination in a state where an index to be viewed is displayed on the display device 10 will be described. The use of the index is not particularly limited, but in the second embodiment, the case where the index is used for a calibration operation will be described.

[0120] As described in the first embodiment, the calibration operation is performed by emphasizing a plurality of indices located at different positions on the display device 10 before capturing an image and having the user view these indices. For example, as Figure 4C shown, indices are displayed at five positions on the display surface of the display device 10 (at the center, above the center, below the center, to the left of the center, and to the right of the center). In the second embodiment, one index is displayed at a time, but all five indices may also be displayed simultaneously, and the indices among the five indices may be emphasized in sequence. Thereby, the user views the displayed index (the emphasized index). In other words, in the calibration operation, the position that the user should view is specified by the index.

[0121] Even during the calibration operation, the Figure 7 gaze detection operation in Figure 8A is performed, and gaze detection errors may be generated for various reasons. For example, as described with reference to Figure 8B and Figure 9A and Figure 9B , when the corneal reflection image or the pupil image is erroneously detected, a gaze detection error is generated. If an incorrect gaze detection result (viewpoint) is used, incorrect gaze correction parameters are determined, and the gaze cannot be properly corrected. If the error determination operation in Figure 10 or Figure 12 is performed during the calibration operation, appropriate gaze correction parameters can be determined by eliminating the incorrect gaze detection result.

[0122] Furthermore, when there is a large change in the position of the pupil image or the corneal reflection image in the eye image (i.e., when the position (viewpoint) where the user is looking changes significantly), the characteristics of the pupil image or the corneal reflection image also change significantly due to the influence of uneven exposure caused by a decrease in the ambient light amount and lens aberration, etc. For example, the shape of the pupil image, the brightness of the pupil image, the shape of the corneal reflection image, and the brightness of the corneal reflection image change significantly. The interval between multiple corneal reflection images and the distance between the center positions of multiple corneal reflection images and the center position of the pupil image also change significantly. Therefore, if the information acquired before the viewpoint change is used for error determination when the actual viewpoint changes significantly, even if the gaze detection is actually successful, it may be erroneously determined that the gaze detection is incorrect. Therefore, it is preferable not to use the information acquired before the viewpoint change. During the calibration operation, the viewpoint may change significantly according to the switching of the display position of the indicator. Therefore, by easily and appropriately eliminating the information acquired before the viewpoint change, the error determination of the gaze detection can be performed with high accuracy. For example, if the eye image captured in a state where the display position of the indicator is the same as the display position of the current image is used as the past eye image, the error determination of the gaze detection can be performed with high accuracy.

[0123] Calibration operation

[0124] The calibration operation according to Embodiment 2 will be described with reference to the Figure 14 flowchart in

[0125] In step S1401, the CPU 3 displays an indicator for the user to gaze at on the display device 10.

[0126] In step S1402, the CPU 3 stands by for a predetermined time.

[0127] In step S1403, the CPU 3 determines whether the user presses (half-presses) the release button 5 to turn on the switch SW1. For example, the user half-presses the release button 5 to indicate that the index is being gazed at and turns on the switch SW1. If the switch SW1 is turned on, the CPU 3 advances the process to step S1404, or if the switch SW1 is not turned on, the process returns to step S1402.

[0128] In step S1404, the CPU 3 performs Figure 7 the line-of-sight detection operation in. Through the line-of-sight detection operation, the coordinates of the pupil image and the corneal reflection image in the eye image are detected, and the viewpoint coordinates on the display surface of the display device 10 are calculated.

[0129] In step S1405, the CPU 3 performs Figure 10 or Figure 12 the error determination operation in.

[0130] In step S1406, the CPU 3 determines whether the current line-of-sight detection is incorrect based on the result of the error determination operation in step S1405. If the current line-of-sight detection is incorrect, the CPU 3 advances the process to step S1407, or if the current line-of-sight detection is correct (successful), the process advances to S1409.

[0131] In step S1407, the CPU 3 determines whether the number of times the line-of-sight detection operation is performed (the error count of the line-of-sight detection) is a predetermined number or more. Then, if the number of times the line-of-sight detection operation is performed is less than the predetermined number, the CPU 3 returns the process to step S1404, or if the number of times the line-of-sight detection operation is performed is a predetermined number or more, the process advances to step S1408. The number of times the line-of-sight detection operation is performed is counted by the CPU 3.

[0132] In step S1408, the CPU 3 determines that calibration cannot be appropriately performed (determination of the line-of-sight correction parameter), and notifies the user that the calibration has failed. Then the CPU 3 ends the calibration operation.

[0133] In step S1409, the CPU 3 determines whether the number of times the viewpoint is detected (the number of times the viewpoint is detected; the number of times the line-of-sight detection is successfully performed) is a predetermined number or more. Then, if the number of times the viewpoint is detected is less than the predetermined number, the CPU 3 returns the process to step S1404, or if the number of times the viewpoint is detected is a predetermined number or more, the process advances to step S1410. The number of times the viewpoint is detected is counted by the CPU 3.

[0134] In step S1410, the CPU 3 determines whether gaze detection has been completed for all metrics (processing in steps S1401 to S1409). Then, if there is at least one metric for which gaze detection has not been performed, the CPU 3 causes the processing to proceed to step S1411, or if gaze detection has been completed for all metrics, the CPU 3 causes the processing to proceed to step S1413.

[0135] In step S1411, the CPU 3 changes (switches) the metric displayed in step S1401 to the next metric.

[0136] In step S1412, the CPU 3 resets the information acquired in the state of the metric before the display change so that the eye image captured in the state of the metric before the display change is not used as the past eye image in the error determination operation. For example, the CPU 3 resets the interval between multiple corneal reflection images and the distance between the center position of multiple corneal reflection images and the center position of the pupil image. The CPU 3 also resets the number of times counted for the processing in steps S1407 and S1409 (the number of times of performing the gaze detection operation and the number of times of detecting the viewing point). Then, the CPU 3 causes the processing to return to step S1401.

[0137] In step S1413, the CPU 3 notifies the user that the calibration operation has been successful.

[0138] In step S1414, the CPU 3 calculates the gaze correction parameter based on the viewing points detected for each metric, stores the gaze correction parameter in the storage unit 4, and ends the calibration operation.

[0139] Conclusion

[0140] As described above, according to Embodiment 2, it is possible to accurately determine the error of gaze detection during the calibration operation, and thus it is possible to accurately determine the gaze correction parameter. In addition, the eye image captured in the state where the display position of the metric that the user should view is the same as the display position of the current eye image is used as the past eye image. Therefore, by easily and appropriately eliminating the information acquired before a large change occurs in the actual viewing point, it is possible to accurately determine the error of gaze detection.

[0141] Embodiments 1 and 2 are merely examples, and configurations obtained by appropriately modifying or changing the configurations of Embodiments 1 and 2 within the spirit of the present invention are also included in the present invention. In addition, configurations obtained by appropriately combining the configurations of Embodiments 1 and 2 are also included in the present invention.

[0142] In addition, an example of detecting the line of sight of a user looking at the viewfinder of a camera has been described above, but the present invention is not limited thereto. For example, even in the case of performing line of sight detection in a head-mounted display (HMD) worn on the user's head to physically perceive virtual reality (VR) or the like, the positional relationship between the HMD and the eyeball (the eyeball of the user wearing the HMD) is usually constant. Therefore, in this case as well, the effects of the present invention described in Embodiments 1 and 2 can be achieved. In the same manner, the present invention can be applied to eyeglass-type line of sight detection devices such as augmented reality (AR) glasses. The present invention can be applied to all electronic devices that estimate a viewpoint using an eye image. In particular, (since the positional relationship between the eyepiece frame and the eyeball (the eyeball looking at the eyepiece frame) is usually constant), the present invention can be applied to electronic devices having an eyepiece frame that limits the field of view (such as the eyepiece frame of a camera, a spectacle frame).

[0143] As described above, according to Embodiments 1 and 2, by performing error determination for line of sight detection using not only the current eye image but also the past eye image, a line of sight detection device capable of achieving highly accurate error determination can be provided to a camera, an HMD, and an eyeglass-type device.

[0144] According to the present disclosure, error determination for line of sight detection can be performed with high accuracy.

[0145] Other embodiments

[0146] Embodiments of the present invention can also be implemented by a method in which software (a program) that executes the functions of the above-described embodiments is provided to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessing unit (MPU) of the system or device reads and executes the program.

[0147] Although the present invention has been described with reference to exemplary embodiments, it should be understood that the present invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications as well as equivalent structures and functions.

Claims

1. A gaze detection device, comprising: a detection unit configured to detect a gaze of the eye based on an eye image obtained by photographing the eye viewing a display surface; and a determination unit configured to determine whether the detection of the gaze is incorrect based on a current eye image and a past eye image, wherein the detection unit detects the gaze by detecting a pupil image and a plurality of corneal reflection images from the eye image; and when a difference between two distances is greater than a threshold value, the determination unit determines that the detection of the gaze is incorrect: a distance from a position of a plurality of corneal reflection images detected from the current eye image to a position of the pupil image detected from the current eye image; and a distance from a position of a plurality of corneal reflection images detected from the past eye image to a position of the pupil image detected from the past eye image.

2. The gaze detection device according to claim 1, wherein when a difference between an interval of a plurality of corneal reflection images detected from the current eye image and an interval of a plurality of corneal reflection images detected from the past eye image is greater than a threshold value, the determination unit determines that the detection of the gaze is incorrect.

3. The gaze detection device according to claim 1 or 2, wherein the determination unit uses an eye image that at least satisfies a condition that a number of detected corneal reflection images is the same as a number of the current eye image as the past eye image.

4. The gaze detection device according to claim 3, wherein the determination unit uses an eye image that further satisfies a condition that an arrangement of a plurality of detected corneal reflection images is the same as an arrangement of the current eye image as the past eye image.

5. The gaze detection device according to claim 1, wherein the current eye image is an eye image obtained by photographing in a state where an index to be viewed by the eye is displayed on the display surface, and the determination unit uses an eye image that at least satisfies a condition that an eye image is obtained by photographing in a state where a display position of the index is the same as a display position of the current eye image as the past eye image.

6. The gaze detection device according to claim 5, wherein the index is an index for a calibration operation to obtain a parameter to be used for detecting the gaze.

7. A gaze detection method, comprising: detecting a gaze of the eye based on an eye image obtained by photographing the eye viewing a display surface; and determining whether the detection of the gaze is incorrect based on a current eye image and a past eye image, wherein the gaze is detected by detecting a pupil image and a plurality of corneal reflection images from the eye image; and When the difference between the following two distances is greater than a threshold value, it is determined that the detection of the gaze is incorrect: the distance from the position representing the plurality of corneal reflection images detected from the current eye image to the position of the pupil image detected from the current eye image; and the distance from the position representing the plurality of corneal reflection images detected from the past eye image to the position of the pupil image detected in the past eye image.

8. A computer-readable storage medium storing a program that causes a computer to execute the steps of the gaze detection method according to claim 7.

9. A computer program product comprising a program that causes a computer to execute the steps of the gaze detection method according to claim 7.

Images