Eye gaze detection device

Abstract

To provide a technique capable of accurately determining an error of sight line detection.SOLUTION: A sight line detection device according to an invention comprises: detection means for detecting a sight line of an eye watching a display surface, based on an eye image obtained by imaging the eye; and determination means which determines whether the detection of the sight line is an error, on the basis of a present eye image and a past eye image.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The present invention relates to a gaze detection device and an imaging device having the gaze detection device. [Background technology]

[0002] In recent years, cameras have become increasingly automated and intelligent. Patent Document 1 proposes a technology that recognizes the subject intended by the photographer based on information about the viewpoint (gaze position) of the photographer looking through the viewfinder and controls the focus, without the need to manually input the subject position. Patent Document 2 proposes a technology that uses captured eye images to determine the reliability of gaze detection results, determines whether gaze detection is possible or not based on the reliability, and, if it is determined that gaze detection is not possible, estimates the cause of the gaze detection failure. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-8323 [Patent Document 2] Japanese Patent Application Publication No. 7-289517 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology of Patent Document 2 uses a single eye image to determine gaze detection errors (determine whether gaze detection is an error (failure) or not; determine whether the gaze was detected correctly or not). However, this method cannot accurately determine an erroneous gaze detection as an error. When the accuracy of error determination is low, for example, inappropriate processing based on an erroneously detected gaze is frequently performed. When an item is displayed at the detected viewpoint, the item is frequently displayed at a position (an erroneously detected viewpoint) that differs from the actual viewpoint, causing the display position of the item to vary (the item swings widely), reducing the usability of the device.

[0005] An object of the present invention is to provide a technique that can perform gaze detection error determination with high accuracy. [Means for solving the problem]

[0006] A first aspect of the present invention is a display device including a detection means for detecting a line of sight of an eye based on an image of an eye looking at a display surface, and a determination means for determining whether or not the detection of the line of sight is an error based on a current eye image and a past eye image. The detection means detects a plurality of corneal reflection images from the eye image to detect the gaze, and the determination means determines whether or not the gaze detection is in error based on the plurality of corneal reflection images detected from the current eye image and the plurality of corneal reflection images detected from the past eye image, and determines that the gaze detection is in error when a difference between an interval between the plurality of corneal reflection images detected from the current eye image and an interval between the plurality of corneal reflection images detected from the past eye image is greater than a threshold value. The gaze detection device is characterized by:

[0007] A second aspect of the present invention is a method for detecting a line of sight of an eye based on an eye image obtained by capturing an image of the eye looking at a display surface. detection and determining whether the gaze detection is an error based on the current eye image and the past eye image. judgement Steps and In the detecting step, a plurality of corneal reflection images are detected from the eye image to detect the line of sight, and in the determining step, it is determined whether or not the detection of the line of sight is an error based on a plurality of corneal reflection images detected from the current eye image and a plurality of corneal reflection images detected from the past eye image, and If a difference between an interval between the plurality of corneal reflection images detected from the image and an interval between the plurality of corneal reflection images detected from the past eye image is greater than a threshold, it is determined that the detection of the line of sight is an error. The gaze detection method is characterized by:

[0008] A third aspect of the present invention is a program for causing a computer to function as each of the means of the gaze detection device described above.A fourth aspect of the present invention is a computer-readable recording medium storing a program for causing a computer to function as each of the means of the gaze detection device described above. [Effects of the Invention]

[0009] According to the present invention, it is possible to perform a highly accurate determination of an error in gaze detection. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is an external view of a camera according to a first embodiment. [Figure 2] FIG. 1 is a cross-sectional view of a camera according to a first embodiment. [Figure 3] FIG. 1 is a block diagram of a camera according to a first embodiment. [Figure 4]FIG. 2 is a diagram showing a field of view within a finder according to the first embodiment. [Figure 5] 1A and 1B are diagrams for explaining the principle of a gaze detection method according to a first embodiment. [Figure 6] FIG. 2 is a diagram showing an eye image according to Example 1. [Figure 7] 10 is a flowchart of a gaze detection operation according to the first embodiment. [Figure 8] FIG. 2 is a diagram showing an eye image according to Example 1. [Figure 9] FIG. 2 is a diagram showing an eye image according to Example 1. [Figure 10] 4 is a flowchart of an error determination operation (first method) according to the first embodiment. [Figure 11] FIG. 2 is a diagram showing an eye image according to Example 1. [Figure 12] 10 is a flowchart of an error determination operation (second method) according to the first embodiment. [Figure 13] 4 is a flowchart of the camera operation according to the first embodiment. [Figure 14] 10 is a flowchart of a calibration operation according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] <<Example 1>> Hereinafter, a first embodiment of the present invention will be described with reference to the accompanying drawings.

[0012] <Configuration explanation> 1A and 1B show the appearance of a camera 1 (digital still camera; interchangeable lens camera) according to a first embodiment. FIG. 1A is a front perspective view, and FIG. 1B is a rear perspective view. As shown in FIG. 1A, the camera 1 has a photographing lens unit 1A and a camera housing 1B. A release button 5, which is an operation member that accepts a photographing operation from a user (photographer), is arranged on the camera housing 1B. As shown in FIG. 1B, an eyepiece lens 12 (eyepiece optical system) is arranged on the back of the camera housing 1B, through which the user looks into a display device 10 (display panel) (described below) contained within the camera housing 1B. The eyepiece optical system may include multiple lenses. Also arranged on the back of the camera housing 1B are operation members 41 to 43 that accept various operations from the user. For example, operation member 41 is a touch panel that accepts touch operations, operation member 42 is an operation lever that can be pushed in any direction, and operation member 43 is a four-way key that can be pressed in each of four directions. The operation member 41 (touch panel) is provided with a display panel such as a liquid crystal panel, and has a function of displaying images on the display panel.

[0013] FIG. 2 is a cross-sectional view of the camera 1 taken along the YZ plane defined by the Y axis and Z axis shown in FIG. 1(A), and shows the general internal configuration of the camera 1. As shown in FIG.

[0014] The photographing lens unit 1A includes two lenses 101 and 102, an aperture 111, an aperture driver 112, a lens drive motor 113, a lens drive member 114, a photocoupler 115, a pulse plate 116, a mount contact 117, and a focus adjustment circuit 118. The lens drive member 114 is comprised of a drive gear and other components, and the photocoupler 115 detects the rotation of the pulse plate 116, which is linked to the lens drive member 114, and transmits this information to the focus adjustment circuit 118. The focus adjustment circuit 118 drives the lens drive motor 113 based on information from the photocoupler 115 and information from the camera housing 1B (lens drive amount information), thereby moving the lens 101 and changing the focus position. The mount contact 117 is an interface between the photographing lens unit 1A and the camera housing 1B. Note that for simplicity, two lenses 101 and 102 are shown, but in reality, more than two lenses are included in the photographing lens unit 1A.

[0015] The camera housing 1B contains an image sensor 2, a CPU 3, a memory unit 4, a display device 10, a display device drive circuit 11, etc. The image sensor 2 is a part of the photographing lens unit 1A. The image forming surface is arranged on the display device 10. The CPU 3 is the central processing unit of the microcomputer, and controls the entire camera 1. The memory unit 4 stores images captured by the image sensor 2. The display device 10 is composed of a liquid crystal display or the like, and displays the captured image (subject image) or the like on the display surface of the display device 10. The display device drive circuit 11 drives the display device 10. The user can view the display surface of the display device 10 through the eyepiece 12.

[0016] The camera housing 1B also contains light sources 13a and 13b, a beam splitter 15, a light-receiving lens 16, an eye image sensor 17, and the like. Light sources 13a and 13b are light sources conventionally used in single-lens reflex cameras and the like to detect the gaze direction from the relationship between the pupil and a reflection image of light reflected by the cornea (corneal reflection image; Purkinje image), and serve to illuminate the user's eyeball 14. Specifically, light sources 13a and 13b are infrared light-emitting diodes or the like that emit infrared light insensitive to the user, and are arranged around the eyepiece 12. An optical image of the illuminated eyeball 14 (eye image; an image formed by light emitted from light sources 13a and 13b and reflected by the eyeball 14) passes through the eyepiece 12 and is reflected by the beam splitter 15. The eye image is then formed by the light-receiving lens 16 on the eye image sensor 17, which is a two-dimensional array of photoelectric elements such as a CMOS. The light receiving lens 16 positions the pupil of the eyeball 14 and the ocular imaging element 17 in a conjugate imaging relationship. Using a predetermined algorithm described later, the line of sight direction of the eyeball 14 (the viewpoint (line of sight position) on the display surface of the display device 10) is detected from the position of the corneal reflection image in the eyeball image formed on the ocular imaging element 17.

[0017] 3 is a block diagram showing the electrical configuration within camera 1. Connected to CPU 3 are gaze detection circuit 201, photometry circuit 202, autofocus detection circuit 203, signal input circuit 204, display device drive circuit 11, light source drive circuit 205, etc. CPU 3 also transmits signals via mount contacts 117 to focus adjustment circuit 118 disposed within photographing lens unit 1A and aperture control circuit 206 included in aperture drive section 112 within photographing lens unit 1A. Memory section 4 associated with CPU 3 has the function of storing image capture signals from image sensor 2 and eye image sensor 17, and the function of storing gaze correction parameters that correct for individual differences in gaze, which will be described later.

[0018] The gaze detection circuit 201 A / D converts the output of the eye imaging element 17 (eye image obtained by capturing an image of the eye) when an eyeball image is formed on the eye imaging element 17, and transmits the result to the CPU 3. The CPU 3 extracts feature points required for gaze detection from the eye image according to a predetermined algorithm described later, and calculates the user's gaze (the viewpoint on the display surface of the display device 10) from the positions of the feature points.

[0019] The photometry circuit 202 amplifies, logarithmically compresses, and A / D converts the signal obtained from the image sensor 2, which also functions as a photometry sensor, specifically the luminance signal corresponding to the brightness of the field, and sends the result to the CPU 3 as field luminance information.

[0020] The autofocus detection circuit 203 A / D converts signal voltages from a plurality of detection elements (a plurality of pixels) included in the image sensor 2 and used for phase difference detection, and sends the converted signal to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to each focus detection point from the signals from the plurality of detection elements. This is a well-known technique known as image plane phase difference AF. In the first embodiment, as an example, it is assumed that there are focus detection points at each of 180 locations on the image plane corresponding to the 180 locations shown in the viewfinder field of view image (display surface of the display device 10) in FIG. 4(A).

[0021] The signal input circuit 204 is connected to a switch SW1 that is turned on by the first stroke of the release button 5 to start photometry, distance measurement, line of sight detection, etc. of the camera 1, and a switch SW2 that is turned on by the second stroke of the release button 5 to start a photographing operation. The ON signals from the switches SW1 and SW2 are input to the signal input circuit 204 and transmitted to the CPU 3.

[0022] 4(A) is a diagram showing the field of view within the viewfinder, and shows the state in which the display device 10 is operating (the state in which an image is displayed). As shown in FIG. 4(A), the field of view within the viewfinder includes a focus detection area 400, 180 ranging point indices 401, a field of view mask 402, etc. Each of the 180 ranging point indices 401 is displayed superimposed on a through image (live view image) displayed on the display device 10 so as to be displayed at a position corresponding to a focus detection point on the imaging surface. Furthermore, of the 180 ranging point indices 401, the ranging point indices 401 that corresponds to the current viewpoint A (estimated position) is displayed highlighted with a frame or the like.

[0023] <Explanation of gaze detection operation> The gaze detection method will be described using Figures 5, 6(A), 6(B), and 7. Figure 5 is a diagram for explaining the principle of the gaze detection method and is a schematic diagram of an optical system for performing gaze detection. As shown in Figure 5, light sources 13a and 13b are arranged approximately symmetrically with respect to the optical axis of light-receiving lens 16 and illuminate user's eyeball 14. A portion of the light emitted from light sources 13a and 13b and reflected by eyeball 14 is collected by light-receiving lens 16 onto ocular imaging element 17. Figure 6(A) is a schematic diagram of an eye image captured by ocular imaging element 17 (eyeball image projected onto ocular imaging element 17), and Figure 6(B) is a diagram showing the output intensity of the CMOS in ocular imaging element 17. Figure 7 is a schematic flowchart of the gaze detection operation.

[0024] 7, light sources 13a and 13b emit infrared light toward user's eyeball 14. An image of the user's eyeball illuminated by the infrared light is formed on ocular imaging element 17 through light receiving lens 16 and is photoelectrically converted by ocular imaging element 17. As a result, a processable electrical signal of the eye image is obtained.

[0025] In step S702, the line-of-sight detection circuit 201 sends the eye image (eye image signal; electric signal of the eye image) obtained from the eye imaging device 17 to the CPU 3.

[0026] In step S703, the CPU 3 obtains the coordinates of the points corresponding to the corneal reflection images Pd and Pe of the light sources 13a and 13b and the pupil center c from the eye image obtained in step S702.

[0027] Infrared light emitted from light sources 13a and 13b illuminates cornea 142 of user's eyeball 14. At this time, corneal reflection images Pd and Pe formed by part of the infrared light reflected from the surface of cornea 142 are collected by light receiving lens 16 and formed on ocular imaging element 17 as corneal reflection images Pd' and Pe' in the eye image. Similarly, light beams from edges a and b of pupil 141 are also formed on ocular imaging element 17 as pupil edge images a' and b' in the eye image.

[0028] FIG. 6B shows luminance information (luminance distribution) of region α in the eye image of FIG. 6A. In FIG. 6B, the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. In Example 1, the X-axis (horizontal) coordinates of corneal reflection images Pd' and Pe' are defined as Xd and Xe, and the X-axis coordinates of pupil edge images a' and b' are defined as Xa and Xb. As shown in FIG. 6B, an extremely high level of luminance is obtained at the X-coordinates Xd and Xe of corneal reflection images Pd' and Pe'. In the region from X-coordinate Xa to X-coordinate Xb, which corresponds to the region of the pupil 141 (the region of the pupil image obtained when the light beam from the pupil 141 is focused on the ocular imaging element 17), an extremely low level of luminance is obtained except for the X-coordinates Xd and Xe. A luminance intermediate between the two types of luminance is obtained in the region of iris 143 outside pupil 141 (the region of the iris image outside the pupil image obtained by focusing the light beam from iris 143). Specifically, a luminance intermediate between the two types of luminance is obtained in a region where the X coordinate (coordinate in the X-axis direction) is smaller than X coordinate Xa and a region where the X coordinate is larger than X coordinate Xb.

[0029] From the luminance distribution shown in FIG. 6B, the X-coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X-coordinates Xa and Xb of the pupil edge images a' and b' can be obtained. Specifically, the coordinates of the corneal reflection images Pd' and Pe' can be obtained as the coordinates of extremely high luminance, and the coordinates of the pupil edge images a' and b' can be obtained as the coordinates of extremely low luminance. Furthermore, when the rotation angle θx of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16 is small, the X-coordinate Xc of the pupil center image c' (center of the pupil image) obtained when the light beam from the pupil center c is focused on the ocular imaging element 17 can be expressed as Xc ≒ (Xa + Xb) / 2. In other words, the X-coordinate Xc of the pupil center image c' can be calculated from the X-coordinates Xa and Xb of the pupil edge images a' and b'. In this way, the coordinates of the corneal reflection images Pd' and Pe' and the coordinates of the pupil center image c' can be estimated.

[0030] In step S704, CPU 3 calculates the imaging magnification β of the eyeball image. The imaging magnification β is determined by the position of eyeball 14 relative to light receiving lens 16, and can be calculated using a function of the distance (Xd-Xe) between corneal reflection images Pd' and Pe'.

[0031] In step S705, CPU 3 calculates the rotation angle of the optical axis of eyeball 14 relative to the optical axis of light receiving lens 16. The X coordinate of the midpoint between corneal reflection images Pd and Pe and the X coordinate of the center of curvature O of cornea 142 approximately coincide. Therefore, if the standard distance from the center of curvature O of cornea 142 to the center c of pupil 141 is Oc, then the rotation angle θx of eyeball 14 in the ZX plane (plane perpendicular to the Y axis) can be calculated using the following equation 1. The rotation angle θy of eyeball 14 in the ZY plane (plane perpendicular to the X axis) can also be calculated using a method similar to that for calculating rotation angle θx. β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc (Formula 1)

[0032] In step S706, CPU 3 uses the rotation angles θx, θy calculated in step S705 to determine (estimate) the user's viewpoint (the position where the line of sight is fixed; the position where the user is looking) on ​​the display surface of display device 10. If the coordinates (Hx, Hy) of the viewpoint are coordinates corresponding to the pupil center c, the coordinates (Hx, Hy) of the viewpoint can be calculated using the following equations 2 and 3. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)

[0033] Parameter m in equations 2 and 3 is a constant determined by the configuration of the viewfinder optical system (light receiving lens 16, etc.) of camera 1, and is a conversion coefficient that converts rotation angles θx and θy into coordinates corresponding to the pupil center c on the display surface of display device 10. Parameter m is determined in advance and stored in memory unit 4. Parameters Ax, Bx, Ay, and By are gaze correction parameters that correct for individual differences in gaze, and are acquired by performing a calibration operation, which will be described later. Parameters Ax, Bx, Ay, and By are stored in memory unit 4 before the gaze detection operation begins.

[0034] In step S707, the CPU 3 stores the coordinates (Hx, Hy) of the viewpoint in the memory unit 4, and ends the line-of-sight detection operation.

[0035] <Calibration procedure explanation> Although the gaze point can be estimated by the gaze detection operation described above, it may not be possible to estimate the gaze point with high accuracy due to factors such as individual differences in the shape of the human eyeball. Specifically, if the gaze correction parameters Ax, Ay, Bx, and By are not adjusted to values ​​suitable for the user, the gaze point shown in Figure 4(B) will not be estimated with high accuracy. As shown, a discrepancy occurs between the actual viewpoint B and the estimated viewpoint C. In FIG. 4(B), the user is gazing at a person, but camera 1 mistakenly estimates that the user is gazing at the background, resulting in a state in which proper focus detection and adjustment cannot be performed.

[0036] Therefore, before the camera 1 captures an image, it is necessary to perform a calibration operation to obtain and store in the camera 1 gaze correction parameters suitable for the user.

[0037] Conventionally, calibration work has been performed by highlighting multiple indices at different positions on the display surface of the display device 10 before capturing an image, as shown in FIG. 4(C), and having the user look at the indices. A known technique involves performing a gaze detection operation when the user gazes at each indices, and then calculating gaze correction parameters suitable for the user from the calculated multiple viewpoints (estimated positions) and the coordinates of each indices. Note that the method for displaying the indices is not particularly limited as long as it suggests the position where the user should look. The indices may be displayed as graphics, or the indices may be displayed by changing the brightness or color of an image (such as a captured image).

[0038] <Explanation of conditions that cause gaze detection errors> As described above, in the gaze detection operation, the corneal reflection image and the pupil image (specifically, the coordinates of the corneal reflection image and the coordinates of the center of the pupil image) are detected from the eye image, and based on these detected images, the rotation angles θx, θy of the eyeball 14 are obtained. Then, the rotation angles θx, θy are converted into coordinates corresponding to the pupil center c on the display surface of the display device 10, thereby estimating the gaze point.

[0039] Therefore, if the corneal reflection image or pupil image is incorrectly detected (if the detection error of the corneal reflection image or pupil image is large), the rotation angles θx and θy will not be calculated correctly, resulting in gaze detection errors (failure to detect gaze; increased gaze detection error; erroneous detection of the gaze).

[0040] An example of a state in which an error in gaze detection occurs will be described using Figures 8(A) and 8(B). When a user wears eyeglasses, light from at least one of light sources 13a and 13b (the light source illuminating eyeball 14) may be reflected on the surface of the eyeglasses, pass through eyepiece lens 12, and enter eye image sensor 17, appearing as a ghost in the eye image. Figure 8(A) is a schematic diagram of an eye image captured by eye image sensor 17 (eyeball image projected onto eye image sensor 17), and Figure 8(B) is a diagram showing the output intensity of the CMOS in eye image sensor 17. In Figure 8(A), a ghost 801 caused by the eyeglasses is captured. As shown in Figure 8(B), the output intensity of ghost 801 is equivalent to the output intensity of the corneal reflection image.

[0041] In the states of FIGS. 8(A) and 8(B), the output intensity of ghost 801 is equivalent to the output intensity of the corneal reflection image, so ghost 801 may be erroneously detected as a corneal reflection image. Specifically, the X-coordinate Xf of ghost 801 may be erroneously detected as the X-coordinate Xd of corneal reflection image Pd' or the X-coordinate Xe of corneal reflection image Pe'. When the corneal reflection image is erroneously detected, the imaging magnification β of the eyeball image is not calculated correctly, and the calculation error of the rotation angles θx and θy increases. In this case, if the estimated viewpoint (the viewpoint obtained by converting the calculated rotation angles θx and θy) is located outside the display surface of display device 10, it is easy to determine that an error has occurred in gaze detection (low viewpoint estimation accuracy). However, if the estimated viewpoint is located inside the display surface of display device 10, the viewpoint may be correctly estimated, making it difficult to correctly determine whether an error has occurred in gaze detection (viewpoint estimation accuracy).

[0042] An example of a state in which an error in gaze detection occurs will be described using Figures 9(A) and 9(B). Figure 9(A) is a schematic diagram of an eye image captured by the eye image sensor 17 (eyeball image projected onto the eye image sensor 17), and Figure 9(B) is a diagram showing the output intensity of the CMOS in the eye image sensor 17. In Figure 9(A), a ghost 901 caused by glasses is captured. In Figure 9(B), As shown, the output intensity of the ghost 901 is comparable to the output intensity of the iris image.

[0043] In the states shown in Figures 9(A) and 9(B), the output intensity of the iris image is equivalent to the output intensity of ghost 901, so the boundary between the pupil image and the iris image (X coordinates Xa and Xb of pupil edge images a' and b') may not be correctly detected. If the X coordinates Xa and Xb are not correctly detected, the coordinate of the center of the pupil image (X coordinate Xc of pupil center image c') may not be correctly calculated, increasing the calculation error of the rotation angles θx and θy. In this case, if the estimated viewpoint is located outside the display surface of display device 10 or if the pupil diameter estimated from the X coordinates Xa and Xb is outside the expected range (e.g., outside the range of pupil diameters for a typical adult), it is easy to determine that an error in gaze detection has occurred. However, if the estimated viewpoint is located inside the display surface of display device 10 and the estimated pupil diameter is within the expected range, the viewpoint may be correctly estimated, making it difficult to correctly determine whether an error in gaze detection has occurred.

[0044] As described above, the accuracy of estimating the viewpoint is significantly affected by the detection error of the corneal reflection image and the detection error of the pupil image. In particular, the detection error of the pupil image is likely to be larger than the detection error of the corneal reflection image due to the influence of the corneal reflection image and noise in the ocular imaging element 17 (the detection results of the pupil image are likely to vary). Therefore, a large detection error of the pupil image is likely to deteriorate the accuracy of estimating the viewpoint (the variation in the detection results of the pupil image is likely to cause variation in the estimated viewpoint).

[0045] Specifically, the accuracy of gaze estimation is significantly affected by the calculation error of the interval "Xd-Xe" between multiple corneal reflection images, the calculation error of the center position "(Xd+Xe) / 2" of the multiple corneal reflection images, the calculation error of the center position "Xc" of the pupil image, etc. Therefore, it is possible to determine whether an error in gaze detection has occurred based on the calculation error of the interval "Xd-Xe" between multiple corneal reflection images, the calculation error of the center position "(Xd+Xe) / 2" of the multiple corneal reflection images, the calculation error of the center position "Xc" of the pupil image, etc.

[0046] Although an example of a gaze detection error caused by ghosting has been described, gaze detection errors can also occur due to various other factors. For example, gaze detection errors can occur when eyelids or eyelashes prevent accurate pupil detection. Furthermore, gaze detection errors can occur when the corneal reflection is reflected in tears, resulting in incorrect detection of the corneal reflection. Gaze detection errors can also occur due to the influence of ambient light around the user.

[0047] <Explanation of the first error determination> A first method for determining an error in gaze detection (determining whether gaze detection by the gaze detection circuit 201 is erroneous; determining whether the gaze is correctly detected) will be described. As described above, if error determination is performed based on a single eye image, it is difficult to perform error determination with high accuracy, except in cases where the detection error is extremely large. Therefore, in the first method, error determination is performed based on a current eye image and a previous eye image. Specifically, error determination is performed based on multiple corneal reflection images detected from the current eye image and multiple corneal reflection images detected from the previous eye image. This allows error determination to be performed with high accuracy. The previous eye image may or may not be an eye image captured immediately before the current eye image. A single eye image or multiple eye images may be used as the previous eye image.

[0048] The distance "Xd-Xe" between the multiple corneal reflection images is strongly correlated with the distance from the viewfinder (light receiving lens 16, etc.) to the cornea. When using camera 1, the eyeball 14 looking through the viewfinder does not normally move back and forth in the Z-axis direction (the direction parallel to the optical axis of light receiving lens 16). Therefore, while looking through the viewfinder, the distance "Xd-Xe" between the multiple corneal reflection images is approximately constant.

[0049] Therefore, in the first method, an error is determined by comparing the intervals between the multiple corneal reflection images detected from the current eye image with the intervals between the multiple corneal reflection images detected from the past eye image. This allows for highly accurate error determination. This error determination can be considered as an error determination in gaze detection, or as an error determination in the detection (calculation) of the interval "Xd-Xe" between the multiple corneal reflection images.

[0050] The operation of determining whether or not there is an error in gaze detection (first method) will be described with reference to the flowchart of FIG.

[0051] In step S1001, CPU 3 determines whether multiple corneal reflection images have been detected from the current eye image, and if multiple corneal reflection images have been detected, proceeds to step S1002; if multiple corneal reflection images have not been detected, proceeds to step S1007.

[0052] In step S1007, the CPU 3 determines that the current gaze detection is an error.

[0053] In step S1002, CPU 3 calculates the intervals between multiple corneal reflection images detected from the current eye image (reflection image intervals) and stores the intervals in memory unit 4. If there are three or more light sources illuminating eyeball 14 and three or more corneal reflection images are detected, a representative value (such as the average, maximum, minimum, mode, or median) of the multiple reflection image intervals obtained from the three or more detected corneal reflection images may be calculated. All reflection image intervals obtained from the three or more corneal reflection images may be calculated, or only some of the reflection image intervals may be calculated. For example, if there are four light sources arranged in two rows and two columns, only the interval between two corneal reflection images aligned in the row direction may be calculated, or only the interval between two corneal reflection images aligned in the column direction may be calculated, or only the interval between two corneal reflection images aligned in a diagonal direction may be calculated. Two of these three types of reflection image intervals may be calculated.

[0054] In step S1003, CPU 3 determines whether the interval between reflected images has been calculated a predetermined number of times (for example, 5 times, 10 times, etc.), that is, whether the process of step S1002 has been performed a predetermined number of times or more. If the interval between reflected images has been calculated a predetermined number of times or more, CPU 3 proceeds to step S1004, but if the number of calculations of the interval between reflected images has not reached the predetermined number of times, CPU 3 ends the error determination operation without performing an error determination.

[0055] In step S1004, the CPU 3 acquires from the memory unit 4 a reflection image interval calculated from a past eye image. Although one reflection image interval calculated from a single past eye image may be acquired, in the first embodiment, the CPU 3 acquires one or more reflection image intervals calculated over a predetermined period of time in the past and calculates the average value of the acquired one or more reflection image intervals. Instead of the average value, other representative values ​​such as a maximum value, minimum value, mode, or median value may be calculated. The predetermined period of time in the past may be, for example, a period of five frames up to the present, or a period from the timing five frames before the present to the timing two frames before. to For example, the period.

[0056] In step S1005, the CPU 3 calculates the difference between the reflection image interval acquired in step S1004 (past reflection image interval; average value in the first embodiment) and the reflection image interval calculated in step S1002 (current reflection image interval).

[0057] In step S1006, CPU 3 compares the difference calculated in step S1005 with a predetermined threshold and determines whether the difference is equal to or greater than the predetermined threshold. If the difference is equal to or greater than the predetermined threshold, CPU 3 proceeds to step S1007 and determines that the current gaze detection is an error. If the difference is less than the predetermined threshold, CPU 3 determines that the current gaze detection is successful and ends the error determination operation.

[0058] <Explanation of the second error determination> A second method for determining an error in gaze detection will now be described. In the second method, an error is determined based on a current eye image and a previous eye image. Specifically, in the second method, an error is determined based on a plurality of corneal reflection images and pupil images detected from the current eye image and a plurality of corneal reflection images and pupil images detected from a previous eye image. This allows for highly accurate error determination.

[0059] There is a strong correlation between the center position "(Xd+Xe) / 2" of the multiple corneal reflection images and the center position "Xc" of the pupil image. If there is no translational movement of the head of the user looking through the viewfinder, when the center position "Xc" of the pupil image moves, the center position "(Xd+Xe) / 2" of the multiple corneal reflection images also moves in the same direction as the center position "Xc" of the pupil image. When using camera 1, the head of the user looking through the viewfinder does not normally move continuously in translation. Therefore, movement of only one of the center position "(Xd+Xe) / 2" of the multiple corneal reflection images and the center position "Xc" of the pupil image does not normally occur.

[0060] Therefore, in the second method, the following two types of distances are compared to perform error determination. This allows for highly accurate error determination. This error determination can be considered as an error determination in gaze detection, an error determination in the detection (calculation) of the center position "(Xd+Xe) / 2" of multiple corneal reflection images, or an error determination in the detection (calculation) of the center position "Xc" of the pupil image. The distance from a representative position (e.g., center position) of multiple corneal reflection images detected from the current eye image to a position (e.g., center position) of the pupil image detected from the current eye image The distance from a position (representative position; for example, a central position) that represents a plurality of corneal reflection images detected from a past eye image to a position (for example, a central position) of a pupil image detected from the past eye image.

[0061] However, when there are three or more light sources illuminating the eyeball 14, it is not necessarily possible to detect the same number of corneal reflection images as the light sources, and the representative position (e.g., the central position) of the multiple corneal reflection images will change depending on the number and arrangement of the detected corneal reflection images.

[0062] FIG. 11 is a schematic diagram of an eye image captured by the ocular imaging element 17 (an eyeball image projected onto the ocular imaging element 17). The eye image in FIG. 11 is an eye image captured when there are four light sources illuminating the eyeball 14, and includes four corneal reflection images P1 to P4. If all of the corneal reflection images P1 to P4 are detected, a central position Pc1 of the corneal reflection images P1 to P4 is calculated as the central position of the multiple corneal reflection images. On the other hand, if the corneal reflection images P1 and P2 cannot be detected and only the corneal reflection images P3 and P4 are detected, the central position Pc1 of the corneal reflection images P1 to P4 is not calculated, and instead the central position Pc2 of the corneal reflection images P3 and P4 is calculated as the central position of the multiple corneal reflection images.

[0063] As described above, the representative positions of the multiple corneal reflection images change depending on the number and arrangement of the detected corneal reflection images, and therefore only the representative positions of the multiple corneal reflection images may change significantly. Furthermore, if an eye image in which the number of detected corneal reflection images differs from that of the current eye image is used as a past eye image, the above-described comparison cannot be performed effectively. Therefore, it is preferable to use an eye image that satisfies at least the condition that the number of detected corneal reflection images is approximately the same as that of the current eye image as the past eye image. It is even more preferable to use an eye image that further satisfies the condition that the arrangement of the detected corneal reflection images is approximately the same as that of the current eye image as the past eye image. Even when performing error determination using the first method, it is preferable to use an eye image that satisfies these conditions as the past eye image.

[0064] The operation of determining whether or not there is an error in gaze detection (second method) will be described with reference to the flowchart of FIG.

[0065] In step S1201, CPU 3 determines whether or not the reflection center (the center position of multiple corneal reflection images) and the pupil center (the center position of the pupil image) have been calculated based on the current eye image. If both the reflection center and the pupil center have been calculated, CPU 3 proceeds to step S1202, and if at least one of the reflection center and the pupil center has not been calculated, CPU 3 proceeds to step S1208.

[0066] In step S1208, CPU 3 determines that the current gaze detection is an error.

[0067] In step S1202, CPU 3 determines the current detection status of the corneal reflection (the number and arrangement of multiple corneal reflection images detected from the current eye image) and determines which light source's corneal reflection image has been detected. It is possible that a ghost has been erroneously detected as a corneal reflection image, but such erroneous detection is determined as an error using the first method described above. To reduce the likelihood of a ghost being erroneously detected as a corneal reflection image, the corneal reflection image may be detected taking into consideration the arrangement and size of multiple images detected as candidates for the corneal detection image. If more images are detected as corneal reflection candidates than light sources, multiple viewpoints may be estimated from multiple combinations of candidates with the same number as the number of light sources. Then, each candidate combination with the most likely viewpoint, such as the viewpoint closest to the previous viewpoint, may be detected as a corneal reflection image.

[0068] In step S1203, the CPU 3 calculates the distance between the reflection center of the current eye image and the pupil center, and stores the calculated distance in the memory unit 4.

[0069] In step S1204, CPU 3 determines whether the center-to-center distance (the distance between the reflection center and the pupil center) has been calculated a predetermined number of times (e.g., 5 times, 10 times, etc.) or more when the corneal reflection detection state (the number and arrangement of multiple detected corneal reflections) is approximately the same as the current state. That is, CPU 3 determines whether the process of step S1203 has been performed a predetermined number of times or more when the corneal reflection detection state is approximately the same as the current state. If the center-to-center distance has been calculated a predetermined number of times or more when the corneal reflection detection state is approximately the same as the current state, CPU 3 proceeds to step S1205. If the center-to-center distance has not been calculated the predetermined number of times when the corneal reflection detection state is approximately the same as the current state, CPU 3 terminates the error determination operation without making an error determination.

[0070] In step S1205, the CPU 3 acquires from the memory unit 4 center-to-center distances calculated from past eye images in which the detection state of the corneal reflection is substantially the same as the current eye image. In the first embodiment, the CPU 3 acquires one or more center-to-center distances calculated during a predetermined period (a period of a predetermined length) in the past in which the detection state of the corneal reflection is substantially the same as the current one, and calculates an average value of the acquired one or more center-to-center distances. Instead of the average value, other representative values ​​such as a maximum value, a minimum value, a mode value, or a median value may be calculated. A single center-to-center distance calculated from a single past eye image may also be acquired.

[0071] In step S1206, the CPU 3 calculates the difference between the center-to-center distance acquired in step S1205 (the past center-to-center distance; the average value in the first embodiment) and the center-to-center distance calculated in step S1203 (the current center-to-center distance).

[0072] In step S1207, the CPU 3 compares the difference calculated in step S1206 with a predetermined threshold and determines whether the difference is equal to or greater than the predetermined threshold. If the difference is equal to or greater than the predetermined threshold, the CPU 3 proceeds to step S1208 and determines that the current gaze detection is an error. If the difference is less than the predetermined threshold, the CPU 3 It is determined that the output is successful, and the error determination operation is terminated.

[0073] <Camera operation explanation> The operation of the camera 1 will be described with reference to the flowchart of FIG.

[0074] When the power of the camera 1 is turned on, in step S1301 the image sensor 2 starts acquiring a through image and transmits an image signal of the through image to the CPU 3, and the CPU 3 displays the acquired through image on the display device 10. The user checks the subject by looking at the through image displayed on the display device 10. The power of the camera 1 is turned on / off in response to user operations on the camera 1.

[0075] In step S1302, CPU 3 determines whether or not to turn off the power to camera 1. If it is to be turned off, the camera operation of FIG. 13 is terminated, and if it is not to be turned off, the process proceeds to step S1303.

[0076] In step S1303, CPU 3 starts acquiring an image of the eye of the user who started viewing the through image in step S1301, and performs the gaze detection operation of Fig. 7. Through the gaze detection operation, the coordinates of the pupil image and the corneal reflection image in the eye image are detected, and the coordinates of the viewpoint on the display surface of display device 10 are calculated.

[0077] In step S1304, the CPU 3 performs the error determination operation shown in FIGS.

[0078] In step S1305, CPU 3 determines whether or not the current gaze detection is an error, depending on the result of the error determination operation in step S1304. If the current gaze detection is an error, CPU 3 proceeds to step S1306, and if the current gaze detection is not an error (successful), CPU 3 proceeds to step S1307.

[0079] In step S1306, CPU 3 predicts the current viewpoint from multiple viewpoints calculated over a predetermined period of time in the past. When the processing of step S1306 is performed, the viewpoint predicted in step S1306 is used as the current viewpoint, rather than the viewpoint calculated in step S1303. Note that the method for predicting the viewpoint is not particularly limited. For example, the current viewpoint can be predicted based on the amount of movement and direction of movement of the past viewpoint. If the viewpoint is moving, the current viewpoint can be predicted so that the trajectory of the viewpoint is smoothly extended. If the viewpoint is substantially stationary at one point (such as when swinging around one point), the center position or average position of the multiple viewpoints can be predicted as the current viewpoint.

[0080] In step S1307, CPU 3 superimposes a viewpoint frame on the through image at a position corresponding to the current viewpoint (estimated position) so that a viewpoint frame (a frame indicating the viewpoint) is displayed at the current viewpoint (estimated position) on the display surface of display device 10. This results in a display such as that shown in Fig. 4(A) (a display in which a viewpoint frame is superimposed on a through image), making it possible to notify the user of the current viewpoint A (estimated position). Instead of the viewpoint frame, a point or the like indicating the viewpoint may be displayed.

[0081] In step S1308, the CPU 3 waits for a predetermined time.

[0082] In step S1309, CPU 3 determines whether or not the user has pressed (half-pressed) release button 5 to turn switch SW1 ON. For example, if the user agrees to focus at the position of the viewpoint frame (a frame indicating the estimated viewpoint) displayed superimposed on the through image, CPU 3 half-presses release button 5 to turn switch SW1 ON. If switch SW1 is ON, CPU 3 proceeds to step S1310, and if switch SW1 is not ON, CPU 3 returns to step S1303 to re-estimate the viewpoint. cormorant.

[0083] In step S1310, CPU 3 performs distance measurement at the current position of the line-of-sight frame, and notifies the user that distance measurement has been performed by highlighting the line-of-sight frame, for example by changing its color.

[0084] In step S1311, CPU 3 drives lens 101 in photographing lens unit 1A in accordance with the distance measurement result obtained in step S1310, thereby achieving focus at the position of the viewpoint frame displayed superimposed on the through image.

[0085] In step S1312, CPU 3 determines whether the user has pressed the release button 5 further (fully pressed) to turn switch SW2 ON. For example, if the user agrees to taking a picture at the current focus position, the user presses the release button 5 fully to turn switch SW2 ON. If switch SW2 is ON, CPU 3 proceeds to step S1313, and if switch SW2 is not ON, CPU 3 returns to step S1309.

[0086] In step S1313, the CPU 3 performs a photographing operation to store in the memory unit 4 an image signal acquired by the image sensor 2.

[0087] In step S1314, the CPU 3 displays the image (captured image) stored in the memory unit 4 in step S1313 on the display device 10 for a predetermined time, and returns the process to step S1302.

[0088] <Summary> As described above, according to the first embodiment, not only the current eye image but also the past eye image is used for determining an error in gaze detection. Specifically, the error determination is performed based on the change in the characteristics of the eye image over time. This allows the gaze detection error determination to be performed with high accuracy. Consequently, it is possible to appropriately eliminate gazes (viewpoints) with large detection errors, appropriately prompt the user to improve the visual recognition state, and achieve focusing at a position based on an accurate gaze.

[0089] <<Example 2>> A second embodiment of the present invention will be described below. Note that the following description will omit the same points as in the first embodiment (such as the configuration and processing), and will focus on the points that are different from the first embodiment. In the first embodiment, an example of determining an error while the camera is operating will be described. In the second embodiment, an example of determining an error will be described in which an index that should be seen with the eyes is displayed on the display device 10. The use of the index is not particularly limited, but in the second embodiment, an example of using an index for calibration work will be described.

[0090] As described in the first embodiment, the calibration process is performed by highlighting a plurality of indices at different positions on the display device 10 before capturing an image and having the user look at the indices. For example, as shown in FIG. 4(C), indices are displayed at five locations on the display surface of the display device 10: the center, above the center, below the center, to the left of the center, and to the right of the center. In the second embodiment, the indices are displayed one by one, but it is also possible to display all five indices and switch the highlighted indices sequentially among the five indices. The user looks at the displayed indices (highlighted indices). In other words, in the calibration process, the positions at which the user should look are specified by the indices.

[0091] Even during the calibration process, the gaze detection operation shown in Figure 7 is performed, and gaze detection errors can occur due to various factors. For example, as explained using Figures 8(A), 8(B), 9(A), and 9(B), gaze detection errors can occur due to incorrect detection of the corneal reflection image or pupil image. If the detection result (viewpoint) is used, incorrect gaze correction parameters will be determined, and the gaze will not be corrected properly. If the error detection operations shown in Figures 10 and 12 are performed during the calibration process, incorrect gaze detection results can be excluded and appropriate gaze correction parameters can be determined.

[0092] Furthermore, when the positions of the pupil image and corneal reflection image in the eye image change significantly (i.e., when the user's viewing position (viewpoint) changes significantly), the characteristics of the pupil image and corneal reflection image change significantly due to factors such as uneven exposure caused by peripheral light falloff and lens aberration. For example, the shape of the pupil image, the brightness of the pupil image, the shape of the corneal reflection image, and the brightness of the corneal reflection image change significantly. The spacing between multiple corneal reflection images and the distance between the centers of the multiple corneal reflection images and the center of the pupil image also change significantly. Therefore, when the actual viewpoint changes significantly, using information obtained before the viewpoint change for error detection may result in an erroneous determination of an error in gaze detection even when it was successful. Therefore, it is preferable not to use information obtained before the viewpoint change. During calibration, the viewpoint is likely to change significantly depending on the change in the display position of the index. Therefore, information obtained before the viewpoint change can be easily and suitably excluded to enable highly accurate gaze detection error detection. For example, by using a past eye image captured with the index display position the same as that of the current eye image as a past eye image, gaze detection error detection can be determined with high accuracy.

[0093] <Calibration operation> The calibration operation according to the second embodiment will be described with reference to the flowchart of Fig. 14. The calibration operation is started in response to, for example, a user operation to instruct the start of a calibration job.

[0094] In step S1401, the CPU 3 displays on the display device 10 an index to be gazed at by the user.

[0095] In step S1402, the CPU 3 waits for a predetermined time.

[0096] In step S1403, CPU 3 determines whether or not the user has pressed (halfway pressed) the release button 5 to turn switch SW1 ON. For example, the user may halfway press the release button 5 to turn switch SW1 ON to indicate that they are gazing at the index. If switch SW1 is ON, CPU 3 proceeds to step S1404, and if switch SW1 is not ON, CPU 3 returns to step S1402.

[0097] In step S1404, CPU 3 performs the gaze detection operation of Fig. 7. By the gaze detection operation, the coordinates of the pupil image and corneal reflection image in the eye image are detected, and the coordinates of the viewpoint on the display surface of display device 10 are calculated.

[0098] In step S1405, the CPU 3 performs the error determination operation shown in FIGS.

[0099] In step S1406, CPU 3 determines whether or not the current gaze detection is an error, depending on the result of the error determination operation in step S1405. If the current gaze detection is an error, CPU 3 proceeds to step S1407, and if the current gaze detection is not an error (successful), CPU 3 proceeds to step S1409.

[0100] In step S1407, CPU 3 determines whether the number of executions of the gaze detection operation (number of gaze detection errors) is equal to or greater than a predetermined number. If the number of executions of the gaze detection operation is less than the predetermined number, CPU 3 returns the process to step S1404 and End If so, the process proceeds to step S1408. The number of times the gaze detection operation is performed is counted by the CPU 3.

[0101] In step S1408, CPU 3 determines that calibration (determination of line-of-sight correction parameters) cannot be performed appropriately, notifies the user that the calibration has failed, and then ends the calibration operation.

[0102] In step S1409, the CPU 3 determines whether the number of times the viewpoint has been detected (the number of times the viewpoint has been detected; the number of times the gaze detection has been successful) is equal to or greater than a predetermined number. If the number of times the viewpoint has been detected is less than the predetermined number, the CPU 3 returns the process to step S1404. End If so, the process proceeds to step S1410. The number of times the viewpoint is detected is counted by the CPU 3.

[0103] In step S1410, CPU 3 determines whether gaze detection (the processing of steps S1401 to S1409) has been completed for all indices. If there are any indices for which gaze detection has not been performed, CPU 3 proceeds to step S1411, and if gaze detection has been completed for all indices, CPU 3 proceeds to step S1413.

[0104] In step S1411, the CPU 3 changes (switches) the index displayed in step S1401 to the next index.

[0105] In step S1412, CPU 3 resets information obtained while the index before the change was displayed so that eye images captured while the index before the change was displayed are not used as past eye images in the error determination operation. For example, CPU 3 resets the intervals between multiple corneal reflection images, the distance between the centers of the multiple corneal reflection images and the center of the pupil image, etc. CPU 3 also resets the number of times counted for the processing of steps S1407 and S1409 (the number of times the gaze detection operation was performed and the number of times the gaze point was detected). Then, CPU 3 returns the processing to step S1401.

[0106] In step S1413, the CPU 3 notifies the user that the calibration has been successful.

[0107] In step S1414, CPU 3 calculates line-of-sight correction parameters based on the viewpoint detected for each index, stores the line-of-sight correction parameters in memory unit 4, and ends the calibration operation.

[0108] <Summary> As described above, according to the second embodiment, it is possible to perform a highly accurate determination of an error in gaze detection during the calibration process, and consequently to determine gaze correction parameters with high accuracy. Furthermore, as the past eye image, an eye image captured in a state where the display position of the index that the user should look at is the same as that of the current eye image is used. This makes it possible to easily and suitably exclude information obtained before a significant change in the actual viewpoint, and to perform a more accurate determination of an error in gaze detection.

[0109] It should be noted that Examples 1 and 2 are merely examples, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of Examples 1 and 2 within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of Examples 1 and 2.

[0110] Although an example of detecting the line of sight of a user looking into a camera viewfinder has been described, the present invention is not limited to this. For example, when detecting the line of sight in an HMD (head mounted display) worn on the head to experience VR (virtual reality), etc., the HMD and the eyeball ( HMD The positional relationship between the eyeballs of a user wearing the glasses (eyepieces) is usually approximately constant. Therefore, the present invention can achieve the effects described in the first and second embodiments. Similarly, the present invention can be applied to glasses-type gaze detection devices such as AR (augmented reality) glasses. The present invention can be applied to all electronic devices that estimate the gaze point using eye images. In particular, the present invention is preferably applicable to electronic devices that have an eye window frame (such as a camera eyepiece frame or a glasses frame) that limits the field of view (because the positional relationship between the eye window frame and the eyeballs (eyeballs looking into the eye window frame) is usually approximately constant).

[0111] As described above, according to the first and second embodiments, it is possible to provide a gaze detection device that can realize highly accurate error detection by using not only the current eye image but also past eye images for gaze detection error detection in a camera, HMD, glasses-type gaze detection device, etc.

[0112] <Other Examples> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0113] 1: Camera 3: CPU

Claims

1. a detection means for detecting a line of sight of the eye based on an image of the eye looking at the display surface; A determination means for determining whether or not the detection of the line of sight is an error based on the current eye image and the past eye image. having the detection means detects the line of sight by detecting a plurality of corneal reflection images from the eye image; The determination means is determining whether or not the detection of the line of sight is an error based on a plurality of corneal reflection images detected from the current eye image and a plurality of corneal reflection images detected from the past eye image; When a difference between an interval between the plurality of corneal reflection images detected from the current eye image and an interval between the plurality of corneal reflection images detected from the past eye image is greater than a threshold value, it is determined that the detection of the line of sight is an error. A gaze detection device comprising:

2. the detection means detects the line of sight by further detecting a pupil image from the eye image; The determination means is a distance from a position representative of a plurality of corneal reflection images detected from the current eye image to a position of a pupil image detected from the current eye image; a distance from a position representative of a plurality of corneal reflection images detected from the past eye image to a position of a pupil image detected from the past eye image; If the difference is greater than a threshold, it is determined that the gaze detection is an error.

2. The gaze detection device according to claim 1,

3. The determining means uses, as the past eye image, an eye image that satisfies at least a condition that the number of detected corneal reflection images is substantially the same as that of the current eye image.

3. The gaze detection device according to claim 1, wherein the gaze detection device detects a gaze of a subject.

4. The determining means uses, as the past eye image, an eye image which further satisfies a condition that an arrangement of a plurality of detected corneal reflection images is substantially the same as that of the current eye image.

4. The gaze detection device according to claim 3.

5. the current eye image is an eye image captured in a state in which an index to be viewed by the eye is displayed on the display surface, The determination means uses, as the past eye image, an eye image that satisfies at least a condition that the display position of the index is captured in the same state as that of the current eye image.

5. The gaze detection device according to claim 1, wherein the gaze detection device detects a gaze of a subject.

6. The index is an index for a calibration operation to obtain parameters used in the gaze detection.

6. The gaze detection device according to claim 5,

7. The determining means determines that the detection of the line of sight is not an error when an interval between a plurality of corneal reflection images detected from the eye image is substantially constant with respect to a change in time.

7. The gaze detection device according to claim 1,

8. The determination means determines whether or not the detection of the line of sight is an error based further on a difference between an interval between a plurality of corneal reflection images detected from the current eye image and an interval between a plurality of corneal reflection images detected from the past eye image.

8. The gaze detection device according to claim 1,

9. The determination means is a distance from a position representative of a plurality of corneal reflection images detected from the current eye image to a position of a pupil image detected from the current eye image; a distance from a position representative of a plurality of corneal reflection images detected from the past eye image to a position of a pupil image detected from the past eye image; and determining whether the gaze detection is an error based on the difference between the 9. The gaze detection device according to claim 1,

10. A control means for performing calibration control to obtain a line of sight correction parameter by using a detection result of the detection means based on the current eye image. and When the determination means determines that the gaze detection is an error, the control means does not use the detection result of the detection means based on the current eye image for the calibration control.

10. The gaze detection device according to claim 1,

11. a notification means for notifying a user that the calibration has failed when the determination means determines that the gaze detection is an error. Further having 11. The gaze detection device according to claim 10.

12. A control means for displaying a gaze position using a detection result of the detection means based on the current eye image. Further having 12. The gaze detection device according to claim 1,

13. an acquisition means for acquiring a detection result of a current gaze position based on a detection result of a past gaze position when the determination means determines that the gaze detection is an error; Further having 13. The gaze detection device according to claim 1,

14. a detection step of detecting a line of sight of the eye based on an image of the eye viewing the display surface; a determining step of determining whether or not the detection of the line of sight is an error based on the current eye image and the past eye image; having In the detection step, a plurality of corneal reflection images are detected from the eye image to detect the line of sight, In the determination step, determining whether or not the detection of the line of sight is an error based on a plurality of corneal reflection images detected from the current eye image and a plurality of corneal reflection images detected from the past eye image; When a difference between an interval between the plurality of corneal reflection images detected from the current eye image and an interval between the plurality of corneal reflection images detected from the past eye image is greater than a threshold value, it is determined that the detection of the line of sight is an error. A gaze detection method comprising:

15. A program for causing a computer to function as each of the means of the gaze detection device according to any one of claims 1 to 13.

16. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the gaze detection device according to any one of claims 1 to 13.

Images