Detection device, detection device control method and program

Abstract

To provide a detection device capable of highly accurately detecting an actual pupil center even in an off-axis gaze detection, a control method and a program.SOLUTION: A detection device 2000 includes a gaze detection sensor 20 as acquisition means for acquiring an image of an eyeball 21 of a user, and a gaze detection unit 201 as detection means for detecting a position of a pupil center c of the eyeball 21 from the image, detecting a direction of a gaze O23 based on the position of the pupil center c and detecting a fixation point of the user based on an angle θ24 formed by an optical axis O22 of the gaze detection sensor 20 and the direction of the gaze O23, and the pupil center c.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a detection device that can be mounted on an imaging device such as a camera, a control method for controlling the detection device, and a program. [Background technology]

[0002] Some conventional cameras and video cameras have a gaze detection function that can detect the gaze direction of a user. Detecting the gaze direction enables functions such as focusing point selection. Head-mounted displays with gaze detection functions have also been put to practical use in XR (XR) technical fields, such as virtual reality (VR) and augmented reality (AR). Many electronic devices with such gaze detection functions employ a gaze detection system in which the optical axis of the gaze detection optical system that detects the user's gaze does not coincide with the user's actual gaze. This system is generally referred to as "off-axis gaze detection." In off-axis gaze detection, a gaze detection sensor in the gaze detection optical system is positioned diagonally below the user's eyeball. When performing gaze detection, the gaze detection sensor captures an image of the user's eyeball from diagonally below and detects the coordinates of the pupil center from this image. For example, Patent Document 1 discloses a technology for approximating the shape of the pupil detected from an image of the eyeball to an ellipse to determine the coordinates of the pupil center. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-126850 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology disclosed in Patent Document 1, the image of the eyeball in off-axis gaze detection is distorted due to the influence of perspective. Therefore, even if the shape of the pupil is approximated to an ellipse, this ellipse is still influenced by perspective. Therefore, it is difficult to accurately determine the coordinates of the pupil center from the ellipse influenced by perspective.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a detection device, a control method, and a program that are capable of detecting the actual pupil center with high accuracy even in off-axis line of sight detection. [Means for solving the problem]

[0006] In order to achieve the above object, the detection device of the present invention comprises: The user interacts with the View the display When The device includes an acquisition means for acquiring an image of a user's eyeball, a detection means for detecting the position of an apparent pupil center of the eyeball from the image and detecting the direction of the user's line of sight based on the position of the apparent pupil center, and a calculation means for calculating the actual pupil center position of the user based on the incident vector of light incident from the cornea of ​​the eyeball to the actual pupil center position, calculated using Snell's law, and the angle formed between the optical axis of the acquisition means and the direction of the line of sight detected by the detection means, the at least one lens The optical axes of the two lenses are not coincident. [Effects of the Invention]

[0007] According to the present invention, the actual pupil center can be detected with high accuracy even in off-axis line of sight detection. [Brief explanation of the drawings]

[0008] [Figure 1] 1A and 1B are external views of a camera according to the present embodiment ((a) is a front perspective view, and (b) is a rear perspective view). [Figure 2] FIG. 1 is a block diagram of a camera according to an embodiment of the present invention. [Figure 3] FIG. 2 is a side cross-sectional view of the internal structure of the camera according to the embodiment. [Figure 4] 3 is a schematic diagram for explaining the angle formed by the optical axis of the line-of-sight detection optical system of the camera according to the embodiment and the line of sight of the user. FIG. [Figure 5A] FIG. 2 is a perspective view showing the configuration of an EVF portion of the camera according to the present embodiment. [Figure 5B] FIG. 2 is a side cross-sectional view showing the configuration of the EVF portion of the camera according to the present embodiment. [Figure 6] 10 is a diagram illustrating an optical path when detecting a user's line of sight using the line of sight detection function of the camera according to the present embodiment. FIG. [Figure 7] 1A and 1B are schematic diagrams for explaining the principle of gaze detection of a camera according to an embodiment of the present invention. [Figure 8] 1A and 1B are schematic diagrams of eye images obtained by a camera according to this embodiment ((a) is a schematic diagram of an eye image captured by a gaze detection sensor, and (b) is a diagram showing the output intensity (brightness distribution) output by the gaze detection sensor when capturing an eye image). [Figure 9] 10 is a flowchart of a gaze detection program executed by the camera according to the present embodiment. [Figure 10A] 10A and 10B are schematic diagrams for explaining a method for correcting the coordinates of the pupil center based on the pupil diameter (when the pupil diameter is small) in the camera according to the present embodiment. [Figure 10B] 10A and 10B are schematic diagrams for explaining a method for correcting the coordinates of the pupil center based on the pupil diameter (when the pupil diameter is large) in the camera according to the present embodiment. [Figure 11A] FIG. 2 is a cross-sectional view of the gaze detection sensor and the user's cornea cut along the YZ plane formed by the Y axis and Z axis of the camera according to the embodiment, illustrating the refraction of light at the cornea. [Figure 11B] 10A and 10B are schematic diagrams for explaining a method for correcting a shift in the coordinates of the pupil center due to refraction at the cornea using the camera according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to FIGS. 1 to 11B. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited to the configurations described in the embodiments. In this embodiment, an example is shown in which a detection device of the present invention is applied to a camera. As a result, the camera has a gaze detection function. The detection device can also be applied to devices (electronic devices) that display information such as images and text. Specific examples of such electronic devices include mobile phones, game consoles, tablet terminals, personal computers, watch-type or eyeglass-type information terminals, head-mounted displays, binoculars, etc.

[0010] As shown in FIG. 1(a), camera 1 is, for example, a lens-interchangeable digital still camera and includes a photographing lens unit 1A and a camera housing 1B. Hereinafter, the width direction of camera housing 1B is defined as the X-axis direction, the height direction as the Y-axis direction, and the optical axis direction of photographing lens unit 1A as the Z-axis direction. Camera housing 1B includes a release button 5 located on the upper front side. Release button 5 is an operation member that accepts image capture operations from the user (the photographer). As shown in FIG. 1(b), camera housing 1B includes an eyepiece window frame 121, operation members 41, 42, and 43 located on the rear surface of camera housing 1B. Eyepiece window frame 121 has a viewing hole 13 formed therein, which protrudes outward (toward the rear surface) from camera housing 1B. The user can view first display panel (display device) 6 by looking through viewing hole 13. Operation members 41 to 43 are members that accept various operations from the user. Operation member 41 is a touch panel that accepts touch operations. The operation member 41 has a display panel such as a liquid crystal panel, and can display images on the display panel. The operation member 42 is an operation lever with markers indicating directions that can be pushed in the direction of each marker. The operation member 43 is a four-way key that can be pushed in each of four directions.

[0011] 2, photographing lens unit 1A has an optical system (not shown) including a zoom lens, a focus lens, an aperture, etc., as well as aperture control unit 118, focus adjustment unit 119, and zoom control unit 120. When photographing lens unit 1A is attached to camera housing 1B, it guides a light beam from a subject to image sensor 2 in camera housing 1B, and forms an image of the subject on the imaging surface of image sensor 2. Aperture control unit 118, focus adjustment unit 119, and zoom control unit 120 each receive instruction signals from CPU 3 in camera housing 1B via mount contact unit 117, and drive and control the aperture, focus lens, and zoom lens in accordance with these instruction signals.

[0012] The camera housing 1B has an image sensor 2. The image sensor 2 is an image sensor such as a CCD or CMOS sensor, and photoelectrically converts an optical image formed on the imaging surface of the image sensor 2 by the optical system of the photographing lens unit 1A, outputs the resulting analog image signal to an A / D converter, and outputs it as image data. The camera housing 1B has a CPU 3 and a memory unit (storage unit) 4 including a ROM, RAM, etc. The CPU 3 reads control programs for each block (each means and each unit) of the camera housing 1B from the ROM of the memory unit 4, expands them into the RAM of the memory unit 4, and executes them. In this embodiment, the memory unit 4 can store image signals from the image sensor 2 and the line-of-sight detection sensor 20. The memory unit 4 also stores programs for causing the CPU 3, which is a computer, to execute each block of the camera 1 (to execute a control method for controlling the detection device).

[0013] The camera housing 1B includes a gaze detection unit (detection means) 201, a photometry unit 202, an autofocus detection unit 203, a signal input unit 204, a light source driver 205, an image processor 206, a recording / output unit 207, and a display device driver 116, all of which are electrically connected to the CPU 3. The gaze detection unit 201 will be described later. The photometry unit 202 processes signals obtained from the image sensor 2, which also functions as a photometry sensor, by amplifying, logarithmically compressing, and A / D converting the luminance signal corresponding to the brightness of the subject, and sends the processed results to the CPU 3 as subject luminance information. The autofocus detection unit 203 A / D converts signal voltages from multiple detection elements (multiple pixels) used for phase difference detection within the image sensor 2 (CCD), and sends the converted signal to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to each focus detection point from the signals from the multiple detection elements. This is a well-known technology known as image-plane phase difference AF (autofocus). In this embodiment, as an example, the viewfinder 13 of the eyepiece window frame 121, i.e., the field of view image (image for visual confirmation) in the viewfinder, is divided, and a focus detection point is located at each of the 180 divided locations on the imaging surface of the image sensor 2. The signal input unit 204 is connected to a switch SW1 that is turned on by the first stroke of the release button 5 to start photometry, distance measurement, line of sight detection, and other operations of the camera 1, and a switch SW2 that is turned on by the second stroke of the release button 5 to start a photographing operation. The ON signals from the switches SW1 and SW2 are input to the signal input unit 204 and then transmitted to the CPU 3. The signal input unit 204 also accepts operation inputs from the operation members 41 to 43. The image processing unit 206 performs various image processing operations on image data stored in the RAM of the memory unit 4. Image processing includes various types of image processing for developing, displaying, and recording digital image data, such as pixel defect correction processing due to the optical system of the photographing lens unit 1A or the image sensor 2, demosaicing processing, white balance correction processing, color interpolation processing, gamma processing, etc. The recording / output unit 207 records data including image data on a recording medium such as a memory card that is detachable from the camera housing 1B, and outputs this data to an external device via an external interface.

[0014] As shown in FIG. 3 , the camera housing 1B includes a shutter 7 disposed forward of the image sensor 2 in the optical axis direction, a first display panel 6 disposed in front of the viewing hole 13, and a second display panel 8 disposed on the rear side of the camera housing 1B. The first display panel 6 is an EVF (Electronic Viewfinder). The first display panel 6 can display, for example, menus and images for operating the camera 1 or viewing and editing images acquired by the camera 1. The first display panel 6 also performs a function of detecting the line of sight of a user looking through the viewing hole 13. This detection result is reflected in the control of the camera 1. Like the first display panel 6, the second display panel 8 can display, for example, menus and images for operating the camera 1 or viewing and editing images acquired by the camera 1. The second display panel 8 is formed, for example, of a backlit LCD panel or an organic EL panel. The viewing hole 13 is a transparent member that transmits visible light. Images and the like displayed on the first display panel 6 are observed through the viewing hole 13. Furthermore, camera housing 1B has lens group 12 arranged between viewing hole 13 and first display panel 6. Lens group 12 is made up of lens 9, lens 10, and lens 11, but the number of lenses is not limited to three.

[0015] As described above, the camera 1 has a gaze detection function. This gaze detection function will be described below. Note that the "gaze detection function" refers to a function of the camera 1 to detect the gaze O23 of the user.

[0016] As shown in FIG. 2, the camera housing 1B has a detection device 2000 mounted therein. The detection device 2000 has a gaze detection unit 201, a gaze detection sensor 20 functioning as an eye image sensor, and infrared LEDs 14 and 15 functioning as light sources for irradiating a user's eyeball 21 with infrared light. The gaze detection unit 201 detects the user's gaze O23 (detection process). The gaze detection unit 201 A / D converts the output of the gaze detection sensor 20 when an eyeball image is formed on the gaze detection sensor 20, i.e., an eye image (image) capturing the eyeball 21, and transmits the result to the CPU 3. The gaze detection unit 201 has a pupil center detection unit (pupil center detection means) 208, a gaze / gazing point detection unit (gaze / gazing point detection means) 209, and a gaze point correction unit (gazing point correction means) 210. The CPU 3 operates the pupil center detection unit 208, the gaze and gaze point detection unit 209, and the gaze point correction unit 210 to extract feature points required for gaze detection from the eye image according to a predetermined algorithm described below. The user's gaze O23 (the gaze point in the viewing image) is then detected from the positions of these feature points. The pupil center detection unit 208 detects the position of the pupil center c of the eyeball 21 from the image of the eyeball 21 (pupil center detection process). The gaze and gaze point detection unit 209 detects the direction of the gaze O23 and the position of the gaze point based on the position of the pupil center c detected by the pupil center detection unit 208 (using the position of the pupil center c) (gaze and gaze point detection process). Here, the "gaze point" refers to the position at which the user's gaze O23 is fixed, i.e., the position at which the user is looking. The gaze point correction unit 210 corrects the position of the gaze point (gaze point correction step) based on the angle θ24 (see FIG. 4) formed between the optical axis O22 of the gaze detection optical system (optical system) 19 (described later) and the direction of the gaze O23 detected by the gaze / gaze point detection unit 209. Then, this corrected position can be set as the actual pupil center c.

[0017] As shown in FIGS. 5A and 5B, the infrared LEDs 14 and 15 are positioned so that they can emit infrared light IL toward the viewing hole 13. As a result, as shown in FIG. 6, the infrared light IL from the infrared LEDs 14 and 15 passes through the viewing hole 13 and is irradiated onto the user's eyeball 21. For gaze detection, a corneal reflection image obtained by specular reflection of the infrared light IL from the cornea 211 is also used in addition to the eyeball image generated by the infrared light illumination. An illumination window 16 is disposed between the infrared LEDs 14 and 15 and the viewing hole 13. The illumination window 16 restricts the visibility of the infrared LEDs 14 and 15 from the user, i.e., it conceals the infrared LEDs 14 and 15. The illumination window 16 is made of a resin material that absorbs visible light and transmits the infrared light IL. The camera housing 1B also includes a gaze detection optical system 19. The gaze detection optical system 19 has an aperture 17 and a gaze imaging lens (lens) 18. The gaze detection optical system 19 is an optical system that forms an image around the eyeball 21 on the gaze detection sensor 20. The gaze detection sensor (acquisition means) 20 is an imaging means that captures (acquires) an image of the user's eyeball 21 via the gaze detection optical system 19 (acquisition process). This allows the gaze detection unit 201 to detect the user's gaze O23 from the image of the eyeball 21 acquired by the gaze detection sensor 20 (acquisition process). The gaze detection sensor 20 is composed of an imaging element such as a CCD or CMOS sensor.

[0018] 7, the infrared LEDs 14 and 15 are arranged approximately symmetrically with respect to the optical axis O18 of the line-of-sight imaging lens 18, and irradiate the cornea 211 of the user's eyeball 21 with infrared light IL. This infrared light IL includes infrared light IL that is reflected by the cornea 211. A portion of this reflected infrared light IL is collected by the line-of-sight imaging lens 18 onto the line-of-sight detection sensor 20. This results in the aforementioned corneal reflection image.

[0019] The pupil 212 and iris 213 of a human eyeball 21 are generally nearly circular in shape. Therefore, when the pupil 212 and iris 213 are imaged from the front, the images of the pupil 212 and iris 213 appear nearly circular. However, in off-axis gaze detection, the gaze detection sensor 20 is positioned so as to look up at the eyeball 21 from diagonally below, and therefore, as shown in FIG. 8( a), the images of the pupil 212 and iris 213 are distorted and appear bulging downward rather than circular.

[0020] The program for the line-of-sight detection operation that demonstrates the line-of-sight detection function will be described with reference to the flowchart shown in FIG.

[0021] When the gaze detection operation program is started, the processes of steps S801 and S802 are executed sequentially. In step S801, the CPU 3 activates the infrared LEDs 14 and 15 via the light source drive unit 205, thereby causing the infrared LEDs 14 and 15 to irradiate the user's eyeball 21 (cornea 211) with infrared light IL. An image of the user's eyeball formed by the illumination of the infrared light IL passes through the gaze imaging lens 18 and is formed on the gaze detection sensor 20. As a result, the eyeball image is photoelectrically converted by the gaze detection sensor 20, and an electrical signal of the eye image (eye image signal) is obtained. In step S802, the gaze detection unit 201 transmits the eye image obtained by the gaze detection sensor 20, i.e., the electrical signal of the eye image.

[0022] Next, step S803 is executed. In step S803, the pupil center detection unit 208 calculates (calculates) the coordinates of points corresponding to the corneal reflection image Pd, the corneal reflection image Pe, and the pupil center c (see FIG. 7 for all) from the eye image obtained in step S802. The calculation process (subroutine) of step S803 will be described later.

[0023] Next, step S804 is executed. In step S804, the pupil center detection unit 208 calculates (detects) the position of the center of curvature O of the cornea 211 and the position of the pupil center c in real space based on the coordinates of the corneal reflection images Pd and Pe detected in step S803 and the coordinates of the pupil center c. The calculation process (subroutine) of step S804 will be described later.

[0024] Next, step S805 is executed. In step S805, the gaze / gazing point detection unit 209 calculates the rotation angle θx of the optical axis O211 of the cornea 211 relative to the optical axis O18 of the gaze imaging lens 18 based on the position of the center of curvature O and the position of the pupil center c calculated in step S804. If the standard distance from the center of curvature O to the pupil center c is Oc (see FIG. 7), the rotation angle θx in the ZX plane (plane perpendicular to the Y axis) can be calculated using the following equation (1). In the equation, Xc indicates the position of the pupil center in real space, and Xo indicates the position of the center of curvature of the cornea 211 in real space. In addition, the rotation angle θy of the cornea 211 in the ZY plane (plane perpendicular to the X axis) can be calculated using a method similar to that for calculating the rotation angle θx. Then, the direction that satisfies both the rotation angles θx and θy is detected as the direction of the user's gaze O23. Oc×Sinθx≒Xc-Xo (1)

[0025] Next, step S806 is executed. In step S806, the gaze / gazing point detection unit 209 uses the rotation angle θx and rotation angle θy calculated in step S805, i.e., based on the direction of the gaze O23, to detect (estimate) the user's gaze point in the viewing image displayed on the first display panel 6. If the coordinates (Hx, Hy) of the gaze point are coordinates corresponding to the pupil center c, the coordinates (Hx, Hy) of the gaze point can be calculated by the following equations (2) and (3). Hx = m × (Ax × θx + Bx) (2) Hy=m×(Ay×θy+By) (3)

[0026] The parameter m in equations (2) and (3) is a constant determined by the configuration of the viewfinder optical system (such as the line-of-sight imaging lens 18) of the camera 1, and is a conversion coefficient that converts the rotation angle θx and the rotation angle θy into coordinates corresponding to the pupil center c in the visual image. This parameter m is determined in advance and stored in the memory unit 4. The parameters Ax, Bx, Ay, and By are each gaze correction parameters that correct individual differences in the gaze, and are acquired by performing a predetermined calibration. Furthermore, the parameters Ax to By are stored in the memory unit 4 in advance before the gaze detection operation program is started. Note that the "predetermined calibration" is performed, for example, by highlighting multiple indices at different positions on the screen displayed on the first display panel 6 before capturing an image and having the user look at the indices. A technology is known in the art that performs gaze detection when gazing at each index, and calculates viewpoint correction parameters suitable for the user from the calculated multiple viewpoints (estimated positions) and the coordinates of each index (see, for example, Patent Publication No. 2021-64928).

[0027] Next, step S807 is executed. In step S807, the gaze detection unit 201 stores the coordinates (Hx, Hy) of the gaze point in the memory unit 4. This ends the gaze detection operation program.

[0028] Next, the process of calculating the coordinates of the corneal reflection image Pd, the corneal reflection image Pe, and the points corresponding to the pupil center c in step S803 will be described.

[0029] Infrared light IL emitted from infrared LED 14 and infrared LED 15 illuminates the user's cornea 211. At this time, corneal reflection images Pd and Pe formed by a portion of the infrared light IL reflected from the surface of the cornea 211 are condensed by gaze imaging lens 18 and formed on gaze detection sensor 20. As a result, corneal reflection images Pd and Pe (see FIG. 7) become corneal reflection images Pd' and Pe' (see FIG. 8(a)), respectively, in the eye image. Similarly, light beams from ends a and b (see FIG. 7) of pupil 212 are also formed on gaze detection sensor 20 and become pupil edge images a' and b' (see FIG. 8(a)) in the eye image.

[0030] FIG. 8(b) shows the luminance distribution (luminance information) of region α' in the eye image in FIG. 8(a). The horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction. In this embodiment, the X-axis (horizontal) coordinates of corneal reflection images Pd' and Pe' are designated Xd and Xe, and the X-axis coordinates of pupil edge images a' and b' are designated Xa and Xb. As shown in FIG. 8(b), an extremely (significantly) high level of luminance is obtained at coordinate Xd of corneal reflection image Pd' and coordinate Xe of corneal reflection image Pe'. Furthermore, in region A1 between coordinates Xa and Xb, which corresponds to the region of pupil 212 (the region of the pupil image obtained by focusing the light beam from pupil 212 on gaze detection sensor 20), an extremely low level of luminance is obtained at each of the coordinates, except for coordinates Xd and Xe. Furthermore, in the region of the iris 213 outside the pupil 212 (the region of the iris image outside the pupil image obtained by focusing the light beam from the iris 213), a luminance intermediate between the two types of luminance is obtained. Specifically, in the X coordinate, region A2 smaller than coordinate Xa and region A3 larger than coordinate Xb each have a luminance intermediate between the two types of luminance. From this luminance distribution, the X coordinate Xd of the corneal reflection image Pd', the X coordinate Xe of the corneal reflection image Pe', the X coordinate Xa of the pupil edge image a', and the X coordinate Xb of the pupil edge image b' can be obtained. Specifically, the X coordinates with the extremely highest luminance can be obtained as the X coordinate Xd of the corneal reflection image Pd' and the X coordinate Xe of the corneal reflection image Pe'. Furthermore, coordinates with extremely low brightness (compared to the brightness of the X coordinates Xd and Xe) can be obtained as the X coordinate Xa of pupil edge image a' and the X coordinate Xb of pupil edge image b'. Then, by moving area α' in the Y-axis direction, the entire area of ​​the eye image can be scanned. This allows a group of edge coordinates of the pupil image to be obtained. The pupil center detection unit 208 detects the coordinate (position) of the brightness center of gravity within this group of edge coordinates in the eye image (image) as the "coordinate (position) of the provisional pupil center c'." The coordinate of pupil center c' is the provisional coordinate of the pupil center c, which is the detection target of the pupil center detection unit 208. The coordinate Xc in FIG. 8(b) is the coordinate of pupil center c' in the X-axis direction and is approximated to (Xa + Xb) / 2.

[0031] Next, the process of calculating the position of the center of curvature O of the cornea 211 and the position of the pupil center c in real space in step S804 will be described. The pupil center detection unit 208 calculates the imaging magnification β of the gaze detection optical system 19. The "imaging magnification β" is a magnification determined by the position of the eyeball 21 relative to the gaze imaging lens 18, and can be obtained using a function of the distance |Xd-Xe| between the corneal reflection image Pd' and the corneal reflection image Pe'. Then, the positions of the center of curvature O of the cornea 211 and the pupil center c in real space can be calculated by multiplying the image height by the imaging magnification β. In this way, in the gaze detection unit 201, the pupil center detection unit 208 is configured to be able to detect the position of the center of curvature O of the cornea 211 and the position of the pupil center c from the eye image (image of the eyeball 21).

[0032] 10A and 10B are diagrams showing a pupil image and an iris image, respectively, of an eye image acquired by the gaze detection sensor 20. As described above, in off-axis gaze detection, the gaze detection sensor 20 is positioned so that the user's eyeball 21 is viewed from diagonally below. Therefore, as shown in FIGS. 10A and 10B, the pupil image acquired by the gaze detection sensor 20 appears bulging downward. Therefore, the coordinates of a tentative pupil center c' detected from the pupil image (eye image) by the pupil center detection unit 208 do not match the coordinates of the actual, i.e., correct, pupil center c. Therefore, in the gaze detection unit 201, the gaze point correction unit 210 corrects the coordinates of the pupil center c' by calculating a correction value v1. As described above, the gaze / gaze point detection unit 209 detects (estimates) the gaze point. In this embodiment, the coordinates (Hx, Hy) of this gaze point correspond to the pupil center c'.

[0033] Humans reflexively adjust the amount of light reaching the retina by changing the pupil diameter according to the light intensity. In response to strong light, the pupil 212 becomes smaller (see FIG. 10A). On the other hand, in response to weak light, the pupil 212 becomes larger (see FIG. 10B). The distance between the coordinates of a virtual pupil center c' estimated from an image and the coordinates of the actual pupil center c' (hereinafter referred to as the "inter-coordinate distance") varies depending on the pupil diameter. There is a correlation between the inter-coordinate distance and the pupil diameter. The inter-coordinate distance can be equivalent to a correction value v1. Thus, the correction value v1 is calculated based on the user's pupil diameter. The correction value v1 is expressed by the following equation (4) using the angle θ24 formed between the optical axis O22 of the gaze detection optical system 19 and the direction of the gaze O23, and the radius r of the pupil 212. v1 = f(θ24, r) (4)

[0034] The function f is a function proportional to the angle θ24 and the radius r. If there is a coefficient, the coefficient is calculated based on data obtained by simulation or experiment and is held as a value specific to the camera.

[0035] Then, the gaze point correction unit 210 moves the gaze point coordinates (Hx, Hy), which are coordinates corresponding to the pupil center c', by the correction value v1. The moved coordinates become the coordinates of the actual pupil center c. In this way, in the gaze detection unit 201, the gaze point correction unit 210 can correct the position of the pupil center c' (gazing point) based on the angle θ24 and the pupil diameter. This makes it possible to detect the actual pupil center c (gazing point) with high accuracy even in off-axis gaze detection. Note that the method and configuration for detecting the pupil diameter are not particularly limited, and for example, the pupil center detection unit 208 may be configured to be able to detect the pupil diameter. In this case, for example, |Xa-Xb| can be used as the pupil diameter.

[0036] Furthermore, since the position of the pupil center c' can be specified to one point within the eye image, the computational load can be reduced compared to, for example, a method in which points in real space corresponding to all points of the edge coordinate group of the pupil image are found and then the pupil center position is calculated. This enables rapid detection of the pupil center c'. Furthermore, in this embodiment, the coordinates of the provisional pupil center c' are estimated from a feature point (brightness center) of the pupil image, but the estimation method is not limited to this. Furthermore, the feature point is not limited to the brightness center, and may be, for example, the geometric center of the pupil image.

[0037] Because the refractive index inside the cornea differs from the refractive index of the air outside the cornea, as shown in Figures 11A and 11B, infrared light IL is refracted at the boundary between the inside of the cornea and the air outside. This causes the actual pupil center position to not coincide with the apparent pupil center position. At the boundary surface, infrared light IL is refracted so that the refraction vector satisfies Snell's law (law of refraction) with respect to the incident vector and normal vector. Snell's law is expressed by the following equation (5) using the refractive index N1 of the cornea 211 and the refractive index N2 of air. θ1 is the angle of incidence from the inside of the cornea to the air, and θ2 is the angle of refraction from the inside of the cornea to the air. N1×Sinθ1=N2×Sinθ2 (5)

[0038] Therefore, the incident vector can be calculated using the refractive index, refraction vector, and normal vector of the cornea 211, and the actual pupil center position can be determined. The gaze detection unit 201 corrects the coordinates of the pupil center c' in the eye image, i.e., the apparent pupil center position, using a correction value v2, thereby making it possible to detect the actual position of the pupil center c in real space while taking refraction at the cornea 211 into consideration. The correction value v2 is calculated based on the angle θ24 formed between the optical axis O22 of the gaze detection optical system 19 and the direction of the gaze O23 (the optical axis of the display optical system: the direction of the gaze when the user looks at the center (centre) of the display unit). The correction value v2 is expressed by the following equation (6) using the angle θ24. v2 = g(θ24, θ1) (6)

[0039] The function g is proportional to the angle θ24 and the angle of incidence θ1. The angle θ24 is a fixed value. If there is a coefficient, this coefficient is calculated based on data obtained through simulation or experiment and is stored as a value specific to the camera.

[0040] Then, the gaze point correction unit 210 moves the gaze point coordinates (Hx, Hy) corresponding to the pupil center c' by the correction value v2. These moved coordinates become the coordinates of the actual pupil center c (gaze point). In this way, in the gaze detection unit 201, the gaze point correction unit 210 can correct the position of the pupil center c' (gaze point) based on the angle θ24, the refractive index of the cornea 211, and the refractive index of air. This allows the actual pupil center c to be detected with high accuracy in off-axis gaze detection, as when the correction value v1 is used. Note that the refractive index of the cornea 211 of the eyeball 21 and the refractive index of air are preferably stored in the memory unit 4 in advance. This allows the correction value v2 to be calculated quickly.

[0041] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications and variations are possible within the scope of the gist of the present invention. The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or storage medium, and having one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions. [Explanation of symbols]

[0042] 1 camera 19. Line of sight detection optical system (optical system) 20. Eye gaze detection sensor 21 Eyeball 201 Gaze detection unit 207 Recording / Output Section 208 Pupil center detection unit (pupil center detection means) 209 Gaze / gazing point detection unit (gaze / gazing point detection means) 210 Gaze point correction unit (gazing point correction means) 2000 Detector

Claims

1. An acquisition means for acquiring an image of a user's eyeball when the user looks at a display unit through at least one lens; a detection means for detecting the position of an apparent pupil center of the eyeball from the image and detecting the direction of the user's line of sight based on the position of the apparent pupil center; a calculation means for calculating the actual pupil center position of the user based on an incident vector of light incident from the cornea of ​​the eyeball to the actual pupil center position, calculated using Snell's law, and an angle formed between the optical axis of the acquisition means and the direction of the line of sight detected by the detection means, A detection device, characterized in that the optical axis of said acquisition means and the optical axis of said at least one lens do not coincide.

2. a memory unit capable of storing the refractive index of the cornea and the refractive index of air, 2. The detection device according to claim 1, wherein the calculation means uses the refractive index of the cornea and the refractive index of air when calculating the actual pupil center position.

3. 3. The detection device according to claim 1, wherein the detection means is capable of detecting the position of the center of curvature of the cornea of ​​the eyeball from the image, and the position of the center of curvature of the cornea is used when detecting the direction of the gaze.

4. 4. The detection device according to claim 1, wherein the detection means detects the position of the luminance center of the image as the position of the apparent pupil center.

5. An acquisition step of acquiring an image of the user's eyeball when the user looks at the display unit through at least one lens by an acquisition means; a detection step of detecting a position of an apparent pupil center of the eyeball from the image and detecting a direction of the user's gaze based on the position of the apparent pupil center; a calculation step of calculating the actual pupil center position of the user based on the incident vector of light incident from the cornea of ​​the eyeball to the actual pupil center position, calculated using Snell's law, and the angle formed by the optical axis of the acquisition means, which does not coincide with the optical axis of the at least one lens, and the direction of the line of sight detected in the detection step.

6. A program for causing a computer to execute each means of the detection device according to any one of claims 1 to 4.

Images