Electronic devices, methods for controlling electronic devices, programs, and storage media

Abstract

To suppress uneven brightness of an image of a captured eyeball.SOLUTION: Electronic equipment of the present invention includes: first acquisition means for acquiring an image of an eyeball illuminated by a plurality of light sources and imaged by imaging means; second acquisition means for acquiring distance information showing a distance from the eyeball to the imaging means; and correction means for correcting uneven brightness of the image with a correction amount according to the distance information.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present invention relates to an electronic device, a control method for an electronic device, a program, and a storage medium.

Background Art

[0002] A method for detecting at which position on an observation surface a user (observer) is observing has been proposed. For example, in the technique disclosed in Patent Document 1, parallel light beams from a plurality of light sources are projected onto the cornea (front part of the eye) of the user's eyeball, and the line of sight is detected by using the positional relationship between the corneal reflection image formed by the reflected light from the cornea and the pupil center. Also, in the technique disclosed in Patent Document 2, the pupil circle is estimated based on the horizontal / vertical coordinates of the corneal reflection image, and the center coordinates of the pupil circle are used for line-of-sight detection.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the techniques disclosed in Patent Document 1 or Patent Document 2, when the distance from a plurality of light sources to the user's eyeball is short, unevenness in the luminance of the eye image (captured image of the eyeball) occurs more significantly than when the distance is long. In particular, unevenness in luminance occurs more significantly in a configuration where a light-receiving lens and an area sensor are arranged outside the eyepiece lens.

[0005] Therefore, an object of the present invention is to suppress unevenness in the luminance of the captured image of the eyeball.

Means for Solving the Problems

[0006] A first aspect of the present invention includes a first acquisition means for acquiring an image of an eyeball illuminated by a plurality of light sources and captured by an imaging means, and a second acquisition means for acquiring distance information indicating the distance from the eyeball to the imaging means, The detection means performs a detection process to detect the line of sight position from the image, and when the calibration operation of the detection process is in progress, it stores either the distance information or a correction value corresponding to the distance information in a storage medium, and when the calibration operation is not in progress, it uses the correction value corresponding to the distance information during the calibration operation. The electronic device is characterized by having a correction means for correcting brightness unevenness in the aforementioned image.

[0007] A second aspect of the present invention comprises a plurality of light sources for illuminating an eyeball, imaging means for imaging the eyeball illuminated by the plurality of light sources, and control means for controlling the plurality of light sources. The plurality of light sources are classified into a plurality of light source groups, each capable of individually controlling the amount of light emitted, and include a first light source group, a second light source group whose distance from the imaging means is longer than the distance from the imaging means to the first light source group, and a third light source group whose distance from the imaging means is greater than or equal to the distance from the imaging means to the second light source group. The number of light sources included in the second light source group is less than the number of light sources included in the third light source group. The control means is The emission amount of the second light source group is made greater than that of the first light source group, but the emission amount of the third light source group is not made greater than that of the second light source group. This is an electronic device characterized by the following features.

[0009] This invention 3 The embodiment involves illuminating the eyeball with multiple light sources and acquiring an image of the eyeball captured by an imaging means. Rus The system acquires distance information indicating the distance from the eyeball to the imaging means. Rus Step and, The steps include: performing a detection process to detect the line of sight position from the image, and, if the calibration operation of the detection process is in progress, the distance information or, according to the distance information The steps include storing one of the correction values ​​in a storage medium, and, if the calibration operation is not in progress, using the correction value corresponding to the distance information during the calibration operation. Correcting brightness unevenness in the aforementioned image. Rus This is a control method for electronic equipment characterized by having a step.

[0011] This invention 4 The aspect is computer The above-described control method for electronic devices is then executed. This is a program designed to do that.

[0012] This invention 5 The aspect is computer The above-described control method for electronic devices is then executed. It is a computer-readable storage medium that contains a program to perform a specific action. [Effects of the Invention]

[0013] According to the present invention, brightness unevenness in the captured image of the eyeball can be suppressed. [Brief explanation of the drawing]

[0014] [Figure 1] It is a cross-sectional view of the camera according to Embodiment 1. [Figure 2] It is a block diagram of the camera according to Embodiment 1. [Figure 3] It is a diagram for explaining the principle of the gaze detection method according to Embodiment 1. [Figure 4] It is a diagram showing an eye image according to Embodiment 1. [Figure 5] It is a diagram showing the principle of generation of luminance unevenness in the eye image according to Embodiment 1. [Figure 6] It is a diagram for explaining the luminance unevenness according to Embodiment 1. [Figure 7] It is a diagram showing an eye image with suppressed luminance unevenness according to Embodiment 1. [Figure 8] It is a diagram showing the arrangement of a plurality of light sources according to Embodiment 1. [Figure 9] It is a flowchart of the calibration operation according to Embodiment 1. [Figure 10] It is a flowchart of the shooting operation according to Embodiment 1. [Figure 11] It is a flowchart of the gaze detection operation according to Embodiment 1. [Figure 12] It is a flowchart of the gaze detection operation according to Embodiment 2. [Figure 13] It is a flowchart of the gaze detection operation according to Embodiment 3. [Figure 14] It is an external view of the display device according to Embodiment 4.

Mode for Carrying Out the Invention

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0016] (Embodiment 1) <Configuration> Figure 1 is a cross-sectional view of camera 1 (imaging device; electronic equipment), showing the general internal configuration of camera 1. The imaging lens unit 1A includes two lenses 101 and 102, an aperture 111, an aperture drive unit 112, a lens drive motor 113, a lens drive member 114, a photocoupler 115, a pulse plate 116, a mount contact 117, a focus adjustment circuit 118, etc. The lens drive member 114 consists of a drive gear, etc., and the photocoupler 115 detects the rotation of the pulse plate 116 which is linked to the lens drive member 114 and transmits it to the focus adjustment circuit 118. The focus adjustment circuit 118 drives the lens drive motor 113 based on information from the photocoupler 115 and information from the camera housing 1B (information on the amount of lens drive), moving the lens 101 to change the focus position. The mount contact 117 is the interface between the imaging lens unit 1A and the camera housing 1B. For simplicity, only two lenses, 101 and 102, are shown; however, the actual image lens unit 1A contains more than two lenses.

[0017] The camera housing 1B contains an image sensor 2, a CPU 3, a memory unit 4, a display device 10, a display device drive circuit 11, etc. The image sensor 2 is positioned at the intended imaging plane of the imaging lens unit 1A. The CPU 3 is the central processing unit of the microcomputer and controls the entire camera 1. The memory unit 4 stores images captured by the image sensor 2, etc. The display device 10 is made up of liquid crystal or the like, and displays captured images (subject images) etc. on the screen (display surface) of the display device 10. The display device drive circuit 11 drives the display device 10. The user can view the screen of the display device 10 through the eyepiece lens 12.

[0018] The camera housing 1B also includes light sources 13a and 13b, a light-receiving lens 15, an eye image sensor 16, etc. Light sources 13a and 13b are light sources that have been conventionally used in SLR cameras and the like to detect the direction of line of sight from the relationship between the reflected image (corneal reflection image) due to corneal reflection of light and the pupil, and are light sources for illuminating the user's eyeball 14. Specifically, light sources 13a and 13b are infrared light-emitting diodes, etc. that emit infrared light that is insensitive to the user, and are arranged around the eyepiece lens 12. The optical image of the illuminated eyeball 14 (eyeball image; an image formed by reflected light emitted from light sources 13a and 13b and reflected by the eyeball 14) passes through the eyepiece lens 12 and is imaged onto the eye image sensor 16, which has a two-dimensional arrangement of photoelectric elements such as a CCD, by the light-receiving lens 15. The light-receiving lens 15 positions the pupil of the eyeball 14 and the ophthalmic image sensor 16 in a conjugate imaging relationship. According to a predetermined algorithm described later, the line of sight direction of the eyeball 14 (line of sight position; viewpoint on the display device 10 screen) is detected from the position of the corneal reflection image in the eyeball image formed on the ophthalmic image sensor 16. Operating members 41-43 for receiving various user operations are also located on the back of the camera housing 1B.

[0019] Figure 2 is a block diagram showing the electrical configuration inside camera 1. The CPU 3 is connected to the gaze detection circuit 201, the photometering circuit 202, the autofocus detection circuit 203, the signal input circuit 204, the display device drive circuit 11, the light source drive circuit 205, and the like. The CPU 3 also transmits signals via the mount contact 117 to the focus adjustment circuit 118 located in the imaging lens unit 1A and the aperture control circuit 206 included in the aperture drive unit 112 in the imaging lens unit 1A. The memory unit 4 attached to the CPU 3 has the function of storing imaging signals from the image sensor 2 and the eye image sensor 16.

[0020] The gaze detection circuit 201 performs A / D conversion on the output of the eye image sensor 16 (CCD-EYE) when an image of the eyeball is formed on the eye image sensor 16 (eye image captured from the eye), and transmits the result to the CPU 3. The CPU 3 extracts feature points necessary for gaze detection from the eye image according to a predetermined algorithm described later, and calculates the user's gaze (viewpoint on the screen of the display device 10) from the positions of the feature points.

[0021] The photometering circuit 202 amplifies, logarithmically compresses, and performs A / D conversion on the signal obtained from the image sensor 2, which also acts as a photometering sensor, specifically the luminance signal corresponding to the brightness of the field of view, and sends the result to the CPU 3 as field of view luminance information.

[0022] The autofocus detection circuit 203 performs A / D conversion on the signal voltages from multiple detection elements (multiple pixels) used for phase difference detection, which are contained within the CCD of the image sensor 2, and sends them to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to each focus detection point from the signals of the multiple detection elements. This is a known technique known as image plane phase difference AF. In Example 1, as an example, it is assumed that there are focus detection points at each of the 180 locations on the image sensor corresponding to the 180 locations shown in the viewfinder image (screen of display device 10) in Figure 4.

[0023] The signal input circuit 204 is connected to switch SW1, which turns ON with the first stroke of the release button and starts the camera 1's metering, distance measurement, and gaze detection operations, and switch SW2, which turns ON with the second stroke of the release button and starts the shooting operation. The ON signals from switches SW1 and SW2 are input to the signal input circuit 204 and transmitted to the CPU 3.

[0024] The light source driving circuit 205 drives the light sources 13a and 13b.

[0025] The image processing circuit 207 applies predetermined image processing to image data to generate signals and image data, and to acquire and / or generate various types of information. The image processing circuit 207 may be a dedicated hardware circuit, such as an ASIC designed to perform a specific function, or it may be configured so that a processor, such as a DSP, performs a specific function by executing software.

[0026] Here, the image processing applied by the image processing circuit 207 includes preprocessing, color interpolation, correction, detection, and data processing. Preprocessing includes signal amplification, reference level adjustment, and defective pixel correction. Color interpolation is the process of interpolating the values ​​of color components not included in the image data, and is also called demosaicing. Correction processing includes white balance adjustment, image brightness correction, optical aberration correction of the shooting lens unit 1A, and color correction. Detection processing includes detection and tracking of feature regions (e.g., face regions, human body regions, object regions), and person recognition. Data processing includes scaling, encoding and decoding, and header information generation. Note that these are examples of image processing that the image processing circuit 207 can perform and do not limit the image processing that the image processing circuit 207 performs.

[0027] <Principle of gaze detection operation> Next, the principle of gaze detection operation will be explained with reference to Figures 3, 4(a) and 4(b). Figure 3 is a diagram illustrating the principle of the gaze detection method and is a schematic diagram of the optical system for gaze detection. As shown in Figure 3, the light sources 13a and 13b are arranged approximately symmetrically with respect to the optical axis of the light-receiving lens 15 and illuminate the user's eyeball 14. A portion of the light emitted from the light sources 13a and 13b and reflected by the eyeball 14 is focused by the light-receiving lens 15 onto the eye image sensor 16.

[0028] Figure 4(a) is a schematic diagram of the eye image (eyeball image projected onto the eye image sensor 16) captured by the eye image sensor 16. The corneal reflection images Pd and Pe of the light sources 13a and 13b, and the pupil center c are detected from the eye image, and the rotation angle θx of the eyeball 14 is calculated. Next, the user's (observer's) line of sight position on the display device 10 is calculated using the rotation angle θx and calibration data.

[0029] First, the detection method for corneal reflection images Pd and Pe will be explained. In Figure 4(a), the horizontal direction of the eye image is the X-axis and the vertical direction is the Y-axis. At this time, the corneal reflection images Pd and Pe from light sources 13a and 13b are focused by the light-receiving lens 15 and imaged onto the eye image sensor 16 to become the corneal reflection images Pd' and Pe' in the eye image. The coordinates of the corneal reflection image Pd' are (Xd, Yd) and the coordinates of the corneal reflection image Pe' are (Xe, Ye). Similarly, the light beams from the ends a and b of the pupil 141 are imaged onto the eye image sensor 16 to become pupil end images a' and b' in the eye image. The coordinates of the pupil end image a' are (Xa, Ya) and the coordinates of the pupil end image b' are (Xb, Yb). In addition, the light beam from the pupil center c is imaged onto the eye image sensor 16 to become the pupil center image c'.

[0030] Figure 4(b) shows the luminance information (luminance distribution) for any single row (hereinafter referred to as the search row) in the eye image of Figure 4(a). As shown in Figure 4(b), extremely strong levels of luminance, above the luminance threshold TH1, are obtained in the corneal reflection images Pd' and Pe'. By sequentially determining the luminance level by changing the search row, coordinates where the luminance is above the luminance threshold TH1 are detected for the entire eye image. Consequently, two regions where the luminance is above the luminance threshold TH1 are detected. By calculating the centroid of each of these two regions, the coordinates (Xd, Yd) of the corneal reflection image Pd' and the coordinates (Xe, Ye) of the corneal reflection image Pe' can be obtained.

[0031] Next, the method for detecting the center (contour center) c of the pupil 141 will be explained. In Figure 4(b), the coordinates of the pupil contour (pupil end) are (Xa,Ya) and (Xb,Yb). In the area corresponding to the pupil 141, except for the positions of coordinates (Xd,Yd) and (Xe,Ye), a low level of brightness below the brightness threshold TH2 is obtained. In contrast, in the area corresponding to the iris 143 outside the pupil 141, a relatively bright level of brightness is obtained between the brightness thresholds TH1 and TH2. In other words, coordinates (Xa,Ya) and (Xb,Yb) that fall between the bright level between the brightness thresholds TH1 and TH2 and below the brightness threshold TH2 are determined to be the pupil contour. By changing the search row and sequentially determining the brightness level, the contour coordinates of the pupil 141 can be detected for the entire eye image. Note that multiple contour coordinates of the pupil 141 are also referred to as the pupil 141 contour coordinate group. Furthermore, the detection of the contour coordinates of pupil 141 is also referred to as contour detection.

[0032] The coordinates of the center c of pupil 141 and the radius r of pupil 141 are calculated using a circular approximation method with the contour coordinates of pupil 141. Here, assuming there are 20 contour coordinates, and representing the contour coordinates as (Xi,Yi) (where i=1 to 20), the coordinates of the center c of pupil 141 (X0,Y0) and the radius r can be calculated using the following equation 1.

number

[0033] Next, we will explain how to calculate the imaging magnification β of the eyeball image. The imaging magnification β is determined by the position of the eyeball 14 relative to the light-receiving lens 15, and can be calculated using a function of the interval (Xd-Xe) between the corneal reflection images Pd' and Pe'.

[0034] The coordinates of the midpoint of the corneal reflection image Pd' and the corneal reflection image Pe' almost coincide with the coordinates of the center of curvature O of the cornea 142. Therefore, if we take Oc as the standard distance from the center of curvature O of the cornea 142 to the center c of the pupil 141, the rotation angle θx of the eyeball 14 in the ZX plane (plane perpendicular to the Y axis) of the optical axis of the eyeball 14 can be calculated using the following equation 2. The rotation angle θy of the eyeball 14 in the ZY plane (plane perpendicular to the X axis) can also be calculated using the same method as for calculating the rotation angle θx. β×Oc×SINθx≒{(Xd+Xe) / 2}-X0 (Formula 2)

[0035] Using the calculated rotation angles θx and θy, the user's viewpoint (the position where the gaze is directed; the position the user is looking at) on the screen of the display device 10 is determined (estimated). If the gaze position (coordinates of the viewpoint) (Hx, Hy) corresponds to the coordinates of the center c of the pupil 141, then the gaze position (Hx, Hy) can be calculated using the following equations 3 and 4. Hx=m×(θx-θx_cal) (Formula 3) Hy=m×(θy-θy_cal) (Formula 4)

[0036] The parameter m in equations 3 and 4 is a constant determined by the configuration of the camera 1's viewfinder optical system (photoreceiving lens 15, etc.), and is a conversion coefficient that transforms the rotation angles θx and θy into coordinates corresponding to the center c of the pupil 141 on the display device 10's screen. The parameter m is assumed to be predetermined and stored in the memory unit 4. The parameters (correction values) θx_cal and θy_cal are gaze correction parameters that correct for individual differences in gaze, and are obtained by performing the calibration work described later. The parameters θx_cal and θy_cal are assumed to be stored in the memory unit 4 before the gaze detection operation starts.

[0037] Calibration is the process of obtaining an offset amount to suppress the positional discrepancy between the position where the user (photographer) is actually fixating and the calculated gaze position, due to factors such as individual differences in eye shape. During calibration, a calibration image is displayed on the display device 10. A target frame for the user to fixate on is displayed in the center of the calibration image. The rotation angles θx_cal and θy_cal during calibration are obtained and stored in the memory unit 4 as the offset amount (correction value). In addition, during calibration, it is possible to improve the accuracy of gaze position detection by displaying the target frame at multiple positions and interpolating between these positions using the rotation angles of each position. The above is an explanation of the principle of gaze detection operation.

[0038] <Principles and methods for suppressing brightness unevenness in eye images> Next, with reference to Figures 5(a), 5(b), 6(a), 6(b), 7(a), 7(b), and 8, the principles of luminance unevenness in eye images and methods for suppressing it will be explained.

[0039] Figures 5(a) and 5(b) illustrate the principle of brightness unevenness. In Figures 5(a) and 5(b), as in Figure 3, light sources 13a and 13b illuminate the eyeball 14. The user is looking at an image displayed on the display device 10. Some of the light reflected by the user's eyeball 14 is focused onto the eye image sensor 16. The eye point (distance from the eyepiece lens 12 to the eyeball 14) 500 in Figure 5(b) is longer than the eye point 500 in Figure 5(a). The light rays 510a and 510b show the optical path from the area mainly illuminated by light sources 13a and 13b to the eye image sensor 16.

[0040] Figure 6(a) is a diagram illustrating the brightness unevenness of the eye image captured by the eye image sensor 16 at the eye point 500 shown in Figure 5(a). Figure 6(b) is a diagram illustrating the brightness unevenness of the eye image captured by the eye image sensor 16 at the eye point 500 shown in Figure 5(b). In the eye image in Figure 6(a), brightness unevenness occurs, with the lower part of the screen being bright and the upper part being dark. Brightness unevenness as shown in Figure 6(a) is due to the fact that the amount of light in the area mainly illuminated by light sources 13a and 13b is inversely proportional to the square of the distance between the light rays 510a and 510b, respectively. The reason why the brightness unevenness of the eye image in Figure 6(b) is smaller than that of the eye image in Figure 6(a) is that the difference between light rays 510a and 510b in Figure 5(b) is smaller than the difference between light rays 510a and 510b in Figure 5(a).

[0041] When performing the gaze detection operation shown in Figure 4(b) using an image with brightness unevenness, it becomes difficult to detect the coordinates (Xa,Ya) and (Xb,Yb) of the pupil contour. When the number of detected pupil contour coordinates decreases, the calculation results of the coordinates (X0,Y0) of the pupil center c and radius r using the circle approximation method also deteriorate. When the calculation results deteriorate, gaze detection errors (failure to detect gaze; increased error in viewpoint detection) may occur, potentially leading to a state where proper focus detection and adjustment are not possible. Although there is a method to change the brightness threshold TH2 in the image according to the eye point, this increases the amount of computation.

[0042] Therefore, in the first method for suppressing brightness unevenness, the CPU 3 of camera 1 performs shading correction on the eye image where brightness unevenness occurs, using a correction value (correction amount) corresponding to the eye point. Figures 7(a) and 7(b) are eye images to which shading correction according to each eye point has been applied to the eye images in Figures 6(a) and 6(b). The brightness unevenness in the eye images in Figures 7(a) and 7(b) is suppressed compared to the eye images in Figures 6(a) and 6(b), and it can be confirmed that the brightness unevenness in Figure 7(a) and Figure 7(b) is almost identical.

[0043] In the second method for suppressing brightness unevenness, the CPU 3 changes the light emission amount (illumination light intensity) of light sources 13a and 13b according to the eye point. For example, as shown in Figure 5(a), when the eye point 500 is shorter than a predetermined distance, the CPU 3 makes the light emission amount of light source 13a relatively larger than that of light source 13b. As shown in Figure 5(b), when the eye point is longer than a predetermined distance, the CPU 3 makes the difference in light emission amount between light sources 13a and 13b smaller than the difference in light emission amount shown in Figure 5(a). The second method also achieves the same effect as the first method. If light sources 13a and 13b are light-emitting diodes, the CPU 3 adjusts the difference in light emission amount between the light sources by changing, for example, the amplitude and duty cycle of the drive signal that drives the light sources.

[0044] Figures 8(a) to 8(c) show examples of arrangements of multiple light sources. When multiple light sources exist around the eyepiece as shown in Figures 8(a) to 8(c), the CPU3 may divide the multiple light sources into multiple light source groups (four light source groups 13c to 13f in Figures 8(a) to 8(c)) and adjust the amount of light emitted for each light source group. Each light source group includes one light source or two or more light sources that are close together. The CPU3 may also adjust the difference in the amount of light emitted between light source groups by changing the number of light sources that emit light (are lit) within each light source group.

[0045] Furthermore, CPU3 may suppress brightness unevenness by combining the first and second methods. The above explains the principle of generation and suppression methods for brightness unevenness in eye images.

[0046] The calibration operation, shooting operation, and gaze detection operation in Embodiment 1 will be described below with reference to Figures 9 to 11.

[0047] <Calibration operation> Figure 9 is a flowchart illustrating the calibration process for gaze detection.

[0048] In step S901, the CPU 3 displays a calibration image on the display device 10. The calibration image can be, for example, an image that shows the position on the display surface of the display device 10 where the user should look.

[0049] In step S902, the CPU 3 emits infrared light from light sources 13a and 13b toward the user's eyeball 14. The image of the eyeball illuminated by the infrared light is formed on the ophthalmic image sensor 16 through the light-receiving lens 15 and converted into an electrical signal by the ophthalmic image sensor 16. This provides an electrical signal of the eye image that can be processed.

[0050] In step S903, the CPU 3 receives an eye image (eyeball image signal; electrical signal of the eye image) from the eye image sensor 16.

[0051] In step S904, the CPU3 obtains the coordinates corresponding to the corneal reflection images Pd and Pe of the light sources 13a and 13b, and the coordinates corresponding to the contour of the pupil 141, from the eye image received in step S903.

[0052] In step S905, the CPU 3 acquires distance information indicating the distance (eye point) from the eyepiece 12 to the user's eyeball 14. For example, the CPU 3 calculates the eye point from the interval between the corneal reflection images Pd and Pe of the eye image received in step S903. The eye point can be calculated using known techniques.

[0053] In step S906, CPU3 uses the contour coordinates of pupil 141 obtained in step S904 to calculate the coordinates of the center c and radius r of pupil 141 using the circle approximation method.

[0054] In step S907, the CPU3 calculates the imaging magnification β of the eyeball image. In step S908, the CPU3 calculates the rotation angles θx and θy of the optical axis of the eyeball 14 with respect to the optical axis of the light-receiving lens 15.

[0055] In step S909, the CPU 3 stores the rotation angles θx and θy calculated in step S908 as correction values ​​θx_cal and θy_cal in the memory unit 4. The CPU 3 also stores the eye point calculated in step S905 in the memory unit 4. In step S909, the CPU 3 may also store shading correction values ​​(shading correction value, peripheral light falloff correction value) or the light emission amounts of multiple light sources (e.g., light sources 13a and 13b) corresponding to the eye point in the memory unit 4. After completing the processing in step S909, the CPU 3 terminates the gaze detection calibration operation.

[0056] <Shooting operation> After the gaze detection calibration operation described in Figure 9, the shooting operation is performed. Figure 10 is a flowchart illustrating the shooting operation of camera 1, which has a gaze detection function. CPU 3 starts the flow in Figure 10 when the power to camera 1 is turned ON.

[0057] In step S1001, the CPU 3 drives the image sensor 2 to acquire (capture) an image. The CPU 3 then displays the acquired image on the display device 10 (live view display; LV display).

[0058] In step S1002, CPU3 determines whether or not to terminate the shooting operation. For example, CPU3 determines to terminate the shooting operation if it receives an instruction to turn off the power of camera 1. If CPU3 determines to terminate the shooting operation, it terminates this flow; otherwise, it proceeds to step S1003.

[0059] In step S1003, CPU3 performs gaze detection. Details of the gaze detection operation will be described later using the flowchart in Figure 11.

[0060] In step S1004, the CPU3 corrects the eye rotation angles θx and θy detected in step S1003. The CPU3 calculates the gaze position (Hx, Hy) from the rotation angles θx and θy detected in step S1003 and the correction values ​​θx_cal and θy_cal stored in step S909.

[0061] In step S1005, the CPU 3 displays an AF (autofocus) frame on the display device 10 at the eye-line position calculated in step S1004.

[0062] In step S1006, CPU3 determines whether switch SW1 is ON or OFF (whether the release button is half-pressed or OFF). If switch SW1 is ON, CPU3 proceeds to step S1007; otherwise, it returns to step S1001 and repeats the image display and gaze detection operation.

[0063] In step S1007, CPU3 performs AF operation. The image sensor 2 is composed of multiple pixels used for phase-detection. From the signals of the multiple pixels, CPU3 calculates the focus state of the subject corresponding to the AF frame determined in step S1006 using image-plane phase-detection AF, a known technique, and controls the lens position.

[0064] In step S1008, CPU3 determines whether the release button has been pressed further and switch SW2 has been turned ON (whether the release button has been fully pressed). If switch SW2 is turned ON, CPU3 proceeds to step S1009; otherwise, it returns to S1006 and repeats the AF operation at the same position.

[0065] In step S1009, the CPU 3 drives the image sensor 2 to acquire an image and stores the acquired image in a storage medium (not shown). After that, the CPU 3 returns to step S1001 and repeats the process.

[0066] Figure 11 is a flowchart showing an example of the gaze detection operation (gaze detection operation during shooting) performed in step S1003 of Figure 10.

[0067] In step S1101, the CPU3 determines whether it is the first time imaging (first time gaze detection operation), whether a predetermined environmental change occurred during the previous imaging, or something else. If it is the first time imaging or a predetermined environmental change occurred during the previous imaging, the CPU3 proceeds to step S1102; otherwise, it proceeds to step S1108. For example, the CPU3 may determine that a predetermined environmental change has occurred if the brightness of the eye image captured during the previous imaging was not at a brightness suitable for detecting the corneal reflection image Pd, Pe and contour coordinates (brightness within a predetermined brightness range). For example, if the eye point suddenly becomes shorter than during imaging before the previous imaging, or if ambient light leaks in, a brightness change of more than a predetermined amount occurs, making it difficult to detect the corneal reflection image Pd, Pe and contour coordinates. Therefore, in order to determine appropriate exposure amount, illumination light amount, and shading correction value, the CPU3 repeats the processing in steps S1102 to S1107.

[0068] In steps S1102 to S1107, the CPU 3 detects (calculates) the eye point and determines the exposure amount, light emission amount, and shading correction value for the next image capture. The processing in steps S1102 to S1105 is the same as the processing in steps S902 to S905 in Figure 9.

[0069] In step S1102, the CPU 3 may set the light emission amount of light source 13a to be the same as that of light source 13b. Alternatively, the CPU 3 may set the light emission amounts of light source 13a and light source 13b with a difference in light emission amount suitable for the exposure amount determination process and the eye point detection process. For example, the CPU 3 may set the light emission amounts of light source 13a and light source 13b so that brightness unevenness in the eye image is suppressed at eye points that are used more frequently than a predetermined threshold. Alternatively, the CPU 3 may set the light emission amounts of light source 13a and light source 13b to the light emission amounts corresponding to the eye points stored in the memory unit 4 during the calibration operation (step S909).

[0070] In step S1106, the CPU 3 determines the exposure amount for the next imaging (the next time the eye image sensor 16 is charged). For example, in step S1106, the CPU 3 determines the accumulation time or gain value of the eye image sensor 16 so that an eye image is obtained in which the outline of the pupil and the corneal reflection image are captured with the desired brightness.

[0071] In step S1107, the CPU3 determines the light emission amount and shading correction value for the next image capture, according to the eye point calculated in step S1105. Here, the correspondence between the eye point and the light emission amounts of multiple light sources, and the correspondence between the eye point and the shading correction value are predetermined. By using these values, even when multiple light sources are provided, eye images suitable for gaze detection can be quickly acquired according to the eye point.

[0072] The amount of light emitted from multiple light sources at each eye point will be explained with reference to Figures 5(a), 5(b), and 8(a). The CPU 3 controls the multiple light sources so that they emit light at each eye point with an amount of light based on the positional relationship between the multiple light sources and the eye image sensor 16. For example, suppose the multiple light sources are classified into multiple light source groups, including light source group 13d (first light source; light source group) and light source group 13c (second light source; light source group), whose light emission amounts can be controlled individually. The distance of light source group 13c from the eye image sensor 16 is longer than the distance from the eye image sensor 16 to light source group 13d. In such a case, the CPU 3 may make the amount of light emitted from light source group 13c greater than the amount of light emitted from light source group 13d. For example, the CPU 3 may light up more light sources in light source group 13c than the number of light sources lit in light source group 13d. Note that the light sources may be arranged so that the number of light sources included in light source group 13c is greater than the number of light sources included in light source group 13d. This prevents the area illuminated by light source group 13c from becoming darker than the area illuminated by light source group 13d. Furthermore, the CPU 3 may increase the difference in light emission between light sources when the eye point is at a first distance (Figure 5(a)) compared to when it is at a second distance longer than the first distance (Figure 5(b)). This allows for more effective suppression of brightness unevenness, even when the eye point is shorter than a predetermined distance and brightness unevenness is likely to occur. It is preferable to have a predetermined set of multiple combinations of eye points and the light emission amounts of multiple light sources.

[0073] Furthermore, if the camera 1 is equipped with a third group of light sources (light sources) whose distance from the eye image sensor 16 is greater than or equal to the distance from the eye image sensor 16 to the second group of light sources (light sources), the CPU 3 may set the light emission amount of the third group of light sources to be greater than that of the second group of light sources. For example, in Figures 8(a) to 8(c), the relative distances from the eye image sensor 16 to each group of light sources are: distance to light source group 13c > distance to light source group 13e (13f) > distance to light source group 13d. In the example in Figure 8(a), the CPU 3 controls the light emission amounts of each group of light sources to be: light emission amount of light source group 13c > light emission amount of light source group 13e (13f) > light emission amount of light source group 13d. Furthermore, CPU3 may control the relative magnitudes of the light emission amounts of the light sources included in each light source group so that the light emission amount of the light source included in light source group 13c > the light emission amount of the light source included in light source group 13e (13f) > the light emission amount of the light source included in light source group 13d. Note that the relative magnitudes of the light emission amounts of the light sources do not have to be such that the light emission amount of the light source included in light source group 13c > the light emission amount of the light source included in light source group 13e (13f) > the light emission amount of the light source included in light source group 13d. Similarly, the relative magnitudes of the light emission amounts of the light sources do not have to be such that the light emission amount of light source group 13c > the light emission amount of light source group 13e (13f) > the light emission amount of light source group 13d.

[0074] For example, CPU3 does not need to make the emission amount of the third light source group greater than that of the second light source group. CPU3 may also control the emission amount of each light source by considering the number of light sources included in each light source group. In the example in Figure 8(b), the number of light sources included in light source group 13e (13f) is less than the number of light sources included in light source group 13c. In a case like Figure 8(b), CPU3 may control the emission amounts of each light source group so that emission amount of light source group 13e (13f) > emission amount of light source group 13c > emission amount of light source group 13d. Alternatively, CPU3 may control the emission amounts of the light sources included in each light source group so that emission amount of light sources included in light source group 13e (13f) > emission amount of light sources included in light source group 13c > emission amount of light sources included in light source group 13d. This makes it possible to suppress the area illuminated by light source group 13e from becoming darker than the areas illuminated by other light source groups.

[0075] Note that, depending on the number of light sources included in the light source group, the light emission amount of the light source group closest to the eye image sensor 16 may not be set to the lowest value. In the example in Figure 8(c), the number of light sources in light source group 13d, which is the light source group closest to the eye image sensor 16, is 13c, 13e. This is less than the number of light sources. In a case like Figure 8(c), the CPU 3 may control the relative magnitudes of the light emission amounts of each light source group so that the light emission amount of light source group 13d > the light emission amount of light source group 13e (13f) > the light emission amount of light source group 13c. Alternatively, the CPU 3 may control the relative magnitudes of the light emission amounts of the light sources included in each light source group so that the light emission amount of the light source included in light source group 13d > the light emission amount of the light source included in light source group 13e (13f) > the light emission amount of the light source included in light source group 13c. This makes it possible to suppress the area illuminated by light source group 13d from becoming darker than the area illuminated by other light source groups. In this way, the CPU 3 can suppress brightness unevenness by controlling the light emission amount of each light source, taking into account the distance from the eye image sensor 16 to each light source or the number of light sources included in each light source group. Note that although the examples in Figures 8(a) to 8(c) describe light source groups 13c to 13f, at least one of them may be a light source instead of a light source group.

[0076] The shading correction values ​​for each eye point will be explained with reference to Figures 5(a), 5(b), and 8. The CPU 3 should set a larger correction amount when the eye point is at a first distance (Figure 5(a)) compared to when it is at a second distance (Figure 5(b)), which is longer than the first distance. For example, when the eye point is at the first distance, the correction amount for the image area (part of the eye image) corresponding to the area illuminated by the light source group 13c (part of the eyeball) should be larger than the correction amount for the image area corresponding to the area illuminated by the light source group 13d, compared to when it is at the second distance. It is desirable that multiple combinations of such eye points and shading correction values ​​be predetermined.

[0077] For example, for each eye point, the light emission amount and shading correction value of light sources 13a and 13b are stored in the memory unit 4. The CPU 3 reads the light emission amount and shading correction value corresponding to the eye point calculated in step S1105 from the memory unit 4 and uses them in the next calculation (imaging). If the eye point calculated in step S1105 is the distance between eye points stored in the memory unit 4, the CPU 3 may perform interpolation. In other words, if the combination of the eye point and the light emission amount or shading correction value is not predetermined for the eye point calculated in step S1105, the CPU 3 performs interpolation. For example, the CPU 3 obtains the light emission amount corresponding to the calculated eye point by performing interpolation using multiple combinations of eye points and light emission amounts that are close to the calculated eye point among the eye points stored in the memory unit 4. The interpolation process for shading correction values ​​is similar. Alternatively, the CPU 3 stores the light emission amount and shading correction value corresponding to the eye point in the memory unit 4 during the calibration operation (step S909). In step S1107, the CPU 3 may read the amount of light emitted and the shading correction value corresponding to the eye point during calibration from the memory unit 4. Alternatively, the CPU 3 may determine the amount of light emitted or the shading correction value by referring to information stored on an external storage medium instead of the memory unit 4.

[0078] In this way, when a predetermined environmental change occurs during the initial or previous imaging, the system proceeds to the next imaging without correcting the brightness unevenness in the eye image, allowing for quick determination of exposure conditions and detection of the eye point, pupil, etc.

[0079] In step S1108, the CPU 3 emits infrared light from light sources 13a and 13b toward the user's eyeball 14 at the amount of light emitted determined in step S1107 or step S1117.

[0080] In step S1110, CPU3 performs shading correction on the eye image obtained in step S1109 using the correction value determined in step S1107 or step S1117.

[0081] The processes in steps S1109 and S1111-S1115 are the same as the processes in steps S903-S908 in Figure 9. Also, the process in step S1116 is the same as the process in step S1106.

[0082] In step S1117, CPU3 determines the amount of light emitted and the shading correction value for the next image capture, in the same way as in step S1107, based on the eye point calculated in step S1112. After completing the process in step S1117, CPU3 terminates the gaze detection operation during shooting.

[0083] Furthermore, if multiple light sources are composed of light-emitting diodes (LEDs) and the amount of light emitted by each LED varies from one unit to another, the CPU 3 may control (adjust) the amount of light emitted from each of the multiple light sources by considering the luminous efficiency of each light source. By considering the luminous efficiency, the CPU 3 can achieve high-precision emission at the amount of light stored in the memory unit 4. Therefore, brightness unevenness can be suppressed with high precision. Alternatively, brightness unevenness may be measured at each eye point for each individual eye-tracking device, and the amount of light emitted to suppress brightness unevenness and the shading correction value may be stored in the memory unit 4.

[0084] In the example shown in Figure 11, if a predetermined environmental change occurs during the initial imaging or the previous imaging, the CPU 3 will not perform the calculation of the pupil's center coordinates, etc. (processing in steps S1113 to S1115), but it may perform the calculation of the pupil's center coordinates, etc. For example, the CPU 3 may determine the amount of light emitted from light sources 13a and 13b so that brightness unevenness in the eye image is suppressed at eye points where the frequency of use is higher than a predetermined threshold, and then perform the calculation of the pupil's center coordinates, etc. from the eye image captured with that amount of light emitted.

[0085] As described above, in Embodiment 1, the CPU 3 changes the amount of light emitted from the light source according to the eye point and performs shading correction. This makes it possible to suppress brightness unevenness in the captured eye image.

[0086] (Embodiment 2) In Embodiment 1, the CPU 3 performs both a change in the amount of light emitted from the light source and shading correction according to the eye point. Alternatively, the CPU 3 may perform only one of either controlling the amount of light emitted from the light source or performing shading correction according to the eye point. Embodiment 2 describes an example in which the CPU 3 performs shading correction according to the eye point. Embodiment 2 differs from Embodiment 1 in that the amount of light emitted from the light source is not changed according to the eye point.

[0087] Figure 12 is a flowchart showing an example of the gaze detection operation performed in step S1003 of Figure 10. The processes in steps S1201 to S1206 and S1209 to S1216 are the same as the processes in steps S1101 to S1106 and S1109 to S1116 of Figure 11.

[0088] In step S1207, CPU3 determines the shading correction value for the next image acquisition based on the eye point calculated in step S1205.

[0089] In step S1208, the CPU 3 emits infrared light from light sources 13a and 13b toward the user's eyeball 14. The processing in step S1208 may be the same as, for example, the processing in step S1102 (S1202). For example, the CPU 3 may emit infrared light from light sources 13a and 13b toward the user's eyeball 14 at an emission amount that suppresses brightness unevenness in the eye image at eye points where the frequency of use is higher than a predetermined threshold. Thus, in Embodiment 2, the CPU 3 does not change the emission amount of the light sources according to the eye point, but uses a fixed value.

[0090] In step S1217, the CPU3 determines the shading correction value for the next image acquisition based on the eye point calculated in step S1212.

[0091] As described above, in Embodiment 2, the CPU 3 uses a fixed value for the amount of light emitted from the light source and determines the shading correction value according to the eye point. This makes it possible to suppress brightness unevenness in the eye image with simpler processing than in Embodiment 1.

[0092] (Embodiment 3) Embodiment 3 describes an example in which the CPU 3 changes the amount of light emitted from the light source according to the eye point. Embodiment 3 differs from Embodiment 1 in that it does not perform shading correction.

[0093] Figure 13 is a flowchart showing an example of the gaze detection operation performed in step S1003 of Figure 10. The processing in steps S1301 to S1306 and S1308 to S1315 is the same as the processing in steps S1101 to S1106, S1108, S1109, and S1111 to S1116 of Figure 11.

[0094] In step S1307, CPU3 determines the amount of light emitted during the next image capture, according to the eye point calculated in step S1305.

[0095] In step S1316, CPU3 determines the amount of light emitted during the next image acquisition based on the eye point calculated in step S1311.

[0096] As described above, in Embodiment 3, the CPU 3 does not perform shading correction, but changes the amount of light emitted from the light source according to the eye point. This makes it possible to suppress brightness unevenness in the eye image with simpler processing than in Embodiment 1.

[0097] (Embodiment 4) Embodiments 1 to 3 described examples of applying the present invention to a camera 1, but the present invention can also be applied to a head-mounted display (HMD), for example. An example of applying the present invention to an HMD will be described below.

[0098] Figures 14(a) and 14(b) show the external appearance of the display device 5 according to Embodiment 4. Figure 14(a) is a front perspective view, and Figure 14(b) is a rear perspective view. The display device 5 is an optical see-through type display device, a type of HMD that can be attached to and removed from the head, and is a glasses-type device that utilizes MR (Mixed Reality) and AR (Augmented Reality). The display device 5 can individually detect the gaze of the right eye and the gaze of the left eye of the user wearing the display device 5 on their head.

[0099] The lens 50 is an optical component that faces the user's eye. The user can see the outside world through the lens 50. The display device 51 displays virtual objects (virtual images of virtual objects) to both of the user's eyes (both the right and left eyes) under control (display control) from the CPU 3 which controls the entire display device 5. The user can see the displayed virtual objects as if they exist in the outside world. The light source driving circuit 205 drives the light sources 13a and 13b. The light sources 13a and 13b are light sources that illuminate the user's eye, and are, for example, infrared light-emitting diodes that emit infrared light that is insensitive to the user. A portion of the light emitted from the light sources 13a and 13b and reflected by the user's eye is focused onto the eye image sensor 16 by the light-receiving lens 15. These components are provided for both the left and right eyes. For example, the eye image sensor 16 is provided as a right image sensor for imaging the right eye and a left image sensor for imaging the left eye. Note that Figure 14(b) shows an example where one light source 13a and one 13b are provided above and below the lens 50 for both the left and right eyes, but there are more than one light source. A light source may be provided, or light sources may be provided on the left and right sides of the lens 50. The dimming device 19 (dimming panel) adjusts the light from the outside. The dimming control circuit 18 is a circuit that changes the transmittance of the dimming device 19, and is, for example, a circuit that controls the voltage applied to the dimming device 19. The dimming device 19 transmits light from the outside with a transmittance controlled by the dimming control circuit 18.

[0100] Since the user rests the display device 5 on their nose, the eye point is approximately constant regardless of the user. Therefore, the CPU 3 controls the amount of light emitted or performs shading correction so that brightness unevenness in the eye image at a specific eye point (for example, the average eye point when the user rests the display device 5 on their nose). For example, the CPU 3 controls multiple light sources to emit light at an amount based on the positional relationship between the multiple light sources and the eye image sensor 16. The CPU 3 also performs shading correction with a correction value that suppresses brightness unevenness at a specific eye point.

[0101] As described above, in Embodiment 4, the CPU 3 can suppress brightness unevenness in the eye image of the user using the HMD.

[0102] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0103] The above embodiments are merely examples, and configurations obtained by appropriately modifying or changing the configuration of the above embodiments within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the configurations of the above embodiments are also included in the present invention.

[0104] This embodiment includes the following configurations and methods. (Composition 1) A first acquisition means that acquires an image of the eyeball, which is illuminated by multiple light sources and captured by an imaging means, A second acquisition means for acquiring distance information indicating the distance from the eyeball to the imaging means, Correction means for correcting brightness unevenness in the image with a correction amount corresponding to the distance information, An electronic device characterized by having the following features. (Configuration 2) The system further comprises the plurality of light sources and the imaging means. The electronic device according to configuration 1, characterized by the features described above. (Composition 3) The correction means increases the correction amount when the distance indicated by the distance information is a first distance compared to when the distance is a second distance that is longer than the first distance. The electronic device according to configuration 1 or 2, characterized by the above. (Composition 4) The correction means increases the amount of correction for the region illuminated by a light source whose distance from the imaging means is longer than a predetermined distance, compared to the case where the distance indicated by the distance information is a first distance and the distance is a second distance that is longer than the first distance. An electronic device according to any one of configurations 1 to 3, characterized by the features described herein. (Composition 5) The correspondence between the distance from the eyeball to the imaging means and the correction value is predetermined. The correction means compensates for the brightness unevenness using a correction value corresponding to the distance indicated by the distance information. Correct An electronic device according to any one of configurations 1 to 4, characterized by the features described herein. (Composition 6) As a correspondence, multiple combinations of the distance from the eyeball to the imaging means and the correction value are predetermined. The correction means, when the combination of the distance indicated by the distance information and the correction value is not predetermined, obtains the correction value corresponding to the distance indicated by the distance information by interpolation processing using the plurality of combinations. The electronic device according to configuration 5, characterized by the features described herein. (Composition 7) The correction means refrains from correcting the brightness unevenness during the initial imaging or when a brightness change of more than a predetermined amount occurs in the image. An electronic device according to any one of configurations 1 to 6, characterized by the features described herein. (Composition 8) The system further includes a detection means for detecting the line of sight position from the aforementioned image, The correction means is When the calibration operation of the detection means is in progress, either the distance information or a correction value corresponding to the distance information is stored in the storage medium. If the calibration operation is not in progress, the brightness unevenness is corrected using the correction value corresponding to the distance information during the calibration operation. An electronic device according to any one of configurations 1 to 7, characterized by the features described herein. (Composition 9) The correction means corrects the brightness unevenness of the image so that areas of the image that are darker than other areas become brighter. An electronic device according to any one of configurations 1 to 8, characterized by the above. (Composition 10) Multiple light sources that illuminate the eyeball, An imaging means for imaging the eyeball illuminated by the aforementioned plurality of light sources, Control means for controlling the plurality of light sources, It has, The control means controls the plurality of light sources to emit light in an amount determined by the positional relationship between the plurality of light sources and the imaging means. An electronic device characterized by the following features. (Composition 11) The plurality of light sources include a first light source and a second light source whose distance from the imaging means is longer than the distance from the imaging means to the first light source. The electronic device according to configuration 10, characterized in that the control means makes the amount of light emitted from the second light source greater than the amount of light emitted from the first light source. (Composition 12) The control means controls, for each of the plurality of light sources, at least one of the amplitude and duty cycle of the drive signal that drives the light source. The electronic device according to configuration 10 or 11, characterized by the above. (Composition 13) The plurality of light sources are classified into a plurality of light source groups, each capable of individually controlling the amount of light emitted, which includes a first group of light sources and a second group of light sources whose distance from the imaging means is longer than the distance from the imaging means to the first group of light sources. The control means makes the amount of light emitted from the second light source group greater than the amount of light emitted from the first light source group. An electronic device according to any one of the configurations 10 to 12, characterized by the features described herein. (Composition 14) The plurality of light source groups further include a third light source group whose distance from the imaging means is greater than or equal to the distance from the imaging means to the second light source group. The number of light sources included in the second light source group is less than the number of light sources included in the third light source group. The control means prevents the amount of light emitted from the third light source group from becoming greater than the amount of light emitted from the second light source group. The electronic device according to configuration 13, characterized by the features described above. (Composition 15) The control means controls, for each of the plurality of light source groups, at least one of the following: the amplitude of the drive signal that drives the light source group, the duty cycle of the drive signal, or the number of light sources that light up within the light source group. The electronic device according to configuration 13 or 14, characterized by the features described herein. (Composition 16) The control means causes the second light source group to light up a greater number of light sources than the first light source group. The electronic device according to configuration 13, characterized by the features described above. (Composition 17) The number of light sources included in the second light source group is greater than the number of light sources included in the first light source group. The electronic device according to configuration 13, characterized by the features described above. (Composition 18) The system further includes a second acquisition means for acquiring distance information indicating the distance from the eyeball to the imaging means, The relationship between the distance from the eyeball to the imaging means and the amount of light emitted from the multiple light sources is predetermined. The control means controls the plurality of light sources to emit light at a light emission amount corresponding to the distance indicated by the distance information. An electronic device according to any one of configurations 10 to 15, characterized by the features described herein. (Composition 19) As a correspondence, multiple combinations of the distance from the eyeball to the imaging means and the amount of light emitted from the multiple light sources are predetermined. When the combination of the distance indicated by the distance information and the emission amount of the multiple light sources is not predetermined, the control means obtains the emission amount of the multiple light sources corresponding to the distance indicated by the distance information by interpolation processing using the multiple combinations. The electronic device according to configuration 18, characterized by the features described above. (Composition 20) The system further includes a first acquisition means for acquiring an image of the eyeball captured by the imaging means, The control means controls the plurality of light sources to emit light at the amount corresponding to the distance from the eyeball to the imaging means, where the frequency of use is higher than a predetermined threshold, either during the initial imaging or when a change in brightness of a predetermined amount or more occurs in the image. The electronic device according to configuration 18 or 19, characterized by the above. (Composition 21) A first acquisition means for acquiring the image of the eyeball captured by the imaging means, A detection means for detecting the line of sight position from the aforementioned image, It further possesses, The control means is When the calibration operation of the detection means is in progress, either the distance information or the amount of light emitted from the plurality of light sources corresponding to the distance information is stored in the storage medium. If the calibration operation is not in progress, the plurality of light sources are controlled to emit light at an amount corresponding to the distance information during the calibration operation. ru The electronic device according to any one of the configurations 18 to 20, characterized by the features described herein. (Composition 22) The control means controls the plurality of light sources, taking into consideration the luminous efficiency of each of the plurality of light sources. An electronic device according to any one of the configurations 10 to 21, characterized by the features described herein. (Composition 23) Control means for controlling the acquisition of an image of the eyeball in which the brightness uniformity is substantially the same whether the distance from the eyeball to the imaging means for imaging the eyeball is a first distance or a second distance which is longer than the first distance. An electronic device characterized by having the following features. (Method 1) A first acquisition step involves acquiring an image of the eyeball, which is illuminated by multiple light sources and captured by an imaging means. A second acquisition step involves acquiring distance information indicating the distance from the eyeball to the imaging means, A correction step in which brightness unevenness in the image is corrected by a correction amount corresponding to the distance information, A method for controlling electronic equipment, characterized by having the following features. (Method 2) An imaging step in which an eyeball illuminated by multiple light sources is imaged by an imaging means, A control step for controlling the plurality of light sources, It has, In the control step, the plurality of light sources are controlled to emit light in an amount based on the positional relationship between the plurality of light sources and the imaging means. A method for controlling electronic equipment characterized by the following features. (program) A program for causing a computer to function as one of the electronic devices described in any one of configurations 1 to 23. (medium) A computer-readable storage medium containing a program for causing the computer to function as one of the electronic devices described in any one of items 1 to 23 of the configuration. [Explanation of Symbols]

[0105] 1: Camera 3: CPU 13a,13b: Light source 14: Eyeball 16: Image sensor for the eye

Claims

1. A first acquisition means that acquires an image of the eyeball, which is illuminated by multiple light sources and captured by an imaging means, A second acquisition means for acquiring distance information indicating the distance from the eyeball to the imaging means, A detection means that performs a detection process to detect the gaze position from the aforementioned image, When the calibration operation of the detection process is in progress, the correction means stores either the distance information or a correction value corresponding to the distance information in a storage medium, and when the calibration operation is not in progress, corrects the brightness unevenness of the image using the correction value corresponding to the distance information during the calibration operation. An electronic device characterized by having the following features.

2. The system further comprises the plurality of light sources and the imaging means. The electronic device according to feature 1.

3. The correction means increases the amount of luminance uniformity correction when the distance indicated by the distance information is a first distance, compared to when the distance is a second distance that is longer than the first distance. The electronic device according to feature 1.

4. The correction means increases the amount of correction for brightness unevenness in the region illuminated by a light source whose distance from the imaging means is longer than a predetermined distance, compared to the case where the distance indicated by the distance information is a first distance. The electronic device according to feature 1.

5. The correspondence between the distance from the eyeball to the imaging means and the correction value is predetermined. The correction means corrects the brightness unevenness using a correction value corresponding to the distance indicated by the distance information. The electronic device according to feature 1.

6. As a correspondence, multiple combinations of the distance from the eyeball to the imaging means and the correction value are predetermined. The correction means, when the combination of the distance indicated by the distance information and the correction value is not predetermined, obtains the correction value corresponding to the distance indicated by the distance information by interpolation processing using the plurality of combinations. The electronic device according to feature 5.

7. The correction means refrains from correcting the brightness unevenness during the initial imaging or when a brightness change of more than a predetermined amount occurs in the image. The electronic device according to feature 1.

8. The correction means corrects the brightness unevenness of the image so that areas of the image that are darker than other areas become brighter. The electronic device according to feature 1.

9. Multiple light sources that illuminate the eyeball, An imaging means for imaging the eyeball illuminated by the aforementioned plurality of light sources, Control means for controlling the plurality of light sources, It has, The plurality of light sources are classified into a plurality of light source groups, each capable of individually controlling the amount of light emitted, including a first light source group, a second light source group whose distance from the imaging means is longer than the distance from the imaging means to the first light source group, and a third light source group whose distance from the imaging means is greater than or equal to the distance from the imaging means to the second light source group. The number of light sources included in the second light source group is less than the number of light sources included in the third light source group. The control means is The amount of light emitted from the second light source group is made greater than the amount of light emitted from the first light source group, The amount of light emitted from the third light source group shall not be greater than the amount of light emitted from the second light source group. An electronic device characterized by the following features.

10. The plurality of light sources include a first light source and a second light source whose distance from the imaging means is longer than the distance from the imaging means to the first light source. The electronic device according to claim 9, characterized in that the control means makes the amount of light emitted from the second light source greater than the amount of light emitted from the first light source.

11. The control means controls, for each of the plurality of light sources, at least one of the amplitude and duty cycle of the drive signal that drives the light source. The electronic device according to feature 9.

12. The control means controls, for each of the plurality of light source groups, at least one of the following: the amplitude of the drive signal that drives the light source group, the duty cycle of the drive signal, or the number of light sources that light up within the light source group. The electronic device according to feature 9.

13. The control means causes the second light source group to light up a greater number of light sources than the first light source group. The electronic device according to feature 9.

14. The number of light sources included in the second light source group is greater than the number of light sources included in the first light source group. The electronic device according to feature 9.

15. The system further includes a second acquisition means for acquiring distance information indicating the distance from the eyeball to the imaging means, The relationship between the distance from the eyeball to the imaging means and the amount of light emitted from the multiple light sources is predetermined. The control means controls the plurality of light sources to emit light at a light emission amount corresponding to the distance indicated by the distance information. The electronic device according to feature 9.

16. As a correspondence, multiple combinations of the distance from the eyeball to the imaging means and the amount of light emitted from the multiple light sources are predetermined. When the combination of the distance indicated by the distance information and the emission amount of the multiple light sources is not predetermined, the control means obtains the emission amount of the multiple light sources corresponding to the distance indicated by the distance information by interpolation processing using the multiple combinations. The electronic device according to feature 15.

17. The system further includes a first acquisition means for acquiring an image of the eyeball captured by the imaging means, The control means controls the plurality of light sources to emit light at the amount corresponding to the distance from the eyeball to the imaging means, where the frequency of use is higher than a predetermined threshold, either during the initial imaging or when a change in brightness of a predetermined amount or more occurs in the image. The electronic device according to feature 15.

18. A first acquisition means for acquiring the image of the eyeball captured by the imaging means, A detection means for detecting the line of sight position from the aforementioned image, It further possesses, The control means is When the calibration operation of the detection means is in progress, either the distance information or the amount of light emitted from the plurality of light sources corresponding to the distance information is stored in the storage medium. If the calibration operation is not in progress, the plurality of light sources are controlled to emit light at an amount corresponding to the distance information during the calibration operation. The electronic device according to feature 15.

19. The control means controls the plurality of light sources, taking into consideration the luminous efficiency of each of the plurality of light sources. The electronic device according to feature 9.

20. The steps include: acquiring an image of the eyeball illuminated by multiple light sources and captured by an imaging means; A step of acquiring distance information indicating the distance from the eyeball to the imaging means, The steps include: performing a detection process to detect the gaze position from the aforementioned image; If the calibration operation of the detection process is in progress, the step of storing either the distance information or a correction value corresponding to the distance information in a storage medium, If the calibration operation is not in progress, the steps include correcting the brightness unevenness of the image using the correction value corresponding to the distance information during the calibration operation. A method for controlling electronic equipment, characterized by having the following features.

21. A program for causing a computer to execute the control method of an electronic device described in claim 20.

22. A computer-readable storage medium storing a program for causing a computer to execute the control method of the electronic device described in claim 20.

Images