Electronic device, its control method, and program

Abstract

When there are multiple combinations of corneal reflection images (P-image pairs) included in the eyeball image, the accuracy of gaze detection may decrease if the time required for gaze detection processing is a predetermined time. [Solution] The electronic device of the present invention includes a plurality of irradiation means for irradiating the user's eyeball with infrared light, an imaging means for capturing an eyeball image obtained when the infrared light irradiated by the plurality of irradiation means is reflected by the eyeball, a first calculation means for calculating the gaze position of the user looking at the display unit based on a combination of a plurality of corneal reflection images included in the eyeball image captured by the imaging means, and a setting means for setting the order of each of the plurality of corneal reflection image combinations based on predetermined conditions during a calibration process to improve the detection accuracy of the gaze detected by the detection means, wherein the first calculation means performs processing using the combination of corneal reflection images based on the order set by the setting means within a predetermined time and calculates the gaze position of the user looking at the display unit.

Description

Technical Field

[0001] The present invention relates to an electronic device, a control method thereof, and a program.

Background Art

[0002] In recent years, electronic devices that use a user's line-of-sight information as a user interface have been used in various fields. Examples of this type of electronic device include a digital still camera and a head-mounted display (HMD). In such an electronic device, the line of sight of a user looking at a display unit can be detected, and various processes can be executed based on the line-of-sight position corresponding to the detected line of sight.

[0003] And, as one method of detecting the line of sight, there is a corneal reflection method. In this corneal reflection method, infrared light from an illumination light source is irradiated onto the eyeball, and an eyeball image obtained from the infrared light reflected from the eyeball is photographed by an image sensor for the eyeball, whereby the line of sight of a user looking at the display unit can be detected.

[0004] Patent Document 1 discloses that when the result of the line of sight calculated based on the first selected pair of corneal reflection images (P images) is clearly not normal and there are further candidate P image pairs, an attempt is made to calculate the line of sight using another P image pair.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] Increasing the number of illuminating light sources improves the accuracy of gaze detection because it increases the probability of accurately determining the imaging magnification of the eyeball image. However, when there are many illuminating light sources, the captured eyeball image may contain multiple P-image pairs, making it difficult to determine which P-image pair to use for gaze calculation. Furthermore, calculating the gaze from each of the multiple P-image pairs can be time-consuming, as it may take considerable time to determine the gaze that is clearly normal.

[0007] Therefore, the present invention aims to prevent a decrease in gaze detection accuracy even when the time required for gaze detection processing is predetermined, when there are multiple combinations of corneal reflection images (P-image pairs) included in the eyeball image. [Means for solving the problem]

[0008] To achieve the above objective, the electronic device of the present invention comprises: a plurality of irradiation means for irradiating the user's eyeball with infrared light; an imaging means for capturing an eyeball image obtained when the infrared light irradiated by the plurality of irradiation means is reflected by the eyeball; a first calculation means for calculating the user's line of sight position when viewing the display unit based on a combination of a plurality of corneal reflection images included in the eyeball image captured by the imaging means; and a setting means for setting an order for each of the plurality of corneal reflection image combinations based on predetermined conditions during a calibration process to improve the detection accuracy of the line of sight detected by the detection means, wherein the first calculation means performs processing using the combination of corneal reflection images based on the order set by the setting means within a predetermined time and calculates the user's line of sight position when viewing the display unit. [Effects of the Invention]

[0009] According to the present invention, when there are multiple combinations of corneal reflection images (P-image pairs) included in the eyeball image, the gaze detection accuracy is less likely to decrease even if the time required for gaze detection processing is a predetermined time. [Brief explanation of the drawing]

[0010] [Figure 1]This is a perspective view showing an example of the external appearance of a digital still camera 1, which is an example of an electronic device according to this embodiment 1. [Figure 2] This is a cross-sectional view showing an example of the configuration of a digital still camera 1, which is an example of an electronic device according to this embodiment 1. [Figure 3] This block diagram shows an example of the configuration of a digital still camera 1, which is an example of an electronic device according to this embodiment 1, focusing on the electrical circuit. [Figure 4] This figure shows an example of a finder image observed through the eyepiece 12. [Figure 5] This is a diagram explaining the principle of gaze detection. [Figure 6] (a) A schematic diagram of the eyeball image formed by the light-receiving lens 16, and (b) A schematic diagram of the brightness distribution in region α. [Figure 7] This is a flowchart illustrating the gaze detection process when there is only one P-image pair. [Figure 8] This is a flowchart related to the calibration process. [Figure 9] This diagram shows how three corneal reflection images (P1-P3) are projected onto the eyeball. [Figure 10] This is a flowchart for the subroutine for rearranging P-image pairs. [Figure 11] This is a flowchart illustrating the gaze detection process when there are multiple P-image pairs. [Figure 12] This is a perspective view showing an example of the external appearance of a head-mounted display (HMD), which is an example of an electronic device according to this second embodiment. [Figure 13] This is a block diagram showing an example of a system configuration of an HMD, which is an example of an electronic device according to this second embodiment. [Modes for carrying out the invention]

[0011] Preferred embodiments of the present invention will be described below with reference to the attached drawings.

[0012] (Embodiment 1) <Description of the exterior> FIG. 1 is a perspective view showing an external appearance example of a digital still camera (hereinafter referred to as “camera”) 1 according to Embodiment 1. FIG. 1(a) is a perspective view seen from the front side, and FIG. 1(b) is a perspective view seen from the back side.

[0013] In FIG. 1, as a camera coordinate system, an XYZ orthogonal coordinate system is defined with the optical axis of the lens unit 1A as the Z axis, a vertical axis orthogonal to the Z axis as the Y axis, and an axis orthogonal to the Z axis and the Y axis as the X axis. Note that the origin of the camera coordinate system may be, for example, the intersection of the imaging surface and the optical axis, but is not limited thereto.

[0014] The camera 1 includes a camera body 1B and a lens unit 1A that is detachable from the camera body 1B. The release button 5 is an operation member that receives an imaging instruction from the user. Operation members such as the release button 5 are hereinafter referred to as “operation units”. An eyepiece lens 12 for the user to look into a display element (to be described later) included inside the camera 1 is disposed on the back surface of the camera 1, and the user can visually recognize a field-of-view image by looking into the eyepiece lens 12.

[0015] <Description of Configuration> FIG. 2 is a cross-sectional view showing a configuration example of a camera 1 which is an example of an electronic device according to the present embodiment. FIG. 2 is a cross-sectional view of the camera 1 cut along the YZ plane formed by the Y axis and the Z axis illustrated in FIG. 1. In FIGS. 1 and 2, corresponding configurations are denoted by the same reference numerals.

[0016] When the lens unit 1A is attached to the camera body 1B, the lens unit 1A and the camera body 1B are electrically connected through the mount contact 117. Power is supplied from the camera body 1B to the lens unit 1A through the mount contact 117. Further, the circuit in the lens unit 1A and the CPU 3 of the camera body 1B can communicate with each other through the mount contact 117.

[0017] The lens unit 1A has a movable lens 101 and a fixed lens 102. In Figure 2, each is shown as a single lens, but in reality, it is composed of multiple lenses. Here, the movable lens 101 is assumed to be a focusing lens, but other movable lenses such as a magnification lens or an image stabilization lens may also be included.

[0018] The movable lens 101 is supported by a lens drive member 114 and driven in the optical axis direction (left-right direction in the drawing) by a lens drive motor 113. The rotation of a pulse plate 116, which is linked to the lens drive member 114, is detected by a photocoupler 115 and output to the focus adjustment circuit 118. The focus adjustment circuit 118 can detect the amount and direction of drive of the movable lens 101 based on the output of the photocoupler 115. When the CPU 3 of the camera body 1B instructs the focus adjustment circuit 118 on the amount and direction of drive of the movable lens 101, it controls the operation of the lens drive motor 113 based on the output of the photocoupler 115.

[0019] In the camera body 1B, the image sensor 2 is a CCD image sensor or a CMOS image sensor. Multiple pixels are arranged in two dimensions on the image sensor 2, and each pixel is provided with one microlens, one color filter, and one or more photoelectric conversion units. In this embodiment, multiple photoelectric conversion units are provided on each pixel, and a signal can be read out from each photoelectric conversion unit. By configuring the pixels in this way, it is possible to generate an captured image, a parallax image pair, and an image signal for phase-detection AF (autofocus) from the signal read out from the image sensor 2. The image sensor 2 converts the optical image formed by the lens unit 1A into a group of pixel signals (analog image signals) by photoelectric conversion by multiple pixels. In addition, in this embodiment, the image sensor 2 has an A / D conversion function and converts the analog image signal into digital image data and outputs it.

[0020] The memory unit 4 includes non-volatile memory (ROM) and volatile memory (RAM). The CPU 3 controls the operation of the camera body 1B and lens unit 1A by reading programs stored in ROM into RAM and executing them, thereby realizing the functions of the camera. The memory unit 4 also includes a recording medium (such as a memory card) for recording image data and audio data obtained during shooting. The CPU 3 controls the operation of the focus adjustment circuit 118 and the aperture drive unit 112 through the mount contact 117.

[0021] The non-volatile memory in memory section 4 may be rewritable. The non-volatile memory stores programs executed by the CPU 3, various setting values, GUI (Graphical User Interface) image data, and gaze correction data to compensate for individual differences in gaze.

[0022] The display element 10 is an LCD (Liquid Crystal Display) or an organic EL display panel, and displays captured images such as live view images, menu screens, and various information.

[0023] The display element driving circuit 11 drives the display element 10 according to the control of the CPU 3. Since the display element 10 is located inside the camera body 1B, an eyepiece is provided for observing the display element 10 from outside the camera body 1B. The eyepiece 119 is equipped with an eyepiece lens 12 and illumination light sources 13a to 13f for line-of-sight detection. The eyepiece 119 is also equipped with a light splitter 15 for capturing an image of the eyeball, a light-receiving lens 16, and an image sensor 17 for the eyeball.

[0024] The illumination sources 13a to 13f are multiple infrared LEDs positioned around the eyepiece 12, illuminating the user's eyeball 14 as they look through the eyepiece with infrared light. The image of the eyeball obtained by the infrared light from the illumination sources 13a to 13f reflecting off the eyeball 14 is reflected by the light divider 15 and captured by the eyeball image sensor 17 via the light-receiving lens 16 located above it. The light-receiving lens 16 positions the pupil of the user's eyeball 14 and the eyeball image sensor 17 in a conjugate imaging relationship. The eyeball image sensor 17 has multiple pixels arranged in a two-dimensional array and is configured to capture an image using infrared light. The number of pixels in the eyeball image sensor 17 may be less than the number of pixels in the image sensor 2. Based on the positional relationship between the corneal reflection and the pupil in the eyeball image obtained by the eyeball image sensor 17, the line of sight of the eyeball 14 can be detected.

[0025] <Explanation of block diagram> Figure 3 is a block diagram showing an example configuration of the camera 1 of this embodiment, focusing on the electrical circuit. The CPU 3 is connected to the gaze detection circuit 201, the photometering circuit 202, the autofocus detection circuit 203, the operation unit 204, the display element drive circuit 11, the illumination light source drive circuit 205, and the liquid crystal display unit 120. In addition, the focus adjustment circuit 118 and the aperture control circuit 206 (included in the aperture drive unit 112) provided on the photographic lens 1 are electrically connected to the CPU 3 via the mount contact 117.

[0026] The gaze detection circuit 201 performs A / D conversion on the analog image signal of the eyeball image obtained from the eyeball image sensor 17 and transmits it to the CPU 3 as digital image data. The CPU 3 detects feature points necessary for gaze detection from the digital image data of the eyeball image according to a known algorithm and detects the user's gaze position from the position of each feature point.

[0027] The photometering circuit 202 generates luminance information as a predetermined evaluation value for exposure control based on image data obtained from the image sensor 2, and outputs it to the CPU 3. The CPU 3 performs automatic exposure control (AE) processing based on the luminance information and determines the shooting conditions. For example, for still image shooting, the shooting conditions are shutter speed, aperture value, and ISO sensitivity. Based on the determined shooting conditions, the CPU 3 controls the aperture value (aperture) of the aperture 111 of the shooting lens 1. The CPU 3 also controls the operation of the mechanical shutter inside the main body 20.

[0028] The autofocus detection circuit 203 generates an image signal for phase-detection autofocus based on image data obtained from the image sensor 2 and outputs it to the CPU 3. The CPU 3 calculates the amount of defocus based on the phase difference of the image signal for phase-detection autofocus. This is a known technique known as image plane phase-detection autofocus. In this embodiment, as an example, it is assumed that there are 180 focus detection points on the image plane corresponding to the locations shown in the viewfinder image (described later) in Figure 4, but this is not limited to this.

[0029] The control unit 204 is a collective term for multiple user-operable input devices (buttons, switches, dials, etc.), including the release button 5 described earlier. When the CPU 3 detects operation of an input device, it executes processing corresponding to the detected operation.

[0030] The release button 5 has a first shutter switch (SW1) that turns ON when half-pressed and a second shutter switch (SW2) that turns ON when fully pressed. When the CPU 3 detects that SW1 is ON, it performs preparatory operations for still image shooting. These preparatory operations include AE ​​processing and AF processing. When the CPU 3 detects that SW2 is ON, it performs still image shooting and recording operations according to the shooting conditions determined by the AE processing.

[0031] The illumination light source drive circuit 205 controls the light emission operation of illumination light sources 13a to 13f according to the control of the CPU 3.

[0032] The liquid crystal display unit 120 displays information on a display device such as an LCD or organic EL display in accordance with the signals from the CPU 3.

[0033] Figure 4 shows an example of a finder image according to this embodiment. Here, the finder image is an image displayed on the display element 10, with various indicators superimposed. The user can observe the finder image in Figure 4 through the eyepiece 12. Figure 4 shows the view within the finder field of view, indicating that the display element 10 is in operation.

[0034] In Figure 4(a), the viewfinder image displays the field mask 300, an index 400 indicating the range in which focus can be detected, and 180 indexes (AF frames) 4001 to 4180 displayed at positions corresponding to points in which focus can be detected (focus detection points). Of these AF frames, the AF frame corresponding to the current line of sight is highlighted as the estimated line of sight A. Here, the estimated line of sight A highlighted in Figure 4(a) is an image displayed based on the estimated line of sight.

[0035] <Explanation regarding eye-tracking detection> We will explain gaze detection using Figures 5 to 7.

[0036] Figure 5 is a diagram illustrating the principle of gaze detection. In Figure 2, the image sensor is positioned so that the optical axis of the image sensor 17 is perpendicular to the optical axis of the eyeball 14. On the other hand, in Figure 5, for convenience, the optical axis of the image sensor 17 is positioned horizontally to the optical axis of the eyeball 14. The illumination light sources 13a to 13f are positioned approximately symmetrically with respect to the optical axis of the light-receiving lens 16 and irradiate the user's eyeball 14 with infrared light. In Figure 5, only the illumination light sources 13a and 13b are shown. The light-receiving lens 16 forms an image of the eyeball on the imaging surface of the image sensor 17, based on the infrared light reflected by the eyeball 14.

[0037] Figure 6(a) is a schematic diagram of the eyeball image formed by the light-receiving lens 16, and Figure 6(b) is a schematic diagram of the brightness distribution in region α of Figure 6(a).

[0038] [Flowchart for gaze detection processing when there is one P-image pair] Figure 7 is a flowchart of the gaze detection process according to this embodiment, where there is one combination of corneal reflection images (P-image pair) included in the eyeball image. The gaze detection process can be executed, for example, when it is detected that an object is close to the eyepiece 12. The proximity of an object to the eyepiece 12 can be detected using any known method, such as using a proximity sensor provided near the eyepiece 12. The gaze detection process may also be started in response to user instructions through the operation unit 204. The process in Figure 7 is executed by the CPU 3 controlling each part.

[0039] In the S701, the CPU 3 lights up the illumination light sources 13a and 13b shown in Figure 5 via the illumination light source drive circuit 205. As a result, infrared light is emitted from the illumination light sources 13a and 13b toward the outside of the camera body 1B. The infrared light is reflected by the user's eyeball looking through the eyepiece 12, further reflected by the light splitter 15, and incident on the light receiving lens 16.

[0040] In S702, CPU3 captures images using the eyeball image sensor 17. The eyeball image formed by the light-receiving lens 16 is converted into an image signal by the eyeball image sensor 17. The image signal is A / D converted by the gaze detection circuit 201 and input to CPU3 as eyeball image data.

[0041] In S703, CPU3 determines the coordinates of the corneal reflection images Pd' and Pe' of the illumination light sources 13a and 13b, and the coordinates of the image point c' of the pupil center c, from the eyeball image data acquired in S702. The eyeball image obtained by the eyeball image sensor 17 includes the corneal reflection images Pd' and Pe' corresponding to the images Pd and Pe of the illumination light sources 13a and 13b projected onto the cornea 142 (Figure 6(a)). Here, it is assumed that there is one combination of corneal reflection images (P-image pair) included in the eyeball image.

[0042] As shown in Figure 6(a), the horizontal direction is defined as the X-axis and the vertical direction as the Y-axis. In this case, the X-axis coordinates of the centers of the corneal reflection images Pd' and Pe' of the illumination light sources 13a and 13b, which are included in the eyeball image, are defined as Xd and Xe. Also, the X-axis coordinates of the images a' and b' of the pupillary apex a and b, which are the ends of the pupil 141, are defined as Xa and Xb.

[0043] As shown in Figure 6(b), the luminance at coordinates Xd and Xe, corresponding to the corneal reflection images Pd' and Pe' of the illumination sources 13a and 13b, is significantly higher than the luminance at other locations. On the other hand, the luminance in the range from coordinate Xa to Xb, corresponding to the pupil 141 region, is significantly lower, except for coordinates Xd and Xe. Furthermore, in the range corresponding to the iris 143 region outside the pupil 141, where the coordinate is smaller than Xa and larger than Xb, the luminance is intermediate between the luminance of the corneal reflection image of the illumination source and the luminance of the pupil.

[0044] Based on these characteristics of brightness levels in the X-axis direction, the CPU 3 can detect the X-axis coordinates Xd and Xe of the corneal reflection images Pd' and Pe' of the illumination light sources 13a and 13b, and the X-axis coordinates Xa and Xb of the images a' and b' of the pupil ends a and b from the eyeball image. Furthermore, in applications such as this embodiment, the rotation angle θx of the optical axis of the eyeball 14 with respect to the optical axis of the light-receiving lens 16 is relatively small. In such cases, the X-axis coordinate Xc of the image c' of the pupil center c in the eyeball image can be expressed as Xc ≈ (Xa + Xb) / 2. In this way, the CPU 3 can determine the coordinates of the corneal reflection images Pd' and Pe' of the illumination light sources 13a and 13b, and the X-axis coordinate of the image c' of the pupil center c, from the eyeball image.

[0045] In S704, CPU3 calculates the imaging magnification β of the eyeball image. β is a magnification determined by the position of the eyeball 14 relative to the light-receiving lens 16, and can be obtained as a function of the interval (Xd-Xe) between the corneal reflection images Pd' and Pe' of the illumination light source.

[0046] In S705, CPU3 calculates the rotation angle of the eyeball. The X-axis coordinate of the midpoint of the images Pd and Pe of the illumination light source on the cornea 142 approximately coincides with the X-axis coordinate of the center of curvature O of the cornea 142. Therefore, if Oc is the standard distance between the center of curvature O of the cornea 142 and the center c of the pupil 141, the rotation angle θx of the optical axis of the eyeball 14 in the ZX plane can be obtained from the relationship β*Oc*SINθx≈{(Xd+Xe) / 2}-Xc.

[0047] Figures 5 and 6 show an example of calculating the rotation angle θx in a plane perpendicular to the Y-axis (ZX plane), but the rotation angle θy in a plane perpendicular to the X-axis (ZY plane) can be calculated similarly. In this way, CPU3 determines the rotation angles θx and θy of the eyeball. Then, the line of sight position can be calculated from these rotation angles of the eyeball.

[0048] In the S706, the CPU 3 obtains correction coefficients (Ax, Bx, Ay, By) from the memory unit 4. These correction coefficients compensate for individual differences such as the shape of the user's eyeballs. The correction coefficients are calculated through a calibration process and stored in the memory unit 4 before starting the gaze detection process shown in Figure 7. If the memory unit 4 stores correction coefficients for multiple users, it uses the correction coefficient corresponding to the current user at any time, for example, by querying the user. The method for calculating the correction coefficients through the calibration process will be described later.

[0049] In S707, CPU3 calculates the user's gaze coordinates (gaze position) on the display element 10 using the eye rotation angles θx and θy calculated in S705. The user's gaze position can also be calculated using the following formulas: Hx = m × (Ax × θx + Bx) and Hy = m × (Ay × θy + By), assuming that the coordinates (Hx, Hy) correspond to the center c of the pupil 141 on the display element 10.

[0050] Here, the coefficient m is a transformation coefficient that converts the rotation angles θx and θy to coordinates corresponding to the center c of the pupil 141 on the display element 10, and is determined by the characteristics of the eyepiece lens 12 of the camera's viewfinder optical system. The coefficient m can be stored in the memory unit 4 beforehand. Also, Ax, Bx, Ay, and By are correction coefficients obtained in S706.

[0051] In S709, CPU3 records the gaze position (Hx, Hy) and the time when eyeball image data was acquired in S702 (gaze detection time) in memory unit 4, and then terminates the gaze detection process.

[0052] As described above, the line of sight position (Hx, Hy) was calculated by obtaining the imaging magnification β of the eyeball image from one P-image pair (corneal reflection images Pd', Pe'), and the rotation angles θx and θy of the eyeball from the imaging magnification β. Thus, if there is only one P-image pair, the line of sight position can be calculated from that P-image pair. However, as the number of illuminating light sources increases, the number of corneal reflection images also increases. Consequently, the number of combinations of corneal reflection images (P-image pairs) corresponding to the number of illuminating light sources also increases. Specifically, as the number of illuminating light sources increases to 2, 3, 4, etc., the number of P-image pairs increases. n The number of C2 (n = number of lit lights, n ≥ 2) increases. If the line of sight position is calculated from each of the multiple P-image pairs, the time required for line of sight detection processing increases. Therefore, it is required that even if the number of lit lights increases, the time required for line of sight detection processing does not increase, and the accuracy of the line of sight position detected by the line of sight detection processing does not decrease.

[0053] In this embodiment, during the calibration process, P-image pairs are rearranged from among multiple P-image pairs in an order that allows for accurate calculation of the gaze position, and this order is stored. Furthermore, a time limit is set for the gaze detection process, and the gaze detection process is performed within this predetermined time. When the gaze detection process is executed, the gaze position is calculated in the order of the stored P-image pairs. This reduces the likelihood of a decrease in the accuracy of the gaze position detected by the gaze detection process, even if calculations using all P-image pairs are not completed within the predetermined time, and allows the gaze detection process to be executed within the predetermined time. Alternatively, instead of rearranging the P-image pairs, an order may be set for the P-image pairs, and the gaze position may be calculated in the set order. Also, if the user has set a priority for the speed of the gaze detection process, the gaze position may be calculated in the order of the stored P-image pairs during the calibration process. In that case, if the user has not set a priority for the speed of the gaze detection process, the gaze position will be calculated in an order other than the order of the stored P-image pairs during the calibration process.

[0054] [Calibration process] Calibration is a process to more accurately detect the gaze position corresponding to the user's line of sight. In other words, calibration is a process to improve the accuracy of gaze detection. There are individual differences in the overall structure of the eye, such as the shape of the user's eyeball, and for some users, it may be difficult to determine the gaze position corresponding to their line of sight. For example, as shown in Figure 4(b), there may be a discrepancy between the gaze position B that the user is actually looking at and the estimated gaze position C detected by the gaze detection process. In that case, even if the system wants to perform AF processing at the person's position based on the gaze position, if the gaze position detected by the gaze detection process is in the background, AF processing will be performed at the background position. This results in AF processing being performed at a position different from the user's intention.

[0055] Therefore, by performing a calibration process, gaze data, which is user-specific gaze information for the user using camera 1, can be obtained. By calculating correction coefficients (Ax, Bx, Ay, By) from the obtained user-specific gaze data, the gaze position corresponding to that user's gaze can be determined more accurately.

[0056] In this embodiment, during the calibration process, correction coefficients (Ax, Bx, Ay, By) are calculated, and if the eye image contains multiple P-image pairs, the P-image pairs are rearranged in an order that allows for accurate calculation of the line of sight position, and this order is stored. This calibration process will be explained in detail using the flowchart in Figure 8.

[0057] The calibration process is executed when Camera 1 is started and a calibration command is issued in the settings menu screen (by selecting the setting item "Perform Calibration"). When a calibration command is issued, the viewfinder image shown in Figure 4(c) is displayed, and indicators D to G, which the user is supposed to focus on, are highlighted in order. The calibration process is executed while each indicator is highlighted. For example, the calibration process in Figure 8 shows the process executed in response to indicator D among indicators D to G, but the process is also executed while each of the other indicators is highlighted. This calibration process is executed by the CPU 3 controlling each part.

[0058] In the S801, the CPU3 lights up the illumination light sources 13a to 13c via the illumination light source drive circuit 205. As a result, infrared light is emitted from the illumination light sources toward the outside of the camera body 1B. The infrared light is reflected by the user's eyeball looking through the eyepiece 12, further reflected by the light splitter 15, and then incident on the light receiving lens 16.

[0059] In S802, the CPU3 captures images using the eyeball image sensor 17. The eyeball image formed by the light-receiving lens 16 is converted into an image signal by the eyeball image sensor 17. The image signal is A / D converted by the gaze detection circuit 201 and input to the CPU3 as eyeball image data.

[0060] In S803, CPU3 obtains the coordinates of the corneal reflection images P1, P2, and P3 of the illumination light source, and the coordinates of the image point c' of the pupil center c, from the eyeball image data acquired in S802. Figure 9 shows how three corneal reflection images (P1~P3) are projected onto the eyeball. When three corneal reflection images are projected in this way, there are three possible combinations of these corneal reflection images (P-image pairs): P1 and P2, P2 and P3, and P1 and P3. In this embodiment, we describe the case where three corneal reflection images are projected onto the eyeball, but if n (≧3) corneal reflection images are projected, the number of P-image pairs can be calculated using the formula nCr = n! / r!(nr)!.

[0061] In the S804, CPU3 initially assigns the total number of P-image pairs to the variable N (N≧2) and assigns "0" to the variable i.

[0062] In the S805, CPU3 obtains the coordinates of the i-th P-image pair. The order in which the coordinates of the P-image pairs are obtained is predetermined by the user, for example, in an order that minimizes the influence of the user's eyelids on the corneal reflection image. If this is the first calibration process, the coordinates of the P-image pairs may be obtained in the order predetermined by the user. However, if this is not the first calibration process, the coordinates of the P-image pairs may be obtained in the order in which the P-image pairs were rearranged in the previous calibration process. Here, the first P-image pair is P1 and P2, and the X-axis coordinates of the centers of P1 and P2 are X1 and X2, respectively.

[0063] In the S806, the CPU3 calculates the imaging magnification β of the eyeball image. β is a magnification determined by the position of the eyeball 14 relative to the light-receiving lens 16, and can be calculated, for example, as a function of the distance (X1-X2) between the coordinates of the first pair of P images, P1 and P2.

[0064] In S807, CPU3 calculates the rotation angle of the eyeball. The X-axis coordinates of the midpoints of the images P1 and P2 of the illumination light source on the cornea 142 and the X-axis coordinates of the center of curvature O of the cornea 142 are approximately coincident. Therefore, if Oc is the standard distance between the center of curvature O of the cornea 142 and the center c of the pupil 141, the rotation angle θx of the optical axis of the eyeball 14 in the ZX plane can be obtained from the relationship β*Oc*SINθx≈{(X1+X2) / 2}-Xc. Similarly, the rotation angle θy in the plane perpendicular to the X axis (ZY plane) can also be calculated. In this way, CPU3 determines the rotation angles θx and θy of the eyeball.

[0065] In S808, CPU3 determines whether it has succeeded in calculating the rotation angle of the i-th P-image pair. If it determines that it has succeeded in calculating the rotation angle of the i-th P-image pair, it proceeds to S809; otherwise, it proceeds to S810. For example, if the coordinates of the pupil center can be detected from the eyeball image data acquired in S802, it is assumed that the calculation of the rotation angle of the i-th P-image pair has succeeded. The rotation angle of the i-th P-image pair is calculated by multiple samplings. If the rotation angle of the i-th P-image pair deviates by a predetermined value or more when compared with the results of multiple samplings, it is assumed that the calculation of the rotation angle of the i-th P-image pair has failed.

[0066] In the S809, the CPU 3 stores the rotation angle of the i-th P-image pair in the memory unit 4. In this embodiment, only the rotation angles of P-image pairs for which the rotation angle calculation was successful are stored in the memory unit 4. However, information regarding P-image pairs for which the rotation angle calculation was unsuccessful may also be stored in the memory unit 4. In that case, it is preferable to store in the memory unit 4 the result of determining whether each P-image pair is a P-image pair for which the rotation angle calculation was successful or unsuccessful, as information regarding the P-image pair.

[0067] In the S810, CPU3 is defined as i = i + 1.

[0068] In S811, CPU3 determines whether i > N. If it is determined that i > N, proceed to S812; otherwise, proceed to S805.

[0069] In S812, CPU3 determines whether a predetermined number of sampling cycles M (≧2) have been completed. Sampling M times means that the rotation angle will be calculated multiple times. If it is determined that sampling has been completed M times, i.e., multiple times, the process proceeds to S813; otherwise, it proceeds to S801.

[0070] In S813, CPU3 calculates the average value and variability of the rotation angle for each P-image pair whose rotation angle is stored, after M samplings. If it is determined that the calculation of the rotation angle was successful for all M samplings, the average values ​​of the rotation angles Avθx and Avθy can be calculated using the following formulas.

[0071]

number

[0072] Furthermore, assuming that the rotation angle was successfully calculated for all M samples, the variance values ​​Vθx and Vθy of the rotation angle can be calculated using the following formulas.

[0073]

number

[0074] In addition, in S813, the calculation process for P-image pairs that were determined in S808 to have failed to calculate the rotation angle is not performed.

[0075] In S814, CPU3 executes a subroutine that rearranges the P-image pairs in order of those that can accurately calculate the line of sight position, based on the average value and variability of the rotation angles calculated in S813, and then proceeds to S815. The subroutine for rearranging the P-image pairs will be described later using Figure 10.

[0076] In S815, the CPU3 stores the order of the P-image pairs in the memory unit 4 after rearranging the P-image pairs, and then proceeds to S816.

[0077] In S816, CPU3 calculates the average rotation angle at index D from the average rotation angle of each P-image pair. Specifically, it calculates the average rotation angles θxd and θyd at index D from the average rotation angles calculated from each of the three P-image pairs: P1 and P2, P2 and P3, and P1 and P3.

[0078] In S817, the correction coefficient is calculated from the average values ​​of the rotation angles θxd and θyd at index D and the coordinate position (Hxd, Hxy) of index D using the formulas Hxd = m × (Ax × θxd + Bx) and Hyd = m × (Ay × θyd + By). Here, the coefficient m is a conversion coefficient that transforms the rotation angles θxd and θyd into coordinates corresponding to the center c of the pupil 141 on the display element 10, and is determined by the characteristics of the eyepiece lens 12 of the camera's viewfinder optical system.

[0079] In S818, the correction coefficients (Ax, Ay, Bx, By) calculated in S817 are stored in the memory unit 4, and the calibration process is terminated.

[0080] (Subroutine for rearranging P-image pairs) Figure 10 is a flowchart of the subroutine for rearranging P-image pairs. There are five ways to rearrange P-image pairs, as shown in Figures 10(a) to (e), which will be explained in order.

[0081] In Figure 10(a), in S1001, CPU3 estimates the line of sight position (Hx, Hy) for each P-image pair from the average value of the rotation angle calculated in S813. The line of sight position (Hx, Hy) is calculated using the formulas Hx = m × (Ax × Avθx + Bx) and Hy = m × (Ay × Avθy + By), where the correction coefficients (Ax, Ay, Bx, By) are arbitrary values.

[0082] In S1002, CPU3 calculates the distance between the line of sight position (Hx, Hy) estimated in S1001 and the coordinate position of the indicator (Hxd, Hxy) for each pair of P images. The distance between the estimated line of sight position and the coordinate position of the indicator may be the distance in the X-axis direction, the distance in the Y-axis direction, or the Euclidean distance.

[0083] In S1003, CPU3 rearranges the P-image pairs in ascending order of the distance calculated in S1002, and then terminates the P-image pair rearrangement subroutine.

[0084] In Figure 10(b), in S1004, CPU3 performs the sorting of P-image pairs in order of increasing variability calculated for each pair in S813, and then terminates the P-image pair sorting subroutine.

[0085] In Figure 10(c), the processing in S1001 to S1003 is the same as the processing in Figure 10(a). In S1005, CPU3 rearranges the P-image pairs rearranged in S1003, selecting those of a predetermined rank or higher, in order of decreasing variability, and then terminates the P-image pair rearrangement subroutine.

[0086] In Figure 10(d), the processes S1001 to S1002 are the same as those in Figure 10(a), and the process S1004 is the same as that in Figure 10(b). In S1006, CPU3 rearranges the P-image pairs rearranged in S1004 in ascending order of distance, with P-image pairs of a predetermined rank or higher, and terminates the P-image pair rearrangement subroutine.

[0087] In Figure 10(e), the processing in S1001 to S1002 is the same as the processing in Figure 10(a). In S1007, CPU3 assigns points to each pair of P images, weighting them in order of increasing distance calculated in S1002.

[0088] In S1008, CPU3 assigns a score to each P-image pair, weighting them in order of increasing variability, as calculated in S813. The weighting method may be the same as in S1007.

[0089] In S1009, CPU3 sorts the P-image pairs in descending order of the sum of distance and variability scores, and then terminates the P-image pair sorting subroutine.

[0090] [Flowchart for gaze detection processing when there are multiple P-image pairs] Figure 11 is a flowchart of the gaze detection process according to this embodiment, where there are multiple combinations of corneal reflection images (P-image pairs) included in the eyeball image. The gaze detection process can be executed, for example, when it is detected that an object is close to the eyepiece 12. The proximity of an object to the eyepiece 12 can be detected using any known method, such as using a proximity sensor provided near the eyepiece 12. The gaze detection process may also be started in response to user instructions through the operation unit 204. The process in Figure 11 is executed by the CPU 3 controlling each part. The process similar to the flowchart in Figure 7 will not be explained.

[0091] In S1101, CPU3 initially assigns the total number of P-image pairs used in the current gaze detection process to the variable Q (Q≧2), and assigns "0" to the variable l. Here, the number of P-image pairs after the order has been rearranged is the total number of P-image pairs.

[0092] In S1102, CPU3 obtains the coordinates of the l-th P-image pair based on the order of the P-image pairs after sorting. The order of the P-image pairs from which coordinates are obtained is the sorted order stored in memory 4 in S815.

[0093] In S1103, CPU3 determines whether a predetermined time has elapsed for the gaze detection process. If it determines that the predetermined time has elapsed, it terminates the gaze detection process without proceeding to S1104. On the other hand, if it determines that the predetermined time has not elapsed, it proceeds to S1104. This allows the gaze detection process to be completed within the predetermined time, while also reducing the likelihood of a decrease in gaze detection accuracy.

[0094] In S1104, CPU3 sets l = l + 1.

[0095] In S1105, CPU3 determines whether l > Q. If it is determined that l > Q, the gaze detection process is terminated; otherwise, the process proceeds to S1102. If there are P-image pairs for which the rotation angle calculation was unsuccessful during the calibration process, the number of P-image pairs after rearrangement is small. In that case, the gaze detection process for Q P-image pairs may be completed before the predetermined time has elapsed. In such cases, S1105 is determined to be Yes.

[0096] As a result, the viewing position (Hx, Hy) on the display element 10 was calculated from each of the multiple P-image pairs. Then, the average value obtained by adding all these viewing positions and dividing by the number of P-image pairs was calculated as a single viewing position, and a pointer indicating the viewing position was displayed or the position of the AF frame was determined based on that viewing position. The average value of the viewing position (Hx, Hy) calculated from each of the multiple P-image pairs was calculated as a single viewing position, but the median value of the viewing position (Hx, Hy) calculated from each of the multiple P-image pairs could also be calculated as a single viewing position. Alternatively, the viewing position (Hx, Hy) calculated from each of the multiple P-image pairs that is closest in distance to the viewing position calculated in the previous frame may be used as the viewing position for determining the pointer display and the position of the AF frame.

[0097] (Embodiment 2) Embodiment 1 described an example of application to a digital still camera 1 as an electronic device, but it can also be applied to an HMD. Embodiment 2 describes an HMD. Figure 12 is a perspective view showing an example of the appearance of a head-mounted display (HMD), which is an example of an electronic device according to Embodiment 2. The HMD 1200 shown in Figure 12 is a display device worn on the user's head. It is equipped with operating members that can perform device operations such as a power switch and buttons to control device settings, and an image processing unit that functions as an image processing unit and performs tasks such as generating virtual objects and compositing images. The head mounting member 1201 is provided to stably fix the HMD 1200 to the user's head, so that the HMD 1200 can operate without shifting in accordance with the user's head movements. Although the head mounting member is configured to be fixed to the user's head, it may also be configured to be fixed by hooking it over the user's ears or supported by the hand.

[0098] The display device 1202 provides a virtual object, or a composite image of a real image and a virtual object, in front of the user's eyes. The display device 1202 may be, for example, a liquid crystal display or an organic EL display. The display device 1202 is composed of, for example, a liquid crystal panel, a driver circuit that controls the liquid crystal panel, and a memory that holds the image to be displayed. The display device 1202 may be a non-transparent display unit or an optically transmissive display unit that allows the outside view to be directly seen.

[0099] The imaging device 1203 is a device (camera unit) that captures images of the environment around the user. The imaging device 1203 may capture images of the environment in front of the user as the environment around the user. In front of the user may mean directly in front of the user's head.

[0100] Figure 13 is a block diagram showing an example of a system configuration of an HMD, which is an example of an electronic device according to this second embodiment.

[0101] The HMD shown in Figure 13 consists of an imaging unit 1300, a gaze detection unit 1301, a movement detection unit 1302, a rotation detection unit 1303, an information processing unit 1304, an image processing unit 1305, and a display unit 1306.

[0102] The information processing unit 1304 consists of a captured image acquisition unit 1307, a virtual object holding unit 1308, a virtual object generation unit 1309, a gaze information acquisition unit 1310, an HMD movement amount calculation unit 1311, an HMD rotation amount calculation unit 1312, and an HMD operation determination unit 1313.

[0103] The image processing unit 1305 consists of a virtual object selection unit 1314, a virtual object control unit 1315, and a display image generation unit 1316. Each block will be described later. Although not shown, the image processing unit 1305 also includes preprocessing, color interpolation, correction, detection, and data processing. Preprocessing includes signal amplification, reference level adjustment, and defective pixel correction. Color interpolation is the process of interpolating the values ​​of color components not included in the image data, and is also called demosaicing. Correction processing includes white balance adjustment, image brightness correction, optical aberration correction of the shooting lens 1A (not shown), and color correction. Detection processing includes detection and tracking of feature areas (for example, face areas, human body areas, animals, or automobiles), and person recognition. Furthermore, by analyzing the difference in image data along a time series, the image processing unit 1305 can acquire motion characteristic information of subjects, such as how subjects in the image data are moving within the image plane. Data processing includes scaling, encoding and decoding, and header information generation. These are examples of image processing that the image processing unit 1305 can perform, and do not limit the image processing performed by the image processing unit 1305.

[0104] The imaging unit 1300 consists of an optical system, an image sensor, a driver circuit for controlling the image sensor, an A / D conversion circuit for converting the signal acquired by the image sensor into a digital signal, and a development circuit for developing the obtained signal as an image. The image data captured by the imaging unit 1300 is acquired by the image acquisition unit 1307 of the information processing unit 1304 and transmitted to the display image generation unit 1316 of the image processing unit 1305. The display image generation unit 1316 performs a composite process with a virtual object, which will be described later, to generate a display image. The image acquisition unit 1307 also transfers the captured image to the virtual object generation unit 1309 to generate a virtual object.

[0105] The gaze detection unit 1301 is a device for detecting the direction of the user's gaze. For example, it may be a device that uses an infrared light-emitting diode, as conventionally used in single-lens reflex cameras, to illuminate the user's eyeball and detects the direction of the gaze from the relationship between the corneal reflection image (P image) and the pupil.

[0106] The gaze information detected by the gaze detection unit 1301 is acquired by the gaze information acquisition unit 1310 of the information processing unit 1304 and transmitted to the virtual object control unit 1315 of the image processing unit 1305. The virtual object control unit 1315 controls the virtual object, as described later, and transmits the information to the display image generation unit 1316 of the image processing unit 1305. The display image generation unit 1316 performs a composite process with the captured image to generate a display image.

[0107] The movement detection unit 1302 is a device for detecting the movement of the HMD. The movement information detected by the movement detection unit 1302 is used to calculate the amount of movement in the HMD movement amount calculation unit 1311 of the information processing unit 1304, and then transferred to the HMD 1313. The movement detection unit 1302 is an HMD movement detection unit, and the movement of the HMD is detected using movement detection means such as GPS (Global Positioning System) position information and acceleration sensors.

[0108] The rotation detection unit 1303 is a device for detecting the rotation of the HMD. The rotation information detected by the rotation detection unit 1303 is used by the HMD rotation amount calculation unit 1312 of the information processing unit 1304 to calculate the amount of rotation, and then transferred to the HMD 1313. The rotation detection unit 1303 is an HMD rotation detection unit, and the rotation of the HMD is detected using a rotation displacement sensor.

[0109] The HMD operation determination unit 1313 determines the operation of the HMD based on the HMD movement information transferred from the HMD movement amount calculation unit 1311 and the HMD rotation amount calculation unit 1312, using the HMD movement direction and amount, and the HMD rotation direction and amount. The operation information determined by the HMD operation determination unit 1313 is transferred to the virtual object control unit 1315 of the image processing unit 1305.

[0110] The virtual object holding unit 1308 holds data related to virtual objects that constitute the virtual space (shape information and position / orientation information), data related to light sources that illuminate the virtual space, and other data of the virtual space. Then, the virtual object selection unit 1314 of the image processing unit 1305 selects a virtual object and transfers it to the virtual object control unit 1315. The virtual object holding unit 1308 also transfers the virtual object to the virtual object generation unit 1309 to generate another virtual object.

[0111] The virtual object generation unit 1309 generates a virtual object based on the image data transferred from the captured image acquisition unit 1307 and the virtual object data transferred from the virtual object holding unit 1308. The generated virtual object is then transferred to the virtual object selection unit 1314 of the image processing unit 1305.

[0112] The virtual object selection unit 1314 selects virtual objects to be displayed to the user from the virtual object holding unit 1308 and the virtual object generation unit 1309, and transfers them to the virtual object control unit 1315 as virtual objects to be displayed. The virtual object data to be displayed includes data such as display image data, placement position data within the display field of view, and overlap order data with other virtual objects.

[0113] The virtual object control unit 1315 controls the virtual object data transferred from the virtual object selection unit 1314 based on the gaze information transferred from the gaze information acquisition unit 1310 and the HMD operation information transferred from the HMD operation determination unit 1313. The virtual object control unit 1315 then transfers the virtual object data to the display image generation unit 1316.

[0114] The display image generation unit 1316 creates a composite image of the captured image data transferred from the captured image acquisition unit 1307 and the virtual object transferred from the virtual object control unit 1315, and transfers it to the display unit 1306.

[0115] (Other embodiments) The various controls described above, which are performed by CPU3, may be performed by a single piece of hardware, or multiple pieces of hardware (for example, multiple processors or circuits) may share the processing to control the entire device.

[0116] Furthermore, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0117] The present invention can also be realized by performing the following process: supplying software (programs) that realize the functions of the embodiments described above to a system or device via a network or various storage media, and having the computer (or CPU, MPU, etc.) of that system or device read and execute the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0118] Furthermore, the disclosure of this embodiment includes the following configurations and methods.

[0119] (Composition 1) Multiple irradiation means for irradiating the user's eyeball with infrared light, An imaging means for capturing an image of an eyeball obtained by the reflection of the infrared light irradiated by the plurality of irradiation means by the eyeball, A first calculation means calculates the user's line of sight position when viewing the display unit based on a combination of multiple corneal reflection images included in the eyeball image captured by the imaging means, During a calibration process to improve the detection accuracy of the line of sight detected by the detection means, the system includes a setting means for setting the order of each of the multiple combinations of corneal reflection images based on predetermined conditions. The electronic device is characterized in that the first calculation means performs processing using the combination of corneal reflection images based on the order set by the setting means within a predetermined time, and calculates the line of sight position of the user viewing the display unit.

[0120] (Configuration 2) The electronic device according to Configuration 1, characterized in that the setting means performs processing using the combination of the plurality of corneal reflection images in a predetermined order during the calibration process, and rearranges the order of the combination of the plurality of corneal reflection images from the predetermined order based on the result of the processing and the predetermined conditions.

[0121] (Composition 3) During the calibration process, a second calculation means is provided for calculating the rotation angle of the eyeball multiple times for each of the multiple combinations of corneal reflection images based on the coordinates of the multiple combinations of corneal reflection images. The electronic device according to configuration 1 or 2, further characterized in that the second calculation means calculates the average value of the rotation angle of the eyeball and the variation of the rotation angle of the eyeball for each of the combinations of the plurality of corneal reflection images.

[0122] (Composition 4) The electronic device according to any one of configurations 1 to 3, characterized in that the predetermined conditions are the average value of the rotation angle of the eyeball calculated by the second calculation means, or conditions based on the variation in the rotation angle of the eyeball.

[0123] (Composition 5) The electronic device according to configuration 4, characterized in that the predetermined condition is the magnitude of the distance between the line of sight position estimated from the average value of the rotation angle of the eyeball calculated by the second calculation means and the index displayed on the display unit during the calibration process.

[0124] (Composition 6) The electronic device according to configuration 4, characterized in that the predetermined condition is the magnitude of the variation in the rotation angle of the eyeball calculated by the second calculation means.

[0125] (Composition 7) An imaging step of capturing an image of the eyeball obtained by reflecting infrared light from multiple irradiation means that irradiate the user's eyeball with infrared light, A first calculation step calculates the user's line of sight when viewing the display unit based on a combination of multiple corneal reflection images included in the eyeball image captured in the aforementioned imaging step, During the calibration process for improving the detection accuracy of the line of sight detected by the detection means, the calibration process includes a setting step of setting an order for each of the multiple combinations of corneal reflection images based on predetermined conditions. A method for controlling an electronic device, characterized in that the first calculation step performs processing using the combination of corneal reflection images based on the order set by the setting step within a predetermined time, and calculates the line of sight position of the user viewing the display unit.

[0126] (Composition 8) A program to cause a computer to perform the control methods of the electronic devices described in Configuration 7.

[0127] (Composition 9) A computer-readable recording medium containing a program that causes the computer to perform the control method of the electronic device described in Configuration 7.

Claims

1. Multiple irradiation means for irradiating the user's eyeball with infrared light, An imaging means for capturing an image of an eyeball obtained by the reflection of the infrared light irradiated by the plurality of irradiation means by the eyeball, A first calculation means calculates the user's line of sight position when viewing the display unit, based on a combination of multiple corneal reflection images included in the eyeball image captured by the imaging means. During a calibration process to improve the detection accuracy of the line of sight detected by the detection means, the system includes a setting means for setting the order of each of the multiple combinations of corneal reflection images based on predetermined conditions. The first calculation means is characterized in that, within a predetermined time, it performs processing using the combination of corneal reflection images based on the order set by the setting means, and calculates the line of sight position of the user viewing the display unit.

2. The electronic device according to claim 1, characterized in that the setting means performs processing using the combination of the plurality of corneal reflection images in a predetermined order during the calibration process, and rearranges the order of the combination of the plurality of corneal reflection images from the predetermined order based on the result of the processing and the predetermined conditions.

3. During the calibration process, a second calculation means is provided for calculating the rotation angle of the eyeball multiple times for each of the multiple combinations of corneal reflection images based on the coordinates of the multiple combinations of corneal reflection images. The electronic device according to claim 1, further characterized in that the second calculation means calculates the average value of the rotation angle of the eyeball and the variation of the rotation angle of the eyeball for each of the combinations of the plurality of corneal reflection images.

4. The electronic device according to claim 3, characterized in that the predetermined conditions are the average value of the rotation angle of the eyeball calculated by the second calculation means, or conditions based on the variation of the rotation angle of the eyeball.

5. The electronic device according to claim 4, characterized in that the predetermined condition is the magnitude of the distance between the line of sight position estimated from the average value of the rotation angle of the eyeball calculated by the second calculation means and the index displayed on the display unit during the calibration process.

6. The electronic device according to claim 4, characterized in that the predetermined condition is the magnitude of the variation in the rotation angle of the eyeball calculated by the second calculation means.

7. An imaging step of capturing an image of the eyeball obtained by reflecting infrared light from multiple irradiation means that irradiate the user's eyeball with infrared light, A first calculation step calculates the user's line of sight when viewing the display unit based on a combination of multiple corneal reflection images included in the eyeball image captured in the aforementioned imaging step, During the calibration process for improving the detection accuracy of the line of sight detected by the detection means, the calibration process includes a setting step of setting an order for each of the multiple combinations of corneal reflection images based on predetermined conditions. A method for controlling an electronic device, characterized in that the first calculation step performs processing using the combination of corneal reflection images based on the order set in the setting step within a predetermined time, and calculates the line of sight position of the user viewing the display unit.

8. A program for causing a computer to execute the control method of an electronic device described in claim 7.

9. A computer-readable recording medium having a program stored on it that causes a computer to execute the control method of the electronic device described in claim 7.

Images