Electronic device, control method of electronic device, program, and medium

Abstract

The present disclosure provides an electronic device, a control method of an electronic device, a program, and a medium. The electronic device includes a plurality of irradiation units configured to irradiate an eyeball of a user with infrared light, an image capturing unit configured to capture an ocular image obtained when the infrared light from the plurality of irradiation units is reflected by the eyeball, one or more memories, and one or more processors in communication with the one or more memories. The one or more processors and the one or more memories are configured to calculate a gaze position of a user viewing a display unit based on a combination of corneal reflection images included in the ocular image, and set an order with respect to each of the combinations of the corneal reflection images based on a predetermined condition during a calibration process for improving a calculation precision of the gaze position.

Description

Technical Field

[0001] This disclosure relates to an electronic device, a control method for the electronic device, a program, and a medium. Background Technology

[0002] In recent years, electronic devices that use user gaze information as a user interface have been used in various fields. Examples of such electronic devices include digital still cameras and head-mounted displays (HMDs). In these devices, the gaze of a user viewing the display unit can be detected, and various types of processing can be performed based on the gaze position corresponding to the detected gaze.

[0003] As a method for detecting gaze, the corneal reflection method has been proposed. According to this method, the eyeball is illuminated with infrared light from an illumination source, and an eye image image obtained from the infrared light reflected from the eyeball is captured by an eye image capturing element, thereby enabling the detection of the user's gaze when viewing the display unit.

[0004] Japanese Patent Application Publication No. 6-125874 describes a technique that, when the gaze result calculated based on a first-selected corneal reflection image (P-image) is significantly abnormal, attempts to perform gaze calculation using another P-image pair when other candidate P-image pairs exist.

[0005] When the number of illuminated light sources is high, the accuracy of gaze detection improves because the probability of accurately determining the magnification of the eye image increases. However, when the number of illuminated light sources is high, the captured eye image contains multiple P-image pairs, and it is difficult to determine which P-image pair to use for gaze calculation. Then, when calculating the gaze based on each of the multiple P-image pairs, it may take time to calculate gazes that appear to be normal. Summary of the Invention

[0006] In view of the above, the embodiments of this disclosure aim to ensure that, in the case where there are multiple combinations (P image pairs) of corneal reflection images contained in an eye image, the accuracy of gaze detection is unlikely to decrease even when the gaze detection processing takes a predetermined time.

[0007] To achieve the above objectives, according to one aspect of this disclosure, an electronic device is provided, comprising: a plurality of illumination units configured to illuminate a user's eyeball with infrared light; an image capturing unit configured to capture an eye image obtained when the infrared light from the plurality of illumination units is reflected by the eyeball; one or more memories; and one or more processors communicating with the one or more memories. The one or more processors and the one or more memories are configured to calculate the gaze position of a user viewing a display unit based on combinations of multiple corneal reflection images included in the eye image, and during calibration processing to improve the accuracy of the gaze position calculation, an order is set for each combination of the plurality of corneal reflection images based on predetermined conditions. Within a predetermined duration, processing is performed using the combinations of the plurality of corneal reflection images based on the set order to calculate the gaze position of the user viewing the display unit.

[0008] The features of various embodiments of this disclosure will become apparent from the following description of the embodiments with reference to the accompanying drawings. The embodiments are described below by way of example. Attached Figure Description

[0009] Figure 1A This is a perspective view illustrating an example of the appearance of a digital still camera, an example of an electronic device according to a first embodiment.

[0010] Figure 1B This is a perspective view illustrating an example of the appearance of a digital still camera, an example of an electronic device according to a first embodiment.

[0011] Figure 2 This is a cross-sectional view illustrating an example of the construction of a digital still camera, which is an example of an electronic device according to the first embodiment.

[0012] Figure 3 This is a block diagram illustrating an example construction of a digital still camera of an electronic device according to the first embodiment when focused on the circuit.

[0013] Figure 4A The illustration shows an example of a viewfinder image viewed through the eyepiece lens.

[0014] Figure 4B The illustration shows an example of a viewfinder image viewed through the eyepiece lens.

[0015] Figure 4C The illustration shows an example of a viewfinder image viewed through the eyepiece lens.

[0016] Figure 5 This is an explanatory diagram used to describe the principle of gaze detection.

[0017] Figure 6A This is a schematic diagram of the eye image formed by a light-receiving lens.

[0018] Figure 6B This is a schematic diagram of the brightness distribution in a region.

[0019] Figure 7 This is a flowchart related to gaze detection processing in the presence of a P-image pair.

[0020] Figure 8 This is a flowchart related to the calibration process.

[0021] Figure 9 The illustration shows three corneal reflection images projected onto the eyeball.

[0022] Figure 10A This is a flowchart related to the subroutine for rearranging the P-image pairs.

[0023] Figure 10B This is a flowchart related to the subroutine for rearranging the P-image pairs.

[0024] Figure 10C This is a flowchart related to the subroutine for rearranging the P-image pairs.

[0025] Figure 10D This is a flowchart related to the subroutine for rearranging the P-image pairs.

[0026] Figure 10E This is a flowchart related to the subroutine for rearranging the P-image pairs.

[0027] Figure 11 This is a flowchart related to gaze detection processing in the presence of multiple P-image pairs.

[0028] Figure 12 This is a perspective view illustrating an example of the appearance of a head-mounted display (HMD) of an electronic device according to a second embodiment.

[0029] Figure 13 This is a block diagram illustrating an example of the system architecture of an HMD for an electronic device according to a second embodiment. Detailed Implementation

[0030] In the following description, typical embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0031] First Embodiment

[0032] Description of appearance

[0033] Figure 1A and Figure 1BThis is a perspective view illustrating an example appearance of a digital still camera (hereinafter referred to as "camera") 1 according to a first embodiment, wherein, Figure 1A It is a perspective view taken from the front, and Figure 1B This is a perspective view taken from the back.

[0034] exist Figure 1A and Figure 1B In this context, the XYZ Cartesian coordinate system is defined as the camera coordinate system, where the Z-axis represents the optical axis of lens unit 1A, the Y-axis represents the axis perpendicular to the Z-axis, and the X-axis represents the axis orthogonal to both the Z and Y axes. It should be noted that the origin of the camera coordinate system can be, for example, the intersection of the image sensing surface and the optical axis, but is not limited to this.

[0035] Camera 1 includes a camera body unit 1B and a lens unit 1A detachably attached to the camera body unit 1B. Release button 5 is an operating component configured to receive image capture commands from the user. Hereinafter, operating components such as release button 5 will be referred to as "operating units". An eyepiece lens 12 is arranged on the back of camera 1 for the user to view the display elements included inside camera 1, which will be described below, and the user can visually identify the field of view image by viewing the eyepiece lens 12.

[0036] Description of the construction

[0037] Figure 2 This is a cross-sectional view illustrating an example of the construction of a camera 1 in an electronic device according to this embodiment. Figure 2 It is along Figure 1A and Figure 1B The image shows a cross-sectional view of camera 1 taken from the YZ plane formed by the Y-axis and Z-axis.

[0038] exist Figure 1A , Figure 1B and Figure 2 In the figures, the corresponding components are given the same reference numerals.

[0039] When lens unit 1A is mounted onto camera body unit 1B, lens unit 1A and camera body unit 1B are electrically connected to each other via mounting contact 117. Power is supplied from camera body unit 1B to lens unit 1A via mounting contact 117. In addition, the circuitry in lens unit 1A and the central processing unit (CPU) 3 of camera body unit 1B can communicate with each other via mounting contact 117.

[0040] Lens unit 1A includes a movable lens 101 and a fixed lens 102. Figure 2 In the image, each is shown as a single lens, but in reality, it consists of multiple lenses.

[0041] Here, the movable lens 101 is a focusing lens, but it may also include other movable lenses such as zoom lenses and image stabilization lenses.

[0042] The movable lens 101 is supported by a lens driving component 114 and driven by a lens driving motor 113 along the optical axis (left-right direction in the figure). The rotation of the pulse plate 116, which is linked to the lens driving component 114, is sensed by an optical coupler 115, and the result is output to the focus adjustment circuit 118. The focus adjustment circuit 118 can detect the driving amount and driving direction of the movable lens 101 based on the output of the optical coupler 115. When the CPU 3 of the camera main unit 1B instructs the CPU 3 of the movable lens 101 on the driving amount and driving direction, the focus adjustment circuit 118 controls the operation of the lens driving motor 113 based on the output of the optical coupler 115.

[0043] In the camera main unit 1B, the image capturing element 2 is a CCD image sensor or a CMOS image sensor. The image capturing element 2 has multiple pixels arranged in a two-dimensional configuration, and each pixel includes a single microlens, a single color filter, and one or more photoelectric conversion units. In this embodiment, a structure is adopted in which each pixel has multiple photoelectric conversion units and signals can be read for each photoelectric conversion unit. By employing this pixel structure, a captured image, a parallax image pair, and an image signal for phase difference autofocus (AF) can be generated based on the signals read from the image capturing element 2. The image capturing element 2 converts the optical image formed by the lens unit 1A into a pixel signal group (analog image signal) through photoelectric conversion of multiple pixels. Furthermore, according to this embodiment, the image capturing element 2 has an analog-to-digital (A / D) conversion function, converting analog image signals into digital image data and outputting the converted digital image data.

[0044] Memory unit 4 includes non-volatile memory (ROM) and volatile memory (RAM). By reading a program stored in the ROM into the RAM and executing that program, CPU 3 controls the operation of the camera body unit 1B and lens unit 1A, thereby realizing the camera's functions. Additionally, memory unit 4 also includes a medium (such as a memory card) configured to store image data and audio data acquired through image capture. CPU 3 controls the operation of the focus adjustment circuit 118 and the aperture drive unit 112 via mounting contacts 117.

[0045] The non-volatile memory of memory unit 4 can be rewritable. The non-volatile memory stores programs executed by CPU 3, various settings, image data of the graphical user interface (GUI), and vision correction data for individual differences in vision correction.

[0046] The display element 10 is a liquid crystal display (LCD) or an organic electroluminescent (EL) display panel, and displays captured images (e.g., live view images), menu screens, various information, etc.

[0047] The display element driving circuit 11, under the control of the CPU 3, controls the display element 10. Since the display element 10 is located inside the camera main unit 1B, an eyepiece unit 119 is provided, configured to observe the display element 10 from outside the camera main unit 1B. The eyepiece unit 119 includes an eyepiece lens 12 and illumination sources 13a to 13f for line-of-sight detection. Additionally, the eyepiece unit 119 includes a beam splitter 15, a light-receiving lens 16, and an eye image capturing element 17, configured to capture an image of the eye.

[0048] Illumination sources 13a to 13f are multiple infrared light-emitting diodes (LEDs) arranged around the eyepiece lens 12, and the illumination sources 13a to 13f illuminate the eyeball 14 of the user viewing the eyepiece unit with infrared light. When the infrared light from the illumination sources 13a to 13f is reflected by the eyeball 14, the resulting eye image is reflected by the beam splitter 15 and captured by the eye image capturing element 17 via the upper-mounted light-receiving lens 16. The light-receiving lens 16 positions the pupil of the user's eyeball 14 and the eye image capturing element 17 in a conjugate imaging relationship. The eye image capturing element 17 includes multiple pixels arranged in two dimensions and is configured to capture infrared images. The number of pixels in the eye image capturing element 17 may be less than the number of pixels in the image capturing element 2. The line of sight of the eyeball 14 can be detected based on the positional relationship between the corneal reflection and the pupil in the eye image obtained by the eye image capturing element 17.

[0049] Description of the block diagram

[0050] Figure 3 This is a block diagram illustrating an example of the construction of the camera 1 of the first embodiment when the focus is on the circuit. The line-of-sight detection circuit 201, metering circuit 202, autofocus detection circuit 203, operation unit 204, display element driving circuit 11, illumination source driving circuit 205, and liquid crystal display unit 120 are connected to the CPU 3. Furthermore, the focus adjustment circuit 118 and aperture control circuit 206 (included in the aperture driving unit 112) provided in the camera lens are electrically connected to the CPU 3 via mounting contacts 117.

[0051] The gaze detection circuit 201 converts the analog image signal of the eye image obtained from the eye image capturing element 17 into digital image data via A / D conversion, and transmits the digital image data to the CPU 3. The CPU 3 detects feature points for gaze detection based on the digital image data of the eye image using existing algorithms, and detects the user's gaze position based on the position of each feature point.

[0052] The metering circuit 202 generates brightness information as an estimated value for previously set exposure control based on image data obtained from the image capturing element 2, and outputs the brightness information to the CPU 3. The CPU 3 performs automatic exposure control (AE) processing based on the brightness information and determines the image capturing conditions. In the case of still image capture, the image capturing conditions include, for example, shutter speed, F-stop, and ISO. Based on the determined image capturing conditions, the CPU 3 controls the F-stop (aperture amount) of the camera lens aperture 111. Additionally, the CPU 3 also controls the operation of the mechanical shutter in the main unit 20.

[0053] The autofocus detection circuit 203 generates an image signal for phase-difference AF based on image data obtained from the image capturing element 2, and outputs the image signal to the CPU 3. The CPU 3 calculates the defocus amount based on the phase difference of the image signal used for phase-difference AF. This is a technique proposed in the prior art for phase-difference AF on the image sensing surface. According to this embodiment, as an example, it is assumed that on the image sensing surface... Figures 4A to 4C The viewfinder image (described below) indicates the location of 180 focus detection points, but the construction is not limited to this.

[0054] Operation unit 204 is a general term for multiple input devices (such as buttons, switches, and dials) that can be operated by a user, including the release button 5 described above. When an operation of an input device is detected, CPU 3 performs processing based on the detected operation.

[0055] Release button 5 includes a first shutter switch (SW1) that is ON when half-pressed and a second shutter switch (SW2) that is ON when fully pressed. When CPU 3 detects that SW1 is ON, CPU 3 performs preparation operations for still image capture. Preparation operations include AE ​​processing, AF processing, etc. In addition, when CPU 3 detects that SW2 is ON, it performs recording operations and still image capture according to the image capture conditions determined in AE processing.

[0056] Under the control of CPU 3, the lighting source driving circuit 205 controls the light emission operation of lighting sources 13a to 13f.

[0057] The liquid crystal display unit 120 displays on a display such as an LCD or organic EL based on signals from the CPU 3.

[0058] Figures 4A to 4C An example viewfinder image according to this embodiment is illustrated. Here, the viewfinder image is an image displayed on the display element 10 with various indicators superimposed on it. The user can observe it through the eyepiece lens 12. Figures 4A to 4C The viewfinder image shown. Figures 4A to 4CThe diagram illustrates the viewfinder field of view representing the operating state of the display element 10.

[0059] exist Figure 4A In the viewfinder image, a field-of-view mask 300, an indicator 400 indicating the range for focus detection, and 180 indicators (AF frames) 4001 to 4180 located at positions corresponding to the focus detection point (focus detection point). Furthermore, the AF frame corresponding to the current viewing position is highlighted as the estimated viewing position A. Here, Figure 4A The estimated gaze position A, which is highlighted in the medium-highlight image, is based on the image displayed at the estimated gaze position.

[0060] Description related to gaze detection

[0061] Reference Figure 5 , Figure 6A , Figure 6B and Figure 7 Describe the gaze detection.

[0062] Figure 5 This is an explanatory diagram used to illustrate the principle of gaze detection. Figure 2 In this configuration, the image capturing element is arranged such that the optical axis of the eye-image capturing element 17 is positioned perpendicular to the optical axis of the eyeball 14. On the other hand, in... Figure 5 For convenience, the optical axis of the eye-image capturing element 17 is arranged horizontally relative to the optical axis of the eyeball 14. The illumination sources 13a to 13f are arranged approximately symmetrically with respect to the optical axis of the light-receiving lens 16, and illuminate the user's eyeball 14 with infrared light. Figure 5 Only illumination sources 13a and 13b are shown. The light-receiving lens 16 forms an infrared eye image reflected by the eyeball 14 on the image sensing surface of the eye image capturing element 17.

[0063] Figure 6A This is a schematic diagram of the eye image formed by the light-receiving lens 16, and Figure 6B yes Figure 6A A schematic diagram of the brightness distribution in region α.

[0064] Flowchart related to gaze detection processing in the presence of a single P-image pair.

[0065] Figure 7This is a flowchart related to gaze detection processing when a single combination (P-image pair) of corneal reflection images contained in an eye image exists according to this embodiment. For example, gaze detection processing can be performed when an object approaches the eyepiece lens 12. The approach of an object to the eyepiece lens 12 can be sensed by any method in the prior art, such as using a proximity sensor located near the eyepiece lens 12. Gaze detection processing can be started in response to a user's instruction via the operation unit 204. Figure 7 The processing in the CPU 3 is performed when the CPU 3 controls the various units.

[0066] In S701, CPU 3 uses the lighting source drive circuit 205 to... Figure 5 The illumination sources 13a and 13b shown are turned on. With this configuration, infrared light is emitted from the illumination sources 13a and 13b to the outside of the camera main unit 1B. The infrared light is reflected by the eyeball of the user viewing the eyepiece lens 12, and further reflected by the beam splitter 15 to be incident on the light-receiving lens 16.

[0067] In S702, CPU 3 captures an image using the eye image capturing element 17. The eye image formed by the light-receiving lens 16 is converted into an image signal by the eye image capturing element 17. The image signal is then converted from digital image to digital image by the gaze detection circuit 201 and input to CPU 3 as eye image data.

[0068] In S703, CPU 3 calculates the coordinates of the corneal reflection images Pd' and Pe' of the illumination sources 13a and 13b, as well as the coordinates of the image c' at the pupil center c, based on the eye image data obtained in S702.

[0069] The eye image acquired by the eye image capturing element 17 includes corneal reflection images Pd' and Pe' corresponding to the images Pd and Pe of the illumination sources 13a and 13b reflected on the cornea 142. Figure 6A Here, it is assumed that the corneal reflection images contained in the eye image exist as a single combination (P image pair).

[0070] like Figure 6A As shown, the horizontal direction is set as the X-axis, and the vertical direction is set as the Y-axis. The X-axis coordinates of the centers of the corneal reflection images Pd' and Pe' of the illumination sources 13a and 13b included in the eye image are then designated as Xd and Xe. Additionally, the X-axis coordinates of the images a' and b' of the pupil edges a and b, which are the ends of the pupil 141, are designated as Xa and Xb.

[0071] like Figure 6BAs shown, the brightness at coordinates Xd and Xe of the corneal reflective images Pd' and Pe' corresponding to illumination sources 13a and 13b is very high relative to the brightness at other locations. On the other hand, the brightness in the region corresponding to pupil 141, from coordinates Xa to Xb, is very low, except for coordinates Xd and Xe. Furthermore, the brightness in the region corresponding to the iris 143 outside pupil 141, within the range of coordinates less than Xa and greater than Xb, is an intermediate brightness between the brightness of the corneal reflective images of the illumination sources and the brightness of the pupil.

[0072] Based on this characteristic of brightness level in the X-axis direction, CPU 3 can detect the X-axis coordinates Xd and Xe of the corneal reflection images Pd' and Pe' of illumination sources 13a and 13b, and the X-axis coordinates Xa and Xb of the images a' and b' of the pupil edges a and b, from the eye image. Furthermore, for this application in this embodiment, the rotation angle θx of the optical axis of the eyeball 14 relative to the optical axis of the light-receiving lens 16 is relatively small. In this case, the X-axis coordinate Xc of the image c' at the pupil center c in the eye image can be expressed as Xc≈(Xa+Xb) / 2. In this way, CPU 3 can calculate the coordinates of the corneal reflection images Pd' and Pe' of illumination sources 13a and 13b, and the X-axis coordinates of the image c' at the pupil center c, based on the eye image.

[0073] In S704, CPU 3 calculates the imaging magnification β of the eye image. Here, β is a magnification determined by the position of the eyeball 14 relative to the light-receiving lens 16, and can be calculated as a function of the distance (Xd-Xe) between the corneal reflective images Pd' and Pe' of the illumination source.

[0074] In S705, CPU 3 calculates the rotation angle of the eyeball. The X-axis coordinates of the midpoints of the illumination light sources Pd and Pe on the cornea 142 approximately coincide with the X-axis coordinates of the curvature center O of the cornea 142. Therefore, when the standard distance from the curvature center O of the cornea 142 to the center c of the pupil 141 is denoted as Oc, the rotation angle θx of the optical axis of the eyeball 14 in the ZX plane can be calculated according to the relationship β*Oc*SINθx≈{(Xd+Xe) / 2}–Xc.

[0075] Figure 5 as well as Figure 6A and Figure 6B The illustration shows an example of calculating the rotation angle θx in a plane perpendicular to the Y-axis (ZX plane), but the rotation angle θy in a plane perpendicular to the X-axis (ZY plane) can be calculated similarly. In this way, CPU 3 calculates the eye rotation angles θx and θy. The gaze position can then be calculated based on these eye rotation angles.

[0076] In the S706, CPU 3 obtains the correction coefficients (Ax, Bx, Ay, By) from memory unit 4. These correction coefficients are used to correct for individual differences such as the shape of a user's eyeballs. The correction coefficients are calculated through a calibration process, and... Figure 7 The correction coefficients are stored in memory unit 4 before the gaze detection process begins. In the case where memory unit 4 stores correction coefficients for multiple users, the correction coefficients corresponding to the current user can be used, for example, by querying the user at any time. The calculation method for the correction coefficients during the calibration process will be described below.

[0077] In S707, CPU 3 calculates the user's gaze coordinates (gaze position) on display element 10 using the eye rotation angles θx and θy calculated in S705. Furthermore, the user's gaze position can be calculated as coordinates (Hx, Hy) corresponding to the center c of the pupil 141 on display element 10, using the equations Hx = m × (Ax × θx + Bx) and Hy = m × (Ay × θy + By).

[0078] Here, coefficient m is a transformation coefficient used to convert rotation angles θx and θy into coordinates corresponding to the center c of the pupil 141 on the display element 10, and is set by the characteristics of the eyepiece lens 12 of the camera's viewfinder optical system. Coefficient m can be pre-stored in the memory unit 4. In addition, Ax, Bx, Ay, and By are correction coefficients obtained in S706.

[0079] In S708, CPU 3 records the gaze position (Hx, Hy) and the time (gaze detection time) of obtaining eye image data in S702 in memory unit 4, and ends the gaze detection process.

[0080] As described above, the gaze position (Hx, Hy) is calculated by obtaining the magnification β of the eye image from a single P-image pair (corneal reflection images Pd' and Pe') and the rotation angles θx and θy of the eyeball from the magnification β. In this way, the gaze position can be calculated from the P-image pair when only a single P-image pair exists. However, as the number of illuminated light sources increases, the number of corneal reflection images also increases. In this case, the number of corneal reflection image combinations (P-image pairs) corresponding to the number of illuminated light sources also increases. Specifically, when the number of illuminated light sources increases to 2, 3, 4, etc., the number of P-image pairs increases. nC2 (n = the number of lit illumination sources, n≥2) increases. The time spent on gaze detection processing increases when calculating the gaze position based on each of multiple P-image pairs. Therefore, it is necessary to satisfy the following: even as the number of lit illumination sources increases, the time spent on gaze detection processing should not increase, and the accuracy of the gaze position detected by gaze detection processing should not decrease.

[0081] According to this embodiment, during the calibration process, the P-image pairs are rearranged from multiple pairs in order of accuracy for calculating the gaze position, and this order is stored. Furthermore, a predetermined duration is set for the gaze detection processing, and the gaze detection processing is performed within this predetermined duration. Then, when the gaze detection processing is executed, the gaze position is calculated according to the stored order of the P-image pairs. With this configuration, even if the calculation using all P-image pairs cannot be completed within the predetermined duration, the accuracy of the gaze position detected by the gaze detection processing is unlikely to decrease, and the gaze detection processing can be performed within the predetermined duration. It should be noted that even without rearranging the P-image pairs, an order can be set for the P-image pairs, and the gaze position can be calculated according to the set order of the P-image pairs. Additionally, when the user sets the speed of the gaze detection processing as a priority, the gaze position can be calculated according to the order of the P-image pairs stored during the calibration process. In the above case, when the user does not set the speed of the gaze detection processing as a priority, the gaze position is calculated in an order that is not based on the order of the P-image pairs stored during the calibration process.

[0082] Calibration Processing

[0083] Calibration processing is used to more accurately detect the position of the gaze corresponding to the user's line of sight. In other words, calibration processing improves the accuracy of gaze detection. Individual differences exist in the overall eye structure, such as the shape of the user's eyeball, making it difficult to determine the position of the gaze corresponding to the user's line of sight. For example, ... Figure 4B As shown, there may be a difference between the user's actual gaze position B and the estimated gaze position C detected by the gaze detection process. In the above case, even if it is expected that AF processing will be performed on the character's position based on the gaze position, if the gaze position detected by the gaze detection process is in the background, then AF processing will be performed at that position in the background. Therefore, AF processing is performed at a position that is not the user's expected position.

[0084] In view of the above, by performing calibration processing, gaze data, which serves as user-specific gaze information using camera 1, can be obtained. By calculating correction coefficients (Ax, Bx, Ay, By) based on the obtained user-specific gaze data, the gaze position corresponding to the user's gaze can be determined more accurately.

[0085] According to this embodiment, in the calibration process, correction coefficients (Ax, Bx, Ay, By) are calculated, and when the eye image contains multiple P-image pairs, the P-image pairs are rearranged according to the order of accuracy of the calculated gaze position, and the order is stored. (Refer to...) Figure 8 The flowchart describes the calibration process in detail.

[0086] The calibration process occurs when Camera 1 is activated and a calibration execution command is issued in the settings menu screen (selection of the "Perform" calibration setting). When a calibration execution command is issued, the following is displayed: Figure 4C The viewfinder image is used to sequentially highlight indicators D through G that the user is interested in, and a calibration process is performed during the highlighting of each indicator. For example, Figure 8 The calibration process represents the process performed corresponding to indicator D among indicators D to G, but also performs separate processes during the highlighting of indicators other than indicator D. This calibration process is performed when CPU 3 controls the various units.

[0087] In S801, CPU 3 turns on illumination sources 13a to 13c via illumination source drive circuit 205. This configuration emits infrared light from the illumination sources towards the outside of camera main unit 1B. The infrared light is reflected by the user's eyeball through viewing eyepiece lens 12 and further reflected by beam splitter 15 before incident on receiving lens 16.

[0088] In S802, CPU 3 captures an image using the eye image capturing element 17. The eye image formed by the light-receiving lens 16 is converted into an image signal by the eye image capturing element 17. The image signal is then converted from digital image to digital image by the gaze detection circuit 201 and input to CPU 3 as eye image data.

[0089] In S803, CPU 3 obtains the coordinates of corneal reflection images P1, P2 and P3 of the illumination source and the coordinates of image c' at the center c of the pupil, based on the eye image data obtained in S802. Figure 9 The illustration shows the projection of three corneal reflective images (P1 to P3) onto the eyeball. When three corneal reflective images are projected in this manner, the combinations of these images (P-image pairs) are three combinations: P1 and P2, P2 and P3, and P1 and P3. According to this embodiment, the projection of three corneal reflective images onto the eyeball will be described; however, when projecting n (n≥3) corneal reflective images, the number of P-image pairs can be calculated using the equation nCr = n! / r! (nr)!.

[0090] In S804, CPU 3 assigns the total number of P image pairs as the initial value to variable N (N≥2) and assigns "0" to variable i.

[0091] In S805, CPU 3 obtains the coordinates of the i-th P-image pair. The order in which the coordinates of the P-image pairs are obtained is predetermined by the user; for example, the order may be determined by the ascending order of the influence of the user's eyelids on the corneal reflection image. It should be noted that in the case of the first calibration process, the coordinates of the P-image pairs can be obtained in the order predetermined by the user. However, in cases where the calibration process is not the first, the coordinates of the P-image pairs can be obtained in the order in which the P-image pairs were rearranged in the previous calibration process. Here, the first P-image pair is set as P1 and P2, and the X-axis coordinates of the centers of P1 and P2 are set as X1 and X2, respectively.

[0092] In S806, CPU 3 calculates the imaging magnification β of the eye image. Here, β is, for example, a magnification determined by the position of the eyeball 14 relative to the light-receiving lens 16, and can be calculated as a function of the coordinate distance (X1-X2) between P1 and P2 of the first P image pair.

[0093] In S807, CPU 3 calculates the rotation angle of the eyeball. The X-axis coordinates of the midpoints of the images P1 and P2 of the illumination source on the cornea 142 approximately coincide with the X-axis coordinates of the center of curvature O of the cornea 142. Therefore, when the standard distance from the center of curvature O of the cornea 142 to the center c of the pupil 141 is denoted as Oc, the rotation angle θx of the optical axis of the eyeball 14 in the ZX plane can be calculated according to the relationship β*Oc*SINθx≈{(X1+X2) / 2}–Xc. Similarly, the rotation angle θy in the plane perpendicular to the X-axis (ZY plane) can also be calculated. In this way, CPU 3 calculates the rotation angles θx and θy of the eyeball.

[0094] In S808, CPU 3 determines whether the rotation angle calculation for the i-th P-image pair was successful. If the calculation is successful, the process proceeds to S809; otherwise, it proceeds to S810. For example, if the pupil center coordinates can be detected based on the eye image data obtained in S802, it is assumed that the rotation angle calculation for the i-th P-image pair was successful. Alternatively, the rotation angle for the i-th P-image pair is calculated by performing resampling. If the rotation angle of the i-th P-image pair deviates from a predetermined value or more than the result of resampling, it is assumed that the rotation angle calculation for the i-th P-image pair failed.

[0095] In S809, CPU 3 stores the rotation angle of the i-th P-image pair in memory unit 4. It should be noted that, according to this embodiment, only the rotation angles of P-image pairs whose rotation angles were successfully calculated are stored in memory unit 4. However, it is also possible to store information related not only to P-image pairs whose rotation angles were successfully calculated but also to P-image pairs whose rotation angles were not calculated in memory unit 4. In the above case, it is preferable to store the determination result of whether each P-image pair is a P-image pair whose rotation angle was successfully calculated or a P-image pair whose rotation angles were not calculated in memory unit 4 as information related to the P-image pair.

[0096] In the S810, CPU 3 is set to i = i + 1.

[0097] In S811, CPU 3 determines whether i>N is true. If i>N is true, the process proceeds to S812; if i>N is false, the process proceeds to S805.

[0098] In S812, CPU 3 determines whether the predetermined M (≥2) samplings have been completed. M samplings refer to repeatedly calculating the rotation angle.

[0099] If it is determined that M samplings (i.e., repeated samplings) have been completed, the process proceeds to S813; if it is determined that M samplings have not been completed, the process proceeds to S801.

[0100] In S813, for each pair of P images that stores the rotation angle, CPU 3 calculates the average and deviation of the rotation angle over M samples. Once it is determined that the rotation angle calculation was successful in all M samples performed, the average values ​​Avθx and Avθy of the rotation angle can be calculated using the following equations.

[0101] and

[0102]

[0103] In addition, regarding the bias, when it is confirmed that the rotation angle calculation is successful in all M samplings, the variances Vθx and Vθy of the rotation angle can be calculated using the following equations.

[0104] and

[0105]

[0106] It should be noted that in S813, no calculation processing is performed on the P-image pairs for which the rotation angle calculation was unsuccessful in S808.

[0107] In S814, CPU 3, based on the average and deviation of the rotation angles calculated in S813, executes a subroutine to rearrange the P-image pairs in order of accuracy of the calculated line-of-sight positions, and the flow proceeds to S815. The following will refer to... Figures 10A to 10E Describes a subroutine for rearranging the P-image pair.

[0108] In S815, CPU 3 stores the P-image pairs after P-image pair rearrangement processing in sequence in memory unit 4, and the process proceeds to S816.

[0109] In S816, CPU 3 calculates the average rotation angle in indicator D based on the average rotation angle of each P-image pair. Specifically, based on the average rotation angle calculated for each of the three P-image pairs P1 and P2, P2 and P3, and P1 and P3, the average rotation angles θxd and θyd in indicator D are calculated.

[0110] In S817, based on the average values ​​θxd and θyd of the rotation angles in indicator D and the coordinate position (Hxd, Hyd) of indicator D, correction coefficients are calculated using the equations Hxd = m × (Ax × θxd + Bx) and Hyd = m × (Ay × θyd + By). Here, coefficient m is a conversion coefficient used to convert the rotation angles θxd and θyd into coordinates corresponding to the center c of the pupil 141 on the display element 10, and is set by the characteristics of the eyepiece lens 12 of the camera's viewfinder optical system.

[0111] In S818, the correction coefficients (Ax, Ay, Bx, By) calculated in S817 are stored in memory unit 4, and the calibration process ends.

[0112] P-image for rearranged subroutines

[0113] Figures 10A to 10E This is a flowchart related to the subroutines for rearranging the P-image pairs. Methods for rearranging the P-image pairs include, for example... Figures 10A to 10E The five methods shown are described below.

[0114] exist Figure 10A In step S1001, CPU 3 estimates the gaze position (Hx, Hy) based on the average rotation angle calculated in S813 for each P image pair. The gaze position (Hx, Hy) is calculated using the equations Hx = m × (Ax × Avθx + Bx) and Hy = m × (Ay × Avθy + By), but at this time, the correction coefficients (Ax, Ay, Bx, By) are arbitrary values.

[0115] In S1002, CPU 3 calculates the distance between the estimated gaze position (Hx, Hy) and the coordinate position (Hxd, Hxy) of the indicator for each P image pair. The estimated distance between the gaze position and the coordinate position of the indicator can be a distance in the X-axis direction, a distance in the Y-axis direction, or another Euclidean distance.

[0116] In S1003, CPU 3 performs the rearrangement of the P image pairs in ascending order of the distances calculated in S1002, and terminates the subroutine for rearranging the P image pairs.

[0117] exist Figure 10B In S1004, CPU 3 performs the rearrangement of P-image pairs in ascending order of the deviations calculated in S813 for each P-image pair, and terminates the subroutine for rearranging the P-image pairs.

[0118] exist Figure 10C In S1001 to S1003, the processing and Figure 10A The processing is similar. In S1005, CPU 3 rearranges the P-image pairs of the predetermined rank or above in ascending order of the deviation of the P-image pairs rearranged in S1003, and ends the subroutine for rearranging the P-image pairs.

[0119] exist Figure 10D In S1001 and S1002, the processing and Figure 10A The processing is similar, and the processing in S1004 is the same as... Figure 10B The processing is similar. In S1006, CPU 3 rearranges P-image pairs of a predetermined rank or higher according to the ascending order of the distance between the P-image pairs rearranged in S1004, and ends the subroutine for rearranging the P-image pairs.

[0120] exist Figure 10E In S1001 and S1002, the processing and Figure 10A The processing is similar. In S1007, CPU 3 weights the data according to the ascending order of the distances calculated in S1002 and assigns a score to each P-image pair.

[0121] In S1008, CPU 3 weights the biases calculated in S813 for each P-image pair in ascending order and assigns a score to each P-image pair. The weighting method can be the same as in S1007.

[0122] In S1009, CPU 3 rearranges the P-image pairs in descending order of the total scores of distance and deviation, and terminates the subroutine for rearranging the P-image pairs.

[0123] Flowchart related to gaze detection processing in the presence of multiple P-image pairs.

[0124] Figure 11 This is a flowchart related to gaze detection processing in the case where there are multiple combinations (P image pairs) of corneal reflection images included in the eye image, according to this embodiment. The gaze detection processing can be performed, for example, when an object approaches the eyepiece lens 12. The object's approach to the eyepiece lens 12 can be sensed by any method in the prior art, such as using a proximity sensor disposed near the eyepiece lens 12. The gaze detection processing can be initiated in response to a command from the user via the operation unit 204. Figure 11 The processing is performed when CPU 3 controls the various units. (The last part, "omitted," appears to be a fragment and doesn't translate directly.) Figure 7 The flowchart describes a similar process.

[0125] In S1101, CPU 3 assigns the total number of P-image pairs used in the current gaze detection process as an initial value to variable Q (satisfying Q≥2), and assigns "0" to variable l. Here, the number of P-image pairs after the rearrangement order is the total number of P-image pairs.

[0126] In S1102, CPU 3 obtains the coordinates of the l-th P-image pair based on the rearranged order of the P-image pairs. The order of the P-image pairs used to obtain the coordinates is the rearranged order stored in memory unit 4 in S815.

[0127] In S1103, CPU 3 determines whether a predetermined time, set beforehand, has elapsed for the gaze detection process. If the predetermined time has elapsed, the process does not proceed to S1104 and ends the gaze detection process. Conversely, if the predetermined time has not elapsed, the process proceeds to S1104. This configuration ensures that the gaze detection process ends within the predetermined time and with minimal reduction in gaze detection accuracy.

[0128] In S1104, CPU 3 is set to l = l + 1.

[0129] In S1105, CPU 3 determines whether l>Q is true. If l>Q is true, the gaze detection process ends; otherwise, the process proceeds to S1102. If there are P-image pairs where rotation angle calculation failed during calibration, the number of rearranged P-image pairs is small. In this case, gaze detection processing for Q P-image pairs may be completed before the predetermined time has elapsed. In this situation, it is determined as "yes" in S1105.

[0130] This construction calculates the gaze position (Hx, Hy) on the display element 10 based on each of multiple P-image pairs. Then, the average value obtained by summing all these gaze positions and dividing by the number of P-image pairs is calculated as a single gaze position. Based on this gaze position, a pointer indicating the gaze position is displayed, or the position of the AF frame is determined. The average value of the gaze positions (Hx, Hy) calculated based on each of the multiple P-image pairs is used to calculate the single gaze position, but the median of the P-image pairs calculating the gaze positions (Hx, Hy) based on each of the multiple P-image pairs can also be used. Furthermore, among the gaze positions (Hx, Hy) calculated based on each of the multiple P-image pairs, the gaze position closest to the gaze position calculated in the previous frame can be used as the gaze position for pointer display or AF frame position determination.

[0131] Second Embodiment

[0132] According to the first embodiment, an example of applying the technology to a digital still camera 1 as an electronic device has been described; however, the technology can also be applied to an HMD. As a second embodiment, an HMD will be described.

[0133] Figure 12 This is a perspective view of an example appearance of a head-mounted display (HMD) of an electronic device according to a second embodiment. Figure 12 The HMD 1200 shown is a display device worn on a user's head. The HMD 1200 includes operating components such as a power switch or buttons for controlling device settings, and an image processing unit configured to function as an image processing device, generating virtual objects or synthesizing images, for example. The head-mounted component 1201 is configured to stably fix the HMD 1200 to the user's head, allowing the HMD 1200 to operate according to the movement of the user's head without shifting. It should be noted that while the head-mounted component has a configuration for being fixed to the user's head, it may also have a configuration for being hung around the user's ears or fixed by hand support.

[0134] Display device 1202 presents virtual objects or composite images of real and virtual objects to the user as a wearer. Display device 1202 can be, for example, a liquid crystal display (LCD) or an organic EL display. For example, display device 1202 comprises a liquid crystal panel, a driving circuit configured to drive the liquid crystal panel, and a memory configured to store the image to be displayed. Display device 1202 can be an opaque display unit or an optically transparent display unit that allows direct visual recognition of the external scenery.

[0135] The image capturing device 1203 is a device (camera unit) configured to capture images of the user's surrounding environment.

[0136] The image capturing device 1203 can capture images of the environment in front of the user as the user's surroundings. The area in front of the user can be the area in front of the user's head.

[0137] Figure 13 This is a block diagram illustrating an example of the system architecture of an HMD for an electronic device according to a second embodiment.

[0138] Figure 13 The HMD shown consists of an image capturing unit 1300, a gaze sensing unit 1301, a motion sensing unit 1302, a rotation sensing unit 1303, an information processing unit 1304, an image processing unit 1305, and a display unit 1306.

[0139] The information processing unit 1304 consists of an image acquisition unit 1307, a virtual object storage unit 1308, a virtual object generation unit 1309, a line-of-sight information acquisition unit 1310, an HMD movement calculation unit 1311, an HMD rotation calculation unit 1312, and an HMD operation determination unit 1313.

[0140] The image processing unit 1305 comprises a virtual object selection unit 1314, a virtual object control unit 1315, and a display image generation unit 1316. Each module will be described below. Although not shown in the figures, the image processing unit 1305 performs processes including preprocessing, color interpolation, correction, detection, and data processing. Preprocessing includes signal amplification, reference level adjustment, and bad pixel correction. Color interpolation is the process of interpolating the values ​​of color components not included in the image data, and is also known as de-mosaicing. Correction includes white balance adjustment, image brightness correction, optical aberration correction of the imaging lens (not shown), and color correction. Detection includes detection and tracking of feature regions (e.g., facial regions, human body regions, animals, cars, etc.) and person recognition. Furthermore, the image processing unit 1305 can obtain motion characteristic information of the subject by analyzing the differences in image data over time, such as how the subject moves within the image plane. Data processing includes scaling, encoding and decoding, and title information generation. It should be noted that these processing types are examples of image processing that can be implemented by the image processing unit 1305, and are not intended to limit the image processing implemented by the image processing unit 1305.

[0141] The image capturing unit 1300 comprises an optical system, an image sensor, a drive circuit configured to control the image sensor, an A / D conversion circuit configured to convert the signal acquired by the image sensor into a digital signal, and a developing circuit configured to develop the acquired signal into an image. Image data captured by the image capturing unit 1300 is acquired by the image acquisition unit 1307 of the information processing unit 1304 and transmitted to the display image generation unit 1316 of the image processing unit 1305. In the display image generation unit 1316, a virtual object compositing process (described below) is performed to generate a display image. Additionally, the image acquisition unit 1307 transmits the captured image to the virtual object generation unit 1309 to generate a virtual object.

[0142] The gaze sensing unit 1301 is a device configured to detect the direction of the gaze of a user as a wearer. For example, the gaze sensing unit 1301 may be a device configured to illuminate the user's eyeball with an infrared light-emitting diode used in conventional single-lens reflex cameras, and detect the direction of the gaze based on the relationship between a corneal reflection image (P-image) reflected by the cornea after passing through the light source and the pupil.

[0143] The gaze information sensed by the gaze sensing unit 1301 is obtained by the gaze information acquisition unit 1310 of the information processing unit 1304 and transmitted to the virtual object control unit 1315 of the image processing unit 1305.

[0144] In the virtual object control unit 1315, the virtual object described below is controlled and transmitted to the display image generation unit 1316 of the image processing unit 1305. In the display image generation unit 1316, captured images are used for composite processing to generate a display image.

[0145] The motion sensing unit 1302 is a device configured to detect HMD motion. The motion information sensed by the motion sensing unit 1302 is obtained by calculating the motion amount using the HMD motion amount calculation unit 1311 of the information processing unit 1304, and then transmitted to the HMD operation determination unit 1313. The motion sensing unit 1302 is an HMD motion detection unit and is configured to detect HMD motion using location information from a Global Positioning System (GPS) or a motion detection unit such as an accelerometer.

[0146] The rotation sensing unit 1303 is a device configured to detect the rotation of the HMD. The rotation information sensed by the rotation sensing unit 1303 is obtained by calculating the rotation amount by the HMD rotation amount calculation unit 1312 of the information processing unit 1304, and is transmitted to the HMD operation determination unit 1313. The rotation sensing unit 1303 is an HMD rotation detection unit and is configured to detect the rotation of the HMD using a rotation displacement sensor.

[0147] The HMD operation determination unit 1313 determines the operation of the HMD based on the motion direction and motion amount of the HMD according to the HMD motion information transmitted by the HMD motion amount calculation unit 1311, and the rotation direction and rotation amount of the HMD according to the HMD rotation amount calculation unit 1312. The operation information determined by the HMD operation determination unit 1313 is transmitted to the virtual object control unit 1315 of the image processing unit 1305.

[0148] The virtual object storage unit 1308 stores data about the virtual space, such as data related to virtual items constituting the virtual space (shape information, position, and pose information) or data related to light sources used for illumination in the virtual space. Then, the virtual object selection unit 1314 of the image processing unit 1305 selects a virtual object to be transmitted to the virtual object control unit 1315. Additionally, the virtual object storage unit 1308 transmits the virtual object to the virtual object generation unit 1309 to generate another virtual object.

[0149] The virtual object generation unit 1309 generates a virtual object based on image data transmitted from the image acquisition unit 1307 and virtual object data transmitted from the virtual object storage unit 1308. The generated virtual object is then transmitted to the virtual object selection unit 1314 of the image processing unit 1305.

[0150] The virtual object selection unit 1314 selects the virtual object to be displayed to the user from the virtual object storage unit 1308 and the virtual object generation unit 1309, and transmits the selected virtual object as the virtual object to be displayed to the virtual object control unit 1315. It should be noted that the virtual object data to be displayed includes display image data, arrangement position data in the display field of view, overlap order data with each virtual object, etc.

[0151] The virtual object control unit 1315 controls the virtual object data transmitted from the virtual object selection unit 1314 based on the gaze information transmitted from the gaze information acquisition unit 1310 and the HMD operation information transmitted from the HMD operation determination unit 1313. Then, the virtual object control unit 1315 transmits the virtual object data to the display image generation unit 1316.

[0152] The display image generation unit 1316 creates a composite image of the captured image data transmitted from the captured image acquisition unit 1307 and the virtual object transmitted from the virtual object control unit 1315, and transmits the composite image to the display unit 1306.

[0153] Other embodiments

[0154] The various types of control performed by CPU 3 described above can be performed by a single piece of hardware, or the control of the entire device can be performed simultaneously by multiple pieces of hardware (e.g., multiple processors or circuits) sharing the processing.

[0155] Furthermore, while this disclosure has been described in detail based on exemplary embodiments, some embodiments are not limited to these exemplary embodiments, and various modes are also included in some embodiments without departing from the spirit and scope of this disclosure. In addition, each of the above embodiments is merely an illustration of an exemplary embodiment of this disclosure, and appropriate combinations of the various embodiments are also possible.

[0156] According to this disclosure, when there are multiple combinations (P image pairs) of corneal reflection images contained in an eye image, the gaze detection accuracy is hardly reduced even when the gaze detection processing takes a predetermined time.

[0157] Embodiments of this disclosure can also be implemented by a computer that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (also more fully referred to as a 'non-transitory computer-readable storage medium') to perform the functions of one or more of the above embodiments and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing the functions of one or more of the above embodiments. Embodiments of this disclosure can also be implemented by a computer that, for example, reads and executes computer-executable instructions from a storage medium to perform the functions of one or more of the above embodiments and / or controls one or more circuits to perform the functions of one or more of the above embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessor unit (MPU)) and may include a network of separate computers or separate processors that read and execute computer-executable instructions. The computer-executable instructions may, for example, be provided to the computer from a network or a storage medium. The storage medium may include, for example, a hard disk, random access memory (RAM), read-only memory (ROM), storage devices for distributed computing systems, and optical discs (such as compressed optical discs (CDs), digital versatile optical discs (DVDs), or Blu-ray discs (BDs). TM One or more of the following: flash memory devices, memory cards, etc.

[0158] While this disclosure has described exemplary embodiments, it should be understood that some embodiments are not limited to those disclosed. The appended claims should be interpreted in the broadest possible sense to cover all such modifications and equivalent structures and functions.

Claims

1. An electronic device, the electronic device comprising: Multiple illumination units, the multiple illumination units being configured to illuminate the user's eyeball with infrared light; An image capturing unit is configured to capture an eye image when the infrared light from the plurality of illumination units is reflected by the eyeball; A first computing unit is configured to calculate the gaze position of a user viewing the display unit based on a combination of multiple corneal reflection images contained in the eye image. as well as A setting unit is configured to, during calibration processing for improving the calculation accuracy of the gaze position calculated by the first calculation unit, set the order of each combination of the plurality of corneal reflection images based on predetermined conditions, wherein... Within a predetermined time period, the first calculation unit processes the combination of the plurality of corneal reflection images based on the order set by the setting unit to calculate the gaze position of the user viewing the display unit.

2. The electronic device according to claim 1, wherein, During the calibration process, the setting unit processes the combination of the plurality of corneal reflection images in a predetermined order, and rearranges the order of the combination of the plurality of corneal reflection images from the predetermined order based on the result of the processing and the predetermined conditions.

3. The electronic device according to claim 1, further comprising: A second calculation unit is configured to repeatedly calculate the rotation angle of the eyeball based on the coordinates of the combinations of the plurality of corneal reflection images for each combination of the plurality of corneal reflection images, wherein For each combination of the plurality of corneal reflection images, the second calculation unit also calculates the average value of the rotation angle of the eyeball and the deviation of the rotation angle of the eyeball.

4. The electronic device according to claim 3, wherein, The predetermined condition is a condition based on the average value of the rotation angle of the eyeball or the deviation of the rotation angle of the eyeball calculated by the second calculation unit.

5. The electronic device according to claim 4, wherein, The predetermined condition is the distance between the gaze position estimated based on the average value of the eyeball rotation angle calculated by the second calculation unit and the indicator displayed on the display unit during the calibration.

6. The electronic device according to claim 4, wherein, The predetermined condition is the magnitude of the deviation of the rotation angle of the eyeball calculated by the second calculation unit.

7. A control method for an electronic device, the control method comprising: The shooting step involves capturing an image of the eyeball when infrared light from multiple illumination units is reflected by the user's eyeball, wherein the multiple illumination units are configured to illuminate the eyeball with the infrared light. The calculation step involves calculating the user's gaze position when viewing the display unit based on a combination of multiple corneal reflection images contained in the eyeball image. as well as In the setup step, during the calibration process used to improve the calculation accuracy of the gaze position, the order of each combination of the plurality of corneal reflection images is set based on predetermined conditions, wherein... In the calculation step, within a predetermined time period, the combination of the plurality of corneal reflection images is processed based on the order set in the setting step to calculate the gaze position of the user viewing the display unit.

8. A program for causing a computer to perform a control method for an electronic device, the control method comprising: The shooting step involves capturing an image of the eyeball when infrared light from multiple illumination units is reflected by the user's eyeball, wherein the multiple illumination units are configured to illuminate the eyeball with the infrared light. The calculation step involves calculating the user's gaze position when viewing the display unit based on a combination of multiple corneal reflection images contained in the eyeball image. as well as In the setup step, during the calibration process used to improve the calculation accuracy of the gaze position, the order of each combination of the plurality of corneal reflection images is set based on predetermined conditions, wherein... In the calculation step, within a predetermined time period, the combination of the plurality of corneal reflection images is processed based on the order set in the setting step to calculate the gaze position of the user viewing the display unit.

9. A non-transitory computer-readable medium storing a program for causing a computer to perform a control method for an electronic device, the control method comprising: The shooting step involves capturing an image of the eyeball when infrared light from multiple illumination units is reflected by the user's eyeball, wherein the multiple illumination units are configured to illuminate the eyeball with the infrared light. The calculation step involves calculating the user's gaze position when viewing the display unit based on a combination of multiple corneal reflection images contained in the eyeball image. as well as In the setup step, during the calibration process used to improve the calculation accuracy of the gaze position, the order of each combination of the plurality of corneal reflection images is set based on predetermined conditions, wherein... In the calculation step, within a predetermined time period, the combination of the plurality of corneal reflection images is processed based on the order set in the setting step to calculate the gaze position of the user viewing the display unit.

Images