Control device, control method of control device, control method, and storage medium

Abstract

The present invention relates to a control device, a control method of a control device, a control method, and a storage medium. A control device capable of quickly and easily reducing a deviation in a position of a point of gaze that occurs each time an eye of a photographer comes into contact with a viewfinder. A position corresponding to a point of gaze of a user is estimated and corrected based on a position of an indicator displayed on a display portion and a position associated with the position of the indicator corresponding to the point of gaze of the user. In a first mode, the correction is made based on a first number of positions of indicators, and in a second mode, the correction is made based on a second number of indicators that is less than the first number.

Description

Technical Field

[0001] This invention relates to a control device for reducing positional deviation of the gaze point, a control device, a control method for the control device, a control method, and a storage medium. Background Technology

[0002] In recent years, camera operation has become automated, and cameras have become intelligent. Japanese Patent Application Publication No. 2004-8323 discloses a camera control device that can identify the subject the photographer intends to photograph and control focus based on information related to the photographer's gaze position in the viewfinder, without requiring manual input of the subject position. When the camera detects the photographer's gaze point position (gaze position) as described above, the photographer's intended gaze point position and the photographer's gaze position identified by the camera sometimes deviate from each other, and in such cases, it is impossible to focus on the photographer's intended subject. To solve this problem, the device described in Japanese Patent Application Publication No. 2004-8323 displays indicators in the viewfinder before shooting and instructs the photographer to look at the displayed indicators. While the photographer is looking at the indicators, the photographer's gaze point position is detected, and the amount of deviation from the position of each indicator is detected. Then, when shooting, the detected photographer's gaze point position identified by the camera is corrected (calibrated) using this deviation amount, thereby detecting the corrected position as the photographer's intended gaze point position. Furthermore, in the device described in Japanese Patent Application Publication No. 2020-106552, the position of the gaze point can be changed so that the photographer can adjust it on the display device.

[0003] In the device described in Japanese Patent Application Publication No. 2004-8323, correction is performed via the aforementioned calibration when a positional deviation occurs between the photographer's intended gaze point position and the gaze point position detected by the camera. However, each time the photographer's eye is brought into contact with the camera's viewfinder, the eye contact position changes slightly, and therefore the gaze point position detected by the camera also changes according to the eye contact position. In this case, calibration is frequently required whenever the photographer's eye is brought into contact with the viewfinder, resulting in a cumbersome calibration process. Furthermore, in the device described in Japanese Patent Application Publication No. 2020-106552, the gaze point position during calibration is preferably fixed rather than variable, taking into account the accuracy of the correction performed via calibration. Summary of the Invention

[0004] The present invention provides a control device, a control method for the control device, a control method, and a storage medium that can quickly and easily reduce the deviation of the gaze point position whenever the photographer's eye comes into contact with the viewfinder and improve the convenience of shooting.

[0005] In a first aspect of the invention, a control device is provided, comprising: at least one processor; and a memory coupled to the at least one processor, the memory having instructions that, when executed by the processor, operate as: an estimation unit configured to estimate a position corresponding to a user's gaze point; and a correction unit configured to correct the position corresponding to the user's gaze point based on the position of an indicator displayed on a display unit and a position corresponding to and associated with the user's gaze point, wherein the correction unit has a first mode and a second mode, the first mode being used to perform the correction based on the positions of a first number of indicators, and the second mode being used to perform the correction based on a second number of indicators less than the first number.

[0006] In a second aspect of the invention, a control device is provided, comprising: at least one processor; and a memory coupled to the at least one processor, the memory having instructions that, when executed by the processor, operate as a unit: a control unit configured to control a display performed by a display unit, the display unit displaying indicators for calibration of a user's gaze point detection, and wherein the control unit causes the display unit to display a first number of indicators in a first calibration, and to display a second number of indicators less than the first number in a second calibration performed after the first calibration.

[0007] In a third aspect of the invention, a control method for a control device is provided, comprising: estimating a position corresponding to a user's gaze point; and correcting the position corresponding to the user's gaze point based on the position of an indicator displayed on a display unit and a position corresponding to and associated with the user's gaze point, wherein the correction is performed in at least one of a first mode and a second mode, the first mode being used to perform the correction based on the positions of a first number of indicators, and the second mode being used to perform the correction based on a second number of indicators less than the first number.

[0008] In a fourth aspect of the invention, a control method is provided, comprising: controlling a display performed by a display unit, the display unit displaying indicators for calibration of a user's gaze point detection, wherein the control comprises: causing the display unit to display a first number of indicators in a first calibration, and to display a second number of indicators less than the first number in a second calibration performed after the first calibration.

[0009] According to the present invention, the deviation of the gaze point that occurs whenever the photographer's eye comes into contact with the viewfinder can be quickly and easily reduced, thereby improving the convenience of shooting.

[0010] Further features of the invention will become apparent from the following description of exemplary embodiments (with reference to the accompanying drawings). Attached Figure Description

[0011] Figures 1A to 1C This is a schematic diagram used to illustrate the appearance of a digital still camera.

[0012] Figure 2 This is a schematic diagram of the interior of a digital still camera.

[0013] Figure 3 This is a block diagram of a digital still camera.

[0014] Figures 4A to 4C Each diagram shows the viewfinder's field of view (the image display status of the display device).

[0015] Figure 5 This is a schematic diagram illustrating the state of the eyeball being illuminated by infrared light from a light source.

[0016] Figure 6 It is used to explain in Figure 5 The diagram illustrates the principle of detecting the line-of-sight position (line of sight) under the shown conditions.

[0017] Figure 7 This is a flowchart of the line-of-sight detection process.

[0018] Figure 8A and Figure 8B Each diagram shows the viewfinder's field of view (the image display status of the display device).

[0019] Figure 9 This is a flowchart of the process for switching between the first correction and the second correction in the first embodiment.

[0020] Figure 10 This is a flowchart of the process for switching between the first correction and the second correction in the second embodiment.

[0021] Figure 11 This is a flowchart of the process for switching between the first correction and the second correction in the third embodiment.

[0022] Figure 12 This is a flowchart of the process for switching between the first correction and the second correction in the fourth embodiment. Detailed Implementation

[0023] The invention will now be described in detail below with reference to the accompanying drawings, which illustrate embodiments of the invention. However, the configurations in the following embodiments are described by way of example only and are in no way intended to limit the scope of the invention.

[0024] First, refer to Figures 1A to 9 The first embodiment is described.

[0025] Figures 1A to 1C These are schematic diagrams showing the appearance of digital still cameras. Figure 1A and Figure 1B These are the front and rear stereoscopic images from a digital still camera. Figure 1C It is a set of stereoscopic images of the joysticks in a digital still camera in various operating states. For example... Figure 1A As shown, the digital still camera (hereinafter referred to as "camera 1"), indicated by reference numeral 1, is a control device having a shooting lens 1A and a camera body 1B. A release button 5 is arranged on the upper side of the camera body 1B for receiving shooting operations from the user (the photographer). Furthermore, as... Figure 1B As shown, an eyepiece lens 12 and operation components 41 to 43 are provided on the rear side of the camera body 1B. The user uses the eyepiece lens 12 to observe the display device arranged inside the camera 1, and the operation components 41 to 43 are used to operate the camera 1. The operation component 41 is a touch panel liquid crystal display (operation component α), the operation component 42 is a lever-type operation component (operation component β), and the operation component 43 is a button cross key (operation component γ).

[0026] Figure 2 This is a schematic diagram of the internal structure of a digital still camera. (For example...) Figure 2 As shown, the imaging lens 1A has lenses 101 and 102. Note that although the number of lenses is two in this embodiment, this is not limiting, and for example, there could be three or more. Furthermore, the imaging lens 1A includes an aperture stop 111, an aperture drive unit 112, a lens drive motor 113, a lens drive member 114, an optocoupler 115, a pulse plate 116, a mounting contact 117, and a lens focus adjustment circuit 118. The lens drive member 114 includes a drive gear. The optocoupler 115 detects the rotation of the pulse plate 116 according to the rotation of the lens drive member 114 and notifies the lens focus adjustment circuit 118 of the rotation information. Based on this rotation information and information received from the camera body side regarding the amount of lens drive, the lens focus adjustment circuit 118 causes the lens drive motor 113 to drive the lens 101 by a predetermined amount. In this way, the lens 101 can be moved to a focused position. The mounting contact 117 is an interface with the camera body.

[0027] The camera body 1B includes an image sensor (imaging unit) 2, a CPU 3, a memory unit 4, a display device (display unit) 10, a display device drive circuit 11, and an eyepiece lens 12. Furthermore, the camera body 1B also includes a light source (illumination source) 13a, a light source (illumination source) 13b, a beam splitter 15, a light-receiving lens 16, and an eye-mounted image sensor (eye-mounted imaging unit) 17. The image sensor 2 is positioned on the intended imaging plane of the camera lens 1A and is capable of capturing images of the subject. The CPU 3 is a central processing unit (controller) of a microcomputer integrated into the camera body and serves as a control unit configured to control the overall operation of the camera (control processing steps, control methods). Furthermore, programs for causing the CPU 3 to execute the operations of the components and units of the camera 1 (control methods for the control device) are pre-stored in a memory (not shown) separate from the memory unit 4. The memory unit 4 records images captured by the image sensor 2. The display device 10 includes a liquid crystal layer forming a display screen with a rectangular shape, and is capable of displaying an image of a subject, for example, captured by the image sensor 2, according to drive control from the display device drive circuit 11. The eyepiece lens 12 is the part of the eyepiece that the user's eye contacts when the user observes (visually recognizes) the image of the subject displayed on the display device 10. Light sources 13a and 13b are configured to illuminate the user's eyeball 14 to detect the direction of gaze based on the relationship between the eyeball image and the image reflected from the cornea, and are implemented, for example, by infrared light-emitting diodes. Light sources 13a and 13b are arranged around the eyepiece lens 12. Light from the eyeball 14 and light from the cornea are transmitted through the eyepiece lens 12, reflected by the beam splitter 15, and formed as an eyeball image and a reflected image on the eyeball image sensor 17 by the light-receiving lens 16, respectively. The light-receiving lens 16 sets the pupil of the user's eyeball 14 and the eyeball image sensor 17 in a conjugate imaging relationship. The eye image sensor 17 is a device formed by photoelectric elements (such as a CCD) arranged in a two-dimensional (matrix) array, and it captures images of the user's eyeball 14. In the camera 1, the gaze direction can be detected based on the positional relationship between the eyeball 14 imaged on the eye image sensor 17 and the reflected image, using a predetermined algorithm described below. Note that the memory unit 4 has the function of storing image signals output from the image sensor 2 and the eye image sensor 17, as well as the function of storing gaze correction data (individual differences) for correcting inter-individual gaze differences, as described below.

[0028] Figure 3 This is a block diagram of a digital still camera. (Example) Figure 3As shown, the gaze detection circuit 201, metering circuit 202, autofocus detection circuit 203, signal input circuit 204, light source driving circuit 205, and display device driving circuit 11 are electrically connected to the CPU 3. The gaze detection circuit 201 converts the output from the eye image sensor 17, which performs eye imaging, from analog to digital and sends the image information of the output to the CPU 3. The CPU 3 extracts feature points of the eye image required for gaze detection according to a predetermined algorithm described below and calculates the user's gaze based on the position of the feature points. Based on the signal obtained from the image sensor 2, which also functions as a metering sensor, the metering circuit 202 amplifies the luminance signal corresponding to the brightness of the field of view, then performs logarithmic compression and analog-to-digital conversion on the amplified luminance signal, and sends the resulting signal as field of view luminance information to the CPU 3. The autofocus detection circuit 203 converts the voltage signals from multiple pixels included in the image sensor 2 (CCD) and used for phase difference detection from analog to digital and sends the digital signals to the CPU 3. The CPU 3 calculates the distance to the subject associated with the focus detection point based on the signals obtained from the multiple pixels. This is a technique known as phase-difference autofocus (AF). In this embodiment, as shown in FIG4, it is assumed, for example, that there are 180 focus detection points. Switches SW1 (not shown) and SW2 (not shown) are connected to signal input circuit 204, wherein switch SW1 is turned on by the first stroke of release button 5, and switch SW2 is turned on by the second stroke of release button 5. When switch SW1 is turned on, the camera's metering, rangefinding, and line-of-sight detection operations begin. When switch SW2 is turned on, the release operation begins. More specifically, the signal generated by operating release button 5 is sent to CPU 3 via signal input circuit 204. Light source drive circuit 205 controls the driving (i.e., ON / OFF) of light sources 13a and 13b. In addition, signals are sent between the lens focus adjustment circuit 118 arranged in the shooting lens 1A via mounting contact 117 and the aperture control circuit 206 included in aperture drive circuit 112 and the camera body.

[0029] Figures 4A to 4C These are diagrams showing the viewfinder's field of view (the image display status of the display device). For example... Figure 4AAs shown, within the viewfinder's field of view, the view mask 300, focus detection area 400, and 180 ranging point indicators 4001 to 4180, displayed in a state superimposed on the through image displayed on the display device 10, can be observed. The ranging point indicators 4001 to 4180 are arranged at positions corresponding to the plurality of focus detection points on the imaging surface of the image sensor 2. Furthermore, the indicator among the ranging point indicators 4001 to 4180 corresponding to the current estimated gaze point position is displayed as estimated gaze point A. Although in this embodiment, estimated gaze point A is represented by a box shape, this is not limiting; for example, estimated gaze point A could be represented by a dot shape.

[0030] Figure 5 This is a diagram showing the state of the eyeball being illuminated by infrared light from a light source. Figure 6 It is used to explain in Figure 5 The diagram illustrates the principle of detecting the line-of-sight position (line of sight) under the shown conditions. As... Figure 6 The upper half Figure 6 (a) schematically shows an image of the eye projected onto an image sensor, and as Figure 6 The lower half Figure 6 (b) shows the brightness distribution of the eye image. Furthermore, in Figure 5 and Figure 6 In the figures, the X-axis is horizontal, the Y-axis is vertical, and the Z-axis is orthogonal to the X-axis and Y-axis (the optical axis of the light-receiving lens 16).

[0031] like Figure 5 As shown, light sources 13a and 13b are arranged linearly symmetrically with respect to the optical axis of the light-receiving lens 16, which serves as the axis of symmetry. Note that this arrangement is shown differently from that shown in FIG. 1 to facilitate understanding of the principle of detecting the position of the gaze. Infrared light IF beams are emitted from light sources 13a and 13b toward the user's eyeball 14, respectively. A portion of the infrared light IF beam reflected from the cornea 142 of the eyeball 14 is then focused by the light-receiving lens 16 to form an image on the ocular image sensor 17. More specifically, a portion of the infrared light IF beam reflected from points Pd and Pe on the surface of the cornea 142 is focused by the light-receiving lens 16 to form images (corneal reflection images) Pd' and Pe', respectively, on the ocular image sensor 17. Similarly, the light beams reflected from the ends (pupil ends) 141a and 141b of the pupil 141 also form images 141a' and 141b' on the ocular image sensor 17, respectively.

[0032] like Figure 6As shown in (a) and (b), images Pd', Pe', 141a', and 141b' are included in region α. ​​The coordinates of image Pd' in the X-axis direction are defined as Xd, and the coordinates of image Pe' in the X-axis direction are defined as Xe. Furthermore, the coordinates of image 141a' in the X-axis direction are defined as Xa, and the coordinates of image 141b' in the X-axis direction are defined as Xb. Figure 6 As shown in (b), very high levels of brightness are detected at coordinates Xd and Xe. Furthermore, in the region between coordinates Xa and Xb corresponding to the region of pupil 141, very low levels of brightness are detected, except for the brightness values ​​at coordinates Xd and Xe. Additionally, intermediate levels of brightness are detected in the region corresponding to the region of iris 143 outside pupil 141, in the X-axis direction less than coordinate Xa and in the X-axis direction greater than coordinate Xb. Therefore, coordinates Xd, Xe, Xa, and Xb can be determined based on the brightness levels. Furthermore, the coordinate of the point on the eye image sensor 17 corresponding to the pupil center 141c in the X-axis direction is defined as Xc. When the rotation angle θx of the optical axis of eyeball 14 relative to the optical axis of light-receiving lens 16 is small, coordinate Xc can be expressed as Xc ≈ (Xa + Xb) / 2. Thus, coordinates Xc, Xd, and Xe can be determined.

[0033] Figure 7 This is a flowchart of the line-of-sight detection process.

[0034] In step S001, the CPU 3 operates the light source driving circuit 205 to emit infrared light from the light sources 13a and 13b toward the user's eyeball 14. Thus, an image of the user's eyeball 14 is formed on the eye image sensor 17 via the light-receiving lens 16. Then, the eye image sensor 17 performs photoelectric conversion, thereby enabling the image of the eyeball 14 to be processed into an electrical signal (eye image signal). In step S002, the gaze detection circuit 201 sends the eye image signal obtained by the eye image sensor 17 to the CPU 3. In step S003, the CPU 3 determines the coordinates Xc, Xd, and Xe based on the eye image signal received in step S002 as described above. In step S004, the imaging magnification β of the eye image is calculated. The imaging magnification β is determined based on the position of the eyeball 14 relative to the light-receiving lens 16, and is calculated essentially as a function of the distance (Xd-Xe) between images Pd' and Pe'.

[0035] In step S005, CPU 3 calculates the rotation angle θx of eyeball 14 about the Y-axis and the rotation angle θy of eyeball 14 about the X-axis. Here, the X-coordinate of the midpoint between image Pd' and image Pe' and the X-coordinate of the curvature center O of cornea 142 are substantially the same. Therefore, assuming that the general distance between the curvature center O of cornea 142 and the center 141c of pupil 141 is represented by Oc, the rotation angle θx can be calculated by the following expression (1). Note that the rotation angle θy can be calculated similarly.

[0036] β×Oc×SINθ x ≈((Xd+Xe) / 2)–Xc…(1)

[0037] In step S006, the CPU 3 reads the gaze correction coefficients Ax, Bx, Ay, and By stored in the memory unit 4. The gaze correction coefficients Ax to By are coefficients used to correct individual differences in the user's gaze, and are determined through calibration (correction) as described below.

[0038] In step S007, CPU 3 uses rotation angles θx and θy to estimate (calculate) the position of the user's gaze on display device 10, i.e., the position of the point the user is looking at (hereinafter referred to as the "gaze point"). Assuming the gaze point is represented by coordinates (Hx, Hy) corresponding to the center 141c of the pupil 141 on display device 10, coordinate Hx is calculated using the following equation (2), and coordinate Hy is calculated using the following equation (3):

[0039] Hx = m × (Ax × θx + Bx) … (2)

[0040] Hy = m × (Ay × θy + By) … (3)

[0041] Note that coefficient m is a constant determined by the configuration of the viewfinder optical system of camera 1, and is a conversion coefficient used to convert rotation angles θx and θy into position coordinates corresponding to the center 141c of the pupil 141 on display device 10. Coefficient m is pre-stored in memory unit 4. Furthermore, as described above, gaze correction coefficients Ax, Bx, Ay, and By are used to correct individual differences in the user's gaze, and are determined through calibration. It is assumed that gaze correction coefficients Ax to By are stored in memory unit 4 before the gaze detection process begins. In step S008, CPU 3 stores the coordinates (Hx, Hy) in memory unit 4.

[0042] As described above, in this embodiment, CPU 3 serves as a gaze point position estimation unit (estimation unit), which is configured to estimate the position corresponding to the user's gaze point on the display device 10 based on an image of the eye 14 captured by the eye image sensor 17 (estimation step). Note that in the camera 1, the CPU used as the gaze point position estimation unit can be set separately from CPU 3.

[0043] As described above, in the gaze detection processing (gaze detection routine), the rotation angles θx and θy of the eyeball 14 are determined based on the eyeball image signal, and the gaze point position is estimated by performing calculations to convert the position of the pupil center 141c into coordinates of the corresponding position on the display device 10. Furthermore, the shape of the eyeball 14 varies from person to person, and for example, in the presence of such factors, unless the values ​​of the gaze correction coefficients Ax, Bx, Ay, and By are adjusted to appropriate values ​​for each user, issues may arise. Figure 4B The possibilities of the states shown. Figure 4B In the state shown, the estimated gaze point C deviates from the user's actual gaze point (actual gaze point) B. In this situation, although the user expects to be looking at a person at gaze point B, camera 1 incorrectly estimates that the user is looking at the background and cannot perform proper focus detection and adequate adjustment. To avoid this, it is preferable to perform calibration to determine appropriate values ​​for the correction coefficients before camera 1 takes a picture.

[0044] During calibration, firstly, a plurality (a first number) of first indicators that the user wants to visually identify are displayed on the display device 10. Then, gaze estimation is performed each time the user visually identifies (gazes at) a first indicator. The CPU 3 determines gaze correction coefficients Ax to By based on the positional relationship between each first indicator on the display device 10 and the estimated gaze point estimated when the user visually identifies each first indicator. For example, as Figure 4CAs shown, a first indicator (c)-L, a first indicator (c)-R, a first indicator (c)-T, and a first indicator (c)-B are displayed on the display device 10. The first indicator (c)-L is positioned in the center of the vertical direction (up-down direction) towards the left edge of the display device 10 (display screen). The first indicator (c)-R is positioned in the center of the vertical direction towards the right edge of the display device 10. The first indicator (c)-T is positioned in the center of the horizontal direction (left-right direction) towards the upper edge of the display device 10. The first indicator (c)-B is positioned in the center of the horizontal direction towards the lower edge of the display device 10. When a user visually recognizes the first indicator (c)-L, an estimated gaze point corresponding to the first indicator (c)-L is detected; and when a user visually recognizes the first indicator (c)-R, an estimated gaze point corresponding to the first indicator (c)-R is detected. Then, based on the results of these tests, the line-of-sight correction coefficients Ax and Bx are calculated.

[0045] More specifically, the X-coordinate of the first indicator (c)-L on the display device 10 is represented by Hx1, the X-coordinate of the first indicator (c)-R is represented by Hx2, the rotation angle of the eyeball 14 when looking at the first indicator (c)-L is represented by θx1, and the rotation angle of the eyeball 14 when looking at the first indicator (c)-R is represented by θx2. Note that preferably, the rotation angles θx1 and θx2 are each values ​​that take into account deviations and noise, such as the average or median of multiple determined values ​​of associated rotation angles. The value Hx1 is calculated by the following equation (4), and the value Hx2 is calculated by the following equation (5):

[0046] Hx1 = m × (Ax × θx1 + Bx) … (4)

[0047] Hx2 = m × (Ax × θx2 + Bx) … (5)

[0048] The gaze correction coefficients Ax and Bx can be determined by equations (4) and (5). Similarly, the gaze correction coefficients Ay and By can also be determined using the first indicator (c)-T and the first indicator (c)-B. The gaze correction coefficients Ax to By are stored as first correction information in the memory unit 4. In the camera 1, by performing a first calibration using the first correction information (i.e., first calibration (calibration in the first mode)), the user's actual gaze point B and the estimated gaze point C are made consistent with each other. That is, the user's gaze point position on the display device 10 can be accurately estimated. Note that although the number of first indicators displayed in this embodiment is set to 4, this is not limiting; for example, it is preferable to set the number of first indicators to 2 to 8, and more preferably to set the number of first indicators to 4 to 5. In this way, the gaze correction coefficients, which are the first correction information, can be accurately determined. Although a frame shape is used as the shape of the first indicator in this embodiment, this is not limiting; for example, a dot shape or the like can be used. Furthermore, in this embodiment, CPU 3 also serves as a correction unit configured to correct the position on the display device 10 corresponding to the gaze point. This correction is performed based on the position of the indicator displayed on the display device 10 and the positions corresponding to and associated with these positions of the indicator and the user's gaze point. Note that in camera 1, the CPU used as the correction unit may be separate from CPU 3.

[0049] As described above, by performing a first correction using the first correction information, the user's gaze point position on the display device 10 can be accurately estimated. However, when the user's eye contact position relative to the eyepiece lens 12 changes, for example due to changes in the user's posture, the first correction information cannot always be used to detect the changed gaze. For example, consider the following situation: as an eye contact position, there exists a first eye contact position and a second eye contact position different from the first eye contact position. At the first eye contact position, when the first correction information is determined (performing a first correction using the first correction information), the user's eye is brought into contact with the eyepiece lens 12. The first correction information is determined at the first eye contact position, and therefore, if the first correction information is used as is for gaze detection at the second eye contact position, there is a possibility that the gaze may be difficult to detect accurately. Therefore, unless the shooting is interrupted and calibration is performed on each eye contact position to determine the correction information whenever the eye contact position changes, it is assumed that the estimation accuracy of the gaze point position is insufficient. In this case, frequent calibration (hereinafter referred to as "normal calibration") makes the control of the CPU 3 cumbersome. Given this inconvenience, camera 1 is configured to perform calibrations that reduce the hassle of control (hereinafter referred to as "eye-based calibration"). The configuration and operation of eye-based calibration will be described below.

[0050] In camera 1, contact between the eye and eyepiece lens 12 is detected based on whether the eye image sensor 17 captures an image of the eye. Therefore, the eye image sensor 17 also functions as an eye contact detection unit configured to detect contact between the eye and eyepiece lens 12. Then, if contact between the eye and eyepiece lens 12 is detected, CPU 3 performs correction by performing eye contact-based calibration (second correction using second correction information as described below). Furthermore, in camera 1, the user's eye contact position relative to eyepiece lens 12 can also be detected based on the position of the eye image captured by the eye image sensor 17. Therefore, the eye image sensor 17 also functions as an eye contact position detection unit capable of detecting a first eye contact position and a second eye contact position. Then, if a second eye contact position different from the first eye contact position is detected, CPU 3 performs correction by performing eye contact-based calibration.

[0051] In order to consider the relationship between the change in fixation point position due to the change in eye contact position and the method for calculating fixation point position described above, equation (2) is transformed into the following equation (6), and equation (3) is transformed into the following equation (7):

[0052] Hx / m = Ax × θx + Bx … (6)

[0053] Hy / m = Ay × θy + By … (7)

[0054] Here, Hx / m clearly represents the corrected rotation angle θx, and is calculated as a linear function of the detected rotation angle θx as a variable, the gaze correction coefficient Ax as the tilt, and the gaze correction coefficient Bx as the intercept. Furthermore, since the gaze correction coefficient Ax is the tilt, it can be calculated using the following equation (8):

[0055] Ax = δ(Hx / m) / δθx … (8)

[0056] It is clear from equations (6) and (8) that the line-of-sight correction coefficient Ax is a correction coefficient for the displacement (unit change) of the detected rotation angle θx, and the line-of-sight correction coefficient Bx is a correction coefficient for the relative position of the detected rotation angle θx. Furthermore, the line-of-sight correction coefficients Ay and By can be considered similarly.

[0057] Incidentally, the change in fixation point position caused by a change in eye contact position refers to the relative change in the detected rotation angles θx and θy. When considering this in conjunction with the above meanings of the various gaze correction coefficients, it is only necessary to determine (calculate) the gaze correction coefficients Bx and By based on eye contact calibration. That is, even when the eye contact position changes, the rotation angles θx and θy change almost nothing (negligible), so although it is not necessary to recalculate the gaze correction coefficients Ax and Ay, it is only necessary to adjust for the change in eye contact position using the gaze correction coefficients Bx and By. Therefore, it is only necessary to determine the gaze correction coefficients Bx and By based on eye contact calibration. Thus, in eye contact-based calibration, for gaze correction coefficients Ax and Ay, the gaze correction coefficients determined through normal calibration can be used as is, and for gaze correction coefficients Bx and By, newly determined gaze correction coefficients can be used.

[0058] As mentioned above, in eye-based calibration, since only the line-of-sight correction factors Bx and By are calculated, it is preferable to perform a normal calibration beforehand. Here, a description of an example of performing eye-based calibration after performing the normal calibration will be given. Figure 8A and Figure 8B Each diagram shows the viewfinder's field of view (the image display status of the display device).

[0059] In eye-based calibration, a first indicator and a second indicator that the user wants to visually identify are displayed on the display device 10. The number of second indicators to be displayed (the second number) can be less than the number of first indicators to be displayed (the first number). For example, as Figure 8A As shown, a second indicator (a)-C is displayed on the display device 10. The second indicator (a)-C is arranged in the center of the display device 10 (display screen). Then, when the user visually recognizes the second indicator (a)-C, the estimated gaze point corresponding to the second indicator (a)-C is detected. In this detection, although it is preferable to substitute the value "0" into the gaze correction coefficients Bx and By without using the value stored in the memory unit 4, this is not limiting. For example, values ​​determined by combining the gaze correction coefficients calculated in the eye-based calibration described below with the values ​​stored in the memory unit 4 can be used. Then, the gaze correction coefficients Bx and By are determined according to the position (coordinates) of the estimated gaze point corresponding to the second indicator (a)-C.

[0060] More specifically, suppose the coordinates of the second indicator (a)-C on the display device 10 are represented by (Hx3, Hy3), and the rotation angle of the eyeball 14 when looking at the second indicator (a)-C is represented by θx3 and θy3. Note that preferably, the rotation angles θx3 and θy3 are each values ​​that take into account bias and noise, such as the average or median of multiple determined values ​​of associated rotation angles. The value Hx3 is calculated by the following equation (9), and the value Hy3 is calculated by the following equation (10):

[0061] Hx3 = m × (Ax × θx3 + Bx) … (9)

[0062] Hy3 = m × (Ay × θy3 + By) … (10)

[0063] Since the gaze correction coefficients Ax and Ay have already been calculated through normal calibration, the gaze correction coefficients Bx and By can be calculated using equations (9) and (10). The gaze correction coefficients Bx and By are stored as second correction information in the memory unit 4. In the camera 1, even when the eye position changes, the user's gaze point position on the display device 10 (display unit) can be accurately estimated by performing a second calibration using the second correction information (i.e., the second calibration (in the second mode)) for each eye position. Note that in the camera 1, the first calibration and the second calibration are performed in the order of the first calibration and the second calibration. Furthermore, the number of gaze correction coefficients to be calculated in the eye-based calibration is less than the number of gaze correction coefficients to be calculated in the normal calibration, which reduces the control complexity of the CPU 3.

[0064] As described above, the second correction information is formed by reflecting the information (gazing correction coefficient Bx and gazing correction coefficient By) determined based on the positional relationship between the second indicator on the display device 10 and the gaze point estimated by the CPU 3 when visually recognizing the second indicator onto the first correction information. Furthermore, in the camera 1, when correcting the gaze point position, it is possible to choose between performing a first correction using the first correction information or a second correction using the second correction information. This allows for a quick and easy reduction of gaze point position deviation caused by eye contact. Moreover, the selection of whether to perform the first correction or the second correction using the second correction information is made by the CPU 3, rather than through manual operation by the user. This improves the convenience of shooting.

[0065] Note that although the number of second indicators to be displayed in this embodiment is 1, it is only required that the number of second indicators is less than the number of first indicators to be displayed, and for example, the number of second indicators is preferably 1 to 4, and more preferably 1 or 2. This allows for the accurate determination of each viewing correction coefficient, which serves as the second correction information. Furthermore, when the number of second indicators to be displayed is multiple, similar to the case where the number of second indicators to be displayed is 1, the second indicators are preferably located at a position biased towards the center portion of the display device 10.

[0066] like Figure 8B As shown, a second indicator (b)-C for eye-based calibration is displayed on the display device 10 as a status superimposed on the through image output from the image sensor 2. This allows eye-based calibration to be performed without stopping or interrupting shooting, even while shooting is in progress, improving shooting convenience. Furthermore, in the camera 1, a quick function to indicate the start / stop of eye-based calibration can be assigned to, for example, the operation member 42 (see...). Figure 1C In this case, for example, when the operating member 42 tilts to the left (see...). Figure 1C (The middle diagram in the image) begins the calibration based on the eyepiece, and when the operating member 42 tilts to the right (see...) Figure 1C (See the right image in the image), interrupting eye-based calibration. This allows for quick initiation or interruption of eye-based calibration as needed. Furthermore, eye-based calibration can also be interrupted by operating the release button 5. Thus, if a shutter release opportunity arises while eye-based calibration is being performed during shooting, pressing the release button 5 can quickly interrupt the calibration and initiate the release operation.

[0067] Figure 9 This is a flowchart of the process for switching between the first and second corrections in the first embodiment. When the camera 1 is powered on and the process begins, in step S101, the CPU 3 causes the image sensor 2 to begin acquiring a live view image, and sends the image signal of the live view image to the CPU 3. Furthermore, the CPU 3 displays the live view image received from the image sensor 2 on the display device 10. The user can visually identify the subject in the live view image displayed on the display device 10. In step S102, the CPU 3 determines whether the power is off. If the CPU 3 determines in step S102 that the power is off, the process terminates immediately. On the other hand, if the CPU 3 determines in step S102 that the power is not off, the process proceeds to step S103. In step S103, the CPU 3 causes the eye to operate using the image sensor 17 to acquire an image of the user's eye that has begun visually identifying the subject in step S101, and performs the aforementioned gaze detection process. In this embodiment, through... Figure 7 The gaze detection process in step S103 determines the coordinates (Hx, Hy) of the estimated gaze point on the display device 10. In step S104, the CPU 3 causes the display device 10 to operate to display an image generated by superimposing the estimated gaze point at the coordinates (Hx, Hy) determined in step S103 onto the live view image. At this time, it is assumed that the user is visually recognizing within the viewfinder, for example, at... Figure 4B The image showing the state is then displayed. The camera 1 then notifies the user of the currently estimated gaze point C, which is overlaid on the live view image. In step S105, the CPU 3 waits for a predetermined time period.

[0068] In step S106, the CPU 3 determines whether it has received an instruction from the user to start eye-based calibration. If the CPU 3 determines in step S106 that it has received an instruction from the user to start eye-based calibration, the process proceeds to step S107. On the other hand, if the CPU 3 determines in step S106 that it has not yet received an instruction from the user to start eye-based calibration, the process proceeds to step S200. As described above, the instruction to start eye-based calibration can be provided by operating the operation member 42. In step S107, the CPU 3 determines whether normal calibration has been performed. If the CPU 3 determines in step S107 that normal calibration has been performed, the process proceeds to step S110. On the other hand, if the CPU 3 determines in step S107 that normal calibration has not been performed, the process proceeds to step S108 without starting eye-based calibration. In step S108, the CPU 3 displays a message on the display device 10, for example, prompting the user to perform normal calibration. After executing step S108, the process proceeds to step S200.

[0069] In step S110, CPU 3 hides the estimated gaze point C displayed on display device 10, and as... Figure 8BAs shown, a second indicator (b)-C for eye-based calibration is displayed on the real-time viewfinder image superimposed on the display device 10. In step S111, the CPU 3 reads the eye-based calibration coefficients Ax to By stored in the memory unit 4, which stores the eye-based calibration coefficients Ax to By on a user-by-user basis, and stores the updated coefficients by substituting "0" into the eye-based calibration coefficients Bx and By in the memory unit 4. In step S112, the CPU 3 determines whether an instruction to stop eye-based calibration has been received from the user. If the CPU 3 determines in step S112 that an instruction to stop eye-based calibration has been received from the user, the process proceeds to step S113. On the other hand, if the CPU 3 determines in step S112 that an instruction to stop eye-based calibration has not been received from the user, the process proceeds to step S120. As described above, the instruction to stop eye-based calibration can be provided by operating the operation member 42 or by pressing the release button 5. In step S113, CPU 3 stores the line-of-sight correction coefficients Ax, Bx, Ay and By before the update in step S111 in memory unit 4, and the process proceeds to step S200.

[0070] In step S120, CPU 3 determines whether the gaze detection in step S103 has been completed a predetermined number of times. If CPU 3 determines in step S120 that the gaze detection has been completed a predetermined number of times, the process proceeds to step S121. On the other hand, if CPU 3 determines in step S120 that the gaze detection has not been completed a predetermined number of times, the process proceeds to step S130. In step S130, CPU 3 causes the eye image sensor 17 to acquire an image of the user's eyeballs gazing at the second indicator (b)-C displayed in step S101, and performs the aforementioned gaze detection processing. In this embodiment, by performing... Figure 7In the gaze detection process, the coordinates (Hx, Hy) of the estimated gaze point on the display device 10 are determined. After executing step S130, the process returns to step S112. In step S121, the CPU 3 calculates the average of a predetermined number of values ​​of each coordinate (Hx, Hy) of the estimated gaze point determined by performing a predetermined number of gaze detections. In step S122, the CPU 3 uses the average coordinates (Hx, Hy) of the estimated gaze point calculated in step S121 and the gaze correction coefficients Ax and Ay obtained in step S111 to perform eye-based calibration. Thus, gaze correction coefficients Bx and By are calculated for eye-based calibration. In step S123, the CPU 3 stores the gaze correction coefficients Ax and Ay read from the memory unit 4 in step S111 and the gaze correction coefficients Bx and By calculated in step S122 in the memory unit 4. After executing step S123, the process proceeds to step S200.

[0071] In step S200, CPU 3 determines whether the user has pressed release button 5 to activate SW1. If the user agrees to focus at the estimated gaze point position displayed on display device 10, the user can activate SW1 by pressing release button 5. If CPU 3 determines in step S200 that SW1 has been activated, the process proceeds to step S201. On the other hand, if CPU 3 determines in step S200 that SW1 has not been activated, the process returns to step S103, and CPU 3 performs gaze detection again. In step S201, CPU 3 performs a ranging operation at the current position of the estimated gaze point and displays information that the ranging operation is in progress on the real-time view image on display device 10. This allows the user to know that a ranging operation is being performed. Note that the ranging operation execution information can be displayed, for example, by changing the color of the estimated gaze point. In step S202, CPU 3 causes lens focus adjustment circuit 118 to operate lens drive member 114 to move lens 101 to a focus position corresponding to the ranging result obtained in step S201. In step S203, the CPU 3 determines whether the user has further pressed the release button 5 to turn on SW2. If the user agrees to focus at the estimated gaze point displayed on the display device 10, the user further presses the release button 5. Thus, in step S203, the CPU 3 can shift SW2 to the on state. If the CPU 3 determines in step S203 that SW2 is already on, the process proceeds to step S204. On the other hand, if the CPU 3 determines in step S203 that SW2 has not yet been turned on, the process returns to step S200, and the CPU 3 executes its subsequent steps sequentially. In step S204, the CPU 3 activates the image sensor 2 to take a picture. Thus, the image sensor 2 acquires an image signal. Furthermore, these image signals are sent to the CPU 3 and stored in the memory unit 4. After executing step S204, the process proceeds to step S205. In step S205, the CPU 3 displays the acquired image on the display device 10 for, for example, a predetermined period of time, and then the process returns to step S102 so that the CPU 3 sequentially executes its subsequent steps.

[0072] As described above, in camera 1, deviations in the gaze point position that may occur due to eye contact can be corrected without requiring cumbersome operation by the user. This correction aligns the user's actual gaze point with the estimated gaze point, thereby achieving focus at that gaze point and taking the picture. This improves the convenience of shooting for the user.

[0073] The following will refer to Figure 10The second embodiment is described. In the first embodiment described above, even when taking a picture by placing the user's eye at a different eye-reception position than the one where normal calibration was performed, the appropriate gaze correction coefficient at the eye-reception position during the picture can be calculated by performing eye-reception-based calibration, thereby estimating the gaze point. Incidentally, if there is a large difference between the first eye-reception distance (at the first eye-reception position) during normal calibration and the second eye-reception distance (at the second eye-reception position) during eye-reception-based calibration, it is sometimes preferable to perform normal calibration again.

[0074] In this embodiment, the "eye-to-eye distance" refers to the actual distance between the light-receiving lens 16 and the eyeball 14. This eye-to-eye distance is generally inversely proportional to the imaging magnification β described in the first embodiment; therefore, the eye-to-eye distance can be determined as a function of the imaging magnification β. Furthermore, as described in the first embodiment, the imaging magnification β can be determined as a function of the distance (Xd-Xe) between the corneal reflected image Pd' and the corneal reflected image Pe', and therefore the eye-to-eye distance can similarly be determined as a function of the distance (Xd-Xe).

[0075] In this embodiment, the camera 1 is equipped with an eye-to-eye distance detection unit configured to detect the eye-to-eye distance when the user's eye is in contact with the eyepiece lens 12. The eye-to-eye distance detection unit is not particularly limited, but an infrared distance sensor can be used, for example. When the eye-to-eye distance detection unit is implemented using an infrared distance sensor, it is preferable to arrange the infrared distance sensor, for example, between light sources 13a and 13b. Furthermore, the eye-to-eye distance detection unit is capable of detecting a first eye-to-eye distance and a second eye-to-eye distance.

[0076] In camera 1, CPU 3 calculates the difference between a first eye contact distance and a second eye contact distance. If this difference is greater than a predetermined value, CPU 3 determines that normal calibration should be performed again. Then, by performing normal calibration, the first correction information is acquired again, and a first correction is performed using the acquired first correction information without performing a second correction. Note that the predetermined value is determined as follows: when designing camera 1, the relationship between the difference between eye contact distances and the consistency rate between the actual gaze point position and the estimated gaze point position is measured, and a predetermined value is determined such that the consistency rate is equal to or higher than the predetermined value. This predetermined value is then stored in memory unit 4 in advance. Note that the predetermined value is not limited to a uniquely determined value, but is preferably appropriately determined based on various conditions. Furthermore, multiple predetermined values ​​of different sizes can be stored in memory unit 4. This allows calibration to be performed differently depending on the magnitude of the difference between eye contact distances. For example, if a first predetermined value and a second predetermined value smaller than the first predetermined value are provided, and the difference between eye contact distances is greater than the first predetermined value, it is determined that normal calibration should be performed again. Furthermore, if the difference between the eye-to-eye distances is equal to or less than a first predetermined value and greater than a second predetermined value, it is determined that an eye-to-eye calibration should be performed to conduct a second correction using the eye-to-eye calibration. Conversely, if the difference between the eye-to-eye distances is equal to or less than the second predetermined value, it is determined that calibration is not required, and the line-of-sight correction coefficient stored in the memory unit 4 is used as is.

[0077] Figure 10 This is a flowchart of the process for switching between the first correction and the second correction in the second embodiment. Figure 10 Switching processing and Figure 9 The difference in the switching process is that steps S300 to S303 have been added, and the other steps are the same as... Figure 9 The switching process is the same. The following descriptions of steps S300 to S303 and step S107 related to these steps will be given, and details related to... Figure 9 The switching process described in the same way as the previous steps.

[0078] In step S107, CPU 3 determines whether normal calibration has been performed. If CPU 3 determines in step S107 that normal calibration has been performed, the process proceeds to step S300. On the other hand, if it determines in step S107 that normal calibration has not been performed, the process proceeds to step S108 without initiating eye-based calibration.

[0079] In step S300, CPU 3 reads a first eye-to-eye distance stored in memory unit 4 as eye-to-eye distance information. In step S301, CPU 3 calculates the difference between the second eye-to-eye distance information, determined in step S103 as the current eye-to-eye distance, and the first eye-to-eye distance obtained in step S300. In step S302, CPU 3 determines whether the difference is greater than a first predetermined value. If CPU 3 determines in step S302 that the difference is greater than the first predetermined value, the process proceeds to step S108. On the other hand, if CPU 3 determines in step S302 that the difference is equal to or less than the first predetermined value, the process proceeds to step S303. In step S303, CPU 3 determines whether the difference is equal to or less than a second predetermined value. If CPU 3 determines in step S303 that the difference is equal to or less than the second predetermined value, the process proceeds to step S200. On the other hand, if CPU 3 determines in step S303 that the difference is greater than the second predetermined value, the process proceeds to step S110. In this case, since normal calibration has already been performed, it is determined that no calibration is required, and therefore the line-of-sight correction coefficient (first calibration information) stored in the memory unit 4 is read out and used as is.

[0080] Based on the above processing, normal calibration or eye-based calibration can be appropriately selected according to the size of the eye-catching distance, which improves the convenience of shooting.

[0081] The following will refer to Figure 11 The third embodiment is described. In the second embodiment, when the eye contact distance varies for each eye contact, normal calibration or eye-based calibration is performed according to the magnitude of the eye contact distance, thereby determining the appropriate gaze correction coefficient and estimating the fixation point. Incidentally, user intervention is required and can be cumbersome in order to initiate the process for selecting the appropriate calibration and recalculating the gaze correction coefficient. Camera 1 is configured to detect contact with the user's eye. In this embodiment, upon detecting eye contact, the process for selecting the appropriate calibration and recalculating the gaze correction coefficient, as described in the second embodiment, can be automatically initiated.

[0082] In this embodiment, gaze detection is performed periodically even in a non-gaze state. If a gaze point is successfully detected, the user is determined to be in a gaze state; otherwise, the non-gaze state continues. Furthermore, the configuration used for gaze detection is not limited to a single configuration, but is preferably determined appropriately based on various conditions. For example, gaze detection can be performed using, for instance, a contact sensor.

[0083] Figure 11 This is a flowchart of the process for switching between the first correction and the second correction in the third embodiment. Figure 11 Switching processing and Figure 10 The difference in the switching process is that steps S400 and S401 are added, step S106 is replaced by step S410, and the other steps are the same as... Figure 10 The steps in the switching process are the same. The following descriptions of steps S400, S401, and S410, as well as steps S103 to S105 related to these steps, will be given, and details related to... Figure 10 The switching process described in the same way as the previous steps.

[0084] In step S103, CPU 3 operates the eye image sensor 17 to acquire an image of the user's eyes, which was already being visually recognized as a subject in step S101. After acquiring this image, CPU 3 performs the aforementioned gaze detection processing. In this embodiment, through... Figure 7 The gaze detection process in step S103 obtains the coordinates (Hx, Hy) of the estimated gaze point on the display device 10. After executing step S103, the process proceeds to step S400.

[0085] In step S400, CPU 3 determines whether the acquisition of the coordinates (Hx, Hy) of the estimated gaze point in step S103 was successful. If CPU 3 determines in step S400 that the acquisition of the coordinates (Hx, Hy) was successful, the process proceeds to step S104. On the other hand, if CPU 3 determines in step S400 that the acquisition of the coordinates (Hx, Hy) of the estimated gaze point has failed, it determines that the user is in a non-eye-focusing state, and the process proceeds to step S401. In step S401, CPU 3 waits for a predetermined time period, and then the process returns to step S103 for CPU 3 to execute subsequent steps. In steps S104 and S105, the process is executed... Figure 10 The switching process involves the same operations in steps S104 and S105. After executing step S105, the process proceeds to step S410. In step S410, CPU 3 automatically begins eye-based calibration, and the process continues to step S107.

[0086] Based on the above process, when eye contact is detected, the process of selecting an appropriate calibration and recalculating the line-of-sight correction factor can be automatically initiated.

[0087] The following will refer to Figure 12The fourth embodiment is described. In the third embodiment, calibration and calculation of the gaze correction coefficient can be performed automatically upon detection of eye contact. Incidentally, as described in the first embodiment, the purpose of calibration is to correct for individual differences among users. Here, consider the case where a user who has already undergone calibration (hereinafter referred to as "first user") and a user different from the first user (hereinafter referred to as "second user") use a camera 1 shared between them. In this case, according to the configuration of the third embodiment, calibration is also automatically performed for the second user.

[0088] To avoid this situation, the camera 1 according to this embodiment is provided with an iris information acquisition unit and a consistency detection unit. The iris information acquisition unit is configured to acquire iris information related to the user's iris, and the consistency detection unit is configured to detect the consistency between multiple iris information items based on the acquisition results of the iris information acquisition unit. Note that in this embodiment, the eye image sensor 17 is used as the iris information acquisition unit, and the CPU 3 is used as the consistency detection unit.

[0089] In this embodiment, when calibration is performed, the iris information, along with the gaze correction coefficient and the eye contact distance information, is stored in the memory unit 4. In this embodiment, the iris information includes first iris information and second iris information, which is different from the first iris information. Each of these is an iris image obtained by extracting the iris portion from the eye image obtained through the gaze detection described above. Note that the iris information is not limited to uniquely determined information, but is preferably appropriately determined based on various conditions. As iris information, for example, images expanded / compressed by convolution integrals, images subjected to digital filtering, or feature quantities determined by segmenting the image into regions can be used.

[0090] Furthermore, the consistency detection unit calculates the consistency between the two iris information items (i.e., the first iris information and the second iris information) obtained through the above method. There are no particular limitations on the consistency calculation method, but for example, the SAD (Sum of Absolute Differences) method can be used. SAD is the sum of the absolute values ​​of the differences, and therefore, the higher the consistency between the two iris information items, the smaller the evaluation value, such that the minimum value equals 0 and the maximum value equals the sum of the values ​​of one of the iris information items. Note that the consistency calculation method is not uniquely determined, but it is preferably determined based on various conditions such as the form of the iris information and the method of capturing the eye image. Then, if the evaluation value of the first iris information and the second iris information is less than a predetermined value pre-stored in the memory unit 4, it can be determined that the two irises belong to the same person. Note that the predetermined value is preferably determined based on a sufficient number of iris samples, and more specifically, based on iris information items obtained under various conditions such as the number of people, eye angle, iris color, and brightness value at the time of capturing the eye image.

[0091] Figure 12 This is a flowchart of the process for switching between the first correction and the second correction in the fourth embodiment. Figure 12 Switching processing and Figure 11 The difference in the switching process is that steps S500 to S502 have been added, and the other steps are the same as... Figure 11 The switching process is the same. The following description will be given of steps S500 to S502 and step S107 related to these steps, and will omit related steps. Figure 11 The switching process described in the same way as the previous steps.

[0092] In step S107, CPU 3 determines whether normal calibration has been performed. If CPU 3 determines in step S107 that normal calibration has been performed, the process proceeds to step S500. On the other hand, if CPU 3 determines in step S107 that normal calibration has not been performed, the process proceeds to step S108 without initiating eye-based calibration.

[0093] In step S500, CPU 3 reads the iris information stored in memory unit 4. The iris information refers to the iris information obtained when normal calibration has been performed as described above. In step S501, CPU 3 calculates the consistency between the current iris information obtained in step S103 and the iris information obtained in step S500. When the SAD method described above is used for consistency calculation, a lower evaluation value indicates a higher consistency between the two iris information items. In step S502, CPU 3 determines whether the evaluation value is equal to or less than a predetermined value (i.e., whether the consistency between the first iris information and the second iris information is equal to or greater than a predetermined value). If CPU 3 determines in step S502 that the consistency is equal to or greater than the predetermined value, the process proceeds to step S300. On the other hand, if CPU 3 determines in step S502 that the consistency is less than the predetermined value, the process returns to step S108, and CPU 2 sequentially executes the subsequent steps.

[0094] According to the above configuration, if the user performing the calibration and the subsequent user who brings their eye into contact with the eyepiece are the same person, the process described in the third embodiment can be started automatically. Furthermore, in this embodiment, if it is determined that the consistency between the first iris information and the second iris information is equal to or higher than a predetermined value, a second correction can be performed; and if it is determined that the consistency is lower than the predetermined value, a first correction can be performed without performing a second correction. This allows for appropriate switching of calibration based on the user, thereby contributing to high-precision estimation of the fixation point.

[0095] Although a description of preferred embodiments of the invention has been provided, the calibration for line-of-sight detection is applicable not only to the cameras described above, but also to, for example, head-mounted displays, see-through glasses, and vehicle cameras.

[0096] Other embodiments

[0097] The embodiments of the present invention can also be implemented by providing the software (program) that performs the functions of the above embodiments to the system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads and executes the program.

[0098] While the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

[0099] This application claims the benefit of Japanese Patent Application 2021-198715, filed on December 7, 2021, the entire contents of which are incorporated herein by reference.

Claims

1. A control device, comprising: An eyepiece is provided so that the user's eyes come into contact with the display unit when the user views it. An eye contact position detection unit is configured to detect the eye contact position of the eyepiece that causes the user's eye to come into contact with it. At least one processor; as well as A memory coupled to the at least one processor, the memory having instructions that, when executed by the processor, operate as a unit: An estimation unit is configured to estimate the position corresponding to the user's gaze point; as well as The correction unit is configured to correct the position corresponding to the user's gaze point based on the position of the indicator displayed on the display unit and the position corresponding to and associated with the user's gaze point. The correction unit has a first mode and a second mode. The first mode is used to perform the correction based on the positions of a first number of indicators, and the second mode is used to perform the correction based on a second number of indicators, which is less than the first number. In the first mode, the correction unit displays the first number of indicators on the display and uses first correction information, which is obtained based on the positional relationship between each of the first number of indicators on the display and the position corresponding to the gaze point estimated by the estimation unit when the user visually recognizes each of the first number of indicators; and in the second mode, the correction unit displays the second number of indicators on the display and uses second correction information, which is generated by reflecting information obtained based on the positional relationship between each of the second number of indicators on the display and the position corresponding to the gaze point estimated by the estimation unit when the user visually recognizes each of the second number of indicators on the display on the first correction information. The eye contact position detection unit is capable of detecting a first eye contact position that the user's eye contacts during a first correction using the first correction information, and a second eye contact position different from the first eye contact position, as the eye contact position. When the eye contact position detection unit detects the second eye contact position, the correction unit performs a second correction using the second correction information.

2. The control device according to claim 1, wherein, The correction unit performs the correction in the order of the first mode and the second mode.

3. The control device according to claim 1, further comprising: An eye contact detection unit is configured to detect the contact between the eye and the eyepiece. Specifically, when the eye contact detection unit detects contact with the eye, the correction unit performs the correction.

4. The control device according to claim 1, further comprising an eye contact distance detection unit, the eye contact distance detection unit being configured to detect the eye contact distance when the user's eye is in contact with the eyepiece portion, and in, The eyelet distance detection unit can detect the first eyelet distance at the first eyelet position and the second eyelet distance at the second eyelet position, and Wherein, if the difference between the first eye contact distance and the second eye contact distance is greater than a first predetermined value, the correction unit performs the first correction but not the second correction.

5. The control device according to claim 4, wherein, When the difference between the first eye contact distance and the second eye contact distance is equal to or less than a second predetermined value that is smaller than the first predetermined value, the correction unit uses the first correction information it has already obtained when performing the first correction.

6. The control device according to claim 4, wherein, If the difference between the first eye contact distance and the second eye contact distance is equal to or less than the first predetermined value, the correction unit performs the second correction.

7. The control device according to claim 1, wherein, The memory has instructions that, when executed by the processor, further operate as units of: An iris information acquisition unit, configured to acquire iris information associated with the user's iris; and A consistency detection unit is configured to detect the consistency among multiple pieces of said iris information, and The iris information acquisition unit is capable of detecting first iris information and second iris information different from the first iris information as the iris information. Specifically, if the consistency between the first iris information and the second iris information detected by the consistency detection unit is equal to or higher than a predetermined value, the correction unit performs the second correction; and if the consistency detected by the consistency detection unit is lower than the predetermined value, the correction unit performs the first correction but not the second correction.

8. The control device according to claim 1, wherein, The display unit includes a display screen with a rectangular shape. The first number of indicators are arranged at various positions offset from the edge of the display screen, and The second number of indicators are each arranged in the center of the display screen.

9. The control device according to claim 1, wherein, The first quantity is the quantity from 2 to 8, and The second quantity is a quantity of 1 to 4.

10. The control device according to claim 1, further comprising: The display unit; The camera unit is configured to capture images of the subject; as well as An eye-mounted camera unit is configured to capture images of the user's eyes.

11. A control device, comprising: An eyepiece is provided so that the user's eyes come into contact with the display unit when the user views it. An eye contact position detection unit is configured to detect the eye contact position of the eyepiece that causes the user's eye to come into contact with it. At least one processor; as well as A memory coupled to the at least one processor, the memory having instructions that, when executed by the processor, operate as a unit: A control unit configured to control the display performed by the display unit, which displays an indicator for calibration of the user's gaze point detection, and The control unit causes the display unit to display a first number of indicators during the first calibration, and a second number of indicators (less than the first number) during a second calibration performed after the first calibration. In the first calibration, the control unit displays the first number of indicators on the display unit and uses first correction information, which is obtained based on the positional relationship between each of the first number of indicators on the display unit and the position corresponding to the gaze point detected when the user visually recognizes each of the first number of indicators; and in the second calibration, the control unit displays the second number of indicators on the display unit and uses second correction information, which is generated by reflecting information obtained based on the positional relationship between each of the second number of indicators on the display unit and the position corresponding to the gaze point detected when the user visually recognizes each of the second number of indicators on the display unit onto the first correction information. The eye contact position detection unit is capable of detecting a first eye contact position that the user's eye contacts during a first calibration using the first correction information, and a second eye contact position different from the first eye contact position, as the eye contact position. When the eye contact position detection unit detects the second eye contact position, the control unit performs a second calibration using the second correction information.

12. A control method for a control device including an eyepiece, wherein when a user views a display, the user's eye contacts the eyepiece, the control method comprising: Detect the eyepiece position where the user's eye contacts the eyepiece. Estimate the position corresponding to the user's gaze point; as well as Based on the position of the indicator displayed on the display unit, and the position corresponding to and associated with the user's gaze point, the position corresponding to the user's gaze point is corrected. The correction is performed in at least one of a first mode and a second mode, wherein the first mode is used to perform the correction based on the position of a first number of indicators, and the second mode is used to perform the correction based on a second number of indicators, which is less than the first number. In the first mode, a first number of indicators are displayed on the display unit, and first correction information is used. This first correction information is obtained based on the positional relationship between each of the first number of indicators on the display unit and the position corresponding to the gaze point estimated when the user visually recognizes each of the first number of indicators. In the second mode, a second number of indicators are displayed on the display unit, and second correction information is used. This second correction information is generated by reflecting information obtained based on the positional relationship between each of the second number of indicators on the display unit and the position corresponding to the gaze point estimated when the user visually recognizes each of the second number of indicators onto the first correction information. Specifically, it can detect a first eye contact position that the user's eye contacts when performing a first correction using the first correction information, and a second eye contact position different from the first eye contact position as the eye contact position. Specifically, when the second contact point is detected, a second correction is performed using the second correction information.

13. A control method for a control device including an eyepiece, wherein when a user views a display, the user's eye contacts the eyepiece, the control method comprising: The display is controlled by a display unit that displays an indicator for calibration of the user's gaze point detection. The control includes: causing the display unit to display a first number of indicators during a first calibration, and to display a second number of indicators less than the first number during a second calibration performed after the first calibration. In the first calibration, the first number of indicators are displayed on the display unit and first correction information is used. This first correction information is obtained based on the positional relationship between each of the first number of indicators on the display unit and the position corresponding to a gaze point detected when the user visually recognizes each of the first number of indicators. In the second calibration, the second number of indicators are displayed on the display unit and second correction information is used. This second correction information is generated by reflecting information obtained based on the positional relationship between each of the second number of indicators on the display unit and the position corresponding to a gaze point detected when the user visually recognizes each of the second number of indicators onto the first correction information. Specifically, it can detect a first eye contact position that the user's eye contacts during a first calibration using the first correction information, and a second eye contact position different from the first eye contact position as the eye contact position. Specifically, when the second contact point is detected, a second calibration is performed using the second correction information.

14. A non-transitory computer-readable storage medium storing a program that, when executed by a computer, implements the method according to claim 12 or 13.

15. A computer program product comprising a program that, when executed by a computer, implements the method according to claim 12 or 13.

Images