Optical device, imaging device, control method for optical device, and program

Abstract

To provide an optical device capable of properly correcting the watching point position of a user without applying a load on the user.SOLUTION: An optical device (100) comprises: estimation means (115) which estimates the watching point position of a user from an image of the eyeballs of the user; display means (111) which displays an index indicative of the watching point position estimated by the estimation means; and analysis means (119) which analyzes the position history of the watching point position and / or the rotation angle history of the eyeballs. The analysis means corrects the position of the index displayed by the display means, based on the position history and / or the rotation angle history.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an optical device, an imaging device, a control method for an optical device, and a program. [Background technology]

[0002] Conventionally, methods for detecting and correcting the deviation between an estimated gaze point position and an actual gaze point position have been known. Patent Document 1 discloses a method for calculating the deviation from the detected gaze point by obtaining a representative value for a specific direction (for example, a point at infinity in the forward direction) when the driver is looking at that direction. Patent Document 2 discloses a method for calculating the deviation from the detected gaze point when an event such as the user pressing a button occurs, assuming that the user is looking at the center of the button, and reflecting the deviation between the position of the gaze pointer and the center of the button in the correction amount. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-004117 A [Patent Document 2] Japanese Patent Application Laid-Open No. 2006-285715 Summary of the Invention [Problem to be solved by the invention]

[0004] The method disclosed in Patent Document 1 may not be able to properly correct the gaze point position because it depends on conditions such as the scenery seen by the user and the user's environment. The method disclosed in Patent Document 2 requires the user to explicitly specify the gaze point position by pressing a button, for example, which places a burden on the user.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an optical device that can appropriately correct the position of the user's gaze point without imposing a burden on the user. [Means for solving the problem]

[0006] An optical device according to one aspect of the present invention includes an estimation means for estimating a gaze point position of a user from an image of the user's eyeball, a display means for displaying an index indicating the gaze point position estimated by the estimation means, and an analysis means for analyzing at least one of a position history of the gaze point position or a rotation angle history of the eyeball, and the analysis means corrects the position of the index displayed on the display means based on at least one of the position history or the rotation angle history. When the analysis means determines that the gaze point position shows a predetermined movement based on at least one of the position history and the rotation angle history, the analysis means corrects the position of the index displayed on the display means, and the analysis means increases an evaluation value related to the predetermined movement as the frequency at which the gaze point position moves in a predetermined direction within a predetermined time increases. do.

[0007] Other objects and features of the present invention will be described in the following embodiments. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an optical device that can appropriately correct the position of the user's point of gaze without imposing a burden on the user. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram of an imaging apparatus according to an embodiment of the present invention. [Figure 2] 10 is a flowchart showing a gaze point detection process in the present embodiment. [Figure 3] 3A and 3B are diagrams illustrating the principle of gaze detection in the present embodiment. [Figure 4] 5A and 5B are explanatory diagrams of an eyeball image projected onto an eyeball image pickup element and an output intensity of the eyeball image pickup element in the present embodiment. [Figure 5] 10 is a flowchart showing a gaze detection process in the present embodiment. [Figure 6] FIG. 3 is an explanatory diagram of a screen visible to a user through a display unit in this embodiment. [Figure 7] 10A and 10B are diagrams illustrating a deviation between an actual gaze point and an estimated gaze point in this embodiment. [Figure 8] 10 is a flowchart of a process for analyzing a gaze point position history in this embodiment. [Figure 9] FIG. 10 is a diagram showing the distribution of the movement of the estimated gaze point in this embodiment. [Figure 10] 10A and 10B are diagrams illustrating a deviation between an actual gaze point and an estimated gaze point in this embodiment. [Figure 11] FIG. 10 is a diagram showing the distribution of estimated gaze points in this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0011] First, the configuration of an optical device according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram of an imaging device (optical device) 100 according to this embodiment. An imaging lens (imaging optical system) 104 forms an optical image (subject image). An imaging element 106 is a photoelectric conversion element such as a CMOS sensor, which photoelectrically converts the optical image formed by the imaging lens 104 and outputs an image signal. Note that in this embodiment, the imaging device 100 is configured integrally with the imaging lens 104, but this is not limiting, and the imaging lens 104 may be a lens device (interchangeable lens) that is detachable from the imaging device 100.

[0012] The CPU 102 controls the display means 111, the illumination light source driving means 112, the storage means 114, the gaze estimation means (estimation means) 115, the eye image sensor 117, the gaze history analysis means (analysis means) 119, and the operation means 124. The display means 111 displays a captured image and also displays information (an index) indicating the gaze point (gaze point position) detected (estimated) by the gaze estimation means 115. The illumination light source 113 is a light source such as a light-emitting diode that emits infrared light that is insensitive to the user, and is driven by the illumination light source driving means 112 to illuminate the user's eye. A portion of the illumination light reflected by the user's eye is focused on the eye image sensor 117. The storage means 114 stores the image signal from the image sensor 106, the image signal from the eye image sensor 117, the gaze point position estimated by the gaze estimation means 115, and gaze correction data that corrects for individual differences in gaze.

[0013] The gaze estimation means 115 is a digital serial interface circuit, and transmits an output signal (a signal acquired by forming an image of the eyeball (eyeball image)) from the eyeball imaging element 117 to the CPU 102. The light receiving lens 116 optically forms an image of the user's eyeball on the eyeball imaging element 117. The gaze history analysis means 119 analyzes the history of the gaze point position of the user estimated by the gaze estimation means 115, and detects characteristic movements (predetermined movements) of the gaze point position (determines whether the gaze point position exhibits the predetermined movement). The operation means 124 is a means for accepting operations performed by the user on the imaging device 100, and includes, for example, a button and a zoom lever (not shown) attached to the imaging device 100, a ring attached to the imaging lens 104, and the like.

[0014] Next, the user's gaze detection process will be described with reference to Figs. 3 to 5. Fig. 3 is an explanatory diagram of the gaze detection principle. In Fig. 3, light sources 113a and 113b (illumination light source 113) are light sources such as light-emitting diodes that emit infrared light that is insensitive to the user. Light sources 113a and 113b illuminate the user's eyeball. A portion of the illumination light reflected by the user's eyeball is collected by light-receiving lens 116 onto eyeball image sensor 117.

[0015] FIG. 4(a) is an explanatory diagram of an eyeball image projected onto the eyeball image sensor 117. FIG. 4(b) is an explanatory diagram of the output intensity of the CCD of the eyeball image sensor 117. FIG. 5 is a flowchart showing the gaze detection process. FIGS. 6(a) to 6(c) are explanatory diagrams of a screen visible to the user through the display means 111. In FIG. 6(a), 300 denotes a visual field mask. The gaze point estimated (calculated) by the gaze estimation means 115 is displayed on the display means 111 by making a frame appear, which is shown as estimated gaze point A in FIG. 6(a).

[0016] 5, when the gaze detection process (gaze detection routine) starts, first, in step S501, illumination light source 113 (light sources 113a and 113b) emits infrared light toward user's eyeball 314. An image of the user's eyeball illuminated by the infrared light is formed on eyeball imaging element 117 through light receiving lens 116. Eyeball imaging element 117 photoelectrically converts the eyeball image formed by light receiving lens 116 and outputs an electrical signal (image signal). Subsequently, in step S502, CPU 102 acquires an image signal (eyeball image signal) from eyeball imaging element 117.

[0017] Next, in step S503, CPU 102 acquires corneal reflection images Pd and Pe of light sources 113a and 113b shown in FIG. 3 and coordinates of a point corresponding to pupil center c (pupil center position, corneal reflection position of light source) based on the eyeball image signal acquired in step S502. Infrared light emitted from light sources 113a and 113b illuminates cornea 342 of user's eyeball 314. At this time, corneal reflection images Pd and Pe formed by part of the infrared light reflected from the surface of cornea 342 are collected by light receiving lens 116 and formed on eyeball image sensor 117 (reflection images Pd', Pe'). Similarly, light beams from the edges of pupil 341 (pupil edges a, b) also form images on eyeball image sensor 117.

[0018] Fig. 4(a) shows an example of a reflection image obtained from the eyeball image sensor 117, and Fig. 4(b) shows example luminance information obtained from the eyeball image sensor 117 in region α of the example image in Fig. 4(a). In Fig. 4(a), the horizontal direction is the X-axis and the vertical direction is the Y-axis. Here, the coordinates in the X-axis direction (horizontal direction) of reflection images Pd' and Pe' formed by corneal reflection images Pd and Pe of light sources 113a and 113b are denoted as Xd and Xe, respectively. Furthermore, the coordinates in the X-axis direction of images (pupil edges a' and b') formed by light beams from pupil edges a and b of pupil 314b are denoted as Xa and Xb, respectively.

[0019] In the example of luminance information in FIG. 4( b), extremely high levels of luminance are obtained at positions Xd and Xe corresponding to the reflection images Pd' and Pe' formed by the corneal reflection images Pd and Pe of light sources 113a and 113b. The region between coordinate Xa (coordinate of pupil edge a') and coordinate Xb (coordinate of pupil edge b'), which corresponds to the region of pupil 341, has extremely low levels of luminance, except for positions Xd and Xe. In contrast, the region corresponding to the region of iris 343 outside pupil 341, which has an X coordinate value lower than coordinate Xa and an X coordinate value higher than coordinate Xb, has intermediate values ​​between the two luminance levels. From information about fluctuations in luminance levels relative to the X coordinate position, the X coordinates Xd and Xe of the reflection images Pd' and Pe' formed by the corneal reflection images Pd and Pe of light sources 113a and 113b and the coordinates Xa and Xb of the pupil edges a' and b' can be obtained.

[0020] Furthermore, when the rotation angle θx of the optical axis of eyeball 314 relative to the optical axis of light receiving lens 116 is small, the coordinate Xc of the point (pupil center c') corresponding to pupil center c imaged on eyeball image sensor 117 can be expressed as Xc ≈ (Xa + Xb) / 2. From the above, it is possible to estimate the X coordinate of pupil center c' imaged on eyeball image sensor 117 and the coordinates of corneal reflection images Pd' and Pe' of light sources 113a and 113b.

[0021] 5, CPU 102 calculates the imaging magnification β of the eyeball image. The imaging magnification β is determined by the position of eyeball 314 relative to light receiving lens 116, and can be calculated substantially as a function of the distance (Xd-Xe) between reflected images Pd' and Pe'. Next, in step S505, CPU 102 calculates the rotation angles (eyeball angles) θx and θy of the eyeball. The X coordinate of the midpoint of corneal reflection images Pd and Pe and the X coordinate of the center of curvature O of cornea 342 approximately coincide. For this reason, if the standard distance from the center of curvature O of cornea 342 to the center c of pupil 341 is Oc, then the rotation angle θ of the optical axis of eyeball 314 in the ZX plane X can be calculated from the following formula (1).

[0022] β*Oc*SINθx≒{(Xd+Xe) / 2}-Xc …(1) 5 and 6, the rotation angle θ when the user's eyeball rotates in a plane perpendicular to the Y axis is X However, the method for calculating the rotation angle θy when the user's eyeball rotates in a plane perpendicular to the X axis is the same.

[0023] After calculating the rotation angles θx, θy of the optical axis of the user's eyeball 314 in step S505, CPU 102 reads correction coefficient data in step S506. Subsequently, in step S507, CPU 102 uses the rotation angles θx, θy to determine the position of the user's line of sight (the position of the point of gaze, referred to as the gaze point position) on display means 111. The gaze point position is calculated as coordinates (Hx, Hy) corresponding to the center c of pupil 341 on display means 111 using the following equations (2) and (3).

[0024] Hx = m × (Ax × θx + Bx) … (2) Hy = m × (Ay × θy + By) … (3) In equations (2) and (3), coefficient m is a constant that represents the relationship between the rotation angles θx, θy of the user's eyeball and their positions on the display means 111. That is, coefficient m is a conversion coefficient that converts the rotation angles θx, θy into position coordinates corresponding to the center c of the pupil 141 on the display means 111, and is determined in advance and stored in the storage means 114. Also, in equations (2) and (3), Ax, Bx, Ay, and By are gaze correction coefficients (correction coefficient data) that correct individual differences in the user's gaze, and are acquired by performing a calibration operation, which will be described later, and are stored in the storage means 114 before the gaze detection routine starts.

[0025] After calculating the coordinates (Hx, Hy) of the center c of the pupil 141 on the display means 111, in step S508, the CPU 102 stores the gaze point position (gazing point coordinates) calculated in step S507 in the storage means 114, and ends the gaze detection routine.

[0026] In this embodiment, a method for acquiring gaze point coordinates on a lens element using the corneal reflection images of light sources 113a and 113b has been described, but this is not limited to this, and any other method for acquiring the rotation angle of the eyeball from the captured eyeball image may be used.

[0027] Next, the calibration process will be described. Calibration is a process (preparatory process) for correcting in advance the deviation that occurs between the actual gaze position of the user (actual gaze position) and the gaze position of the user estimated by the imaging device (estimated gaze position). In this embodiment, the CPU 102 functions as an acquisition unit that acquires calibration data regarding the deviation between the index displayed on the display unit 111 and the gaze position estimated by the line-of-sight estimation unit 115.

[0028] As described above, in this embodiment, the gaze detection routine acquires the rotation angles θx and θy of the eyeball from the eyeball image, and performs a calculation to convert the pupil center position to a corresponding position on the display unit 111 to estimate the gaze point position. However, due to factors such as individual differences in the shape of the human eyeball, the values ​​of the gaze correction coefficients Ax, Ay, Bx, and By must be adjusted to appropriate values ​​for each user. Without such adjustment, as shown in FIG. 6(b), the position where the user actually gazes (the position of the user's gaze point B) will differ from the estimated (calculated) position of the estimated gaze point C (the positions of gaze points B and C will differ). For example, in the example of FIG. 6(b), the user intends to gaze at the person (subject) located at gaze point B, but the gaze estimation unit 115 erroneously estimates that the user is gazing at the background. Therefore, before using the imaging device 100, a calibration operation must be performed to acquire appropriate correction coefficient values ​​for each user and store them in the storage unit 114.

[0029] Conventionally, calibration work is performed by highlighting multiple indices at different positions and having the user look at the indices before using the imaging device 100, as shown in Fig. 6(c). A known technique is to perform a gaze point detection process when the user gazes at each indices, and to calculate appropriate correction coefficients (calibration data) from the calculated multiple estimated gaze point coordinates and the positions of the index coordinates.

[0030] However, even if the deviation between the actual gaze position (actual gaze point) and the estimated gaze point estimated by gaze detection means 115 is corrected using the correction coefficient obtained in the calibration process, a deviation may still remain. This may occur when the positional relationship between the user and light receiving lens 116 changes significantly between the time of the calibration process and the time of shooting. A method for detecting and correcting this deviation (gaze point detection process) will be described below with reference to Figure 2. Figure 2 is a flowchart showing the gaze point detection process.

[0031] First, in step S201, the CPU 102 detects the gaze point (gazing point position) by the gaze detection process described with reference to Fig. 5. Subsequently, in step S202, the CPU 102 (gaze history analysis means 119) analyzes the history (position history) of the gaze point position detected in step S201.

[0032] Here, the analysis process of the gaze point position history will be described with reference to Fig. 8. Fig. 8 is a flowchart of the analysis process of the gaze point position history. First, in step S801, the CPU 102 (gaze history analysis means 119) analyzes whether the gaze point position continues to shift in a predetermined direction (the gaze point position moves in a predetermined direction for a predetermined time) as a first characteristic movement of the gaze point.

[0033] 7(a) to 7(c) are diagrams showing the deviation between the actual gaze point and the estimated gaze point. In Fig. 7(a), a gaze frame is displayed at the estimated gaze point 701 on the display means 111. Fig. 7(a) shows that a deviation (deviation vector 703) has occurred between the actual gaze point 702, which is the point the user is actually looking at, and the estimated gaze point 701.

[0034] In this state, the user is likely to look at the estimated gaze point 701, which displays a gaze frame. However, once the user looks at the gaze frame, in the next frame, the actual gaze point 706 moves to the position where the gaze point marker 701 was displayed, as shown in FIG. 7(b). Accordingly, the estimated gaze point 705 also moves to a position shifted by the displacement vector 703 from the actual gaze point 706. If this is repeated for each frame, the user's actual gaze point 708 moves in the direction of the displacement vector 703, and eventually moves to the edge of the display means 111, as shown in FIG. 7(c).

[0035] To detect such a state, the horizontal (x direction) and vertical (y direction) movement amounts of the estimated gaze point between frames are observed for a predetermined time (constant period), and respective histograms are created, as shown in Figures 9(a) and 9(b). Figures 9(a) and 9(b) are diagrams showing the distribution (histogram) of the movement of the estimated gaze point. In Figure 9(a), the horizontal axis represents the horizontal (x direction) movement amount of the estimated gaze point on the display means 111 between frames, and the vertical axis represents the number of times (frequency) observed over a certain period, normalized to a total of 1. Similarly, in Figure 9(b), the vertical axis represents the vertical (y direction) movement amount of the estimated gaze point on the display means 111 between frames, and the vertical axis represents the number of times (frequency) observed over a certain period, normalized to a total of 1.

[0036] The histograms shown in Figures 9(a) and (b) are analyzed, and the evaluation value (feature evaluation value) f is increased as the horizontal movement amount mode p_x_1 approaches 1 and the vertical movement amount mode p_y_1 approaches 1. For example, the evaluation value f is calculated using the following equation (4). In equation (4), k_x and k_y are weighted addition coefficients, respectively.

[0037] f=k_x·p_x_1+k_y·p_y_1 …(4) The larger the evaluation value f, the more characteristic the movement. Also, it is determined that a deviation occurs between the actual and estimated gaze points, with the horizontal component of the vector x_1 and the vertical component y_1 representing the amount of movement of the estimated gaze point.

[0038] 8, the CPU 102 (the gaze history analysis means 119) corrects the deviation of the gaze point position based on the evaluation value f calculated in step S801 and the detected deviation vector 703 (x_1, y_1). In this embodiment, for the sake of explanation, v is added to the deviation vector 703. diff The symbol is assigned to the displacement vector v diff The deviation between the actual gaze point and the estimated gaze point can be corrected by adding a value obtained by multiplying the deviation correction strength α(f) expressed by the following equation (5) to the estimated gaze point.

[0039]

number

[0040] The correction strength α(f) is a function of the evaluation value f, and the larger the evaluation value f, the larger the correction strength α. diff If the deviation vector v is corrected in a short time, the gaze point marker displayed at the estimated gaze point on the display means 111 may suddenly move, which may cause discomfort to the user. diff The magnitude of the threshold th diff If it is larger than diff is corrected by the value obtained by multiplying the coefficient (correction coefficient) k by the evaluation value f. However, the value obtained by multiplying the coefficient k by the evaluation value f is set to be between 0 and 1. diff The magnitude of the threshold th diff If it becomes smaller than diff Correct only the size of

[0041] In this way, in this embodiment, the higher the frequency with which the gaze point position moves in a predetermined direction within a predetermined time, the larger the evaluation value (feature evaluation value) f for the predetermined movement. Also, when it is determined that the gaze point position shows a predetermined movement, the gaze history analysis means 119 calculates the movement speed vector (deviation vector v) of the gaze point position. diff The gaze point position is corrected using a correction vector obtained by multiplying the calculated value (i.e., the distance (k)) by a correction coefficient (coefficient k) based on the evaluation value. Note that correction of the gaze point position can be performed only when a characteristic movement is detected, but is not limited to this. In addition, the correction coefficient stored in the storage means 114 may be modified (updated) based on the detected deviation.

[0042] Next, in step S803, the CPU 102 (gaze history analysis means 119) analyzes whether the gaze point position alternates between two locations (the gaze point position moves back and forth between two different locations within a predetermined time), as a second characteristic movement of the gaze point. A case where this state occurs will be described with reference to FIG. 10. FIG. 10 is a diagram showing a deviation between the actual gaze point and the estimated gaze point. As shown in FIG. 10, there is a deviation between the actual gaze point 1002 and the estimated gaze point 1001. If the user is aware of this deviation, the user periodically looks at the estimated gaze point 1001 while gazing at the actual gaze point 1002. In this way, the second characteristic movement may occur when checking the gaze frame displayed at the estimated gaze point 1001.

[0043] In order to detect such a state, the estimated gaze point between frames is observed for a certain period of time in terms of horizontal position (x) and vertical position (y), and respective histograms are created as shown in Fig. 11. Fig. 11 is a diagram showing the distribution (histogram) of the estimated gaze point. In Fig. 11, with respect to the horizontal direction, the horizontal axis indicates the horizontal position (x) of the estimated gaze point on the display means 111, and the vertical axis indicates the number of times (frequency) observed in a certain period of time normalized so that the total sum is 1. Similarly, with respect to the vertical direction, the vertical axis indicates the vertical position (y) of the estimated gaze point on the display means 111, and the horizontal axis indicates the number of times (frequency) observed in a certain period of time normalized so that the total sum is 1.

[0044] The CPU 102 (gaze history analysis means 119) determines the position with the highest horizontal frequency to be the horizontal position (x_2) of the actual gaze point, and the position with the highest vertical frequency to be the vertical position (y_2) of the actual gaze point. The CPU 102 also determines the position with the second highest horizontal frequency to be the horizontal position (x_3) of the estimated gaze point, and the position with the second highest vertical frequency to be the vertical position (y_3) of the estimated gaze point.

[0045] The closer the mode p_x_2 of the horizontal position at the actual gaze point is to 1, and the closer the mode p_y_2 of the vertical position is to 1, the larger the evaluation value f is. For example, the evaluation value f is calculated using the above-mentioned formula (4). A larger evaluation value f indicates more distinctive movement. The CPU 102 (gaze history analysis means 119) analyzes the histogram and determines the difference between the horizontal coordinates x_2 and x_3 corresponding to the frequencies p_x_2 and p_x_3 as the deviation in the x direction, and the difference between the vertical coordinates y_2 and y_3 corresponding to the frequencies p_y_2 and p_y_3 as the deviation in the y direction.

[0046] Next, in step S804 of FIG. 8, the deviation of the gaze point position is corrected based on the evaluation value f calculated in step S804 and the detected deviation vector (x_3-x_2, y_3-y_2). diff The correction method is the same as in step S802, so a description thereof will be omitted.

[0047] As described above, in this embodiment, the gaze history analysis means 119 increases the evaluation value (feature evaluation value) f for the predetermined movement as the frequency of the gaze point position moving back and forth between two different positions within a predetermined time increases. Furthermore, when the gaze history analysis means 119 determines that the gaze point position exhibits the predetermined movement, it corrects the gaze point position using a correction vector obtained by multiplying the vector between the two different positions by a correction coefficient based on the evaluation value. Note that correction of the gaze point position can be performed only when a characteristic movement is detected, but is not limited to this. Furthermore, the correction coefficient stored in the storage means 114 may be further corrected (updated) using the detected deviation.

[0048] In this embodiment, as shown in Figures 7(a) to 7(c) or 11, when the estimated gaze point is shifted downward and to the right of the display means 111 relative to the actual gaze point, the user may actually move their gaze downward and to the right to view a subject or a menu display at the lower right. Therefore, even if the user's gaze movement corresponds to the first characteristic movement (S801) or the second characteristic movement (S803), if a distinctive subject such as a person, an animal, or a high-contrast subject is present at the moved estimated gaze point position, correction of the estimated gaze point position may not be performed. In other words, the gaze history analysis means 119 may be configured not to correct the position of the index displayed on the display means 111 when the gaze point position estimated by the gaze estimation means 115 moves and reaches a predetermined position and a distinctive subject is detected around the predetermined position.

[0049] Alternatively, if the operation means 124 is operated after the estimated point of gaze position has moved, and some instruction (predetermined operation) is given to the imaging device 100 regarding the point of gaze position, the CPU 102 may determine that there was an intention to move the point of gaze to that position, and may not correct the estimated point of gaze position. In other words, the gaze history analysis means 119 may be configured not to correct the position of the index displayed on the display means 111 when the point of gaze position estimated by the gaze estimation means 115 has moved and reached a predetermined position and a predetermined operation has been performed on the imaging device 100.

[0050] Alternatively, if the estimated gaze point position is moving faster than a predetermined speed, the CPU 102 may determine that the user is not following the gaze point frame but is quickly moving their gaze to a distinctive subject or menu display, and may not need to correct the estimated gaze point position.

[0051] In this embodiment, a calibration operation is performed in advance, and correction coefficients are stored in the storage means 114. Also, in this embodiment, the gaze history analysis means 119 may correct the calibration data based on at least one of the position history and the rotation angle history. However, this embodiment is not limited to this. The deviation between the actual gaze point and the estimated gaze point may be detected and corrected during shooting by detecting the characteristic movements described above without performing a calibration operation in advance.

[0052] In this embodiment, characteristic movements are detected based on the history of estimated gaze point positions, but the history of the rotation angles θx, θy of the user's eyeballs (rotation angle history) may also be used. The relationship between the gaze point (Hx, Hy) and the rotation angles θx, θy of the eyeballs is expressed by equations (2) and (3). Note that the detection method may be similar to steps S801 and S803 in FIG. 8, and therefore a description thereof will be omitted.

[0053] As described above, the imaging device 100 of this embodiment includes a gaze estimation means 115, a display means 111, and a gaze history analysis means 119. The gaze estimation means 115 estimates the position of the user's gaze point from an image signal of the user's eyeball. The display means 111 displays an index indicating the gaze point position estimated by the gaze estimation means 115. The gaze history analysis means 119 analyzes at least one of the position history of the gaze point position or the rotation angle history of the eyeball. The gaze history analysis means 119 also corrects the position of the index displayed on the display means 111 based on at least one of the position history or the rotation angle history. Preferably, the gaze history analysis means 119 corrects the position of the index displayed on the display means 111 when it determines that the gaze point position shows a predetermined movement based on at least one of the position history or the rotation angle history.

[0054] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0055] According to this embodiment, it is possible to provide an optical device, a control method for an optical device, and a program that can appropriately (dynamically) correct the position of the user's gaze point without imposing a burden on the user.

[0056] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0057] For example, the present invention is also applicable to optical devices for VR and optical see-through MR / AR. [Explanation of symbols]

[0058] 100 Imaging device (optical device) 111 Display means 115 Line of sight estimation means (estimation means) 119 Gaze history analysis means (analysis means)

Claims

1. an estimation means for estimating a gaze point position of a user from an image signal of the user's eyeball; a display means for displaying an indicator indicating the gaze point position estimated by the estimation means; and an analysis means for analyzing at least one of the position history of the gaze point position and the rotation angle history of the eyeball, the analysis means corrects the position of the index displayed on the display means based on at least one of the position history and the rotation angle history; when it is determined that the gaze point position exhibits a predetermined movement based on at least one of the position history and the rotation angle history, the analysis means corrects the position of the index displayed on the display means; The optical device is characterized in that the analysis means increases an evaluation value relating to the predetermined movement as the frequency with which the gaze point position moves in a predetermined direction within a predetermined time period increases.

2. The optical device described in claim 1, characterized in that when the analysis means determines that the gaze point position shows the specified movement, the analysis means corrects the gaze point position using a correction vector obtained by multiplying the movement velocity vector of the gaze point position by a correction coefficient based on the evaluation value.

3. 2. The optical device according to claim 1, wherein the analysis means increases the evaluation value for the predetermined movement as the frequency with which the gaze point position moves back and forth between two different positions within a predetermined time increases.

4. The optical device described in claim 3, characterized in that when the analysis means determines that the gaze point position shows the predetermined movement, the analysis means corrects the gaze point position using a correction vector obtained by multiplying the vector between the two different positions by a correction coefficient based on the evaluation value.

5. The apparatus further includes an acquisition means for acquiring calibration data relating to a deviation between the index displayed on the display means and the gaze point position estimated by the estimation means, 5. The optical device according to claim 1, wherein the analysis means corrects the calibration data based on at least one of the position history and the rotation angle history.

6. The optical device according to any one of claims 1 to 5, characterized in that the analysis means does not correct the position of the index displayed on the display means when the gaze point position estimated by the estimation means moves and reaches a predetermined position and a predetermined operation is performed on the optical device.

7. The optical device according to any one of claims 1 to 6, characterized in that the analysis means does not correct the position of the index displayed on the display means when the gaze point position estimated by the estimation means moves and reaches a predetermined position and a subject is detected in the vicinity of the predetermined position.

8. 8. The optical device according to claim 1, further comprising an eye image pickup element for photoelectrically converting an eye image formed by the lens.

9. an imaging element that photoelectrically converts a subject image formed by an imaging optical system; An imaging device comprising: the optical device according to claim 1 .

10. an estimation step of estimating a gaze point position of the user from an image of the user's eyeball; a display step of displaying an indicator indicating the gaze point position estimated in the estimation step; an analysis step of analyzing at least one of a position history of the gaze point position or a rotation angle history of the eyeball; correcting the position of the index based on at least one of the position history and the rotation angle history; a step of correcting a position of the index displayed in the display step when it is determined that the gaze point position shows a predetermined movement based on at least one of the position history or the rotation angle history; a step of increasing an evaluation value relating to the predetermined movement as the frequency with which the gaze point position moves in a predetermined direction within a predetermined time period increases.

11. A program causing a computer to execute the control method according to claim 10.

Images