Line-of-sight information acquisition device, imaging device, line-of-sight information acquisition method, program, and storage medium

Abstract

To provide a technology which can suitably reduce change of visual line information due to involuntary eye movement.SOLUTION: A visual line information acquisition device comprises first acquisition means which acquires visual line information which is information on a visual line of an eye seeing a captured image, and reduction means which reduces change of the visual line information due to involuntary eye movement. The reduction means, if a focal distance when the image is captured is short, reduces the change of the visual line information due to the involuntary eye movement, by a greater reduction degree than when the focal distance is long.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present invention relates to a gaze information acquisition device, an imaging device, a gaze information acquisition method, a program, and a storage medium.

Background Art

[0002] In recent years, the automation and intelligentization of imaging devices have advanced, and devices have been proposed that select positions in the space in the direction the photographer is looking, positions on the screen, etc. based on the gaze information (gaze information) of the photographer looking through the viewfinder without manually inputting the position.

[0003] One of the error factors in gaze detection is the physiological eye movement of humans. For example, even when a person intends to look at a single point, the human eyeball involuntarily shakes slightly. Such involuntary movement is called fixation microsaccade.

[0004] In Patent Document 1, a configuration is proposed for determining whether the movement of the operator's eyeball is a fixation microsaccade or a voluntary movement / saccade based on the moving distance of the pupil center position within a predetermined time. Further, in Patent Document 1, based on the determination result, a configuration is proposed for changing the number of data of the pupil center position, calculating the average value of the pupil center position, and detecting the gaze direction based on this calculation result.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the configuration of Patent Document 1, when the size and movement amount of the object the operator is looking at change dynamically, the accuracy of gaze detection decreases due to fixation microsaccades, and unintended changes occur in the gaze information.

[0007] An object of the present invention is to provide a technique capable of suitably reducing changes in gaze information caused by fixational eye movement. [Means for solving the problem]

[0008] A first aspect of the present invention includes a first acquisition means for acquiring gaze information, which is information on the gaze of an eye that views a captured image, and a reduction means for reducing a change in the gaze information due to fixational eye movement, the reduction means being configured to reduce a change in the gaze information due to fixational eye movement when the focal length when the image is captured is short. as much as This is a gaze information acquisition device that reduces the change in the gaze information caused by the fixational eye movement with a large degree of reduction.

[0010] The present invention 2 The aspect of the present invention is a method for acquiring information on a line of sight by a camera, comprising: The information acquisition device is an imaging device characterized by controlling the image captured by the imaging means to be displayed on a display unit, and detecting the line of sight of an eye viewing the image.

[0011] The present invention 3 The aspect of the present invention includes an acquisition step of acquiring gaze information, which is information on the gaze of an eye that views a captured image, and a reduction step of reducing a change in the gaze information due to fixational eye movement, and in the reduction step, a focal length when capturing the image is short. as much as This is a method for acquiring gaze information, characterized in that the change in the gaze information caused by the fixational eye movement is reduced to a large extent.

[0013] The present invention 4 The present invention is also directed to a program for causing a computer to function as each of the means of the above-mentioned line-of-sight information acquisition device. 5 The third aspect is a computer-readable storage medium storing a program for causing a computer to function as each of the means of the line-of-sight information acquisition device described above. Effect of the Invention

[0014] According to the present invention, it is possible to suitably reduce changes in line-of-sight information due to fixation microsaccades.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Modes for Carrying Out the Invention

[0016] <<Embodiment 1>> Hereinafter, Embodiment 1 of the present invention will be described with reference to the accompanying drawings.

[0017] <Description of the Configuration> Figs. 1(A) and 1(B) show the appearance of the camera 1 (digital still camera; interchangeable-lens camera) according to Embodiment 1. Fig. 1(A) is a front perspective view, and Fig. 1(B) is a rear perspective view. As shown in Fig. 1(A), the camera 1 has a photographing lens unit 1A and a camera body 1B. A release button 5, which is an operation member for receiving a photographing operation from a user (photographer), is arranged on the camera body 1B. As shown in Fig. 1(B), an eyepiece lens 12 (eyepiece optical system) for a user to look into a display device 10 (display panel), which will be described later and is included in the camera body 1B, is arranged on the rear surface of the camera body 1B. Note that The eyepiece optical system may include a plurality of lenses. Operation members 41 to 43 for receiving various operations from a user are also arranged on the rear surface of the camera body 1B. For example, the operation member 41 is a touch panel that receives a touch operation, the operation member 42 is an operation lever that can be pushed down in each direction, and the operation member 43 is a four-way key that can be pushed in each of the four directions. The operation member 41 (touch panel) includes a display panel such as a liquid crystal panel and has a function of displaying an image on the display panel.

[0018] Fig. 2 is a cross-sectional view of the camera 1 cut along the YZ plane formed by the Y-axis and the Z-axis shown in Fig. 1(A), and shows a general internal configuration of the camera 1.

[0019] Inside the imaging lens unit 1A, there are included two lenses 101 and 102, a diaphragm 111, a diaphragm driving unit 112, a lens driving motor 113, a lens driving member 114, a photocoupler 115, a pulse plate 116, mount contacts 117, a focus adjustment circuit 118, etc. The lens driving member 114 consists of a driving gear, etc. The photocoupler 115 detects the rotation of the pulse plate 116 interlocked with the lens driving member 114 and transmits it to the focus adjustment circuit 118. The focus adjustment circuit 118 drives the lens driving motor 113 based on the information from the photocoupler 115 and the information from the camera body 1B (information on the lens driving amount), and moves the lens 101 to change the focusing position. The mount contacts 117 are the interface between the imaging lens unit 1A and the camera body 1B. For simplicity, two lenses 101 and 102 are shown, but actually, more than two lenses are included in the imaging lens unit 1A.

[0020] Inside the camera body 1B, there are included an imaging element 2, a CPU 3, a memory unit 4, a display device 10, a display device driving circuit 11, etc. The imaging element 2 is arranged at the planned imaging plane of the imaging lens unit 1A. The CPU 3 is the central processing unit of a microcomputer and controls the entire camera 1. The memory unit 4 stores images captured by the imaging element 2, etc. The display device 10 is composed of liquid crystal, etc., and displays the captured image (subject image) etc. on the display surface of the display device 10. The display device driving circuit 11 drives the display device 10. The user can view the image (such as the image captured by the imaging element 2) displayed on the display surface of the display device 10 through the eyepiece 12.

[0021] Inside the camera housing 1B, there are also included a light sources 13a, 13b, a beam splitter 15, a light receiving lens 16, an eye imaging device 17, etc. The light sources 13a, 13b are light sources for illuminating the user's eyeball 14. The light sources 13a, 13b have conventionally been used in a single-lens reflex camera or the like for detecting the line-of-sight direction (the direction of the line of sight; the direction the user is looking at) from the relationship between the reflected image (corneal reflection image; Purkinje image) due to the corneal reflection of light and the pupil. Specifically, the light sources 13a, 13b are infrared light-emitting diodes or the like that emit infrared light imperceptible to the user, and are arranged around the eyepiece lens 12. The optical image of the illuminated eyeball 14 (eyeball image; an image formed by the reflected light emitted from the light sources 13a, 13b and reflected by the eyeball 14) passes through the eyepiece lens 12 and is reflected by the beam splitter 15. Then, the eyeball image is formed on the eye imaging device 17 in which a two-dimensional array of photoelectric elements such as CCD or CMOS is arranged by the light receiving lens 16. The light receiving lens 16 positions the pupil of the eyeball 14 and the eye imaging device 17 in a conjugate imaging relationship. The line-of-sight direction of the eyeball 14 is detected from the position of the corneal reflection image in the eyeball image formed on the eye imaging device 17 by a predetermined algorithm described later.

[0022] Figure 3 is a block diagram showing the electrical configuration inside the camera 1. Connected to the CPU 3 are a line-of-sight detection circuit 201, a photometry circuit 202, an autofocus detection circuit 203, a signal input circuit 204, a display device drive circuit 11, a light source drive circuit 205, etc. Further, the CPU 3 transmits signals to a focus adjustment circuit 118 arranged inside the photographic lens unit 1A and a diaphragm control circuit 206 included in the diaphragm drive unit 112 inside the photographic lens unit 1A via the mount contact 117. The memory unit 4 associated with the CPU 3 has a function of storing imaging signals from the imaging device 2 and the eye imaging device 17, and a function of storing line-of-sight correction parameters for correcting individual differences in the line of sight described later.

[0023] ​The line-of-sight detection circuit 201 A / D-converts the output of the eye imaging element 17 (the eye image obtained by imaging the eye) in a state where the eye image is formed on the eye imaging element 17, and transmits the result to the CPU 3. The CPU 3 extracts feature points necessary for line-of-sight detection from the eye image according to a predetermined algorithm described later, and calculates the user's viewpoint (line-of-sight position; the position where the line of sight is directed; the position where the user is looking) on the display surface of the display device 10 from the positions of the feature points.

[0024] The photometry circuit 202 performs amplification, logarithmic compression, A / D conversion, etc. on the signal obtained from the imaging element 2 that also serves as a photometry sensor, specifically, the luminance signal corresponding to the brightness of the subject, and sends the result to the CPU 3 as subject luminance information.

[0025] The autofocus detection circuit 203 A / D-converts the signal voltages from a plurality of detection elements (a plurality of pixels) included in the imaging element 2 and used for phase difference detection, and sends them to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to each focus detection point from the signals of the plurality of detection elements. This is a known technique known as imaging plane phase difference AF. In Embodiment 1, as an example, it is assumed that there are focus detection points at 180 locations on the imaging plane corresponding to 180 locations shown on the display surface of the display device 10 in the finder internal view (the view when looking into the finder) of Fig. 4(A).

[0026] The switch SW1 and the switch SW2 are connected to the signal input circuit 204. The switch SW1 is turned on at the first stroke of the release button 5, and is a switch for starting the photometry, distance measurement, line-of-sight detection operations, etc. of the camera 1. The switch SW2 is turned on at the second stroke of the release button 5, and is a switch for starting the shooting operation. The ON signals from the switches SW1 and SW2 are input to the signal input circuit 204 and transmitted to the CPU 3.

[0027] Figure 4(A) is a diagram showing the viewfinder internal field of view, and shows the state in which the display device 10 is operating (the state in which an image is displayed). As shown in Figure 4(A), in the viewfinder internal field of view, there are a focus detection area 400, 180 distance measurement point indicators 401, a field mask 402, and the like. Each of the 180 distance measurement point indicators 401 is displayed superimposed on the through image (live view image) displayed on the display device 10 so as to be displayed at a position corresponding to the focus detection point on the imaging surface. Further, among the 180 distance measurement point indicators 401, the distance measurement point indicator 401 corresponding to the current viewpoint A (estimated position) is highlighted and displayed with a frame or the like.

[0028] In addition, as an example of an item indicating the current viewpoint (estimated position), an example of displaying a frame or the like that highlights the distance measurement point indicator corresponding to the viewpoint has been described, but it is not limited to this. For example, as shown in Figure 4(D), the item indicating the current viewpoint may be an item (gaze pointer) that only indicates the viewpoint, rather than an item that discriminably indicates the distance measurement point indicator corresponding to the viewpoint. In that case, both the display of the item indicating the current viewpoint and the highlighting display of the distance measurement point indicator corresponding to the viewpoint and various buttons may be performed. The display position of the item indicating the current viewpoint may change continuously or may change stepwise.

[0029] <Explanation of the gaze detection operation> The gaze detection method will be described with reference to Figures 5, 6(A), 6(B), and 7. Figure 5 is a diagram for explaining the principle of the gaze detection method, and is a schematic diagram of an optical system for performing gaze detection. As shown in Figure 5, the light sources 13a and 13b are arranged substantially symmetrically with respect to the optical axis of the light receiving lens 16 and illuminate the user's eyeball 14. A part of the light emitted from the light sources 13a and 13b and reflected by the eyeball 14 is condensed by the light receiving lens 16 onto the eye imaging device 17. Figure 6(A) is a schematic diagram of the eye image (the eyeball image projected onto the eye imaging device 17) captured by the eye imaging device 17, and Figure 6(B) is a diagram showing the output intensity of the CMOS in the eye imaging device 17. Figure 7 is a flowchart of the gaze detection operation. A schematic diagram, and Figure 6(B) is a diagram showing the output intensity of the CMOS in the eye imaging device 17. Figure 7 is a flowchart of the gaze detection operation.

[0030] When the line-of-sight detection operation starts, at step S701 in FIG. 7, the light sources 13a and 13b emit infrared light toward the user's eyeball 14. The user's eyeball image illuminated by the infrared light is imaged on the eye image sensor 17 through the light receiving lens 16 and is photoelectrically converted by the eye image sensor 17. Thereby, an electrical signal of an eye image that can be processed is obtained.

[0031] At step S702, the line-of-sight detection circuit 201 sends the eye image (eye image signal; electrical signal of the eye image) obtained from the eye image sensor 17 to the CPU 3.

[0032] At step S703, the CPU 3 obtains the coordinates of the points corresponding to the corneal reflection images Pd and Pe of the light sources 13a and 13b and the pupil center c from the eye image obtained at step S702.

[0033] The infrared light emitted from the light sources 13a and 13b illuminates the cornea 142 of the user's eyeball 14. At this time, the corneal reflection images Pd and Pe formed by a part of the infrared light reflected on the surface of the cornea 142 are condensed by the light receiving lens 16 and are imaged on the eye image sensor 17 to become the corneal reflection images Pd' and Pe' in the eye image. Similarly, the light beams from the ends a and b of the pupil 141 are also imaged on the eye image sensor 17 to become the pupil end images a' and b' in the eye image.

[0034] FIG. 6(B) shows the luminance information (luminance distribution) of region α in the eye image of FIG. 6(A). In FIG. 6(B), the horizontal direction of the eye image is the X-axis direction, the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. In Embodiment 1, the coordinates in the X-axis direction (horizontal direction) of the corneal reflection images Pd' and Pe' are Xd and Xe, and the coordinates in the X-axis direction of the pupil edge images a' and b' are Xa and Xb. As shown in FIG. 6(B), extremely high levels of luminance are obtained at the X coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. In the region from the X coordinate Xa to the X coordinate Xb, which corresponds to the region of the pupil 141 (the region of the pupil image obtained by imaging the light beam from the pupil 141 on the eye imaging device 17), extremely low levels of luminance are obtained except at the X coordinates Xd and Xe. And in the region of the iris 143 outside the pupil 141 (the region of the iris image outside the pupil image obtained by imaging the light beam from the iris 143), luminance intermediate between the above two kinds of luminance is obtained. Specifically, luminance intermediate between the above two kinds of luminance is obtained in the region where the X coordinate (coordinate in the X-axis direction) is smaller than the X coordinate Xa and in the region where the X coordinate is larger than the X coordinate Xb.

[0035] From the luminance distribution as shown in FIG. 6(B), the X coordinates Xd and Xe of the corneal reflection images Pd' and Pe', and the X coordinates Xa and Xb of the pupil edge images a' and b' can be obtained. Specifically, the coordinates with extremely high luminance can be obtained as the coordinates of the corneal reflection images Pd' and Pe', and the coordinates with extremely low luminance can be obtained as the coordinates of the pupil edge images a' and b'. Also, when the rotation angle θx of the optical axis of the eyeball 14 with respect to the optical axis of the light receiving lens 16 is small, the X coordinate Xc of the pupil center image c' (the center of the pupil image) obtained by imaging the light beam from the pupil center c on the eye imaging device 17 can be expressed as Xc≒(Xa + Xb) / 2. That is, the X coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa and Xb of the pupil edge images a' and b'. In this way, the coordinates of the corneal reflection images Pd' and Pe', and the coordinates of the pupil center image c' can be estimated.

[0036] In step S704, the CPU 3 calculates the imaging magnification β of the eye image. The imaging magnification β is a magnification determined by the position of the eyeball 14 with respect to the light receiving lens 16, and can be obtained using a function of the interval (Xd - Xe) between the corneal reflection images Pd' and Pe'.

[0037] In step S705, the CPU 3 calculates the rotation angle of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16 as information on the line of sight of the eyeball 14. The X coordinate of the midpoint between the corneal reflection images Pd and Pe and the X coordinate of the center of curvature O of the cornea 142 are almost the same. If the standard distance from the center of curvature O of 42 to the center c of the pupil 141 is Oc, the rotation angle θx of the eyeball 14 in the ZX plane (plane perpendicular to the Y axis) can be calculated by the following formula 1. The rotation angle θy of the eyeball 14 in the ZY plane (plane perpendicular to the X axis) can also be calculated in a similar manner to the method for calculating the rotation angle θx. β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc (Formula 1)

[0038] In step S706, the CPU 3 uses the rotation angles θx, θy calculated in step S705 to determine (estimate) the user's viewpoint on the display surface of the display device 10. If the coordinates (Hx, Hy) of the viewpoint are coordinates corresponding to the pupil center c, the coordinates (Hx, Hy) of the viewpoint can be calculated by the following formulas 2 and 3. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)

[0039] Parameter m in Equations 2 and 3 is a constant determined by the configuration of the viewfinder optical system (light receiving lens 16, etc.) of camera 1, and is a conversion coefficient that converts rotation angles θx, θy into coordinates corresponding to the pupil center c on the display surface of display device 10. Parameter m is determined in advance and stored in memory unit 4. Parameters Ax, Bx, Ay, and By are gaze correction parameters that correct individual differences in gaze, and are acquired by performing a calibration operation described later. Parameters Ax, Bx, Ay, and By are stored in memory unit 4 before the gaze detection operation starts.

[0040] In step S707, the CPU 3 stores the coordinates (Hx, Hy) of the viewpoint in the memory unit 4 and ends the gaze detection operation.

[0041] Although an example using a corneal reflection image has been described, the method of gaze detection is not limited to this, and any method of detecting a gaze based on an eye image may be used. Also, although an example of obtaining the coordinates (Hx, Hy) of the viewpoint as the result (final result) of gaze detection has been described, any information related to the gaze (gaze information) such as the rotation angles θx, θy, etc. may be obtained as the result of gaze detection.

[0042] <Explanation of calibration work> Although the gaze can be estimated by the above-described gaze detection operation, due to factors such as individual differences in the shape of the human eyeball, it may not be possible to estimate the gaze with high accuracy. Specifically, if the gaze correction parameters Ax, Ay, Bx, By are not adjusted to values suitable for the user, as shown in FIG. 4(B), a deviation will occur between the actual viewpoint B and the estimated viewpoint C. In FIG. 4(B), although the user is gazing at a person, the camera 1 erroneously estimates that the background is being gazed at, and thus falls into a state where appropriate focus detection and adjustment cannot be performed.

[0043] Therefore, before the camera 1 performs imaging, it is necessary to perform a calibration operation to obtain gaze correction parameters suitable for the user and store them in the camera 1.

[0044] Conventionally, the calibration operation has been performed by highlighting a plurality of indicators at different positions as shown in FIG. 4(C) on the display surface of the display device 10 before imaging and having the user look at the indicators. Then, a technique for obtaining gaze correction parameters suitable for the user from the calculated plurality of viewpoints (estimated positions) and the coordinates of each indicator during the gaze detection operation when each indicator is gazed at is known as a known technique. Note that if the position where the user should look is suggested, the display method of the indicator is not particularly limited, and a graphic that is an indicator may be displayed, or an indicator may be displayed by changing the luminance or color of an image (such as a captured image).

[0045] <Explanation of Human Physiological Eye Movements> One of the error factors in gaze detection is human physiological eye movements. For example, even when a person intends to look at a point, their eyes involuntarily shake slightly. Such involuntary movements are called saccadic microtremors. Since the change in gaze due to saccadic microtremors is not intended by the user, saccadic microtremors become an error factor in gaze detection.

[0046] Therefore, in Embodiment 1, the change in gaze information (detected gaze) due to saccadic microtremors is reduced (saccadic microtremor reduction processing). The saccadic microtremor reduction processing is performed, for example, by the CPU 3. The saccadic microtremor reduction processing is not particularly limited, but in Embodiment 1, it is assumed to be a low-pass filter processing that attenuates the components in a specific frequency band (a frequency band higher than the cut-off frequency) in the detected change in gaze.

[0047] Figs. 8(A) and 8(B) are graphs showing the time change of the X coordinate of the detected viewpoint. The vertical axis of Figs. 8(A) and 8(B) indicates the X coordinate of the detected viewpoint, and the horizontal axis of Figs. 8(A) and 8(B) indicates time. Fig. 8(A) shows the state before the low-pass filter processing, which is the saccadic microtremor reduction processing, and Fig. 8(B) shows the state after the low-pass filter processing. It can be seen from Fig. 8(A) that before the low-pass filter processing, the X coordinate of the detected viewpoint fluctuates violently (finely). As shown in Fig. 8(B), by performing the low-pass filter processing, the high-frequency components of the fluctuation of the detected viewpoint (the fluctuation in Fig. 8(A)) are attenuated, and the fluctuation of the X coordinate of the detected viewpoint is reduced.

[0048] However, the change in line-of-sight information (detected line of sight) due to fixational microsaccades does not necessarily need to be reduced at the same reduction rate all the time. That is, the fixational microsaccade reduction process does not necessarily need to be performed with the same processing intensity all the time. Human physiological eye movements include not only fixational microsaccades. When a human follows an object moving at a slow speed with the eyes, the person performs an eye movement (a type of voluntary movement) that smoothly moves the line of sight. When following an object moving at a high speed with the eyes, the person tends to perform a saccade that rapidly changes the line of sight. When the fixational microsaccade reduction process is performed, the change in the detected line of sight (line-of-sight information) becomes dull compared to the actual change in the line of sight (eye movement). That is, the followability of the detected line of sight with respect to the object decreases. Therefore, if the processing intensity of the fixational microsaccade reduction process is strong (if the reduction rate due to the fixational microsaccade reduction process is large), the line-of-sight information will not accurately represent voluntary movements, saccades, etc.

[0049] Therefore, in Embodiment 1, the CPU 3 changes the intensity of the fixational microsaccade reduction process (the reduction rate due to the fixational microsaccade reduction process) so as to preferably reduce the change in line-of-sight information due to fixational microsaccades. The details will be described later. The line-of-sight pointer, etc. is displayed so as to indicate the viewpoint after the fixational microsaccade reduction process.

[0050] <Explanation of Control of Fixational Microsaccade Reduction Process> In Embodiment 1, the CPU 3 changes the intensity of the fixational microsaccade reduction process (the reduction rate due to the fixational microsaccade reduction process) according to the focal length (the zoom magnification of the camera 1). When the focal length is changed, for example, by changing the imaging lens unit 1A, information about the change in the focal length and information (value) about the focal length are transmitted from the focus adjustment circuit 118 in the imaging lens unit 1A to the CPU 3 in the camera housing 1B. However, it is not limited to this. For example, when the correspondence between the imaging lens unit 1A and the focal length is one-to-one, the memory unit 4 in the camera housing 1B may store the correspondence between the identification information (model number, etc.) of the imaging lens unit 1A and the focal length. In that case, the imaging lens unit 1A transmits the identification information of the imaging lens unit 1A to the CPU 3, and the CPU 3 uses the received identification information and the information stored in the memory unit 4 The current focal length information may be determined from the obtained information (the correspondence between the identification information and the focal length).

[0051] When the focal length is short (when the zoom ratio is low), the angle of view (imaging range) is wide, so the size of the object (subject) relative to the angle of view tends to be small. In other words, the object tends to be small in the captured image. And when the object is small in the captured image, the relative magnitude (shaking amount) of fixational eye movement to the size of the object becomes large.

[0052] From the above viewpoints, when the focal length is short, the intensity of the fixation eye movement reduction process is increased. In other words, the degree of reduction by the fixation eye movement reduction process is increased. Specifically, the cutoff frequency of the low-pass filter process, which is the fixation eye movement reduction process, is lowered (shifted to the lower frequency side). This makes it possible to stabilize the gaze information (detected gaze).

[0053] This makes the change in the detected line of sight (line of sight information) slower compared to the actual change in line of sight (eye movement). However, if the object is small in the captured image, the amount of movement of the object relative to the angle of view tends to be small, and the object can continue to be captured by the detected line of sight, so this is not a problem.

[0054] On the other hand, when the focal length is long (when the zoom ratio is high), the angle of view is narrow, so the size of the object (subject) tends to be large relative to the angle of view. In other words, the object tends to be large in the captured image. And when the object is large in the captured image, the amount of movement of the object relative to the angle of view tends to be large.

[0055] From the above viewpoints, when the focal length is long, the intensity of the fixation micro-vibration reduction process is weakened. In other words, the degree of reduction by the fixation micro-vibration reduction process is made smaller. Specifically, the cut-off frequency of the low-pass filter process, which is the fixation micro-vibration reduction process, is increased (shifted to the high-frequency side). As a result, the responsiveness of the detected line of sight (line of sight information) to the actual change in the line of sight (eye movement) increases, and the followability of the detected line of sight to the object improves.

[0056] In this way, the stability of the line of sight information (detected line of sight) decreases. However, if the object is large in the captured image, even if the detected line of sight is slightly blurred, the object can continue to be captured with the line of sight (the detected viewpoint can be kept within the area of the object), so there is no problem.

[0057] In Embodiment 1, in order to realize the above-described control, information (such as a function or a table) indicating the correspondence relationship between the focal length and the cut-off frequency is stored in advance in the memory unit 4. FIG. 9 is a graph showing the correspondence relationship between the focal length and the cut-off frequency. The vertical axis of FIG. 9 indicates the cut-off frequency, and the horizontal axis of FIG. 9 indicates the focal length. According to the correspondence relationship in FIG. 9, the cut-off frequency of the low-pass filter process, which is the fixation micro-vibration reduction process, becomes lower as the focal length becomes shorter. Therefore, the degree of reduction by the fixation micro-vibration reduction process becomes larger as the focal length becomes shorter.

[0058] Note that in FIG. 9, the cut-off frequency increases linearly with the increase in the focal length. However, the cut-off frequency may increase non-linearly with the increase in the focal length. The cut-off frequency may increase stepwise (discontinuously) with the continuous increase in the focal length. Similarly, the degree of reduction by the fixation micro-vibration reduction process may decrease linearly or non-linearly with the increase in the focal length. The degree of reduction by the fixation micro-vibration reduction process may decrease stepwise (discontinuously) with the continuous increase in the focal length.

[0059] Note that the fixation tremor reduction process is not limited to low-pass filter processing. For example, the fixation tremor reduction process may be another filter processing that attenuates components in a specific frequency band, such as band-pass filter processing. The fixation tremor reduction process may be Kalman filter processing, which is a sequential Bayesian filter processing. For example, in Kalman filter processing, the posterior estimate is calculated using the following Equation 4. The posterior estimate corresponds to the line-of-sight information after the current fixation tremor reduction process, the prior estimate corresponds to the line-of-sight information after the previous fixation tremor reduction process, and the observation value corresponds to the line-of-sight information before the current fixation tremor reduction process. [Number]

[0060] As shown in FIG. 10, the posterior estimate corresponds to the midpoint between the prior estimate and the observation value, and the mixing ratio is k:1-k (0 < k < 1). Increasing the uncertainty of the observation, that is, decreasing k, causes the posterior estimate to approach the prior estimate. As described above, the prior estimate corresponds to the line-of-sight information after the previous fixation tremor reduction process. Therefore, by increasing the uncertainty of the observation, the change in the line-of-sight information (detected line of sight) due to fixation tremor can be significantly reduced. From this, by increasing the uncertainty of the observation when the focal length is short, the line-of-sight information can be stabilized so that small objects in the captured image can be captured by the line of sight.

[0061] On the other hand, increasing the uncertainty of the prediction, that is, increasing k, causes the posterior estimate to approach the observation value. As described above, the observation value corresponds to the line-of-sight information before the current fixation tremor reduction process. Therefore, increasing the uncertainty of the prediction cannot significantly reduce the change in the line-of-sight information (detected line of sight) due to fixation tremor, but can increase the responsiveness of the change in the detected line of sight to the actual change in the line of sight (eye movement). From this, by increasing the uncertainty of the prediction when the focal length is long, the followability of the detected line of sight with respect to an object with a large amount of movement in the captured image can be improved.

[0062] [Summary] As described above, in the first embodiment, the degree of reduction by the fixation eye movement reduction process is changed depending on the focal length, thereby making it possible to suitably reduce changes in gaze information due to fixation eye movement.

[0063] <<Embodiment 2>> A second embodiment of the present invention will be described below. Note that, in the following, a description of the same points (such as configuration and processing) as in the first embodiment will be omitted, and only points different from the first embodiment will be described. In the second embodiment, in addition to the focal length, the fixation eye movement reduction process is controlled taking into consideration the depth to the object (subject) in the captured image (object distance; subject distance). Note that, although an example in which both the focal length and the object distance are taken into consideration will be described, only the object distance may be taken into consideration.

[0064] FIG. 11 is a flowchart showing the control of the second embodiment.

[0065] In step S1101, the CPU 3 in the camera housing 1B acquires information (value) of the focal length from the focus adjustment circuit 118 in the photographing lens unit 1A.

[0066] In step S1102, the CPU 3 calculates the distance from the camera 1 (image capture position) to the object. For example, the CPU 3 detects a phase difference or contrast using the image sensor 2 to obtain a focal position, and obtains the object distance based on the focal position. Note that the method of obtaining the object distance information is not limited to this, and for example, the object distance information may be obtained using a distance measuring sensor.

[0067] In step S1103, the CPU 3 determines the cutoff frequency in accordance with the information (focal length and object distance) acquired in steps S1101 and S1102.

[0068] In step S1104, the CPU 3 changes the cutoff frequency of the low-pass filter process, which is the fixation eye movement reduction process, to the cutoff frequency determined in step S1103.

[0069] Assuming that conditions other than the object distance (e.g., focal length) are constant, when the object distance is short, the size of the object in the captured image is larger than when the object distance is long. From this perspective, when the focal length is constant, if the object distance is long, the reduction degree by the fixation jitter reduction process is increased, and if the object distance is short, the reduction degree by the fixation jitter reduction process is decreased.

[0070] In Embodiment 2, in order to realize the above-described control, information (such as a function or a table) indicating the correspondence relationship among the focal length, the object distance, and the cut-off frequency is stored in advance in the memory unit 4. FIG. 12 is a graph showing the correspondence relationship among the focal length, the object distance, and the cut-off frequency. The vertical axis in FIG. 12 indicates the cut-off frequency, and the horizontal axis in FIG. 12 indicates the object distance. In FIG. 12, the correspondence relationship between the object distance and the cut-off frequency is shown for each focal length. In each correspondence relationship, the cut-off frequency becomes lower as the object distance becomes longer. Therefore, when the focal length is constant, the reduction degree by the fixation jitter reduction process becomes larger as the object distance becomes longer.

[0071] In FIG. 12, the cut-off frequency decreases linearly as the object distance increases. However, the cut-off frequency may decrease non-linearly as the object distance increases. The cut-off frequency may decrease stepwise (discontinuously) with the continuous increase of the object distance. Similarly, the reduction degree by the fixation jitter reduction process may decrease linearly or non-linearly as the object distance increases. The reduction degree by the fixation jitter reduction process may decrease stepwise (discontinuously) with the continuous increase of the object distance.

[0072] Note that also in Embodiment 2, the fixation jitter reduction process is not limited to the low-pass filter process, and may be a band-pass filter process, a Kalman filter process, or the like. In the case of the Kalman filter process, when the focal length is constant, if the object distance is short, the uncertainty of prediction may be increased, and if the object distance is long, the uncertainty of observation may be increased. That is, when the focal length is constant, if the object distance is short, k may be increased, and if the object distance is long, k may be decreased.

[0073] As described above, in Embodiment 2, in consideration of the object distance, the degree of reduction by the fixation micro-vibration reduction process is changed. Thereby, the change in the line-of-sight information due to the fixation micro-vibration can be suitably reduced. Further, by considering both the focal length and the object distance, the change in the line-of-sight information due to the fixation micro-vibration can be reduced more suitably than in the case of considering only one of them.

[0074] Note that Embodiments 1 and 2 are merely examples, and configurations obtained by appropriately modifying or changing the configurations of Embodiments 1 and 2 within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the configurations of Embodiments 1 and 2 are also included in the present invention.

[0075] For example, although an example of applying the present invention to an imaging device (camera) has been described, the present invention is not limited to this. That is, the line-of-sight information acquisition device of the present invention may be a device separate from the imaging device having an imaging unit and may be a separate device from the line-of-sight information acquisition device. The display unit that displays the image viewed by the user may be a separate device from the line-of-sight information acquisition device, or the control unit (CPU) of the line-of-sight information acquisition device may control the display unit, which is an external device, to display an image, an item, etc. The image viewed by the user may be an image different from the captured image, such as a three-dimensional computer graphic. The line-of-sight information acquisition device of the present invention only needs to have at least a function of acquiring line-of-sight information, a function of performing a fixation micro-vibration reduction process, and a function of changing the degree of reduction by the fixation micro-vibration reduction process. Further, although an example in which the fixation micro-vibration reduction process is a process of correcting (adjusting; changing) the acquired line-of-sight information so that the change in the line-of-sight information due to the fixation micro-vibration is reduced has been described, the present invention is not limited to this. For example, the fixation micro-vibration reduction process may be a process of changing the method of acquiring line-of-sight information so that the change in the line-of-sight information due to the fixation micro-vibration is reduced.

[0076] <<Examples of Application to Other Electronic Devices>> FIG. 13A is an external view of a notebook type personal computer 1310 (notebook PC) to which the present invention is applied. In FIG. 13A, an imaging unit 1315 for imaging a user looking at a display unit 1311 of the notebook PC 1310 is connected to the notebook PC 1310, and the notebook PC 1310 acquires an imaging result from the imaging unit 1315. The notebook PC 1310 detects the user's line of sight based on the imaging result, and changes the degree of reduction by the fixation eye movement reduction process according to the focal length and object distance corresponding to the image displayed on the display unit 1311. Note that the electronic device to which the present invention is applied only needs to have an interface for receiving the result of the line of sight detection (line of sight information) as a line of sight input, and the line of sight detection may be performed by an external device. In other words, the imaging unit 1315 may perform the line of sight detection, and the notebook PC 1310 may acquire the result of the line of sight detection from the imaging unit 1315.

[0077] Fig. 13B is an external view of a smartphone 1320 to which the present invention is applied. In Fig. 13B, the smartphone 1320 detects the user's line of sight based on the imaging result of an in-camera 1321 (front camera), and changes the reduction degree by fixation eye movement reduction processing according to the focal length and object distance corresponding to the image displayed on a display unit 1322.

[0078] Figure 13(C) is an external view of a game machine 1330 to which the present invention is applied. In Figure 13(C), a head-mounted display 1335 (HMD) for displaying VR (Virtual Reality) images of a game is connected to the game machine 1330. The HMD 1335 has a camera 1337 for imaging the eyes of a user wearing the HMD 1335, and the game machine 1330 acquires the imaging result from the HMD 1335. Then, the game machine 1330 detects the user's line of sight based on the imaging result, and changes the degree of reduction by fixation micro-vibration reduction processing according to the depth (object distance) to an object in the VR image displayed on the display unit 1336 of the HMD 1335. Here, the object in the VR image is, for example, an object existing in the direction closest to the user's line of sight direction. Similar to the case where the present invention is applicable when viewing the VR image displayed on the HMD, the present invention is also applicable when viewing an AR (Augmented Reality) image displayed on, for example, the lens portion of a glasses-type wearable terminal. Similar to the case where the present invention is applicable to VR technology and AR technology, the present invention is also applicable to other xR technologies such as MR (Mixed Reality) technology and SR (Substitutional Reality) technology.

[0079] Note that the imaging unit 1315 in Fig. 13(A), the HMD 1335 in Fig. 13(C), etc. may change the degree of reduction by fixation micro-vibration reduction processing and perform the fixation micro-vibration reduction processing. That is, the present invention is also applicable to the imaging unit 1315 in Fig. 13(A), the HMD 1335 in Fig. 13(C), etc.

[0080] <<Other Embodiments>> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Explanation of Reference Numerals

[0081] 1: Camera 3: CPU 201: Gaze detection circuit 1310: Notebook personal computer 1320: Smartphone 1330: Game console 1335: Head-mounted display

Claims

1. A first acquisition means for acquiring line-of-sight information which is information on a line of sight of an eye viewing a captured image; A reduction means for reducing a change in the line of sight information due to fixational eye movement; having The reduction means reduces the change in the line of sight information caused by the fixational eye movement to a greater degree as the focal length when capturing the image is shorter. A gaze information acquisition device comprising:

2. The imaging device further includes a second acquisition unit for acquiring information on the focal length.

2. The line-of-sight information acquisition device according to claim 1.

3. The reduction means reduces the change in the line of sight information caused by the fixational eye movement to a greater degree when an object distance, which is a depth to an object in the image, is long than when the object distance is short when the focal length is constant.

3. The line-of-sight information acquisition device according to claim 1 or 2.

4. The object distance measuring device further includes a third obtaining unit for obtaining information on the object distance.

4. The line-of-sight information acquisition device according to claim 3.

5. The process of reducing the change in the gaze information due to the fixational eye movement is a process of attenuating a component of a specific frequency band in the change in the gaze.

5. The line-of-sight information acquiring device according to claim 1,

6. the process of reducing the change in the gaze information due to the fixational eye movement is a low-pass filter process of attenuating a component of the specific frequency band higher than a cutoff frequency in the change in the gaze, The reduction means increases the degree of reduction by lowering the cutoff frequency.

6. The line-of-sight information acquisition device according to claim 5.

7. The image display device further includes a display unit for displaying the image.

7. The line-of-sight information acquiring device according to claim 1,

8. A control means for controlling a display unit that displays the image so as to display an item indicating a position where the gaze is fixed after the change due to the fixational eye movement is reduced. Further having 8. The line-of-sight information acquiring device according to claim 1,

9. An imaging means; The line-of-sight information acquisition device according to any one of claims 1 to 8, having The line-of-sight information acquisition device controls the image captured by the imaging means to be displayed on a display unit, and detects the line of sight of an eye viewing the image.

1. An imaging device comprising:

10. An acquisition step of acquiring line-of-sight information which is information on the line of sight of an eye viewing a captured image; A reduction step of reducing a change in the gaze information due to fixational eye movement; having In the reducing step, the change in the line of sight information caused by the fixational eye movement is reduced to a greater degree as the focal length when capturing the image is shorter. A method for acquiring line-of-sight information.

11. A program for causing a computer to function as each of the means of the line-of-sight information acquisition device according to any one of claims 1 to 8.

12. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the line-of-sight information acquisition device according to any one of claims 1 to 8.

Images