Control device, imaging device, control method, and program

Abstract

To provide a control device capable of always performing visual line detection with high accuracy.SOLUTION: A control device (1) includes display means (28) for displaying an image, visual line acquisition means (701) for acquiring visual line information of a user observing the display means, subject detection means (50a) for detecting subject information in the image, scene determination acquisition means (50b) for acquiring scene determination information in the image, and calculation means (50c) for calculating corrected visual line location information on the basis of the visual line information, the subject information, and the scene determination information.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a control device, an imaging device, a control method, and a program. [Background technology]

[0002] Patent Document 1 discloses a method for selecting a focus point by detecting the gaze position within the shooting range of a user looking into the field of view of an optical viewfinder. Patent Document 2 discloses a method for detecting the gaze of a subject by accurately calculating the pupil center by detecting the size of the pupils of each eye and correcting the left and right gaze points according to the pupil size. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-022208 [Patent Document 2] Japanese Patent Application Laid-Open No. 2017-102688 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when taking a photograph using the viewfinder of an imaging device such as a single-lens camera, one must look through the viewfinder with one eye, and therefore the method disclosed in Patent Document 2 cannot be used. Furthermore, with an imaging device such as a single-lens camera, one looks through the viewfinder mainly only when taking a photograph, and therefore the pupil position changes depending on whether the eye is brought close to the subject or not. For this reason, unless the viewpoint is constantly corrected, the accuracy of viewpoint detection decreases.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a control device, an imaging device, a control method, and a program that are capable of always performing gaze detection with high accuracy. [Means for solving the problem]

[0006] As one aspect of the present inventionImaging The device is Live view display captured by the image sensor a display means for displaying an image; a reliability acquisition means for acquiring reliability information regarding the line of sight of a user observing the display means; Observe the display means The aforementioned User's gaze position of detection gaze Position detection means and Live View Display In the image It Subject Body detection and obtain subject information Subject Information acquisition means and Operation information of the imaging device before a line-of-sight determination operation is performed by a user, and the live view display before the line-of-sight determination operation is performed by the user image In the above, the motion of the subject is different from the subject information. a scene determination acquisition means for acquiring information; In response to the line of sight confirmation operation being performed by the user, The aforementioned Reliability information 、 The subject information 、 The aforementioned Operation information of the imaging device and operation information of the subject Based on The gaze position detected by the gaze position detecting means and a calculation means for calculating correction information of the gaze position.

[0007] Other objects and features of the present invention will be described in the following embodiments. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a control device, an imaging device, a control method, and a program that are capable of always performing gaze detection with high accuracy. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram of an imaging apparatus according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating the configuration of an eye-gaze input operation unit according to the present embodiment. [Figure 3] FIG. 10 is a configuration diagram of another eye-gaze input operation unit in the present embodiment. [Figure 4] 4 is a flowchart of a control method according to the present embodiment. [Figure 5] 10A and 10B are explanatory diagrams of eye gaze position correction in the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. [Imaging device] First, an imaging device (control device) 1 in this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram of the imaging device 1. The imaging device 1 is configured to include a digital camera (camera body) 100 and a lens unit (lens device) 150 that is detachable from the digital camera 100. However, this embodiment is not limited to this, and can also be applied to an imaging device in which the camera body and lens device are configured as an integrated unit.

[0011] Lens unit 150 is a lens unit equipped with an interchangeable photographic lens. Lens (optical system) 103 is usually composed of multiple lenses, but for simplicity, it is shown here as only one lens. Communication terminal 6 is a communication terminal that allows lens unit 150 to communicate with digital camera 100, and communication terminal 10 is a communication terminal that allows digital camera 100 to communicate with lens unit 150. Lens unit 150 communicates with system control unit 50 via communication terminals 6 and 10, controls aperture 102 via aperture drive circuit 2 using an internal lens system control circuit 4, and adjusts focus by displacing the position of lens 103 via AF drive circuit 3.

[0012] The shutter 101 is a focal plane shutter whose exposure time for the imaging unit 22 can be freely controlled by the system control unit 50. The imaging unit 22 is an imaging element formed of a CCD, CMOS, or other element that converts an optical image into an electrical signal. The A / D converter 23 converts analog signals into digital signals. The A / D converter 23 is used to convert analog signals output from the imaging unit 22 into digital signals. The signals obtained from the imaging unit 22 are used not only for imaging but also for exposure control and focus detection control. The imaging unit 22 has pixels in which a photoelectric conversion unit is divided for each microlens. By dividing the photoelectric conversion unit, the entrance pupil is divided, and a phase difference detection signal can be obtained from each photoelectric conversion unit. Furthermore, an imaging signal can be obtained by adding the signals from the divided photoelectric conversion units. Such pixels have the advantage of being able to serve as both focus detection pixels and imaging pixels.

[0013] The image processing unit 24 performs predetermined pixel interpolation, resizing such as reduction, and color conversion processing on data from the A / D converter 23 or data from the memory control unit 15. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data. The system control unit 50 performs exposure control and distance measurement control based on the arithmetic results obtained from the image processing unit 24. This results in TTL (through-the-lens) AF (autofocus) processing, AE (autoexposure) processing, and EF (flash pre-flash) processing. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data, and performs TTL AWB (auto white balance) processing based on the obtained arithmetic results.

[0014] The output data from the A / D converter 23 is written into the memory 32 via the image processing unit 24 and the memory control unit 15, or directly via the memory control unit 15. The memory 32 stores image data obtained by the imaging unit 22 and converted into digital data by the A / D converter 23, as well as image data to be displayed on the display unit (display means) 28. The memory 32 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio.

[0015] The memory 32 also serves as a memory (video memory) for image display. The D / A converter 19 converts image display data stored in the memory 32 into an analog signal and supplies it to the display unit 28. As a result, the display image data written to the memory 32 is displayed on the display unit 28 via the D / A converter 19. The display unit 28 displays an image on a display device such as an LCD in accordance with the analog signal from the D / A converter 19. The digital signal that has been A / D converted by the A / D converter 23 and stored in the memory 32 is converted to analog in the D / A converter 19 and sequentially transferred to the display unit 28 for display. As a result, the display unit 28 functions as an electronic viewfinder and can display a through image (live view display). The display unit 28 may be provided with an electronic viewfinder that is viewed through an eyepiece (not shown), or with a display on the back of the digital camera 100. Alternatively, both an electronic viewfinder and a display on the back may be provided.

[0016] The nonvolatile memory 56 is an electrically erasable and recordable memory, such as an EEPROM. The nonvolatile memory 56 stores constants, programs, etc. for the operation of the system control unit 50. The programs referred to here are programs for executing the flowcharts described later in this embodiment.

[0017] The system control unit 50 controls the entire digital camera 100. The system control unit 50 also has a subject detection unit 50a, a scene determination acquisition unit 50b, and a calculation unit 50c. The subject detection unit 50a detects subject information in an image. The scene determination acquisition unit 50b acquires scene determination information for the image. The calculation unit 50c calculates gaze position correction information based on gaze information, subject information, and scene determination information. The system control unit 50 also executes programs recorded in a nonvolatile memory 56 to realize each process of this embodiment, which will be described later.

[0018] The system memory 52 is, for example, a RAM. Constants and variables for operation of the system control unit 50, programs read from the non-volatile memory 56, and the like are loaded into the system memory 52. ​​The system control unit 50 also performs display control by controlling the memory 32, the D / A converter 19, the display unit 28, and the like. The display unit 28 is either an EVF that the user looks into or a TFT LCD screen that can be operated by touch or the like and is not looked into. An eye sensor installed in the EVF of the display unit 28 controls the display unit 28 based on information indicating whether the user has their eye in close proximity or away from the screen.

[0019] The system timer 53 is a timing unit that measures the time used for various controls and the time of a built-in clock. The power switch 72 is an operating member that switches the power of the digital camera 100 on and off. The mode selector switch 60, the first shutter switch 62, the second shutter switch 64, and the operation unit 70 are operating units for inputting various operational instructions to the system control unit 50. The operation unit 70 includes a gaze input operation unit (gaze acquisition means) 701 and a gaze confirmation operation unit (gaze confirmation means) 702. The gaze input operation unit 701 acquires gaze information of the user observing the display unit 28. The gaze confirmation operation unit 702 allows input and confirmation operations based on the user's gaze information, and confirms the gaze information.

[0020] The mode selector switch 60 switches the operating mode of the system control unit 50 to one of still image recording mode, video shooting mode, playback mode, etc. Modes included in the still image recording mode include auto shooting mode, auto scene determination mode, manual mode, aperture priority mode (Av mode), and shutter speed priority mode (Tv mode). Other modes include various scene modes, program AE mode, and custom mode, which are shooting settings for different shooting scenes. The mode selector switch 60 can directly switch to one of these modes included in the menu button. Alternatively, after switching to the menu button with the mode selector switch 60, another operating member can be used to switch to one of these modes included in the menu button. Similarly, the video shooting mode may also include multiple modes.

[0021] The first shutter switch 62 is turned on and generates a first shutter switch signal SW1 when the shutter button 61 provided on the digital camera 100 is pressed halfway (a shooting preparation command) during operation. The first shutter switch signal SW1 starts operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing.

[0022] The second shutter switch 64 is turned on when the shutter button 61 is fully pressed (photographing instruction) and generates a second shutter switch signal SW2. The system control unit 50 starts a series of photographing processing operations, from reading out a signal from the imaging unit 22 to writing image data to the recording medium 200, in response to the second shutter switch signal SW2.

[0023] Each operation member of the operation unit 70 is assigned a function appropriate for each situation by selecting and operating various function icons displayed on the display unit 28, and functions as various function buttons. Examples of the function buttons include an end button, a back button, an image forward button, a jump button, a filter button, and an attribute change button. For example, when the menu button is pressed, a menu screen in which various settings can be made is displayed on the display unit 28. The user can intuitively make various settings using the menu screen displayed on the display unit 28, the four-way buttons (up, down, left, and right), and the SET button. The operation unit 70 is made up of various operation members as an input unit that accepts operations from the user. The operation unit 70 is provided with electronic buttons, a cross key, and the like for selecting a menu, selecting a mode, playing back captured moving images, and the like.

[0024] The power supply control unit 80 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between powered blocks, and the like, and detects whether a battery is installed, the type of battery, and the remaining battery power. The power supply control unit 80 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the required voltage for the required period to each unit, including the recording medium 200.

[0025] The power supply unit 30 is composed of primary batteries such as alkaline batteries or lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries or Li batteries, an AC adapter, etc. The recording medium I / F 18 is an interface with a recording medium 200 such as a memory card or hard disk. The recording medium 200 is a recording medium such as a memory card for recording captured images, and is composed of a semiconductor memory, a magnetic disk, etc.

[0026] The communication unit 54 is connected wirelessly or via a wired cable and transmits and receives video and audio signals. The communication unit 54 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 54 can transmit images (including through images) captured by the imaging unit 22 and images recorded on the recording medium 200, and can also receive image data and various other information from external devices.

[0027] The orientation detection unit 55 detects the orientation of the digital camera 100 with respect to the direction of gravity. Based on the orientation detected by the orientation detection unit 55, it is possible to determine whether an image captured by the imaging unit 22 was captured with the digital camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the orientation detected by the orientation detection unit 55 to the image file of the image captured by the imaging unit 22, or rotate and record the image. The orientation detection unit 55 includes, for example, an acceleration sensor, a gyro sensor, etc.

[0028] [Gaze detection] Next, gaze detection in this embodiment will be described with reference to Fig. 2. Fig. 2 is a configuration diagram of a gaze input operation unit (gaze acquisition means) 701. In this embodiment, the gaze input operation unit 701 is provided as one of the operation units 70. The gaze input operation unit 701 is an operation member for detecting at which point on the display unit 28 the user's gaze is directed.

[0029] The configuration shown in FIG. 2 realizes a method disclosed in Patent Document 1, in which the rotation angle of the optical axis of a user's eyeball 501a looking into a viewfinder field is detected and the user's line of sight is detected from the detected rotation angle. A live view image captured through the lens unit 150 is displayed on the display unit 28. Reference numeral 701a denotes an image sensor, 701b denotes a light-receiving lens, 701c denotes a dichroic mirror, 701d denotes an eyepiece, and 701e denotes an illumination light source. The illumination light source 701e projects infrared light onto the eyeball 501a. The infrared light reflected by the eyeball 501a is reflected by the dichroic mirror 701c and captured by the image sensor 701a. The captured eyeball image is converted into a digital signal by an A / D converter (not shown) and transmitted to the system control unit 50. The system control unit 50, which serves as a line of sight acquisition unit, extracts the pupil area and other information from the captured eyeball image to calculate the user's line of sight.

[0030] Note that the gaze input operation unit 701 is not limited to this method, and may be a method of capturing an image of both of the user's eyes and detecting the gaze. FIG. 3 is a configuration diagram of the gaze input operation unit 701, which is different from that shown in FIG. 2. As shown in FIG. 3, a live view display image (captured image) captured through the lens unit 150 is displayed on the display unit 28 provided on the rear surface of the digital camera 100. In FIG. 3, a camera 701f is provided on the rear surface of the digital camera 100, capturing an image of the user's face 500 viewing the display unit 28. In FIG. 3, the angle of view captured by the camera 701f is indicated by a dotted line. Light is projected onto the user's face from an illumination light source 701e (not shown), and an image of the user's eyeballs is acquired by the camera 701f. This allows the user's gaze to be calculated. Note that the gaze input operation unit 701 is not limited to this method, and may be configured in any manner as long as it can detect which part of the display unit 28 the user is gazing at.

[0031] [Gaze detection and dynamic calibration method] Next, the gaze position correction method of this embodiment will be described with reference to Figs. 4 and 5. Fig. 4 is a flowchart of the gaze detection processing method and the gaze position correction method (control method). Fig. 4 shows the operation when a user performs a gaze confirmation operation and implements a focus adjustment means while looking at an image displayed on the display unit 28, which is the display means of this embodiment, and is realized mainly by the system control unit 50. Note that this embodiment will be described using the block diagram of the imaging device 1 in Fig. 1, but similar operations are possible in other devices that have at least the system control unit 50, the display unit 28, and the operation unit 70.

[0032] First, in step S1, the system control unit 50 starts a process (image display process) of displaying image data (captured image) on the display unit 28. The image to be displayed is, for example, a reduced image that matches the resolution of the display unit 28. While the image is being displayed, it is displayed at a predetermined frame rate, and the user can check the displayed image through the display unit 28.

[0033] Next, in step S2, the system control unit 50 determines whether or not gaze information can be acquired using the gaze input operation unit (gaze acquisition means) 701. Furthermore, if the display unit 28 has an eye sensor installed in the EVF, it determines whether or not gaze information can be acquired based on information on whether the user has their eyes in close contact with the EVF, and acquires reliability information (gaze reliability information) of the gaze position information. In this way, gaze information can include reliability information. If the user is looking into the EVF, the eye sensor determines that the user has their eyes in close contact with the EVF. In this case, the gaze input operation unit 701 determines that gaze information can be acquired. On the other hand, if the user is not looking into the EVF, the eye sensor determines that the user has their eyes in close contact with the EVF. In this case, the gaze input operation unit 701 determines that gaze information cannot be acquired. Furthermore, if the display unit 28 has a TFT liquid crystal display, the gaze input operation unit 701 also determines whether or not gaze information can be acquired.

[0034] Next, in step S3, if it is determined in step S2 that the gaze information can be acquired, the system control unit 50 starts gaze detection and acquires gaze information (gaze position information) using the gaze input operation unit 701. From step S4 onwards, the gaze input operation unit 701 acquires and stores gaze information (gaze position information) regarding which position on the display unit 28 the user is observing (gaze position) at predetermined time intervals, in association with the display image the user was observing. Note that if it is determined in step S2 that the gaze information cannot be acquired, the gaze information at the determined timing is not acquired or stored.

[0035] Next, in step S4, the subject detection means 50a of the system control unit 50 acquires subject information, such as subject position information, which is position information of the subject displayed on the display unit 28, the number of detected subjects, the type of detected subject, and the size of the detected subject. The subject detection means 50a also acquires rating information for the captured image as subject information. The rating information may be a value determined by the imaging device 1 based on focus information or the like, or a manual value determined by the user. Note that this subject information is updated each time the scene changes, such as by panning the imaging device 1. The update timing may also be every scene change. During continuous shooting, the subject information may be updated only when a change occurs, based on information indicating whether the subject information acquired for the previous frame has been updated. After acquiring the subject information, the process proceeds to step S5.

[0036] Next, in step S5, the scene determination acquisition means 50b of the system control unit 50 acquires, as scene determination information, camera operation information at the time of shooting the scene displayed on the display unit 28 and subject operation information displayed on the display unit 28. The information acquired as camera operation information at the time of shooting includes panning information acquired using a gyro sensor or the like, lens information such as the focal length and zoom position of the lens, shutter settings such as single shot / continuous shot, and camera setting information such as Tv and ISO. The information acquired as subject operation information also includes the in-plane movement speed v of the subject on the display unit 28. t , the movement vector V t , past movement trajectory information P t After acquiring the camera operation information and subject operation information, the process proceeds to step S6.

[0037] Next, in step S6, the system control unit 50 determines whether or not the user has performed a gaze confirmation operation. If the user has performed a gaze confirmation operation, the display of the gaze position on the display unit 28 is stopped, and the calculation means 50c proceeds to step S7 to correct the gaze position. Note that the display of the gaze position on the display unit 28 may be continued, or the display method may be changed. On the other hand, if the user has not performed a gaze confirmation operation, the system control unit 50 returns to step S2 and acquires new gaze information.

[0038] In step S7, the calculation means 50c of the system control unit 50 calculates a correction value for the gaze position. Here, calculation of the correction value for the gaze position (gaze position correction) will be described with reference to Figs. 5(a) to 5(c). Figs. 5(a) to 5(c) are explanatory diagrams of gaze position correction in this embodiment, showing various scenes on the display unit 28 when the calculation means 50d is used. EP is a gaze display that displays the gaze position of the user, and the gaze position coordinates on the display unit 28 displayed at this time are expressed as (x eye ,y eye ) where Trg is the detected object, Area_dT represents the range of Trg, and the Trg coordinates are (x trg ,y trg ) and CAL eye represents the gaze position correction value, which is divided into a horizontal position correction value and a vertical position correction value, and can be expressed as the following equation (1).

[0039]

number

[0040] Figure 5(a) shows the moment when the user adjusts his / her gaze position to the flying airplane and fixes his / her gaze. eye ,y eye ) is the Trg coordinate (x trg ,y trg) is displayed at a different position from the Trg coordinate (x ). In this case, it is considered that the user has confirmed the line of sight after recognizing the airplane, and therefore the difference between these two coordinates is considered to be the amount of error. Therefore, dynamic calibration is performed from the difference between these two coordinates using line of sight information, subject information, and scene determination information. Note that if the subject is not detected by the subject detection means 50a, the Trg coordinate (x trg ,y trg ) cannot be acquired, it is determined that dynamic calibration will not be performed.

[0041] First, the gaze reliability information acquired in step S2 is used as gaze information. The gaze gain Gain eye The value (weighting) of changes. When the gaze reliability information is low (Gain eye <1, the acquired gaze position coordinates (x eye ,y eye ) is considered to have a large error for some reason, so it cannot be used as it is. On the other hand, when the gaze reliability information is high (Gain eye =1 is the acquired gaze position coordinate (x eye ,y eye ) can be used.

[0042] Next, the subject gain Gain at the time of calculating the gaze position correction value is calculated based on the detected subject type, subject size, and rating result as subject information. Trg If the subject is considered difficult to photograph based on its type and size, the Gain Trg On the other hand, if the subject is considered to be easy to photograph based on the subject type and size, change the subject gain to Gain <1. Trg Change the subject gain to =1.

[0043] Finally, as scene determination information, the scene determination gain Gain SceneThe value (weighting) changes depending on the difficulty of the scene. When the scene difficulty is determined to be high based on the camera operation information and subject movement information acquired in step S5 as the scene determination information, Gain Scene On the other hand, if the scene difficulty level is determined to be low based on the camera operation information and subject movement information acquired in step S5, the Gain Scene =1 and change the scene determination gain.

[0044] From these gains, the eye gaze position correction value is calculated by the following equations (2a) to (2d) using the distance between the two points (ΔX′, ΔY′).

[0045]

number

[0046] Similarly, in Figure 5(b), the gaze confidence is high (Gain eye = 1), and the subject is also a stationary subject. Trg =1), and the scene difficulty is low (Gain Scene = 1), so the gaze position correction value CAL eye is the gaze position coordinate (x eye ,y eye ) and Trg coordinates (x trg ,y trg ) is considered to be the exact difference in coordinates.

[0047] In addition, in Figure 5(c), the gaze position coordinate (x eye ,y eye In this case, the object (coffee) at the Trg coordinate (x trg ,y trg ) cannot be acquired, it is determined that dynamic calibration will not be performed. eye In order to correct the gaze position, the process proceeds to step S8.

[0048] In step S8, the system control unit 50 calculates the eye gaze position correction value CAL calculated in step S7. eye The eye gaze position is corrected by the eye gaze position correction value CAL. eye includes both horizontal and vertical position correction values. Therefore, each coordinate is converted into the gaze position correction value CAL eye After the line-of-sight correction position coordinates are calculated, the process proceeds to step S9 to determine whether or not the first shutter switch (Sw1) 62 has been detected as being on.

[0049] If, in step S9, on-state of Sw1 is not detected (or off-state is detected), the system control unit 50 proceeds to step S17. In step S17, the system control unit 50 determines whether or not the main switch included in the operation unit 70 has been turned off. On the other hand, if on-state of Sw1 is detected in step S9, the system control unit 50 proceeds to step S10. In step S10, the system control unit 50 sets a focus detection area to be focused and performs focus detection. Here, the focus detection area is set using the gaze position corrected in step S8. The gaze position acquired in step S3 contains an error with respect to the position of the subject intended by the user due to various factors, so the gaze position corrected in step S8 is used. Note that even if correction is made in step S8, it is difficult to completely eliminate the error, but it is possible to reduce it.

[0050] In step S10, the system control unit 50 sets a focus detection area using gaze position information that has been processed as described below. At this time, the gaze position and the center position of the focus detection area may be aligned, or if there are candidates for focus detection areas detected by the subject detection unit 50a or the like, the gaze position may be associated with the detected subject area and the closest subject area may be selected as the focus detection area. After step S10, setting of the focus detection area using gaze position information and focus detection processing are repeatedly performed each time an image is captured. Here, the focus detection processing will be described.

[0051] The defocus amount and direction are calculated for each focus detection area using focus detection data corresponding to the set focus detection area. In this embodiment, the system control unit 50 generates image signals for focus detection, calculates the amount of deviation (phase difference) between the focus detection signals, and calculates the amount of defocus and direction from the calculated amount of deviation.

[0052] Shading correction and filter processing are performed on the first and second focus detection signals obtained from the set focus detection area as image signals for focus detection, reducing the difference in light intensity between the pair of signals and extracting spatial frequency signals for phase difference detection.Next, shift processing is performed to relatively shift the first and second focus detection signals after filter processing in the pupil division direction, and a correlation amount that indicates the degree of signal coincidence is calculated.

[0053] The kth first focus detection signal after filter processing is denoted by A(k), the second focus detection signal is denoted by B(k), and the range of number k corresponding to the focus detection area is denoted by W. Furthermore, if the shift amount due to shift processing is denoted by s1 and the shift range of shift amount s1 is denoted by Γ1, the correlation amount COR is calculated by equation (3).

[0054]

number

[0055] By shifting the shift amount s1, the k-th first focus detection signal A(k) and the k-s1-th second focus detection signal B(k-s1) are matched and subtracted to generate a shift subtraction signal. The absolute value of the generated shift subtraction signal is calculated, and the sum of the number k is taken within the range W corresponding to the focus detection area to calculate the correlation amount COR(s1). If necessary, the correlation amount calculated for each row may be added across multiple rows for each shift amount.

[0056] Next, the real-valued shift amount that minimizes the correlation amount is calculated from the correlation amount using sub-pixel calculations, and is set as the image shift amount p1. The calculated image shift amount p1 is then multiplied by a conversion coefficient K1 that corresponds to the image height of the focus detection area, the F-number of the imaging lens (imaging optical system), and the exit pupil distance to determine the detected defocus amount.

[0057] Next, in step S11, the system control unit 50 drives the lens based on the defocus amount detected in the selected focus detection area. If the detected defocus amount is smaller than a predetermined value, it is not necessarily necessary to drive the lens.

[0058] Next, in step S12, the system control unit 50 acquires the image for detection and display performed in step S1, displays the live view, and performs focus detection processing. As described above, the detected subject area and gaze position information are also superimposed on the live view display. The processing performed in step S12 may be performed in parallel with the lens driving in step S11. Furthermore, the focus detection area may be changed in accordance with the obtained gaze position in accordance with the live view display, which is updated as needed. After the focus detection processing is completed, the system proceeds to step S13.

[0059] In step S13, the system control unit 50 detects the on / off state of the second shutter switch 64 (Sw2), which indicates an instruction to start shooting. The release (shooting trigger) switch, which is one of the operation units 70, can detect two levels of on / off depending on the amount of depression, and the aforementioned Sw2 corresponds to the second level of on / off of the release (shooting trigger) switch. If the system control unit 50 does not detect that Sw2 is on in step S13, it returns to step S9 and detects the on / off state of Sw1. On the other hand, if the system control unit 50 detects that Sw2 is on in step S13, the system control unit 50 proceeds to step S14.

[0060] In step S14, the system control unit 50 determines whether or not to record images. In this embodiment, image acquisition during continuous shooting is switched between processing for recording images and processing for image capture / display and focus detection. The switching may be alternate, or, for example, image capture / display and focus detection may be performed once every three shots. This allows for highly accurate focus detection without significantly reducing the number of shots taken per unit time. If it is determined in step S14 that image recording is to be performed, the process proceeds to step S15. In step S15, the system control unit 50 executes shooting processing. After the shooting subroutine is executed in step S15, the process returns to step S13, where the system control unit 50 determines whether or not Sw2 has been turned on, i.e., whether or not a continuous shooting instruction has been issued.

[0061] If it is determined in step S14 that image recording is not to be performed, the process proceeds to step S16. In step S16, the system control unit 50 executes image capture / display and focus detection processing during continuous shooting. The image capture / display and focus detection processing during continuous shooting are the same as those executed in step S12. The difference is that the display period, display update rate (interval), and display delay of the images captured in step S16 are different from those in the processing of step S12, depending on the frame rate of continuous shooting, the process of generating recorded images, etc. This display control is executed by the system control unit 50. As described above, the obtained gaze position information is used for setting the focus detection area and linking it to the detected subject area.

[0062] When imaging / display and focus detection processing during continuous shooting are performed in step S16, the process returns to step S13, where the system control unit 50 determines whether or not Sw2 is detected as being on, i.e., whether or not a continuous shooting instruction has been issued. If Sw1 is not detected as being on (or is detected as being off) in step S9, and the main switch is detected as being off in step S17, focus detection and shooting operations are terminated. On the other hand, if the main switch is not detected as being off in step S17, the process returns to step S1, and image display processing is performed.

[0063] As described above, in this embodiment, the control device (imaging device 1) includes a display unit (display unit 28) that displays an image, and a gaze acquisition unit (gaze input operation unit 701) that acquires gaze information of a user observing the display unit. The control device also includes a subject detection unit 50a that detects subject information in an image, a scene determination acquisition unit 50b that acquires scene determination information for the image, and a calculation unit 50c that calculates gaze position correction information based on the gaze information, subject information, and scene determination information.

[0064] Preferably, the calculation means changes a weighting (gain) of the gaze position information when calculating the correction information based on the reliability information (gaze reliability information). Also preferably, the subject information includes at least one of the number of subjects in the image, the subject type, the subject size, or the rating information. More preferably, the calculation means changes a weighting (gain) of the subject information (subject position information) when calculating the correction information based on at least one of the subject type, the subject size, or the rating information. Also preferably, the calculation means does not calculate the correction information when the subject detection means acquires subject information indicating that at least two subjects exist within a predetermined range in the image. Also preferably, the calculation means does not calculate the correction information when the subject detection means cannot acquire the subject information from the image. Also preferably, the calculation means initializes the correction information when the user is not observing the display means. Also preferably, the scene determination acquisition means changes a weighting (gain) of the gaze information (gaze position information) when calculating the correction information based on the scene determination information.

[0065] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0066] According to each embodiment, it is possible to provide a control device, an imaging device, a control method, and a program that can always perform gaze detection with high accuracy. Note that in this embodiment, dynamic calibration correction of the gaze position is performed using all of gaze information, subject information, and scene determination information, but this is not limited to this, and correction may be performed using at least one of these. Furthermore, calibration data may be reset depending on whether the user's eye is in contact with or away from the viewfinder.

[0067] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0068] 1. Imaging device (control device) 28 Display section (display means) 50a Subject detection means 50b Scene determination acquisition means 50c calculation method 701 Eye-gaze input operation unit (eye-gaze acquisition means)

Claims

1. An imaging device, a display means for displaying a live view image captured by the imaging element; a reliability acquisition means for acquiring reliability information regarding the line of sight of a user observing the display means; a gaze position detection means for detecting a gaze position of the user observing the display means; a subject information acquisition means for detecting a subject in the live view display image and acquiring subject information; a scene determination acquisition means for acquiring operation information of the imaging device before the user performs a line-of-sight determination operation and operation information of the subject, the operation information being different from the subject information in the live view display image before the user performs the line-of-sight determination operation; a calculation means for calculating correction information for the gaze position detected by the gaze position detection means based on the reliability information, the subject information, operation information of the imaging device, and operation information of the subject in response to the user performing the gaze confirmation operation.

2. The imaging device according to claim 1 , wherein the correction information includes a horizontal position correction value and a vertical position correction value.

3. An imaging device as described in claim 1 or 2, characterized in that the subject information includes the position of the subject.

4. The imaging device according to any one of claims 1 to 3, characterized in that the calculation means calculates the correction information of the gaze position by changing a weighting, which is a coefficient by which the difference between the gaze position detected by the gaze position detection means and the position of the subject acquired by the subject information acquisition means, is multiplied.

5. The imaging device described in Claim 4, characterized in that the weighting is a value that changes based on the reliability information, the subject information, operation information of the imaging device, and operation information of the subject.

6. 6. The imaging device according to claim 1, wherein the subject information includes at least one of the number of subjects in the live view display image, the subject type, the subject size, and rating information.

7. The imaging device according to claim 6, characterized in that the calculation means does not calculate the correction information when the subject information acquisition means acquires the subject information indicating that at least two or more subjects exist within a predetermined range in the live view display image.

8. 8. The imaging apparatus according to claim 1, wherein the calculation means does not calculate the correction information when the subject information acquisition means is unable to acquire the subject information.

9. A control method for an imaging device, comprising: a display step of displaying a live view display image captured by the imaging element on a display means; a reliability acquisition step of acquiring reliability information regarding the line of sight of a user observing the display means; a gaze position detecting step of detecting a gaze position of the user observing the display means; a subject information acquisition step of detecting a subject in the live view display image and acquiring subject information; a scene determination acquisition step of acquiring operation information of the imaging device before the user performs a line-of-sight determination operation and operation information of the subject in the live view display image before the user performs the line-of-sight determination operation, the operation information being different from the subject information; a calculation step of calculating correction information for the gaze position detected in the gaze position detection step based on the reliability information, the subject information, operation information of the imaging device, and operation information of the subject in response to the user performing the gaze confirmation operation.

10. A program causing a computer to execute the method for controlling an imaging device according to claim 9.

Images