Imaging device, imaging method, and program

Abstract

The present invention is provided with: a detection unit that detects a pupil with respect to a captured image; an autofocus control unit that executes pupil autofocus control such that autofocus is performed on the pupil detected by the detection unit, in accordance with a predetermined manipulation, in a still image capturing mode, and executes pupil autofocus control in accordance with detection of the pupil by the detection unit, in a video image capturing mode; and a display control unit that performs control such that a pupil frame indicating the focused pupil is displayed on a captured monitor image in accordance with focusing through the pupil autofocus control.

Description

[Technical field]

[0001] The present technology relates to an imaging device, an imaging method, and a program, and in particular to a technical field regarding autofocus operation and display related to the autofocus operation. [Background technology]

[0002] Autofocus is a widely used function in the field of imaging devices. For example, there are imaging devices in which the autofocus operation is performed when the shutter button is pressed halfway. Recently, some cameras use captured image analysis and subject recognition technology to track a target subject and perform autofocus on the subject. The following Patent Document 1 describes a technique relating to autofocus and tracking operations. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-111746 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, recent imaging devices such as digital cameras are sometimes provided with a still image capture mode and a moving image capture mode, allowing the user to selectively capture still images or moving images. However, it is considered that the appropriate autofocus operation and the appropriate manner of presenting information to the user do not necessarily match between the still image capture mode and the video image capture mode. Therefore, the present disclosure proposes a technique for performing appropriate autofocus control and display control in a still image capture mode and a video capture mode. [Means for solving the problem]

[0005] An imaging device according to the present technology includes a detection unit that detects a pupil in an captured image, and an autofocus control unit that, in a still image imaging mode, executes pupil autofocus control to perform autofocus on the pupil detected by the detection unit in response to a predetermined operation, and, in a video imaging mode, executes the pupil autofocus control in response to the pupil being detected by the detection unit, and a display control unit that controls displaying a pupil frame indicating the focused pupil on an imaging monitor image in response to focusing by the pupil autofocus control. Autofocus control is performed on the pupils detected in the captured image, such as the pupils of a person or an animal, in such a manner that different controls are performed in the still image capture mode and the video capture mode.

[0006] In the imaging device related to the above-described present technology, it is considered that the display control unit performs control to display a pupil focusing frame as the pupil frame in accordance with focusing by the pupil autofocus control in a still image imaging mode, and to display a pupil detection frame in a display mode different from the pupil focusing frame as the pupil frame in accordance with focusing by the pupil autofocus control in a video imaging mode. The pupil focusing frame and the pupil detection frame refer to pupil frames that are displayed in different manners, and the conditions and timing under which each is displayed are not strictly defined. As one example, the pupil focusing frame is a pupil frame that is displayed when presenting the focus state for the pupil to the user. The pupil detection frame is a pupil frame that is presented to the user when at least the pupil has been detected; in other words, it may be displayed even before the pupil is focused on, and it may be displayed when the pupil is focused on.

[0007] In the imaging device related to the above-mentioned present technology, it is considered that, in a still image imaging mode, when the pupil autofocus control is performed in response to the specified operation, the display control unit performs control not to display the pupil frame until focus is achieved. That is, in the case of the still image capturing mode, the pupil frame is displayed after the pupil autofocus is focused.

[0008] In the imaging device according to the present technology described above, it is considered that the pupil detection frame and the pupil focusing frame are displayed as frames having different colors. For example, the color of the pupil detection frame may be changed to a white pupil frame and the color of the pupil focusing frame may be changed to a green pupil frame.

[0009] In the imaging device according to the present technology described above, the predetermined operation may be a half-press of the shutter button. In the still image capture mode, pupil autofocus control is started by half-pressing the shutter button.

[0010] In the imaging device related to the present technology described above, as a mode related to pupil autofocus control, an auto mode in which it is automatically set as to whether the target of the pupil autofocus control is the left or right pupil, and a fixed mode in which the target of the pupil autofocus control is an pupil specified by a user operation can be selected, and it is considered that when the pupil autofocus control in video imaging mode is not focused on a pupil, the display control unit performs control not to display a pupil frame in the auto mode, and to display a pupil frame in the fixed mode. For example, the fixed modes may be one or both of a right eye fixed mode in which the pupil autofocus control is performed on the right pupil selected by user operation, and a left eye fixed mode in which the pupil autofocus control is performed on the left pupil selected by user operation. In the case of the moving image capture mode, pupil autofocus control is performed in response to pupil detection, but in the stage before focusing, the presence or absence of display of the pupil frame is changed depending on the mode related to the pupil autofocus control.

[0011] In the imaging device related to the above-described present technology, in the auto mode, a temporary selection state can be set in which a pupil specified by a user operation is temporarily fixed as a target and the pupil autofocus control is performed, and it is considered that the display control unit performs control to display a pupil frame in the temporary selection state when the pupil autofocus control in video imaging mode is not focused on the pupil. Even in the auto mode, the target of pupil autofocus control can be temporarily fixed. For example, a temporary right-eye selection state can be established in which the right pupil is temporarily fixed as the target and pupil autofocus control is performed by user operation, or a temporary left-eye selection state can be established in which the left pupil is temporarily fixed as the target and pupil autofocus control is performed by user operation. In this case, the pupil frame is displayed in the same manner as in the fixed mode.

[0012] In the imaging device related to the above-described present technology, it is considered that, in a video imaging mode, when the pupil autofocus control is performed, the display control unit performs control to display a pupil detection frame as the pupil frame until focusing is achieved, and then, upon focusing, to display a pupil focusing frame in a display mode different from the pupil detection frame as the pupil frame. That is, in the case of a moving image shooting mode, the pupil frame is switched from the pupil detection frame to the pupil focusing frame before and after focusing by pupil autofocus.

[0013] In the imaging device relating to the present technology described above, it is possible to select whether to display a face frame in response to face detection by the detection unit, and it is considered that the display control unit, in a video imaging mode, controls the display of the face frame in response to the on / off selection. For example, the user may be allowed to arbitrarily set the display of the face frame to on or off.

[0014] In the imaging device related to the present technology described above, as a mode related to pupil autofocus control, an auto mode in which it is automatically set as to which of the left and right pupils will be the target of the pupil autofocus control, and a fixed mode in which the target of the pupil autofocus control is an eye specified by a user operation can be selected, and when turning off the display of the face frame is selected, it is considered that the display control unit performs control not to display the face frame in the auto mode, and not to display the face frame in the fixed mode, and to display the pupil frame regardless of whether it is in focus through pupil detection. When the face frame display is set to "off" in the moving image capture mode, the face frame is not displayed, but the display conditions of the pupil frame are made different depending on the mode.

[0015] In the imaging device related to the above-described present technology, in the auto mode, a temporary selection state can be set in which an eye specified by a user operation is temporarily fixed as a target and the pupil autofocus control is performed, and when turning off the display of a face frame is selected, it is considered that the display control unit performs control not to display a face frame in the temporary selection state, but to display an pupil frame regardless of whether focus is achieved by pupil detection. Even in the auto mode, the target of the pupil autofocus control can be temporarily fixed to the right or left pupil. In this case, when the face frame display is set to "off," the same pupil frame display as in the fixed mode is performed.

[0016] In the imaging device related to the above-mentioned present technology, when the selection to turn off the display of the face frame is made, in the auto mode, it is considered that the display control unit performs control not to display the face frame, but to display the pupil frame in response to focusing on the pupil. Even in the auto mode, the pupil frame is displayed on condition that the subject is in focus.

[0017] In the imaging device related to the above-described present technology, an auto mode can be selected as one of the modes related to pupil autofocus control, in which either the left or right pupil will be the target of the pupil autofocus control, and in the auto mode, a temporary selection state can be set in which a pupil specified by a user operation is temporarily fixed as the target and the pupil autofocus control is performed, and in a still image imaging mode, the temporary selection state can be released in response to a release operation that instructs recording of a still image, and in a video imaging mode, the temporary selection state can be released in response to an operation other than the release operation. For example, in the still image shooting mode, the temporary selection state is released by a release operation using the shutter button, and in the moving image shooting mode, the temporary selection state is released by an operation of another operator such as the enter button.

[0018] In the imaging device related to the above-mentioned present technology, in a still image imaging mode, the autofocus control unit starts the pupil autofocus control based on pupil detection targeting the entire captured image in response to an operation instructing an pupil autofocus operation, and the display control unit displays an pupil focusing frame as a pupil frame in response to focusing, and in a video imaging mode, the autofocus control unit switches from autofocus control targeting the face to the pupil autofocus control based on pupil detection targeting the entire captured image in response to the operation, and the display control unit displays an pupil detection frame as a pupil frame in response to the pupil detection. For example, the user can use a predetermined operator to perform an operation to instruct pupil autofocus control.

[0019] In the imaging device related to the above-described present technology, it is possible that a touch operation on a display screen of an imaging monitor image can be detected, and the autofocus control unit, in a still image imaging mode, tracks an eye in response to a touch operation to perform the pupil autofocus control, and, in a video imaging mode, tracks a face in response to a touch operation and then performs the pupil autofocus control on a selected eye. For example, tracking can be started when the user touches the face part of the display.

[0020] In the imaging device related to the above-mentioned technology, it is thought that the autofocus control unit, in the case of a still image imaging mode, performs the pupil autofocus control to track the pupil closer to the touch position in response to a touch operation on a face in the imaging monitor image, and performs autofocus control on the face by tracking the face closer to the touch position in response to a touch operation on a part other than the face in the imaging monitor image. For example, the target to be tracked can be selected depending on the position that the user touches on the display.

[0021] An imaging method according to the present technology is an imaging method in which an imaging device performs a detection process for detecting an pupil in a captured image, and in a still image imaging mode, executes pupil autofocus control such that autofocus is performed on the pupil detected in the detection process in response to a predetermined operation, and in a video imaging mode, performs an autofocus control process for executing the pupil autofocus control in response to the pupil being detected in the detection process, and a display control process for displaying a pupil frame indicating the focused pupil on an imaging monitor image in response to focusing by the pupil autofocus control. This allows for proper autofocus control in both the still image capture mode and the video image capture mode, and also allows for proper display control for the user. The program according to the present technology is a program for causing a processor, for example a processor within an imaging device, to execute processing corresponding to such an imaging method. This makes it possible to execute the autofocus control and display control of the present disclosure in various imaging devices. [Brief description of the drawings]

[0022] [Figure 1] 1 is a perspective view of an appearance of an imaging device according to an embodiment of the present technology; [Diagram 2] FIG. 2 is a rear view of the imaging device according to the embodiment. [Diagram 3]1 is a block diagram of an internal configuration of an imaging device according to an embodiment. [Figure 4] FIG. 2 is an explanatory diagram of a functional configuration of a camera control unit according to the embodiment. [Diagram 5] 11A and 11B are explanatory diagrams of a display example during autofocus according to an embodiment. [Figure 6] 11A and 11B are explanatory diagrams of a display example during tracking according to an embodiment. [Figure 7] 4 is an explanatory diagram of a focus area icon according to the embodiment. FIG. [Figure 8] FIG. 4 is an explanatory diagram of mode transition regarding pupil AF according to the embodiment. [Figure 9] 11 is an explanatory diagram of an example of a frame display in a still image imaging mode according to the first embodiment; FIG. [Figure 10] 11 is an explanatory diagram of an example of a frame display in a moving image shooting mode according to the first embodiment; FIG. [Figure 11] 4 is a flowchart of a process in a still image imaging mode according to the first embodiment. [Figure 12] 4 is a flowchart of a process in a still image imaging mode according to the first embodiment. [Figure 13] 4 is a flowchart of a process in a moving image shooting mode according to the first embodiment. [Figure 14] 4 is a flowchart of a process in a moving image shooting mode according to the first embodiment. [Figure 15] 13 is an explanatory diagram of an example of a frame display in a moving image shooting mode according to the second embodiment. FIG. [Figure 16] 13 is a flowchart of a process in a moving image shooting mode according to the second embodiment. [Figure 17] 13 is a flowchart of a process in a moving image shooting mode according to the second embodiment. [Figure 18] 13 is an explanatory diagram of an example of a frame display in a moving image shooting mode according to the third embodiment. FIG. [Figure 19] 13 is a flowchart of a process in a moving image shooting mode according to the third embodiment. [Figure 20] 13 is a flowchart of a process in a moving image shooting mode according to the third embodiment. [Figure 21]13A to 13C are explanatory diagrams of an example of a frame display during pupil AF operation according to the fourth embodiment. [Figure 22] 13 is a flowchart of a process during pupil AF operation according to the fourth embodiment. [Figure 23] 13 is an explanatory diagram of an example of a frame display when a touch operation is performed according to the fifth embodiment. FIG. [Figure 24] 13 is an explanatory diagram of an example of a frame display when a touch operation is performed according to the fifth embodiment. FIG. [Diagram 25] 13 is a flowchart of a tracking process in a still image imaging mode according to the fifth embodiment. [Figure 26] 13 is a flowchart of a tracking process in a still image imaging mode according to the fifth embodiment. [Figure 27] 13 is a flowchart of a tracking process in a moving image imaging mode according to a fifth embodiment. [Figure 28] 23 is a flowchart of a process in a moving image shooting mode according to the sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] The embodiments will be described below in the following order. <1. Configuration of imaging device> <2. Eye AF mode transition> <3. First embodiment> <4. Second embodiment> <5. Third embodiment> <6. Fourth embodiment> <7. Fifth embodiment> 8. Sixth embodiment <9. Summary and variations>

[0024] In this disclosure, the term "autofocus" is also written as "AF." The basic operation modes of the imaging device include a "still image capturing mode" and a "moving image capturing mode." The "still image capture mode" is an operation mode in which the imaging device captures and records still images. In the still image capture mode, for example, an image of the subject is displayed on the display unit as a capture monitor image (so-called through image), so that the user can check the subject. Then, the user operates the shutter button (release operation) to record still image data.

[0025] The "moving image capture mode" is an operation mode in which the image capture device captures and records moving images. Even in the moving image capture mode, for example, an image of the subject is displayed on the display unit as a captured monitor image, so that the user can check the subject. When the camera is in a standby state in the moving image capture mode, moving image recording is started when the user performs a recording start operation, and moving image recording is ended when the user performs a recording stop operation. Also, when the still image capture mode is selected, the user may perform a recording start operation to switch to the moving image capture mode, thereby starting recording of the moving image.

[0026] "Eye AF" refers to the control and operation of AF that targets the eyes of a person, etc. It is also written as "Eye AF control" when referring to the control, and "Eye AF operation" when referring to the operation. The term "pupil AF control" as used herein refers to a process in which AF control unit 52, which will be described later, generates and transmits a control signal for driving a focus lens with the pupil as a target, based on distance measurement results and the like. The "pupil AF operation" refers to the entire AF operation targeting the pupil, including the above-mentioned pupil AF control and the operation of the focus lens driven thereby. In addition, the control and operation of AF targeting the face is sometimes referred to as "face AF" to distinguish it from eye AF.

[0027] There are three eye AF modes: "auto mode," "right eye fixed mode," and "left eye fixed mode."

[0028] "Auto mode" is a mode that performs autofocus on the eye, and automatically performs AF on the appropriate eye depending on the direction of the face, etc. The "right eye fixed mode" is a mode that is selected in response to, for example, a user operation, and is a mode in which AF is performed on the pupil of the right eye (sometimes written as "right pupil"). The "left eye fixed mode" is a mode that is selected, for example, in response to a user operation, and is a mode in which AF is performed on the pupil of the left eye (sometimes written as "left pupil"). The term "fixed mode" is a general term for "right eye fixed mode" and "left eye fixed mode," and refers to either or both of the "right eye fixed mode" and the "left eye fixed mode."

[0029] In the embodiment, an example will be given in which automatic selection of the left and right pupils is temporarily stopped in the auto mode, and the AF target can be fixed. The state in which the right eye (pupil of the right eye) is temporarily and fixedly selected as the AF target during auto mode is referred to as the "temporary right-eye selection state," and the state in which the left eye (pupil of the left eye) is temporarily and fixedly selected as the AF target during auto mode is referred to as the "temporary left-eye selection state." Furthermore, the term "temporary selection state" is a general term for "temporary right eye selection state" and "temporary left eye selection state," and refers to either or both of the "temporary right eye selection state" and the "temporary left eye selection state."

[0030] When considering an imaging device that can be set to such a temporary selection state, the term “auto mode” includes the “temporary selection state” in the broad sense, but does not include the “temporary selection state” in the narrow sense. In other words, the auto mode in the narrow sense is a mode that automatically sets whether the target of pupil AF control is the left or right eye. On the other hand, in the case of an imaging device in which a temporary selection state is not possible, there is no difference between the broad and narrow definitions of "auto mode."

[0031] <1. Configuration of imaging device> Fig. 1 shows a front perspective view of an image capturing device 1 according to an embodiment, and Fig. 2 shows a rear view. The image capturing device 1 is a so-called digital still camera, and is capable of capturing both still images and moving images by switching the image capturing mode. In this embodiment, the imaging device 1 is not limited to a digital still camera, but may be a video camera that is mainly used for capturing moving images and can also capture still images.

[0032] In the imaging device 1, a lens barrel 2 is disposed on the front side of a main body housing 100 constituting a camera body, or is detachable.

[0033] On the rear side (user side) of the imaging device 1, a display panel 101 is provided, which is a display device such as a liquid crystal display (LCD) or an organic EL (Electro-Luminescence) display. A display unit formed using an LCD, an organic EL display, or the like is also provided as the viewfinder 102. The viewfinder 102 is not limited to an electronic viewfinder (EVF), and may be an optical viewfinder (OVF).

[0034] A user can view images and various information through the display panel 101 and the viewfinder 102 . In this example, the imaging device 1 is provided with both a display panel 101 and a viewfinder 102, but this is not limited to this, and the imaging device 1 may be configured to have only one of the display panel 101 and the viewfinder 102, or both or either of the display panel 101 and the viewfinder 102 may be detachable.

[0035] On the main body housing 100 of the imaging device 1, various controls 110 are provided. For example, various types of operators 110, such as keys, dials, and composite operators for pressing / rotating, are provided to realize various operation functions. For example, menu operation, playback operation, mode selection operation, focus operation, zoom operation, parameter selection operation such as shutter speed and F-number, etc. are possible. Although detailed description of each operator 110 will be avoided, in the case of this embodiment, among the operators 110, the shutter button 110S, the assignable button 110C, and the enter button 110E are particularly shown.

[0036] The shutter button 110S is used for shutter operation (release operation) and also for AF operation when pressed halfway. For example, in the still image capture mode, the AF operation is started by pressing the shutter button 110S halfway.

[0037] The assignable button 110C is an operator also called a custom button, and is a button to which the user can assign any operation function. In this embodiment, an operation function for instructing pupil AF is assigned to this assignable button 110C, and there are cases in which assignable button 110C serves as a pupil AF operation button. Further, a switching operation function among "auto mode," "right eye fixed mode," and "left eye fixed mode" may be assigned to assignable button 110C, and assignable button 110C may be used as a mode switching operation button. In addition, an operation function for transitioning to a "temporary right eye selection state" and a "temporary left eye selection state" in the "auto mode" may be assigned to the assignable button 110C, and the assignable button 110C may be used as a state switching operation button in the auto mode.

[0038] The number of assignable buttons 110C is not limited to one, and a plurality of assignable buttons may be provided. For example, three assignable buttons 110C may be provided, each of which is assigned one of the above functions. Furthermore, the operators for the above-mentioned functions may be formed as dedicated operators provided in advance, rather than as the assignable buttons 110C.

[0039] The enter button 110E is a button used for various confirmation operations, such as confirming an option on a menu screen. In this embodiment, an example will be described later in which the Enter button 110E is assigned an operating function for releasing the temporary fixation from the "temporary right eye selection state" or the "temporary left eye selection state" and returning to normal (narrowly defined) auto mode.

[0040] Shutter button 110S is arranged on the upper right side of main body housing 100, and can be pressed with the index finger of the user's right hand, for example, when holding grip portion 103 with the user's right hand. Also, the assignable button 110C is arranged on the upper part of the rear side of the main body housing 100 as shown in FIG. 2, for example, and can be operated by the user by pressing it with the thumb of the right hand. In other words, the shutter button 110S and the assignable button 110C are positioned so that they can be operated with the index finger and thumb when the user holds the main housing 100 by gripping the grip part 103 with his or her right hand, i.e., they can be pressed individually or simultaneously as desired.

[0041] It should be noted that the shutter button 110S and the assignable button 110C are not necessarily limited to being operable simultaneously with one hand. For example, the shutter button 110S may be pressed with the right hand and the assignable button 110C may be operated with the left hand. For this reason, although not shown, for example, the lens barrel 2 may be provided with an assignable button 110C.

[0042] 3 shows the internal configuration of the imaging device 1 including the lens barrel 2. Note that, although an example will be described here in which the imaging device 1 is configured separately into the main body housing 100 and the lens barrel 2, the portion corresponding to the lens barrel 2 may be integrated with the main body housing 100.

[0043] The imaging device 1 has an imaging element (image sensor) 12, a camera signal processing unit 13, a recording control unit 14, a display unit 15, an output unit 16, an operation unit 17, a power supply unit 18, a camera control unit 30, and a memory unit 31 in a main body housing 100. The lens barrel 2 also has a lens system 21, a driver unit 22, and a lens control unit .

[0044] The lens system 21 in the lens barrel 2 includes lenses such as a zoom lens and a focus lens, and an aperture mechanism. Light (incident light) from a subject is guided by this lens system 21 and collected on the imaging element 12.

[0045] The imaging element 12 is configured as, for example, a charge coupled device (CCD) type, a complementary metal oxide semiconductor (CMOS) type, or the like. The image sensor 12 performs, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, etc., on the electrical signal obtained by photoelectrically converting the received light, and further performs A / D (Analog / Digital) conversion processing. Then, the image sensor 12 outputs the imaging signal as digital data to the downstream camera signal processor 13 and camera controller 30.

[0046] The camera signal processing unit 13 is configured as an image processor, for example, a DSP (Digital Signal Processor) etc. The camera signal processing unit 13 performs various types of signal processing on the digital signal (captured image signal) from the imaging element 12. For example, as camera processes, the camera signal processing unit 13 performs pre-processing, synchronization processing, YC generation processing, resolution conversion processing, codec processing, etc.

[0047] In the pre-processing, the captured image signal from the imaging element 12 is subjected to a clamping process for clamping the R, G, and B black levels to a predetermined level, a correction process between the R, G, and B color channels, and the like. In the synchronization process, a color separation process is performed so that the image data for each pixel has all the color components R, G, and B. For example, in the case of an image sensor that uses a Bayer array color filter, a demosaic process is performed as the color separation process. In the YC generation process, a luminance (Y) signal and a color (C) signal are generated (separated) from R, G, B image data. In the resolution conversion process, the image data that has been subjected to various types of signal processing is subjected to the resolution conversion process.

[0048] The recording control unit 14 performs recording and reproduction on a recording medium, such as a non-volatile memory, etc. The recording control unit 14 performs processing for recording image files MF such as moving image data and still image data, thumbnail images, etc. on the recording medium, for example. There are various possible actual forms for the recording control unit 14. For example, the recording control unit 14 may be configured as a flash memory built into the imaging device 1 and a write / read circuit for the memory memory, or may be in the form of a card recording / playback unit that performs recording / playback access to a recording medium that can be detached from the imaging device 1, such as a memory card (such as a portable flash memory). Also, the recording control unit 14 may be realized as an HDD (Hard Disk Drive) built into the imaging device 1.

[0049] The display unit 15 is a display unit that displays various information to the person taking the picture, and specifically represents the display panel 101 and the viewfinder 102 shown in FIG. The display unit 15 executes various displays on the display screen based on instructions from the camera control unit 30. For example, the display unit 15 displays a reproduced image of image data read from a recording medium by the recording control unit 14. The display unit 15 is also supplied with image data of a captured image whose resolution has been converted for display by the camera signal processing unit 13. In response to instructions from the camera control unit 30, the display unit 15 performs display based on the image data of the captured image, thereby displaying a so-called through image (a captured monitor image based on the subject light received by the imaging element 12). Further, based on instructions from the camera control unit 30, the display unit 15 executes display of various operation menus, icons, messages, etc., that is, GUI (Graphical User Interface) on the screen.

[0050] The output unit 16 performs data communication and network communication with external devices via wire or wirelessly. For example, the captured image data (still image files or video files) is transmitted to an external display device, recording device, playback device, or the like. The output unit 16 may also be a network communication unit that communicates via various networks such as the Internet, a home network, or a LAN (Local Area Network), and transmits and receives various data to and from servers, terminals, etc. on the network.

[0051] The operation unit 17 collectively indicates an input device for a user to input various operations. Specifically, the operation unit 17 indicates various operators 110 (including a shutter button 110S, an assignable button 110C, and an enter button 110E) provided on the main body housing 100. The operation unit 17 detects the operation of the user, and sends a signal corresponding to the input operation to the camera control unit 30 . As the operation unit 17, not only the operator 110 but also a touch panel may be used. For example, a touch panel may be formed on the display panel 101, and various operations may be made possible by touch panel operations using icons, menus, and the like displayed on the display panel 101. Alternatively, the operation unit 17 may have a configuration in which a user's tap operation is detected using a touch pad or the like. Furthermore, the operation unit 17 may be configured as a receiving unit for an external operation device such as a separate remote controller.

[0052] The power supply unit 18 generates a power supply voltage Vcc required for each unit, for example from an internally mounted battery, and supplies it as an operating voltage. When the lens barrel 2 is attached to the imaging device 1, the power supply voltage Vcc from the power supply unit 18 is also supplied to the circuits within the lens barrel 2. The power supply unit 18 may be provided with a circuit for charging the battery and a circuit for generating the power supply voltage Vcc, using a DC voltage converted and input by an AC adapter connected to a commercial AC power supply as a power source.

[0053] The camera control unit 30 is configured by a microcomputer (arithmetic processing device) equipped with a CPU (Central Processing Unit). The memory unit 31 stores information and the like used for processing by the camera control unit 30. The illustrated memory unit 31 collectively includes, for example, a Read Only Memory (ROM), a Random Access Memory (RAM), a flash memory, and the like. The memory unit 31 may be a memory area built into a microcomputer chip serving as the camera control unit 30, or may be configured by a separate memory chip. The camera control unit 30 executes a program stored in a ROM, flash memory, or the like of a memory unit 31, thereby controlling the entire imaging device 1 and lens barrel 2. For example, camera control unit 30 controls the operation of each required unit with respect to control of the shutter speed of image sensor 12, instructions for various signal processing in camera signal processing unit 13, imaging operation and recording operation in response to user operation, playback operation of recorded image files, operation of lens system 21 such as zoom, focus, and aperture adjustment in lens barrel 2, user interface operation, etc. With respect to aperture adjustment, camera control unit 30 performs variable control of the F-number in response to user operation, and instructs the F-number for automatic control (auto iris).

[0054] The RAM in the memory unit 31 is used as a working area for various data processing by the CPU of the camera control unit 30, for temporarily storing data, programs, and the like. The ROM and flash memory (non-volatile memory) in the memory unit 31 are used to store the OS (Operating System) that the CPU uses to control each part, content files such as image files, application programs for various operations, firmware, etc.

[0055] When the lens barrel 2 is attached to the main body housing 100, the camera control unit 30 communicates with the lens control unit 23 and issues various instructions. The lens barrel 2 is equipped with a lens control unit 23 implemented by, for example, a microcomputer, and various types of data communication is possible between the lens control unit 23 and the camera control unit 30. In the case of this embodiment, the camera control unit 30 issues drive instructions to the lens control unit 23 for the zoom lens, focus lens, aperture mechanism, etc. The lens control unit 23 controls the driver unit 22 in response to these drive instructions, causing the lens system 21 to perform operations. When the lens barrel 2 is attached to the main body housing 100, wired communication is performed between the camera control unit 30 and the lens control unit 23. However, the camera control unit 30 and the lens control unit 23 may be configured to perform wireless communication.

[0056] The driver unit 22 includes, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a motor driver for a diaphragm mechanism motor, and the like. These motor drivers apply drive currents to the corresponding drivers in response to instructions from the lens control unit 23, thereby moving the focus lens and zoom lens, opening and closing the aperture blades of the aperture mechanism, and so on.

[0057] FIG. 4 extracts a part of the configuration in FIG. 3 and shows the functions of the camera control unit 30 for performing the operation of this embodiment. The camera control unit 30 has the following functions: an AF distance measuring unit 51, an AF control unit 52, an image recognition unit 53, an operation recognition unit 54, a display control unit 55, and an assignable button setting unit 56. As an example for the purpose of explanation, these functions are provided by software in the camera control unit 30, but some or all of these functions may be provided in a DSP or microcomputer used as the camera signal processing unit 13 or the like, or may be provided in other locations, such as a microcomputer on a different chip from the camera control unit 30 or the camera signal processing unit 13. In any case, it is sufficient that a configuration for performing these functions is provided within the imaging device 1.

[0058] The AF distance measurement unit 51 performs distance measurement processing for autofocusing. For example, using signals of image plane phase difference pixels obtained by the image sensor 12 and captured image signals, it performs calculations for AF control as a so-called phase difference detection method, contrast detection method, or a hybrid AF method that has features of both of these.

[0059] The AF control unit 52 generates an AF control signal based on the control value obtained by the AF distance measurement unit 51, and performs a process of executing an AF operation. The AF control signal from the AF control unit 52 is transmitted to the lens control unit 23. Then, the lens control unit 23 controls the driver unit 22 based on the AF control signal, and causes the driver unit 22 to drive the focus lens in the lens system 21, thereby performing the AF operation.

[0060] The image recognition unit 53 performs image analysis on the captured image signal obtained by the image sensor 12 and performs recognition processing of the image content. In particular, in the case of this embodiment, the recognition processing is performed by a face / eye detection unit 61 and a tracking processing unit 62.

[0061] The face / eyes detection unit 61 performs processing to detect the face and eyes of a person (or animal) as a subject within a frame of a captured image signal. The face / eye determination unit 61 may further determine the face of the subject. This is an example of determining the area of ​​a person's face within the frame and further processing the individual recognition of the face. For example, when a large number of people are subjects, it is expected that the face of a targeted individual will be recognized.

[0062] The tracking processing unit 62 performs processing for determining the position of a target subject within a frame of the captured image signal and tracking the subject. The target to be tracked is, for example, a subject to be subjected to AF. The selection of the tracking target by the user (photographer) can also be performed by an operation of selecting a subject to be subjected to AF.

[0063] The display control unit 55 controls the display operation of the display unit 15. For example, the display control unit 55 has a function of controlling the execution of display of an imaging monitor image, and the execution of display of various operation menus, icons, messages, and the like on the screen. In particular, as displays related to the pupil AF operation, display control is performed for a face frame 150 and a pupil frame 152, which will be described later.

[0064] The operation recognition unit recognizes the operation performed by the operation unit 17. The camera control unit 30 performs necessary control processing in response to the user operation recognized by the operation recognition unit . In particular, with regard to the processing of the embodiment, the operation recognition unit 54 recognizes user operations such as the operation of the shutter button 110S, the assignable button 110C, and the enter button 110E on the operation unit 17, and touch operations detected on a touch panel provided on the display unit 15 (in this case, the display panel 101). In response to the recognition of these operations, the AF control section 52, the face / eye detection section 61 and the tracking processing section 62 in the image recognition section 53, the display control section 55, and the like carry out corresponding processes.

[0065] The assignable button setting section 56 indicates a function for setting the operation function of the assignable button 110C in accordance with a setting operation by the user. When the operation recognition section 54 detects an operation of the assignable button 110C, the camera control section 30 recognizes that an operation of an assigned function based on the setting of the assignable button setting section 56 has been performed.

[0066] 5 and 6 show examples of display on the display unit 15. These are examples in which various icons are displayed superimposed on the captured monitor image. In FIG. 5, a person HM is captured as a subject at approximately the center of the screen of the display unit 15 (that is, the angle of view at which the image is captured). Here, a focus frame 90 is displayed in the center of the screen. The focus frame 90 is a frame that indicates the AF target, or a frame that indicates a so-called AF distance measuring point. The display position of the focus frame 90 varies depending on the various modes, and multiple focus frames 90 may be displayed, but here, an example is shown in which one focus frame 90 is displayed in the center of the screen.

[0067] Fig. 6 shows a state in which tracking is being performed. For example, assume that AF control is performed on the face of person HM based on the focus frame 90 in Fig. 5, and that the face is set as a tracking target. 6, suppose that person HM is positioned to the left of the screen. A tracking frame 91 is displayed for this person HM. The tracking frame 91 tracks the targeted subject (the face of person HM) and is displayed so as to overlap the subject.

[0068] As shown in FIGS. 5 and 6, various icons and letters / numbers indicating the mode and operation state of the imaging device 1 are displayed on the screen. For example, the ISO sensitivity 71, exposure value 72, aperture value 73, shutter speed 74, and focus mark 75 are displayed at the bottom of the screen. The in-focus mark 75 indicates that the target is in focus, and in FIG. 7 shows that the tracking target is in focus.

[0069] On the left side of the screen, a viewfinder frame rate 76, a shutter method 77, a focus area 78, a focus mode 79, a drive mode 80, and a shooting mode 81 are displayed. The viewfinder frame rate 76 is an icon indicating the frame rate of the image displayed on the viewfinder 102 . The shutter mode 77 is an icon showing the mechanical shutter mode, the electronic shutter mode, and the auto mode. The icon shown in the figure is the auto mode, which indicates a mode in which the mechanical shutter mode and the electronic shutter mode are automatically switched depending on the shooting conditions and the shutter speed.

[0070] The focus area 78 is an icon that indicates a focus area setting as an area for performing focus control. The focus area settings include "wide", "zone", "center", "flexible spot", etc. The icons that indicate each focus area setting include icons 78a, 78b, 78c, 78d, etc., as shown in FIG. 7. "Wide" is a focus area mode in which automatic focusing is performed based on the entire monitor as a through image, and is indicated by, for example, an icon 78a. In FIG. "Zone" is a focus area mode in which the focus is automatically adjusted within a zone selected on the monitor where the subject is to be focused, and is indicated by, for example, icon 78b. "Center" is a focus area mode that automatically focuses on a subject near the center of the monitor, and is indicated by, for example, icon 78c. "Flexible Spot" is a focus area mode that allows you to move the focus frame to any location on the monitor and focus on very small subjects or narrow areas, and is indicated by icon 78d, for example.

[0071] In this way, the icon of the focus area 78 is switched according to the focus area setting at that time. Note that the image examples of the icons displayed as the focus area 78 shown in FIGS. 5, 6, and 7 are merely illustrative examples, and the present invention is not limited to these image designs.

[0072] 5 and 6, a focus mode 79 is an icon representing an AF operation mode, such as a single AF (AF-S) mode or a continuous AF (AF-C) mode. In single AF, autofocus is activated when the shutter button (110S) is pressed halfway, and once focus is achieved, the focus is locked at that point. Continuous AF is a mode in which the autofocus function continues to adjust the focus while the shutter button 110S is pressed halfway. Figures 5 and 6 show an example in which an icon indicating Continuous AF is displayed.

[0073] Regarding the pupil AF control of this embodiment, in the case of the still image capturing mode, pupil AF control as AF-S is performed by half-pressing the shutter button 110S. In addition, in video capture mode, if pupil detection is possible, pupil AF control is performed continuously as AF-C.

[0074] The drive mode 80 shown in FIG. 5 and FIG. 6 is an icon showing single shooting, continuous shooting, bracket shooting, and the like. The shooting modes 81 are icons indicating shooting modes such as "program auto", "aperture priority", "shutter speed priority", "manual exposure", "movie", and "slow / quick motion".

[0075] 5 and 6, the storage medium / number of images 82, aspect ratio 83, image size 84, image stabilization 85, image quality 86, and remaining battery level 87 are displayed at the top of the screen. Storage media / number of pictures 82 displays the number of pictures stored on the storage media and the number that can be stored. The aspect ratio 83 indicates the aspect ratio of the captured image. The image size 84 indicates the size of the captured image. Image Stabilization 85 displays the on / off status and mode of the image stabilization function. Image quality 86 displays the image quality setting of the captured image. The battery remaining capacity 87 indicates the remaining battery capacity, for example, by a picture and a percentage.

[0076] Various icons and the like displayed on the display unit 15 have been exemplified above, but in addition to these, icons, numbers, letters, or graphics for presenting various types of information may also be displayed. Also, these displays are just examples.

[0077] <2. Eye AF mode transition> An example of mode transition regarding pupil AF will be described. FIG. 8 shows a schematic diagram of the mode transition. As described above, the operation modes of the imaging device 1 can be broadly divided into a still image imaging mode and a video imaging mode. In addition to these, there may be other modes, such as various settings and maintenance modes, but a description of the modes that are not directly related to the pupil AF control and UI control of the present disclosure will be omitted.

[0078] The still image capture mode and the video capture mode are switched by a mode selection operation by the user. Alternatively, the switching may be performed by means of a method other than a direct mode selection operation. For example, if a "record button" is provided as the operator 110, even in the still image capture mode, there may be a mode transition in which the mode selection operation is omitted and the video capture mode is switched to by operating the "record button", and video capture is started. In any case, the still image capture mode and the moving image capture mode are selected in response to some kind of user operation.

[0079] In the still image capture mode and in the video capture mode, there are auto mode, right eye fixed mode, and left eye fixed mode for pupil AF control, as described above. In FIG. 8, "auto mode" indicates both a broad definition that includes "temporary selection state" and a narrow definition that does not include it. The auto mode, the right eye-fixed mode, and the left eye-fixed mode are selected in response to a user operation, for example, the mode may be selected in sequence by a toggle operation using the assignable button 110C.

[0080] When in the auto mode, it is possible to temporarily switch to a temporary right eye selection state or a temporary left eye selection state, that is, while maintaining the auto mode state. When the "temporary selection state" as the temporary right eye selection state or the temporary left eye selection state is released, the mode returns to the auto mode in the narrow sense. In still image capture mode, if the camera is in a temporary selection state in auto mode, for example, performing a release operation with the shutter button 110S will cancel the temporary selection state and return to auto mode in the strict sense. For example, if the user wants to focus on either the right eye or the left eye while in auto mode to capture and record a still image, the user can set the camera to a temporary selection state. This temporary selection state is released by a release operation, and the camera can continue in auto mode without performing any other operation to return to auto mode.

[0081] On the other hand, in the moving image shooting mode, if the temporary selection state is entered in the auto mode, there is no appropriate timing for canceling the release operation or the like. Therefore, for example, when the user operates the enter button 110E, this is recognized as an operation to cancel the temporary selection state, and the temporary selection state is cancelled, returning to the auto mode in the narrow sense.

[0082] <3. First embodiment> The following describes pupil AF control and its UI (user interface) display as a first embodiment. The UI display refers to the display performed on the display unit 15 under the control of the display control unit 55, and the following description will focus on the display of the face frame 150 and pupil frame 152 in particular. In the following description of each embodiment, the term "auto mode" is basically used in a narrow sense. When it is used in a broad sense, an explanation or a note will be provided in each case.

[0083] FIG. 9 shows an example of the UI display during pupil AF in still image capture mode. FIG. 9A is an example of a display when the pupil is successfully detected in auto mode. FIG. 9B is an example of the display when the pupil AF is in focus in auto mode. FIG. 9C is a display example when the pupil is successfully detected in the right eye fixed mode or in the temporary right eye selected state. FIG. 9D is an example of a display when pupil AF is in focus in the right eye fixed mode or in a state where the right eye is temporarily selected.

[0084] Note that the left eye fixed mode or temporary left eye selected state differs from the right eye fixed mode or temporary right eye selected state only in that the eye displayed with pupil frame 152 is the left eye, and therefore illustrations of UI display examples are omitted in each embodiment. The left eye fixed mode or temporary left eye selected state can be considered to be similar to the right eye fixed mode or temporary right eye selected state, with the only difference being the right eye / left eye.

[0085] As shown in FIG. 9A, when the subject is not in focus in the auto mode but the pupils are detected, a face frame 150 is displayed. A face frame 150 is displayed for each recognized face, with a distinction between the priority face and other faces. A priority face is a face that is given priority as an AF target. When multiple faces are detected, a variety of selection algorithms can be considered to determine which face is the priority face, and although we will not go into detail here, one of the faces is selected as the priority face. The face frame 150 of the priority face is displayed, for example, in a white frame. In the drawing, the face frame 150 of the priority face is marked with the symbol "WDp." The face frame 150 of a face other than the priority face is displayed, for example, in a gray frame. In the figure, it is shown with a hatched frame. The symbol "WDe" is added to the face frame 150 of a face other than the priority face (non-priority face). Of course, the frame color is just an example. Hereinafter, the face frame 150 may be referred to as a "priority face frame WDp" and a "non-priority face frame WDe" to distinguish them.

[0086] Eye AF control is performed on the eyes of the priority face. 9A is the pre-focus stage, and even if the pupils have been detected, it is not necessarily clear which of the left or right pupils is the most suitable for the AF target. In auto mode, a specific algorithm is used to select either the right or left pupil as the AF target. For this reason, the eye selected as the AF target may often fluctuate between the left and right due to changes in the subject's condition or facial direction, in which case it is expected that the position of the pupil frame 152 will frequently change between the left and right pupils, causing flickering. Therefore, when the subject is not in focus, the pupil frame 152 is not displayed, and instead the face frame 150 (priority face frame WDp) is displayed.

[0087] When pupil AF is focused in auto mode, pupil frame 152 is displayed as shown in Fig. 9B. In this embodiment, pupil frame 152 includes pupil focusing frame WHf and pupil detection frame WHd, which have different display modes. For example, both are pupil frames but have different colors. The pupil focusing frame WHf is, for example, a green frame, and the pupil detection frame WHd is, for example, a white frame. Of course, the frame colors are just examples. The pupil focusing frame and the pupil detection frame refer to pupil frames that are displayed in different manners, and the conditions and timing under which each is displayed are not strictly defined. As an example, the pupil focus frame WHf is a pupil frame 152 that indicates the focus state. Furthermore, the pupil detection frame WHd may indicate a focused state, but may also be displayed before the pupil is focused on. Therefore, the pupil detection frame WHd is a pupil frame 152 that may be displayed at least in a state where the pupil is detected. There are cases where both the pupil focusing frame WHf and the pupil detection frame WHd are displayed in an in-focus state, but this is due to differences in the process leading to the in-focus state, for example differences in modes or operations related to pupil AF. In the case of FIG. 9B, the pupil focusing frame WHf is displayed as the pupil frame 152. The face frame 150 is not displayed at all. This makes the pupil frame 152 stand out, and also makes it easier for the user to recognize that AF control is being performed with the pupil frame 152 as the target.

[0088] When the right eye is fixed or the right eye is temporarily selected and the focus is not achieved but the pupil can be detected, the priority face frame WDp is not displayed and the pupil frame 152 is displayed as the pupil detection frame WHd around the pupil of the priority face as shown in Fig. 9C. By not displaying both the pupil detection frame WHd and the priority face frame WDp, the display is simplified. Also, the non-priority face frame WDe is displayed. In this case, the reason why the pupil frame 152 (pupil detection frame WHd) is displayed is because the AF target is fixed to the right eye, and therefore the pupil frame 152 does not move frequently between the left and right eyes. By displaying the pupil frame 152, it is possible to clearly indicate to the user that pupil AF control is being performed.

[0089] When the pupil AF is focused in the right-eye fixed mode or in the temporary right-eye selected state, the pupil frame 152 is displayed as shown in Fig. 9D. In this case, the pupil focusing frame WHf is displayed as the pupil frame 152. Furthermore, by making all of the face frames 150 invisible, the pupil frames 152 stand out and the user can easily recognize that AF control is being performed with the pupil frames 152 as the target.

[0090] FIG. 10 shows an example of the UI display during pupil AF in video capture mode. FIG. 10A is an example of a display when the pupil is successfully detected in auto mode. FIG. 10B is an example of the display when the pupil AF is in focus in auto mode. FIG. 10C is a display example when the pupil is successfully detected in the right eye fixed mode or in the temporary right eye selected state. FIG. 10D is an example of a display when pupil AF is in focus in the right eye fixed mode or in a state where the right eye is temporarily selected.

[0091] As shown in FIG. 10A, when the subject is not in focus in the auto mode but the pupils have been detected, a priority face frame WDp and a non-priority face frame WDe are displayed as the face frame 150. As in the case of the still image shooting mode, in the auto mode, it is not necessarily stable whether the left or right eye is suitable as the AF target before focusing. For this reason, if the pupil frame 152 is displayed, its position will likely change frequently between the left and right pupils, causing flickering. For this reason, the pupil frame 152 is not displayed when the subject is not in focus. In video capture mode, pupil AF control is continuously performed by AF-C in response to pupil detection, and the period from pupil detection to focusing is usually extremely short, so it is unlikely that the user will feel uncomfortable with the face frame 150 being displayed while pupil AF control is in progress.

[0092] When pupil AF is in focus in auto mode, pupil detection frame WHd is displayed as pupil frame 152, as shown in Fig. 10B. The reason why pupil detection frame WHd is used instead of pupil focusing frame WHf is that pupil AF control is performed continuously in AF-C in video shooting mode. In other words, since AF operation is always in progress, the display is stabilized by displaying only pupil detection frame WHd as pupil frame 152. In addition, by displaying pupil detection frame WHd during pupil AF control regardless of whether the subject is in focus or not, the user is prevented from misunderstanding that the subject is not in focus unless it is the pupil focusing frame WHf. Furthermore, the display is simplified by not displaying the priority face frame WDp for the priority face for which the pupil frame 152 is displayed. Also, the non-priority face frame WDe is displayed, although it is also possible to configure the non-priority face frame WDe not to be displayed.

[0093] When the right eye is fixed or the right eye is temporarily selected and the focus is not achieved but the pupil can be detected, the priority face frame WDp is not displayed and the pupil frame 152 is displayed as the pupil detection frame WHd around the pupil of the priority face as shown in Fig. 10C. By not displaying the priority face frame WDp, the display is simplified. Furthermore, the non-priority face frame WDe is displayed. In this case, the reason why the pupil frame 152 is displayed is so that the AF target is fixed to the right eye, and the pupil frame 152 does not move frequently between the left and right eyes. By displaying the pupil frame 152, it is possible to clearly indicate to the user that pupil AF control is being performed.

[0094] Fig. 10D shows a case where pupil AF is focused in right eye fixed mode or in the temporary right eye selection state, and similarly to Fig. 10C, pupil detection frame WHd is displayed as pupil frame 152. Pupil focusing frame WHf is not displayed. Also, non-priority face frame WDe is displayed.

[0095] That is, in video shooting mode, pupil AF control is performed in AF-C, and in right eye fixed mode or temporary right eye selected state, the pupil AF target is fixed to the right eye, so the pupil detection frame WHd is displayed for the right eye regardless of the focus state. Also, the display is simplified by not displaying the priority face frame WDp for the priority face for which the pupil frame 152 (pupil detection frame WHd) is displayed.

[0096] 9 and 10, the display manner of pupil frame 152 according to the situation differs between still image capture mode and video capture mode. In particular, as can be seen by comparing Fig. 9D and Fig. 10D, in still image capture mode, pupil focusing frame WHf is displayed as pupil AF comes into focus, but in video capture mode, pupil detection frame WHd continues to be displayed from before focusing even if pupil AF comes into focus. Furthermore, in the still image capture mode, pupil AF control is started as AF-S by a predetermined user operation, for example, half-pressing the shutter button 110S, whereas in the video capture mode, pupil AF control is continued as AF-C if pupil detection is successful.

[0097] An example of processing for performing pupil AF control and display control according to the still image capture mode and the moving image capture mode will be described below. Note that, hereinafter, processing examples described with reference to flowcharts in each embodiment can be considered as processing of the camera control unit 30. In particular, these can be considered as processing performed by the AF control unit 52, face / eye detection unit 61, and display control unit 55 in the camera control unit 30 in response to user operation recognition by the operation recognition unit 54. Moreover, each flowchart shows only the processing for pupil AF control and the associated display control.

[0098] Fig. 11 and Fig. 12 show an example of processing in the still image capture mode. Fig. 11 and Fig. 12 show a series of flowcharts in separate steps, with connections indicated by "c1" and "c2".

[0099] The camera control unit 30 performs face detection and pupil detection processing in a captured image in step S101 of Fig. 11. Since the processing of Fig. 11 and Fig. 12 is continuously and repeatedly performed, it can be considered that the face detection and pupil detection processing is performed at all times. Note that the following process is explained on the assumption that pupil detection has been successful in step S101. If pupil detection has not been successful, pupil AF control is not performed in the first place.

[0100] In step S102, the camera control unit 30 monitors the release operation by the shutter button 110S. If a release operation is detected, the camera control unit 30 performs a release process in step S103. That is, a still image recording process of a frame corresponding to the operation timing is performed.

[0101] In step S104, the camera control unit 30 checks whether or not it is in a temporary selection state (temporary right eye selection state or temporary left eye selection state), and if it is not in a temporary selection state, the process returns to step S101. If the selection is temporary, the camera control unit 30 cancels the temporary selection in step S105, i.e., returns to the auto mode in the narrow sense, and then returns to step S101.

[0102] When a release operation is not detected, the camera control unit 30 proceeds from step S102 to S110, and branches the process depending on whether the current mode is auto mode (broad definition). In the case of the auto mode (broad definition), in step S111, the camera control unit 30 checks whether or not a cancellation operation for the temporary selection state has been performed. For example, it checks whether or not the enter button 110E has been operated in the temporary selection state. If a release operation for the temporary selection state is detected, the camera control unit 30 releases the temporary selection state in step S112, and then proceeds to step S114. It is not necessary to perform the processes of steps S111 and S112. In the case of the still image shooting mode, since the temporary selection state is released by the release operation as described above (steps S104 and S105), it is also possible to consider not operating a specific operator such as the enter button 110E for releasing the selection state.

[0103] If the operation to cancel the temporary selection state is not detected in step S111, the camera control unit 30 branches the process in step S113 depending on whether the current state is the temporary selection state or not. If the selection is not temporary, that is, if the mode is the auto mode in the strict sense, the camera control unit 30 proceeds to step S114.

[0104] In step S114, the camera control unit 30 performs eye AF target setting in the narrow auto mode. That is, it is a setting of whether the pupil of the left eye or the pupil of the right eye is to be the AF target. For example, the current frame image data is analyzed, and if one eye of the subject of the priority face is detected, the pupil of that eye is set as the AF target. If both eyes of the subject of the priority face are detected, the pupil of the eye determined to be the more desirable AF target by a predetermined algorithm is set as the AF target.

[0105] In step S115, the camera control unit 30 determines whether or not the shutter button 110S is half-pressed, and branches the process. If the button is not pressed halfway, the camera control unit 30 performs display control of the face frame 150 in step S116, for example, as shown in Fig. 9A. In this case, both the priority face frame WDp and the non-priority face frame WDe are displayed. Then, the camera control unit 30 returns to step S101.

[0106] If the camera control unit 30 detects a half-press in step S115, the process proceeds to step S117, and executes pupil AF control for the pupil that was set as the AF target in step S114. In this case, eye AF control as AF-S will be performed when the shutter is half-pressed. While the user continues to press the button halfway, the process of FIG. 11 is repeated and the process proceeds to step S117, thereby continuing pupil AF control.

[0107] During the period from when the pupil AF control is started until focusing, the camera control unit 30 proceeds from step S118 to step S116 and continues display control of the face frame 150. In this case, both the priority face frame WDp and the non-priority face frame WDe are displayed. Then, the camera control unit 30 returns to step S101.

[0108] When the pupil AF operation is focused at a certain point, the camera control unit 30 proceeds from step S118 to step S119, and performs control to display the pupil focusing frame WHf as the pupil frame 152 for the pupil that is the target of the pupil AF, as shown in Fig. 9B. In this case, all face frames 150 are hidden. Then, the camera control unit 30 returns to step S101.

[0109] If the mode is not the auto mode in the broad sense, that is, if the mode is the right eye fixed mode or the left eye fixed mode, the process of the camera control unit 30 proceeds from step S110 to step S130 in FIG. 12 except during a release operation. Even if the auto mode is in a broad sense, when it is in a temporary selection state, the process of the camera control unit 30 proceeds from step S113 in FIG. 11 to step S130 in FIG.

[0110] In step S130, the process branches depending on whether the target of the fixed pupil AF control is the right eye or the left eye. If the mode is right-eye fixed or the right-eye is temporarily selected and the right eye is the target, the camera control unit 30 proceeds from step S130 to step S141, determines whether the shutter button 110S is half-pressed, and branches the process.

[0111] If the shutter button 110S is not half-pressed, the camera control unit 30 performs display control of the pupil frame 152 and the non-priority face frame WDe in step S142, for example, as shown in FIG. 9C. In this case, the pupil frame 152 is the pupil detection frame WHd, and the priority face frame WDp is not displayed. Then, the camera control unit 30 returns to step S101.

[0112] If the camera control unit 30 detects a half-press in step S141, the process proceeds to step S143, where it executes pupil AF control for the pupil of the right eye. In this case, pupil AF control as AF-S is performed in response to the half-press. While the user continues to press the button halfway, the processes in FIGS. 11 and 12 are repeated and the process proceeds to step S143, thereby continuing pupil AF control.

[0113] During the period from when the pupil AF control is started until focusing, the camera control unit 30 proceeds from step S144 to step S142, and performs display control of the pupil frame 152 (in this case, the pupil detection frame WHd) and the non-priority face frame WDe. Then, the camera control unit 30 returns to step S101.

[0114] When the pupil AF operation is focused at a certain point, the camera control unit 30 proceeds from step S144 to step S145, and performs control to display the focusing frame WHf as the pupil frame 152 for the right pupil, as shown in Fig. 9D. In this case, all face frames 150 are hidden. Then, the camera control unit 30 returns to step S101.

[0115] If the camera is in the left eye fixed mode or in the temporary left eye selected state and the left eye is the target, the camera control unit 30 proceeds from step S130 to step S151. The camera control unit 30 then determines whether or not the shutter button 110S is half-pressed, and branches the process accordingly.

[0116] If the shutter button 110S is not half-pressed, the camera control unit 30 performs display control of the pupil frame 152 and the non-priority face frame WDe in step S152. In this case, the pupil frame 152 is displayed at the position of the left eye as the pupil detection frame WHd, and the priority face frame WDp is not displayed. Then, the camera control unit 30 returns to step S101.

[0117] If the camera control unit 30 detects a half-press in step S151, the process proceeds to step S153 and executes pupil AF control for the left pupil. In this case, pupil AF control as AF-S is performed in response to the half-press. Then, while the user continues to half-press the button, the processes in Figs. 11 and 12 are repeated and the process proceeds to step S153, thereby continuing pupil AF control.

[0118] During the period from when the pupil AF control is started until focusing, the camera control unit 30 proceeds from step S154 to step S152, and performs display control of the pupil frame 152 (in this case, the pupil detection frame WHd) and the non-priority face frame WDe. Then, the camera control unit 30 returns to step S101.

[0119] When the pupil AF operation is focused at a certain point, the camera control unit 30 proceeds from step S154 to step S155, and performs control to display the focusing frame WHf for the left eye pupil as the pupil frame 152. In this case, all face frames 150 are hidden. Then, the camera control unit 30 returns to step S101.

[0120] Fig. 13 and Fig. 14 show an example of processing in the moving image shooting mode. Fig. 13 and Fig. 14 show a series of flowcharts in separate steps, with connections indicated by "c11" and "c12".

[0121] The camera control unit 30 performs face detection and pupil detection processing in a captured image in step S201 of Fig. 13. Furthermore, the camera control unit 30 performs pupil AF control as AF-C in response to pupil detection. Since the processes in Figs. 13 and 14 are continuously and repeatedly performed, it can be considered that face detection and pupil detection processes are performed at all times, and pupil AF control is performed in response to pupil detection. The following processing will be described on the assumption that pupil detection and pupil AF control as AF-C corresponding to the pupil detection have been performed in step S201.

[0122] In step S202, the camera control unit 30 monitors a user's operation to start recording. If a recording start operation is detected, the camera control unit 30 performs control to start video recording in step S203, and returns to step S201.

[0123] In step S204, the camera control unit 30 monitors whether the user has performed an operation to end recording. If an operation to end recording is detected, the camera control unit 30 performs control to end video recording in step S203, and returns to step S201.

[0124] At any time other than when the recording start operation or the recording end operation is detected, the camera control unit 30 proceeds to step S210, and branches the process depending on whether the current mode is auto mode (broad definition). In the case of the auto mode (broad definition), in step S211, the camera control unit 30 checks whether or not a cancellation operation for the temporary selection state has been performed. For example, it checks whether or not the enter button 110E has been operated in the temporary selection state. If a release operation for the temporary selection state is detected, the camera control unit 30 releases the temporary selection state in step S212, and then proceeds to step S214. In the moving image shooting mode, it is preferable to cancel the temporary selection state in the processing of steps S211 and S212. This is because in the moving image shooting mode, it is not possible to determine the appropriate timing to cancel the temporary selection state, as is the case with the release operation in the still image shooting mode. Therefore, a specific operator such as the enter button 110E is provided to the user as an operator for canceling the temporary selection state, so that the temporary selection state can be canceled according to the user's intention, even during moving image recording, for example.

[0125] If a release operation for the temporary selection state is not detected in step S211, the camera control unit 30 branches the process in step S213 depending on whether the current state is the temporary selection state or not. If the selection is not temporary, that is, if the mode is the auto mode in the strict sense, the camera control unit 30 proceeds to step S214.

[0126] In step S214, the camera control unit 30 performs pupil AF target setting in the narrow auto mode. That is, similar to step S114 in Fig. 11 in the still image imaging mode, it sets whether the pupil of the left eye or the pupil of the right eye is to be the AF target.

[0127] In step S215, the camera control unit 30 determines whether or not the camera is currently in focus on the pupil that is the target of the pupil AF control. If the focus is not achieved, the camera control unit 30 proceeds to step S216 and performs display control of the face frame 150, for example, as shown in Fig. 10A. In this case, both the priority face frame WDp and the non-priority face frame WDe are displayed. Then, the camera control unit 30 returns to step S201.

[0128] When it is determined in step S215 that the focus is achieved, the camera control unit 30 proceeds to step S217, and performs control to display the pupil frame 152 for the pupil that is the target of the pupil AF, as shown in Fig. 10B, for example. In this case, the pupil frame 152 is the pupil detection frame WHd. Also, as the face frame 150, the priority face frame WDp is not displayed, and the non-priority face frame WDe is displayed. It is also possible to hide the non-priority face frame WDe. Then, the camera control unit 30 returns to step S201.

[0129] If the mode is not the broad auto mode, that is, if the mode is the right eye fixed mode or the left eye fixed mode, the process of the camera control unit 30 proceeds from step S210 to step S220 in FIG. Even if the auto mode is in a broad sense, when it is in a temporary selection state, the process of the camera control unit 30 proceeds from step S213 in FIG. 13 to step S220 in FIG.

[0130] In step S220, the process branches depending on whether the target of the fixed pupil AF control is the right eye or the left eye. If the camera is in right eye fixed mode or temporary right eye selection state and the right eye is the target, the camera control unit 30 proceeds to step S221 and performs control to display an pupil frame 152 for the right eye of the priority face and also display a non-priority face frame WDe, as shown in Figure 10C or 10D. In this case, the pupil frame 152 is the pupil detection frame WHd, and the priority face frame WDp is not displayed. Then, the camera control unit 30 returns to step S201.

[0131] If the camera is in left eye fixed mode or temporary left eye selection state and the left eye is the target, the camera control unit 30 proceeds to step S231 and performs control to display an pupil frame 152 for the left eye of the priority face and also display a non-priority face frame WDe. In this case, the pupil frame 152 is the pupil detection frame WHd, and the priority face frame WDp is not displayed. Then, the camera control unit 30 returns to step S201.

[0132] <4. Second embodiment> A second embodiment will be described as an example in which the UI display of the first embodiment described above is partially changed. FIG. 15 shows an example of a UI display during pupil AF in video capture mode. FIG. 15A is an example of a display when the pupil is successfully detected in auto mode. FIG. 15B is an example of the display when the pupil AF is in focus in auto mode. FIG. 15C is a display example when the pupil is successfully detected in the right eye fixed mode or in the temporary right eye selection state. FIG. 15D is a display example when pupil AF is in focus in the right eye fixed mode or in the right eye temporary selection state.

[0133] The UI display in FIG. 15A is similar to the example in FIG. 10A. When the pupil AF is in focus in auto mode, as shown in Fig. 15B, the pupil focus frame WHf is displayed as the pupil frame 152. This is different from Fig. 10B. The UI display in FIG. 15C is similar to the example in FIG. 10A. When the pupil AF is in focus in the right eye fixed mode or in the temporary right eye selected state, as shown in Fig. 15D, the pupil focusing frame WHf is displayed as the pupil frame 152. This also differs from Fig. 10D.

[0134] 15, in video capture mode, the pupil-focusing frame WHf is displayed when the subject is in focus. Also, in fixed mode or temporary selection state, the display of the pupil frame 152 is switched between the pupil-detection frame WHd and the pupil-focusing frame WHf depending on whether the subject is in focus or not.

[0135] An example of processing in the moving image shooting mode is shown in Figures 16 and 17. In Figures 16 and 17, the same processes as those in Figures 13 and 14 are given the same step numbers and their explanations are omitted. The processes from step S201 to step S216 in Fig. 16 are the same as those in Fig. 13. The only difference is the process in step S217A.

[0136] In the auto mode, when it is determined in step S215 that the focus is not on the target pupil of the pupil AF control, the camera control unit 30 proceeds to step S216 and performs display control of the face frame 150, for example, as shown in Fig. 15A. In this case, both the priority face frame WDp and the non-priority face frame WDe are displayed. Then, the camera control unit 30 returns to step S201.

[0137] In auto mode, when it is determined in step S215 that the focus is on the target pupil of the pupil AF control, the camera control unit 30 proceeds to step S217A, and performs control to display the pupil frame 152 for the pupil that is the target of the pupil AF, as shown in Fig. 15B, for example. In this case, the pupil frame 152 is the pupil focusing frame WHf. In addition, as the face frame 150, the priority face frame WDp is not displayed, and the non-priority face frame WDe is displayed. It is also possible to hide the non-priority face frame WDe. Then, the camera control unit 30 returns to step S201.

[0138] If the mode is not the auto mode in the broad sense, that is, if the mode is the right eye fixed mode or the left eye fixed mode, the process of the camera control unit 30 proceeds from step S210 in FIG. 16 to step S220 in FIG. Even if the auto mode is in a broad sense, when it is in a temporary selection state, the process of the camera control unit 30 proceeds from step S213 in FIG. 16 to step S220 in FIG.

[0139] In step S220, the process branches depending on whether the target of the fixed pupil AF control is the right eye or the left eye. If the camera is in the right-eye fixed mode or in the temporary right-eye selected state and the right eye is the target, the camera control unit 30 proceeds to step S225 and determines whether or not the focus is on the pupil of the right eye. If the focus is not on the right eye pupil, the camera control unit 30 proceeds to step S226, and performs control to display the pupil frame 152 for the right eye of the priority face and to display the non-priority face frame WDe, for example, as shown in Fig. 15C. In this case, the pupil frame 152 is the pupil detection frame WHd. Also, the priority face frame WDp is not displayed. Then, the camera control unit 30 returns to step S201.

[0140] If the right eye pupil is in focus, the camera control unit 30 proceeds from step S225 to step S227, and performs control to display the pupil frame 152 for the right eye of the priority face and to display the non-priority face frame WDe, for example, as shown in Fig. 15D. In this case, the pupil frame 152 is the pupil focusing frame WHf. Also, the priority face frame WDp is not displayed. Then, the camera control unit 30 returns to step S201.

[0141] If the camera is in the left eye fixed mode or in the temporary left eye selected state and the left eye is the target, the camera control unit 30 proceeds to step S235 and determines whether or not the focus is on the pupil of the left eye. If the focus is not on the left eye pupil, the camera control unit 30 proceeds to step S236, and performs control to display the pupil frame 152 for the left eye of the priority face, and also to display the non-priority face frame WDe. In this case, the pupil frame 152 is the pupil detection frame WHd. Also, the priority face frame WDp is not displayed. Then, the camera control unit 30 returns to step S201.

[0142] If the left eye pupil is in focus, the camera control unit 30 proceeds from step S235 to step S237, and performs control to display the pupil frame 152 for the left eye of the priority face, and also to display the non-priority face frame WDe. In this case, the pupil frame 152 is the pupil focusing frame WHf. Also, the priority face frame WDp is not displayed. Then, the camera control unit 30 returns to step S201.

[0143] <5. Third embodiment> As the third embodiment, a processing example in which the display of the face frame 150 can be selected to be turned on or off by a user operation will be described. The face frame display can be switched on / off by the user, for example, by operating a menu or a specific operator.

[0144] Fig. 18A, Fig. 18B, Fig. 18C, and Fig. 18D are examples of UI displays when the face frame display is set to "ON" in the video capture mode. Also, Fig. 18E, Fig. 18F, Fig. 18G, and Fig. 18H are examples of UI displays when the face frame display is set to "OFF" in the video capture mode.

[0145] 18A and 18E are display examples when the pupil is successfully detected in auto mode. 18B and 18F are display examples when the pupil AF is in focus in auto mode. 18C and 18G are display examples when the pupil is successfully detected in the right eye fixed mode or in the temporary right eye selection state. 18D and 18H are display examples when pupil AF is in focus in the right eye fixed mode or in the right eye temporary selection state.

[0146] The UI display examples shown in FIGS. 18A, 18B, 18C, and 18D when the face frame display is set to "ON" are similar to those in FIGS. 10A, 10B, 10C, and 10D of the first embodiment.

[0147] The UI display examples shown in Figures 18E, 18F, 18G, and 18H when the face frame display is turned "off" are those in which the face frame 150 has been removed from the displays in Figures 18A, 18B, 18C, and 18D.

[0148] As shown in FIG. 18E, when the subject is not in focus in auto mode but the pupil is detected, neither face frame 150 nor pupil frame 152 is displayed. When the pupil AF is in focus in auto mode, neither the face frame 150 nor the pupil frame 152 is displayed as shown in Fig. 18F. In other words, no frame is displayed before or after focus is achieved. In this way, in the case of the auto mode, when the face frame display is set to "off", both the face frame 150 and the pupil frame 152 are hidden. In the auto mode, it can be said that the user does not need to be particularly aware of it, and therefore, when the face frame display is set to "off" by the user's will, it may be preferable for the user not to display both the face frame 150 and the pupil frame 152. 18F, when the pupil AF is in focus in auto mode, for example, the pupil detection frame WHd may be displayed as the pupil frame 152. This can be adopted when notifying the user of the focus in auto mode is considered to be more suitable as a UI display. When the subject is not in focus in the right eye fixed mode or in the temporary right eye selected state but the pupil can be detected, as shown in FIG. 18G, a pupil frame 152 is displayed around the pupil of the priority face as pupil detection frame WHd. FIG. 18H shows a case where the pupil AF is focused in the right eye fixed mode or in the temporary right eye selected state, and the pupil detection frame WHd is displayed as the pupil frame 152 in the same manner as in FIG. 18G.

[0149] An example of processing for displaying a UI as shown in Fig. 18 is shown in Fig. 19 and Fig. 20. In Fig. 19 and Fig. 20, the same processes as those in Fig. 13 and Fig. 14 of the first embodiment are given the same step numbers and their explanations are omitted. The processes from step S201 to step S214 in FIG. 19 are similar to those in FIG. In FIG. 19, the process of step S240 following step S214 is carried out.

[0150] In step S240, the camera control unit 30 checks whether the face frame display is turned "off". If the face frame display is "ON", the camera control unit 30 proceeds to step S215 and determines whether or not the focus is on the pupil that is the target of the pupil AF control. If the focus is not achieved, the camera control unit 30 proceeds to step S216 and performs display control of the face frame 150 (priority face frame WDp, non-priority face frame WDe) as shown in FIG. 18A, for example.

[0151] If it is determined in step S215 that the focus is achieved, the camera control unit 30 proceeds to step S217, and displays an pupil frame 152 (pupil detection frame WHd) for the pupil that is the target of pupil AF, as shown in FIG. 18B, and also hides the priority face frame WDp as the face frame 150, and displays the non-priority face frame WDe.

[0152] If the face frame display is "OFF", the camera control unit 30 proceeds from step S240 to step S241, and determines whether or not the camera is currently in focus on the pupil that is the target of the pupil AF control. If the subject is not in focus, the face frame 150 is hidden in step S242. For example, the display without the face frame 150 will continue as shown in FIG.

[0153] When it is determined in step S241 that the focus is achieved, the camera control unit 30 proceeds to step S243 and performs control to hide both the face frame 150 and the pupil frame 152, for example, as shown in FIG. 18F. However, in this step S243, it is also possible to perform control such that the pupil frame 152 (pupil detection frame WHd) is displayed for the pupil that is the target of pupil AF, and the face frame 150 (priority face frame WDp and non-priority face frame WDe) is not displayed.

[0154] If the mode is not the broad auto mode, that is, if the mode is right eye fixed or left eye fixed, or if the mode is the broad auto mode but is in a temporary selection state, the processing of the camera control unit 30 proceeds from step S210 or step S213 in FIG. 19 to step S220 in FIG. 20.

[0155] In step S220, the process branches depending on whether the target of the fixed pupil AF control is the right eye or the left eye. If the camera is in the right-eye fixed mode or in the temporary right-eye selected state and the right eye is the target, the camera control unit 30 proceeds to step S251 and checks whether the face frame display is turned "OFF". If the face frame display is "ON", the camera control unit 30 proceeds to step S221 and performs control to display an pupil frame 152 (pupil detection frame WHd) for the right eye of the priority face, and also to display a non-priority face frame WDe, as shown in, for example, FIG. 18C or FIG. 18D.

[0156] If the face frame display is "OFF", the camera control unit 30 proceeds from step S251 to step S252, and performs control to display the pupil frame 152 (pupil detection frame WHd) for the right eye of the priority face, and to hide the face frame 150 (priority face frame WDp and non-priority face frame WDe), as shown in, for example, FIG. 18G or FIG. 18H.

[0157] If the camera is in the left eye fixed mode or in the temporary left eye selected state and the left eye is the target, the camera control unit 30 proceeds from step S220 to step S261 and checks whether the face frame display is turned "OFF". If the face frame display is "ON", the camera control unit 30 proceeds to step S231, and performs control to display the pupil frame 152 (pupil detection frame WHd) for the left eye of the priority face, and also to display the non-priority face frame WDe.

[0158] If the face frame display is “off”, the camera control unit 30 proceeds from step S261 to step S262, and controls the display of the pupil frame 152 (pupil detection frame WHd) for the left eye of the priority face, and the display of the face frame 150 (priority face frame WDp and non-priority face frame WDe) is suppressed.

[0159] In this example, when the face frame display is set to "off," the display of the face frame 150 is turned off and the pupil frame 152 is not affected; however, it is also possible to consider an example in which the pupil frame 152 is not displayed in auto mode, for example.

[0160] In addition, the UI display example of FIG. 18 is an example in which the face frame 150 is not displayed in accordance with the UI display of the first embodiment, but the face frame 150 may not be displayed in accordance with the UI display of the second embodiment (FIGS. 15A, 15B, 15C, and 15D).

[0161] <6. Fourth embodiment> As a fourth embodiment, an example in which, for example, the assignable button 110C or another specific operator is used as an operator for instructing the pupil AF operation will be described.

[0162] 21A and 21B show the UI display in the still image capture mode. FIG. 21A shows an example in which focus control is performed on an object other than the face, with the focus frame 90 indicating the focus target. Here, when the user performs an operation to instruct pupil AF using, for example, assignable button 110C, pupil AF control is activated, and pupil frames 152 are displayed for the eyes of the priority face, as shown in Fig. 21B. In the case of still image imaging mode, pupil focusing frame WHf is displayed in response to focusing.

[0163] 21C and 21D show the UI display in the video shooting mode. FIG. 21C shows an example of a case where focus control is performed on an object other than the face, with the focus frame 90 indicating the focus target. Here, when the user performs an operation to instruct pupil AF using, for example, the assignable button 110C, pupil AF control is activated, and pupil frames 152 are displayed for the eyes of the priority face, as shown in Fig. 21D. In the video imaging mode, pupil detection frame WHd is displayed.

[0164] For example, assuming that the pupil AF instruction function is assigned to assignable button 110C, the process of camera control unit 30 when that operation is performed is shown in FIG.

[0165] In step S301, the camera control unit 30 monitors the pupil AF operation, for example, the operation of the assignable button 110C.

[0166] When the pupil AF operation is performed by the assignable button 110C, the camera control unit 30 proceeds from step S301 to step S302. In step S302, the end of pupil AF is monitored. For example, when the user performs an operation to end pupil AF or when pupil AF is ended due to some control circumstance, the camera control unit 30 proceeds from step S302 to S304 and returns to the state before the pupil AF operation.

[0167] While the pupil AF operation is being performed by the pupil AF operation, the process proceeds to step S303, where it branches depending on the still image capture mode / moving image capture mode, and proceeds to step S310 or step S340.

[0168] In the case of the still image capturing mode, in step S310, the camera control unit 30 performs face detection and pupil detection from the entire captured image. In step S311, the camera control unit 30 checks whether or not a face has been detected. If a face has not been detected, the camera control unit 30 blinks the pupil frame 152, for example, in the center of the screen, to notify the user that detection for pupil AF is in progress. The camera control unit 30 then returns to step S302.

[0169] If a face has been detected, the process proceeds from step S311 to step S312 to check whether or not the pupils have been detected. When a single face is detected, the process checks whether or not the pupils of that face have been detected, and when multiple faces are detected, a priority face is determined from among the multiple faces, and the process checks whether or not the pupils of the priority face have been detected.

[0170] If the pupils have not been detected, the camera control unit 30 proceeds to step S330 and performs face AF control in AF-S mode. That is, AF control is performed with the detected face (priority face) as the target. Then, the camera control unit 30 checks the in-focus state in step S331, and if not in-focus, proceeds to step S333 to control the display of the face detection frame, and then returns to step S302.

[0171] When focusing is achieved by face AF control, the camera control unit 30 proceeds from step S331 to step S332, and performs display control of the face focusing frame, and then returns to step S302. Although the face detection frame and face focusing frame are not shown, for example, the white priority face frame WDp is the face detection frame, and the green priority face frame WDp is the face focusing frame.

[0172] Note that cases where a face can be detected but the eyes cannot be detected include the period from face detection to eye detection, when the face is too small to detect the eyes, and when the subject turns away after the face has been detected and the eyes are no longer visible. In such cases, AF control will be performed with the face as the target.

[0173] If the pupil has been detected, the process of the camera control unit 30 proceeds from step S312 to S313, and pupil AF control is performed in AF-S mode, that is, AF control is performed with the detected pupil as the target. Then, the camera control unit 30 checks the in-focus state in step S314, and if not in-focus, proceeds to step S316 to control the display of the pupil-detection frame WHd, and then returns to step S302. When focusing is achieved by the pupil AF control, the camera control unit 30 proceeds from step S314 to step S315, and performs display control of the pupil focusing frame WHf as shown in Fig. 21B, and then returns to step S302.

[0174] In the video shooting mode, the process of the camera control unit 30 proceeds from step S303 to step S340, and performs face detection and pupil detection from the entire captured image. Note that in the video shooting mode, AF control continues in AF-C. In step S341, the camera control unit 30 checks whether or not a face has been detected. If a face has not been detected, the camera control unit 30 blinks the pupil frame 152, for example, in the center of the screen, to notify the user that detection for pupil AF is in progress. Then, the camera control unit 30 returns to step S302.

[0175] If a face has been detected, the process proceeds from step S341 to step S342, where it is determined whether or not pupil detection has been performed. If the pupils have not been detected, the camera control unit 30 proceeds to step S344 and performs face AF control in AF-C. That is, AF control is performed on the detected face (priority face). Also, display control of the face detection frame is performed. Then, the process returns to step S302.

[0176] If the pupil has been detected, the process of the camera control unit 30 proceeds from step S342 to S343, and performs pupil AF control in AF-C. That is, AF control is performed with the detected pupil as the target. Also, display control of the pupil detection frame WHd is performed as shown in FIG. 21D. Then, the process returns to step S302.

[0177] In the above processing example, the following operations are executed. The pupil AF operation is initiated by operating the assignable button 110C or the like (step S301). Face / eye detection is performed from the entire screen, and eye AF control is performed (steps S310, S340). At this time, it is possible to give priority to the search from the area closest to the focus area set by the user. The focus area settings include the above-mentioned "wide," "zone," "center," and "flexible spot," and the priority search is performed according to the setting. Even if the eyes cannot be detected, if a face can be detected, AF is performed on the face and in-focus display is performed (steps S330, S331, S332, S333, step S344). If a face cannot be detected, the AF operation is not performed and the display blinks (step S320). The focus mode is AF-S in still image capture mode and AF-C in video capture mode. It is also possible that even with AF-S, there will be only one eye frame / face frame to indicate focus.

[0178] And whether the camera is in still image capture mode or video capture mode, the behavior will be the same: "AF on the eye using the current AF mode (AF-C or AF-S)." However, in still image capture mode, AF operation is started by triggering assignable button 110C, and after focusing, the pupil focusing frame WHf is displayed, whereas in video capture mode, AF operation is always ongoing, so in response to pupil AF operation, the focus target is simply changed to the pupil, and the pupil frame 152 is also displayed as pupil detection frame WHd.

[0179] <7. Fifth embodiment> As a fifth embodiment, an example of a combination of pupil AF operation and tracking will be described. In the case of a still image capturing mode, pupil AF control is performed to track the pupil in response to a touch operation, and in the case of a video capturing mode, pupil AF control is performed on the selected pupil after tracking the face in response to a touch operation.

[0180] 23 and 24 show examples of UI display when tracking is instructed by a touch operation in the still image capturing mode. 23A, it is assumed that the user performs a touch operation on a face in the image with a finger F or the like. A touch on the face in the image may be considered as a touch inside the face frame 150. When this operation is performed, the camera control unit 30 interprets this as an instruction to track, as a target, the eye that is closer to the touched position on the face touched by the user. In the case of Fig. 23A, the touch operation is performed at a position close to the left eye. Therefore, the camera control unit 30 performs tracking processing with the left eye as the target. The UI display displays an pupil frame 152 (for example, pupil detection frame WHd) for the left eye as shown in Fig. 23B. In addition, a tracking icon 155 is displayed when tracking begins.

[0181] Even during tracking, the tracking target can be switched between the right eye and the left eye by a specific user operation. For example, the tracking target can be alternately switched between the right eye and the left eye by a toggle operation of a certain operator.

[0182] For example, when the user performs a right / left eye switching operation from the state of FIG. 23B, the camera control unit 30 switches the tracking target to the right eye, and displays the pupil frame 152 for the right eye as shown in FIG. 23C. Also, the fact that the tracking target has been switched between the right eye and the left eye is indicated by a switching display 157. Note that "right eye auto" in the figure indicates that the right eye has been temporarily selected in auto mode.

[0183] 24A shows a case where the touch operation is not directed to a face in the image. In this case, if the camera control unit 30 determines that the touched image is a person, it interprets this as a user's instruction to track that person, and performs processing to track the person's face as a target. Therefore, in the case of the touch in Fig. 24A, a face frame 150 (for example, a face detection frame) is displayed only around the face of the person who was touched, as shown in Fig. 24B, and tracking of the face is started. In addition, a tracking icon 155 is displayed as tracking starts.

[0184] Also, for example, when the user switches between the right eye and the left eye from the state of Fig. 24B, the camera control unit 30 switches the tracking target from the face to either the left or right eye. For example, when switching to the right eye, the pupil frame 152 is displayed for the right eye as shown in Fig. 24C. When the user switches between the right eye and the left eye while in face tracking mode, which eye is to be tracked may be determined according to the operation method, or the appropriate eye may be automatically selected, similar to the auto mode processing described above. Furthermore, when the user performs a right eye / left eye switching operation in the state of FIG. 24C, the camera control unit 30 switches the tracking target to the left eye.

[0185] 23 and 24 are UI display examples in still image capture mode, but in video capture mode, it is difficult for the user to touch the area around the face or the body with a touch operation. Therefore, in video capture mode, the appropriate eye is selected as the tracking target by touch detection on the face.

[0186] An example of processing by the camera control unit 30 for such an operation will be described with reference to FIGS. 25 and 26 show an example of a series of processes in the still image capture mode, and connections are indicated by "c31", "c32", "c33", "c34", "c35", "c36", and "c37".

[0187] In the case of the still image imaging mode, when a touch operation is detected, the camera control unit 30 proceeds from step S401 to step S402 in FIG. 25, and sets the pupil AF to the auto mode. Then, in step S403, the camera control unit 30 determines the relationship between the touch position and the image, i.e., whether the position touched by the user is on the face or other than the face, and if it is on the face, whether the position is closer to the left or right eye.

[0188] If it is determined that the user has touched the vicinity of the right eye of a face in the image, the camera control unit 30 proceeds from step S410 to step S411, and starts tracking the pupil of the right eye as the tracking target. Furthermore, in step S412, the camera control unit 30 performs pupil AF control by temporarily selecting the right eye in the auto mode. In step S413, the camera control unit 30 performs control so that the tracking icon 155 and the pupil frame 152 (for example, pupil detection frame WHd) for the right eye that is the tracking target are displayed as UI display.

[0189] Next, the camera control unit 30 proceeds to step S415 in FIG. 26, and checks whether or not an operation for switching to the left eye has been performed. In addition, the camera control unit 30 determines whether or not tracking has been released in step S416. For example, when a user performs an operation to end tracking, or when a subject person goes out of frame and becomes unable to be tracked, the camera control unit 30 determines that tracking has been released.

[0190] If there is no switching operation and tracking is not released, the camera control unit 30 returns to step S411 in FIG. 25, and continues the tracking process and pupil AF control for the right pupil.

[0191] If it is determined in step S403 that the user's touch operation has been performed near the left eye of a face in the image, the camera control unit 30 proceeds from step S420 to S421 and starts tracking the pupil of the left eye as the tracking target. Furthermore, in step S422, the camera control unit 30 performs pupil AF control by setting the pupil AF control mode to a temporary left eye selection state in auto mode. In step S423, the camera control unit 30 performs control so that the tracking icon 155 and the pupil frame 152 (for example, pupil detection frame WHd) for the left eye that is the tracking target are displayed as UI display.

[0192] Next, the camera control unit 30 proceeds to step S425 in FIG. 26, and checks whether or not an operation for switching to the right eye has been performed. Furthermore, in step S426, the camera control unit 30 determines whether or not tracking has been released. If there is no switching operation and tracking is not released, the camera control unit 30 returns to step S421 in FIG. 25, and continues the tracking process and pupil AF control for the left pupil.

[0193] If it is determined in step S403 that the user's touch operation was performed on a part other than the face, the camera control unit 30 proceeds to step S430 and checks whether the touched position is in a person's area (i.e., the area of ​​the image of the person's body) as shown in FIG. 24A. If it is a person's area, the camera control unit 30 proceeds to step S431 and starts tracking the touched person's face as the tracking target. Furthermore, the camera control unit 30 starts face AF control targeting the face in step S432. Then, in step S433, the camera control unit 30 performs control so that, as the UI display, a tracking icon 155 and a face frame 150 for the face that is the tracking target are displayed.

[0194] Next, the camera control unit 30 proceeds to step S425 in FIG. 26, and checks whether or not an operation for switching to the right eye has been performed. Furthermore, in step S426, the camera control unit 30 determines whether or not tracking has been released. If there is no switching operation and tracking is not released, the camera control unit 30 returns to step S421 in FIG. 25, and continues the tracking process and pupil AF control for the left pupil.

[0195] The processing of the camera control unit 30 proceeds to step S433 in Fig. 26, where it is confirmed whether an operation to select the left eye or automatic selection has been performed. For example, this applies when the left eye is selected by a user's operation to switch between the right eye and the left eye, or when the left eye is selected as the pupil AF target by processing in auto mode. If the left eye has not been selected, the camera control unit 30 checks in step S434 whether an operation to select the right eye or automatic selection has been performed. For example, this applies when the right eye is selected by a user's operation to switch between the right eye and the left eye, or when the right eye is selected as the pupil AF target by processing in auto mode.

[0196] If neither the left eye nor the right eye is selected, the camera control unit 30 determines in step S435 whether tracking has been released. If no operation or automatic determination for left eye selection or right eye selection has occurred, and tracking has not been canceled, the camera control unit 30 returns to step S431 in FIG. 25, and continues the face tracking process and face AF control.

[0197] In this processing example, face tracking and face AF are performed when it is determined that the touch position of the user is in an area of ​​a person. If the touch position of the user is not in the area of ​​a person, the process returns from step S430 to step S401, i.e., the process does not proceed to steps S431, S432, and S433, and the tracking and AF operations are not started. However, if the touch position is not in a person's area, it may be assumed that the closest person is specified and the process proceeds to steps S431, S432, and S433.

[0198] When tracking and pupil AF control is being performed on the right pupil, if the left eye is specified by the user's right / left eye switching operation, the camera control unit 30 proceeds from step S415 in FIG. 26 to steps S421, S422, and S423 in FIG. 25, and begins tracking and pupil AF control on the left pupil, and performs UI display control to display a pupil frame 152 for the left eye.

[0199] In addition, if left eye selection is detected while face tracking and face AF control are being performed, the camera control unit 30 proceeds from step S433 in FIG. 26 to steps S421, S422, and S423 in FIG. 25, and starts tracking and pupil AF control for the left pupil, and also performs UI display control to display a pupil frame 152 for the left eye.

[0200] When tracking and pupil AF control is being performed on the left pupil, if the right eye is specified by the user's right / left eye switching operation, the camera control unit 30 proceeds from step S425 in FIG. 26 to steps S411, S412, and S413 in FIG. 25, and begins tracking and pupil AF control on the right pupil, and performs UI display control to display a pupil frame 152 for the right eye.

[0201] Furthermore, if right eye selection is detected while face tracking and face AF control are being performed, the camera control unit 30 proceeds from step S434 in FIG. 26 to steps S411, S412, and S413 in FIG. 25, and starts tracking and pupil AF control for the right pupil, and also performs UI display control to display a pupil frame 152 for the right eye.

[0202] When the camera control unit 30 detects the release of tracking in any one of steps S416, S426, and S435 in FIG. 26, the process proceeds to step S440, and the state returns to the state before tracking.

[0203] Next, the process in the moving image shooting mode will be described with reference to FIG. In the case of the video shooting mode, the camera control unit 30 monitors the face touch operation in step S450. Here, only touching on the face area (for example, within the face frame 150) is recognized as a tracking operation. In addition, since it may be difficult to accurately touch the face area in the image, it is not necessary to make a strict judgment, and it may be judged as a face touch if it is a touch at a position close to the face to some extent. Alternatively, if a face image is present, all touches may be considered as face touches and may be judged as a touch operation on the nearest face.

[0204] When a face touch operation is detected, the camera control unit 30 proceeds from step S450 to step S451 and sets the eye AF to auto mode. This is auto mode in the strict sense, and if it is in a temporary selected state at this time, it is released.

[0205] Then, in step S452, the camera control unit 30 starts tracking the touched person's face as the tracking target. In step S453, the camera control unit 30 performs a process of selecting whether the target of the pupil AF control should be the left or right eye of the face of the tracking target. This is the same process as in step S214 in Fig. 13. Then, the target of the pupil AF control as AF-C is switched to (or maintained as) the selected eye.

[0206] If the right eye is selected in step S453, the camera control unit 30 proceeds from step S460 to step S461, and controls to display the pupil detection frame WHd for the right eye of the face being tracked, and also displays the tracking icon 155.

[0207] If the left eye is selected in step S453, the camera control unit 30 proceeds from step S470 to step S471, and controls to display an pupil detection frame WHd for the left eye of the face being tracked, and also displays the tracking icon 155.

[0208] There may be cases where neither the right eye nor the left eye can be selected in step S453. This may be the case when the direction of the face being tracked changes, or when the face size becomes smaller as the face moves away from the imaging device 1. In such cases, the camera control unit 30 proceeds to step S480, where it performs AF control targeting the face, and also performs control to display the face frame 150 and tracking icon 155.

[0209] In step S440, the camera control unit 30 checks whether tracking has been completed, and if tracking has been completed, the process proceeds to step S491 to return to the state before tracking. If tracking is continuing, the process of the camera control unit 30 returns to step S452.

[0210] As described above, when a tracking instruction is detected, the camera control unit 30 performs the processes shown in Fig. 25 and Fig. 26 in the still image shooting mode, and performs the process shown in Fig. 27 in the video shooting mode. Therefore, the following operation is executed.

[0211] In still image capture mode, if the user's touch position is a face, the closer eye is selected. If the right eye is selected, right eye tracking is performed and pupil AF control is performed for the right eye with the right eye temporarily selected. If the left eye is selected, left eye tracking is performed and pupil AF control is performed for the left eye with the left eye temporarily selected. In still image capture mode, if the user's touch position is not on the face, the face will be tracked and face AF control will be performed. The user can switch between the right and left eye as the target for tracking and AF. In video mode, the camera tracks faces by detecting when the user touches the face. Eye AF is performed on the right or left eye of the face being tracked in strict auto mode.

[0212] It should be noted that there are many other possible examples of the control process. For example, the tracking target may be selected according to the size of the face. When a touch operation on the face is detected, if the face in the image is larger than a predetermined size, the eye is selected and tracking and eye AF control are performed, and if the face is smaller than the predetermined size, the face is selected and tracking and face AF control are performed.

[0213] 8. Sixth embodiment An example of processing in the sixth embodiment will be described with reference to FIG. In FIG. 13 of the first embodiment and FIG. 16 of the second embodiment described above, an example has been described in which in step S201, face detection and pupil detection processing is performed in a captured image, and pupil AF control as AF-C is performed in response to the pupil detection. 13 and 16 are explained under the assumption that pupil detection has been performed and AF-C has been performed in step S201, but in the sixth embodiment, an example of the flow up to face detection, pupil detection, and pupil AF until that prerequisite is met will be explained.

[0214] In Fig. 28, steps S202 to S217 are the same as in Fig. 13. Also, Fig. 28, when combined with Fig. 14, forms a series of processing flows, which is also the same as in Fig. 13. 13. The process corresponding to step S201 in FIG. 13 is shown as steps S10 to S17 in FIG. In addition, step S18 is added.

[0215] In step S10, the camera control unit 30 branches the process depending on whether or not pupil AF control is currently being performed. If pupil AF control is in progress, face detection and pupil detection have already been performed, so the process proceeds to step S202 and subsequent steps, and the same process as in FIG. 13 described above is performed.

[0216] If pupil AF control is not in progress, the camera control unit 30 proceeds from step S10 to step S11 and performs face detection. For example, the camera control unit 30 searches for an image of the face of, for example, a person (or an animal) within the entire range of the frame. During the period when face detection has not been performed, the camera control unit 30 proceeds from step S12 to steps S202 and S204, and if recording has started or ended, performs the above-mentioned corresponding processing (steps S203 and S205), and then returns to step S10. If there is no particular processing, the camera control unit 30 returns from step S18 to step S10 since pupil detection has not been performed yet.

[0217] If a face is detected, the camera control unit 30 proceeds from step S12 to step S13, and performs display control of the face frame 150. For example, both the priority face frame WDp and the non-priority face frame WDe are displayed. Furthermore, the camera control unit 30 starts face AF as AF-C in step S14.

[0218] In step S15, the camera control unit 30 performs pupil detection. For example, pupil detection is performed on a priority face among the detected faces. During the period in which the pupil is not detected, the camera control unit 30 proceeds from step S16 to step S202. If an eye is detected, the camera control unit 30 proceeds from step S16 to step S17 to switch the target of AF-C from the face to the eye, i.e., starts eye AF. In this case, whether the target of pupil AF will be the right eye or the left eye will be determined by the user in fixed mode or temporary fixed state, but in auto mode it will be determined in step S214.

[0219] For example, by performing face detection, face AF, pupil detection, and pupil AF in this stepwise manner, the appropriate AF operation can be executed flexibly depending on the subject and the conditions of the captured image.

[0220] <9. Summary and variations> According to the above embodiment, the following effects can be obtained. As described in the first, second and third embodiments, the imaging device 1 comprises a face / eye detection unit 61 which detects an eye in a captured image, an AF control unit 52 which, in a still image imaging mode, executes eye AF control to perform AF on the eye detected by the face / eye detection unit 61 in response to a specified operation, and, in a video imaging mode, executes eye AF control in response to the eye being detected by the face / eye detection unit 61, and a display control unit 55 which controls the display of an eye frame 152 indicating the focused eye on the imaging monitor image in response to focusing by the eye AF control.

[0221] As a result, pupil AF operation is performed in response to user operation when capturing still images, and pupil AF is activated automatically when capturing video, providing the user with easy-to-use pupil AF operation that matches the capture mode. In particular, in the case of video, assuming that AF operation is performed continuously as AF-C, automatic pupil AF operation when the pupil is detected is a very effective control for capturing high-quality video.

[0222] As described in the first embodiment, in the still image capturing mode, the display control unit 55 displays the pupil focusing frame WHf as the pupil frame 152 in response to focusing by pupil AF control (step S119 in FIG. 11, steps S145 and S155 in FIG. 12). On the other hand, in the video capturing mode, the display control unit 55 displays the pupil detection frame WHd as the pupil frame 152 even in response to focusing by pupil AF control (step S217 in FIG. 13, steps S221 and S231 in FIG. 14).

[0223] In video capture mode, AF control is performed continuously (AF-C), so even if the pupil focus frame WHf is displayed, the display is likely to be unstable. In that case, there is a concern that the information display screen will be difficult for the user to understand. Therefore, by continuing the pupil detection frame WHd, the capture monitor image can be stabilized, providing the user with a screen that is easy to see. Furthermore, in the case of moving image capture mode, if focus is achieved, the pupil frame 152 can be stably displayed. For this reason, displaying the pupil frame 152 using the pupil detection frame WHd is suitable as information display for the user.

[0224] As described in the first embodiment, when the display control unit 55 automatically sets whether the left or right pupil will be the target of pupil AF control in still image imaging mode, that is, in the auto mode in the narrow sense, when pupil AF control is performed in response to a specified operation, the display control unit 55 does not display the pupil detection frame WHd as the pupil frame 152 until focus is achieved (FIGS. 9A, 9B, and steps S118 and S116 in FIG. 11).

[0225] In auto mode, even if the eye is detected before focusing, it may not be possible to determine which of the left or right eye is the most suitable AF target. In such cases, it is expected that the position of the pupil frame 152 will frequently change between the left and right eyes, causing flickering. Therefore, by not displaying the pupil frame 152 and displaying the face frame 150 when the subject is not in focus, the display will not become unstable. In addition, after pupil AF control is performed by half-pressing the shutter button, the user can recognize the focus state on the pupil by the display of the pupil frame 152. It should be noted that if the pupil AF operation is completed in an extremely short time, the time during which the pupil frame 152 does not appear is extremely short, and it is assumed that the user will not actually be aware that the pupil frame 152 does not appear. However, if it is assumed that for some reason the pupil AF operation is difficult to focus and the time before focusing is long, a processing example in which, for example, the pupil detection frame WHd is displayed before focusing is also suitable.

[0226] Furthermore, in right eye fixed mode, left eye fixed mode, or temporary selection state, the position of the pupil frame 152 does not frequently change between the left and right pupils and does not flicker. Therefore, in these cases, when pupil AF control is performed in response to a predetermined operation such as half-pressing, the pupil detection frame WHd is displayed as the pupil frame 152 until focusing is achieved (steps S145 and S155 in FIG. 12). This allows the user to confirm that pupil AF control is being performed, for example, when half-pressing, but the pupil has not yet been focused on.

[0227] Note that if the pupil AF operation is completed in an extremely short time, the time during which the user can recognize the pupil detection frame WHd is also extremely short, and it is conceivable that in fact the user may not be able to recognize it at all. However, if for some reason the pupil AF operation is difficult to focus, i.e., if there is a long period of time when the subject is not in focus, the display of the pupil detection frame WHd is a useful display.

[0228] In the embodiment, the pupil detection frame WHd and the pupil focusing frame WHf are displayed as frames of different colors (see FIGS. 9 and 10, etc.). The difference in color makes it easier for the user to recognize the difference between the pupil detection frame WHd and the pupil focusing frame WHf. Furthermore, the shapes of the pupil detection frame WHd and the pupil focusing frame WHf are almost the same, such as a square that surrounds the pupil. It often takes only a short time for the pupil AF operation to be performed until the focus is achieved, and for some reason the focused and unfocused states may frequently switch. In this case, if the pupil detection frame WHd and the pupil WHf have different frame shapes, the pupil frame 152 may change frequently on the imaging monitor image, making it difficult to see. Therefore, the pupil detection frame WHd and the pupil WHf can be displayed in a manner in which only the color changes, making them easier for the user to see.

[0229] In this embodiment, in the still image capturing mode, pupil AF control is started by half-pressing the shutter button 110S. This allows the AF operation commonly used in imaging devices to be applied as the pupil AF operation. In other words, the user can activate the pupil AF operation that is more suitable for capturing images of people, etc., in the same way as the previous AF operation.

[0230] As described in the first embodiment, in the case of a narrow auto mode in which pupil AF control is performed in video capture mode and it is automatically set whether the target of pupil AF control is the left or right pupil, the display control unit 55 does not display the pupil frame 152 (step S216 in FIG. 13). On the other hand, in a fixed mode in which the target of pupil autofocus control is a pupil specified by a user operation, for example, a right eye fixed mode or a left eye fixed mode, the display control unit 55 displays the pupil frame 152 (pupil detection frame WHd) even when the pupil is not in focus (steps S221 and S231 in FIG. 14).

[0231] In video imaging mode, pupil AF control is performed automatically in response to pupil detection, but in auto mode, there is a concern that flickering from side to side may occur if the subject is not in focus, so the pupil frame is not displayed to maintain the display quality of the captured monitor image. On the other hand, in the case of the right eye fixed mode or the left eye fixed mode, the pupil frame 152 is fixed to either the left eye side or the right eye side, and flickering between left and right does not occur even if the focus is not achieved. It is also preferable to clearly indicate whether the focus target specified by the user is the left or right side. Therefore, by displaying the pupil frame 152 to match the right eye in the right eye fixed mode and displaying the pupil frame 152 to match the left eye in the left eye fixed mode, the display quality is not impaired and appropriate information can be provided to the user.

[0232] In the embodiment, the auto mode, right eye fixed mode, and left eye fixed mode can be selected by user operation, but for example, only the auto mode and right eye fixed mode may be selectable, or only the auto mode and left eye fixed mode may be selectable.

[0233] As described in the first embodiment, when in auto mode, it is possible to enter a temporary selection state in which an eye specified by a user operation is temporarily fixed as the target and pupil AF control is performed, and in the temporary selection state, for example a temporary right eye selection state or a temporary left eye selection state, display control unit 55 displays pupil frame 152 (pupil detection frame WHd) even when the pupil is not in focus, as in the right eye fixed mode or left eye fixed mode (steps S221 and S222 in FIG. 14).

[0234] When the camera is in the temporary right eye selection state or the temporary left eye selection state, the pupil frame 152 is fixed to either the left eye side or the right eye side, as in the right eye fixation mode or the left eye fixation mode. In other words, no flickering occurs between the left and right sides. In addition, since the display is in response to user operation, it is preferable to clearly indicate the target of pupil AF control. Therefore, by displaying the pupil frame 152 for the right eye in the temporary right eye selection state and displaying the pupil frame 152 for the left eye in the temporary left eye selection state, the display quality is not impaired and appropriate information can be provided to the user.

[0235] In the embodiment, both the temporary right eye selection state and the temporary left eye selection state are selectable by user operation, but it is also possible to make it so that either the temporary right eye selection state or the temporary left eye selection state is selectable by user operation.

[0236] As described in the second embodiment, in video imaging mode, when pupil AF control is performed, display control unit 55 displays pupil detection frame WHd as pupil frame 152 until focusing is achieved (steps S226 and S236 in FIG. 17), and then displays pupil focusing frame WHf as pupil frame 152 in response to focusing (steps S227 and S237 in FIG. 17, step S217A in FIG. 16).

[0237] This allows the user to check the focus state on the pupil when capturing video. In the case of auto mode in the strict sense, the pupil detection frame WHd is not displayed during the out-of-focus period (step S216 in FIG. 16). This is to prevent the pupil frame 152 from flickering left and right in auto mode.

[0238] As described in the third embodiment, the display of the face frame 150 in response to face detection by the face / eye detection unit 61 can be selected to be turned on or off, and the display control unit 55 may control the display of the face frame in response to the selection of on or off in the video imaging mode (see Figures 18, 19, and 20). In particular, when capturing video, the face frame can make the screen display complicated. By allowing the user to select whether to turn the frame on or off, the display can be made easier to see.

[0239] In the example of the third embodiment, when the face frame display "OFF" is selected in the video capture mode, in the narrow auto mode, the display control unit 55 performs control not to display the face frame 150 (step S242 in FIG. 19). Also, in the right eye fixed mode or the left eye fixed mode, the face frame 150 is not displayed, and the pupil frame is displayed by pupil detection regardless of whether the focus is achieved (steps S252, S262 in FIG. 20).

[0240] That is, in either mode, the user can select to turn off the face frame display. In the case of auto mode in the strict sense, there is a concern that flickering from side to side may occur when the subject is not in focus, so the pupil frame 152 is not displayed, or is at least displayed only after the subject is in focus, thereby maintaining the display quality of the captured monitor image. In the case of the right eye fixed mode or the left eye fixed mode, the pupil frame 152 is fixed to either the left eye side or the right eye side, and flickering between the left and right does not occur. It is also preferable to clearly indicate the target specified by the user. Therefore, by displaying the pupil frame 152, the display quality is not impaired and appropriate information can be provided to the user.

[0241] As described in the third embodiment, when the display of the face frame 150 is set to "off," the display control unit 55 does not display the face frame 150 in the temporary right eye selection state or the temporary left eye selection state, but displays the pupil frame 152 (pupil detection frame WHd) even when the pupil is not in focus, as in the right eye fixed mode or left eye fixed mode (steps S221, S231 in FIG. 20). When in the temporary right eye selection state or temporary left eye selection state, as in the right eye fixed mode or left eye fixed mode, the pupil frame 152 is fixed to either the left eye side or the right eye side, and there is no flickering from side to side of the pupil frame 152. In addition, since this is in response to user operation, it is preferable to clearly indicate the target of pupil AF control. Therefore, by displaying the pupil frame 152, display quality is not compromised and appropriate information can be provided to the user.

[0242] In the third embodiment, an example was described in which when the display of the face frame 150 is selected to be “off,” in the auto mode, the face frame 150 is not displayed, and the pupil frame is displayed in response to focusing on the pupil (step S242 in FIG. 19). Since it is expected that the display of pupil frame 152 will be stable upon focusing even in auto mode, pupil frame 152 is displayed to present the focused state to the user.

[0243] In the first, second and third embodiments, examples have been given in which, in the still image capture mode, the temporary right eye selection state or the temporary left eye selection state is released in response to a release operation that instructs still image recording (step S105 in FIG. 11), and, in the video capture mode, the temporary right eye selection state or the temporary left eye selection state is released in response to an operation other than the release operation (operation of the Enter button 110E) (step S212 in FIGS. 13, 16 and 19).

[0244] In still image capture mode, the release operation indicates the timing for canceling the temporary selection state according to the user's intention, but there is no trigger for such cancellation during video recording. For the user, a specific operation different from the release operation is provided as an operation for canceling the temporary selection state, which is preferable when returning to auto mode in the narrow sense. For example, it becomes possible to return to auto mode without performing an operation such as stopping video recording. In addition, the temporary selection state can be released at any time by pressing a fixed key such as the enter button 110E, which makes it easy for the user to use.

[0245] In FIG. 11, the temporary selection state can be cancelled by pressing a fixed key such as the enter button 110E even in the still image capture mode (steps S111, S112). In this way, the temporary selection state can be cancelled at any time at will, providing the user with an easy-to-understand operating environment.

[0246] As described in the fourth embodiment, for example, the assignable button 110C is used as an operator for instructing the pupil AF operation. In the still image capturing mode, the AF control unit 52 starts pupil AF control based on pupil detection targeting the entire captured image in response to the operation of the assignable button 110C, and the display control unit 55 displays the pupil focusing frame WHf as the pupil frame 152 in response to focusing. In the video capturing mode, the AF control unit 52 switches the AF control targeting the face to pupil AF control based on pupil detection targeting the entire captured image in response to the operation of the assignable button 110C, and the display control unit 55 displays the pupil detection frame WHd as the pupil frame 152 in response to the pupil detection (see FIG. 22).

[0247] As a result, pupil AF control based on user operation is performed according to the still image capture mode and the video capture mode, and the appropriate pupil frame 152 is displayed for each mode. That is, in the still image capture mode, pupil AF control as AF-S is performed and in-focus is displayed. In the video capture mode, the target of AF-C is switched to the pupil, and the detection status is displayed by the pupil detection frame WHd. This makes it possible to present information that is easy for the user to understand.

[0248] Note that the operation to instruct the pupil AF operation is not limited to the operation of the assignable button 110C, and may be various other operations, such as the operation of a specific operation button or the operation by the user's touch, behavior, voice, or the like.

[0249] As described in the fifth embodiment, for example, an imaging monitor image is displayed and touch operation is possible on the display panel 101. In the still image imaging mode, the AF control unit 52 performs pupil AF control to track the pupil in response to the touch operation, and in the video imaging mode, the AF control unit 52 performs pupil AF control on the selected pupil after tracking the face in response to the touch operation. Since it is difficult for users to perform detailed operations while shooting video, the camera tracks the face specified by touch and automatically selects the left and right eyes of that face for eye AF control, making it easier for users to operate the camera.

[0250] As described in the fifth embodiment, in the case of a still image imaging mode, the AF control unit 52 can perform pupil AF control to track the pupil closer to the touch position in response to a touch operation on a face in the imaging monitor image, and can perform AF control to track the face closer to the touch position in response to a touch operation on a part other than the face in the imaging monitor image. In still image shooting mode, it is expected that the user has more leeway in operation than in video shooting mode, and touch operations are often possible with accuracy. Therefore, it is possible to specify whether the tracking target is the right eye, the left eye, or the face depending on the touch position. This allows the user to specify the tracking target in detail using only touch operations.

[0251] The program of the embodiment is a program that causes, for example, a CPU, a DSP, or a device including these to execute any of the processes in Figures 10, 11, 13, 14, 16, 17, 19, 20, 25, 26, and 27.

[0252] That is, the program of the embodiment causes the arithmetic processing device to execute image recognition processing which performs pupil detection in a captured image, and in still image imaging mode, pupil AF control which performs AF on the pupil detected in the image recognition processing in response to a specified operation, and in video imaging mode, AF control processing which performs pupil AF control in response to the pupil being detected in the image recognition processing, and display control processing which displays a pupil frame indicating the focused pupil on the imaging monitor image in response to focusing by the pupil AF control.

[0253] The arithmetic processing device is, for example, an arithmetic processing device (microprocessor) serving as the camera control unit 30. Such a program can realize a device that performs the operations of the imaging device 1 of this embodiment.

[0254] Such a program can be recorded in advance in a HDD as a recording medium built into a device such as a computer device, or in a ROM within a microcomputer having a CPU. Alternatively, the software may be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magnet optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such removable recording media may be provided as a so-called package software.

[0255] Furthermore, such a program can be installed in a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0256] Furthermore, such a program is suitable for widespread provision of the imaging device 1 of the embodiment. For example, by downloading the program to devices equipped with an imaging function, such as personal computers, portable information processing devices such as smartphones and tablets, mobile phones, game devices, video devices, and PDAs (Personal Digital Assistants), these devices can function as the imaging device 1 of the present disclosure.

[0257] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0258] The present technology can also be configured as follows. (1) A detection unit that detects pupils in a captured image; an autofocus control unit that, in a still image capture mode, executes pupil autofocus control in response to a predetermined operation so as to perform autofocus on the pupil detected by the detection unit, and, in a video capture mode, executes the pupil autofocus control in response to the pupil being detected by the detection unit; a display control unit that controls displaying a pupil frame indicating a focused pupil on an imaging monitor image in response to focusing by the pupil autofocus control. Imaging device. (2) The display control unit is In a still image imaging mode, a pupil focusing frame is displayed as the pupil frame in response to focusing by the pupil autofocus control; In a moving image imaging mode, control is performed to display, as the pupil frame, a pupil detection frame in a display mode different from that of the pupil focusing frame in response to focusing by the pupil autofocus control. The imaging device according to (1) above. (3) The display control unit is In the case where it is automatically set whether the target of the pupil autofocus control is the left or right pupil in the still image imaging mode, when the pupil autofocus control is performed in response to the predetermined operation, control is performed in which the pupil frame is not displayed until the focus is achieved. The imaging device according to (1) or (2) above. (4) The pupil detection frame and the pupil focusing frame are displayed as frames of different colors. The imaging device according to (2) or (3) above. (5) The predetermined operation is a half-press of the shutter button. An imaging device according to any one of (1) to (4) above. (6) Eye autofocus control modes include: an auto mode in which the target of the pupil autofocus control is automatically set to either the left or right pupil; a fixed mode in which the target of the pupil autofocus control is a pupil designated by a user operation; is selectable, The display control unit is When the pupil is not focused on by the pupil autofocus control in the video imaging mode, In the case of the above auto mode, the pupil frame is not displayed, In the case of the fixed mode, control is performed to display the pupil frame. An imaging device according to any one of (1) to (5) above. (7) In the auto mode, A temporary selection state can be set so that the pupil autofocus control is performed by temporarily fixing a pupil designated by a user operation as a target, The display control unit is When the pupil is not focused on by the pupil autofocus control in the video imaging mode, In the case of the temporary selection state, control is performed to display the pupil frame. The imaging device according to (6) above. (8) The display control unit is In the video imaging mode, when the pupil autofocus control is performed, a pupil detection frame is displayed as the pupil frame until focusing is achieved, and a pupil focusing frame in a display mode different from the pupil detection frame is displayed as the pupil frame upon focusing. The imaging device according to (1) above. (9) a face frame display according to face detection by the detection unit can be selectively turned on / off; In a video imaging mode, the display control unit Depending on the on / off selection, the face frame display is controlled. An imaging device according to any one of (1) to (8) above. (10) Eye autofocus control modes include: an auto mode in which the target of the pupil autofocus control is automatically set to either the left or right pupil; a fixed mode in which the target of the pupil autofocus control is a pupil designated by a user operation; is selectable, When the display control unit selects to turn off the display of the face frame, In the case of the above-mentioned auto mode, the face frame is not displayed, In the fixed mode, the face frame is not displayed, and the pupil frame is displayed regardless of whether the subject is in focus or not by pupil detection. The imaging device according to (9) above. (11) In the auto mode, A temporary selection state can be set so that the pupil autofocus control is performed by temporarily fixing a pupil designated by a user operation as a target, When the display control unit selects to turn off the display of the face frame, In the case of the temporary selection state, the face frame is not displayed, and the pupil frame is displayed by pupil detection regardless of whether the subject is in focus. The imaging device according to (10) above. (12) When the display control unit selects to turn off the display of the face frame, In the case of the auto mode, the face frame is not displayed, and the pupil frame is displayed in response to focusing on the pupil. The imaging device according to (10) or (11) above. (13) One of the modes for pupil autofocus control is An auto mode is selectable in which the target of the pupil autofocus control is automatically set to either the left or right pupil, In the auto mode, a temporary selection state can be set in which a pupil designated by a user operation is temporarily fixed as a target and the pupil autofocus control is performed, In the case of a still image capture mode, the temporary selection state is released in response to a release operation instructing recording of a still image, In the case of a video shooting mode, the temporary selection state is released in response to an operation different from the release operation. An imaging device according to any one of (1) to (12) above. (14) In still image capture mode, the autofocus control unit starts the pupil autofocus control based on pupil detection targeting an entire captured image in response to an operation instructing a pupil autofocus operation; the display control unit causes a pupil focusing frame to be displayed as a pupil frame in response to focusing; In the video capture mode, the autofocus control unit switches, in response to the operation, autofocus control targeting a face to the pupil autofocus control based on pupil detection targeting an entire captured image; The display control unit displays an pupil detection frame as a pupil frame in response to pupil detection. An imaging device according to any one of (1) to (13) above. (15) A touch operation on a display screen of the captured monitor image is detectable, The autofocus control unit is In the case of a still image capturing mode, the pupil autofocus control is performed by tracking the pupil in response to a touch operation, In the case of a video capture mode, the face is tracked in response to a touch operation and the pupil autofocus control is performed on the selected pupil. An imaging device according to any one of (1) to (14) above. (16) The autofocus control unit is In still image capture mode, performing the pupil autofocus control to track the pupil closer to the touch position in response to a touch operation on the face in the captured monitor image; In response to a touch operation on a part of the image captured by the monitor other than the face, the face closest to the touch position is tracked and autofocus control is performed on the face. The imaging device according to (15) above. (17) A detection process for detecting an eye in a captured image; an autofocus control process that, in a still image capture mode, executes a pupil autofocus control so as to perform autofocus on the pupil detected in the detection process in response to a predetermined operation, and, in a video capture mode, executes the pupil autofocus control in response to the pupil being detected in the detection process; a display control process for displaying a pupil frame indicating a focused pupil on an imaging monitor image in response to focusing by the pupil autofocus control; The imaging method is performed by an imaging device. (18) A detection process for detecting an eye in a captured image; an autofocus control process that, in a still image capture mode, executes a pupil autofocus control so as to perform autofocus on the pupil detected in the detection process in response to a predetermined operation, and, in a video capture mode, executes the pupil autofocus control in response to the pupil being detected in the detection process; a display control process for displaying a pupil frame indicating a focused pupil on an imaging monitor image in response to focusing by the pupil autofocus control; A program that causes a processor to execute the above. [Explanation of symbols]

[0259] 1 imaging device, 2 lens barrel, 12 imaging element, 13 camera signal processing unit, 14 recording control unit, 15 display unit, 16 output unit, 17 operation unit, 18 power supply unit, 21 lens system, 22 driver unit, 23 lens control unit, 24 operation unit, 30 camera control unit, 31 memory unit, 51 AF distance measurement unit, 52 AF control unit, 53 image recognition unit, 54 operation recognition unit, 55 display control unit, 56 assignable button setting unit, 61 face / eye detection unit, 62 tracking processing unit, 71 ISO sensitivity, 72 exposure value, 73 aperture value, 74 shutter speed, 75 focus mark, 76 viewfinder frame rate, 77 shutter method, 78 focus area, 79 focus mode, 80 drive mode, 81 shooting mode, 82 storage media / number of frames, 83 aspect ratio, 84 image size, 85 Image stabilization, 86 Image quality, 87 Battery level, 90 Focus frame, 91 Tracking frame, 150 Face frame, 152 Eye frame, 155 Tracking icon, 157 Switching display, 100 Body, 101 Display panel, 102 Viewfinder, 103 Grip, 110 Controls, 110S Shutter button, 110C Assignable button, 110E Enter button, WDp Priority face frame, WDe Non-priority face frame, WHf Eye focus frame, WHd Eye detection frame

Claims

1. A detection unit that detects pupils in a captured image; an autofocus control unit that, in a still image capture mode, executes pupil autofocus control in response to a predetermined operation so as to perform autofocus on the pupil detected by the detection unit, and, in a video capture mode, executes the pupil autofocus control in response to the pupil being detected by the detection unit; a display control unit that performs control to display a pupil frame indicating a focused pupil on an imaging monitor image in response to focusing by the pupil autofocus control, The display control unit is In a still image imaging mode, a pupil focusing frame is displayed as the pupil frame in response to focusing by the pupil autofocus control; In a moving image imaging mode, control is performed to display, as the pupil frame, a pupil detection frame in a display mode different from that of the pupil focusing frame in response to focusing by the pupil autofocus control. Imaging device.

2. The display control unit is In the case where it is automatically set whether the target of the pupil autofocus control is the left or right pupil in the still image imaging mode, when the pupil autofocus control is performed in response to the predetermined operation, control is performed in which the pupil frame is not displayed until the focus is achieved. The imaging device according to claim 1 .

3. The pupil detection frame and the pupil focusing frame are displayed as frames of different colors. The imaging device according to claim 1 .

4. The predetermined operation is a half-press of the shutter button. The imaging device according to claim 1 .

5. Eye autofocus control modes include: an auto mode in which the target of the pupil autofocus control is automatically set to either the left or right pupil; a fixed mode in which the target of the pupil autofocus control is a pupil designated by a user operation; is selectable, The display control unit is When the pupil is not focused on by the pupil autofocus control in the video imaging mode, In the case of the above auto mode, the pupil frame is not displayed, In the case of the fixed mode, control is performed to display the pupil frame. The imaging device according to claim 1 .

6. In the auto mode, A temporary selection state can be set so that the pupil autofocus control is performed by temporarily fixing a pupil designated by a user operation as a target, The display control unit is When the pupil is not focused on by the pupil autofocus control in the video imaging mode, In the case of the temporary selection state, control is performed to display the pupil frame. The imaging device according to claim 5 .

7. A detection unit that detects pupils in a captured image; an autofocus control unit that, in a still image capture mode, executes pupil autofocus control in response to a predetermined operation so as to perform autofocus on the pupil detected by the detection unit, and, in a video capture mode, executes the pupil autofocus control in response to the pupil being detected by the detection unit; a display control unit that performs control to display a pupil frame indicating a focused pupil on an imaging monitor image in response to focusing by the pupil autofocus control, The display control unit is In the video imaging mode, when the pupil autofocus control is performed, a pupil detection frame is displayed as the pupil frame until focusing is achieved, and a pupil focusing frame having a display mode different from that of the pupil detection frame is displayed as the pupil frame upon focusing. Imaging device.

8. a face frame display according to face detection by the detection unit can be selectively turned on / off; In a video imaging mode, the display control unit Depending on the on / off selection, the face frame display is controlled. The imaging device according to claim 1 .

9. Eye autofocus control modes include: an auto mode in which the target of the pupil autofocus control is automatically set to either the left or right pupil; a fixed mode in which the target of the pupil autofocus control is a pupil designated by a user operation; is selectable, When the display control unit selects to turn off the display of the face frame, In the case of the above-mentioned auto mode, the face frame is not displayed, In the fixed mode, the face frame is not displayed, and the pupil frame is displayed regardless of whether the subject is in focus or not by pupil detection. The imaging device according to claim 8.

10. In the auto mode, A temporary selection state can be set so that the pupil autofocus control is performed by temporarily fixing a pupil designated by a user operation as a target, When the display control unit selects to turn off the display of the face frame, In the case of the temporary selection state, the face frame is not displayed, and the pupil frame is displayed by pupil detection regardless of whether the subject is in focus. The imaging device according to claim 9.

11. When the display control unit selects to turn off the display of the face frame, In the case of the auto mode, the face frame is not displayed, and the pupil frame is displayed in response to focusing on the pupil. The imaging device according to claim 9.

12. One of the modes for pupil autofocus control is An auto mode is selectable in which the target of the pupil autofocus control is automatically set to either the left or right pupil, In the auto mode, a temporary selection state can be set in which a pupil designated by a user operation is temporarily fixed as a target and the pupil autofocus control is performed, In the case of a still image capture mode, the temporary selection state is released in response to a release operation instructing recording of a still image, In the case of a video shooting mode, the temporary selection state is released in response to an operation different from the release operation. The imaging device according to claim 1 .

13. In still image capture mode, the autofocus control unit starts the pupil autofocus control based on pupil detection targeting an entire captured image in response to an operation instructing a pupil autofocus operation; the display control unit causes a pupil focusing frame to be displayed as a pupil frame in response to focusing; In the video capture mode, the autofocus control unit switches, in response to the operation, autofocus control targeting a face to the pupil autofocus control based on pupil detection targeting an entire captured image; The display control unit displays an pupil detection frame as a pupil frame in response to pupil detection. The imaging device according to claim 1 .

14. A touch operation on a display screen of the captured monitor image is detectable, The autofocus control unit is In the case of a still image capturing mode, the pupil autofocus control is performed by tracking the pupil in response to a touch operation, In the case of a video capture mode, the face is tracked in response to a touch operation and the pupil autofocus control is performed on the selected pupil. The imaging device according to claim 1 .

15. The autofocus control unit is In still image capture mode, performing the pupil autofocus control to track the pupil closer to the touch position in response to a touch operation on the face in the captured monitor image; In response to a touch operation on a part of the image captured by the monitor other than the face, the face closest to the touch position is tracked and autofocus control is performed on the face. The imaging device according to claim 14.

16. A detection process for detecting an eye in a captured image; an autofocus control process that, in a still image capture mode, executes a pupil autofocus control so as to perform autofocus on the pupil detected in the detection process in response to a predetermined operation, and, in a video capture mode, executes the pupil autofocus control in response to the pupil being detected in the detection process; a display control process that performs control to display a pupil focusing frame as a pupil frame indicating a focused pupil on an imaging monitor image in response to focusing by the pupil autofocus control in a still image imaging mode, and to display a pupil detection frame in a display mode different from the pupil focusing frame as the pupil frame in response to focusing by the pupil autofocus control in a video imaging mode; The imaging method is performed by an imaging device.

17. A detection process for detecting an eye in a captured image; an autofocus control process that, in a still image capture mode, executes a pupil autofocus control so as to perform autofocus on the pupil detected in the detection process in response to a predetermined operation, and, in a video capture mode, executes the pupil autofocus control in response to the pupil being detected in the detection process; a display control process that performs control to display a pupil focusing frame as a pupil frame indicating a focused pupil on an imaging monitor image in response to focusing by the pupil autofocus control in a still image imaging mode, and to display a pupil detection frame in a display mode different from the pupil focusing frame as the pupil frame in response to focusing by the pupil autofocus control in a video imaging mode; A program that causes a processor to execute the above.

Images