Focusing device and method, and imaging apparatus

Abstract

To automatically determine a main subject to be focused and flexibly change the main subject according to a user's intention.SOLUTION: A focusing device comprises: detection means that photoelectrically converts light incident through a photographic optical system with imaging means and detects main subject candidates to be the candidates for a main subject from image signals that are repeatedly output; selection means that selects a main subject from the main subject candidates; acquisition means that acquires operation information on operating means for moving the position of a focus lens included in the photographic optical system; and focusing means that, when an autofocus mode is set and the operating means is not operated, controls the focus lens so that the main subject selected by the selection means is focused. When the operating means is operated, the selection means selects the main subject based on the focusing position of the focus lens and fixes the selected main subject.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a focusing device and method, and an imaging device. [Background technology]

[0002] In conventional autofocus (AF) control, the user must specify the AF frame for the subject to be focused on. However, for example, in shoulder-mounted cameras used for reporting, it can be difficult to install an operating device for operating the AF frame in a location that is easy for the operator to operate.

[0003] Therefore, Patent Document 1 discloses a technique for automatically setting an AF frame for a subject focused on by a manual focus (MF) operation performed by an operator.

[0004] Furthermore, Patent Document 2 discloses a technology in which, if the position where the focus is adjusted as a result of manual focus operation performed by an operator is within a face detection area, an AF frame is set on the face if a face is detected, or on a subject other than the face if a face is not detected, and automatic tracking and AF are performed.

[0005] On the other hand, image processing that automatically detects a specific subject from an image is known. For example, in digital cameras and digital video cameras, the face of a person is detected from a captured image, and the focus and exposure are optimized based on the detection result. However, in general face detection technology, stable detection is not possible when the facial features cannot be obtained sufficiently, such as when the person is facing away from the camera or wearing an ornament on the face.

[0006] Therefore, Patent Document 3 proposes a method of detecting the upper body or head of a person from an image and controlling the area of ​​the person's upper body or head, or the area of ​​the face estimated from the detection results. With this method, it is possible to stably identify the face area of ​​a person even if the person is facing away from the camera or has an accessory on their face. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2010-97167 A [Patent Document 2] Patent No. 5474528 [Patent Document 3] JP 2017-126915 A Summary of the Invention [Problem to be solved by the invention]

[0008] However, focusing with MF operation requires skilled technique, and it is difficult to smoothly move the focus lens to the in-focus position. Furthermore, in recent years, the number of pixels has increased, leading to 4K and 8K resolutions, making it very difficult to check the in-focus position on a monitor attached to a camera. Therefore, as with the technique described in Patent Document 2, if an operator performs manual focus operation until the face is approximately in focus, the quality of the video may be reduced in terms of focusing technique and the occurrence of blurring due to resolution.

[0009] On the other hand, in the method of tracking a subject using face detection technology described in Patent Document 3, if a user wants to focus on an arbitrary subject other than the tracked subject on the screen, the user cannot change the focus.

[0010] The present invention has been made in consideration of the above problems, and has an object to automatically determine a main subject to be focused on, and to enable a user to flexibly change the main subject in accordance with his or her intention. [Means for solving the problem]

[0011] In order to achieve the above object, the focus adjustment device of the present invention comprises: a detection means for photoelectrically converting light incident through a photographing optical system by an imaging means, and detecting main subject candidates that are candidates for the main subject from image signals that are repeatedly output based on a predetermined first condition; a selection means for selecting the main subject from the main subject candidates; an acquisition means for acquiring operation information of an operation means for moving the position of a focus lens included in the photographing optical system; and a focus adjustment means for controlling the focus lens to focus on the main subject selected by the selection means when autofocus mode is set and the operation means is not operated, and is characterized in that when the operation means is operated, the selection means selects the main subject based on the focus position of the focus lens and fixes the selected main subject. Effect of the Invention

[0012] According to the present invention, it is possible to automatically determine the main subject to be focused on, and to flexibly change the main subject in accordance with the user's intention. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing the configuration of a digital video camera system according to a first embodiment of the present invention. [Diagram 2] 5 is a flowchart of a focus adjustment process according to the first embodiment. [Diagram 3] 5 is a flowchart of a parameter setting process according to the first embodiment. [Figure 4] 5A to 5C are views for explaining focus detection frame setting processing according to the first embodiment. [Diagram 5] 5 is a flowchart of a main subject candidate determination process according to the first embodiment. [Figure 6] 6 is a flowchart of a main subject change process according to the first embodiment. [Figure 7] 6 is a flowchart of a switching subject identification process according to the first embodiment. [Figure 8]6 is a flowchart of a main subject lock determination process in the first embodiment. [Figure 9] 5A and 5B are conceptual diagrams showing an example of a frame display that indicates a subject detected during MF operation and when MF operation is not being performed in the first embodiment. [Figure 10] 5A and 5B are conceptual diagrams showing an example of frame display indicating a main subject and main subject candidates when MF operation is in progress and when MF operation is not in progress in the first embodiment. [Figure 11] 5A and 5B are conceptual diagrams showing examples of frame display showing main subject candidates and the main subject in the first embodiment. [Figure 12] 10 is a flowchart of a main subject candidate determination process according to the second embodiment. [Figure 13] 10 is a flowchart showing main subject candidate reset time setting processing in the second embodiment. [Figure 14] 10 is a flowchart showing a main subject reset time adjustment process in the second embodiment. [Figure 15] 10 is a flowchart showing a reset time adjustment process for a main subject candidate in the second embodiment. [Figure 16] 13A and 13B are conceptual diagrams showing areas for adjusting a reset time for a main subject and a reset time for a main subject candidate, and distances between subjects in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0015] <First embodiment> A first embodiment of the present invention will be described in detail below. Note that in the following embodiment, a digital video camera system will be taken as an example of an image capturing apparatus, but the present invention is not limited to this.

[0016] FIG. 1 is a block diagram showing the configuration of a lens-interchangeable digital video camera system according to an embodiment of the present invention. The camera system according to this embodiment is composed of a lens unit 10 and a camera body 20, and data is exchanged between a lens control unit 106 that controls the overall operation of the lens unit 10 and a camera control unit 207 that controls the overall operation of the camera system. In this embodiment, the camera control unit 207 is composed of a processor such as a CPU or MPU, and a storage unit such as a memory. The camera control unit 207 may also include an arithmetic circuit, which may execute some of the arithmetic functions performed by the processor. Note that, in this embodiment, an example of a lens-interchangeable camera system in which the lens unit 10 is detachable from the camera body 20 will be described, but the present invention can also be applied to a camera that is integrated with a lens.

[0017] First, the configuration of the lens unit 10 will be described. The lens unit 10 has an imaging optical system including a fixed lens 101, a zoom lens 108, an aperture 102, and a focus lens 103. The aperture 102 is driven by an aperture drive unit 104 to control the amount of light incident on an image sensor 201, which will be described later. The focus lens 103 is driven by a focus lens drive unit 105 to adjust the focus. The zoom lens 108 is driven by a zoom lens drive unit 109 to adjust the zoom magnification. Note that a lens unit 10 without a zoom function may be used.

[0018] The aperture driving unit 104, the focus lens driving unit 105, and the zoom lens driving unit 109 are controlled by the lens control unit 106, which determines the aperture amount of the aperture 102 and the positions of the focus lens 103 and the zoom lens 108. When a user operates the aperture, focus, zoom, or the like via the lens operation unit 107, the lens control unit 106 performs control according to the user operation. Furthermore, the lens control unit 106 controls the aperture driving unit 104, the focus lens driving unit 105, and the zoom lens driving unit 109 according to a control command and control information received from a camera control unit 207, which will be described later. Furthermore, the lens control unit 106 transmits lens information (for example, MF operation information and information about the photographing optical system) to the camera control unit 207.

[0019] Next, the configuration of the camera body 20 will be described. The camera body 20 is configured to be able to acquire an image signal and an AF signal from a light beam that has passed through the photographing optical system of the lens unit 10. The image sensor 201 is configured using a CCD, a CMOS sensor, or the like. The light beam that has passed through the photographing optical system is incident on the light receiving surface of the image sensor 201, and the formed subject image is converted by the photodiodes into charges according to the amount of incident light (photoelectric conversion). The charges accumulated in each photodiode are sequentially read out from the image sensor 201 as an image signal and an AF signal, based on a drive pulse provided by a timing generator 209 in accordance with a command from a camera control unit 207, and a voltage signal according to the charges is read out.

[0020] The image signal and AF signal read out from the image sensor 201 are input to a CDS / AGC circuit 202, which performs correlated double sampling to remove reset noise, adjusts the gain, and digitizes the signal. The CDS / AGC circuit 202 then outputs the obtained image data to a camera signal processing unit 203, a face area detection unit 210, and a head area detection unit 211, and outputs AF data to an AF signal processing unit 204.

[0021] The camera signal processing unit 203 transmits the image data output from the CDS / AGC circuit 202 to the display unit 205. The display unit 205 is a display device (display member) configured using an LCD, an organic EL, or the like, and displays an image based on the image data. In addition, in a mode for recording an image, the image data is recorded in a recording unit 206.

[0022] The AF signal processing unit 204 calculates information on the focus state and reliability by a known method based on the AF data output from the CDS / AGC circuit 202. The information on reliability may include contrast information, saturation information, scratch information, etc., and in the case of the image sensor 201 that supports a so-called image plane phase difference method, the degree of coincidence of two images, the degree of steepness of two images, etc. Then, the calculated focus state and reliability information are output to the camera control unit 207.

[0023] The face area detection unit 210 (first detection means) performs a known face detection process on the image data output from the CDS / AGC circuit 202 to detect the face area (first area) of a person in the shooting screen. That is, the face area detection unit 210 detects a predetermined subject (here, a face) from the image data. The obtained face detection result is transmitted to the camera control unit 207. The face detection process performed by the face area detection unit 210 includes, for example, a method of extracting a skin color area from the gradation color of each pixel represented by the image data and detecting a face based on the degree of matching with a face outline plate prepared in advance. There is also a method of detecting a face by extracting facial feature points such as eyes, nose, and mouth using a known pattern recognition technology, but the present invention is not limited to the method of face detection process, and any method may be used. In addition, it is not limited to a face, and a subject other than a human face may be used.

[0024] Head region detection section 211 (second detection means) detects a target subject region from within an image as a head region (second region), which is a region related to a face region (first region) detected by face region detection section 210. That is, head region detection section 211 detects a predetermined subject (here, a head) related to the predetermined subject detected by face region detection section 210 from image data. In head region detection in this embodiment, a head region is detected from image data based on learning data, and the detection result is transmitted to camera control section 207.

[0025] The camera control unit 207 transmits information to the AF signal processing unit 204 to set a focus detection area at a position including the face area and head area in the shooting screen based on the face detection result of the face area detection unit 210 and the head detection result of the head area detection unit 211. In the first embodiment, if a face is detected, the focus detection area is set based on the position and size of the face area. On the other hand, if a face is not detected and only the head area is detected, the focus detection area is set based on the position and size of the head area. Note that if only the head area is detected, the position and size of the subject's face area may be estimated from the head detection result, and the focus detection area may be set based on the estimated position and size.

[0026] The camera control unit 207 also exchanges information with each component in the camera body 20 to perform control, and executes not only the processing in the camera body 20 but also the camera functions operated by the user in response to input from the camera operation unit 208. The camera functions operated by the user include, for example, turning the power on / off, changing settings, starting recording, switching between AF (autofocus) / MF (manual focus) control, and checking recorded images. As described above, the camera control unit 207 also exchanges information with the lens control unit 106 in the lens unit 10, sends control commands and control information for the photographing optical system, and acquires information in the lens unit 10. The camera control unit 207 also performs processing to calculate the drive amount of the focus lens 103 to achieve a focused state, using the information in the lens unit 10 from the lens control unit 106 and the focus state and reliability information from the AF signal processing unit 204.

[0027] Furthermore, camera control unit 207 includes main subject candidate determination unit 220, main subject lock unit 221, and main subject candidate reset determination unit 222. Main subject candidate determination unit 220 determines whether or not a subject can be a main subject candidate based on the face detection result obtained by face area detection unit 210 and the head detection result obtained by head area detection unit 211. Then, from among the subjects determined to be main subject candidates, a main subject is selected by a main subject determination process described below, and a process is performed to superimpose a frame display indicating the main subject and the main subject candidate on the captured image, and displayed on display unit 205 or the like.

[0028] Next, the focus adjustment process in the first embodiment will be described with reference to Fig. 2. The focus adjustment process in the first embodiment is characterized by control based on the detection results of the face area and head area by the face area detection unit 210 and the head area detection unit 211. This focus adjustment process is executed according to a computer program stored in the camera control unit 207. Also, the series of focus adjustment processes described below are assumed to be performed in a system synchronized with the imaging cycle, but this is not limiting.

[0029] First, in S201, the face area detection unit 210 performs a parameter setting process for subject detection, which sets parameters to be used for subject detection. Details of the parameter setting process will be described later with reference to FIG. 3. Then, using the set parameters depending on whether AF or MF is selected, the face area detection unit 210 detects a face area and the head area detection unit 211 detects a head area. Then, the process proceeds to S202.

[0030] Next, in S202, the AF signal processing unit 204 performs a focus detection frame setting process to set a focus detection frame for the AF signal output from the CDS / AGC circuit 202, and detects the focus state for each of the set focus detection frames. Note that the focus detection frame setting process performed here will be described later with reference to Fig. 4. Then, the process proceeds to S203.

[0031] In S203, the face detection result by face area detection unit 210, the head detection result by head area detection unit 211, and the tracking and search result are acquired, subject information is generated, and the process proceeds to S204. Note that the subject information generated here includes information on the detection ID, face detection result, head detection result, tracking and search result, position and size of the subject area, main subject candidate flag, main subject flag, main subject lock flag, and focus detection result.

[0032] The camera control unit 207 detects a subject based on the acquired face detection result and head detection result, and assigns the detection ID to the detected subject. At that time, the camera control unit 207 determines the face area and the head area, and assigns the same detection ID to the face area and the head detection result determined to be of the same subject. Furthermore, a tracking search process is performed to determine whether or not the same subject as the subject detected in the current frame is also detected in the previous frame. This tracking search process can be performed using a known method such as template matching or histogram matching using the image of the previous frame. If it is determined that the same subject is present in the image of the previous frame as a result of the tracking search process, the same detection ID as the detection ID assigned to that subject is assigned to the subject detected in the current frame.

[0033] The face detection result is information relating to the position and size of the face region detected by face region detection section 210, and the head detection result is information relating to the position and size of the head region detected by head region detection section 211. Note that instead of the face detection result and the head detection result, information indicating whether or not corresponding information exists and a link for accessing the corresponding information if it exists may be held.

[0034] The tracking and search result indicates information as to whether or not the same subject was found in the image of the previous frame by the above-mentioned tracking and search process.

[0035] The subject area indicates information about the position and size of the subject determined based on the face detection results and head detection results.

[0036] The main subject candidate flag indicates information added by the main subject candidate determination process in S204, which will be described later. The main subject flag is information added by the main subject change process performed in S206, which will be described later, and the main subject lock determination process performed in S208, and is set for only one subject. The main subject lock flag is also information added by the main subject change process performed in S206, which will be described later, and the main subject lock determination process performed in S208, and is a flag used when fixing the main subject flag. Note that, when the same subject is present in the image of the previous frame, the information of these flags for the same detection ID is carried over in S203.

[0037] The focus detection result indicates the focus state detected in S202.

[0038] In S204, the camera control unit 207 performs a main subject candidate determination process in which a main subject candidate is determined from the subjects detected in S203 and a main subject candidate flag is set. This process will be described later with reference to Fig. 5. Then, the process proceeds to S205.

[0039] In S205, it is determined whether or not MF operation is in progress. MF operation information can be acquired by the lens operation unit 107 or the camera operation unit 208, and there are basically three patterns as shown below.

[0040] (1) Set to MF and operate the lens unit 10 directly The lens is set to a state in which MF operation is possible by switching an AF / MF switch (not shown) attached to the lens unit 10 using the lens operation unit 107, or by switching between an AF mode (autofocus mode) and an MF mode (manual focus mode) (not shown) from a camera menu using the camera operation unit 208. In this state, the lens operation unit 107 is directly operated to drive the focus lens driving unit 105.

[0041] (2) Setting to MF and operating the lens unit 10 from the camera body 20 From the camera operation unit 208, switching to MF mode (not shown) is performed through a menu of the camera body 20, and the lens unit 10 is set to a state in which MF operation is possible. In this state, the user sets drive information for manual operation, and the information is transmitted to the lens control unit 106 via the camera control unit 207, thereby driving the focus lens drive unit 105.

[0042] (3) Directly operate the lens unit 10 while keeping the AF setting With the lens unit 10 and the camera body 20 left in the AF setting, the focus lens driving unit 105 is driven by directly operating the lens operation unit 107. In this state, the mode returns to AF when the operation is completed.

[0043] If an MF operation is being performed, the process proceeds to S206, and if not, the process proceeds to S207.

[0044] In S206, a main subject change process is performed to specify a subject that the user wants to photograph from among the subjects for which the main subject candidate flag has been set, and to set the main subject flag. This process will be described later with reference to Fig. 6. Then, the process proceeds to S211.

[0045] On the other hand, in S207, it is determined whether or not a subject designation instruction has been given by the user from the camera operation unit 208. If it is determined that a subject designation instruction has been given, the process proceeds to S208, and if not, the process proceeds to S209. Note that possible subject designation methods include, for example, a method of designating a subject using a touch panel, or a method of designating a subject by detecting a direction or pressure, such as a cross key.

[0046] In S208, main subject locking processing is performed by main subject locking unit 221. The main subject locking processing is processing for determining a main subject based on a user instruction, regardless of the main subject candidate result determined in S204. This processing will be described later with reference to Fig. 8. Then, the process proceeds to S211.

[0047] In S209, it is determined whether or not the main subject is locked by determining whether or not a main subject lock flag is set by the process described later in S208. If the main subject is not locked, the process proceeds to S210, and if it is locked, the process proceeds to S211.

[0048] In S210, a main subject is determined from among the main subject candidates determined in the main subject candidate determination process in S204, based on the position and size of the person. Note that the parameters for determining the main subject may be determined not only based on the position and size, but also based on the reliability and orientation of the face detection and head detection results, whether both the face and head are detected, or only one of them is detected, etc. Also, the main subject may be determined from among the main subject candidates using a known algorithm.

[0049] In S211, a frame display process is performed in which a frame is displayed on the display unit 205 around a subject for which a main subject candidate flag is set, and a user is notified. Specifically, a main subject frame indicating that the subject is a main subject is displayed around the face or head of the main subject determined in S206, S208, and S210. Also, among subjects for which the main subject candidate flag is ON, a candidate frame indicating that the subject is a main subject candidate is displayed around the face or head for which the main subject flag is not ON. Furthermore, when the main subject lock flag is ON, only the main subject frame is displayed, and a candidate frame is not displayed around the face or head for which the main subject candidate flag is ON other than the main subject.

[0050] In S212, it is determined whether MF is set or MF operation is in progress. If MF is set or MF operation is in progress, the process proceeds to S213, and lens control processing is performed to send a drive command for focus lens 103 to lens control unit 106 based on the focus detection result of the main subject for which the main subject flag is ON. If there is no subject for which the main subject flag is ON, for example, a focus detection frame is set in the center of the screen, and lens control processing is performed based on the focus detection result, etc. Note that since conventional methods may be used for the focus detection method and lens control processing, detailed explanations are omitted here.

[0051] It should be noted that if MF is set or if MF operation is in progress, the user is operating the focus lens 103, and so the process ends immediately.

[0052] Next, the parameter setting process performed in S201 will be described with reference to Fig. 3. The parameter setting process is a process for setting parameters referenced in the face area detection unit 210. In this embodiment, the threshold setting is made different between when MF operation is in progress and when MF operation is not in progress, and during MF operation, even small faces or faces with low reliability are made easier to detect in order to detect the subject targeted by the user.

[0053] First, in S301, it is determined whether or not an MF operation is in progress. Note that this determination is made by determining whether an operation according to any of the three patterns described above is being performed. If an MF operation is in progress, the process proceeds to S302, and if not, the process proceeds to S305.

[0054] In S302, a threshold value FaceTh1 for the minimum detectable face size is set. For example, when the threshold value is set based on the pixel size on the image sensor 201, if a face of 15 pixels or more is to be detected, the threshold value is set to "15".

[0055] Next, in S303, a threshold value RelianceTh1 of the reliability of face detection is set. For example, when the reliability of the face detection result is output in 10 stages and a face with a reliability of 3 or more is to be detected, the threshold value RelianceTh1 is set to "3".

[0056] Then, in S304, a threshold value AreaTh1 for the range of face detection is set, and the process ends. For example, if it is desired to detect the target range of face detection in 90% of the screen, the threshold value is set to "90".

[0057] On the other hand, if it is determined in S301 that MF operation is not in progress, the process proceeds to S305, and a threshold value FaceTh2 for the minimum detectable face size is set. For example, when the threshold value is set based on the pixel size on the image sensor 201, if a face of 25 pixels or more is to be detected, the threshold value is set to "25."

[0058] Next, in S306, a threshold value RelianceTh2 of the reliability of face detection is set. For example, when the reliability of the face detection result is output in 10 stages and a face with a reliability of 6 or more is to be detected, the threshold value RelianceTh2 is set to "6".

[0059] Then, in S307, a threshold value AreaTh2 for the range of face detection is set, and the process ends. For example, if the target range of face detection is to be detected in 80% of the screen, it is set to "80".

[0060] The above threshold relationship is FaceTh1 <FaceTh2、RelianceTh1<RelianceTh2、AreaTh1> AreaTh2 is set to make face detection easier during MF. In this way, during MF operation, the lower limit of the minimum face size is made smaller, the lower limit of the reliability of face detection is lowered, and the face detection range is made wider than when MF operation is not in progress.

[0061] The subject detection result obtained when the parameters are set as above will be described with reference to Fig. 9. Fig. 9(a) shows a case where MF operation is not in progress and subjects 901 and 902 are included on the screen. In this case, subject 901 is detected by subject detection, but subject 902 is not detected because its face is smaller than the minimum face size threshold FaceTh2 set in S305. Furthermore, if subject 902 is located on the periphery of the screen and is outside the face detection range threshold AreaTh2 set in S307, it is unlikely to become the main subject in the first place, so subject detection itself is restricted.

[0062] If subject 901 is determined to be the main subject by the processing in S208 or S210 based on such subject detection results, a main subject frame 903 indicating that subject 901 is the main subject is displayed on subject 901 in S211. On the other hand, no frame is displayed on subject 902. Also, only subject 901 is subject to focus detection processing.

[0063] 9B shows a case where the same subject is in MF operation. During MF operation, there is a possibility that the user intentionally performs focus operation on subject 902. Therefore, by lowering the face detection threshold and widening the face detection range to change to a state where face detection is easier, subjects 901 and 902 are both detected.

[0064] If subject 901 is determined to be the main subject by the process in S206 based on such subject detection results, then in S211, a main subject frame 903 indicating the main subject is displayed on subject 901. Furthermore, a candidate frame 904 is displayed on subject 902, and both subjects 901 and 902 become targets of focus detection processing. In this way, during MF operation, subjects that were not selected as candidates for the main subject when MF operation was not in progress can also be subject to focus detection processing, making it possible to switch the main subject in a way that reflects the user's intention.

[0065] Next, the focus detection frame setting process performed in S202 will be described with reference to Fig. 4. The focus detection frame setting process is a process for setting focus detection frames in a mesh pattern on the imaging surface, and focus detection is performed based on the AF signal obtained from each of the frames.

[0066] In this embodiment, the method of setting the focus detection frame differs depending on whether or not MF operation is in progress. In S202, if MF operation according to any of the above-mentioned patterns (1) to (3) is detected, it is determined that MF operation is in progress, and a focus detection frame is set that corresponds to whether MF operation is in progress or not.

[0067] In Fig. 4, subjects 401, 402, and 403 are examples of subjects detected in S202 by the face area detection unit 210 and the head area detection unit 211 using the parameters set in S201. Fig. 4(a) shows a method of setting a focus detection frame when MF operation is not in progress, in which focus detection frames 405, 406, and 407 are set for subjects 401, 402, and 403 according to the size of the subjects. This setting method has the following advantages. Efficient focus detection processing is possible, eliminating unnecessary areas other than the subject. The size of the frame varies depending on the size of the subject, so there is no need to set a large focus detection frame for a small face. By reducing perspective conflicts within the frame, stable focus detection results can be achieved.

[0068] In the example shown in Figure 4(a), for the sake of explanation, a focus detection frame is set for each subject, but when MF operation is not in progress, focus adjustment is only performed on subjects whose main subject flag is ON, so it is sufficient to set one focus detection frame.

[0069] 4B shows a method for setting the focus detection frame during MF operation. During MF operation, a mesh-shaped frame 408 is set centered on the center of the screen, regardless of the subject detection result. This setting method has the following advantages. -Focus detection results for multiple subjects can be obtained at once. The number of focus detection frames does not increase even if the number of subjects detected by subject detection increases.

[0070] In this way, by changing the method of setting the focus detection frame depending on whether MF operation is in progress or not, stable focus adjustment is possible when MF operation is not in progress. On the other hand, during MF operation, focus detection results for multiple subjects are obtained efficiently, allowing for quick response to switching of the main subject.

[0071] Next, the main subject candidate determination process performed in S204 will be described with reference to Fig. 5. This process is a process in which the camera control unit 207 sets a main subject candidate flag for a subject that is a candidate for the main subject among the subjects detected in S203. The main subject candidate flag is a flag that indicates a subject that has the possibility of becoming the main subject based on the detection result.

[0072] First, in S501, it is determined whether or not there is a face detection result for one detection ID among the subject information acquired in S203. If there is a face detection result, the process proceeds to S502, and if there is no face detection result, the process proceeds to S503. In S502, the position and size of the face area are stored as information on the subject area. Then, the process proceeds to S510.

[0073] In S503, it is determined whether or not an MF operation is in progress. If an MF operation is in progress, the process proceeds to S504, and if not, the process proceeds to S506.

[0074] In S504, it is determined whether or not there is a head detection result. If there is a head detection result, the process proceeds to S505. On the other hand, if there is no head detection result, the process proceeds to S512 to turn off the main subject candidate flag, and then proceeds to S517.

[0075] In S505, the position and size of the subject's head area are stored as subject area information. In this way, during MF operation, even if a face cannot be detected, if the head area is detected, it is stored as subject area information and the process proceeds to S510.

[0076] In S506, it is determined whether the main subject candidate flag is ON. This is ON if the main subject candidate flag is set for a subject having the same detection ID in the previous frame, and OFF otherwise. If the main subject candidate flag is ON, the process proceeds to S505, and the position and size of the subject's head region are stored as subject region information. In this way, if the face region of the same subject has been detected in the past, or the head region has been detected during MF operation, and the main subject candidate flag is ON, even if the face region is not detected in the current frame, the head region information is stored as subject region information.

[0077] On the other hand, if the main subject candidate flag is OFF, the process proceeds to S512, where the main subject candidate flag is turned OFF, and then the process proceeds to S517.

[0078] In S510, the main subject candidate flag is set to ON, and the process proceeds to S513.

[0079] In S513, it is determined whether or not MF operation is in progress, and if MF operation is in progress, the process proceeds to S514, and if not, the process proceeds to S516.

[0080] In S514, the position of the subject being processed for which the main subject candidate flag is ON is identified on the focus detection frame. In this process, it is identified in which area of ​​the focus detection frame 408 in FIG. 4B, which is arranged in a mesh pattern, the center position of the subject being processed is located. This is a process required to identify which main subject candidate is desired to be focused on during MF operation. Then, the process proceeds to S515.

[0081] In S515, focus detection is performed based on the AF signal within the focus detection frame 408 at the position specified in S514, the focus detection result is stored, and the process proceeds to S517.

[0082] Meanwhile, in S516, focus detection is performed based on the AF signal within the position and size stored in the subject area in either S502, S505, or S509, and the focus detection result is stored. Since MF operation is not in progress here, the focus detection result of the focus detection frame set for each subject, such as focus detection frames 405, 406, and 407 in Fig. 4(a), is obtained. Then, the process transitions to S517.

[0083] In S517, it is determined whether there are any unprocessed subjects (detection IDs), and if the main subject candidate flag setting process and the focus detection result storage process are completed for all detection results, the process ends. If there are any unprocessed subjects (detection IDs), the process returns to S501 and the above-mentioned process is repeated.

[0084] In this manner, in this embodiment, the determination of the head detection result is changed depending on whether MF operation is in progress or not. That is, during MF operation, in order to detect the subject targeted by the user, even a small face in which the eyes and nose cannot be detected or a face that does not face forward is flagged as a candidate for the main subject. On the other hand, when MF operation is not in progress, a small face in which the eyes and nose cannot be detected or a face that does not face forward is generally unlikely to become the main subject, so a flag is not set as a candidate for the main subject. However, when face detection of a subject that was previously possible becomes impossible, if the head detection result is valid, the main subject candidate flag is still set. This makes it easier to capture the subject targeted by the user during MF operation, and when not, it becomes possible to perform focus adjustment control on a face that is likely to be the main subject.

[0085] An example of setting the main subject candidate flag as described above will be described with reference to FIGS.

[0086] FIG. 9(c) shows a case where MF operation is not in progress and subjects 905 and 906 are included on the screen. Since a face region has been detected for subject 905, the main subject candidate flag is ON. On the other hand, this shows a case where a face region has not been detected for subject 906, but a head region has been detected. In this case, if the main subject candidate flag for the previous frame was OFF, the main subject candidate flag will continue to be OFF. This is because subject 906 has only had its head region detected, and its face has not been detected in the past, so it is unlikely to become the main subject.

[0087] If subject 905 is determined to be the main subject by the processing in S208 or S210 based on the main subject candidate flag thus set, then in S211 a main subject frame 907 indicating that subject 905 is the main subject is displayed around subject 905. On the other hand, no frame is displayed around subject 906. Also, only subject 905 is subject to focus detection processing.

[0088] 9(d) shows the same subject during MF operation. A face region has been detected for subject 905, so the main subject candidate flag is ON. Similarly, a face region has not been detected for subject 906, but a head region has been detected, so the main subject candidate flag is ON.

[0089] If subject 905 is determined to be the main subject by the process in S206 based on the main subject candidate flag thus set, then in S211, a main subject frame 907 indicating the main subject is displayed around subject 905. Also, a candidate frame 908 is displayed around subject 906, and both subjects 905 and 906 become targets of focus detection process.

[0090] In this state, if the user intentionally performs a focus operation on subject 906, there is a possibility that the user wants to make subject 906 the main subject. Therefore, even if a face has not been detected, if there is a head detection result, candidate frame 908 is displayed and the subject is made the target of focus detection processing. In this way, during MF operation, subjects that were not selected as main subject candidates when not in MF operation can be made the target of focus detection processing, making it possible to switch the main subject in a way that reflects the user's intention.

[0091] Next, the main subject change process performed in S206 in Fig. 2 will be described with reference to Fig. 6. This process is executed when MF operation is in progress.

[0092] In S601, it is determined whether there is one or more subjects with the main subject candidate flag set to ON. If there is one or more subjects with the main subject candidate flag set to ON, the process proceeds to S604, and if there is none, the process proceeds to S602.

[0093] In S602, the main subject lock flags of all detected subjects are set to OFF, and the process proceeds to S603. In S603, the main subject flags of all detected subjects are set to OFF, and the main subject change process ends. In this case, neither the main subject frame nor the candidate frame is displayed in S211.

[0094] On the other hand, in S604, it is determined whether there are two or more subjects with the main subject candidate flag set to ON. In order to switch the main subject, at least two or more subjects are required. Therefore, if there are two or more subjects, the process proceeds to S606, and if not, the process proceeds to S605.

[0095] In S605, since there is only one subject whose main subject candidate flag is ON, the main subject flag of that subject is set to ON, and the main subject change process ends.

[0096] In S606, a changeover subject identification process is performed to identify a changeover of the main subject from the operation information of the lens unit 10 and the focus detection result, and the process proceeds to S607. The process of S606 will be described later with reference to FIG.

[0097] In S607, it is determined whether or not a change in the main subject has occurred. If a change has occurred, the process proceeds to S608, and if not, the process proceeds to S610.

[0098] In S608, the main subject lock flag is set ON for the switching destination subject identified in S606, and further in S609, the main subject flag of the switching destination subject identified in S606 is set ON, and the main subject change process ends.

[0099] In S610, since the main subject has not been switched, the main subject flag of the subject having the same detection ID as the previous main subject is turned ON, and the main subject change process ends.

[0100] As described above, if the main subject changes during MF operation, the lock flag for that subject can be set to ON to prevent the main subject from changing when AF is set after MF operation is completed. This makes it possible to continuously focus on the subject that the user intended through MF operation as the main subject.

[0101] Next, the switching subject identification process performed in S606 will be described with reference to FIG. First, in S701, the focus detection result of the main subject whose main subject flag is ON is obtained and held as DEF_Main, and the process proceeds to S702.

[0102] In S702, the focal depth is calculated from information obtainable from the lens control unit 106, and N times the focal depth is set as the switching threshold DEF_Th. This switching threshold DEF_Th is a threshold for determining that the user wants to switch subjects when the amount of blur of the main subject becomes N times the depth or more. Note that this threshold is only an example, and other values ​​may be used as long as they can detect the switching of the main subject.

[0103] In S703, it is determined whether there is a focus detection result for a subject whose main subject candidate flag is ON, excluding a subject whose main subject flag is ON, to thereby determine whether there is a focus detection result for another subject that may be a switching destination. If there is a focus detection result, the process transitions to S704, and if there is not, the process transitions to S706.

[0104] In S704, the focus detection result DEF_Main of the main subject whose main subject flag is ON is compared with the switching threshold DEF_Th. If DEF_Main>DEF_Th, it is determined that switching by the user has occurred, and the process proceeds to S705. If switching has not occurred, the process proceeds to S712, where it is determined that there has been no switching of the main subject, and the process ends.

[0105] In S705, the subject with the smallest amount of blur among the subjects whose main subject candidate flags are ON is selected as the new main subject.

[0106] Meanwhile, in S706, the operation direction of the focus ring is obtained, and information on the movement direction (close / infinity direction) of the focus position of the focus lens 103, which can be obtained from the lens control unit 106, is obtained, and the process proceeds to S707. In S707, the relative distance from the camera system is determined from the size of the subject whose main subject candidate flag is ON. Here, it is determined from the size of the face whether it is closer to the main subject or on the infinity side. Comparing with the size of the main subject, it is basically determined that if the size is large, it is a subject on the close side, and if the size is small, it is a subject on the infinity side. Then, the process proceeds to S708.

[0107] In S708, the focus detection result DEF_Main of the main subject whose main subject flag is ON is compared with the switching threshold DEF_Th. If DEF_Main>DEF_Th, it is determined that switching by the user has occurred, and the process proceeds to S709. If switching has not occurred, the process proceeds to S712, where it is determined that there has been no switching of the main subject, and the process ends.

[0108] In S709, it is determined whether the lens operation direction is the close-up direction. If the operation direction acquired in S706 is the close-up direction, the process proceeds to S710, and if not, the process proceeds to S711.

[0109] In S710, the main subject is identified from the closest subjects. This makes it possible to select the main subject from subjects for which focus detection results could not be obtained due to small subject size or shooting conditions. One of the subjects in the operation direction of the focus lens is selected and identified as the main subject. As a method of identification, as described above, a subject that is likely to be the main subject is selected based on the size and position of the face.

[0110] In S711, the main subject is identified from among the subjects on the infinity side. The process is the same as in S710, so a description thereof will be omitted. After that, the process is switched over and the subject identification process is terminated.

[0111] This makes it possible to automatically switch the main subject during MF operation by acquiring the focus detection result of the subject during MF operation and monitoring the change in the focus detection result. Also, depending on the shooting conditions, there are cases where the focus detection result cannot be acquired, so even in such cases, it is possible to determine the switching of the main subject based on the lens operation direction and face size.

[0112] Here, the switching of the main subject when the result of S704 is YES will be described with reference to FIG.

[0113] Fig. 10(a) shows an example of the display state before MF operation, showing a case where there are two subjects, a first subject 1001 (out of focus) and a second subject 1002 (in focus). In this case, the main subject flag of the second subject 1002 is ON, so in S211 of Fig. 2, a main subject frame 1004 indicating that the second subject 1002 is the main subject is displayed above the second subject 1002. On the other hand, the main subject candidate flag of the first subject 1001 is ON, so a candidate frame 1003 indicating that the first subject is the main subject candidate is displayed.

[0114] FIG. 10(b) shows a case where the focus lens 103 is driven by MF operation from a state where the second subject 1002 is in focus toward the first subject 1001. When the focus detection result DEF_Main of the second subject 1002 becomes greater than the switching threshold DEF_Th, the main subject flag is turned OFF. Therefore, in S211, instead of the main subject frame 1004 indicating that the subject is the main subject, a candidate frame 1006 indicating that the subject is a main subject candidate is displayed. On the other hand, since the main subject flag for the first subject 1001 is turned ON, a main subject frame 1005 indicating that the subject is the main subject is displayed.

[0115] Fig. 10(c) is a diagram explaining the above states in a chronological order. In Fig. 10(c), the vertical axis represents the focus detection result, with 0 being the in-focus state. The horizontal axis represents time. The focus detection result of the first subject 1001 is shown by curve 1008, and the focus detection result of the second subject 1002 is shown by curve 1007.

[0116] 10(a) is a state based on the focus detection result at time t1, and from that state, by driving the focus lens 103, the focus detection result 1008 of the first subject 1001 decreases while the focus detection result 1007 of the second subject 1002 improves. At point 1012 (time t2) where the focus detection result 1007 exceeds the switching threshold DEF_Th, the camera determines that the user wants to focus on the first subject 1001, and switches the main subject (the state shown in FIG. 10(b)).

[0117] In this way, by switching the main subject while driving the focus lens, it is possible to simplify the operation of the focus lens for the user. In other words, by setting AF, even if the user stops operating the focus ring when the main subject frame 1005 indicating the main subject is displayed, AF control is performed for that subject, making it possible to perform continuous focus adjustment from MF operation.

[0118] Next, the main subject lock determination process performed in S208 of FIG. 2 will be described with reference to FIG.

[0119] When a frame is displayed so that the face area of ​​a person other than the main subject is visible, it can be annoying if the frame continues to be displayed on unintended subjects. However, depending on the shooting situation, there are some scenes where it is better to track the subject for a certain period of time. For example, if there are multiple people you want to shoot on the screen, if you continue to track a subject other than the main subject as a main subject candidate for a certain period of time, when the main subject disappears and you are no longer able to track it, the main subject will be switched to another subject, preventing the focus and exposure from being set on the background.

[0120] Therefore, in this process, control is performed to select an appropriate main subject according to the shooting situation while improving the tracking performance of the main subject when the main subject becomes invisible.

[0121] First, in S801, it is determined whether or not there is a subject with the main subject candidate flag determined in S204 being ON within a predetermined range from the position of the subject (e.g., touch position) according to the subject designation instruction received from the camera operation unit 208 in S207. The above-mentioned predetermined range may be the size of the face area or head area stored in the subject area of ​​the subject information, or may be within a distance, for example, twice the size of the face area or head area. If there is a subject that is a main subject candidate within the predetermined range, the process proceeds to S802, and if not, the process proceeds to S809.

[0122] In S802, it is determined whether there are multiple subjects that are candidates for the main subject within a specified range of the touch position. If it is determined that there is only one person, the process proceeds to S803. If it is determined that there are multiple people, the process proceeds to S804.

[0123] In S803, it is determined whether the reliability of the face region or head region stored in the subject region of the subject of the main subject candidate determined to be within a predetermined range of the touch position is equal to or greater than a predetermined threshold value. If it is determined to be equal to or greater than the threshold value, the process proceeds to S805, and if it is determined to be less than the threshold value, the process proceeds to S806.

[0124] On the other hand, in S804, it is determined whether the size of the face region or head region of the subject of all main subject candidates determined to be within a predetermined range of the touch position is equal to or smaller than a threshold value. If it is determined that it is equal to or smaller than the threshold value, the process proceeds to S807, and if it is determined that there is a subject of a size larger than the threshold value, the process proceeds to S808.

[0125] Next, in S805 and S808, a subject that is within the range of the touched position and is closest to the touched position and is a main subject candidate is locked.

[0126] On the other hand, in S806, S807, and S809, the features of the subject are extracted based on the touch position, and an object other than the face or head of a person is tracked (hereinafter, referred to as "object tracking").

[0127] Next, in S810, the locked main subject candidate is determined to be the main subject, and the main subject candidate flag of the locked main subject is set to ON. Then, in S811, the main subject candidate flags of subjects other than the locked main subject are set to OFF, thereby preventing subjects other than the locked main subject from becoming main subject candidates, and the process ends.

[0128] The reason for performing object tracking in S806 when it is determined in S803 that the reliability is less than the threshold is that, for example, when part of the head is out of frame, and the head detection reliability is low, the detection position and size may become unstable, and AF may become unstable. Therefore, when the reliability of the detected face area or head area is low, the area is specified by tracking using color information and brightness information at the time of touch, and the subject at the position specified by the user is temporarily tracked as an object. Then, if a face or a highly reliable head is detected nearby during tracking, the tracking target can be switched to the face or head, thereby reducing AF instability.

[0129] Also, in S804, the reason for performing object tracking when there are multiple small-sized face or head detection results within the specified range of the touch position is that there is a possibility that an unintended person among multiple people will become the tracking target due to the position specified by the user being shifted from the touch position, etc. Therefore, by temporarily tracking a subject at a position specified by the user as an object, and then switching the tracking target to the face or head when a face or head of a specified size or larger is detected within the specified range, it is possible to reduce focusing on an unintended person.

[0130] The above-mentioned processing will be explained with reference to FIG. 11A shows a scene in which a person 1101 comes into the frame facing away from the camera when no MF operation is performed. In this case, the main subject candidate flag for the person 1101 is turned OFF in S512 of FIG. 5, so a frame is not displayed.

[0131] Fig. 11(b) shows a case where the same person 1101 faces forward from the state shown in Fig. 11(a). In this case, the main subject candidate flag is turned ON in S510 of Fig. 5, and since there is only one main subject candidate, the person 1101 is automatically selected as the main subject in S605 of Fig. 6, and a main subject frame 1102 is displayed.

[0132] Fig. 11(c) shows a case where the person 1101 turns away from the forward facing state shown in Fig. 11(b). In this case, the person 1101 turns forward once, so the main subject candidate flag is ON, and the head area is detected, so the main subject candidate flag is ON in S510 of Fig. 5. Also, the person 1101 is automatically selected as the main subject in S605 of Fig. 6, and the main subject frame 1102 continues to be displayed. By continuing to set the person 1101 as a main subject candidate, focus on the person 1101 continues, making it possible to prevent focus and brightness from being adjusted to an unintended area.

[0133] Fig. 11(d) shows a scene in which there are multiple people (person 1103 and person 1105) when MF operation is not performed. In this case, the main subject candidate flag for person 1103, who is facing forward, is turned ON in S510 of Fig. 5, and since there is only one main subject candidate, person 1103 is automatically selected as the main subject in S605 of Fig. 6, and a main subject frame 1104 is displayed. On the other hand, since person 1105 remains facing away, the main subject candidate flag is turned OFF in S512, and no frame is displayed.

[0134] Fig. 11(e) shows a case where person 1105 faces forward from the state shown in Fig. 11(d). In this case, the face area of ​​person 1105 is detected, the main subject candidate flag is turned ON in S510 of Fig. 5, and a candidate frame 1106 is displayed. In this case, person 1105 can also become the main subject, but unless there is a change by MF operation (S206), a change by a subject designation instruction (S208), or a change of the main subject by automatic determination (S210), person 1103, which has already been selected as the main subject, will continue to be the main subject for the time being.

[0135] Figure 11(f) shows a case where person 1105 turns away from the forward facing state of Figure 11(e). In this case, person 1105 turns forward once, the main subject candidate flag is ON, and the head area is detected, so the main subject candidate flag remains ON in S510 of Figure 5. By continuing to set person 1105 as a main subject candidate in this way, it is possible to instantly make person 1105 the main subject when person 1103 goes out of frame.

[0136] 11(g) to (i) show a case where there are multiple people (person 1103 and person 1105) and the user designates person 1105 by touch from the states of FIG. 11(d) to (f), respectively.

[0137] In the case of FIG. 11(g), since the person 1105 has not been detected as a subject, the person 1103 closest to the touch position continues to be selected as the main subject, and the main subject frame 1104 continues to be displayed.

[0138] 11(h) and (i), if the reliability of the face region or head region is high, the person 1105 is locked as the main subject region by the processes of S801, S802, S803, S805, S810, and S811, and the person 1105 is selected as the main subject. Also, a main subject frame 1109 is displayed. Also, the subject candidate flag of the person 1103 is turned OFF by S811, and therefore the main subject frame 1104 that was displayed around the person 1103 is erased.

[0139] As described above, when a person facing away from the camera is touched, if there is a head detection result at the touched position, the head detection result is used to lock the subject, regardless of whether the subject has already been set as a main subject candidate area or not. When a person who is not yet a main subject area or main subject candidate area is touched, if there is a head detection result at the position specified by the touch, the head detection result is used to lock the subject as a main subject area, thereby enabling robust tracking of changes in the shape and brightness of the subject.

[0140] As described above, according to the first embodiment, it is possible to switch the subject to be focused on without driving the lens to the in-focus position. Furthermore, when AF is set, a representative subject is selected as the main subject, thereby realizing highly versatile AF, and MF operation is used as a trigger to change the main subject selection method, thereby allowing the user to select the subject he or she is aiming for as the main subject, thereby enabling AF that suits the user's intentions. Furthermore, the subject selected by MF operation can be continued as the AF target and focus adjustment can be performed even after MF operation. Furthermore, it is possible to easily switch the main subject.

[0141] In addition, the focus detection device provided in the first embodiment can maintain focus on the subject over a wide range by changing the focus detection area used for focus driving depending on the position of the subject within a preset AF frame range, and can prevent difficulty in focusing on a new main subject that appears nearby when focusing on a subject other than the main subject, such as the background (hereinafter referred to as a "miscellaneous subject")

[0142] <Second embodiment> Next, a second embodiment of the present invention will be described.

[0143] When the main subject is determined by the method described in the first embodiment, it is very useful, for example, when the operator has decided on a person to photograph and wants to continue to track that person even when the face is no longer visible, and adjust the focus and exposure. However, when the operator basically wants to photograph a person as the main subject, and also wants to adjust the focus and exposure for other subjects and scenery without switching the shooting mode, if the person continues to be tracked even when the face is no longer visible, the focus and exposure cannot be adjusted for other subjects and scenery as long as the person is in the screen.

[0144] Furthermore, if a face is removed from the list of candidates for the main subject as soon as it becomes invisible, the following problem occurs: If an unintended person other than the person the operator is trying to photograph enters the screen and remains within the screen, the person will not be removed from the list of candidates for the main subject, and the unintended person will be selected as the main subject, causing changes in focus and exposure.

[0145] Therefore, in the above-mentioned first embodiment, even if a face cannot be detected, if a head area can be detected, the subject is continued to be tracked as a main subject candidate, the main subject is determined according to the designation method, and focus adjustment and exposure adjustment are performed. However, in a situation where the main subject turns away during person-centered shooting and the face becomes temporarily invisible, it is desirable to keep the subject as a main subject candidate for as long as possible, but when shooting things other than people or when multiple people are captured, the cases where it is desirable to keep the subject as a main subject candidate and not to keep it will vary depending on the shooting situation.

[0146] Therefore, in the second embodiment, in addition to the first embodiment, the time that each main subject candidate is to remain as a main subject candidate is set, and the time is changed according to the status of the subjects including the main subject. The process in the second embodiment will be described below with reference to Figs. 12 to 16. Note that the second embodiment also uses the digital video camera system shown in Fig. 1, and therefore the description will be omitted.

[0147] Fig. 12 is a flowchart showing the main subject candidate determination process in the second embodiment, which is performed instead of the process shown in Fig. 5. This process is the main subject candidate determination process in the first embodiment described above with reference to Fig. 5, to which steps S1201 to S1203, which are a feature of the second embodiment, are added to set, for each subject, the time from when a face area cannot be detected until the main subject candidate flag is turned OFF. Since the other processes are the same as those shown in Fig. 5, the same step numbers are used and their explanations are omitted.

[0148] In S1201, a reset time RT (>0) from when a face area cannot be detected until the main subject candidate flag is turned OFF is set. The reset time RT is set according to conditions for each subject, and details will be described later with reference to FIG.

[0149] In S1202, if a face area cannot be detected, the reset time RT set in S1201 is decremented. In S1203, main subject candidate reset determination section 222 determines whether or not the reset time RT is greater than 0, and if it is greater than 0, main subject candidate determination section 220 turns the main subject candidate flag ON in S511. On the other hand, if it is equal to or less than 0, it is determined that the time set in S1201 until the main subject candidate flag is turned OFF has expired, and the main subject candidate flag is turned OFF in S512.

[0150] By the process of S1201 to S1203, the reset time RT according to the conditions is always set while the face is detected, and if the situation where the face cannot be detected continues for the reset time RT, it is possible to remove the person from the main subject candidates. This is because in general video shooting, there are many situations and scenes where a person whose face is not visible for a long time is not suitable for selection as a target for focus adjustment control or exposure control, that is, as a main subject, and it is considered better to remove the person from the main subject candidates after a certain amount of time has passed. However, when there is only one person to be shot, or when there are multiple people near the center of the screen, there are cases where it is desired to continue selecting the person as the main subject even in a situation where the face cannot be seen. Therefore, the reset time RT is adjusted by changing it according to the situation.

[0151] Fig. 13 is a flowchart showing details of the process of setting the main subject candidate reset time RT in S1201 in Fig. 12. In S1301, it is determined whether the number n of detected subjects is 1 or not, and if it is 1, the process proceeds to S1302, and if it is more than 1, the process proceeds to S1303. In S1302, the value of MainTimer1 is set as the reset time RT. Here, since the value of MainTimer1 is a set time when only the main subject is present within the shooting screen, it is preferable to set a relatively long time, but in this embodiment, it is set to 10 seconds as an example.

[0152] In S1303, it is determined whether the subject being processed has been selected as the main subject in the main subject determination process among the multiple detected subjects. If the subject has been selected as the main subject, the process proceeds to S1304. If the subject has not been selected, the process proceeds to S1305. In S1304, the value of MainTimer2 is set as the reset time RT. Here, the value of MainTimer2 is the time for which the subject being processed is continued to be set as the main subject in a situation where multiple people are photographed in the shooting screen. Since there is a higher possibility that other main subject candidates may become the main subject compared to the case where there is only one person, the time may be set shorter than the case where there is only one person, or the same time as the case where there is only one person. In this embodiment, as an example, it is set to 10 seconds, the same as MainTimer1.

[0153] In S1305, it is determined whether the number n of detected subjects is less than a threshold value NumTh. If it is less, the process proceeds to S1306. If it is more, the process proceeds to S1307. In S1306 and S1307, the time for tracking the main subject candidate is set, and a time shorter than the time MainTimer1 and MainTimer2 for tracking the main subject is set. This is because the subject of the main subject candidate is less likely to be a subject to be photographed than the main subject, and the main subject is prevented from being unintentionally switched by setting the main subject candidate for too long. In S1306, the value of SubTimer1 is set as the reset time RT. Here, the value of SubTimer1 is the time for continuing to set the main subject candidate when the number of people is relatively small, and in this embodiment, it is set to 3 seconds as an example.

[0154] On the other hand, in S1307, the value of SubTimer2 is set as the reset time RT. The value of SubTimer2 is the time for which the camera continues to set the camera as a main subject candidate when there are many people in the shooting screen, and since there is a high possibility that a subject not intended by the user is included in the shot, a time shorter than SubTimer1 is set. In this embodiment, the setting is made to 1 second as an example.

[0155] In S1308, the time period during which the subject is set as the main subject, which was set in S1302 or S1304, is adjusted to be longer or shorter depending on the conditions. In S1309, similar to S1308, the time period during which the subject is set as the main subject candidate, which was set in S1306 and S1307, is adjusted depending on the conditions. After S1308 or S1309 is completed, the process returns to the process in FIG.

[0156] Next, the details of the process in S1308 will be described with reference to FIG. Fig. 14 is a flow chart showing a process for adjusting the time for tracking a main subject according to conditions. In S1401, an area is set for determining whether or not the main subject is in a predetermined area in the screen in S1403 described later. The concept of the predetermined area will be described later with reference to Fig. 16.

[0157] In S1402, a size threshold SizeTh is set for determining whether the size of the face region and head region of the main subject is relatively large in S1404. This is because, when photographing a person as the main subject, the face is often photographed relatively large, and a small face is likely to be a person not intended by the user, that is, not the main subject.

[0158] In S1403, it is determined whether or not the main subject is present within the predetermined area set in S1401. If it is determined that the main subject is present, the process proceeds to S1404, and if it is determined that the main subject is not present, the process ends.

[0159] In S1404, it is determined whether the size of the face region and head region is larger than the size threshold SizeTh set in S1402. If it is determined that they are larger, the process proceeds to S1405, and if it is determined that they are smaller, the process ends.

[0160] In S1405, the time RT until the tracking of the main subject is reset, which was set in S1302 and S1304 in FIG. 13, is reset to a time longer than MainTimer1 and MainTimer2, and the process ends.

[0161] Here, the predetermined area will be explained with reference to Fig. 16. Fig. 16 is a schematic diagram showing a scene in which a person is the subject within the shooting screen, Fig. 16(a) to (c) show a case where there is only one subject, that is, only the main subject is shot and the face of the main subject is not visible, and Fig. 16(d) to (f) show a case where there are multiple subjects.

[0162] When shooting a video with a person as the main subject, the subject is generally framed near the center of the screen, so if the subject is detected near the center of the screen, it is highly likely that the subject is the person the user wants to shoot. Therefore, it is determined whether the face area and head area of ​​the detected subject are within area 1601 shown by the dashed line in Fig. 16, and if so, tracking is continued for a longer period of time even if the face becomes invisible. In other words, the time RT until the tracking of the main subject is reset, which was set in S1302 and S1304 in Fig. 13, is re-set to a time longer than MainTimer1 and MainTimer2.

[0163] In this embodiment, 15 seconds is set as an example, but the present invention is not limited to this, and may be set to, for example, 30 seconds or infinity (tracking continues until both detection and tracking are no longer possible), or the times of MainTimer1 and MainTimer2 may be left as they are. Note that FIG. 16(a) shows a case where the face area is visible, and FIG. 16(b) shows a case where the main subject turns away and the face area is no longer visible, and both are within the area 1601, so the time RT until the tracking of the subject is reset is 15 seconds. However, if the subject is near the boundary of the area 1601 and frequently enters and exits the area 1601, the set time becomes unstable, becoming 10 seconds or 15 seconds. Therefore, once the main subject enters the area 1601, the area may be expanded to a larger area than 1601, as shown in area 1602 in FIG. 16(c), and the time may be extended until the main subject exits the area 1602. In this way, it is possible to prevent the set time from becoming unstable near the boundary of the area.

[0164] 14, the conditions for extending the MainTimer have been described, but if it is not desirable to track the main subject for a long time, the time until the main subject is reset may be shortened. For example, if panning of the camera is detected, it may be determined that there is an intention to change the main subject, and the time may be shortened to exclude main subjects whose faces are not visible.

[0165] In this embodiment, a rectangular area of ​​a predetermined size is set as area 1601 with the center of the screen as the center, but considering that the area at the top of the screen is likely to contain the main subject, the center position may be shifted upward or may be expanded upward. Furthermore, the area may be changed according to the camera direction. In this embodiment, the time is adjusted according to whether or not the subject is within the area, but this is not limiting and the area may be changed according to the image height.

[0166] Next, the details of the process in S1309 will be described with reference to FIG. Fig. 15 is a flowchart showing a process for adjusting the time for tracking a main subject candidate according to conditions. In S1501, an area is set for determining whether or not the main subject candidate is in a predetermined area within the screen in S1503 described later. In this embodiment, this area is described as being set to be the same area as area 1601 in Figs. 16(a) to (c), but it may be set narrower or wider than that of the main subject.

[0167] In S1502, the difference D between the detection position and the subject area selected as the main subject is calculated. This is to determine whether each main subject candidate detected in S1504 is close to the main subject, and the distance between the center positions of the face area and head area on the screen is calculated, as shown in Fig. 16(d) and (e).

[0168] In S1503, it is determined whether or not a subject that is a candidate for the main subject is present within the predetermined area set in S1501. If it is determined that the subject is present, the process proceeds to S1504, and if it is determined that the subject is not present, the process proceeds to S1505.

[0169] In S1504, it is determined whether the difference D between the detection position and the main subject calculated in S1502 is equal to or smaller than a predetermined value DTh, and if it is equal to or smaller, it is determined that the distance to the main subject is close and the process proceeds to S1507, and if it is greater, it is determined that the distance to the main subject is far away and the process proceeds to S1505. Note that the predetermined value DTh may be a predetermined fixed value, or may be calculated based on the sizes of the face region and head region.

[0170] In S1507, the time RT until the tracking of the subject of the main subject candidate, which was set in S1306 and S1307 in FIG. 13, is reset to a time longer than SubTimer1 and SubTimer2, and the process ends.

[0171] As described above, if a subject is detected near the center of the screen, it is highly likely that the subject is a person that the user wants to photograph. Also, if the subject is a main subject candidate that exists near the main subject, it is highly likely that the subject will be a candidate for the next main subject when the main subject cannot be detected. Therefore, it is considered better to continue tracking a subject that meets these conditions for a longer period of time even if the face becomes invisible. In this embodiment, 5 seconds is set as an example, but the present invention is not limited to this, and the times of SubTimer1 and SubTimer2 may be left as they are.

[0172] Fig. 16(d) shows a case where there is one subject whose face region is not visible within area 1601 near a main subject whose face region is visible, and one subject whose face region is not visible outside area 1601. Fig. 16(e) shows a case where there is one subject whose face region is not visible within area 1601, but is far from the main subject. In the case of Fig. 16(d), the subject whose face region is not visible is within predetermined area 1601 and near the main subject, so the time RT until the tracking of the subject is reset is 5 seconds, but in the case of Fig. 16(e), the subject whose face region is not visible is far from the main subject, so the time RT until the tracking of the subject is reset is not extended and remains at 3 seconds.

[0173] 16(f) shows a scene with many people in the screen, and since there are many subjects, it is easier for the main subject to be unintentionally switched, so it is better not to track subjects that are candidates for the main subject for too long. This corresponds to the case where the number of detections in S1305 of FIG. 13 is greater than the predetermined number NumTh and a short time is set in S1307.

[0174] In S1505, it is determined whether or not the person has been selected as a main subject in the past. If the person has been selected, then there is a high possibility that the person is someone the user wants to photograph, so the process proceeds to S1507, where the time until the subject tracking is reset is adjusted; if the person has not been selected, the process proceeds to S1506.

[0175] In S1506, it is determined whether the time during which face detection was possible is longer than a predetermined time, and if it is longer, it is highly likely that the subject is intentionally framed within the screen by the user, and it is considered better to track the subject as a main subject candidate for a certain period of time. Therefore, the process proceeds to S1507, and the time until the subject tracking is reset is adjusted. On the other hand, if it is shorter, it is highly likely that the subject is a person not intended by the user, such as when the subject appears on the screen for a moment and then turns around, and the process ends without adjusting the time.

[0176] In the flowchart of Fig. 15, the conditions for extending the SubTimer are described, but in cases where it is better not to track a main subject candidate subject for a long time, like the main subject, the time until the subject is reset may be shortened. For example, if panning of the camera is detected, it may be determined that there is an intention to change the subject, and the time may be shortened to exclude subjects whose faces are not visible from the main subject candidates.

[0177] As described above, according to the second embodiment, the time from when face detection becomes impossible until the tracking of the subject is reset is set for each subject. In particular, the main subject is tracked for a long time, and the tracking time for other subjects is set to be shorter than that of the main subject. This makes it possible to improve the tracking performance of the main subject when the face becomes invisible, while realizing the selection of an appropriate main subject according to the shooting situation.

[0178] In the above-described embodiment, the detection results of a human face region and head region are used. However, a combination of a human face region and body region, or an animal's face region and head region or body region may also be used.

[0179] <Other embodiments> The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0180] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0181] 10: lens unit, 20: camera body, 103: focus lens, 105: focus lens driving section, 106: lens control section, 107: lens operation section, 201: image sensor, 204: AF signal processing section, 205: display section, 207: camera control section, 208: camera operation section, 210: face area detection section, 211: head area detection section, 220: main subject candidate determination section, S221: main subject lock section, 222: main subject candidate reset determination section

Claims

1. a detection means for detecting main subject candidates based on a first predetermined condition from image signals repeatedly outputted by the image pickup means through a photographing optical system, the main subject candidates being candidates for the main subject; a selection means for selecting the main subject from the main subject candidates; an acquisition unit that acquires operation information of an operation unit for moving a position of a focus lens included in the photographing optical system; a focus adjustment unit that controls the focus lens so as to focus on the main subject selected by the selection unit when an autofocus mode is set and the operation unit is not operated, a focus adjustment device, characterized in that, when the operation means is operated, the selection means selects the main subject based on a focus position of the focus lens, and fixes the selected main subject.

2. 2. The focus adjustment device according to claim 1, wherein the selection means selects, when the operation means is operated, a main subject candidate that is closest to a focus position of the focus lens as the main subject.

3. 3. The focus adjustment device according to claim 1, wherein the selection means reselects the main subject when the operation means is operated and the focus state of the selected main subject is lower than a state set based on the focal depth of the focus lens.

4. the acquisition means acquires a moving direction of a focus position of the focus lens in response to an operation of the operation means, The selection means reselects the main subject from main subject candidates existing in the moving direction of the focus position.

4. The focus adjustment device according to claim 3.

5. 5. The focus adjustment device according to claim 1, wherein the selection means determines the main subject based on a second predetermined condition when the operation means is not operated and the main subject is not fixed.

6. The detection means includes: a first detection means for detecting a first region having a first characteristic from the image signal; a second detection means for detecting a second region related to the first region and having a second characteristic from the image signal; 6. The focus adjustment device according to claim 1, wherein the main subject candidate is detected based on information on the first region and the second region.

7. 7. The focus adjustment device according to claim 6, wherein the first detection means detects the first area using a third condition that differs depending on whether the operation means is operated or not.

8. 8. The focus adjustment device according to claim 7, wherein the third condition includes size, and when the operation means is operated, a lower limit of the size of the area detected as the first area is made smaller than when the operation means is not operated.

9. The focus adjustment device according to claim 7 or 8, characterized in that the third condition includes a range of an image in which the first area is detected, and when the operation means is operated, the first area is detected over a wider range than when the operation means is not operated.

10. 10. The focus adjustment device according to claim 7, wherein the third condition includes reliability, and when the operation means is operated, a lower limit of the reliability of the area detected as the first area is set lower than when the operation means is not operated.

11. 11. The focus adjustment device according to claim 6, wherein the first condition is different depending on whether the operation means is operated or not.

12. 12. The focus adjustment device according to claim 11, wherein, when the operation means is operated, the detection means detects a subject included in at least one of the first area and the second area as a main subject candidate, and when the operation means is not operated, the detection means detects a subject included in the first area and a subject included in the second area that has been previously detected as a main subject candidate, as main subject candidates.

13. The imaging device further includes a display unit for displaying an image based on the image signal output from the imaging unit, when an autofocus mode is set and the operation means is not operated, a first display is performed on the image displayed on the display means, the first display showing the main subject selected by the selection means; When the operation means is operated, the first display and a second display showing the main subject candidate detected by the detection means are displayed on the image displayed on the display means.

13. The focus adjustment device according to claim 1, wherein the focus adjustment device is a focusing device.

14. 14. The focus adjustment device according to claim 13, further comprising a setting unit that sets, for each of the main subject candidates, a time period during which the second display is performed after the main subject candidate is detected by the detection unit.

15. A focus adjustment device according to any one of claims 1 to 14, The imaging means; An imaging device comprising:

16. 16. The imaging apparatus according to claim 15, further comprising the imaging optical system.

17. 16. The image pickup apparatus according to claim 15, wherein the photographing optical system is detachable.

18. a detection step in which the detection means performs photoelectric conversion on light incident via the photographing optical system by the imaging means, and detects main subject candidates that are candidates for the main subject from image signals that are repeatedly output based on a first predetermined condition; a selection step of selecting the main subject from the main subject candidates by a selection means; an acquisition step in which an acquisition unit acquires operation information of an operation unit for moving a position of a focus lens included in the photographing optical system; a focus adjustment step of controlling the focus lens so as to focus on the main subject selected in the selection step when an autofocus mode is set and the operation means is not operated, a focus adjustment method, characterized in that in the selection step, when the operating means is operated, the main subject is selected based on a focus position of the focus lens, and the selected main subject is fixed.

19. A program for causing a computer to function as each of the means of the focus adjustment device according to any one of claims 1 to 14.

20. A computer-readable storage medium storing the program according to claim 19.

Images