Focusing device and method, and imaging device

Abstract

To automatically determine a main subject to be focused and flexibly change the main subject according to a user's intention.SOLUTION: A focusing device comprises: detection means that photoelectrically converts light incident through a photographic optical system with imaging means and detects main subject candidates to be the candidates for a main subject from image signals that are repeatedly output; selection means that selects a main subject from the main subject candidates; acquisition means that acquires operation information on operating means for moving the position of a focus lens included in the photographic optical system; and focusing means that, when an autofocus mode is set and the operating means is not operated, controls the focus lens so that the main subject selected by the selection means is focused. When the operating means is operated, the selection means selects the main subject based on the focusing position of the focus lens and fixes the selected main subject.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a focus adjustment device and method, and an imaging device.

Background Art

[0002] In conventional autofocus (AF) control, the user sometimes has to specify an AF frame for the subject to be focused. However, for example, in a shoulder-held type camera used for news gathering or the like, it may be difficult to install an operating device for operating the AF frame at a location easily accessible to the operator.

[0003] Therefore, Patent Document 1 discloses a technique for automatically setting an AF frame for a subject that has been focused by a manual focus (MF) operation performed by an operator.

[0004] Furthermore, Patent Document 2 discloses a technique for setting an AF frame on a face if a face is detected, or on a subject other than the face if no face is detected, and performing automatic tracking and AF for a subject whose focus has been adjusted as a result of a manual focus operation performed by an operator, provided that the focused position is within a face detection area.

[0005] On the other hand, image processing for automatically detecting a specific subject from an image is known. For example, in a digital camera or a digital video camera, a face of a person is particularly detected from a captured image, and the detection result is used as a control target to optimize focus and exposure. However, in general face detection technology, stable detection cannot be achieved when facial features cannot be sufficiently obtained, such as when a person is facing backward or wearing ornaments on the face.

[0006] Therefore, Patent Document 3 proposes a method of detecting an upper body or a head portion of a person from an image and using the area of the upper body or head of the person, or the area of a face estimated from these detection results, as a control target. With this method, the face area of a person can be stably specified even when the person is facing backward or wearing ornaments on the face.

Prior Art Documents

[0007] [Patent Document 1] Japanese Patent Publication No. 2010-97167 [Patent Document 2] Patent No. 5474528 [Patent Document 3] Japanese Patent Publication No. 2017-126915 [Overview of the project] [Problems that the invention aims to solve]

[0008] However, focusing using manual focus (MF) requires skilled technique, and smoothly moving the focus lens to the in-focus position is difficult. Furthermore, with the recent increase in pixel count, resolutions have risen to 4K and 8K, making it extremely difficult to confirm the in-focus position on the camera's monitor. Therefore, as described in Patent Document 2, having the operator manually focus until the subject is nearly in focus on the face may degrade the quality of the video due to the difficulty of focusing and the resulting blurring caused by the high resolution.

[0009] On the other hand, in the method of subject tracking using face detection technology described in Patent Document 3, the user cannot change the focus to an arbitrary subject different from the tracked subject on the screen if they wish to do so.

[0010] This invention was made in view of the above-mentioned problems, and aims to automatically determine the main subject to focus on, while also allowing the main subject to be flexibly changed according to the user's intentions. [Means for solving the problem]

[0011] To achieve the above objective, the focusing device of the present invention converts light incident through the imaging optical system by the imaging means into photoelectric form, and repeatedly outputs image signals to identify candidate main subjects. facial area The system includes a detection means for detecting a main subject, a selection means for selecting a main subject from the main subject candidates, an acquisition means for acquiring operation information of an operation means for moving the position of a focus lens included in the imaging optical system, and a focus adjustment means for controlling the focus lens to focus on the main subject selected by the selection means when the autofocus mode is set and the operation means is not operated, wherein the selection means is controlled when the operation means is operated and when the operation means is not operated. Rather, reduce the size of the face that can be detected. It is characterized by the following: [Effects of the Invention]

[0012] According to the present invention, the main subject to be focused on is automatically determined, and the main subject can be flexibly changed according to the user's intention. [Brief explanation of the drawing]

[0013] [Figure 1] A block diagram showing the configuration of a digital video camera system in the first embodiment of the present invention. [Figure 2] Flowchart of the focus adjustment process in the first embodiment. [Figure 3] A flowchart of the parameter setting process in the first embodiment. [Figure 4] A diagram illustrating the focus detection frame setting process in the first embodiment. [Figure 5] Flowchart of the main subject candidate determination process in the first embodiment. [Figure 6] Flowchart of the main subject change process in the first embodiment. [Figure 7] Flowchart of the process for identifying the switching subject in the first embodiment. [Figure 8]Flowchart of main subject lock determination processing in the first embodiment. [Figure 9] Conceptual diagram showing an example of frame display of subjects detected when MF operation is in progress and not in progress in the first embodiment. [Figure 10] Conceptual diagram showing an example of frame display of the main subject and main subject candidates when MF operation is in progress and not in progress in the first embodiment. [Figure 11] Conceptual diagram showing an example of frame display of main subject candidates and the main subject in the first embodiment. [Figure 12] Flowchart of main subject candidate determination processing in the second embodiment. [Figure 13] Flowchart showing the main subject candidate reset time setting processing in the second embodiment. [Figure 14] Flowchart showing the main subject reset time adjustment processing in the second embodiment. [Figure 15] Flowchart showing the main subject candidate reset time adjustment processing in the second embodiment. [Figure 16] Conceptual diagram showing the area for adjusting the main subject reset time and the main subject candidate reset time and the distance between subjects in the second embodiment.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and duplicate explanations are omitted.

[0015] <The First Embodiment> The first embodiment of the present invention will be described in detail below. In the following embodiment, a digital video camera system will be used as an example of an imaging device, but the invention is not limited thereto.

[0016] Figure 1 is a block diagram showing the configuration of a lens-interchangeable digital video camera system in an embodiment of the present invention. The camera system in this embodiment consists of a lens unit 10 and a camera body 20, and a lens control unit 106 that comprehensively controls the operation of the entire lens unit 10 and a camera control unit 207 that oversees the operation of the entire camera system communicate data. In this embodiment, the camera control unit 207 consists of a processor such as a CPU or MPU and storage means such as memory. The camera control unit 207 also includes an arithmetic circuit, and the arithmetic circuit may execute some of the arithmetic functions performed by the processor. In this embodiment, a lens-interchangeable camera system in which the lens unit 10 can be attached to and detached from the camera body 20 is described as an example, but the present invention can also be applied to cameras with integrated lenses.

[0017] First, the configuration of the lens unit 10 will be described. The lens unit 10 has an imaging optical system comprising a fixed lens 101, a zoom lens 108, an aperture 102, and a focus lens 103. The aperture 102 is driven by an aperture drive unit 104 and controls the amount of light incident on the image sensor 201, which will be described later. The focus lens 103 is driven by a focus lens drive unit 105 and adjusts the focus. The zoom lens 108 is driven by a zoom lens drive unit 109 and adjusts the zoom magnification. Note that a lens unit 10 without a zoom function may also be used.

[0018] The aperture drive unit 104, the focus lens drive unit 105, and the zoom lens drive unit 109 are controlled by the lens control unit 106, which determines the aperture size of the aperture 102 and the positions of the focus lens 103 and the zoom lens 108. When the user operates the aperture, focus, zoom, etc. via the lens operation unit 107, the lens control unit 106 performs control according to the user's operation. In addition, the lens control unit 106 controls the aperture drive unit 104, the focus lens drive unit 105, and the zoom lens drive unit 109 according to the control commands and control information received from the camera control unit 207, which will be described later. Furthermore, the lens control unit 106 transmits lens information (for example, MF operation information and information about the shooting optical system) to the camera control unit 207.

[0019] Next, the configuration of the camera body 20 will be described. The camera body 20 is configured to acquire image signals and AF signals from the light beam that has passed through the imaging optical system of the lens unit 10. The image sensor 201 is configured using a CCD or CMOS sensor, etc. The light beam that has passed through the imaging optical system is incident on the light-receiving surface of the image sensor 201, and the formed subject image is converted into an electric charge (photoelectric conversion) by a photodiode according to the amount of incident light. The charge accumulated in each photodiode is then sequentially read out from the image sensor 201 as an image signal and an AF signal based on a voltage signal corresponding to the charge, based on a drive pulse provided from the timing generator 209 according to a command from the camera control unit 207.

[0020] The image signal and AF signal read from the image sensor 201 are input to the CDS / AGC circuit 202, where correlated double sampling to remove reset noise, gain adjustment, and signal digitization are performed. The CDS / AGC circuit 202 then outputs the obtained image data to the camera signal processing unit 203, the face area detection unit 210, and the head area detection unit 211, and the AF data to the AF signal processing unit 204.

[0021] The camera signal processing unit 203 transmits image data output from the CDS / AGC circuit 202 to the display unit 205. The display unit 205 is a display device (display component) made using an LCD or organic EL, etc., and displays an image based on the image data. In addition, when in image recording mode, the image data is recorded in the recording unit 206.

[0022] The AF signal processing unit 204 calculates information regarding the focus state and reliability based on the AF data output from the CDS / AGC circuit 202 using known methods. Reliability information can include contrast information, saturation information, scratch information, and, in the case of an image sensor 201 compatible with the so-called image plane phase-detection method, bi-image coincidence and bi-image steepness can be used. The calculated focus state and reliability information are then output to the camera control unit 207.

[0023] The face region detection unit 210 (first detection means) applies known face detection processing to the image data output from the CDS / AGC circuit 202 to detect the face region (first region) of a person in the captured image. That is, the face region detection unit 210 detects a predetermined subject (in this case, a face) from the image data. The obtained face detection result is transmitted to the camera control unit 207. As an example of the face detection processing performed by the face region detection unit 210, one method is to extract the skin tone region from the gradation color of each pixel represented by the image data and detect the face based on the degree of matching with a pre-prepared face contour plate. In addition, there are methods of performing face detection by extracting facial feature points such as eyes, nose, and mouth using well-known pattern recognition technology, but the present invention is not limited by the method of face detection processing, and any method may be used. Furthermore, it is not limited to faces, but may also be a subject other than a person's face.

[0024] The head region detection unit 211 (second detection means) detects the target subject region from the image as a head region (second region), which is a region related to the face region (first region) detected by the face region detection unit 210. In other words, the head region detection unit 211 detects a predetermined subject (in this case, the head) from the image data that is related to a predetermined subject detected by the face region detection unit 210. In the head region detection in this embodiment, the head region is detected from the image data based on the learning data, and the detection result is transmitted to the camera control unit 207.

[0025] The camera control unit 207 transmits information to the AF signal processing unit 204 to set a focus detection area at a position including the face area and head area within the shooting screen, based on the face detection result of the face area detection unit 210 and the head detection result of the head area detection unit 211. In this first embodiment, if a face is detected, the focus detection area is set based on the position and size of the face area. On the other hand, if a face is not detected and only the head area is detected, the focus detection area is set based on the position and size of the head area. In the case where only the head area is detected, the position and size of the subject's face area may be estimated from the head detection result, and the focus detection area may be set based on the estimated position and size.

[0026] Furthermore, the camera control unit 207 exchanges information with each component within the camera body 20 to perform control, and executes camera functions operated by the user in response to input from the camera operation unit 208, as well as processing within the camera body 20. Camera functions operated by the user include, for example, turning the power on / off, changing settings, starting recording, switching between AF (autofocus) / MF (manual focus) control, and checking recorded images. Also, as described above, the camera control unit 207 exchanges information with the lens control unit 106 within the lens unit 10, sending control commands and control information for the shooting optical system, and acquiring information from within the lens unit 10. It also performs processing to calculate the amount of drive for the focus lens 103 to achieve focus, using the information from within the lens unit 10 received from the lens control unit 106 and the focus state and reliability information from the AF signal processing unit 204.

[0027] The camera control unit 207 also includes a main subject candidate determination unit 220, a main subject lock unit 221, and a main subject candidate reset determination unit 222. The main subject candidate determination unit 220 determines whether a subject can be a candidate for the main subject based on the face detection result obtained by the face region detection unit 210 and the head detection result obtained by the head region detection unit 211. Then, from among the subjects determined to be candidates for the main subject, it selects the main subject through a main subject determination process described later, and performs a process to superimpose a frame display indicating that it is the main subject or a candidate for the main subject onto the captured image, which is then displayed on the display unit 205, etc.

[0028] Next, the focus adjustment process in the first embodiment will be explained with reference to Figure 2. The focus adjustment process in the first embodiment is characterized by control based on the detection results of the face region and head region by the face region detection unit 210 and the head region detection unit 211. This focus adjustment process is executed according to a computer program stored in the camera control unit 207. Furthermore, the series of focus adjustment processes described below are assumed to be performed in synchronization with the imaging cycle, but are not limited to this.

[0029] First, in S201, the face region detection unit 210 performs a parameter setting process for subject detection, which involves setting the parameters to be used for subject detection. Details of the parameter setting process will be described later with reference to Figure 3. Then, depending on whether AF or MF is selected, the face region detection unit 210 performs face region detection and the head region detection unit 211 performs head region detection using the set parameters. After that, the process proceeds to S202.

[0030] Next, in S202, the AF signal processing unit 204 performs a focus detection frame setting process to set focus detection frames for the AF signal output from the CDS / AGC circuit 202, and also detects the focus state for each set focus detection frame. The focus detection frame setting process performed here will be described later with reference to Figure 4. After that, the process proceeds to S203.

[0031] In S203, the face detection result from the face region detection unit 210, the head detection result from the head region detection unit 211, and the tracking search result are acquired, subject information is generated, and the process proceeds to S204. The subject information generated here includes the detection ID, face detection result, head detection result, tracking search result, position and size of the subject region, main subject candidate flag, main subject flag, main subject lock flag, and focus detection result information.

[0032] The detection ID is assigned by the camera control unit 207 when it detects a subject based on the acquired face detection results and head detection results. At that time, the camera control unit 207 determines the face region and the head region, and assigns the same detection ID to the face region and head detection results that are determined to belong to the same subject. Furthermore, it performs a tracking search process to determine whether the same subject detected in the current frame was also detected in the previous frame. This tracking search process can be performed, for example, using the image of the previous frame and employing known methods such as template matching or histogram matching. If the tracking search process determines that the same subject exists in the image of the previous frame, the same detection ID that was assigned to that subject is assigned to the subject detected in the current frame.

[0033] The face detection result is information regarding the position and size of the face region detected by the face region detection unit 210, and the head detection result is information regarding the position and size of the head region detected by the head region detection unit 211. Alternatively, instead of the face detection result and head detection result, information indicating whether or not the relevant information exists, and a link to access the relevant information if it exists, may be stored.

[0034] The tracking search results indicate whether or not the same subject was present in the image of the previous frame as a result of the tracking search process described above.

[0035] The subject area indicates information about the subject's position and size, determined based on face detection and head detection results.

[0036] The primary subject candidate flag indicates information added by the primary subject candidate determination process in S204, which will be described later. The primary subject flag is information added by the primary subject change process performed in S206 and the primary subject lock determination process performed in S208, which will be described later, and is set for only one subject. The primary subject lock flag is also information added by the primary subject change process performed in S206 and the primary subject lock determination process performed in S208, which will be described later, and is a flag used to fix the primary subject flag. Note that if the same subject is present in the image of the previous frame, the information for the same detection ID is carried over in S203.

[0037] The focus detection result shows the focus state detected in S202.

[0038] In S204, the camera control unit 207 performs a main subject candidate determination process, which involves determining the main subject candidates from among the subjects detected in S203 and setting a main subject candidate flag. This process will be described later using Figure 5. After that, the process proceeds to S205.

[0039] The S205 determines whether or not manual focus (MF) operation is in progress. MF operation information can be obtained by the lens control unit 107 or the camera control unit 208, and there are basically three patterns as shown below.

[0040] (1) A pattern in which the lens unit 10 is operated directly by setting it to MF. The lens is set to a state where it can be operated manually by switching between an AF and MF switch (not shown) on the lens unit 10 using the lens operation unit 107, or by switching between an AF mode (autofocus mode) and an MF mode (manual focus mode) (not shown) from the camera menu using the camera operation unit 208. In that state, the focus lens drive unit 105 is driven by directly operating the lens operation unit 107.

[0041] (2) Setting to MF and operating the lens unit 10 from the camera body 20. The camera control unit 208 switches to an MF mode (not shown) via the camera body 20's menu, setting the lens unit 10 to a state where MF operation is possible. In this state, the user sets the manual operation drive information and transmits this information to the lens control unit 106 via the camera control unit 207, thereby driving the focus lens drive unit 105.

[0042] (3) Pattern in which the lens unit 10 is operated directly while the AF setting remains unchanged. With the lens unit 10 and camera body 20 still in AF mode, the focus lens drive unit 105 is driven by directly operating the lens control unit 107. In this state, the camera returns to AF mode when the operation is finished.

[0043] If MF operation is in progress, proceed to S206; otherwise, proceed to S207.

[0044] In S206, a main subject change process is performed in which the user identifies the subject they want to photograph from among the subjects with the main subject candidate flag set, and sets the main subject flag for that subject. This process will be described later using Figure 6. After that, the process proceeds to S211.

[0045] Meanwhile, in S207, the system determines whether or not the camera control unit 208 has issued a subject selection instruction from the user. If it determines that a subject selection instruction has been issued, the system proceeds to S208; otherwise, it proceeds to S209. Possible methods for selecting a subject include, for example, selecting it via a touch panel or by detecting direction and pressure, similar to a directional pad.

[0046] In S208, the main subject locking unit 221 performs the main subject locking process. The main subject locking process determines the main subject based on user instructions, regardless of the results of the main subject candidates determined in S204. This process will be described later using Figure 8. After that, the process proceeds to S211.

[0047] In S209, the presence or absence of the main subject lock flag, which is set in S208 by a process described later, is checked to determine whether the main subject is locked or not. If the main subject is not locked, the process proceeds to S210; if it is locked, the process proceeds to S211.

[0048] In S210, the primary subject is selected from the primary subject candidates determined in the primary subject candidate determination process in S204, based on the position and size of the person. The parameters for determining the primary subject may also be determined based on factors other than position and size, such as the reliability and orientation of the face detection and head detection results, whether both the face and head are detected, or whether only one is detected. Alternatively, the primary subject may be determined from the primary subject candidates using a known algorithm.

[0049] In S211, the display unit 205 displays a frame around subjects for which the main subject candidate flag has been set, and performs frame display processing to notify the user. Specifically, for the main subjects determined in S206, S208, and S210, a main subject frame is displayed on the face or head portion to indicate that it is the main subject. In addition, among subjects for which the main subject candidate flag is ON, a candidate frame is displayed on the face or head portion for which the main subject candidate flag is not ON to indicate that it is a candidate for the main subject. Furthermore, if the main subject lock flag is ON, only the main subject frame is displayed, and no candidate frames are displayed on the face or head portion of subjects other than the main subject for which the main subject candidate flag is ON.

[0050] In S212, it is determined whether MF is set or if MF operation is in progress. If MF is set or MF operation is in progress, the process proceeds to S213, where lens control processing is performed by sending a drive command for the focus lens 103 to the lens control unit 106 based on the focus detection result of the main subject for which the main subject flag is ON. If there is no subject with the main subject flag ON, for example, a focus detection frame is set in the center of the screen, and lens control processing is performed based on the focus detection result, etc. Note that the focus detection method and lens control processing can be done using conventional methods, so a detailed explanation is omitted here.

[0051] If manual focus (MF) is selected or in manual focus operation, the process will terminate immediately because the user is operating the focus lens 103.

[0052] Next, the parameter setting process performed in S201 will be explained using Figure 3. The parameter setting process is the process of setting parameters that are referenced in the face region detection unit 210. In this embodiment, the threshold settings are different depending on whether MF operation is in progress or not. During MF operation, the system makes it easier to detect small faces or faces with low reliability in order to detect the subject that the user is aiming at.

[0053] First, in S301, it is determined whether or not an MF operation is in progress. This determination is made by determining whether or not an operation is being performed according to one of the three patterns described above. If an MF operation is in progress, the process proceeds to S302; otherwise, it proceeds to S305.

[0054] In S302, you set the threshold FaceTh1 for the minimum face size that can be detected. For example, if you set it using the pixel size on the image sensor 201, you would set it to "15" if you want to detect faces that are 15 pixels or larger.

[0055] Next, in S303, you set the face detection confidence threshold, RelianceTh1. For example, if the confidence level of the face detection result is output on a 10-point scale, and you want to detect faces with a confidence level of 3 or higher, you would set it to "3".

[0056] Then, in S304, the threshold AreaTh1 for face detection is set, and the process ends. For example, if you want to detect faces in 90% of the screen area, set it to "90".

[0057] On the other hand, if it is determined in S301 that manual focus (MF) operation is not in progress, the system proceeds to S305 to set the threshold FaceTh2, which is the minimum face size threshold that can be detected. For example, if the threshold is set using the pixel size on the image sensor 201, the threshold should be set to "25" if you want to detect faces that are 25 pixels or larger.

[0058] Next, in S306, you set the face detection confidence threshold, RelianceTh2. For example, if the confidence level of the face detection result is output on a 10-point scale, and you want to detect faces with a confidence level of 6 or higher, you set it to "6".

[0059] Then, in S307, the threshold AreaTh2 for face detection is set, and the process ends. For example, if you want to detect faces in an area of ​​80% of the screen, set it to "80".

[0060] Note that the above threshold relationship is FaceTh1 <FaceTh2、RelianceTh1<RelianceTh2、AreaTh1> AreaTh2 is configured to make face detection easier during manual focus (MF) operation. In this way, during MF operation, the minimum face size threshold is reduced, the minimum confidence threshold for face detection is lowered, and the face detection range is widened compared to when MF operation is not in progress.

[0061] The subject detection results obtained when the parameters are set as described above will be explained using Figure 9. Figure 9(a) shows the case when MF operation is not in progress and subjects 901 and 902 are on the screen. In this case, subject 901 is detected by subject detection, but subject 902 is not detected because its face is smaller than the minimum face size threshold FaceTh2 set in S305. Furthermore, if subject 902 is located on the periphery of the screen and is outside the face detection range threshold AreaTh2 set in S307, it is unlikely to be the main subject in the first place, so subject detection itself is restricted from being performed.

[0062] Based on these subject detection results, if subject 901 is determined to be the primary subject through processing in S208 or S210, a primary subject frame 903 indicating that subject 901 is the primary subject is displayed in S211. On the other hand, no frame is displayed for subject 902. Furthermore, only subject 901 is subject to focus detection processing.

[0063] Figure 9(b) shows the same subject when manual focus (MF) operation is in progress. When manual focus is in progress, the user may intentionally focus on subject 902. Therefore, by lowering the face detection threshold and widening the face detection range, the system can be made more responsive to face detection, allowing both subjects 901 and 902 to be detected.

[0064] Based on these subject detection results, if the processing in S206 determines that subject 901 is the primary subject, in S211, a primary subject frame 903 indicating the primary subject is displayed for subject 901. In addition, a candidate frame 904 is displayed for subject 902, and both subjects 901 and 902 become targets for focus detection processing. In this way, during MF operation, subjects that were not candidates for the primary subject when not in MF operation are also subject to focus detection processing, making it possible to switch the primary subject in a way that reflects the user's intentions.

[0065] Next, the focus detection frame setting process performed in S202 will be explained using Figure 4. The focus detection frame setting process is the process of setting focus detection frames in a mesh pattern on the image sensor, and focus detection is performed based on the AF signal obtained from each of these frames.

[0066] In this embodiment, the method for setting the focus detection frame differs depending on whether or not manual focus (MF) operation is in progress. In S202, if manual focus operation is detected according to any of the patterns (1) to (3) described above, it is determined that manual focus operation is in progress, and the focus detection frame corresponding to the case of manual focus operation and the case of non-MF operation are set accordingly.

[0067] In Figure 4, subjects 401, 402, and 403 are examples of subjects detected in S202 by the face region detection unit 210 and the head region detection unit 211 using the parameters set in S201. Figure 4(a) shows how to set the focus detection frame when MF operation is not in progress, and for subjects 401, 402, and 403, focus detection frames 405, 406, and 407 are set according to the size of the subjects. This setting method has the following advantages. • Enables efficient focus detection processing by eliminating unnecessary parts other than the subject. • Because the frame size is variable according to the size of the subject, stable focus detection results can be obtained by reducing near-far competition within the frame without having to set a large focus detection frame for small faces.

[0068] Note that in the example shown in Figure 4(a), a focus detection frame is set for each subject for illustrative purposes. However, when not in manual focus operation, focus adjustment is performed only on the subject for which the main subject flag is ON, so one focus detection frame is sufficient.

[0069] Figure 4(b) shows how to set the focus detection frame during manual focus (MF) operation. During MF operation, regardless of the subject detection result, a mesh-like frame 408 is set centered on the screen. This setting method has the following advantages. • Focus detection results for multiple subjects can be obtained at once. • Even if the number of subjects detected by subject detection increases, the number of focus detection frames will not increase.

[0070] In this way, by changing the method of setting the focus detection frame depending on whether manual focus (MF) operation is in progress or not, stable focus adjustment becomes possible when not in MF operation. On the other hand, when in MF operation, the camera can efficiently acquire focus detection results for multiple subjects, allowing for quick response to changes in the main subject.

[0071] Next, the main subject candidate determination process performed in S204 will be explained using Figure 5. This process involves the camera control unit 207 setting a main subject candidate flag for subjects among those detected in S203 that are candidates for the main subject. The main subject candidate flag is a flag that indicates a subject that has the potential to become the main subject based on the detected results.

[0072] First, in S501, the system determines whether or not a face detection result exists for one of the detection IDs obtained in S203. If a face detection result exists, the system proceeds to S502; otherwise, it proceeds to S503. In S502, the system stores the position and size of the face region as information about the subject region. After that, the system proceeds to S510.

[0073] In S503, it is determined whether or not an MF operation is in progress. If an MF operation is in progress, the process proceeds to S504; otherwise, it proceeds to S506.

[0074] In S504, it is determined whether or not a head detection result is obtained. If a head detection result is obtained, the process proceeds to S505. On the other hand, if no head detection result is obtained, the process proceeds to S512, where the main subject candidate flag is turned OFF, and then to S517.

[0075] In S505, the position and size of the subject's head region are stored as subject region information. Thus, during MF operation, even if a face is not detected, if the head region is detected, it is stored as subject region information, and the process proceeds to S510.

[0076] In S506, it is determined whether the main subject candidate flag is ON. This is ON if the main subject candidate flag is set for a subject with the same detection ID in the previous frame, and OFF otherwise. If the main subject candidate flag is ON, the process transitions to S505, where the position and size of the subject's head region are stored as subject region information. Thus, if the face region of the same subject has been detected in the past, or if the head region was detected during MF operation and the main subject candidate flag is ON, the head region information is stored as subject region information even if the face region is not detected in the current frame.

[0077] On the other hand, if the main subject candidate flag is OFF, the process proceeds to S512, where the main subject candidate flag is turned OFF, and then to S517.

[0078] In S510, the main subject candidate flag is turned ON, and the process proceeds to S513.

[0079] In S513, it is determined whether or not an MF operation is in progress. If an MF operation is in progress, the process proceeds to S514; otherwise, it proceeds to S516.

[0080] In S514, the position of the subject being processed, for which the main subject candidate flag is set to ON, is identified on the focus detection frame. This process identifies which area of ​​the focus detection frame 408 in Figure 4(b), which is arranged in a mesh pattern, the center position of the subject being processed is located. This is a necessary process to identify which main subject candidate to focus on during MF operation. The process then proceeds to S515.

[0081] In S515, focus detection is performed based on the AF signal within the focus detection frame 408 at the position identified in S514, the focus detection result is stored, and the process transitions to S517.

[0082] Meanwhile, in S516, focus detection is performed based on the AF signals within the position and size stored in the subject area in either S502, S505, or S509, and the focus detection result is stored. Here, since MF operation is not in progress, the focus detection results for the focus detection frames set for each subject are obtained, as shown in focus detection frames 405, 406, and 407 in Figure 4(a). After that, the process proceeds to S517.

[0083] In S517, it is determined whether there are any unprocessed subjects (detection IDs). If the process of setting the main subject candidate flag and storing the focus detection results has been completed for all detection results, the process ends. If there are any unprocessed subjects (detection IDs), the process returns to S501 and the above process is repeated.

[0084] Thus, in this embodiment, the judgment on the head detection result is changed depending on whether or not manual focus (MF) operation is in progress. Specifically, during MF operation, in order to detect the subject the user is aiming at, even small faces where the eyes and nose cannot be detected, or faces that are not facing forward, are flagged as candidates for the main subject. On the other hand, when not in MF operation, small faces where the eyes and nose cannot be detected, or faces that are not facing forward, are generally less likely to be the main subject, so they are not flagged as candidates for the main subject. However, if face detection of a subject that was previously detectable becomes impossible, the main subject candidate flag is continued to be set if the head detection result is valid. This makes it easier to capture the subject the user is aiming at during MF operation, and when not in MF operation, it becomes possible to control focus adjustment on faces that are more likely to be the main subject.

[0085] An example of setting the main subject candidate flag as described above will be explained using Figures 9(c) and (d).

[0086] Figure 9(c) shows the case where the screen includes subjects 905 and 906, but not during manual focus operation. Subject 905 has a face region detected, so the primary subject candidate flag is ON. On the other hand, subject 906 does not have a face region detected, but its head region is detected. In this case, if the primary subject candidate flag was OFF in the previous frame, the primary subject candidate flag will remain OFF. This is because subject 906, which only has a head region detected and has not had a face detected in the past, is unlikely to become the primary subject.

[0087] Based on the main subject candidate flag set in this manner, if the processing in S208 or S210 determines that subject 905 is the main subject, in S211, a main subject frame 907 is displayed on subject 905 to indicate that it is the main subject. On the other hand, no frame is displayed on subject 906. In addition, only subject 905 is subject to focus detection processing.

[0088] Figure 9(d) shows the same subject during manual focus operation. Subject 905 has a face region detected, so the primary subject candidate flag is turned ON. Similarly, for subject 906, although no face region is detected, a head region is detected, so the primary subject candidate flag is also turned ON.

[0089] Based on the main subject candidate flag set in this way, if the processing in S206 determines that subject 905 is the main subject, in S211, a main subject frame 907 indicating the main subject is displayed on subject 905. In addition, a candidate frame 908 is displayed on subject 906, and both subjects 905 and 906 become targets for focus detection processing.

[0090] If the user intentionally focuses on subject 906 in this state, it is possible that they intend to make subject 906 the main subject. Therefore, even if face detection has not been performed, if a head detection result is available, candidate frame 908 will be displayed and included in the focus detection process. In this way, during manual focus operation, subjects that were not considered as the main subject when not in manual focus operation will also be included in the focus detection process, making it possible to switch the main subject in a way that reflects the user's intentions.

[0091] Next, the main subject change process performed in S206 of Figure 2 will be explained using Figure 6. This process is executed when manual focus (MF) operation is in progress.

[0092] In S601, it is determined whether there is one or more subjects with the main subject candidate flag set to ON. If there is one or more subjects with the main subject candidate flag set to ON, the process proceeds to S604; otherwise, the process proceeds to S602.

[0093] In S602, the main subject lock flag for all detected subjects is set to OFF, and the process proceeds to S603. In S603, the main subject flag for all detected subjects is set to OFF, and the main subject change process is terminated. In this case, neither the main subject frame nor the candidate frame will be displayed in S211.

[0094] On the other hand, in S604, it is determined whether there are two or more subjects with the main subject candidate flag turned ON. At least two or more subjects are required to switch the main subject. Therefore, if there are two or more, the process proceeds to S606; otherwise, it proceeds to S605.

[0095] In S605, since there is only one subject with the main subject candidate flag set to ON, the main subject flag for that subject is set to ON, and the main subject change process is terminated.

[0096] In S606, a switching subject identification process is performed to identify the change in the main subject based on the operation information of the lens unit 10 and the focus detection result, and then the process proceeds to S607. The processing in S606 will be described later with reference to Figure 7.

[0097] In S607, it is determined whether or not a change in the main subject has occurred. If a change has occurred, the program proceeds to S608; otherwise, it proceeds to S610.

[0098] In S608, the main subject lock flag is turned ON for the target subject identified in S606. Furthermore, in S609, the main subject flag is turned ON for the target subject identified in S606, and the main subject change process is terminated.

[0099] In S610, since no change in the main subject has occurred, the main subject flag for the subject with the same detection ID as the previous main subject is turned ON, and the main subject change process is terminated.

[0100] As described above, if the main subject changes during manual focus (MF) operation, setting a lock flag for that subject prevents the main subject from changing when AF settings are applied after MF operation is complete. This allows the camera to continuously focus on the subject intended by the user through MF operation.

[0101] Next, using Figure 7, we will explain the process of identifying the switching subject that takes place in S606. First, in S701, the focus detection result for the main subject with the main subject flag ON is obtained, stored as DEF_Main, and the program transitions to S702.

[0102] In the S702, the depth of field is calculated from the information obtained from the lens control unit 106, and N times the depth of field is set as the switching threshold DEF_Th. This switching threshold DEF_Th is the threshold at which the user decides to switch subjects when the amount of blur of the main subject exceeds N times the depth of field. Note that this threshold is just an example, and other values ​​may be used as long as they can detect a change in the main subject.

[0103] In S703, the system determines whether focus detection results are available for subjects with the "Candidate Main Subject" flag ON, excluding subjects with the "Main Subject" flag ON, in order to determine if focus detection results are available for other subjects that could potentially be the target of the switch. If focus detection results are available, the system proceeds to S704; otherwise, it proceeds to S706.

[0104] In S704, the focus detection result DEF_Main of the main subject for which the main subject flag is ON is compared with the switching threshold DEF_Th. If DEF_Main > DEF_Th, it is determined that a user switch has occurred, and the process proceeds to S705. If no switch has occurred, the process proceeds to S712, and it is determined that there has been no switch of the main subject, and the process ends.

[0105] In the S705, among subjects with the main subject candidate flag turned ON, the subject with the smallest amount of bokeh is selected as the new main subject.

[0106] Meanwhile, in S706, the operating direction of the focus ring is obtained, and information on the movement direction (near / infinity direction) of the focus position of the focus lens 103, which can be obtained from the lens control unit 106, is acquired, and the process transitions to S707. In S707, the relative distance from the camera system is determined from the size of the subject for which the main subject candidate flag is ON. Here, the size of the face is used to determine whether the subject is closer or at infinity than the main subject. By comparing it with the size of the main subject, it is generally determined that if the size is larger, the subject is closer, and if the size is smaller, the subject is at infinity. After that, the process transitions to S708.

[0107] In S708, the focus detection result DEF_Main of the main subject for which the main subject flag is ON is compared with the switching threshold DEF_Th. If DEF_Main > DEF_Th, it is determined that a user switch has occurred, and the process proceeds to S709. If no switch has occurred, the process proceeds to S712, and it is determined that there has been no switch of the main subject, and the process ends.

[0108] In S709, it is determined whether the lens is operating in the near-focus direction or not. If the operating direction obtained in S706 is the near-focus direction, the process proceeds to S710; otherwise, it proceeds to S711.

[0109] The S710 identifies the main subject from among the closest subjects. This makes it possible to select the main subject from subjects that are small in size or for which focus detection results could not be obtained depending on the shooting conditions. One of the subjects in the direction of the focus lens operation is selected and identified as the main subject. As mentioned above, the identification method involves selecting the subject that is most likely to be the main subject based on the size and position of the face.

[0110] In S711, the main subject is identified from among the subjects at infinity. The processing is the same as in S710, so the explanation is omitted. After that, the process switches and the subject identification process ends.

[0111] This allows for automatic switching of the main subject during manual focus (MF) operation by acquiring the focus detection result of the subject and monitoring changes in the focus detection result. Furthermore, since focus detection results may not be available depending on the shooting conditions, even in such cases, it is possible to determine the change in the main subject based on the direction of lens operation and face size.

[0112] Here, we will explain the switching of the main subject when the answer to S704 is YES, using Figure 10.

[0113] Figure 10(a) shows an example of the display state before MF operation, showing a case with two subjects: a first subject 1001 (out of focus) and a second subject 1002 (in focus). Here, since the main subject flag for the second subject 1002 is ON, in S211 of Figure 2, a main subject frame 1004 is displayed on the second subject 1002 to indicate that it is the main subject. On the other hand, since the main subject candidate flag for the first subject 1001 is ON, a candidate frame 1003 is displayed to indicate that it is a candidate for the main subject.

[0114] Figure 10(b) shows the case where the focus lens 103 is driven in the direction of the first subject 1001 from a state where the second subject 1002 is in focus via MF operation. When the focus detection result DEF_Main of the second subject 1002 becomes greater than the switching threshold DEF_Th, the main subject flag is turned OFF. Therefore, in S211, instead of the main subject frame 1004 which indicates that it is the main subject, the candidate frame 1006 which indicates that it is a candidate for the main subject is displayed. On the other hand, since the main subject flag for the first subject 1001 is turned ON, the main subject frame 1005 which indicates that it is the main subject is displayed.

[0115] Figure 10(c) is a diagram illustrating the above state in chronological order. In Figure 10(c), the vertical axis represents the focus detection result, with 0 indicating focus. The horizontal axis represents time. The focus detection result for the first subject 1001 is shown in curve 1008, and the focus detection result for the second subject 1002 is shown in curve 1007.

[0116] The state shown in Figure 10(a) is the state based on the focus detection result at time t1. From this state, by driving the focus lens 103, the focus detection result 1008 of the first subject 1001 decreases, while the focus detection result 1007 of the second subject 1002 improves. At point 1012 (time t2), when the focus detection result 1007 exceeds the switching threshold DEF_Th, the camera determines that the user wants to focus on the first subject 1001 and switches the main subject (the state shown in Figure 10(b)).

[0117] In this way, by switching the main subject while the focusing lens is being driven, it becomes possible to simplify the user's operation of the focusing lens. In other words, by setting AF, even if the user stops operating the focus ring when the main subject frame 1005 indicating the main subject is displayed, AF control will be performed on that subject, allowing for continuous focus adjustment from MF operation.

[0118] Next, referring to Figure 8, the main subject lock determination process performed in S208 of Figure 2 will be explained.

[0119] When superimposing a frame to show the face area of ​​people other than the main subject, it can be annoying if the frame continues to be displayed on unintended subjects. However, depending on the shooting situation, there are scenes where it is better to track the subject for a certain amount of time. For example, if there are multiple people you want to photograph in the frame, if you continue to track subjects other than the main subject as potential main subjects for a certain amount of time, when the main subject disappears or is no longer trackable, the main subject will switch to one of the other subjects, which can prevent the focus and exposure from being set on the background.

[0120] Therefore, this process aims to improve the tracking performance of the main subject when it is no longer visible, while also controlling the selection of an appropriate main subject according to the shooting situation.

[0121] First, in S801, it is determined whether there is a subject with the main subject candidate flag (determined in S204) within a predetermined range from the position of the subject (e.g., touch position) determined by the subject specification instruction received from the camera operation unit 208 in S207. The predetermined range mentioned above may be the size of the face area or head area stored in the subject area of ​​the subject information, or it may be within a distance of, for example, twice the size of the face area or head area. If there is a main subject candidate subject within the predetermined range, the process proceeds to S802; otherwise, the process proceeds to S809.

[0122] In S802, the system determines whether there are multiple potential main subjects within a predetermined range of the touch location. If it determines there is only one person, the process proceeds to S803; if it determines there are multiple people, the process proceeds to S804.

[0123] In S803, it is determined whether the confidence level of the face area or head area stored in the subject area of ​​the main subject candidate subject, which is determined to be within a predetermined range of the touch position, is above a predetermined threshold. If it is determined to be above the threshold, the process proceeds to S805; if it is determined to be below the threshold, the process proceeds to S806.

[0124] Meanwhile, in S804, it is determined whether the size of the face or head region of all primary subject candidates that are determined to be within a predetermined range of the touch position is below a threshold. If it is determined that the size is below the threshold, the process proceeds to S807; if it is determined that there are subjects larger than the threshold, the process proceeds to S808.

[0125] Next, S805 and S808 lock onto the main subject candidate that is closest to the touched location within the range of the touched location.

[0126] On the other hand, the S806, S807, and S809 extract features of the subject based on the touch position and track objects other than a person's face or head (hereinafter referred to as "object tracking").

[0127] Next, in S810, the locked primary subject candidate is designated as the primary subject, and the primary subject candidate flag for the locked primary subject is turned ON. Then, in S811, the primary subject candidate flags for subjects other than the locked primary subject are turned OFF, so that subjects other than the locked primary subject do not become primary subject candidates, and the process ends.

[0128] The reason why S806 performs mono tracking when S803 determines that the reliability is below the threshold is that if the reliability of head detection is low, for example, if part of the head is out of frame, the detection position and size may become unstable, potentially leading to unstable autofocus. Therefore, if the reliability of the detected face or head area is low, the system uses color and brightness information from the touch screen to specify the area and temporarily tracks the subject at the user's specified location as a mono. Then, if a face or a head with high reliability is detected nearby during tracking, the system switches the tracking target to the face or head, thereby mitigating the instability of autofocus.

[0129] Furthermore, in the S804, the reason for performing object tracking when multiple small faces or heads are detected within a predetermined range of the touch position is that, due to shifts in the touch position, the user-specified location may result in an unintended person being tracked. Therefore, by temporarily tracking the subject at the user-specified location as an object, and then switching the tracking target to a face or head when a face or head of a predetermined size or larger is detected within the predetermined range, the risk of focusing on an unintended person can be reduced.

[0130] The process described above will be explained using Figure 11. Figure 11(a) shows a scene where person 1101 enters the frame alone and facing away from the camera, when manual focus (MF) operation is not performed. In this case, as indicated by S512 in Figure 5, the flag for person 1101 as the main subject candidate is turned OFF, and therefore no frame is displayed.

[0131] Figure 11(b) shows the same person 1101 facing forward from the state shown in Figure 11(a). In this case, the main subject candidate flag is turned ON at S510 in Figure 5, and since there is only one main subject candidate, person 1101 is automatically selected as the main subject at S605 in Figure 6, and the main subject frame 1102 is displayed.

[0132] Figure 11(c) shows the case where person 1101 turns around from the front-facing position shown in Figure 11(b). In this case, since person 1101 has once faced forward, the main subject candidate flag is ON, and the head region has been detected, so the main subject candidate flag is turned ON at S510 in Figure 5. Also, at S605 in Figure 6, person 1101 is automatically selected as the main subject, and the main subject frame 1102 continues to be displayed. By continuing to set person 1101 as the main subject candidate, the focus is maintained on person 1101, preventing the focus or brightness from being set to an unintended area.

[0133] Figure 11(d) shows a scene with multiple people (person 1103 and person 1105) when manual focus is not being used. In this case, person 1103, who is facing forward, has the main subject candidate flag turned ON at S510 in Figure 5. Since there is only one main subject candidate, person 1103 is automatically selected as the main subject at S605 in Figure 6, and the main subject frame 1104 is displayed. On the other hand, since person 1105 is facing away, the main subject candidate flag is turned OFF at S512, and no frame is displayed for person 1105.

[0134] Figure 11(e) shows the case where person 1105 is facing forward from the state shown in Figure 11(d). In this case, the face area of ​​person 1105 is detected, the main subject candidate flag is turned ON at S510 in Figure 5, and the candidate frame 1106 is displayed. In this case, person 1105 could also be the main subject, but unless there is a change by MF operation (S206), a change by subject specification instruction (S208), or a change of the main subject by automatic determination (S210), person 1103, which has already been selected as the main subject, will continue to be the main subject for the time being.

[0135] Figure 11(f) shows the case where person 1105 turns around from the front-facing position in Figure 11(e). In this case, since person 1105 has once faced forward, the main subject candidate flag is ON, and the head region has been detected, the main subject candidate flag remains ON in S510 of Figure 5. In this way, by continuing to set person 1105 as the main subject candidate, person 1105 can be immediately made the main subject when person 1103 goes out of frame.

[0136] Figures 11(g) to (i) show the cases where there are multiple people (person 1103 and person 1105), and the user selects person 1105 by touch from the state shown in Figures 11(d) to (f).

[0137] In Figure 11(g), since person 1105 is not detected as a subject, person 1103, which is closest to the touch position, continues to be selected as the main subject, and the main subject frame 1104 continues to be displayed.

[0138] Furthermore, in the cases of Figures 11(h) and (i), if the confidence level of the face region or head region is high, the person 1105 is locked as the main subject region by the processing of S801, S802, S803, S805, S810, and S811, and person 1105 is selected as the main subject. The main subject frame 1109 is also displayed. In addition, S811 turns off the subject candidate flag for person 1103, so the main subject frame 1104 that was displayed for person 1103 is deleted.

[0139] As described above, when a person facing away is touched, if a head detection result is found at the touched location, the subject will be locked using the head detection result, regardless of whether it has already been set as a primary subject candidate area or not. When a person that is not yet in the primary subject area or primary subject candidate area is touched, if a head detection result is found at the touched location, the subject will be locked as the primary subject area using the head detection result, enabling strong tracking of changes in the subject's shape and brightness.

[0140] As described above, according to the first embodiment, it becomes possible to switch the subject to be focused on without having to drive the lens to the focus position. Furthermore, when AF is set, a highly versatile AF is achieved by selecting a representative subject as the main subject, and by changing the method of selecting the main subject triggered by MF operation, it becomes possible to select the subject the user intended as the main subject, enabling AF that matches the user's intentions. In addition, the subject selected by MF operation can be continuously focused on as the AF target even after the MF operation. Moreover, it becomes possible to easily switch the main subject.

[0141] Furthermore, the focus detection device of the first embodiment can maintain focus on the subject over a wide range by changing the focus detection area used for autofocus drive according to the position of the subject within the range of a preset AF frame, and can also prevent difficulty in focusing on a new main subject that appears in close proximity when the focus is on a subject other than the main subject, such as the background (hereinafter referred to as "miscellaneous subject").

[0142] <Second Embodiment> Next, a second embodiment of the present invention will be described.

[0143] When determining the main subject using the method described in the first embodiment, it is very useful, for example, when the operator has decided on a person they want to photograph and wants to continuously track that person and adjust the focus and exposure even when their face is no longer visible. However, if the primary goal is to photograph a person as the main subject while also adjusting the focus and exposure for other subjects and scenery without switching shooting modes, continuing to track the person even when their face is no longer visible means that, as long as the person is in the frame, the focus and exposure for other subjects and scenery cannot be adjusted.

[0144] Furthermore, if a subject is immediately removed from the list of potential main subjects once their face disappears from view, the following problem arises: If an unintended person, different from the person the operator is trying to photograph, enters the frame and remains in the frame, the unintended person will remain in the list of potential main subjects indefinitely, resulting in the camera selecting the unintended person as the main subject and altering the focus and exposure.

[0145] Therefore, in the first embodiment described above, even if a face cannot be detected, if the head region can be detected, it will continue to track it as a candidate for the main subject, determine the main subject according to the specified method, and adjust the focus and exposure. However, in situations where the main subject is facing away and their face is temporarily obscured during person-centered photography, it is desirable to keep them as a candidate for the main subject for as long as possible. On the other hand, when photographing things other than people, or when multiple people are included in the shot, whether or not to keep them as a candidate for the main subject depends on the shooting situation.

[0146] Therefore, in the second embodiment, in addition to the first embodiment, the time to be kept as a candidate for the main subject is set for each candidate for the main subject, and this time is further changed according to the situation of the subjects, including the main subject. The processing in the second embodiment will be explained below with reference to Figures 12 to 16. Note that the digital video camera system shown in Figure 1 is used in the second embodiment as well, so the explanation will be omitted.

[0147] Figure 12 is a flowchart showing the main subject candidate determination process in the second embodiment, which is performed instead of the process shown in Figure 5. This process adds the process S1201 to S1203, which is a feature of the second embodiment, to the main subject candidate determination process in the first embodiment described above with reference to Figure 5, and sets the time for each subject from when the face region can no longer be detected until the main subject candidate flag is turned OFF. The other processes are the same as those shown in Figure 5, so the same step numbers are used and the explanation is omitted.

[0148] In S1201, the reset time RT (>0) is set, which is the time from when the face region can no longer be detected until the main subject candidate flag is turned OFF. The reset time RT is set according to the conditions for each subject, and the details will be described later using Figure 13.

[0149] In S1202, if a face region cannot be detected, the reset time RT set in S1201 is decremented. In S1203, the main subject candidate reset determination unit 222 determines whether the reset time RT is greater than 0. If it is greater than 0, the main subject candidate determination unit 220 turns the main subject candidate flag ON in S511. On the other hand, if it is 0 or less, the time until the main subject candidate flag set in S1201 is turned OFF is considered to have expired, and the main subject candidate flag is turned OFF in S512.

[0150] The processing in S1201-S1203 keeps a reset time RT set according to the conditions as long as face detection is possible. If the situation where a face cannot be detected continues for the reset time RT, the subject can be excluded from the main subject candidates. This is because, in general video shooting, a person whose face is not visible for a long time is often not suitable to be selected as the main subject, i.e., a subject for which focus adjustment and exposure control are performed. Therefore, it is considered better to exclude them from the main subject candidates after a certain amount of time has passed. However, in cases where there is only one person being filmed, or even with multiple people but they are near the center of the screen, it may be desirable to continue selecting them as the main subject even if their faces are not visible. Therefore, the reset time RT is adjusted according to the situation.

[0151] Figure 13 is a flowchart detailing the process for setting the main subject candidate reset time RT in S1201 of Figure 12. In S1301, it is determined whether the number of detected subjects n is 1 or not. If it is 1, the process proceeds to S1302; if it is more than 1, the process proceeds to S1303. In S1302, the value of MainTimer1 is set as the reset time RT. Here, the value of MainTimer1 is set for a situation where only the main subject is present in the shooting screen, so it is preferable to set a relatively long time. In this embodiment, however, it will be explained as being set to 10 seconds as an example.

[0152] In S1303, it is determined whether the subject being processed has been selected as the main subject in the main subject determination process among the multiple detected subjects. If it has been selected as the main subject, the process proceeds to S1304; otherwise, it proceeds to S1305. In S1304, the value of MainTimer2 is set as the reset time RT. Here, the value of MainTimer2 is the time for which the subject being processed is kept as the main subject when multiple people are in the shooting screen. Compared to the case of a single person, there is a higher possibility that other main subject candidates may become the main subject, so a shorter time may be set than in the case of a single person, or the same time may be set as in the case of a single person. In this embodiment, as an example, it is set to 10 seconds, the same as MainTimer1.

[0153] In S1305, it is determined whether the number of detected subjects n is less than the threshold NumTh. If it is less, the process proceeds to S1306; otherwise, it proceeds to S1307. In S1306 and S1307, the time for tracking the main subject candidate is set, and this time is set to be shorter than the tracking time for the main subject, MainTimer1 and MainTimer2. This is because subjects that are main subject candidates are less likely to be the target of photography than the main subject, and setting them as main subject candidates for too long prevents the main subject from switching unintentionally. In S1306, the value of SubTimer1 is set as the reset time RT. Here, the value of SubTimer1 is the time for which a subject remains set as a main subject candidate when the number of people is relatively small. In this embodiment, it is explained as being set to 3 seconds as an example.

[0154] On the other hand, in S1307, the value of SubTimer2 is set as the reset time RT. Here, the value of SubTimer2 is the time for which the subject remains set as a candidate for the main subject when there are many people in the shooting screen, and since there is a high possibility that subjects not intended by the user may be captured in the image, a shorter time is set than that of SubTimer1. In this embodiment, as an example, it will be explained that it is set to 1 second.

[0155] In S1308, the process adjusts the time set in S1302 or S1304 for the main subject, making it longer or shorter depending on the conditions. In S1309, similar to S1308, the process adjusts the time set in S1306 and S1307 for the main subject candidate, making it longer depending on the conditions. After the completion of S1308 or S1309, the process returns to the process shown in Figure 12.

[0156] Next, the details of the processing in S1308 will be explained using Figure 14. Figure 14 is a flowchart showing the process of adjusting the tracking time of the main subject according to the conditions. In S1401, an area is set for determining whether or not the main subject is in a predetermined area within the screen, which will be described later in S1403. The concept of the predetermined area will be explained later using Figure 16.

[0157] In S1402, a size threshold, SizeTh, is set to determine whether the face and head regions of the main subject are large enough in S1404. This is because when photographing a person as the main subject, the face is often photographed at a certain size, and a small face is likely to represent a person not intended by the user, i.e., not the main subject.

[0158] In S1403, it is determined whether or not the main subject exists within the predetermined area set in S1401. If it is determined that the subject exists, the process proceeds to S1404; otherwise, the process ends.

[0159] In S1404, it is determined whether the size of the face region and head region is greater than the size threshold SizeTh set in S1402. If it is determined to be greater, the process proceeds to S1405; otherwise, the process terminates.

[0160] In S1405, the time RT until the tracking of the main subject, which was set in S1302 and S1304 in Figure 13, is reset to a longer time than that of MainTimer1 and MainTimer2, and the process is terminated.

[0161] Here, the designated area will be explained using Figure 16. Figure 16 is a schematic diagram representing a scene in which a person who is the subject is in the shooting frame. Figures 16(a) to (c) show the case where there is only one subject, that is, only the main subject is photographed and the main subject's face is not visible, while Figures 16(d) to (f) show the case where there are multiple subjects.

[0162] When shooting video with a person as the main subject, the subject is generally framed near the center of the screen. Therefore, if the subject is detected near the center of the screen, it is highly likely that the subject is the person the user wants to film. To determine if the face and head regions of the detected subject are within area 1601, indicated by the dashed line in Figure 16, the system will continue to track the subject for a longer period of time even if the face is no longer visible. Specifically, the time RT until the tracking of the main subject is reset, as set in S1302 and S1304 in Figure 13, is set to a longer time than that of MainTimer1 and MainTimer2.

[0163] In this embodiment, we will explain using 15 seconds as an example, but it is not limited to this, and you may set it to, for example, 30 seconds or infinite (continue tracking until both detection and tracking are no longer possible), or leave the time of MainTimer1 and MainTimer2 as is. Figure 16(a) shows the case when the face area is visible, and Figure 16(b) shows the case when the main subject turns its back and the face area is no longer visible. In both cases, the subject is within the area of ​​area 1601, so the time RT until the subject tracking is reset is 15 seconds. However, if the subject is near the boundary of area 1601 and frequently enters and exits area 1601, the set time may become unstable, sometimes becoming 10 seconds and sometimes 15 seconds. Therefore, once the main subject enters area 1601, the area may be expanded to be larger than 1601, as shown in area 1602 in Figure 16(c), and the time may be extended until the subject leaves area 1602. In this way, it is possible to prevent the set time from becoming unstable near the boundary of the area.

[0164] Furthermore, while the process shown in Figure 14 explains the conditions for extending MainTimer, if it is not desirable to track the main subject for a long time, the time until the main subject is reset can be shortened. For example, if camera panning is detected, it can be assumed that there is an intention to change the main subject, and the time can be shortened to exclude the main subject whose face is not visible.

[0165] Furthermore, in this embodiment, area 1601 is defined as a rectangular area of ​​a predetermined size centered on the screen. However, considering that the main subject is likely to be in the upper part of the screen, the center position may be shifted upwards, or the area may be expanded upwards. Moreover, the area may be changed according to the orientation of the camera. In addition, in this embodiment, the time is adjusted depending on whether or not a subject is in the area, but this is not the only option, and it may be adjusted according to the image height.

[0166] Next, the details of the process in S1309 will be explained using Figure 15. Figure 15 is a flowchart showing the process of adjusting the tracking time of the main subject candidate according to the conditions. In S1501, an area is set for determining whether or not the main subject candidate is in a predetermined area on the screen in S1503, which will be described later. In this embodiment, this area is described as being the same area as area 1601 in Figures 16(a) to (c), but it may be set to be narrower or wider than in the case of the main subject.

[0167] In S1502, the difference D between the detected position and the subject area selected as the main subject is calculated. This is done to determine whether each of the main subject candidates detected in S1504 is close to the main subject, by determining the distance to the center positions of the face area and head area on the screen, as shown in Figures 16(d) and (e).

[0168] In S1503, it is determined whether or not a candidate subject for the main subject exists within the predetermined area set in S1501. If it is determined that a subject exists, the process proceeds to S1504; otherwise, it proceeds to S1505.

[0169] In S1504, it is determined whether the difference D between the detection position and the main subject, calculated in S1502, is less than or equal to a predetermined value DTh. If it is less than or equal to DTh, it is determined that the distance to the main subject is close and the process proceeds to S1507; if it is greater, it is determined that the object is far from the main subject and the process proceeds to S1505. The predetermined value DTh may be a fixed value determined in advance, or it may be calculated based on the size of the face region and head region.

[0170] In S1507, the time RT until the tracking of the main subject candidate, which was set in S1306 and S1307 in Figure 13, is reset to a time longer than that of SubTimer1 and SubTimer2, and the process is terminated.

[0171] As mentioned above, if a subject is detected near the center of the screen, it is highly likely that the subject is the person the user wants to photograph. Also, if a subject that is a candidate for the main subject is located near the main subject, it is highly likely to be the next candidate for the main subject if the main subject can no longer be detected. For this reason, it is considered better to continue tracking subjects that meet these conditions for a longer period of time even if their faces are no longer visible. In this embodiment, we will explain by setting it to 5 seconds as an example, but it is not limited to this, and the durations of SubTimer1 and SubTimer2 may be left as they are.

[0172] Figure 16(d) shows the case where there is one subject whose face is not visible within area 1601, near the main subject whose face is visible, and another subject whose face is not visible outside area 1601. Figure 16(e) shows the case where there is one subject whose face is not visible within area 1601, but far from the main subject. In the case of Figure 16(d), the subject whose face is not visible is within the designated area 1601 and near the main subject, so the time RT until the subject tracking is reset is 5 seconds. However, in the case of Figure 16(e), the subject whose face is not visible is far from the main subject, so the time RT until the subject tracking is reset is not extended and remains at 3 seconds.

[0173] Furthermore, Figure 16(f) shows a scene with many people in the frame. When there are many subjects, the main subject is more likely to switch unintentionally, so it is best not to track the candidate main subject for too long. This is the case when the number of detections in S1305 in Figure 13 is greater than the predetermined number NumTh, and a short time is set in S1307.

[0174] In S1505, it is determined whether the subject has been selected as the main subject in the past. If it has been selected, it is highly likely that the person the user wants to photograph is the subject, so the process proceeds to S1507, where the time until subject tracking is reset is adjusted. If it has not been selected, the process proceeds to S1506.

[0175] In S1506, it is determined whether the time for which face detection was possible is longer than a predetermined time. If it is longer, it is highly likely that the subject is one that the user intentionally framed in the screen, and it is considered better to track it for a certain amount of time as a potential main subject. Therefore, the process proceeds to S1507, and the time until subject tracking is reset is adjusted. On the other hand, if it is short, it is highly likely that the subject is not one that the user intended to capture, such as when the subject briefly appeared in the screen and then turned away, so the process is terminated without adjusting the time.

[0176] While Figure 15's flowchart explains the conditions for extending the SubTimer, if it's better not to track a candidate subject for the main subject for too long, similar to the main subject, you can shorten the time until the subject is reset. For example, if camera panning is detected, you can assume there is an intention to change the subject and shorten the time to exclude subjects whose faces are not visible from the main subject candidates.

[0177] As described above, according to this second embodiment, the time from when face detection becomes impossible until subject tracking is reset is set for each subject. In particular, the main subject is tracked for a longer period, while the tracking time for other subjects is shorter than that of the main subject. This improves the tracking performance of the main subject when the face is no longer visible, while enabling appropriate selection of the main subject according to the shooting situation.

[0178] In the embodiments described above, the detection results of the human face region and head region were used for explanation, but combinations such as the human face region and body region, or the animal's face region and head region or body region may also be used.

[0179] <Other Embodiments> Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0180] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0181] 10: Lens unit, 20: Camera body, 103: Focus lens, 105: Focus lens drive unit, 106: Lens control unit, 107: Lens operation unit, 201: Image sensor, 204: AF signal processing unit, 205: Display unit, 207: Camera control unit, 208: Camera operation unit, 210: Face area detection unit, 211: Head area detection unit, 220: Main subject candidate determination unit, S221: Main subject lock unit, 222: Main subject candidate reset determination unit

Claims

1. A detection means that uses an imaging means to convert light incident through the imaging optical system into photoelectric signals and repeatedly outputs image signals to detect the facial region of a candidate main subject, which is a candidate for the main subject. A selection means for selecting the main subject from the main subject candidates, An acquisition means for acquiring operation information of an operation means for moving the position of a focus lens included in the imaging optical system, The system includes a focus adjustment means that controls the focus lens to focus on the main subject selected by the selection means when the autofocus mode is set and the operating means is not being operated, The detection means is characterized in that, when the operating means is operated, it reduces the detectable face size compared to when the operating means is not operated.

2. A detection means that converts light incident through the imaging optical system into photoelectric signals and detects the face region of a candidate main subject from the repeatedly output image signal, A selection means for selecting the main subject from the main subject candidates, An acquisition means for acquiring operation information of an operation means for moving the position of a focus lens included in the imaging optical system, The system includes a focus adjustment means that controls the focus lens to focus on the main subject selected by the selection means when the autofocus mode is set and the operating means is not being operated, The detection means is characterized in that, when the operating means is operated, the lower limit of the reliability of detecting the face region is lower than when the operating means is not operated.

3. A detection means for detecting a candidate main subject from an image signal that is repeatedly output by an imaging means, which converts light incident through the imaging optical system into photoelectric signals, A selection means for selecting the main subject from the main subject candidates, An acquisition means for acquiring operation information of an operation means for moving the position of a focus lens included in the imaging optical system, The system includes a focus adjustment means that controls the focus lens to focus on the main subject selected by the selection means when the autofocus mode is set and the operating means is not being operated, The focus adjustment device according to claim 2, characterized in that when the operating means is operated, the detection range within the screen for detecting at least one of the face region and the head region is wider than when the operating means is not operated.

4. A detection means for detecting a candidate main subject from an image signal that is repeatedly output by an imaging means, which converts light incident through the imaging optical system into photoelectric signals, A selection means for selecting the main subject from the main subject candidates, An acquisition means for acquiring operation information of an operation means for moving the position of a focus lens included in the aforementioned imaging optical system, The system includes a focus adjustment means that controls the focus lens to focus on the main subject selected by the selection means when the autofocus mode is set and the operating means is not being operated, The detection means is characterized in that, when the operating means is operated, even if the face region is not detected, if the head region is detected, it is stored as a candidate for the main subject.

5. A detection means that converts light incident through the imaging optical system into photoelectric signals and detects the face region of a candidate main subject from the repeatedly output image signal, A selection means for selecting the main subject from the main subject candidates, An acquisition means for acquiring operation information of an operation means for moving the position of a focus lens included in the aforementioned imaging optical system, The system includes a focus adjustment means that controls the focus lens to focus on the main subject selected by the selection means when the autofocus mode is set and the operating means is not being operated, When the operating means is operated, the detection means detects a candidate for the main subject under different conditions than when the operating means is not operated. The selection means is characterized in that, when the operating means is operated, it selects the main subject based on the focus position of the focus lens, and does not change the main subject even if multiple candidates for the main subject are detected after the selection of the main subject.

6. The focus adjustment device according to claim 5, characterized in that when the operating means is operated and multiple main subject candidates are detected, the selection means selects the main subject candidate with the smallest amount of blur as the main subject.

7. A detection means that converts light incident through the imaging optical system into photoelectric signals and detects a face region of a candidate main subject from the repeatedly output image signal, A selection means for selecting the main subject from the main subject candidates, An acquisition means for acquiring operation information of an operation means for moving the position of a focus lens included in the imaging optical system, The system includes a focus adjustment means that controls the focus lens to focus on the main subject selected by the selection means when the autofocus mode is set and the operating means is not being operated, When the operating means is operated, the detection means detects a candidate for the main subject under different conditions than when the operating means is not operated. The selection means is a focus adjustment device characterized in that, when the operating means is operated and the in-focus state of the selected main subject is lower than the state set based on the depth of focus of the focus lens, the main subject is re-selected.

8. The acquisition means acquires the direction of movement of the focus position of the focus lens due to the operation of the operating means, The selection means re-selects the main subject from among the main subject candidates that exist in the direction of movement of the focus position. The focus adjustment device according to feature 7.

9. A detection means for detecting a candidate main subject from an image signal that is repeatedly output by an imaging means, which converts light incident through the imaging optical system into photoelectric signals, A selection means for selecting the main subject from the main subject candidates, An acquisition means for acquiring operation information of an operation means for moving the position of a focus lens included in the imaging optical system, The system includes a focus adjustment means that controls the focus lens to focus on the main subject selected by the selection means when the autofocus mode is set and the operating means is not being operated, When the operating means is operated, the detection means detects a candidate for the main subject under different conditions than when the operating means is not operated. The detection means is A first detection means for detecting a first region having a first characteristic from the image signal, The system includes a second detection means for detecting a second region relating to the first region and having a second characteristic from the image signal, A focus adjustment device characterized by detecting the main subject candidate based on information from the first region and the second region.

10. The focus adjustment device according to claim 9, characterized in that the detection means detects subjects included in the first region and subjects included in the second region that have been previously detected as main subject candidates when the operating means is not operated, and when the operating means is operated, detects subjects included in at least one of the first region and the second region as main subject candidates.

11. A detection means for detecting a candidate main subject from an image signal that is repeatedly output by an imaging means, which converts light incident through the imaging optical system into photoelectric signals, A selection means for selecting the main subject from the main subject candidates, An acquisition means for acquiring operation information of an operation means for moving the position of a focus lens included in the imaging optical system, When autofocus mode is set and the operating means is not operated, a focus adjustment means controls the focus lens to focus on the main subject selected by the selection means, The system includes a display control means that displays an image on a display means based on an image signal output from the imaging means, When the operating means is operated, the detection means detects a candidate for the main subject under different conditions than when the operating means is not operated. When autofocus mode is set and the operating means is not operated, a first display indicating the main subject selected by the selection means is shown on the image displayed on the display means. When the operating means is operated, the first display and a second display indicating the main subject candidate detected by the detection means are displayed on the image shown on the display means. A focus adjustment device characterized by the following features.

12. The focus adjustment device according to claim 11, further comprising setting means for setting the time for each of the main subject candidates to perform the second display after being detected as such by the detection means.

13. A focus adjustment device according to any one of claims 1 to 12, The imaging means and An imaging device characterized by having the following features.

14. The imaging apparatus according to claim 13, further comprising the aforementioned imaging optical system.

15. The imaging apparatus according to claim 13, characterized in that the imaging optical system is detachable.

16. A detection step involves using an imaging means to convert light incident through the imaging optical system into photoelectric signals, and repeatedly detecting the facial region of a candidate main subject from the output image signal, and A selection step of selecting the main subject from the aforementioned main subject candidates, An acquisition step to acquire operation information of an operating means for moving the position of a focus lens included in the imaging optical system, The system includes a focus adjustment step, in which, when the autofocus mode is set and the operating means is not operated, the focus lens is controlled to focus on the main subject selected in the selection step, The detection step is characterized in that, when the operating means is operated, the detectable face size is made smaller than when the operating means is not operated.

17. A detection step in which light incident through the imaging optical system is photoelectrically converted by an imaging means, and a candidate for the main subject is detected from the repeatedly output image signal, A selection step of selecting the main subject from the aforementioned main subject candidates, An acquisition step to acquire operation information of an operating means for moving the position of a focus lens included in the imaging optical system, The system includes a focus adjustment step, in which, when the autofocus mode is set and the operating means is not operated, the focus lens is controlled to focus on the main subject selected in the selection step, The focus adjustment method is characterized in that, when the operating means is operated, the detection range within the screen for detecting at least one of the face region and the head region is widened compared to when the operating means is not operated.

18. A detection step in which light incident through the imaging optical system is photoelectrically converted by an imaging means, and a candidate for the main subject is detected from the repeatedly output image signal, A selection step of selecting the main subject from the aforementioned main subject candidates, An acquisition step to acquire operation information of an operating means for moving the position of a focus lens included in the imaging optical system, The system includes a focus adjustment step, in which, when the autofocus mode is set and the operating means is not operated, the focus lens is controlled to focus on the main subject selected in the selection step, The focus adjustment method is characterized in that, in the detection step, when the operating means is operated, even if the face region is not detected, if the head region is detected, it is stored as a candidate for the main subject.

19. A detection step in which light incident through the imaging optical system is photoelectrically converted by an imaging means, and a candidate for the main subject is detected from the repeatedly output image signal, A selection step of selecting the main subject from the aforementioned main subject candidates, An acquisition step to acquire operation information of an operating means for moving the position of a focus lens included in the imaging optical system, The system includes a focus adjustment step, in which, when the autofocus mode is set and the operating means is not operated, the focus lens is controlled to focus on the main subject selected in the selection step, In the detection step, when the operating means is operated, the main subject candidate is detected under different conditions than when the operating means is not operated. The selection step is characterized in that, when the operating means is operated, the main subject is selected based on the focus position of the focus lens, and the main subject is not changed even if multiple candidates for the main subject are detected after the selection of the main subject.

20. A program for causing a computer to function as one of the means of a focus adjustment device according to any one of claims 1 to 12.

21. A computer-readable storage medium storing the program described in claim 20.

Images