Control device, imaging device, control method, and program

Abstract

To provide a control unit that, even when an obstacle and the background are mixed near and far with respect to a subject area, can accurately perform focusing in an area with higher priority, while avoiding an area in which focus detection is difficult.SOLUTION: A control unit has: a setting unit that sets a plurality of focus detection frames by using subject information including information on parts of a subject based on an image signal converted from a subject image; a selection unit that selects, from the plurality of focus detection frames in a search area, a main focus detection frame for performing focusing; a subject determination unit that, by using a focus state of the focus detection frame of a part with the highest priority of the priorities set to the parts of the subject, determines whether the subject included in the focus detection frame is the same as a subject included in the main focus detection frame selected in the past; and a decision unit that decides the search area according to the determination made by the subject determination unit.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to a control device, an imaging device, a control method, and a program.

Background Art

[0002] In recent years, in various AF methods such as an imaging plane phase difference AF method and a contrast AF method, techniques for specifying a region of a main subject and focusing have been proposed. Patent Document 1 discloses a method of setting a focus detection region according to the reliability of detection results of organs included in a face.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the method of Patent Document 1, since the focus detection region is set with respect to the position of the main subject, focus detection may not be performed in a region suitable for focus detection. In particular, in the case of a subject with intense movement, if obstacles or backgrounds are mixed in the foreground and background with respect to the focus detection region, desired results may not be obtained.

[0005] An object of the present invention is to provide a control device capable of accurately performing focus adjustment in a region with a higher priority while avoiding a region where focus detection is difficult even when obstacles or backgrounds are mixed in the foreground and background with respect to the subject region.

Means for Solving the Problems

[0006] A control device as one aspect of the present invention is characterized by comprising: a setting unit that sets a plurality of focus detection frames using subject information including information about parts of the subject based on an image signal converted from a subject image; a selection unit that selects a primary focus detection frame for adjusting the focus from the plurality of focus detection frames in the search area; a subject determination unit that determines whether the subject included in the focus detection frame is the same as the subject included in a previously selected primary focus detection frame, using the focus state of the focus detection frame of the part with the highest priority among the priorities set for each part of the subject; and a determination unit that determines the search area according to the determination by the subject determination unit.

[0007] One aspect of the present invention is a control device which includes: a setting unit that sets up a plurality of focus detection frames using subject information including information about parts of the subject based on an image signal converted from a subject image; a selection unit that selects a primary focus detection frame for adjusting the focus from the plurality of focus detection frames in the search area; an obstacle determination unit that determines whether an obstacle exists in an area determined based on the priority set for each part of the subject; and a determination unit that determines the search area according to the determination by the obstacle determination unit. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a control device that can accurately adjust focus in a higher-priority area while avoiding areas where focus detection is difficult, even when obstacles and backgrounds are mixed together at near and far from the subject area. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing the configuration of the camera system in Example 1. [Figure 2] This is a flowchart showing the processing during image capture for the camera system in Example 1. [Figure 3] This is a flowchart showing the AF frame settings for Example 1. [Figure 4] This figure shows the state where the main region has been detected. [Figure 5]This figure shows an image of the AF frame settings in the state shown in Figure 4. [Figure 6] This is a flowchart showing the AF operation in Example 1. [Figure 7] This is a flowchart showing the focus detection process in Example 1. [Figure 8] This is a flowchart showing the selection of the AF main frame in Example 1. [Figure 9] This flowchart shows the selection of the AF main frame in Example 2. [Figure 10] This is a flowchart showing the obstacle detection method for Example 2. [Modes for carrying out the invention]

[0010] The embodiments of the present invention will be described in detail below with reference to the drawings. In each figure, the same reference numeral is used for identical components, and redundant explanations are omitted. [Examples]

[0011] Figure 1 is a block diagram showing the configuration of the camera system in this embodiment. The camera system includes a lens device (interchangeable lens) 100 and a camera body (imaging device) 200. The lens device 100 is detachably (replaceable) attached to the camera body 200 via a mount part (not shown) including an electrical contact unit 106.

[0012] The lens device 100 includes an optical system, a motor 104, and a lens controller 105. The optical system includes a photographic lens 101, an aperture and shutter 102, and a focusing lens 103. The photographic lens 101 includes a zoom mechanism. The aperture and shutter 102 control the amount of light. The focusing lens 103 focuses onto the light-receiving surface of the image sensor 201. The motor 104 drives the focusing lens 103.

[0013] The camera body 200 includes an image sensor 201, an A / D conversion unit 202, an image processing unit 203, an AF signal processing unit 204, a format conversion unit 205, a memory (hereinafter referred to as DRAM) 206, an image recording unit 207, and a timing generator 208. The camera body 200 also includes a system control unit 209, a lens communication unit 210, a subject detection unit 211, an image display memory (VRAM) 212, and an image display unit 213. Furthermore, the camera body 200 includes an operation unit 214, a shooting mode switch 215, a main switch 216, a switch 217 (hereinafter referred to as SW1), and a shooting switch 218 (hereinafter referred to as SW2).

[0014] The image sensor 201 is composed of a CCD sensor or a CMOS sensor and converts the subject image formed through the optical system into an image signal. The A / D conversion unit 202 includes a CDS circuit and a nonlinear amplification circuit to remove output noise from the image sensor 201. The DRAM 206 is an internal memory, such as a random access memory, and is used as a temporary buffer and working memory for image compression and decompression. The image recording unit 207 consists of a recording medium such as a memory card and its interface. The lens communication unit 210 communicates with the lens device 100. The image display unit 213 displays images, provides operation assistance, and displays the status of the camera system. In addition, the image display unit 213 displays the shooting screen and the focus detection area when shooting. The operation unit 214 includes menu switches for setting various settings such as the shooting function of the camera body 200 and settings for image playback, and a switch for switching between shooting mode and playback mode, and is used to operate the camera system from the outside. The shooting mode switch 215 is used to select a shooting mode such as macro mode or sports mode. The main switch 216 is used to power on the camera system. SW1 is used to perform shooting standby operations such as AF and AE. SW2 is used to take a picture after SW1 has been operated.

[0015] The system control unit 209 controls the entire camera system. FIG. 1(b) is a block diagram of the system control unit 209. The system control unit 209 includes a setting unit 209a, a selection unit 209b, a subject determination unit 209c, and a determination unit 209d. In this embodiment, the system control unit 209 is mounted in the camera body 200, but it may be configured as a control device separate from the camera body 200.

[0016] The light beam incident on the light receiving surface of the image sensor 201 from the lens device 100 is converted into signal charges corresponding to the incident light amount by photodiodes. The signal charges accumulated in each photodiode are sequentially read out from the image sensor 201 as voltage signals based on drive pulses given from the timing generator 208 according to the commands of the system control unit 209.

[0017] Each pixel of the image sensor 201 used in this embodiment is composed of two (a pair) of photodiodes A and B and one microlens provided for the pair of photodiodes A and B. Each pixel divides the incident light beam with the microlens to form a pair of optical images on the pair of photodiodes A and B, and outputs a pair of pixel signals (A signal and B signal) used for the AF signal from the pair of photodiodes A and B. Also, an imaging signal (A + B signal) can be obtained by adding the outputs of the pair of photodiodes A and B.

[0018] By synthesizing the plurality of A signals and the plurality of B signals output from the plurality of pixels, a pair of image signals as AF signals (focus detection signals) used for autofocus (imaging plane phase difference AF) by the imaging plane phase difference detection method are obtained. The AF signal processing unit 204 obtains the phase difference (hereinafter, image shift amount), which is the shift amount of the pair of image signals, by performing a correlation operation on the pair of image signals, and obtains focus information including the defocus amount (focus state), defocus direction, and reliability of the optical system using the image shift amount. The AF signal processing unit 204 obtains a plurality of defocus amounts in a plurality of specified predetermined regions.

[0019] Figure 2 is a flowchart showing the processing of the camera system during shooting in this embodiment.

[0020] In step S201, the system control unit 209 determines whether SW1 is ON. If it is determined that SW1 is ON, the process proceeds to step S202; otherwise, the process of this step is repeated.

[0021] In step S202 (setting step), the system control unit 209 (setting unit 209a) performs AF frame settings to set up multiple AF frames (focus detection frames).

[0022] In step S203, the system control unit 209 performs AF operation.

[0023] In step S204, the system control unit 209 determines whether SW1 is ON. If it is determined that SW1 is ON, the process proceeds to step S205; otherwise, it returns to step S201.

[0024] In step S205, the system control unit 209 determines whether SW2 is ON. If it is determined that SW2 is ON, the process proceeds to step S206; otherwise, it returns to step S204.

[0025] In step S206, the system control unit 209 causes the camera system to perform a shooting operation.

[0026] Figure 3 is a flowchart showing the AF frame settings in step S202 of Figure 2.

[0027] In step S301, the system control unit 209 acquires subject information from the subject detection unit 211. In this embodiment, the subject detection unit 211 uses deep learning and image processing means to detect subjects such as people and animals (e.g., dogs and wild birds), and the main regions of the subjects such as their eyes, face, and body. The subject information includes the type of subject (whether it is a person or an animal), and the center coordinates, horizontal size, and vertical size of the main regions.

[0028] In step S302, the system control unit 209 determines whether multiple major regions have been detected using the subject information. Figure 4 shows the state in which major regions have been detected. Figure 4(a) shows the state in which a single major region (face a) has been detected when the subject is a person. Figure 4(b) shows the state in which multiple major regions (pupil A, face B, body C) have been detected when the subject is a person. Figures 4(c) and 4(d) show the state in which multiple major regions (pupil A, face B, body C) have been detected when the subject is an animal. If it is determined that multiple major regions have been detected, the process proceeds to step S303; otherwise, the process proceeds to step S306.

[0029] In step S303, the system control unit 209 sets the size of the AF frame to the smaller of the horizontal and vertical sizes (MinA) of the smallest main area (for example, pupil A in Figures 4(b) to (d)).

[0030] In step S304, the system control unit 209 first obtains the horizontal size of the area encompassing all major areas from the horizontal coordinates and horizontal sizes of each major area. Next, the system control unit 209 sets the number of horizontal AF frames by dividing the obtained horizontal size by the AF frame size MinA set in step S304. For example, as shown in Figures 5(b) to 5(d), the system control unit 209 sets the number of horizontal AF frames (H / MinA) by dividing the horizontal size H of the area encompassing multiple major areas (pupil A, face B, body C) by the AF frame size MinA.

[0031] In step S305, the system control unit 209 first obtains the vertical size of the area encompassing all major areas from the vertical coordinates and vertical sizes of each major area. Next, the system control unit 209 sets the number of vertical AF frames as the value obtained by dividing the obtained vertical size by the AF frame size MinA set in step S304. For example, as shown in Figures 5(b) to 5(d), the system control unit 209 sets the number of vertical AF frames (H / MinA) by dividing the vertical size V of the area encompassing multiple major areas (pupil A, face B, body C) by the AF frame size MinA.

[0032] In this embodiment, the AF frame is square in shape, but it may also be rectangular in shape, with different horizontal and vertical lengths. Furthermore, the number of AF frames may be set to a number that can be calculated by the system control unit 209.

[0033] In step S306, the system control unit 209 sets an AF frame of a predetermined size for the main area. For example, as shown in Figure 5(a), the system control unit 209 sets an AF frame of a predetermined size X for face a. The predetermined size may be the size of the pupil estimated from the face, or it may be a size that ensures a sufficient signal-to-noise ratio and focusing performance, taking into account low-light environments.

[0034] In step S307, the system control unit 209 sets the number of AF frames that can encompass the main area using the AF frames set in step S306. For example, in Figure 5(a), the system control unit 209 sets the number of AF frames Y that can encompass face a.

[0035] Figure 6 is a flowchart showing the AF operation in step S203 of Figure 2.

[0036] In step S401, the system control unit 209 instructs the AF signal processing unit 204 to perform focus detection processing and acquires focus information including the amount of defocus and reliability.

[0037] In step S402 (selection step), the system control unit 209 (selection unit 209b) selects the AF main frame using the reliability acquired in step S401.

[0038] In step S403, the system control unit 209 determines whether the confidence level obtained in step S401 is higher than a preset first threshold. The first threshold is set so that the accuracy of the defocus amount cannot be guaranteed if the confidence level is lower than the first threshold, but the focus position direction of the subject can be guaranteed. If it is determined that the confidence level is higher than the first threshold, the system proceeds to step S404; otherwise, it proceeds to step S413. Note that if the confidence level is equal to the first threshold, the system can arbitrarily set which step to proceed to.

[0039] In step S404, the system control unit 209 determines whether the amount of defocus acquired in step S401 is smaller than a preset second threshold. The second threshold is a value (for example, five times the depth of focus) that allows the focus lens 103 to be controlled within the depth of focus by driving for a predetermined number of times (for example, three times) equal to the amount of defocus, if the amount of defocus is smaller than the second threshold. If it is determined that the amount of defocus is smaller than the second threshold, the system proceeds to step S405; otherwise, it proceeds to step S412. Note that if the amount of defocus is equal to the second threshold, the system can arbitrarily set which step to proceed to.

[0040] In step S405, the system control unit 209 determines whether the focus lens 103 is stopped. If it is determined that the focus lens 103 is stopped, the process proceeds to step S406; otherwise, the process proceeds to step S410.

[0041] In step S406, the system control unit 209 determines whether the confidence level obtained in step S401 is higher than a preset third threshold. The third threshold is set such that when the confidence level is higher than the third threshold, the accuracy variation of the defocus amount falls within a predetermined range (for example, within the depth of focus). If it is determined that the confidence level is higher than the third threshold, the system proceeds to step S407; otherwise, it proceeds to step S410. Note that if the confidence level is equal to the third threshold, the system can arbitrarily set which step to proceed to.

[0042] In step S407, the system control unit 209 determines whether the amount of defocus acquired in step S401 is smaller than a preset fourth threshold. The fourth threshold is set so that the focus lens 103 is controlled within the depth of field when the amount of defocus is smaller than the fourth threshold. If it is determined that the amount of defocus is smaller than the fourth threshold, the system proceeds to step S408; otherwise, it proceeds to step S409. Note that if the amount of defocus is equal to the fourth threshold, the system can arbitrarily set which step to proceed to.

[0043] In step S408, the system control unit 209 displays the focus frame on the image display unit 213.

[0044] In step S409, the system control unit 209 drives the focus lens 103 via the lens controller 105 by the amount of defocus acquired in step S401. By performing the processing in steps S405 to S409, if the confidence level acquired in step S401 is higher than the third threshold, the amount of defocus can be detected again with the focus lens 103 stopped.

[0045] In step S410, the system control unit 209 drives the focus lens 103 via the lens controller 105 by a predetermined percentage of the defocus amount acquired in step S401.

[0046] In step S411, the system control unit 209 stops the focus lens 103 via the lens controller 105.

[0047] In step S412, the system control unit 209 drives the focus lens 103 via the lens controller 105 by a predetermined percentage of the defocus amount acquired in step S401. The predetermined percentage is set such that the amount of drive of the focus lens 103 is small relative to the defocus amount, for example, 80%. The speed of the focus lens 103 is set to be slower than the speed at which the drive is completed in the time of one frame. This prevents the focus from exceeding the subject's focus position if the defocus amount is incorrect, and also allows the next drive to be performed while the focus lens 103 is being driven without stopping (overlap control).

[0048] In step S413, the system control unit 209 determines whether the non-focusing condition is satisfied, which is necessary to determine that there is no subject to focus on. For example, the non-focusing condition is set as the condition that the drive is completed throughout the entire movable range of the focus lens 103, that is, when the focus lens 103 has detected both the far and near lens ends and returned to its initial position. If it is determined that the non-focusing condition is satisfied, the process proceeds to step S414; otherwise, the process proceeds to step S415.

[0049] In step S414, the system control unit 209 causes the image display unit 213 to display an out-of-focus image.

[0050] In step S415, the system control unit 209 determines whether the focus lens 103 has reached the far or near end of the lens. If it is determined that the lens end has been reached, the system proceeds to step S416; otherwise, the system proceeds to step S417.

[0051] In step S416, the system control unit 209 reverses the drive direction of the focus lens 103 via the lens controller 105.

[0052] In step S417, the system control unit 209 drives the focus lens 103 in a predetermined direction via the lens controller 105. The speed of the focus lens 103 is set to the fastest speed within a range of speeds that will not overshoot the focus position when the amount of defocus can be detected.

[0053] In step S418, the system control unit 209 determines whether the AF mode set by the operation unit 214 is the servo AF mode and whether SW1 is ON, i.e., the servo AF state. If it is determined to be the servo AF state, the process proceeds to step S419; otherwise, this flow terminates.

[0054] In step S419, the system control unit 209 instructs the AF signal processing unit 204 to perform focus detection processing and acquires focus information including the amount of defocus and reliability.

[0055] In step S420, the system control unit 209 (selection unit 209b) selects the AF main frame using the reliability acquired in step S419.

[0056] In step S421, the system control unit 209 performs motion prediction AF processing to predict the focus position (predicted position) of the subject in the target frame from the history information of the subject's position in multiple past frames.

[0057] In step S422, the system control unit 209 drives the focus lens 103 to the predicted position predicted in step S421.

[0058] Figure 7 is a flowchart showing the focus detection process in steps S401 and S419 of Figure 6.

[0059] In step S501, the AF signal processing unit 204 sets a focus detection area of ​​any range within the image sensor 201.

[0060] In step S502, the AF signal processing unit 204 acquires a pair of image signals from the image sensor 201 in the focus detection region set in step S501.

[0061] In step S503, the AF signal processing unit 204 performs vertical row averaging on the pair of image signals acquired in step S502 in order to reduce the influence of noise on the pair of image signals.

[0062] In step S504, the AF signal processing unit 204 performs a filter process to extract signal components of a predetermined frequency band from the signal that has undergone vertical row averaging in step S503.

[0063] In step S505, the AF signal processing unit 204 obtains a correlation amount using the signal that was filtered in step S504.

[0064] In step S506, the AF signal processing unit 204 obtains the correlation change amount using the correlation amount obtained in step S505.

[0065] In step S507, the AF signal processing unit 204 obtains the image shift amount using the correlation change amount obtained in step S506.

[0066] In step S508, the AF signal processing unit 204 obtains a reliability score that indicates how reliable the amount of image shift acquired in step S507 is.

[0067] In step S509, the AF signal processing unit 204 converts the image shift amount into a defocus amount.

[0068] Figure 8 is a flowchart showing the selection of the AF main frame in steps S402 and S420 of Figure 6 in this embodiment.

[0069] In step S601, the system control unit 209 determines whether the subject detection unit 211 has detected the subject's pupils. If it is determined that the subject detection unit 211 has detected the subject's pupils, the system proceeds to step S602; otherwise, the system proceeds to step S603.

[0070] In step S602, the system control unit 209 (decision unit 209d) sets the search area of ​​the AF main frame (main focus detection frame) to the pupil area.

[0071] In step S603, the system control unit 209 determines whether the subject detection unit 211 has detected the subject's face. If it is determined that the subject detection unit 211 has detected the subject's face, the process proceeds to step S604; otherwise, the process proceeds to step S616.

[0072] In step S604, the system control unit 209 (decision unit 209d) sets the search area of ​​the AF main frame to the face area.

[0073] The processing in steps S601 and S603 is performed based on the priority given to the parts of the subject detected by the subject detection unit 211. In this embodiment, the priority is set in the order of pupil, face, and body, but different priorities may be set as needed. For example, priorities may be set according to the posture of the subject or the posture of the camera system.

[0074] In step S605, the system control unit 209 predicts the subject position (predicted position) at the time of defocus detection in the target frame from the historical information of the subject positions of multiple past frames.

[0075] In step S606 (determination step), the system control unit 209 (determination unit 209c) determines whether the amount of defocus of the AF frame closest to the predicted position within the search area of ​​the AF main frame is within a predetermined range relative to the predicted position. The predetermined range is a range (for example, twice the depth of field) for considering the subject as the same subject as the subject that was in focus in past frames, and it is desirable that this range be set considering prediction errors and focus detection errors. Furthermore, the predetermined range does not need to be a fixed value and may be changed according to the subject's speed, distance, and other conditions. In addition, in this step, only AF frames with a defocus amount reliability higher than a predetermined threshold may be selected. If it is determined that the defocus amount is within the predetermined range, the process proceeds to step S607; otherwise, the process proceeds to step S608.

[0076] In step S607, the system control unit 209 (selection unit 209b) selects the AF frame that is closest to the predicted position within the search area of ​​the AF main frame as the AF main frame.

[0077] In step S608, the system control unit 209 determines whether the subject detection unit 211 has detected the subject's body. If it is determined that the subject detection unit 211 has detected the subject's body, the process proceeds to step S609; otherwise, the process proceeds to step S612.

[0078] In step S609 (decision step), the system control unit 209 (decision unit 209d) sets the search area of ​​the AF main frame to an area that encompasses all major areas (all parts of the subject). The area that encompasses all major areas can be determined by the method described in Figure 5. Alternatively, the search area of ​​the AF main frame may be set to an area obtained by multiplying the area determined by the method described in Figure 5 by a predetermined magnification.

[0079] In step S610, the system control unit 209 counts the defocus amount acquired in step S509 for each predetermined depth in the search area of ​​the AF main frame and creates a histogram. In this embodiment, the defocus amount itself is histogrammed, but considering the moving subject, a histogram may be created of predicted values ​​corresponding to the subject position acquired using the defocus amount for each AF frame.

[0080] In step S611, the system control unit 209 (selection unit 209b) selects the AF frame that is closest in coordinate position to the AF main frame of the previous frame from among the AF frames counted as histogram peaks. This increases the likelihood of selecting an AF frame of the same subject that was in focus in previous frames.

[0081] In step S612 (decision step), the system control unit 209 (decision unit 209d) sets the search area of ​​the AF main frame to an area within a predetermined range from the center of the subject detection area. The predetermined range is set to be wider than the size of the subject detected by the subject detection unit 211. This allows for the selection of a more suitable AF frame from a wider area even if an AF frame for the same subject as in a previous frame cannot be selected in a narrow area such as a face or pupil. The predetermined range may be a fixed value or a value obtained by multiplying the size of the subject by a magnification factor.

[0082] In step S613, the system control unit 209 (determination unit 209c) determines whether the amount of defocus of the AF frame closest to the predicted position, among the AF frames whose distance to the previous AF main frame is less than or equal to a predetermined value within the search area of ​​the AF main frame, is within a predetermined range relative to the predicted position. By selecting from AF frames whose distance to the previous AF main frame is less than or equal to a predetermined value within the search area of ​​the AF main frame, the likelihood of selecting an AF frame of the same subject as in a previous frame increases. If it is determined that the amount of defocus is within the predetermined range, the system proceeds to step S614; otherwise, it proceeds to step S615.

[0083] In step S614, the system control unit 209 (selection unit 209b) selects as the AF main frame the AF frame that is closest to the predicted position among the AF frames whose distance to the previous AF main frame determined in step S613 is less than or equal to a predetermined value.

[0084] In step S615, the system control unit 209 (selection unit 209b) selects the AF frame that is closest to the predicted position from among the search areas of the AF main frame as the AF main frame.

[0085] In step S616, the system control unit 209 (selection unit 209b) selects the AF main frame from a predetermined area on the screen without using subject information.

[0086] As described above, with the configuration of this embodiment, when multiple parts of a subject are detected, it becomes possible to accurately adjust the focus on a higher-priority area while avoiding areas where focus detection is difficult. For example, when the subject is an animal such as a dog, cat, or bird, or a person participating in a sporting event, it is possible to select an AF frame that is highly likely to be the same subject as a previous frame and adjust the focus. [Examples]

[0087] The basic configuration of the camera system in this embodiment is the same as that of the camera system in Embodiment 1. In this embodiment, we will explain the differences from the configuration of Embodiment 1.

[0088] In this embodiment, the system control unit 209 has an obstacle detection unit 209e instead of a subject detection unit 209c.

[0089] Figure 9 is a flowchart showing the selection of the AF main frame in steps S402 and S420 of Figure 6 in this embodiment.

[0090] In step S701, the system control unit 209 determines whether the subject detection unit 211 has detected a part of the subject (at least one of the pupil, face, and body). If it is determined that a part of the subject has been detected, the system proceeds to step S702; otherwise, it proceeds to step S714.

[0091] In step S702, the system control unit 209 performs obstacle detection. Obstacle detection involves determining whether there are any obstructing objects mixed in with the subject area detected by the subject detection unit 211 or in the area near it.

[0092] In step S703 (determination step), the system control unit 209 (obstacle determination unit 209e) determines whether an obstacle exists using obstacle determination. If it is determined that an obstacle exists, the process proceeds to step S704; otherwise, the process proceeds to step S711.

[0093] In step S704, the system control unit 209 determines whether the subject detection unit 211 has detected the subject's face. If it is determined that the subject detection unit 211 has detected the subject's face, the system proceeds to step S705; otherwise, it proceeds to step S706.

[0094] In step S705 (decision step), the system control unit 209 (decision unit 209d) sets the search area of ​​the AF main frame to an area that is a predetermined multiple of the size of the face detected by the subject detection unit 211 (an area larger than the size of the face). This makes it less susceptible to the influence of obstacles.

[0095] In step S706, the system control unit 209 determines whether the subject detection unit 211 has detected the subject's pupils. If it is determined that the subject detection unit 211 has detected the subject's pupils, the system proceeds to step S707; otherwise, it proceeds to step S710.

[0096] In step S707, the system control unit 209 determines whether the subject detection unit 211 has detected the subject's body. If it is determined that the subject detection unit 211 has detected the subject's body, the system proceeds to step S708; otherwise, it proceeds to step S709.

[0097] In step S708 (decision step), the system control unit 209 (decision unit 209d) sets the search area of ​​the AF main frame to an area that encompasses all major areas (all parts of the subject). The area that encompasses all major areas can be determined by the method described in Figure 5. Alternatively, the search area of ​​the AF main frame may be set to an area obtained by multiplying the area determined by the method described in Figure 5 by a predetermined magnification.

[0098] In step S709 (decision step), the system control unit 209 (decision unit 209d) sets the search area of ​​the AF frame to an area that is a predetermined multiple of the pupil size detected by the subject detection unit 211 (an area larger than the face size estimated from the pupil size). This makes it less susceptible to the influence of obstacles.

[0099] In step S710 (decision step), the system control unit 209 (decision unit 209d) sets the search area of ​​the AF frame to an area that is a predetermined multiple of the body size detected by the subject detection unit 211 (an area that is larger than 1 times the body size). This makes it less susceptible to the influence of obstacles.

[0100] In step S711 (decision step), the system control unit 209 (decision unit 209d) sets the search area of ​​the AF frame to the detection area of ​​the part with the highest priority among the parts detected by the subject detection unit 211. In this embodiment, the priority is set in the order of pupil, face, and body, but different priorities may be set as needed. For example, priorities may be set according to the posture of the subject or the posture of the camera system.

[0101] In step S712, the system control unit 209 predicts the subject position (predicted position) at the time of defocus detection in the target frame from the historical information of the subject positions of multiple past frames.

[0102] In step S713, the system control unit 209 selects the AF frame closest to the predicted position within the AF frame search area as the AF main frame. Alternatively, the system control unit 209 may count the amount of defocus at predetermined depths to create a histogram, and select the AF frame closest to the coordinate position of the AF main frame of the previous frame from among the AF frames counted as histogram peaks as the AF main frame.

[0103] In step S714, the system control unit 209 selects the AF main frame from a predetermined area on the screen without using subject information.

[0104] Figure 10 is a flowchart showing the obstacle detection in step 702 of Figure 9.

[0105] In step S801, the system control unit 209 determines whether the subject detection unit 211 has detected the subject's face. If it is determined that the subject detection unit 211 has detected the subject's face, the system proceeds to step S802; otherwise, it proceeds to step S803.

[0106] In step S802, the system control unit 209 sets the search area for obstacle detection to an area that is a predetermined multiple of the size of the face detected by the subject detection unit 211.

[0107] In step S803, the system control unit 209 determines whether the subject detection unit 211 has detected the subject's pupils. If it is determined that the subject detection unit 211 has detected the subject's pupils, the system proceeds to step S804; otherwise, it proceeds to step S805.

[0108] In step S804, the system control unit 209 sets the search area for obstacle detection to an area that is a predetermined multiple of the pupil size detected by the subject detection unit 211.

[0109] In step S805, the system control unit 209 determines whether the subject detection unit 211 has detected the subject's face. If it is determined that the subject detection unit 211 has detected the subject's face, the process proceeds to step S806; otherwise, the process proceeds to step S812.

[0110] In step S806, the system control unit 209 sets the search area for obstacle detection to an area that is a predetermined multiple of the size of the body detected by the subject detection unit 211.

[0111] In step S807, the system control unit 209 predicts the subject position (predicted position) at the time of defocus detection in the target frame from the historical information of the subject positions of multiple past frames.

[0112] In step S808, the system control unit 209 sets the AF frame that is closest to the predicted position within the obstacle detection search area as the reference point.

[0113] In step S809, the system control unit 209 counts the number of front-focused AF frames within the obstacle detection search area by a predetermined amount compared to the defocus amount of the reference point. The predetermined amount is the range (for example, twice the depth of field) used to determine that an object is an obstacle and not the same subject that was in focus in previous frames. It is desirable that this amount be set considering prediction errors and focus detection errors. Furthermore, the predetermined amount does not need to be a fixed value and may be changed according to the subject's speed, distance, and other conditions. Note that since obstacles are considered to be closer to the subject, only the number of front-focused AF frames is counted.

[0114] In step S810, the system control unit 209 determines whether the count is equal to or greater than a predetermined percentage of the number of AF frames in the obstacle detection search area. Since obstacles have a certain size, a higher count increases the likelihood of an obstacle being present. Therefore, by determining the presence of an obstacle only when the count exceeds a certain percentage (for example, 20%), false detections can be reduced. Alternatively, the determination may be made based on a fixed number instead of a percentage. In this embodiment, obstacles are determined based on the amount of defocus within the subject area, but the presence of obstacles in the subject area may also be determined using a deep learning learning method or image processing means. If it is determined that the count is equal to or greater than the predetermined percentage, the process proceeds to step S811; otherwise, the process proceeds to step S812.

[0115] In step S811, the system control unit 209 determines that an obstacle is present.

[0116] In step S812, the system control unit 209 determines that no obstacles are present.

[0117] As described above, according to the configuration of this embodiment, when multiple parts of a subject are detected and there are obstacles in the subject detection area, it is possible to accurately adjust the focus on a higher priority area while avoiding difficult areas. [Other examples] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0118] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist. [Explanation of symbols]

[0119] 209 System Control Unit (Control Device) 209a Setting section 209b Selection Section 209c Subject determination section 209d Decision section

Claims

1. a setting unit that sets a plurality of focus detection frames using subject information including information about parts of the subject based on an image signal converted from an image of the subject; a selection unit that selects a main focus detection frame for performing focus adjustment from the plurality of focus detection frames in the search area; a subject determination unit that determines whether the subject included in the focus detection frame is the same as a subject included in a previously selected main focus detection frame, using the focus state of the focus detection frame of the part with the highest priority among the priorities set for each part of the subject; A control device comprising: a determination unit that determines the search area in accordance with the determination by the subject determination unit.

2. 2. The control device according to claim 1, wherein the subject determination unit determines that the subject included in the focus detection frame is the same as the subject included in the previously selected main focus detection frame when the focus state of the focus detection frame is smaller than a predetermined value based on the focus states of multiple previously selected main focus detection frames, and determines that the subject included in the focus detection frame is not the same as the subject included in the previously selected main focus detection frame when the focus state of the focus detection frame is larger than the predetermined value.

3. The control device according to claim 1 or 2, characterized in that, when the subject determination unit determines that the subject included in the focus detection frame is the same as the subject included in the principal focus detection frame selected in the past, the determination unit determines the area including the part with the highest priority as the search area.

4. The control device according to any one of claims 1 to 3, characterized in that when the subject determination unit determines that the subject included in the focus detection frame is not the same as the subject included in the previously selected main focus detection frame, if the body of the subject is detected, the determination unit determines an area that includes all detected parts of the subject as the search area, and if the body of the subject is not detected, determines a predetermined range centered on the area that includes the part with the highest priority as the search area.

5. a setting unit that sets a plurality of focus detection frames using subject information including information about parts of the subject based on an image signal converted from an image of the subject; a selection unit that selects a main focus detection frame for performing focus adjustment from the plurality of focus detection frames in the search area; an obstacle determination unit that determines whether an obstacle exists in the area determined based on the priority set for each part of the subject; A control device comprising: a determination unit that determines the search area in accordance with the determination by the obstacle determination unit.

6. 6. The control device according to claim 5, wherein the obstacle determining unit determines whether the obstacle is present using focus states of a plurality of focus detection frames in the area determined based on the priority.

7. 7. The control device according to claim 5, wherein the determination unit determines, when the obstacle determination unit determines that the obstacle exists, an area that is a predetermined multiple of an area that includes any part of the subject as the search area, and when the obstacle determination unit determines that the obstacle does not exist, determines, when the obstacle determination unit determines that the obstacle does not exist, an area that includes the part with the highest priority as the search area.

8. 8. The control device according to claim 5, wherein, when the obstacle determination unit determines that the obstacle exists and the body of the subject has been detected, the determination unit determines an area that includes all of the detected parts of the subject as the search area, and when the obstacle determination unit determines that the obstacle does not exist, the determination unit determines an area that includes the part with the highest priority as the search area.

9. 9. The control device according to claim 1, wherein the priority is set in descending order of pupil, face, and body.

10. an image sensor that converts a subject image into an image signal; An imaging device comprising the control device according to any one of claims 1 to 9.

11. a setting step of setting a plurality of focus detection frames using subject information including information about parts of the subject based on an image signal converted from an image of the subject; a determination step of determining whether the subject included in the focus detection frame is the same as the subject included in a main focus detection frame for focus adjustment selected in the past, according to the focus state of the focus detection frame of the part with the highest priority among the priorities set for each part of the subject; a determination step of determining a search area for selecting the main focus detection frame in accordance with the determination made in the determination step; a selection step of selecting the main focus detection frame from the plurality of focus detection frames in the search area.

12. a setting step of setting a plurality of focus detection frames using subject information including information about parts of the subject based on an image signal converted from an image of the subject; a determining step of determining whether an obstacle exists in the area determined based on the priority set for each part of the subject; a determination step of determining a search area for selecting a main focus detection frame for performing focus adjustment in accordance with the determination made in the determination step; a selection step of selecting the main focus detection frame from the plurality of focus detection frames in the search area.

13. A program that causes a computer to execute the control method according to claim 11 or 12.