Focus adjustment apparatus, control method therefor, and electronic device

Abstract

Focus adjustment apparatus capable of improving accuracy of automatic focus adjustment for small feature region and control method thereof SOLUTION: The focus adjustment device performs focus adjustment of a photographing optical system so that a subject area detected from a photographed image is focused. A focus adjustment device sets a plurality of regions for calculating a defocus amount as regions in an image, and selects a region to be used for focus adjustment from among the plurality of regions. Then, the focus adjustment device performs focus adjustment of the imaging optical system based on the defocus amount calculated for the area used for focus adjustment. The focus adjustment device sets a plurality of regions in a range that includes the subject region, and if the subject region is smaller than a specified size the range is made narrower than if it is not smaller.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a focus adjustment device that adjusts the focal distance of an imaging device, a control method for the same, and electronic equipment. [Background technology]

[0002] BACKGROUND ART There is known an imaging device that detects a characteristic area such as a person's face from a captured image and performs automatic focus adjustment so that the detected characteristic area is brought into focus (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7154758 Summary of the Invention [Problem to be solved by the invention]

[0004] Advances in detection technology have made it possible to detect small feature regions (e.g., face regions). Therefore, for example, when focus adjustment is performed to focus on a detected feature region, the feature region may be smaller than the focus detection region set for the feature region. In this case, the focus detection region includes not only the feature region but also the background region. Because the background is significantly different in distance from the subject in the feature region, the accuracy of focus adjustment using the background region as the focus detection region is likely to decrease.

[0005] In addition, for example, a scene in which multiple athletes are photographed within the frame can be considered as a scene in which a small subject is photographed. In this case, multiple characteristic areas (for example, facial areas) are detected, so there is a risk that the athlete being focused on may be mistakenly switched from the intended athlete to another athlete.

[0006] In one aspect, the present invention provides a focus adjustment device and a control method thereof that can improve the accuracy of automatic focus adjustment for small feature areas, thereby alleviating one or more of the problems of the prior art. [Means for solving the problem]

[0007] In one aspect, the present invention provides a focus adjustment device that adjusts the focus of a photographic optical system so that a subject area detected from a photographed image is in focus, the focus adjustment device comprising: a setting means that sets multiple areas within the image for which a defocus amount is calculated; a selection means that selects an area to be used for focus adjustment from the multiple areas; and an adjustment means that adjusts the focus of the photographic optical system based on the defocus amount calculated for the area to be used for focus adjustment, wherein the setting means sets multiple areas in a range that includes the subject area, and when the subject area is smaller than a predetermined size, the range is narrower than when the subject area is not small. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a focus adjustment device and a control method thereof that can improve one or more of the problems of the prior art by improving the accuracy of automatic focus adjustment for small feature regions. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of a digital camera to which a focus adjustment device according to an embodiment is applied; [Figure 2] Flowchart regarding the operation of the digital camera according to the embodiment [Figure 3] Flowchart regarding the operation of the digital camera according to the embodiment [Figure 4] FIG. 10 is a diagram for explaining an AF frame setting process in an embodiment. [Figure 5] Flowchart regarding the operation of the digital camera according to the embodiment [Figure 6] Flowchart regarding the operation of the digital camera according to the embodiment [Figure 7] Flowchart regarding the operation of the digital camera according to the embodiment [Figure 8] Schematic diagram of a histogram used in the embodiment [Figure 9] Flowchart regarding the operation of the digital camera according to the embodiment [Figure 10] FIG. 1 is a diagram showing an example of a prediction curve used in an embodiment. [Figure 11] FIG. 1 is a diagram for explaining erroneous tracking prevention control according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] The following description will be given of an embodiment in which the focus adjustment device according to the present invention is implemented in a digital camera. However, the present invention can be implemented in any electronic device having an imaging function. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), smartphones, smart watches, game consoles, robots, drones, drive recorders, etc. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0012] ●Digital camera configuration 1 is a block diagram showing an example of the functional configuration of a digital camera (hereinafter simply referred to as a camera) to which a focus adjustment device according to an embodiment of the present invention can be applied. The camera is composed of a lens device (interchangeable lens) 100 and a camera body 200. Here, an interchangeable lens camera is shown in which the lens device 100 is detachable from the camera body 200. However, the lens device 100 may also be fixed to the camera body 200.

[0013] The lens device 100 and the camera body 200 have mounts that are detachably engaged with each other. The lens device 100 is attached to the camera body 200 by engaging the mount of the lens device 100 with the mount of the camera body 200. The electrical contact unit 106 includes a group of electrical contacts that are provided on the mount of the lens device 100 and the mount of the camera body 200 so as to face each other.

[0014] When the lens device 100 is attached to the camera body 200, the electrical contacts of the electrical contact unit 106 come into contact, and power is supplied from the camera body 200 to the lens device 100. In addition, mutual communication between the lens device 100 (lens controller 105) and the camera body 200 (system control unit 209) becomes possible through the electrical contacts that have come into contact.

[0015] The lens device 100 includes a photographing lens 101, an aperture 102 that controls the amount of light, a focus lens 103, a motor 104 that drives the focus lens 103, and a lens controller 105. The photographing lens 101 and the focus lens 103 constitute a photographing optical system that forms a subject image on the imaging surface of an image sensor 201. In this embodiment, the aperture 102 also serves as a mechanical shutter. The photographing optical system may also include a lens that changes the focal length (angle of view) of the lens device 100, an image blur correction lens, etc.

[0016] The camera body 200 has an image sensor 201. The image sensor 201 may be, for example, a known CCD or CMOS color image sensor with a primary color Bayer array color filter. The image sensor 201 has a pixel array in which multiple pixels are arranged two-dimensionally, and peripheral circuits for reading out signals from each pixel. Each pixel accumulates charge according to the amount of incident light through photoelectric conversion. By reading out from each pixel a signal having a voltage according to the amount of charge accumulated during the exposure period, a group of pixel signals (analog image signals) representing the subject image formed on the imaging surface can be obtained.

[0017] In this embodiment, each pixel of the image sensor 201 has one microlens and a photoelectric conversion region divided into multiple regions. Signals can be selectively read out from each pixel from multiple photoelectric conversion regions. Here, the photoelectric conversion region is divided into two equal parts, and each photoelectric conversion region is designated as photodiodes (or sub-pixels) A and B. For multiple pixels included in any rectangular region of the pixel array, an analog image signal (signal A) read out from photodiode A and an analog image signal (signal B) read out from photodiode B form a parallax image pair.

[0018] The defocus amount of the focus detection area can be obtained using the A signal and B signal read from multiple pixels within a rectangular area set as the focus detection area. Therefore, the A signal and B signal are also called focus detection signals. On the other hand, the signal obtained by adding the A signal and the B signal for each pixel (A+B signal) can be treated in the same way as a signal obtained from a pixel whose photoelectric conversion area is not divided, so it is also called a captured image signal. Note that the A signal (or B signal) may also be generated by subtracting the B signal (or A signal) from the A+B signal.

[0019] The A / D conversion unit 202 has a circuit that applies preprocessing to the analog image signal obtained from the image sensor 201 and a circuit that A / D converts the signal to which the preprocessing has been applied. The preprocessing may be, for example, correlated double sampling (CDS) and nonlinear amplification. The A / D conversion unit 202 outputs digital data (image data) after A / D conversion for the A+B signal to the image processing unit 203. The A / D conversion unit 202 outputs digital data (image data) after A / D conversion for the A or B signal to the AF signal processing unit 204. Note that when the A or B signal is generated using the A+B signal, the A / D conversion unit 202 also outputs the A+B signal to the AF signal processing unit 204.

[0020] The AF signal processing unit 204 calculates the phase difference (image shift amount) between the A signal and the B signal. The AF signal processing unit 204 further calculates the defocus amount (and defocus direction and reliability) of the photographic optical system from the image shift amount. If multiple focus detection areas are set, the AF signal processing unit 204 performs these calculation processes for each focus detection area (AF frame).

[0021] The system control unit 209 is, for example, one or more processors (CPU, MPU, microprocessor, etc.) capable of executing programs. The system control unit 209 loads programs stored in the ROM 219 into the DRAM 206 and executes them, thereby controlling the operation of each unit of the camera body 200 and the lens apparatus 100 and realizing the functions of the digital camera.

[0022] The ROM 219 is a rewritable non-volatile memory that stores programs executed by the system control unit 209, various setting values ​​for the camera body 200, GUI data, and the like.

[0023] The image processing unit 203 applies predetermined image processing to the image data output by the A / D conversion unit 202, generates signals and image data according to the application, and acquires and / or generates various types of information. The image processing unit 203 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the image processing unit 203 may be configured such that a processor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) executes software to realize a specific function. The image processing unit 203 outputs the acquired or generated information and data to the system control unit 209, DRAM 206, or the like according to the application.

[0024] The image processing applied by the image processing unit 203 can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include reference level adjustment, defective pixel correction, etc. Color interpolation processing is performed when a color filter is provided on the image sensor 201, and is processing that interpolates the values ​​of color components that are not included in the individual pixel data that make up the image data. Color interpolation processing is also called demosaic processing. The correction processing can include white balance adjustment, tone correction, correction of image degradation caused by optical aberrations in the imaging optical system (image restoration), correction of the effects of vignetting in the imaging optical system, color correction, and the like. The detection process may include detection of a characteristic region (for example, a region of a specific subject) and its movement, person recognition processing, and the like. Data processing can include processes such as area extraction (trimming), compositing, scaling, encoding and decoding, header information generation (data file generation), etc. Data processing also includes the generation of image data for display or image data for recording. The evaluation value calculation process can include processes such as generating an evaluation value used for automatic exposure control (AE). Special effect processing can include adding a blur effect, changing color tones, relighting, and the like. Note that these are examples of processes that the image processing unit 203 can apply, and do not limit the processes that the image processing unit 203 can apply.

[0025] In this embodiment, the image processing unit 203 can detect regions of humans, animals (dogs, cats, birds, etc.), and vehicles (airplanes, trains, ships, automobiles, motorcycles, bicycles, etc.) as feature regions. The image processing unit 203 can detect feature regions using any known method. For example, the image processing unit 203 can detect feature regions by using a machine learning model trained according to the type of subject, or by using template matching using a template with a shape or pattern characteristic of the subject. These are merely examples, and other known methods may also be used. Data used to detect feature regions, such as trained machine learning models and templates, is assumed to be stored in advance in the ROM 219. Note that the image processing unit 203 detects face (head) regions, pupil regions, and torso regions for human and animal subjects. Furthermore, for automobile, motorcycle, and bicycle subjects, it detects the vehicle region and the head or helmet region of the occupant.

[0026] The image processing unit 203 stores, as detection results, identification information (ID), type of subject, position and size in the image, detection reliability, and the like for each feature region in the DRAM 206. The size of a feature region may be, for example, the horizontal and vertical size (number of pixels) of a rectangular region circumscribing the subject region. Furthermore, the image coordinates of one vertex of this rectangular region (for example, the upper left vertex) can be used as the position of the feature region in the image.

[0027] The DRAM 206 is used as a main memory for the system control unit 209, a buffer for temporarily storing captured image data, a work memory for temporarily storing data being processed by the image processing unit 203, and the like.

[0028] The VRAM 212 is a video memory that stores image data to be displayed on the image display unit 213 .

[0029] The image recording unit 207 has a recording medium such as a memory card and an interface circuit for accessing the recording medium. Image data to be recorded is recorded on the recording medium by the image recording unit 207. Note that the recording medium is not limited to removable media such as a memory card, and may be a recording medium built into the camera body 200.

[0030] A timing generator 208 supplies each unit of the camera body 200 with a clock signal that serves as a reference for operation timing.

[0031] The lens communication unit 210 supplies a synchronization signal to the lens device 100. In addition, the system control unit 209 communicates bidirectionally with the lens controller 105 over a communication bus established between the lens controller 105 and the lens communication unit 210.

[0032] The image display unit 213 is, for example, a color liquid crystal display (LCD) provided on the surface of the camera body 200. The image display unit 213 displays captured images, playback images, menu screens, information about the settings and status of the digital camera, and the like. In a shooting standby state, the image display unit 213 functions as an electronic viewfinder (EVF) by continuously shooting video, generating image data for display, and displaying the image on the image display unit 213. A series of operations for causing the image display unit 213 to function as an EVF is called a live view display operation, and the video displayed is called a live view image.

[0033] The shooting mode switch (SW) 215 is a switch for selecting one of a plurality of shooting modes selectable on the camera body 200. The shooting modes, such as night scene mode, sports mode, portrait mode, etc., are determined according to the type of scene or subject to be photographed. When a shooting mode is selected, multiple items such as aperture, shutter speed, sensitivity, autofocus operation mode, and image processing content are changed to settings according to the shooting mode.

[0034] The main SW216 is a power switch that turns the digital camera on and off.

[0035] SW1 217 and SW2 218 are switches that are turned on when the release button is pressed halfway and fully, respectively. The system control unit 209 recognizes that turning on SW1 is an instruction to prepare for still image shooting, and turning on SW2 is an instruction to start shooting a still image.

[0036] The operation unit 214 is a general term for all input devices (buttons, switches, dials, etc.) provided for the user to input various instructions to the camera body 200, excluding the switches 215 to 218 described above. The operation unit 214 includes a video recording switch, menu button, directional keys, an enter key, etc. The system control unit 209 recognizes the video recording switch pressed in shooting standby mode as an instruction to start video recording, and recognizes the switch pressed during video recording as an instruction to stop recording. The input device may be software buttons or keys using a touch display. The operation unit 214 may also include an input device compatible with non-contact input methods such as voice input or eye-gaze input.

[0037] Of the components of the camera body 200 described above, the image processing unit 203, the AF signal processing unit 204, and the system control unit 209 implement a focus adjustment device.

[0038] ●Digital camera operation Next, the operation of the digital camera when capturing a still image will be described using the flowchart shown in Fig. 2. This operation starts when the camera body 200 is powered on by the main switch 216 and enters a shooting standby state. In the shooting standby state, the system control unit 209 controls each unit of the digital camera to perform a live view display operation.

[0039] In S201, the system control unit 209 determines whether or not SW1 217 is on. If it is determined that SW1 217 is on, the system control unit 209 executes S202, and if it is not determined that SW1 217 is on, the system control unit 209 executes S201 repeatedly.

[0040] In S202, the system control unit 209 sets a focus detection area (AF frame) and notifies the AF signal processing unit 204. Details of the processing will be described later.

[0041] In S203, the system control unit 209 performs an automatic focus adjustment operation (AF operation) that drives the focus lens 103 based on the defocus information obtained from the AF signal processing unit 204. Details of the process will be described later. The system control unit 209 also determines the exposure conditions (aperture value, shutter speed, and shooting sensitivity) based on the evaluation value obtained from the image processing unit 203 and settings such as the exposure mode. The exposure conditions can be determined using any known method known as automatic exposure control (AE), and therefore detailed description thereof will be omitted.

[0042] In S204, the system control unit 209 determines whether SW2 218 is on, and if it is determined that SW2 218 is on, proceeds to S205, and if it is not determined that SW2 218 is on, repeats the processing from S201. Note that if it is not determined that SW2 218 is on, and SW1 217 remains on, S204 may be repeated, and if SW1 217 is turned off, repeats the processing from S201.

[0043] In S205, the system control unit 209 controls the operations of each unit to capture and record a still image. When the image recording unit 207 finishes recording the still image data, the system control unit 209 repeatedly executes the process from S201. Note that the system control unit 209 may start the process of S201 before the recording of the still image data finishes.

[0044] AF frame setting operation Next, the AF frame setting operation in S202 of FIG. 2 will be described in detail with reference to the flowchart shown in FIG.

[0045] In a shooting standby state, the image processing unit 203 generates display image data to be used for live view display operation, and also continuously executes subject detection processing on the display image data. The image processing unit 203 sequentially updates the data stored in the DRAM 206 so that the results of the most recent predetermined number of subject detection processing runs are stored in the DRAM 206. Here, as an example, it is assumed that the characteristic areas detected by the subject detection processing are the eyes, face, and torso of a human or animal.

[0046] In S301, the system control unit 209 acquires the results of the subject detection process from the DRAM 206. Here, it is assumed that an eye region A, a face region B, and a torso region C of the same human subject have been detected, as shown in Fig. 4(a). Here, since each part is detected individually, it is assumed that they are parts of the same subject, but if multiple parts of the same type are detected, it is possible to determine whether they are parts of the same subject based on, for example, their relationship with the detected positions of other parts.

[0047] In S302, the system control unit 209 determines region D that encompasses the entire detected pupil region A, face region B, and torso region C. Region D encompasses pupil region A, face region B, and torso region C of the same subject, and therefore corresponds to the subject region. Region D may be a rectangular region circumscribing pupil region A, face region B, and torso region C. As shown in FIG. 4(a), region D may be a region that is a slightly enlarged rectangular region circumscribing pupil region A, face region B, and torso region C, taking into account the movement of the subject and detection errors in feature regions.

[0048] In S303, the system control unit 209 sets the number of horizontal AF frames to WnH. In S304, the system control unit 209 sets the number of vertical AF frames to WnV. WnH and WnV are integers equal to or greater than 2, and the product of WnH and WnV, i.e., the total number of AF frames, is set to a value that exceeds a predetermined threshold (for example, several tens to several hundred). The threshold can be set so that sufficient accuracy can be obtained in identifying the subject using a histogram of defocus amounts, which will be described later with reference to FIG. 8.

[0049] In S305, the system control unit 209 sets the size of the AF frame to an initial value (reference value). Initial value of AF frame size (integer) = size of long side of area D × predetermined magnification A / number of AF frames Here, the number of AF frames is WnV when area D is vertically long, and WnH when area D is horizontally long. Also, the predetermined magnification A is a predetermined value equal to or greater than 1, and can take different values ​​depending on, for example, the AF operation mode (for example, single-shot AF or continuous AF).

[0050] In S306, the system control unit 209 determines whether a subject area (hereinafter, small subject) smaller than a predetermined size on the image has been detected. If it is determined that a subject area has been detected, S309 is executed; if not, S307 is executed. The system control unit 209 can determine that a small subject has been detected if, for example, the size of a region of a part corresponding to the type of subject included in region D is less than a threshold size. The system control unit 209 can determine that a small subject has been detected if, for example, the size of a face region in the case of a person subject, or the size of a rider's head or helmet region in the case of an automobile or motorcycle, is less than a threshold size. The threshold size may be the number of pixels or a percentage (%) of the entire screen taken as 100%. For example, the threshold size for a face or head region can be set to 3 to 4%. The threshold size may vary depending on the type of subject. Also, S306 may not be executed depending on the type of subject.

[0051] In S307, the system control unit 209 determines whether the entire AF frame is larger than a predetermined area E set within the screen, indicated by H and V in FIG. 4B. Here, area E is assumed to be a pre-defined area whose center coincides with the center of the screen, has the same aspect ratio as the screen, and occupies a predetermined percentage (e.g., approximately 70% to 80%) of the entire screen. This is just an example, and area E may be set according to other conditions. The entire AF frame is defined as an area in which WnV AF frames of the initial size set in S305 are arranged vertically and WnH AF frames of the initial size are arranged horizontally. If the system control unit 209 determines that the entire AF frame is larger than area E, it does not change the size of the AF frame (leaving it at the initial size) and ends the AF frame setting process. On the other hand, if the system control unit 209 does not determine that the entire AF frame is larger than area E, it executes S308. In S308, the system control unit 209 changes the size of the AF frame from the initial size to the first lower limit size MinA and ends the AF frame setting process.

[0052] The first lower limit size MinA can be determined as a value that allows AF frames to be arranged in the entire area of ​​a predetermined size that includes area D when WnH AF frames are arranged horizontally and WnV AF frames are arranged vertically. Here, the area of ​​the predetermined size is area E with a horizontal size H and a vertical size V.

[0053] For example, if AF frames are arranged with no gaps in the horizontal and vertical directions, the system control unit 209 can determine the larger of H / WnH and V / WnV as the first lower limit size MinA. If AF frames are arranged with gaps in the horizontal and vertical directions, the system control unit 209 can similarly determine the first lower limit size MinA using values ​​obtained by subtracting the sizes corresponding to the gaps from H and V, respectively.

[0054] In S309, the system control unit 209 determines whether or not moving object prediction is in progress, and if it is determined that moving object prediction is in progress, executes S307, and if not, executes S310. Moving object prediction is a process of predicting the distance to a subject (main subject) to be focused on. Details of moving object prediction will be described later. For example, if a history including the number of prediction results required for moving object prediction is saved, the system control unit 209 can determine that moving object prediction is in progress. Note that the determination may also be made based on other conditions.

[0055] In S310, the system control unit 209 determines whether or not this is the first AF operation. If it is determined to be the first AF operation, S313 is executed; if not, S311 is executed. The first AF operation is an AF operation performed on a different area than the previous AF operation. This corresponds to the AF operation performed when the subject detection processing results are used for the first time or when the feature area to be focused on is switched.

[0056] In S311, the system control unit 209 determines whether the entire AF frame is larger than a predetermined area F on the screen, which is indicated by H' and V' in FIG. 4C. If the system control unit 209 determines that the entire AF frame is larger than area F, it ends the AF frame setting process without changing the size of the AF frame (leaving it at its initial size). On the other hand, if the system control unit 209 does not determine that the entire AF frame is larger than area F, it executes S312.

[0057] In S312, the system control unit 209 changes the size of the AF frame from the initial size to the second lower limit size MiNB, and ends the AF frame setting process.

[0058] The second lower limit size MinB can be determined as a value that allows AF frames to be arranged in the entire area of ​​a predetermined size that includes area D when WnH AF frames are arranged horizontally and WnV AF frames are arranged vertically. Here, the area of ​​the predetermined size is area F with a horizontal size H' and a vertical size V'.

[0059] Here, S311 is executed when it is determined that a small subject has been detected. Therefore, the horizontal size H' and the vertical size V' of the region with a predetermined size are made smaller than the horizontal size H and the vertical size V of the region E used when determining the first lower limit size MinA of the AF frame. That is, H' < H and V' < V, and the region F is smaller than the region E.

[0060] In addition, in the case of a shooting scene where there are a plurality of small subjects within the screen, the region F is set to a size such that the frequency distribution of the defocus amount can be obtained for the plurality of small subjects, in order to prevent the subject to be focused on from being accidentally switched from the intended main subject to another subject. The process of suppressing an unintended switch of the main subject will be described later.

[0061] The system control unit 209 can determine the second lower limit size MinB in the same manner as the first lower limit size MinA. Therefore, the second lower limit size MinB is smaller than the first lower limit size MinA. However, it is determined so as to ensure the detection accuracy of the deviation amount between the A signal and the B signal. When AF frames of the determined sizes are arranged in the horizontal and vertical directions without gaps and overlaps, and there is an AF frame that falls outside the region of a predetermined size, it may be allowed that adjacent AF frames partially overlap so that there is no AF frame that falls outside the region of the predetermined size.

[0062] In S313, the system control unit 209 changes the size of the AF frame from the initial size to the third lower limit size MinC and ends the AF frame setting process.

[0063] As shown in FIG. 4(d), the third lower limit size MinC of the AF frame can be determined as a value such that WnH AF frames can be arranged in the horizontal direction and WnV AF frames can be arranged in the vertical direction with respect to the entire region D (subject region). The horizontal size and the vertical size of the region D are smaller than the horizontal size H' and the vertical size V' of the region F used when determining the second lower limit size MinB of the AF frame.

[0064] The system control unit 209 can determine the third lower limit size MinC in the same way as the first and second lower limit sizes MinA and MinB. Therefore, the third lower limit size MinC is equal to or smaller than the second lower limit size MinB. However, the third lower limit size MinC is determined so as to ensure the accuracy of detecting the amount of deviation between the A and B signals. Note that if AF frames of a determined size are arranged horizontally and vertically without gaps or overlaps and some AF frames fall outside the area of ​​a predetermined size, adjacent AF frames may be allowed to partially overlap so that no AF frames fall outside the area of ​​the predetermined size.

[0065] The system control unit 209 determines the position of each AF frame based on the size and arrangement range of the AF frame determined in S305, S308, S312, or S313, and then notifies the AF signal processing unit 204 of the size and position of each AF frame.

[0066] In this way, when a small subject is detected, the area in which the AF frame is placed is narrower than when a small subject is not detected, and the size of the AF frame is also reduced. Furthermore, when a small subject is detected and the first AF operation is performed, the area in which the AF frame is placed is narrower than when the second or subsequent AF operations are performed, and the size of the AF frame is also reduced.

[0067] Therefore, when a small subject is detected, the detection density of the defocus amount is higher than when a small subject is not detected, and the accuracy of separating the subject from the background based on the distribution of the defocus amount can be improved. Furthermore, when a small subject is detected, the AF frame is less likely to include the background when the first AF operation is performed, allowing for highly accurate focus adjustment. When performing AF operations from the second time onwards, the AF frame placement range is expanded, allowing for continued appropriate focus adjustment even if the subject moves.

[0068] According to this embodiment, the size of the range in which the AF frames are arranged is changed, so that the focus adjustment accuracy for small subjects can be improved without increasing the total number of AF frames (i.e., without increasing the calculation load).

[0069] ●AF operation Next, the AF operation in S203 of FIG. 2 will be described in detail with reference to the flowchart shown in FIG.

[0070] In step S401, the AF signal processing unit 204 acquires the defocus amount and its reliability for each AF frame. Details of the operation will be described later.

[0071] In S402, the system control unit 209 selects an AF frame (called the AF main frame) to be used for focus adjustment based on the defocus amount acquired in S401 by the AF signal processing unit 204. Details of the operation will be described later.

[0072] In S403, the system control unit 209 saves the history of the defocus amount of the AF main frame, including the defocus amount of the AF main frame selected in S402.

[0073] In step S404, the system control unit 209 performs moving object prediction using the history of defocus amounts. Details of this operation will be described later.

[0074] In S405, the system control unit 209 drives the focus lens 103 so as to focus on the distance according to the prediction result in S404.

[0075] Defocus amount acquisition process Next, the defocus amount acquisition process in S401 of FIG. 5 will be described in detail with reference to the flowchart shown in FIG.

[0076] In S501, the AF signal processing unit 204 sets each AF frame arranged by the AF frame setting process as an area within the image. For example, when detecting the defocus amount using a live view image, the AF signal processing unit 204 sets each AF frame as an area within the live view image. It is assumed that the image sensor 201 reads out an A signal and a B signal, or one of the A signal and the B signal and an A+B signal.

[0077] In S502, the AF signal processing unit 204 generates, for each horizontal line, a waveform (image A) obtained by connecting signal group A and a waveform (image B) obtained by connecting signal group B for pixels included in each of the set AF frames, thereby generating multiple pairs of images A and B.

[0078] In S503, the AF signal processing unit 204 converts multiple images A into a single image A, for example by averaging them. The AF signal processing unit 204 converts the images B into a single image B in the same way. This makes it possible to suppress the effects of noise in the images A and B. In this way, the AF signal processing unit 204 obtains a pair of images A and B for each AF frame.

[0079] In S504, the AF signal processing unit 204 applies filtering to the pair of images A and B obtained in S503 to extract signal components in a predetermined frequency band.

[0080] In S505, the AF signal processing unit 204 calculates the correlation value for the filtered A and B images by a known method while shifting the relative positions of the A and B images.

[0081] In S506, the AF signal processing unit 204 calculates the amount of change in the correlation value due to the relative positions of the A and B images.

[0082] In S507, the AF signal processing unit 204 calculates the amount of image shift at which the correlation value between the image A and the image B becomes maximum, based on the amount of change in the correlation value.

[0083] In S508, the AF signal processing unit 204 calculates the reliability of the calculated image shift amount using any known method.

[0084] In step S509, the AF signal processing unit 204 converts the image shift amount into a defocus amount using any known method. The sign of the defocus amount indicates the defocus direction.

[0085] The AF signal processing unit 204 executes the processes of S504 to S509 for each AF frame. The AF signal processing unit 204 stores the defocus amount and the reliability of the image shift amount as the reliability of the defocus amount in, for example, the DRAM 206, and ends the defocus amount acquisition process.

[0086] AF main frame selection processing Next, the AF main frame selection process in S402 of FIG. 5 will be described in detail with reference to the flowchart shown in FIG.

[0087] In S601, the system control unit 209 determines whether or not to use a histogram of the defocus amount (subject distance) to select the AF main frame. If it is determined that a histogram should be used, S603 is executed; if not, S602 is executed. For example, if it is determined that a small subject has been detected in S303, the system control unit 209 can determine that a histogram should be used. Note that a histogram may always be used, in which case S601 and S602 are unnecessary.

[0088] In S602, the system control unit 209 performs normal main frame selection (without using a defocus amount histogram). For example, the system control unit 209 selects an AF frame with a high reliability of the defocus amount obtained in S508 that is closest to the center of the subject area as the main AF frame. If the subject distance is predicted by moving object prediction, the system control unit 209 may select the main AF frame based on the predicted subject distance. For example, the system control unit 209 compares the subject distance corresponding to the defocus amount of the AF frame closest to the center of the subject area with the predicted subject distance. If the difference in subject distance is equal to or greater than a threshold, the system control unit 209 selects an AF frame within the subject area that has a defocus amount corresponding to the subject distance closest to the predicted subject distance as the main AF frame.

[0089] In S603, the system control unit 209 instructs the image processing unit 203 to generate a histogram of the defocus amounts obtained in S401. The image processing unit 203 sets the conditions for generating the histogram (the number of bins and the range of subject distances corresponding to each bin). The method for dividing the bins can be determined in advance, for example. Furthermore, the width of the bins does not need to be constant.

[0090] In S604, the image processing unit 203 converts the defocus amount for each AF frame into a subject distance and generates a histogram.

[0091] 8 is a schematic diagram showing an example of a photographic scene and a histogram generated by the image processing unit 203 from the defocus amount obtained for the photographic scene. The image processing unit 203 generates the histogram shown in the bottom row based on the defocus amount detected for each AF frame by the AF signal processing unit 204 from an image captured of the photographic scene shown in the top row. The histogram represents the frequency distribution of the defocus amount for each subject distance range (or the frequency distribution of the AF frame for each defocus amount range). The image processing unit 203 generates histogram data that associates bin numbers with frequencies and stores the data in the DRAM 206.

[0092] In S605, the system control unit 209 selects, based on the histogram data, the bin with the smallest (closest) range of corresponding subject distances from among the bins having a frequency exceeding a predetermined frequency threshold.

[0093] In S606, the system control unit 209 determines whether a bin was selected in S605, and if it is determined that a bin was selected, it executes S607, and if not, it executes S608. A bin cannot be selected in S605 if, for example, there is no bin with a frequency exceeding the frequency threshold.

[0094] In S607, the system control unit 209 selects one AF frame (e.g., the AF frame closest to the center) from among the AF frames classified into the selected bin, whose distance from the center of the subject area (area D in Figure 4(a)) is less than a threshold value, as the main AF frame.

[0095] In S608, the system control unit 209 determines whether or not a moving object is being predicted. If it is determined that a moving object is being predicted, S609 is executed, and if not, S610 is executed.

[0096] In S609, the system control unit 209 selects, as the primary AF frame, one AF frame that has a defocus amount corresponding to a subject distance that differs from the subject distance predicted by moving object prediction by less than a threshold value from among the AF frames that exist within the subject area. For example, the system control unit 209 can select, as the primary AF frame, one AF frame that has a subject distance corresponding to a defocus amount that is closest to the subject distance predicted by moving object prediction from among the AF frames that exist within the subject area.

[0097] In S610, the system control unit 209 selects, from all AF frames, the AF frame with the shortest subject distance corresponding to the defocus amount as the main AF frame. Note that if a lower limit for the subject distance is set as a histogram generation condition, the system control unit 209 selects, as the main AF frame, the AF frame with the shortest subject distance corresponding to the defocus amount that is equal to or greater than the lower limit.

[0098] If a bin can be selected, it is assumed that a sufficient number of AF frames are positioned in the subject area, and the AF main frame is selected based on its position within the subject area. On the other hand, if a bin cannot be selected, it is assumed that the number of AF frames positioned in the subject area is insufficient, or that a large proportion of AF frames include the background. Therefore, if moving object prediction is in progress, an AF frame with a defocus amount corresponding to a subject distance close to the predicted subject distance is selected from among the AF frames in the subject area. This makes it possible to select an AF frame that is less likely to be affected by the background. Furthermore, if moving object prediction is not in progress, an AF frame with a defocus amount corresponding to a subject close to the camera is selected. This is because, in general photography, subjects close to the camera are often the subject intended by the user.

[0099] If a bin can be selected, it is assumed that a sufficient number of AF frames are positioned in the subject area, and the AF main frame is selected based on its position within the subject area. On the other hand, if a bin cannot be selected, it is assumed that the number of AF frames positioned in the subject area is insufficient, or that a large proportion of AF frames include the background. Therefore, if moving object prediction is in progress, an AF frame with a defocus amount corresponding to a subject distance close to the predicted subject distance is selected from among the AF frames in the subject area. This makes it possible to select an AF frame that is less likely to be affected by the background. Furthermore, if moving object prediction is not in progress, an AF frame with a defocus amount corresponding to a subject close to the camera is selected. This is because, in general photography, subjects close to the camera are often the subject intended by the user.

[0100] Up to this point, we have explained the shooting scene with a single subject. However, there are also shooting scenes in which multiple subject areas of similar size are detected. For example, in track and field sports, a scene in which multiple athletes are close to each other may be captured.

[0101] 11 is a schematic diagram showing an example of subject area detection in a soccer scene where three players are close together. Here, the three players are detected as small subjects, and in the AF frame setting process, an AF frame targeting the small subject is set in S313 during the first AF operation, and then multiple AF frames are set in area F in S312.

[0102] For example, suppose that an eye area A, a face area B, and a body area C are detected for each of the main subject and sub-subject. If one of the sub-subjects crosses in front of the main subject during tracking photography that continuously focuses on the main subject, the sub-subject may be mistakenly recognized as the main subject, resulting in an unintended change in the subject being tracked.

[0103] According to this embodiment, it is possible to prevent such unintentional switching of the subject to be tracked. First, the system control unit 209 obtains the inclusion regions described with reference to FIG.

[0104] Next, the system control unit 209 determines a representative AF frame for each inclusion area. The representative AF frame may be, for example, the AF frame closest to the center of the inclusion area, or may be selected from the AF frames classified into the bin with the highest frequency in the defocus amount histogram of the AF frames within the inclusion area. Alternatively, for simplicity, the representative AF frame may be the AF frame with the defocus amount closest to the median value of the defocus amount of the inclusion area.

[0105] The system control unit 209 then calculates the difference in subject distance from the defocus amount of the representative AF frame determined for each subject. If the difference in subject distance is equal to or greater than a threshold, the system control unit 209 excludes the area of ​​the sub-subject from the tracking target. In other words, the system control unit 209 uses the difference in subject distance to search for an area to track within the tracking candidate area, excluding clearly different people. This makes it possible to prevent a sub-subject at a different distance from being mistakenly recognized as the main subject, even when shooting at a time when the main subject is obscured by a sub-subject in the foreground. This makes it possible to prevent erroneous switching of the AF main frame.

[0106] In this embodiment, when it is determined that a small subject has been detected, a smaller AF frame is placed in a narrower range than when it has not been determined. This improves the accuracy of separating the subject from the background using the defocus amount histogram compared to when the AF frame placement range and size are not changed. As a result, it is possible to improve the likelihood of selecting a bin in S605 when a small subject is detected, i.e., the likelihood of selecting an appropriate AF frame within the subject area. As a result, it is possible to properly track the main subject even in scenes where multiple small subjects are close together or where the main subject is temporarily hidden by a secondary subject among multiple small subjects.

[0107] ●Motion prediction processing Next, details of the moving object prediction process in S404 of FIG. 5 will be described with reference to the flowchart shown in FIG.

[0108] In S801, the system control unit 209 determines whether a history including a predetermined number Num_a or more of moving object prediction results is stored in the DRAM 206, and if it is determined that the history is stored, executes S802, otherwise executes S805. The predetermined number Num_a is an integer equal to or greater than 2, and can be determined in advance by experiment or the like.

[0109] In S802, the system control unit 209 determines whether to perform moving object prediction. If it is determined that moving object prediction should be performed, the system control unit 209 executes S803; if not, the system control unit 209 executes S805. Here, as an example, the system control unit 209 determines to perform moving object prediction if the number of times that the amount of change per predetermined unit time exceeds a threshold is equal to or greater than a predetermined number of times among the stored prediction results (subject distance). If the condition is not satisfied, it is considered that there is little change in the distance of the main subject, and therefore there is little need for prediction, or that prediction based on history is not suitable. For example, if moving object prediction is performed periodically, the system control unit 209 can determine to perform moving object prediction if the history contains a predetermined number or more prediction results whose difference from the most recent prediction result is equal to or greater than a threshold.

[0110] In S803, the system control unit 209 generates a prediction curve for predicting future subject distances based on the history of prediction results. The system control unit 209 can generate the prediction curve by applying any known method (such as the least squares method or polynomial interpolation) for approximating a curve passing through multiple points to the execution time of moving object prediction and the prediction result.

[0111] In S804, the system control unit 209 uses the generated prediction curve to predict the subject distance at the timing of performing the next AF operation. Then, the system control unit 209 sets the drive target position of the focus lens 103 to a position corresponding to the predicted subject distance. In other words, the system control unit 209 determines the drive amount and drive direction of the focus lens 103 based on the current position of the focus lens 103.

[0112] In S805, the system control unit 209 determines the driving amount and driving direction of the focus lens 103 based on the defocus amount detected in the AF main frame.

[0113] ●Prediction curve calculation process 10 is a diagram showing an example of a prediction curve generated based on the history of moving object prediction results saved in S403. The vertical axis represents the subject distance to the main subject area where the main AF frame is set, obtained as a result of moving object prediction, and the horizontal axis represents time.

[0114] Each of times T1 to T5 is a time when focus adjustment processing (driving of focus lens 103) is performed. The subject distances at times T1 to T4 are a history of predicted results, and the next focus adjustment processing is performed at time T5.

[0115] The prediction curve shown in FIG. 10 indicates that the main subject is approaching the camera. System control unit 209 sets the target position of focus lens 103 based on the difference between the most recent moving object prediction result and the subject distance at the next focus adjustment execution time predicted using the prediction curve. In the example shown in FIG. 10, system control unit 209 sets the target position of focus lens 103 at time T4 to a position that focuses on distance d1 based on the difference x between the prediction result for time T3 and the prediction result for time T4. Similarly, system control unit 209 sets the target position of focus lens 103 at time T5 to a position that focuses on distance d2 based on the difference y between the prediction result for time T4 and the prediction result for time T5.

[0116] As described above, according to this embodiment, the range in which the AF frame is placed is dynamically changed depending on the size of the detected subject area. Specifically, when a small subject is detected, the range in which the AF frame is placed is narrower than when a non-small (normal) subject is not detected. Furthermore, the size of the AF frame can also be made smaller when a small subject is detected than when a non-small (normal) subject is not detected.

[0117] By arranging the AF frames in this way, it is possible to obtain a defocus amount distribution over a range appropriate for the size of the subject without increasing the total number of AF frames. Therefore, even if the subject is small, it is possible to accurately separate the subject from the background based on the defocus amount distribution, and as a result, it is possible to accurately adjust the focus on the intended subject.

[0118] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0119] The disclosure of the present embodiment includes the following focus adjustment device, control method thereof, electronic device, and program. (Item 1) A focus adjustment device that adjusts the focus of an imaging optical system so that a subject area detected from a captured image is in focus, a setting means for setting a plurality of regions in an image for which a defocus amount is to be calculated; a selection means for selecting an area to be used for focus adjustment from among the plurality of areas; an adjustment unit that performs focus adjustment of the photographing optical system based on the defocus amount calculated for the area used for focus adjustment, a setting unit that sets the plurality of regions in a range that includes the subject region, and when the subject region is smaller than a predetermined size, narrows the range more than when the subject region is not small. (Item 2) The focus adjustment device according to item 1, characterized in that when the subject area is smaller than a predetermined size, the setting means narrows the range when performing a first focus adjustment on the subject area compared to when performing a second focus adjustment. (Item 3) 3. The focus adjustment device according to item 2, wherein the setting means sets the plurality of regions within the subject region when performing the first focus adjustment. (Item 4) 4. The focus adjustment device according to any one of items 1 to 3, wherein the setting means sets the plurality of regions by allowing partial overlap of adjacent regions when the subject region is smaller than a predetermined size. (Item 5) the subject region is a region that includes regions of different parts, 5. The focus adjustment device according to any one of items 1 to 4, wherein the setting means determines the sizes of the plurality of regions based on the size of the region of the part. (Item 6) 6. The focus adjustment device according to any one of items 1 to 5, wherein the selection means selects an area to be used for the focus adjustment from the plurality of areas based on the distribution of defocus amounts calculated for each of the plurality of areas. (Item 7) 7. The focus adjustment device according to any one of items 1 to 6, wherein the selection unit selects an area to be used for the focus adjustment from among the plurality of areas, an area whose object distance range is classified as the closest bin among bins that exceed a frequency threshold in a histogram of object distances corresponding to the defocus amount. (Item 8) 8. The focus adjustment device according to any one of items 1 to 7, wherein, when there is no bin exceeding a frequency threshold in the histogram of the object distance corresponding to the defocus amount among the plurality of areas, if the object distance is being predicted, the selection means selects an area corresponding to an object distance whose difference from the predicted object distance is less than a threshold as the area to be used for the focus adjustment. (Item 9) Item 9. The focus adjustment device according to any one of items 1 to 8, wherein the selection means selects, among the plurality of regions, a region corresponding to a shortest object distance as the region to be used for the focus adjustment if there is no bin exceeding a frequency threshold in a histogram of object distances corresponding to the defocus amount and if the object distance is not being predicted. (Item 10) An imaging element; a detection means for detecting a subject area from an image captured using the image sensor; A focus adjustment device according to any one of items 1 to 9, An electronic device comprising: (Item 11) A control method executed by a focus adjustment device that adjusts the focus of an imaging optical system so that a subject area detected from a captured image is in focus, comprising: setting a plurality of regions in an image for calculating a defocus amount; selecting an area to be used for focus adjustment from among the plurality of areas; and performing focus adjustment of the photographing optical system based on the defocus amount calculated for the area used for focus adjustment, a control method for a focus adjustment device, wherein the setting includes setting the plurality of regions in a range that includes the subject region, and when the subject region is smaller than a predetermined size, the range is narrower than when the subject region is not small. (Item 12) A program for causing a computer to function as each of the means included in the focus adjustment device according to any one of items 1 to 9.

[0120] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention. [Explanation of symbols]

[0121] 100... lens device, 103... focus lens, 105... lens controller, 200... camera body, 201... image sensor, 204... AF signal processing unit, 209... system control unit, 210... lens communication unit, 211... subject detection unit

Claims

1. A focus adjustment device that adjusts the focus of an imaging optical system so that a subject area detected from a captured image is in focus, a setting means for setting a plurality of regions in an image for which a defocus amount is to be calculated; a selection means for selecting an area to be used for focus adjustment from among the plurality of areas; an adjustment unit that performs focus adjustment of the photographing optical system based on the defocus amount calculated for the area used for focus adjustment, a setting unit that sets the plurality of regions in a range that includes the subject region, and when the subject region is smaller than a predetermined size, narrows the range more than when the subject region is not small.

2. 2. The focus adjustment device according to claim 1, wherein when the subject area is smaller than a predetermined size, the setting means narrows the range when performing a first focus adjustment on the subject area compared to when performing a second focus adjustment.

3. 3. The focus adjustment device according to claim 2, wherein the setting unit sets the plurality of regions within the subject region when performing the first focus adjustment.

4. 2. The focus adjustment device according to claim 1, wherein the setting means sets the plurality of regions by allowing partial overlap of adjacent regions when the subject region is smaller than a predetermined size.

5. the subject region is a region that includes regions of different parts, 2. The focus adjustment device according to claim 1, wherein the setting means determines the sizes of the plurality of regions based on the size of the region of the part.

6. 2. The focus adjustment device according to claim 1, wherein the selection unit selects an area to be used for the focus adjustment from the plurality of areas based on a distribution of defocus amounts calculated for each of the plurality of areas.

7. 2. The focus adjustment device according to claim 1, wherein the selection unit selects, from among the plurality of regions, a region whose subject distance range is classified as the shortest bin among bins that exceed a frequency threshold in a histogram of subject distances corresponding to the defocus amount, to be used for the focus adjustment.

8. 2. The focus adjustment device according to claim 1, wherein, when there is no bin exceeding a frequency threshold in a histogram of object distances corresponding to the defocus amount among the plurality of regions, if the selection unit is in the process of predicting an object distance, the selection unit selects, as the region to be used for the focus adjustment, a region corresponding to an object distance whose difference from the predicted object distance is less than a threshold.

9. 2. The focus adjustment device according to claim 1, wherein the selection unit selects, among the plurality of regions, a region corresponding to a shortest object distance as the region to be used for the focus adjustment when there is no bin exceeding a frequency threshold in a histogram of object distances corresponding to the defocus amount and when the object distance is not being predicted.

10. An imaging element; a detection means for detecting a subject area from an image captured using the image sensor; A focus adjustment device according to any one of claims 1 to 9; An electronic device comprising:

11. A control method executed by a focus adjustment device that adjusts the focus of an imaging optical system so that a subject area detected from a captured image is in focus, comprising: setting a plurality of regions in an image for calculating a defocus amount; selecting an area to be used for focus adjustment from among the plurality of areas; and performing focus adjustment of the photographing optical system based on the defocus amount calculated for the area used for focus adjustment, a control method for a focus adjustment device, wherein the setting includes setting the plurality of regions in a range that includes the subject region, and when the subject region is smaller than a predetermined size, the range is narrower than when the subject region is not small.

12. A program for causing a computer to function as each of the means included in the focus adjustment device according to any one of claims 1 to 9.

Images