Focus adjustment device and method, imaging apparatus, program, and storage medium

Abstract

To provide an imaging apparatus that can focus on a subject at a timing appropriate for photographing.SOLUTION: A focus adjustment device includes: a subject detection section that detects a subject from an image; a state detection section that detects a specific state of the subject from the image; a focus detection section that acquires a history of a focus detection result corresponding to the subject detected by the subject detection section; a focus adjustment section that adjust a focus using the focus detection result; a control section that performs acquisition of the history of the focus detection result of the subject by the focus detection section in parallel with detection processing of the specific state by the state detection section; and a prediction section that predicts, when the focus adjustment section shifts from a state of focusing on a first subject to a state of focusing on a second subject of which specific state has been detected, a focus adjustment position for the focus adjustment section to focus on the second subject using a history of a focus detection result of the second subject.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a focus adjustment device and method for use in an imaging device. [Background technology]

[0002] In recent years, various AF (autofocus) methods have been put to practical use, such as image-sensor phase-difference AF and contrast AF, which use image sensors. Furthermore, in these AF methods, techniques for identifying and focusing on the area of ​​the main subject are known.

[0003] Patent Document 1 discloses a technology that detects changes in the size and position of a feature area within a photographic screen based on multiple images generated in time series, estimates the direction of movement of a subject, and performs a focusing operation based on the estimation result.

[0004] Furthermore, Patent Document 2 discloses an imaging device equipped with a capture determination means that performs focus adjustment when the amount of change in image plane position during focus detection is less than a preset threshold. When predetermined conditions are met and the subject and imaging device are stationary, the threshold of the capture determination means is set to a value greater than the preset value, thereby improving focus tracking accuracy. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-160991 [Patent Document 2] Japanese Patent Application Laid-Open No. 2012-203207 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in Patent Document 1, the motion of the subject is determined based only on the size, position, and focus detection information of the subject, and it is not possible to determine whether the subject is in a desirable state as a main subject.

[0007] In Patent Document 2, focus tracking can be improved only at the start of AF, but it cannot be applied to focus tracking on a subject that is preferable as the main subject after AF has started.

[0008] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an imaging device that can focus on a subject at an appropriate timing for photographing. [Means for solving the problem]

[0009] A focus adjustment device according to the present invention comprises: subject detection means for detecting a subject from an image; state detection means for detecting a specific state of the subject from the image; focus detection means for acquiring a history of focus detection results corresponding to the subject detected by the subject detection means; focus adjustment means for adjusting focus using the focus detection results; control means for acquiring the history of focus detection results of the subject by the focus detection means in parallel with the detection process of the specific state by the state detection means; and prediction means for, when the focus adjustment means transitions from a state in which it is focused on a first subject to a state in which it is focused on a second subject for which the specific state has been detected, predicting a focus adjustment position for the focus adjustment means to focus on the second subject using the history of focus detection results of the second subject. [Effects of the Invention]

[0010] According to the present invention, it is possible to focus on a subject at an appropriate timing for photographing. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing the configuration of an imaging apparatus according to an embodiment of the present invention; [Figure 2] 4 is a flowchart illustrating the operation of the imaging apparatus. [Figure 3] 10 is a flowchart illustrating an operation of setting an AF frame. [Figure 4A] Conceptual diagram of detection areas for a person's eyes, face, and body. [Figure 4B] Conceptual diagram of AF frame settings for a person's eyes, face, and body. [Figure 5] 10 is a flowchart illustrating an AF operation. [Figure 6] 10 is a flowchart illustrating focus detection processing. [Figure 7] 10 is a flowchart illustrating an AF main frame selection process. [Figure 8A] 10 is a flowchart showing a motion history saving process. [Figure 8B] Conceptual diagram of the subject frame when saving the motion history. [Figure 9] 10 is a flowchart showing the operation of moving object prediction. [Figure 10] Conceptual diagram of a prediction curve. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0013] <Configuration of imaging device> 1 is a block diagram showing the configuration of an interchangeable lens camera 1 (hereinafter simply referred to as a camera) that is an embodiment of the imaging device of the present invention. The camera 1 of this embodiment is capable of performing focus adjustment by an imaging surface phase difference detection method using an output signal from an imaging element 201 that captures an image of a subject.

[0014] 1, camera 1 is composed of a lens device (interchangeable lens) 100 and a camera body 200. When lens device 100 is attached to camera body 200 via a mount unit (not shown) having an electrical contact unit 106, a lens controller 105 that controls the overall operation of lens device 100 and a system control unit 209 that controls the overall operation of the camera become able to communicate with each other.

[0015] First, we will explain the configuration of the lens device 100. The lens device 100 includes a photographing lens 101 with a zoom mechanism, an aperture and shutter 102 that controls the amount of light, a focus lens 103 for focusing on an image sensor 201, a motor 104 that drives the focus lens, and a lens controller 105.

[0016] Next, the configuration of the camera body 200 will be described. The camera body 200 is configured to acquire an imaging signal from a light beam that passes through the imaging optical system of the lens apparatus 100. The camera body 200 includes an image sensor 201 that photoelectrically converts reflected light from a subject into an electrical signal, an A / D conversion unit 202 that includes a CDS circuit that removes output noise from the image sensor 201 and a nonlinear amplifier circuit that performs A / D conversion before the A / D conversion, an image processing unit 203, and an AF signal processing unit 204. The camera body 200 also includes a format conversion unit 205, a high-speed internal memory (e.g., random access memory, hereinafter referred to as DRAM) 206, and an image recording unit 207 that includes a recording medium such as a memory card and its interface. The camera body 200 also includes a timing generator 208, a system control unit 209 that controls the system, such as the shooting sequence, a lens communication unit 210 that communicates between the camera body 200 and the lens apparatus 100, an object detection unit 211 that detects an object from an image, and an image display memory (hereinafter referred to as VRAM) 212.

[0017] The camera body 200 also has an image display unit 213 that displays images, operational assistance, and the camera status, and also displays the shooting screen and focus detection area during shooting. It also has an operation unit 214 for externally operating the camera 1, a shooting mode switch 215 for selecting a shooting mode such as macro mode or sports mode, and a main switch 216 for powering on the camera 1.

[0018] It also has a switch (hereinafter referred to as SW1) 217 ​​for performing shooting standby operations such as AF (autofocus) and AE (autoexposure control), and a shooting switch (hereinafter referred to as SW2) 218 ​​for shooting after operating switch SW1. DRAM 206, which is an internal memory, is used as temporary image storage means, i.e., a high-speed buffer, or as working memory when processing images. The operation unit 214 includes, for example, the following: a menu switch for setting various settings such as the shooting functions of the imaging device and settings for image playback, and an operation mode changeover switch between shooting mode and playback mode.

[0019] The image sensor 201 is configured with a CCD or CMOS sensor. Each pixel of the image sensor 201 used in this embodiment is configured with two (a pair) of photodiodes A and B and one microlens provided for the pair of photodiodes A and B. Each pixel splits incident light with the microlens to form a pair of optical images on the pair of photodiodes A and B. The pair of photodiodes A and B then outputs a pair of pixel signals (signals A and B) used as an AF (autofocus) signal, which will be described later. In addition, an image signal (signal A+B) can be obtained by adding the outputs of the pair of photodiodes A and B.

[0020] By combining multiple A signals and multiple B signals output from multiple pixels, a pair of image signals (A image signal, B image signal) is obtained as an AF signal (focus detection signal) used for AF using the image plane phase difference detection method (hereinafter referred to as image plane phase difference AF).

[0021] The AF signal processing unit 204 performs correlation calculations on the pair of image signals to calculate a phase difference (hereinafter referred to as image shift amount) that is the amount of shift between the pair of image signals, and further calculates the defocus amount (and defocus direction and reliability) of the photographing optical system from the image shift amount. Note that the AF signal processing unit 204 calculates multiple defocus amounts in a specified area that can be specified.

[0022] <Operation of the imaging device> The operation of the imaging device of this embodiment will be described below with reference to Fig. 2. Fig. 2 is a flowchart showing the flow of imaging control processing when capturing a still image from a state in which a live view image is displayed. A system control unit 209 as a computer executes this processing in accordance with a control program as a computer program.

[0023] First, in step S201, the system control unit 209 checks the state of the switch SW1 217. If the switch SW1 is ON, the system control unit 209 proceeds to step S202, and if not, it waits.

[0024] In step S202, the system control unit 209 performs AF frame setting for the AF signal processing unit 204. Details of the AF frame setting will be described later.

[0025] In step S203, the system control unit 209 performs an AF operation, the details of which will be described later.

[0026] In step S204, the system control unit 209 checks the state of the switch SW1 217. If the switch SW1 is ON, the system control unit 209 proceeds to step S205, and if not, returns the process to step S201.

[0027] In step S205, the system control unit 209 checks the state of switch SW2 (218). If switch SW2 is ON, the system control unit 209 proceeds to step S206; otherwise, it returns the process to step S204.

[0028] In step S206, the system control unit 209 executes a shooting operation and returns the process to step S201.

[0029] <AF Frame Setting> FIG. 3 is a flowchart for explaining the operation of AF frame setting in step S202 of FIG. 2.

[0030] First, in step S301, the system control unit 209 acquires subject detection information from the subject detection unit 211. The subject in this embodiment is a person, and furthermore, the main area within the subject is to be detected. Here, the main area refers to the pupils, face, and body of a person or animal. For these detection methods, known learning methods based on machine learning or recognition processing by image processing means are used.

[0031] For example, the types of machine learning are as follows. (1) Support Vector Machine (2) Convolutional Neural Network (3) Recurrent Neural Network Also, as an example of recognition processing, a method of extracting a skin color area from the gradation color of each pixel represented by image data and detecting a face based on the degree of matching with a face contour plate prepared in advance is known. Also, a method of performing face detection by extracting feature points of a face such as eyes, nose, mouth, etc. using well-known pattern recognition technology is also well-known.

[0032] These techniques can also be applied to estimate the posture of a subject from features such as the joints of the human body, and detect important states (hereafter referred to as actions) of the subject in sports scenes. This makes it possible to detect, for example, a shooting posture in soccer or basketball, or an attacking posture in volleyball. This action detection method is used in the action detection process (state detection process) in step S800 of FIG. 9, which will be described later.

[0033] The subject detection unit 211 can detect multiple subjects, and manages the detection results by subject ID to identify which subject the detection information is for. In other words, different subjects have different subject IDs. When multiple subjects are detected, a main subject to track focus is selected from among them. Subjects other than the main subject are called sub-subjects.

[0034] In this embodiment, a subject determined to have taken an action using the above-described action detection method is determined to be the main subject. There are several possible methods for determining the main subject when all subjects are not taking an action, such as center-priority or size-priority, but detailed explanations of these methods are omitted as they are not relevant to the gist of the present invention. Furthermore, the main region detection methods applicable to the present invention are not limited to these methods, and other methods may also be used.

[0035] In step S302, the system control unit 209 determines whether multiple main regions have been detected based on the detection results of the subject detection unit 211. If multiple main regions have been detected, the system control unit 209 proceeds to step S303; otherwise, the system control unit 209 proceeds to step S304. The main regions here refer to the subject's face, body, eyes, nose, mouth, etc.

[0036] The concept of the detected main area will now be explained using Fig. 4A. Fig. 4A shows a state in which pupil A, face B, and body C have been detected. From the subject detection unit 211, it is possible to acquire the type of subject, such as a person or an animal, as well as the center coordinates, horizontal size, and vertical size of each detected main area.

[0037] In step S303, the system control unit 209 inputs the minimum detectable main area, that is, the smaller of the horizontal and vertical sizes of pupil A in FIG. 4A, into MinA, and sets MinA as the size of one AF frame.

[0038] In step S305, the system control unit 209 calculates the horizontal size H in Fig. 4A that encompasses all of the main areas from the horizontal coordinates and horizontal sizes of each detected main area, and then determines the number of horizontal AF frames by dividing the horizontal size H by the AF frame size MinA, as shown in Fig. 4B.

[0039] In step S307, the system control unit 209 calculates the vertical size V in Fig. 4A that encompasses all of the detected main areas from the vertical coordinates and vertical sizes of each detected main area. Then, as shown in Fig. 4B, the system control unit 209 determines the number of vertical AF frames by dividing the vertical size V by the AF frame size MinA, and ends the AF frame setting.

[0040] Here, the above AF frame size is used to determine the inclusion area D. This inclusion area D may be a size that encompasses the pupil A, face B, and torso C, with a margin provided as shown in FIG. 4A to take into account subject movement and detection errors. The method of using this inclusion area D will be described later with reference to FIG. 8B. In this embodiment, the size of a square area based on the smallest detectable main area is set as the AF frame size, but the AF frame size may be different horizontally and vertically, or a size may be set that is divided by the number of AF frames that can be calculated by the system control unit 209.

[0041] In step S304, the system control unit 209 sets an AF frame of a predetermined size X for the detected face. The size X may be set to a pupil size estimated from the face, or a frame size may be set that ensures an S / N ratio and provides sufficient focusing performance in a low-light environment. In this embodiment, X is set using the estimated pupil size.

[0042] In step S306, the system control unit 209 sets the number of AF frames Y that includes the area of face B with the AF frame size X and can cope with the movement of the face.

[0043] <AF operation> FIG. 5 is a flowchart for explaining the AF operation in step S203 of FIG. 2.

[0044] First, in step S401, the system control unit 209 performs focus detection processing in each AF frame as shown in FIG. 4B, and detects the defocus amount and reliability. The focus detection processing will be described later.

[0045] In step S402, the system control unit 209 performs AF main frame selection using the focus detection information obtained in step S401. The details of the AF main frame selection will be described later using FIG. 7.

[0046] In step S403, the system control unit 209 stores the defocus amount of the AF frame selected in step S402, the defocus history of the AF frame selected in the past, and the subject information. The details of the storage of this movement history information will be described later using FIG. 8A.

[0047] In step S404, the system control unit 209 predicts the movement position of the moving object using the movement history information in step S403. The details of the prediction of the movement position of the moving object will be described later using FIG. 9.

[0048] In step S405, the system control unit 209 performs lens driving according to the prediction result calculated in step S404.

[0049] <Focus detection processing> FIG. 6 is a flowchart for explaining the focus detection processing in step S401 of FIG. 5.

[0050] First, in step S501, the system control unit 209 sets a focus detection area in an arbitrary range within the image data output from the imaging device 201.

[0051] In step S502, the system control unit 209 acquires a pair of image signals (A image signal, B image signal) for focus detection from the imaging device 201 corresponding to the focus detection area set in step S501.

[0052] In step S503, the system control unit 209 performs row addition and averaging processing on the pair of image signals acquired in step S502 in the vertical direction. This processing can reduce the influence of noise in the image signals.

[0053] In step S504, the system control unit 209 performs filter processing to extract signal components in a predetermined frequency band from the signals vertically row-added and averaged in step S503.

[0054] In step S505, the system control unit 209 calculates a correlation amount from the signals filter-processed in step S504.

[0055] In step S506, the system control unit 209 calculates a correlation change amount from the correlation amount calculated in step S505.

[0056] In step S507, the system control unit 209 calculates an image shift amount from the correlation change amount calculated in step S506.

[0057] In step S508, the system control unit 209 calculates a reliability indicating how reliable the image shift amount calculated in step S507 is.

[0058] In step S509, the image shift amount is converted into a defocus amount to end the focus detection process.

[0059] <AF Main Frame Selection> FIG. 7 is a flowchart for explaining the operation of main frame selection in step S402 of FIG. 5. In this embodiment, a histogram, which is an image analysis means, is used. However, since it is a general technique, the details of the histogram are omitted from the description.

[0060] First, in step S601, the system control unit 209 determines whether or not a human face has been detected by the subject detection unit 211. If a face has been detected, the system control unit 209 proceeds to step S602, and if not, the system control unit 209 proceeds to step S603.

[0061] In step S602, the system control unit 209 determines whether or not a person's torso has been detected by the subject detection unit 211. If a torso has been detected, the system control unit 209 proceeds to step S604, and if not, the system control unit 209 proceeds to step S605.

[0062] In step S604, the system control unit 209 counts the defocus amounts calculated for each AF frame in the area that includes all the main areas in step S509 of Fig. 6 for each specified depth to create a histogram. This area may be the same as inclusion area D shown in Fig. 4A, or may be smaller than inclusion area D to exclude components such as the background.

[0063] In this embodiment, the defocus amount itself is converted into a histogram, but taking into account moving subjects, a predicted value corresponding to the subject position may be calculated based on the defocus amount calculated for each AF frame, and this predicted value may be converted into a histogram.

[0064] In step S605, the system control unit 209 counts the defocus amount calculated for each AF frame set within an area that is a predetermined multiple of the face frame for each predetermined depth, and creates a histogram.

[0065] In step S606, the system control unit 209 determines whether the peak value (the number of AF frames of the histopeak) of the histogram created in step S604 or step S605 is equal to or greater than a predetermined number.

[0066] In this embodiment, the peak value of the histogram is normalized for the total number of AF frames and converted to a ratio for use. If the peak value is equal to or greater than a predetermined ratio, the system control unit 209 proceeds to step S612, and if it is less than the predetermined ratio, the system control unit 209 proceeds to step S607.

[0067] In step S607, the system control unit 209 determines whether the pupil has been detected and whether the pupil-centered AF frame is within a predetermined depth from the focus lens position when the defocus amount was calculated. Generally, subject detection information is more accurate when the subject is in focus, and if the defocus amount is greater than or equal to the predetermined depth, the subject detection information may be an erroneous detection. For this reason, this condition is used. If the pupil has been detected and the pupil-centered frame is within the predetermined depth, the system control unit 209 proceeds to step S609, where it sets the pupil-centered AF frame as the main frame; otherwise, it proceeds to step S608.

[0068] In step S608, the system control unit 209 performs a loop process for all frames to select a main frame from the set AF frames. The initial value of the main frame is assumed to be already set to information (such as the total number of frames + 1) that allows determining that no main frame has been selected, and is not shown in the figure.

[0069] In step S610, the system control unit 209 determines whether the AF frame being processed is closer than the currently selected main frame and has a defocus amount within a predetermined depth. If the conditions are met, the system control unit 209 proceeds to step S611; if not, the system control unit 209 returns to step S608 and repeats the loop.

[0070] In step S611, the system control unit 209 updates the main frame. By repeating this loop, the closest AF frame within a predetermined depth is selected as the main frame.

[0071] In step S612, similar to step S608, the system control unit 209 performs loop processing for all frames to select a main frame from the set AF frames.

[0072] In step S613, the system control unit 209 determines whether the AF frame is counted as a histogram peak. If so, the system control unit 209 proceeds to step S614; if not, the system control unit 209 repeats the loop process of step S612.

[0073] In step S614, the system control unit 209 determines whether or not the pupil has been detected. If the pupil has been detected, the system control unit 209 proceeds to step S616, and if not, the system control unit 209 proceeds to step S615.

[0074] In step S615, the system control unit 209 determines whether the AF frame being processed is closer to the face detection center than the currently selected main frame. If it is closer, the system control unit 209 proceeds to step S617 and updates the main frame. If not, the system control unit 209 repeats the loop process of step S612.

[0075] In step S616, the system control unit 209 determines whether the AF frame being processed is one whose coordinates are closer to the pupil detection center than the currently selected main frame. If it is closer, the system control unit 209 proceeds to step S617 and updates the main frame. If not, the system control unit 209 repeats the loop processing of step S612. By repeating this loop, the AF frame closest to the pupil center or the face center among the AF frames counted as histogram peaks is detected as the main frame.

[0076] In step S603, the system control unit 209 determines whether or not a torso has been detected by the subject detection unit 211. If a torso has been detected, the system control unit 209 proceeds to step S618, and if not, the system control unit 209 proceeds to step S624.

[0077] In step S618, the system control unit 209 creates a histogram in the body detection area and finds the histogram peak.

[0078] In step S619, the system control unit 209 determines whether the peak value of the histogram created in step S618 is equal to or greater than a predetermined ratio. If it is equal to or greater than the predetermined ratio, the system control unit 209 proceeds to step S620, and if it is less than the predetermined ratio, the system control unit 209 proceeds to step S626.

[0079] In step S626, the system control unit 209 selects the center of the body as the main frame.

[0080] In step S620, the system control unit 209 performs loop processing for all frames to select a main frame from the set AF frames, similar to step S608.

[0081] In step S621, the system control unit 209 determines whether the AF frame is counted as a histogram peak. If so, the system control unit 209 proceeds to step S622; if not, the system control unit 209 repeats the loop process of step S620.

[0082] In step S622, the system control unit 209 determines whether the AF frame being processed is a frame whose coordinates are closer to the body detection center than the currently selected main frame. If it is closer, the system control unit 209 proceeds to step S623; if not, the system control unit 209 repeats the loop processing of step S620.

[0083] In step S623, the system control unit 209 updates the main frame. By repeating this loop, the AF frame closest to the center of the moving object among the AF frames counted as the histogram peak is detected as the main frame.

[0084] In step S624, the system control unit 209 determines whether a main frame has been selected according to the flow described above by checking whether the main frame is at its initial value. If the main frame is at its initial value, the main frame has not been selected, and the process proceeds to step S625; if not, the main frame selection process ends.

[0085] In step S625, the system control unit 209 performs processing such as selecting a main frame in a predetermined area within the screen without using the detection information.

[0086] <Movement history saving process> FIG. 8A is a flowchart illustrating the operation of saving the motion history in step S403 of FIG.

[0087] First, in step S700, the system control unit 209 determines whether or not a person has been detected by the subject detection unit 211. If a person has been detected, the system control unit 209 proceeds to step S701, and if a person has not been detected, the system control unit 209 proceeds to step S704.

[0088] In step S701, the system control unit 209 performs a loop process the number of times equal to the number of detected people.

[0089] In step S702, the system control unit 209 selects an AF frame for the person being processed that has calculated the closest focus detection result within the inclusion area D determined in FIG. 4A.

[0090] This concept is shown in Figure 8B. For each detected subject in Figure 8B, an AF frame is searched for that calculates the closest focus detection information from within inclusion frame D (dotted line). If the subject falls outside the AF area during this process, the AF frame size and number may be increased so that as many AF frames as possible cover the detected subject. Also, if there are multiple subjects in action, as shown in Figure 8B, an object that can identify the main subject in a sports scene, such as a ball 850, may be detected, and the action subject that is close to the object may be identified as the main subject.

[0091] In step S703, the system control unit 209 stores the subject ID and the focus detection information (focus position information) of the AF frame selected in step S702 in the DRAM as a sub-subject movement history (hereinafter referred to as sub-history). This process is repeated for the number of detected people.

[0092] In step S704, the system control unit 209 stores the subject ID of the main subject and the focus detection result of the AF frame corresponding to the main subject in the DRAM as a movement history of the main subject (hereinafter referred to as the main history). Note that the main subject here refers to the subject corresponding to the AF frame (main frame) selected in the main frame selection process in FIG.

[0093] <Moving object prediction processing> FIG. 9 is a flowchart for explaining the prediction operation using the motion history stored in steps S703 and S704 of FIG. 8A.

[0094] In step S800, the system control unit 209 determines whether or not an action has been detected in the subject on the screen using the method described above. If an action has been detected, the system control unit 209 proceeds to step S801; if not, the system control unit 209 proceeds to step S808.

[0095] In step S801, the system control unit 209 determines whether the subject ID of the currently focused subject has changed when focusing on the subject whose action has been detected. If the subject ID has changed, the system control unit 209 proceeds to step S802; otherwise, the system control unit 209 proceeds to step S804.

[0096] This is because, although it is desirable to instantly shift the focus to track a subject that has taken action, it is necessary to predict the movement of that subject using the motion history information for that subject and adjust the focus accordingly. Therefore, the process proceeds to step S802.

[0097] In step S802, the system control unit 209 searches the sub-history stored in step S703 of Fig. 8A to determine whether there is a motion history that matches the subject ID of the action detected. If there is a sub-history with a matching subject ID, the system control unit 209 proceeds to step S803; otherwise, the system control unit 209 proceeds to step S804.

[0098] In step S803, the system control unit 209 updates the main history with the found sub-history. At this time, the conditions that restrict subject transitions are relaxed. For example, control that restricts focus tracking by detecting a change in focus detection results that is greater than a predetermined value is not performed.

[0099] In step S804, the system control unit 209 determines whether or not Num_a or more histories have been saved (whether or not a predetermined number or more have been stored). If Num_a or more histories have been saved, the system control unit 209 proceeds to step S805, and if Num_a or more histories have not been saved, the system control unit 209 proceeds to step S809.

[0100] In step S805, the system control unit 209 determines whether the movement is predictable. In this embodiment, prediction is deemed impossible when the change in the value of the focus detection information in the saved history is equal to or greater than a predetermined threshold (for example, a subject whose position is separated by 10 depths or more in terms of depth conversion) or when the variation in the value is equal to or greater than a predetermined value (for example, calculated by 3σ or the like from the focus detection result). This condition is determined based on factors that may cause a large error in the predicted position, which will be described later.

[0101] In step S806, the system control unit 209 calculates a predicted movement curve as shown in FIG.

[0102] In step S807, the system control unit 209 sets the predicted position of the subject based on the prediction curve obtained in step S806 as the target position for lens drive.

[0103] In step S808, the system control unit 209 determines whether or not Num_b or more histories have been saved. In this case, the magnitude relationship between Num_a and Num_b is set as follows:

[0104] Num_a < Num_b …(1) The reason for using the relationship in formula (1) above is that the subject at the time of action detection is likely to be in a state that is suitable for the timing of shooting sports scenes, the duration of the action is shorter than normal movement, and there is also a high possibility that the subject is moving in the distance direction, making it as easy to use prediction as possible. This increases the success rate when shooting difficult sports scenes.

[0105] In step S809, since the conditions are not predictable, the system control unit 209 sets the focus detection position as the target position for lens drive.

[0106] In this embodiment, by storing the history of sub-subjects that are candidates for the main subject, when the focus is shifted from the main subject to the sub-subject due to action detection, the movement of the sub-subject at the new location can be predicted and the focus can be tracked immediately, making it possible to provide an image that is in focus at the ideal timing in sports scenes, etc.

[0107] <Prediction curve calculation process> Figure 10 shows a prediction curve calculated using the main history saved in step S704 of Figure 8A. The vertical axis represents the subject image plane position calculated from the defocus amount calculated in step S401 of Figure 5, and the horizontal axis represents time. The larger the image plane position on the vertical axis, the greater the distance, and the history in the figure shows the tracking of a subject approaching the photographer (camera).

[0108] The focus adjustment process is performed periodically, and times T1 to T5 are the times when the focus adjustment process is performed. The predicted drive amount can be obtained by, for example, deriving a prediction curve using a batch least-squares method using past image plane positions and each focus detection time, and then calculating the image plane position (focus adjustment position) at the predicted future time based on this curve. In other words, the lens is driven at time T4, and a lens drive equivalent to defocus y is requested to achieve focus at time T5.

[0109] As described above, the method of this embodiment makes it possible to provide an image that is in focus at an appropriate timing in a sports scene or the like.

[0110] The disclosure of this specification includes the following focus adjustment device and method, imaging device, program, and storage medium.

[0111] (Item 1) an object detection means for detecting an object from an image; a motion detection means for detecting motion of the detected subject and storing a history of the motion; a focus adjustment means for focusing on a subject; a prediction means for predicting a focus adjustment position for the focus adjustment means to focus on the second object based on the history of the movement of the second object stored by the movement detection means, when the focus adjustment means shifts from a state in which the focus adjustment means focuses on the first object to a state in which the focus adjustment means focuses on the second object; A focus adjustment device comprising:

[0112] (Item 2) The focus adjustment device according to item 1, further comprising a state detection means for detecting a specific state of the subject from a feature amount of the subject, wherein the focus adjustment means transitions from a state in which the first subject is focused to a state in which the second subject is focused when the specific state of the second subject is detected.

[0113] (Item 3) 3. The focus adjustment device according to item 2, wherein the specific state is a state in which the subject has taken a specific action.

[0114] (Item 4) 4. The focus adjustment device according to any one of items 1 to 3, further comprising a setting unit that sets a plurality of focus detection areas within a screen, wherein the motion detection unit detects motion of the subject based on changes in focus position in the plurality of focus detection areas and stores a history of the motion.

[0115] (Item 5) 5. The focus adjustment device according to item 4, wherein the prediction means predicts a focus adjustment position for focusing on the second subject when a predetermined number or more of movement histories have been stored for the second subject.

[0116] (Item 6) 6. The focus adjustment device according to item 4 or 5, characterized in that if the difference or variation in focus position in the movement history of the second subject is equal to or greater than a predetermined value, the prediction means does not predict the focus adjustment position for focusing on the second subject.

[0117] (Item 7) The focus adjustment device described in any one of items 1 to 6, characterized in that the motion detection means stores the closest focus detection information among multiple focus detection areas for the subject as movement history information.

[0118] (Item 8) an imaging means for imaging a subject; A focus adjustment device according to any one of items 1 to 7, An imaging device comprising:

[0119] (Item 9) a subject detection step of detecting a subject from an image; a motion detection step of detecting motion of the detected subject and storing a history of the motion; a focus adjustment step of focusing on an object; a prediction step of predicting a focus adjustment position for focusing on the second object in the focus adjustment step based on the history of the movement of the second object stored in the movement detection step, when the focus adjustment step shifts from a state in which the first object is focused to a state in which the second object is focused; A focus adjustment method comprising:

[0120] (Item 10) 10. A program for causing a computer to execute each step of the focus adjustment method according to item 9.

[0121] (Item 11) A computer-readable storage medium storing a program for causing a computer to execute each step of the focus adjustment method according to item 9.

[0122] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0123] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0124] 100: lens device, 103: focus lens, 105: lens controller, 200: camera body, 201: image sensor, 204: AF signal processing unit, 209: system control unit, 210: lens communication unit, 211: subject detection unit

Claims

1. an object detection means for detecting an object from an image; a state detection means for detecting a specific state of a subject from an image; a focus detection unit that acquires a history of focus detection results corresponding to the subject detected by the subject detection unit; a focus adjustment means for adjusting a focus using the focus detection result; a control means for acquiring a history of focus detection results of the subject by the focus detection means in parallel with the detection process of a specific state by the state detection means; a prediction means for predicting a focus adjustment position for the focus adjustment means to focus on the second object using a history of focus detection results for the second object, when the focus adjustment means shifts from a state in which the focus adjustment means focuses on a first object to a state in which the specific state is detected and the focus adjustment means focuses on the second object; A focus adjustment device comprising:

2. 2. The focus adjustment device according to claim 1, wherein the specific state is a state in which the subject has taken a specific action.

3. 2. The focus adjustment device according to claim 1, wherein the prediction means predicts a focus adjustment position for focusing on the second subject when a predetermined number or more of focus detection result histories for the second subject have been stored.

4. 2. The focus adjustment device according to claim 1, wherein when a difference or variation in focus position in the history of focus detection results for the second subject is equal to or greater than a predetermined value, the prediction means does not predict a focus adjustment position for focusing on the second subject.

5. 2. The focus adjustment device according to claim 1, wherein the focus detection means stores focus detection information for the closest of a plurality of focus detection areas for the subject as information on the history of focus detection results.

6. an imaging means for imaging a subject; A focus adjustment device according to any one of claims 1 to 5; An imaging device comprising:

7. a subject detection step of detecting a subject from an image; a state detection step of detecting a specific state of a subject from an image; a focus detection step of acquiring a history of focus detection results corresponding to the subject detected by the subject detection step; a focus adjustment step of adjusting a focus using the focus detection result; a control step of controlling acquisition of a history of focus detection results of the subject by the focus detection step to be performed in parallel with detection processing of a specific state by the state detection step; a prediction step of predicting a focus adjustment position for focusing on the second object by the focus adjustment step, when the state of focusing on the first object by the focus adjustment step is shifted to a state of focusing on the second object for which the specific state has been detected, using a history of focus detection results for the second object; A focus adjustment method comprising:

8. A program for causing a computer to execute each step of the focus adjustment method according to claim 7.

9. A computer-readable storage medium storing a program for causing a computer to execute each step of the focus adjustment method according to claim 7.

Images