Focus adjustment device and method, imaging device, program, and storage medium

Abstract

To provide an imaging apparatus capable of focusing on a subject at a timing appropriate for photographing.SOLUTION: A focus adjustment device includes: a subject detection unit for detecting a subject from an image; a motion detection unit for detecting a motion in the detected subject and storing a history of the motion; a focus adjustment unit for focusing on the subject; and a prediction unit for predicting a focus adjustment position where the focus adjustment unit focuses on a second subject on the basis of the history of the motion of the second subject stored by the motion detection unit, when shifting from a state where the focus adjustment unit focuses on a first subject to a state where the focus adjustment unit focuses on the second subject.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a focus adjustment device and method for use in an imaging device. [Background technology]

[0002] In recent years, various AF (autofocus) methods have been put to practical use, such as image-sensor phase-difference AF and contrast AF, which use image sensors. Furthermore, in these AF methods, techniques for identifying and focusing on the area of the main subject are known.

[0003] Patent Document 1 discloses a technology that detects changes in the size and position of a feature area within a photographic screen based on multiple images generated in time series, estimates the direction of movement of a subject, and performs a focusing operation based on the estimation result.

[0004] Furthermore, Patent Document 2 discloses an imaging device equipped with a capture determination means that performs focus adjustment when the amount of change in image plane position during focus detection is less than a preset threshold. When predetermined conditions are met and the subject and imaging device are stationary, the threshold of the capture determination means is set to a value greater than the preset value, thereby improving focus tracking accuracy. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-160991 [Patent Document 2] Japanese Patent Application Laid-Open No. 2012-203207 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in Patent Document 1, the motion of the subject is determined based only on the size, position, and focus detection information of the subject, and it is not possible to determine whether the subject is in a desirable state as a main subject.

[0007] In Patent Document 2, focus tracking can be improved only at the start of AF, but it cannot be applied to focus tracking on a subject that is preferable as the main subject after AF has started.

[0008] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an imaging device that can focus on a subject at an appropriate timing for photographing. [Means for solving the problem]

[0009] The focus adjustment device of the present invention is characterized by comprising: subject detection means for detecting a subject from an image; motion detection means for detecting movement of the detected subject and storing a movement history; focus adjustment means for focusing on the subject; and prediction means for predicting, when the focus adjustment means transitions from a state in which it is focused on a first subject to a state in which it is focused on a second subject, a focus adjustment position for the focus adjustment means to focus on the second subject based on the movement history of the second subject stored by the motion detection means. [Effects of the Invention]

[0010] According to the present invention, it is possible to focus on a subject at an appropriate timing for photographing. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing the configuration of an imaging apparatus according to an embodiment of the present invention; [Figure 2] 4 is a flowchart illustrating the operation of the imaging apparatus. [Figure 3] 10 is a flowchart illustrating an operation of setting an AF frame. [Figure 4A] Conceptual diagram of detection areas for a person's eyes, face, and body. [Figure 4B] Conceptual diagram of AF frame settings for a person's eyes, face, and body. [Figure 5]10 is a flowchart illustrating an AF operation. [Figure 6] 10 is a flowchart illustrating focus detection processing. [Figure 7] 10 is a flowchart illustrating an AF main frame selection process. [Figure 8A] 10 is a flowchart showing a motion history saving process. [Figure 8B] Conceptual diagram of the subject frame when saving the motion history. [Figure 9] 10 is a flowchart showing the operation of moving object prediction. [Figure 10] Conceptual diagram of a prediction curve. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0013] <Configuration of imaging device> 1 is a block diagram showing the configuration of an interchangeable lens camera 1 (hereinafter simply referred to as a camera) that is an embodiment of the imaging device of the present invention. The camera 1 of this embodiment is capable of performing focus adjustment by an imaging surface phase difference detection method using an output signal from an imaging element 201 that captures an image of a subject.

[0014] 1, camera 1 is composed of a lens device (interchangeable lens) 100 and a camera body 200. When lens device 100 is attached to camera body 200 via a mount unit (not shown) having an electrical contact unit 106, a lens controller 105 that controls the overall operation of lens device 100 and a system control unit 209 that controls the overall operation of the camera become able to communicate with each other.

[0015] First, we will explain the configuration of the lens device 100. The lens device 100 includes a photographing lens 101 with a zoom mechanism, an aperture and shutter 102 that controls the amount of light, a focus lens 103 for focusing on an image sensor 201, a motor 104 that drives the focus lens, and a lens controller 105.

[0016] Next, the configuration of the camera body 200 will be described. The camera body 200 is configured to acquire an imaging signal from a light beam that passes through the imaging optical system of the lens apparatus 100. The camera body 200 includes an image sensor 201 that photoelectrically converts reflected light from a subject into an electrical signal, an A / D conversion unit 202 that includes a CDS circuit that removes output noise from the image sensor 201 and a nonlinear amplifier circuit that performs A / D conversion before the A / D conversion, an image processing unit 203, and an AF signal processing unit 204. The camera body 200 also includes a format conversion unit 205, a high-speed internal memory (e.g., random access memory, hereinafter referred to as DRAM) 206, and an image recording unit 207 that includes a recording medium such as a memory card and its interface. The camera body 200 also includes a timing generator 208, a system control unit 209 that controls the system, such as the shooting sequence, a lens communication unit 210 that communicates between the camera body 200 and the lens apparatus 100, an object detection unit 211 that detects an object from an image, and an image display memory (hereinafter referred to as VRAM) 212.

[0017] The camera body 200 also has an image display unit 213 that displays images, operational assistance, and the camera status, and also displays the shooting screen and focus detection area during shooting. It also has an operation unit 214 for externally operating the camera 1, a shooting mode switch 215 for selecting a shooting mode such as macro mode or sports mode, and a main switch 216 for powering on the camera 1.

[0018] It also has a switch (hereinafter referred to as SW1) 217 for performing shooting standby operations such as AF (autofocus) and AE (autoexposure control), and a shooting switch (hereinafter referred to as SW2) 218 for shooting after operating switch SW1. DRAM 206, which is an internal memory, is used as temporary image storage means, i.e., a high-speed buffer, or as working memory when processing images. The operation unit 214 includes, for example, the following: a menu switch for setting various settings such as the shooting functions of the imaging device and settings for image playback, and an operation mode changeover switch between shooting mode and playback mode.

[0019] The image sensor 201 is configured with a CCD or CMOS sensor. Each pixel of the image sensor 201 used in this embodiment is configured with two (a pair) of photodiodes A and B and one microlens provided for the pair of photodiodes A and B. Each pixel splits incident light with the microlens to form a pair of optical images on the pair of photodiodes A and B. The pair of photodiodes A and B then outputs a pair of pixel signals (signals A and B) used as an AF (autofocus) signal, which will be described later. In addition, an image signal (signal A+B) can be obtained by adding the outputs of the pair of photodiodes A and B.

[0020] By combining multiple A signals and multiple B signals output from multiple pixels, a pair of image signals (A image signal, B image signal) is obtained as an AF signal (focus detection signal) used for AF using the image plane phase difference detection method (hereinafter referred to as image plane phase difference AF).

[0021] The AF signal processing unit 204 performs correlation calculations on the pair of image signals to calculate a phase difference (hereinafter referred to as image shift amount) that is the amount of shift between the pair of image signals, and further calculates the defocus amount (and defocus direction and reliability) of the photographing optical system from the image shift amount. Note that the AF signal processing unit 204 calculates multiple defocus amounts in a specified area that can be specified.

[0022] <Operation of the imaging device> Hereinafter, the operation of the imaging device according to this embodiment will be described with reference to FIG. 2. FIG. 2 is a flowchart showing the flow of imaging control processing when performing still image imaging from the state of displaying a live view image. The system control unit 209 as a computer executes this processing according to a control program as a computer program.

[0023] First, in step S201, the system control unit 209 checks the state of the switch SW1 (217). If the switch SW1 is ON, the system control unit 209 proceeds to step S202 for processing; otherwise, it waits as it is.

[0024] In step S202, the system control unit 209 performs AF frame setting on the AF signal processing unit 204. Details of the AF frame setting will be described later.

[0025] In step S203, the system control unit 209 performs an AF operation. Details of the AF operation will be described later.

[0026] In step S204, the system control unit 209 checks the state of the switch SW1 (217). If the switch SW1 is ON, the system control unit 209 proceeds to step S205 for processing; otherwise, it returns the processing to step S201.

[0027] In step S205, the system control unit 209 checks the state of the switch SW2 (218). If the switch SW2 is ON, the system control unit 209 proceeds to step S206 for processing; otherwise, it returns the processing to step S204.

[0028] In step S206, the system control unit 209 executes a shooting operation and returns the processing to step S201.

[0029] <AF Frame Setting> FIG. 3 is a flowchart for explaining the operation of AF frame setting in step S202 of FIG. 2.

[0030] First, in step S301, the system control unit 209 acquires subject detection information from the subject detection unit 211. In this embodiment, the subject is a person, and a main area within the subject is detected. Here, the main area refers to the eyes, face, and body of a person or animal. These detection methods use known learning techniques using machine learning or recognition processing using image processing means.

[0031] For example, the types of machine learning include: (1) Support Vector Machine (2) Convolutional Neural Network (3) Recurrent Neural Network As an example of recognition processing, a method is known in which a skin color area is extracted from the gradation color of each pixel represented by image data and a face is detected based on the degree of matching with a prepared face outline plate. Another known method is to use well-known pattern recognition technology to extract facial features such as the eyes, nose, and mouth to detect a face.

[0032] These techniques can also be applied to estimate the posture of a subject from features such as the joints of the human body, and detect important states (hereafter referred to as actions) of the subject in sports scenes. This makes it possible to detect, for example, a shooting posture in soccer or basketball, or an attacking posture in volleyball. This action detection method is used in the action detection process (state detection process) in step S800 of FIG. 9, which will be described later.

[0033] The subject detection unit 211 can detect multiple subjects, and manages the detection results by subject ID to identify which subject the detection information is for. In other words, different subjects have different subject IDs. When multiple subjects are detected, a main subject to track focus is selected from among them. Subjects other than the main subject are called sub-subjects.

[0034] In this embodiment, a subject determined to have taken an action using the above-described action detection method is determined to be the main subject. There are several possible methods for determining the main subject when all subjects are not taking an action, such as center-priority or size-priority, but detailed explanations of these methods are omitted as they are not relevant to the gist of the present invention. Furthermore, the main region detection methods applicable to the present invention are not limited to these methods, and other methods may also be used.

[0035] In step S302, the system control unit 209 determines whether multiple main regions have been detected based on the detection results of the subject detection unit 211. If multiple main regions have been detected, the system control unit 209 proceeds to step S303; otherwise, the system control unit 209 proceeds to step S304. The main regions here refer to the subject's face, body, eyes, nose, mouth, etc.

[0036] The concept of the detected main area will now be explained using Fig. 4A. Fig. 4A shows a state in which pupil A, face B, and body C have been detected. From the subject detection unit 211, it is possible to acquire the type of subject, such as a person or an animal, as well as the center coordinates, horizontal size, and vertical size of each detected main area.

[0037] In step S303, the system control unit 209 inputs the minimum detectable main area, that is, the smaller of the horizontal and vertical sizes of pupil A in FIG. 4A, into MinA, and sets MinA as the size of one AF frame.

[0038] In step S305, the system control unit 209 calculates the horizontal size H in Fig. 4A that encompasses all of the main areas from the horizontal coordinates and horizontal sizes of each detected main area, and then determines the number of horizontal AF frames by dividing the horizontal size H by the AF frame size MinA, as shown in Fig. 4B.

[0039] In step S307, the system control unit 209 obtains the vertical size V in FIG. 4A that encompasses all the main regions from the vertical coordinates and vertical sizes of the respective detection main regions. Then, as shown in FIG. 4B, the vertical AF frame number is determined by dividing the vertical size V by the AF frame size MinA, and the AF frame setting is completed.

[0040] Here, the inclusion region D is obtained using the above AF frame size. This inclusion region D may have a margin as shown in FIG. 4A in consideration of the movement of the subject and detection errors with respect to the size that encompasses the pupil A, face B, and body C. The usage method of this inclusion region D will be described later with reference to FIG. 8B. In the present embodiment, the size of the square region based on the minimum detection main region is used as the AF frame size, but the AF frame sizes may be different for horizontal and vertical, or the sizes divided by the AF frame number that can be calculated by the system control unit 209 may be set.

[0041] In step S304, the system control unit 209 sets an AF frame of a predetermined size X for the detected face. The size X may be set to the pupil size estimated from the face, or may be set to a frame size that can ensure S / N and obtain sufficient focusing performance in consideration of low illuminance environments. In the present embodiment, X is set using the estimated pupil size.

[0042] In step S306, the system control unit 209 sets an AF frame number Y that encompasses the region of the face B with the AF frame size X and can also cope with the case where the face moves.

[0043] <AF operation> FIG. 5 is a flowchart for explaining the AF operation of step S203 in FIG. 2.

[0044] First, in step S401, the system control unit 209 performs focus detection processing in each AF frame as shown in FIG. 4B, and detects the defocus amount and reliability. The focus detection processing will be described later.

[0045] In step S402, the system control unit 209 selects the AF main frame using the focus detection information obtained in step S401. Details of the AF main frame selection will be described later with reference to FIG.

[0046] In step S403, the system control unit 209 saves the defocus amount of the AF frame selected in step S402, the defocus history of previously selected AF frames, and subject information. Details of saving this movement history information will be described later with reference to FIG. 8A.

[0047] In step S404, the system control unit 209 predicts the movement position of the moving object using the movement history information in step S403. Details of the prediction of the movement position of the moving object will be described later with reference to FIG.

[0048] In step S405, the system control unit 209 drives the lens in accordance with the prediction result calculated in step S404.

[0049] <Focus detection processing> FIG. 6 is a flowchart illustrating the focus detection process in step S401 of FIG.

[0050] First, in step S501 , the system control unit 209 sets a focus detection area of an arbitrary range within the image data output from the image sensor 201 .

[0051] In step S502, the system control unit 209 acquires a pair of image signals (image signal A and image signal B) for focus detection from the image sensor 201 corresponding to the focus detection area set in step S501.

[0052] In step S503, the system control unit 209 performs row averaging processing in the vertical direction on the pair of image signals acquired in step S502. This processing can reduce the influence of noise on the image signals.

[0053] In step S504, the system control unit 209 performs a filtering process to extract a signal component in a predetermined frequency band from the signal vertically added and averaged in step S503.

[0054] In step S505, the system control unit 209 calculates a correlation amount from the signal filtered in step S504.

[0055] In step S506, the system control unit 209 calculates a correlation change amount from the correlation amount calculated in step S505.

[0056] In step S507, the system control unit 209 calculates an image displacement amount from the correlation change amount calculated in step S506.

[0057] In step S508, the system control unit 209 calculates a reliability representing how reliable the image displacement amount calculated in step S507 is.

[0058] In step S509, the image displacement amount is converted into a defocus amount to end the focus detection process.

[0059] <AF Main Frame Selection> FIG. 7 is a flowchart for explaining the operation of the main frame selection in step S402 of FIG. 5. In this embodiment, a histogram which is an image analysis means is used, but since it is a general technique, the details of the histogram are omitted from the description.

[0060] First, in step S601, the system control unit 209 determines whether a person's face has been detected by the subject detection unit 211. If the face has been detected, the system control unit 209 proceeds to step S602, and if not, it proceeds to step S603.

[0061] In step S602, the system control unit 209 determines whether or not a person's torso has been detected by the subject detection unit 211. If a torso has been detected, the system control unit 209 proceeds to step S604, and if not, the system control unit 209 proceeds to step S605.

[0062] In step S604, the system control unit 209 counts the defocus amounts calculated for each AF frame in the area that includes all the main areas in step S509 of Fig. 6 for each specified depth to create a histogram. This area may be the same as inclusion area D shown in Fig. 4A, or may be smaller than inclusion area D to exclude components such as the background.

[0063] In this embodiment, the defocus amount itself is converted into a histogram, but taking into account moving subjects, a predicted value corresponding to the subject position may be calculated based on the defocus amount calculated for each AF frame, and this predicted value may be converted into a histogram.

[0064] In step S605, the system control unit 209 counts the defocus amount calculated for each AF frame set within an area that is a predetermined multiple of the face frame for each predetermined depth, and creates a histogram.

[0065] In step S606, the system control unit 209 determines whether the peak value (the number of AF frames of the histopeak) of the histogram created in step S604 or step S605 is equal to or greater than a predetermined number.

[0066] In this embodiment, the peak value of the histogram is normalized for the total number of AF frames and converted to a ratio for use. If the peak value is equal to or greater than a predetermined ratio, the system control unit 209 proceeds to step S612, and if it is less than the predetermined ratio, the system control unit 209 proceeds to step S607.

[0067] In step S607, the system control unit 209 determines whether the pupil has been detected and whether the pupil-centered AF frame is within a predetermined depth from the focus lens position when the defocus amount was calculated. Generally, subject detection information is more accurate when the subject is in focus, and if the defocus amount is greater than or equal to the predetermined depth, the subject detection information may be an erroneous detection. For this reason, this condition is used. If the pupil has been detected and the pupil-centered frame is within the predetermined depth, the system control unit 209 proceeds to step S609, where it sets the pupil-centered AF frame as the main frame; otherwise, it proceeds to step S608.

[0068] In step S608, the system control unit 209 performs a loop process for all frames to select a main frame from the set AF frames. Also, the initial value of the main frame is assumed to be already set to information (such as the total number of frames + 1) that allows determining that no main frame has been selected, and is not shown in the figure.

[0069] In step S610, the system control unit 209 determines whether the AF frame being processed is closer than the currently selected main frame and has a defocus amount within a predetermined depth. If the conditions are met, the system control unit 209 proceeds to step S611; if not, the system control unit 209 returns to step S608 and repeats the loop.

[0070] In step S611, the system control unit 209 updates the main frame. By repeating this loop, the closest AF frame within a predetermined depth is selected as the main frame.

[0071] In step S612, similar to step S608, the system control unit 209 performs loop processing for all frames to select a main frame from the set AF frames.

[0072] In step S613, the system control unit 209 determines whether the AF frame is counted as a histogram peak. If so, the system control unit 209 proceeds to step S614; if not, the system control unit 209 repeats the loop process of step S612.

[0073] In step S614, the system control unit 209 determines whether or not the pupil has been detected. If the pupil has been detected, the system control unit 209 proceeds to step S616, and if not, the system control unit 209 proceeds to step S615.

[0074] In step S615, the system control unit 209 determines whether the AF frame being processed is closer to the face detection center than the currently selected main frame. If it is closer, the system control unit 209 proceeds to step S617 and updates the main frame. If not, the system control unit 209 repeats the loop process of step S612.

[0075] In step S616, the system control unit 209 determines whether the AF frame being processed is one whose coordinates are closer to the pupil detection center than the currently selected main frame. If it is closer, the system control unit 209 proceeds to step S617 and updates the main frame. If not, the system control unit 209 repeats the loop processing of step S612. By repeating this loop, the AF frame closest to the pupil center or the face center among the AF frames counted as histogram peaks is detected as the main frame.

[0076] In step S603, the system control unit 209 determines whether or not a torso has been detected by the subject detection unit 211. If a torso has been detected, the system control unit 209 proceeds to step S618, and if not, the system control unit 209 proceeds to step S624.

[0077] In step S618, the system control unit 209 creates a histogram in the body detection area and finds the histogram peak.

[0078] In step S619, the system control unit 209 determines whether the peak value of the histogram created in step S618 is equal to or greater than a predetermined ratio. If it is equal to or greater than the predetermined ratio, the system control unit 209 proceeds to step S620, and if it is less than the predetermined ratio, the system control unit 209 proceeds to step S626.

[0079] In step S626, the system control unit 209 selects the center of the body as the main frame.

[0080] In step S620, the system control unit 209 performs loop processing for all frames to select a main frame from the set AF frames, similar to step S608.

[0081] In step S621, the system control unit 209 determines whether the AF frame is counted as a histogram peak. If so, the system control unit 209 proceeds to step S622; if not, the system control unit 209 repeats the loop process of step S620.

[0082] In step S622, the system control unit 209 determines whether the AF frame being processed is a frame whose coordinates are closer to the body detection center than the currently selected main frame. If it is closer, the system control unit 209 proceeds to step S623; if not, the system control unit 209 repeats the loop processing of step S620.

[0083] In step S623, the system control unit 209 updates the main frame. By repeating this loop, the AF frame closest to the center of the moving object among the AF frames counted as the histogram peak is detected as the main frame.

[0084] In step S624, the system control unit 209 determines whether a main frame has been selected according to the flow described above by checking whether the main frame is at its initial value. If the main frame is at its initial value, the main frame has not been selected, and the process proceeds to step S625; if not, the main frame selection process ends.

[0085] In step S625, the system control unit 209 performs processing such as selecting a main frame in a predetermined area within the screen without using the detection information.

[0086] <Movement history saving process> FIG. 8A is a flowchart illustrating the operation of saving the motion history in step S403 of FIG.

[0087] First, in step S700, the system control unit 209 determines whether or not a person has been detected by the subject detection unit 211. If a person has been detected, the system control unit 209 proceeds to step S701, and if a person has not been detected, the system control unit 209 proceeds to step S704.

[0088] In step S701, the system control unit 209 performs a loop process the number of times equal to the number of detected people.

[0089] In step S702, the system control unit 209 selects an AF frame for the person being processed that has calculated the closest focus detection result within the inclusion area D determined in FIG. 4A.

[0090] This concept is shown in Figure 8B. For each detected subject in Figure 8B, an AF frame is searched for that calculates the closest focus detection information from within inclusion frame D (dotted line). If the subject falls outside the AF area during this process, the AF frame size and number may be increased so that as many AF frames as possible cover the detected subject. Also, if there are multiple subjects in action, as shown in Figure 8B, an object that can identify the main subject in a sports scene, such as a ball 850, may be detected, and the action subject that is close to the object may be identified as the main subject.

[0091] In step S703, the system control unit 209 stores the subject ID and the focus detection information (focus position information) of the AF frame selected in step S702 in the DRAM as a sub-subject movement history (hereinafter referred to as sub-history). This process is repeated for the number of detected people.

[0092] In step S704, the system control unit 209 stores the subject ID of the main subject and the focus detection result of the AF frame corresponding to the main subject in the DRAM as a movement history of the main subject (hereinafter referred to as the main history). Note that the main subject here refers to the subject corresponding to the AF frame (main frame) selected in the main frame selection process in FIG.

[0093] <Moving object prediction processing> FIG. 9 is a flowchart for explaining the prediction operation using the motion history stored in steps S703 and S704 of FIG. 8A.

[0094] In step S800, the system control unit 209 determines whether or not an action has been detected in the subject on the screen using the method described above. If an action has been detected, the system control unit 209 proceeds to step S801; if not, the system control unit 209 proceeds to step S808.

[0095] In step S801, the system control unit 209 determines whether the subject ID of the currently focused subject has changed when focusing on the subject whose action has been detected. If the subject ID has changed, the system control unit 209 proceeds to step S802; otherwise, the system control unit 209 proceeds to step S804.

[0096] This is because, although it is desirable to instantly shift the focus to track a subject that has taken action, it is necessary to predict the movement of that subject using the motion history information for that subject and adjust the focus accordingly. Therefore, the process proceeds to step S802.

[0097] In step S802, the system control unit 209 searches the sub-history stored in step S703 of Fig. 8A to determine whether there is a motion history that matches the subject ID of the action detected. If there is a sub-history with a matching subject ID, the system control unit 209 proceeds to step S803; otherwise, the system control unit 209 proceeds to step S804.

[0098] In step S803, the system control unit 209 updates the main history with the found sub-history. At this time, the conditions that restrict subject transitions are relaxed. For example, control that restricts focus tracking by detecting a change in focus detection results that is greater than a predetermined value is not performed.

[0099] In step S804, the system control unit 209 determines whether or not Num_a or more histories have been saved (whether or not a predetermined number or more have been stored). If Num_a or more histories have been saved, the system control unit 209 proceeds to step S805, and if Num_a or more histories have not been saved, the system control unit 209 proceeds to step S809.

[0100] In step S805, the system control unit 209 determines whether the movement is predictable. In this embodiment, prediction is deemed impossible when the change in the value of the focus detection information in the saved history is equal to or greater than a predetermined threshold (for example, a subject whose position is separated by 10 depths or more in terms of depth conversion) or when the variation in the value is equal to or greater than a predetermined value (for example, calculated by 3σ or the like from the focus detection result). This condition is determined based on factors that may cause a large error in the predicted position, which will be described later.

[0101] In step S806, the system control unit 209 calculates a predicted movement curve as shown in FIG.

[0102] In step S807, the system control unit 209 sets the predicted position of the subject based on the prediction curve obtained in step S806 as the target position for lens drive.

[0103] In step S808, the system control unit 209 determines whether or not Num_b or more histories have been saved. In this case, the magnitude relationship between Num_a and Num_b is set as follows:

[0104] Num_a < Num_b …(1) The reason for using the relationship in formula (1) above is that the subject at the time of action detection is likely to be in a state that is suitable for the timing of shooting sports scenes, the duration of the action is shorter than normal movement, and there is also a high possibility that the subject is moving in the distance direction, making it as easy to use prediction as possible. This increases the success rate when shooting difficult sports scenes.

[0105] In step S809, since the conditions are not predictable, the system control unit 209 sets the focus detection position as the target position for lens drive.

[0106] In this embodiment, by storing the history of sub-subjects that are candidates for the main subject, when the focus is shifted from the main subject to the sub-subject due to action detection, the movement of the sub-subject at the new location can be predicted and the focus can be tracked immediately, making it possible to provide an image that is in focus at the ideal timing in sports scenes, etc.

[0107] <Prediction curve calculation process> Figure 10 shows a prediction curve calculated using the main history saved in step S704 of Figure 8A. The vertical axis represents the subject image plane position calculated from the defocus amount calculated in step S401 of Figure 5, and the horizontal axis represents time. The larger the image plane position on the vertical axis, the greater the distance, and the history in the figure shows the tracking of a subject approaching the photographer (camera).

[0108] The focus adjustment process is performed periodically, and times T1 to T5 are the times when the focus adjustment process is performed. The predicted drive amount can be obtained by, for example, deriving a prediction curve using a batch least-squares method using past image plane positions and each focus detection time, and then calculating the image plane position (focus adjustment position) at the predicted future time based on this curve. In other words, the lens is driven at time T4, and a lens drive equivalent to defocus y is requested to achieve focus at time T5.

[0109] As described above, the method of this embodiment makes it possible to provide an image that is in focus at an appropriate timing in a sports scene or the like.

[0110] The disclosure of this specification includes the following focus adjustment device and method, imaging device, program, and storage medium.

[0111] (Item 1) an object detection means for detecting an object from an image; a motion detection means for detecting motion of the detected subject and storing a history of the motion; a focus adjustment means for focusing on a subject; a prediction means for predicting a focus adjustment position for the focus adjustment means to focus on the second object based on the history of the movement of the second object stored by the movement detection means, when the focus adjustment means shifts from a state in which the focus adjustment means focuses on the first object to a state in which the focus adjustment means focuses on the second object; A focus adjustment device comprising:

[0112] (Item 2) The focus adjustment device according to item 1, further comprising a state detection means for detecting a specific state of the subject from a feature amount of the subject, wherein the focus adjustment means transitions from a state in which the first subject is focused to a state in which the second subject is focused when the specific state of the second subject is detected.

[0113] (Item 3) 3. The focus adjustment device according to item 2, wherein the specific state is a state in which the subject has taken a specific action.

[0114] (Item 4) 4. The focus adjustment device according to any one of items 1 to 3, further comprising a setting unit that sets a plurality of focus detection areas within a screen, wherein the motion detection unit detects motion of the subject based on changes in focus position in the plurality of focus detection areas and stores a history of the motion.

[0115] (Item 5) 5. The focus adjustment device according to item 4, wherein the prediction means predicts a focus adjustment position for focusing on the second subject when a predetermined number or more of movement histories have been stored for the second subject.

[0116] (Item 6) 6. The focus adjustment device according to item 4 or 5, characterized in that if the difference or variation in focus position in the movement history of the second subject is equal to or greater than a predetermined value, the prediction means does not predict the focus adjustment position for focusing on the second subject.

[0117] (Item 7) The focus adjustment device described in any one of items 1 to 6, characterized in that the motion detection means stores the closest focus detection information among multiple focus detection areas for the subject as movement history information.

[0118] (Item 8) an imaging means for imaging a subject; A focus adjustment device according to any one of items 1 to 7, An imaging device comprising:

[0119] (Item 9) a subject detection step of detecting a subject from an image; a motion detection step of detecting motion of the detected subject and storing a history of the motion; a focus adjustment step of focusing on an object; a prediction step of predicting a focus adjustment position for focusing on the second object in the focus adjustment step based on the history of the movement of the second object stored in the movement detection step, when the focus adjustment step shifts from a state in which the first object is focused to a state in which the second object is focused; A focus adjustment method comprising:

[0120] (Item 10) 10. A program for causing a computer to execute each step of the focus adjustment method according to item 9.

[0121] (Item 11) A computer-readable storage medium storing a program for causing a computer to execute each step of the focus adjustment method according to item 9.

[0122] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0123] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0124] 100: lens device, 103: focus lens, 105: lens controller, 200: camera body, 201: image sensor, 204: AF signal processing unit, 209: system control unit, 210: lens communication unit, 211: subject detection unit

Claims

1. an object detection means for detecting an object from an image; a motion detection means for detecting motion of the detected subject and storing a history of the motion; a focus adjustment means for focusing on a subject; a prediction means for predicting a focus adjustment position for the focus adjustment means to focus on the second object based on the history of the movement of the second object stored by the movement detection means, when the focus adjustment means shifts from a state in which the focus adjustment means focuses on the first object to a state in which the focus adjustment means focuses on the second object; A focus adjustment device comprising:

2. 2. The focus adjustment device according to claim 1, further comprising: state detection means for detecting a specific state of the subject from a feature amount of the subject, wherein the focus adjustment means transitions from a state in which the first subject is focused to a state in which the second subject is focused when the specific state of the second subject is detected.

3. 3. The focus adjustment device according to claim 2, wherein the specific state is a state in which the subject has taken a specific action.

4. 2. The focus adjustment device according to claim 1, further comprising a setting unit that sets a plurality of focus detection areas within a screen, wherein the motion detection unit detects motion of the subject based on changes in focus position in the plurality of focus detection areas and stores a history of the motion.

5. 5. The focus adjustment device according to claim 4, wherein the prediction means predicts a focus adjustment position for focusing on the second subject when a predetermined number of movement histories or more have been stored for the second subject.

6. 5. The focus adjustment device according to claim 4, wherein when a difference or variation in focus position in the movement history of the second subject is equal to or greater than a predetermined value, the prediction means does not predict a focus adjustment position for focusing on the second subject.

7. 2. The focus adjustment device according to claim 1, wherein the movement detection means stores, as movement history information, focus detection information for the closest of a plurality of focus detection areas for the subject.

8. an imaging means for imaging a subject; A focus adjustment device according to any one of claims 1 to 7; An imaging device comprising:

9. a subject detection step of detecting a subject from an image; a motion detection step of detecting motion of the detected subject and storing a history of the motion; a focus adjustment step of focusing on an object; a prediction step of predicting a focus adjustment position for focusing on the second object in the focus adjustment step based on the history of the movement of the second object stored in the movement detection step, when the focus adjustment step shifts from a state in which the first object is focused to a state in which the second object is focused; A focus adjustment method comprising:

10. A program for causing a computer to execute each step of the focus adjustment method according to claim 9.

11. A computer-readable storage medium storing a program for causing a computer to execute each step of the focus adjustment method according to claim 9.

Images