Focus detection device and method, electronic apparatus, program, and storage medium

Abstract

To perform more stable focus adjustment when photographing a moving image.SOLUTION: A focus detection device includes: acquisition means for repeatedly acquiring an image signal and a focus detection signal for each frame from imaging means; detection means for detecting a main region of a predetermined subject on the basis of the image signal; generation means for synthesizing the focus detection signal of a predetermined number of frames including the latest frame to generate a synthesized focus detection signal; setting means for setting a main frame in an overlapping region where the main region detected in the predetermined number of frames including the latest frame overlaps; and focus detection means for detecting a focus state based on the synthesized focus detection signal of a region corresponding to the main frame.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to a focus detection apparatus and method, an electronic device, a program and a storage medium, and more particularly to a focus detection technique. [Background technology]

[0002] Conventionally, a technology has been disclosed for realizing high-speed phase difference focus adjustment (hereinafter referred to as "image plane phase difference AF") by providing an image sensor with a function for detecting the phase difference of a subject image, thereby eliminating the need for a sensor dedicated to focus detection. Patent Document 1 discloses a configuration for performing image plane phase difference AF by providing a pupil division function by decentering the sensitive area of ​​the light receiving unit with respect to the optical axis of an on-chip microlens in some light receiving elements (pixels) of the image sensor.

[0003] On the other hand, there is known an imaging device equipped with a video recording mode that records a video to a recording medium at the same time as displaying the output of the imaging pixels on a display in live view. Patent Document 2 proposes a method of performing focus detection calculation by adding image data of focus detection pixels stored in multiple past readouts in order to achieve both short-period refresh operation and focus detection for low-brightness subjects.

[0004] Furthermore, a technology has been established for identifying the area of ​​the main subject in live view and video recording and focusing using an AF method. In Patent Document 3, a mass within a predetermined depth adjacent to a plurality of AF frames is detected, and a main frame is selected from among them. In Patent Document 4, in addition to the process of detecting a mass within a predetermined depth, color information is added to improve the accuracy of identifying the main subject area.

[0005] Patent Document 5 discloses a method for detecting the pupils and other parts of the face, determining the reliability of the detection results for the detected parts, and setting the focus detection area in an area with high detection reliability. This makes it possible to focus on the main subject with a more accurate position. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2000-292686 A [Patent Document 2] JP 2014-071290 A [Patent Document 3] JP 2010-191073 A [Patent Document 4] JP 2015-041901 A [Patent Document 5] JP 2019-121860 A Summary of the Invention [Problem to be solved by the invention]

[0007] When the imaging device of Patent Document 2 is used for video shooting, it is possible to store image data of focus detection pixels in each frame, and perform focus detection calculation based on image data obtained by adding image data of focus detection pixels stored in the past multiple readouts. In this case, when the subject moves at high speed, the subject position differs between the multiple frames to be added, so that depending on the position of the focus detection area, focus detection is performed from an image to which information other than the main subject is added, which may cause an error in the calculation result. Therefore, if the lens is driven based on the obtained defocus information, there is a problem that it is difficult to drive the lens appropriately.

[0008] Furthermore, in the methods of Patent Documents 3 to 5, the area of ​​the main subject is detected with high accuracy and a focus detection area is set based on the detection result. However, when image data from multiple frames is added as in Patent Document 2, focus detection is not necessarily performed in an area suitable for focus detection.

[0009] The present invention has been made in consideration of the above problems, and an object of the present invention is to enable more stable focus adjustment during video shooting. [Means for solving the problem]

[0010] In order to achieve the above-mentioned object, the focus detection device of the present invention comprises an acquisition means for repeatedly acquiring an image signal and a focus detection signal for each frame from an imaging means, a detection means for detecting a predetermined main area of ​​a subject based on the image signal, a generation means for generating a composite focus detection signal by combining the focus detection signals of a predetermined number of frames including the most recent frame, a setting means for setting a main frame in an overlap area where the main areas detected in a predetermined number of frames including the most recent frame overlap, and a focus detection means for detecting a focus state based on the composite focus detection signal of the area corresponding to the main frame. Effect of the Invention

[0011] According to the present invention, more stable focus adjustment can be performed during video shooting. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing a configuration of an imaging apparatus according to an embodiment of the present invention. [Diagram 2] 2 is a schematic diagram of a pixel array of an image sensor according to the embodiment. [Diagram 3] 4 is a flowchart for explaining the operation of the imaging apparatus according to the embodiment. [Figure 4] 5A and 5B are conceptual diagrams illustrating subject detection results in the embodiment. [Diagram 5] 5 is a flowchart for explaining focus detection processing in the embodiment. [Figure 6] 5A and 5B are views showing examples of focus detection areas for the eyes, face, and body of a person according to an embodiment. [Figure 7] 5A to 5C are diagrams showing positions of people in each frame when photographing a person according to an embodiment. [Figure 8] 11 is a flowchart illustrating a main frame selection process in the embodiment. [Figure 9] 11 is a flowchart for explaining a process for setting a main frame selection area in the embodiment. [Figure 10]6A and 6B are conceptual diagrams illustrating a method for acquiring a main frame selection area in the embodiment. [Figure 11] 11 is a flowchart for explaining a main frame selection process that prioritizes a detection center in the embodiment. [Figure 12] 13 is a flowchart for explaining a main frame selection process that prioritizes reliability in a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0014] Imaging device configuration 1 is a block diagram showing the configuration of an electronic camera as an imaging device according to an embodiment of the present invention. The present invention is also applicable to, for example, lens-integrated digital cameras and video cameras. It can also be implemented in any electronic device equipped with a camera, such as a mobile phone, a personal computer (laptop, tablet, desktop, etc.), a game machine, etc.

[0015] As shown in FIG. 1, a lens apparatus 100 (interchangeable lens) is detachably (replaceably) attached to a digital camera body 200 via a mount portion (not shown) having an electrical contact unit 106.

[0016] The lens device 100 includes a photographing lens 101 including a zoom mechanism, an aperture and shutter 102 for controlling the amount of light, a focus lens 103, a motor 104 for driving the focus lens, and a lens controller 105.

[0017] In the digital camera body 200, the image sensor 201 is composed of a CCD or CMOS sensor, and receives reflected light from a subject that has passed through the photographing optical system of the lens apparatus 100, converts the light into signal charges according to the amount of incident light by photodiodes, and accumulates the signal charges. The signal charges accumulated in the photodiodes are sequentially read out from the image sensor 201 as voltage signals according to the signal charges based on drive pulses provided by a timing generator 208 in accordance with commands from a system control unit 209.

[0018] The A / D conversion unit 202 performs A / D conversion on the voltage signal output from the image sensor 201. The A / D conversion unit 202 includes a CDS circuit that removes output noise from the image sensor 201, and a non-linear amplification circuit that performs A / D conversion before the A / D conversion.

[0019] The digital camera body 200 further has an image processing unit 203, an AF signal processing unit 204, a format conversion unit 205, and a high-speed built-in memory 206 such as a random access memory (hereinafter referred to as "DRAM"). The DRAM 206 is used as a high-speed buffer for temporarily storing images, or as a working memory for compressing and expanding images. The image recording unit 207 consists of a removable recording medium such as a memory card and its interface.

[0020] A system control unit 209 controls the entire imaging device, including the shooting sequence. The digital camera body 200 also has a lens communication unit 210 that communicates between the digital camera body 200 and the lens device 100, a subject detection unit 211, and an image display memory 212 (VRAM). In addition to displaying images, the image display unit 213 also displays operation assistance and camera status, and during shooting, displays the shooting screen and focus detection area.

[0021] The digital camera body 200 also has various operating members for the user to operate the imaging device. The operating section 214 includes, for example, a menu switch for performing various settings such as the imaging function of the imaging device and settings for image playback, and an operation mode changeover switch between the imaging mode and the playback mode. The imaging mode switch 215 is a switch for selecting an imaging mode such as macro mode or sports mode, and the main switch 216 is a switch for turning on the power of the imaging device. The operation imaging switch 217 is a switch for instructing the start and end of video recording, and when pressed while video recording is not being performed, outputs an instruction to start video recording, and when pressed during video recording, outputs an instruction to end video recording.

[0022] Image sensor configuration FIG. 2 is a diagram showing an outline of the pixel array of the image sensor 201 in this embodiment, and shows the pixel array of a two-dimensional CMOS sensor used as the image sensor 201 in an area of ​​4 columns x 4 rows of imaging pixels (an area of ​​8 columns x 4 rows as the array of focus detection pixels).

[0023] In this embodiment, the pixel group 300 is composed of pixels arranged in 2 columns by 2 rows, and is covered with color filters in a Bayer array. In each pixel group 300, a pixel 300R having a spectral sensitivity of R (red) is arranged at the upper left position, a pixel 300G having a spectral sensitivity of G (green) is arranged at the upper right and lower left positions, and a pixel 300B having a spectral sensitivity of B (blue) is arranged at the lower right position. Furthermore, in the image sensor 201 of this embodiment, in order to perform focus detection using an image plane phase difference method, each pixel holds multiple photodiodes (photoelectric conversion units) for one microlens 315. In this embodiment, each pixel is composed of two photodiodes 311 and 312 arranged in 2 columns by 1 row.

[0024] The image sensor 201 is capable of acquiring an image signal and an AF signal by arranging a large number of pixel groups 300, each of which is made up of 2 columns x 2 rows of pixels (4 columns x 2 rows of photodiodes) as shown in FIG. 2, on the imaging surface.

[0025] In each pixel having such a configuration, the light beams passing through different pupil regions are separated by the microlens 315 and imaged on the photodiodes 311 and 312. Then, a signal (signal A+B) obtained by adding the signals from the two photodiodes 311 and 312 can be used as an imaging signal, and two signals (signal A, signal B) read out from each of the photodiodes 311 and 312 can be used as an AF signal (signal for focus detection). Note that the imaging signal and the AF signal may be read out separately, but in this embodiment, the following may be used in consideration of the processing load. That is, the imaging signal (signal A+B) and the AF signal (for example, signal A) of one of the photodiodes 311 and 312 are read out, and the difference is taken to obtain the other AF signal (for example, signal B) having parallax.

[0026] In this embodiment, each pixel has two photodiodes 311 and 312 for one microlens 315, but the number of photodiodes is not limited to two and may be more than two. The pupil division direction is not limited to the horizontal direction and may be the vertical direction. A plurality of pixels having different opening positions of the light receiving portion with respect to the microlens 315 may be provided. In other words, it is sufficient that the configuration results in two signals for phase difference detection, such as A signal and B signal. In addition, the present invention is not limited to the configuration in which all pixels have a plurality of photodiodes as shown in FIG. 2, and may be a configuration in which focus detection pixels as shown in FIG. 2 are discretely provided within normal pixels constituting the image sensor 201. A plurality of types of pixels divided by different division methods may be included in the same image sensor.

[0027] In addition, in this embodiment, the A signals and the B signals output from the multiple pixels are added in a direction perpendicular to the pupil division direction. In this embodiment, the A signals and the B signals read out at a predetermined cycle are stored in time series, and the A signals and the B signals of multiple frames are added for each corresponding pixel between frames to generate a pair of composite A signals and composite B signals used in focus detection calculation. The AF signal processor 204 collects the multiple composite A signals and multiple composite B signals obtained in this way to generate a pair of image signals (A image signal, B image signal). Then, a correlation calculation is performed between the A image signal and the B image signal to calculate a phase difference (hereinafter referred to as "image shift amount") which is the amount of shift between the A image signal and the B image signal, and the defocus amount and defocus direction of the photographing optical system are obtained from the calculated image shift amount. Note that the AF signal processor 204 in this embodiment calculates the defocus amount for each of multiple focus detection areas specified within the screen of the image sensor 201.

[0028] ●Video recording processing Next, the overall flow of the moving image shooting process performed by the image capture device of this embodiment will be described with reference to the flowchart of FIG.

[0029] When the photographer operates the main switch 216 to turn on the power of the imaging device, the video shooting process starts. First, in S201, the system control unit 209 checks the operation of each actuator and the image sensor 201 in the camera, initializes memory contents and execution programs, and executes shooting preparation operations.

[0030] Next, in S202, a live view mode is set to start periodic imaging operation of the image sensor 201. Then, in S203, a signal for live view (LV) display on the image display unit 213 (hereinafter referred to as an "LV signal"), a signal for recording moving images to be recorded in the image recording unit 207 (hereinafter referred to as a "moving image signal"), and an AF signal for performing focus detection in the AF signal processing unit 204 are read out.

[0031] In the next step S204, photometry processing is performed. Here, an automatically determined aperture value is determined according to the brightness of the subject measured by a photometry sensor (not shown) or the brightness of the subject obtained from the LV signal, that is, an aperture value that will make the image signal have a level suitable for display when the image signal is periodically read out. Alternatively, the aperture value may be manually set by the user using an operation member included in the operation unit 214. Then, aperture control information according to the obtained aperture value is sent to the lens controller 105, and the aperture opening diameter of the aperture and shutter 102 is set based on the aperture control information. Also, the exposure time in each frame may be controlled based on the obtained brightness. For example, the exposure time can be controlled by controlling the reset timing of the image sensor 201.

[0032] In S205, the state of the video capture switch 217 is checked. If video capture has been instructed, the process proceeds to S206, where the video image data obtained by processing the video signal read out in S203 by the A / D conversion unit 202, image processing unit 203, and format conversion unit 205 is recorded, and the process proceeds to S207. If video capture has not been instructed, the process proceeds to S207 without recording the video image data.

[0033] In S207, the subject detection unit 211 performs subject detection using the LV signal read out in S203, and obtains subject detection information. In this embodiment, a person, a dog, a wild bird, or other animal is detected as the subject, and a main area within the subject is detected. Note that the main area in this embodiment refers to the pupil, face, and body area of ​​a person or animal. As a detection method for these, for example, learning by deep learning, image processing, and the like, which are known technologies, can be used.

[0034] In S208, subject detection information by subject detection unit 211 is temporarily stored as history for each type of subject and each main area.

[0035] FIG. 4 is a conceptual diagram showing the subject detection result by the subject detection unit 211, and FIG. 4(a) shows an example in which the subject is a person, FIG. 4(b) shows an example in which the subject is a dog, and FIG. 4(c) shows an example in which the subject is a wild bird. In FIG. 4, 401a, 402a, and 403a show pupil detection regions, 401b, 402b, and 403b show face detection regions, and 401c, 402c, and 403c show body detection regions, and these regions are main regions. The subject detection information stored in S208 includes coordinate information and region size of each main region of each subject. In this embodiment, the main region of each subject is defined as a square region of the minimum size including the pupil, face, and body, but may be defined as a region of other shapes such as a rectangle, a circle, or an ellipse.

[0036] Next, in S209, focus detection processing is performed to detect the defocus amount and reliability, and the process proceeds to S210. Details of the focus detection processing in S209 will be described later with reference to FIG.

[0037] In S210, a main frame selection process is performed using the subject detection information stored in S208 and the focus detection result obtained in S209, and the process proceeds to S211. Details of the main frame selection process in S210 will be described later with reference to Fig. 8. In S211, the AF operation is performed as described with reference to Fig. 2.

[0038] Thereafter, the process returns to S203, and the processes of S203 to S211 are periodically repeated.

[0039] Focus detection processing Next, the focus detection process in S209 will be described with reference to FIG. First, in S401, the AF signals (signals A and B) read from the image sensor 201 in S203 are stored in the DRAM 206 in sequence. It is assumed that the DRAM 206 is capable of storing AF signals for a predetermined number of frames. In S402, of the AF signals stored in S401, the AF signals for a predetermined number of frames going back from the current frame are added for each of the signals A and B (hereinafter referred to as "frame addition processing") to generate a pair of composite signals A and B used in focus detection calculation. Hereinafter, the generated pair of composite signals A and B will be referred to as "composite AF signals" (composite focus detection signals). This processing can reduce the effects of noise in each AF signal.

[0040] In S404, the focus detection area is set using the latest subject detection information acquired from the subject detection unit 211 in S207.

[0041] FIG. 6 is a diagram showing a method of setting the focus detection area, and shows a case where a person is detected as the subject. The focus detection area is preferentially arranged in the main area of ​​the subject. As described above, the main area is an area showing the pupil, face, and body of a person, and it is assumed that a plurality of main areas have been detected. In FIG. 6(a), as in FIG. 4(a), the pupil, face, and body are detected, and the pupil detection area 401a, face detection area 401b, and body detection area 401c are shown. In this case, the type of subject, such as a person or an animal, the center coordinates of each detected main area, and the horizontal and vertical sizes are output from the subject detection unit 211 as subject detection information. Based on this subject detection information, as shown in FIG. 6(b), the size of one AF frame is set to the size of the smallest main area (here, the pupil detection area) among the detected main areas, and a plurality of focus detection areas 601 are arranged at equal intervals in a rectangular area 600 (partial area) that includes all the main areas.

[0042] In S405, a composite AF signal corresponding to one of the multiple focus detection areas 601 set in S404 is obtained from the composite AF signals generated in S402, and the process proceeds to S406. In S406, a filter process is performed to extract signal components in a predetermined frequency band from the composite AF signal obtained in S405, and the process proceeds to S407.

[0043] In S407, the correlation amount of the composite AF signal filtered in S406 is calculated, and in the next S408, the correlation change amount is calculated from the correlation amount calculated in S407, and the process proceeds to S409. In S409, the image shift amount (focus state) is calculated from the correlation change amount calculated in S408, and the process proceeds to S410. In S410, a reliability indicating how reliable the image shift amount calculated in S409 is is obtained, and in S411, the image shift amount is converted into a defocus amount (focus state).

[0044] In S412, it is determined whether or not the processes in S405 to S411 have been performed for all of the focus detection areas 601 set in S404. If there are any unprocessed focus detection areas 601, the process returns to S405 and repeats the above processes. If all focus detection areas 601 have been processed, the focus detection process ends.

[0045] Main frame selection process Next, the main frame selection process in S210 will be described. In this embodiment, when the frame addition process is performed in S402, the addition process of the AF signals for the past three frames including the latest frame is performed sequentially. Note that the number of frames to be added may be dynamically changed according to the signal level of the composite AF signal.

[0046] Figure 7 is a diagram showing the position of a person in each frame when the person is photographed. In the same scene, 701 indicates the person when photographed in the latest frame, and similarly 702 and 703 indicate the person in the previous frame. 704 indicates the face detection area in the latest frame. Although not shown in Figure 7, multiple focus detection areas are arranged around the entire person, the subject, including the body, as shown in Figure 6(b).

[0047] When frame addition processing of AF signals is performed, the above-mentioned three frames of AF signals are combined. At this time, if the subject moves quickly, a focus detection area corresponding to a combined AF signal in which an AF signal in a person's area and an AF signal in a background area are combined is generated among the combined AF signals corresponding to the multiple focus detection areas set. Hereinafter, a focus detection area corresponding to a combined AF signal in which AF signals of different types of areas are combined is called a "mixed area." When focus detection processing is performed on this mixed area, there is a possibility that the defocus amount between the person and the background is calculated. In that case, even though the area is a region in which a person's face is detected in the latest frame, an error is added to the focus detection result by performing frame addition processing. Therefore, when AF control is performed using the focus detection result of the focus detection area in the mixed area, the focus may shift from the person to the background, resulting in an unstable AF operation. Therefore, in this embodiment, a main frame selection process is performed as described below.

[0048] The main frame selection process performed in S210 will be described below with reference to the flowchart in Fig. 8. Note that, in this description, it is assumed that, among the main regions, the pupils have the highest priority, the face has the next highest priority, and the body has the lowest priority.

[0049] First, in S601, as a preparation for main frame selection, the main frame is set to an arbitrary initial position. In S602, it is determined whether or not the subject's eyes have been detected by the subject detection unit 211. If the eyes have been detected, the process proceeds to S603, where a main frame selection area is set, and if not, the process proceeds to S607. The process of setting the main frame selection area in S603 will be described later with reference to FIG. 9. Then, in the following S604, it is determined whether or not there is a focus detection area (main frame candidate area) in the main frame selection area set in S603, and if there is, the process proceeds to S605, and if there is not, the process proceeds to S607. In S605, main frame selection is performed with priority given to the detection center, and the process proceeds to S606. The process of selecting the main frame with priority given to the detection center in S605 will be described later with reference to FIG. 11.

[0050] In S606, it is determined whether the reliability of the main frame selected in S605 is higher than a predetermined threshold. If it is determined that the reliability is equal to or higher than the threshold, the process proceeds to S616, and if it is not determined that the reliability is equal to or higher than the threshold, the process proceeds to S607. The purpose of the process in S606 is to determine whether the defocus amount of the selected main frame is within a predetermined variation.

[0051] On the other hand, if it is determined in S602 that an eye has not been detected, the main frame cannot be selected from the eye detection area, and therefore in S607, it is determined whether or not the face of the subject has been detected by the subject detection unit 211. If a face has been detected, the process proceeds to S608, and if not, the process proceeds to S612. Note that the process of setting the main frame selection area in S608 will be described later with reference to FIG. 9. Then, in the following S609, it is determined whether or not there is a focus detection area (main frame candidate area) in the main frame selection area set in S608, and if there is, the process proceeds to S610, and if there is not, the process proceeds to S612. In S610, main frame selection with detection center priority is performed, and the process proceeds to S611. Note that the process of selecting the main frame with detection center priority in S610 will be described later with reference to FIG. 11.

[0052] In S611, it is determined whether the reliability of the main frame selected in S610 is higher than a predetermined threshold. If it is determined that the reliability is equal to or higher than the threshold, the process proceeds to S616, and if it is not determined that the reliability is equal to or higher than the threshold, the process proceeds to S612. The purpose of the process in S611 is to determine whether the detected defocus amount of the selected main frame is within a predetermined variation.

[0053] On the other hand, if it is determined in S607 that a face has not been detected, the main frame cannot be selected from the face detection area, and therefore in S612, it is determined whether or not the body of the subject has been detected by the subject detection unit 211. If the body has been detected, the process proceeds to S613, and if not, the process proceeds to S616. The process of setting the main frame selection area in S613 will be described later with reference to FIG. 9. Then, in the following S614, it is determined whether or not there is a focus detection area (main frame candidate area) in the main frame selection area set in S613, and if there is, the process proceeds to S615, and if there is not, the process proceeds to S616. In S615, main frame selection with detection center priority is performed, and the process proceeds to S616. The process of selecting the main frame with detection center priority in S615 will be described later with reference to FIG. 11.

[0054] In S616, it is determined whether the main frame is in its initial position to confirm whether the main frame has finally been set to the eye, face, or body area. If the main frame is not in its initial position, it is determined that the main frame has been selected, and if it is in its initial position, the process proceeds to S617, where the position of the main frame is released and the main frame selection process ends. In S617, a method of selecting the main frame in a predetermined area within the screen without using subject detection information is conceivable, but since this is not a key technology of the present invention, details will be omitted.

[0055] By the above processing, if a main frame is selected in S210, AF operation is performed based on the defocus amount of the selected main frame in S211, and if a main frame is not selected, AF operation is performed, for example, by a conventional method.

[0056] 9 is a flowchart showing the process of setting the main frame selection region performed in S603, S608, and S613. In this embodiment, when frame addition of AF signals is performed, the mixed region is excluded from the main frame selection region.

[0057] In S801, information on the number of frames to which the AF signal is to be added (3 frames in this embodiment) is obtained, and in S802, the frame ID (=k) of the current frame (closest frame) is obtained. In S803, subject detection information of the frame to which the AF signal is to be added is obtained from the history of subject detection information of each frame stored in S208. In the following S804, the main frame selection area is obtained using the subject detection information obtained in S803. Note that how to obtain the main frame selection area will be described later with reference to FIG. 10.

[0058] In S806, for each of the multiple focus detection areas set in S404, it is determined whether or not it is within the main frame selection area determined in S804, and if it is within the main frame selection area, in S807, the focus detection area is stored as a main frame candidate area. If it is not within the main frame selection area, the process proceeds to S808, where it is determined whether or not all focus detection areas have been judged, and if all focus detection areas have been judged, the process returns to the process in FIG. 8, and if there is any focus detection area that has not been judged, the process returns to S806 and the above process is repeated.

[0059] Next, the method of determining the main frame selection area in S804 will be described with reference to Fig. 10. Fig. 10(a) is a diagram showing the results of detecting the eyes of a person in each frame when the person is photographed. Similarly, Fig. 10(b)2 is a diagram showing the trajectory of the results of detecting the face of a person in each frame when the person is photographed. In Fig. 10(a) and Fig. 10(b), 1001a and 1001b indicate the main area in the latest frame, and similarly 1002a and 1002b indicate the main area in the previous frame, and 1003a and 1003b indicate the main area in two frames before.

[0060] Then, the main frame selection area is obtained from the overlapping area of ​​the main areas extracted from each frame to be added. As shown in Fig. 10(a), when a person is moving quickly and the pupil detection area is small, there is no overlapping area of ​​the main areas. Therefore, the center coordinates of the pupil detection area of ​​the latest frame are set as the center coordinates of the main frame selection area, and 0 is set as the size of the main frame selection area.

[0061] 10(b), when the face detection area is relatively large, an overlapping area 1004 of the main areas exists between the frames. Therefore, the center coordinates of the overlapping area 1104 are set as the center coordinates of the main frame selection area, and the horizontal and vertical lengths of the overlapping area 1104 are set as the size of the main frame selection area.

[0062] As with the face, the overlapping areas of the main areas for the pupils and body are determined as the main frame selection areas.

[0063] 10C is a diagram showing a face portion among the multiple focus detection areas set in S404. In S806, a determination is made based on whether or not the center coordinates of each focus detection area 601 are included in the main frame selection area 1005. Then, in S604, S609, and S614, a determination is made as to whether or not there is a focus detection area included in the main frame selection area 1005.

[0064] Next, the main frame selection process in S605, S610, and S615 will be described with reference to the flowchart in Fig. 11. Here, the main frame selection process with priority given to the detection center will be described.

[0065] In S902, for one of the main frame candidate areas selected in either S604, S608, or S613, it is determined whether its center is closer to the center of the main frame selection area set in S804 than the currently set main frame. If it is closer, the process proceeds to S903, and the main frame candidate area being processed is updated as the main frame. If it is farther, the process proceeds to S904. As described above, the determination process based on the distance between the center of the main frame selection area and the center of the main frame candidate area is performed on all main frame candidate areas (until YES is obtained in S904), and then the process returns to FIG. 8.

[0066] As described above, according to this embodiment, when focus detection is performed using AF signals of multiple frames, it is possible to select an appropriate area for focus detection even for a subject moving within the screen, and stable focus adjustment is possible. This makes it possible to perform focus adjustment using AF signals of a more stable focus detection area, while avoiding the selection of a focus detection area in which AF signals corresponding to distant and near subjects are added by frame addition when the subject moves quickly.

[0067] In this embodiment, an example has been described in which the subject is primarily a person, with the eyes, face, and body detected as the main subject. However, as shown in FIG. 4, the effects of the present invention can also be obtained by implementing a similar method when the detected subject is an animal, with the eyes, face, and body detected as the main subject.

[0068] In the above embodiment, the defocus amounts are detected for all focus detection areas set in S404, and the defocus amount corresponding to the main frame is selected from the detected defocus amounts, but the present invention is not limited to this. For example, after the main frame is selected, the defocus amount of the focus detection area corresponding to the main frame may be calculated.

[0069] <Modification> In the above-described embodiment, when focus detection is performed using a composite AF signal obtained by adding together AF signals from multiple frames, the focus detection result of a main frame candidate area close to the center of the main frame selection area is selected in order to avoid focus detection areas where errors are likely to occur in the focus detection result. However, in cases where it is necessary to take into account the deviation of the subject detection position in each frame, it is also possible to select a process of selecting a main frame candidate area with a higher reliability of the defocus amount as the main frame.

[0070] FIG. 12 is a flowchart showing a main frame selection process in this modified example in which a main frame candidate region having a high reliability of a detected defocus amount is given priority, and is performed in place of the process shown in FIG. In S1003, the reliability of one of the main frame candidate areas selected in either S604, S608, or S613 is compared with the reliability of the currently set main frame. If the reliability of the main frame candidate area is higher than the reliability of the currently set main frame, the process proceeds to S1004, and the main frame candidate area being processed is updated as the main frame. On the other hand, if the reliability is the same as or lower than the currently set main frame, the process proceeds to S1005. The above processing is performed on the main frame candidate areas (until YES is obtained in S1004), and the process returns to FIG. 8.

[0071] The present invention has been described above in detail based on preferred embodiments thereof, but the present invention is not limited to these specific embodiments, and various forms that do not deviate from the gist of the present invention are also included in the present invention.

[0072] <Other embodiments> The present invention may be applied to a system made up of a plurality of devices, or to an apparatus made up of a single device.

[0073] The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0074] <Summary> The disclosure of this embodiment includes the following configuration.

[0075] (Item 1) an acquisition means for repeatedly acquiring an image signal and a focus detection signal for each frame from the imaging means; a detection means for detecting a main area of ​​a predetermined subject based on the image signal; a generating means for generating a composite focus detection signal by combining the focus detection signals of a predetermined number of frames including an immediately preceding frame; a setting means for setting a main frame in an overlapping area where the main areas detected in a predetermined number of frames including the most recent frame overlap; a focus detection means for detecting a focus state based on the composite focus detection signal of an area corresponding to the main frame; A focus detection device comprising: (Item 2) the focus detection means detects a focus state for each of a plurality of focus detection areas obtained by dividing a partial area including the main area of ​​the most recent frame, based on the composite focus detection signal; the setting means sets one of the focus detection areas included in the overlapping area as the main frame, out of the plurality of focus detection areas; 2. The focus detection device according to item 1, characterized in that a focus state corresponding to the focus detection area corresponding to the main frame is selected from among the focus states of the plurality of focus detection areas. (Item 3) 3. The focus detection device according to item 2, wherein the setting means sets the main frame based on the distance between the center of a focus detection area included in the overlapping area and the center of the overlapping area. (Item 4) The focus detection device described in item 3, characterized in that the setting means sets, among the focus detection areas included in the overlapping area, the focus detection area having a center closest to the center of the overlapping area as the main frame. (Item 5) The focus detection means further determines a reliability of the focus state, 3. The focus detection device according to item 2, wherein the setting means sets the main frame based on the reliability. (Item 6) 6. The focus detection device according to item 5, wherein the focus detection means sets the focus detection area with the highest reliability among the focus detection areas included in the overlapping area as the main frame. (Item 7) The detection means detects a plurality of predetermined main regions of the subject, The plurality of key areas are prioritized; The setting means sets the main frame in order from the main region with the highest priority, and when the main frame cannot be set, sets the main frame for the main region with the next highest priority. 7. The focus detection device according to any one of items 2 to 6, (Item 8) 8. The focus detection device according to item 7, wherein the predetermined subjects include people and animals. (Item 9) 9. The focus detection device according to item 8, wherein the plurality of main regions include, in order of priority, eye, face, and body regions. (Item 10) The focus detection device described in any one of items 7 to 9, characterized in that the partial areas are divided by equally spaced focus detection areas having the smallest size of the main areas detected by the detection means. (Item 11) A focus detection device according to any one of items 1 to 10, a focus adjustment means for performing focus adjustment based on the focus state of the area corresponding to the main frame; 1. An electronic device comprising: (Item 12) 12. The electronic device according to item 11, further comprising the imaging means. (Item 13) an acquisition step in which an acquisition means repeatedly acquires an image signal and a focus detection signal for each frame from the imaging means; a detection step in which a detection means detects a predetermined main area of ​​a subject based on the image signal; a generating step in which a generating means generates a composite focus detection signal by combining the focus detection signals of a predetermined number of frames including an immediately preceding frame; a setting step in which a setting means sets a main frame to an overlapping area where the main areas detected in a predetermined number of frames including the most recent frame overlap; a focus detection step in which a focus detection means detects a focus state based on the composite focus detection signal of an area corresponding to the main frame; 13. A focus detection method comprising: (Item 14) A program for causing a computer to function as each of the means of the focus detection device according to any one of items 1 to 10. (Item 15) Item 15. A computer-readable storage medium storing the program according to item 14.

[0076] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0077] 100: lens device, 101: photographing lens, 102: aperture and shutter, 103: focus lens, 104: motor, 105: lens controller, 106: electrical contact unit, 200: digital camera body, 201: imaging element, 202: A / D conversion unit, 203: image processing unit, 204: AF signal processing unit, 205: format conversion unit, 206: DRAM, 207: image recording unit, 208: timing generator, 209: system control unit, 210: lens communication unit, 211: subject detection unit, 212: image display memory, 213: image display unit, 214: operation unit, 215: shooting mode switch, 216: main switch, 217: video shooting switch, 601: focus detection area, 1005: main frame selection area

Claims

1. an acquisition means for repeatedly acquiring an image signal and a focus detection signal for each frame from the imaging means; a detection means for detecting a main area of ​​a predetermined subject based on the image signal; a generating means for generating a composite focus detection signal by combining the focus detection signals of a predetermined number of frames including an immediately preceding frame; a setting means for setting a main frame in an overlapping area where the main areas detected in a predetermined number of frames including the most recent frame overlap; a focus detection means for detecting a focus state based on the composite focus detection signal of an area corresponding to the main frame; A focus detection device comprising:

2. the focus detection means detects a focus state for each of a plurality of focus detection areas obtained by dividing a partial area including the main area of ​​the most recent frame, based on the composite focus detection signal; the setting means sets one of the focus detection areas included in the overlapping area as the main frame, out of the plurality of focus detection areas; 2. The focus detection device according to claim 1, wherein a focus state corresponding to the focus detection area corresponding to the main frame is selected from among the focus states of the plurality of focus detection areas.

3. 3. The focus detection device according to claim 2, wherein the setting means sets the main frame based on a distance between a center of a focus detection area included in the overlapping area and a center of the overlapping area.

4. 4. The focus detection device according to claim 3, wherein the setting means sets, as the main frame, a focus detection area having a center closest to a center of the overlapping area among the focus detection areas included in the overlapping area.

5. The focus detection means further determines a reliability of the focus state, 3. The focus detection device according to claim 2, wherein the setting means sets the main frame based on the reliability.

6. 6. The focus detection device according to claim 5, wherein the focus detection means sets, as the main frame, a focus detection area having the highest reliability among the focus detection areas included in the overlapping area.

7. The detection means detects a plurality of predetermined main regions of the subject, The plurality of key areas are prioritized; The setting means sets the main frame in order from the main region with the highest priority, and when the main frame cannot be set, sets the main frame for the main region with the next highest priority.

3. The focus detection device according to claim 2.

8. 8. The focus detection device according to claim 7, wherein the predetermined subjects include people and animals.

9. 9. The focus detection device according to claim 8, wherein the plurality of main regions include, in order of priority, eye, face, and body regions.

10. 8. The focus detection device according to claim 7, wherein the partial regions are divided by arranging at equal intervals focus detection regions having a size of the smallest main region among the main regions detected by the detection means.

11. A focus detection device according to any one of claims 1 to 10, a focus adjustment means for performing focus adjustment based on the focus state of the area corresponding to the main frame; 1. An electronic device comprising:

12. 12. The electronic device according to claim 11, further comprising the imaging means.

13. an acquisition step in which an acquisition means repeatedly acquires an image signal and a focus detection signal for each frame from the imaging means; a detection step in which a detection means detects a predetermined main area of ​​a subject based on the image signal; a generating step in which a generating means generates a composite focus detection signal by combining the focus detection signals of a predetermined number of frames including an immediately preceding frame; a setting step in which a setting means sets a main frame to an overlapping area where the main areas detected in a predetermined number of frames including the most recent frame overlap; a focus detection step in which a focus detection means detects a focus state based on the composite focus detection signal of an area corresponding to the main frame; 13. A focus detection method comprising:

14. A program for causing a computer to function as each of the means of the focus detection apparatus according to any one of claims 1 to 10.

15. A computer-readable storage medium storing the program according to claim 14.

Images