Imaging device and control method therefor

Abstract

To provide an imaging device with which it is possible to accurately adjust the focus to a subject without the need for accurately framing to the subject by optionally selected AF.SOLUTION: The present invention comprises: a focus detection step for outputting a focus detection result on the basis of an image signal; a setting step for setting the region of an AF frame designated by a user to within the image signal; an acquisition step for acquiring the region of a subject; and a control step for controlling the movement of a focus lens on the basis of the result of focus detection. In the control step, the drive of the focus lens based on a focus detection result corresponding to the AF frame is not performed, when there is a transition from a state where the region of the subject exists in the region of the AF frame to a state where the region of the subject no longer exists in the region of the AF frame.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an imaging device with a focus adjustment function. [Background technology]

[0002] Conventionally, as a means for selecting an area to be focused, there is a method of arbitrary selection AF, which focuses on an arbitrary area (AF frame) within a photographic screen represented by a rectangle or the like.

[0003] With manual selection AF, even if there are multiple subjects, such as people, within the shooting screen, the user can focus on any subject they want by setting the position and size of the AF frame so that the subject they want to focus on is included in the AF frame, or by moving the imaging device itself so that the subject is included in a pre-set AF frame.

[0004] However, when photographing a fast-moving subject, it can be difficult to keep the AF frame correctly positioned on the subject, resulting in the subject being out of focus. Conversely, if an obstacle gets inside the AF frame, the AF will focus on that obstacle, resulting in the subject not being in focus as intended.

[0005] Patent Document 1 has a function for detecting defocus over a range wider than a predetermined area, performs clustering based on the defocus value, determines whether there is an obstacle subject near the outside of the predetermined area, and controls the lens so as not to focus on the obstacle.

[0006] Furthermore, in Patent Document 2, a subject recognition function is provided, and if the defocus amount obtained from the subject recognition area is equal to or less than a predetermined value, the defocus amount is used to control the focus lens to focus on the subject, and if the defocus amount obtained from the subject recognition area is equal to or more than a predetermined value, the focus lens is controlled using the defocus amount obtained from an area other than the subject recognition area, such as an AF frame.Regarding the display of the shooting screen, Patent Document 3, similar to Patent Document 2, hides the AF frame and displays a frame indicating the subject recognition area when the defocus amount obtained from the subject recognition area is equal to or less than a predetermined value, and displays the AF frame and hides the frame indicating the subject recognition area when the defocus amount obtained from the subject recognition area is equal to or more than a predetermined value. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2002-207156 [Patent Document 2] Patent No. 5447549 [Patent Document 3] Patent Publication No. 2021-21857 Summary of the Invention [Problem to be solved by the invention]

[0008] However, with the conventional technology disclosed in Patent Document 1, if the correct subject is outside the specified area due to framing, that position cannot be selected as the AF position, resulting in the problem of focusing on a different position.

[0009] Furthermore, with the conventional technologies disclosed in Patent Documents 2 and 3, even when a user intends to focus on a subject using an AF frame, if a subject unintended by the user is recognized and the defocus amount obtained from that subject recognition area is small, the frame indicating that subject recognition area is displayed, the AF frame is hidden, and the focus lens is controlled using the defocus amount obtained from the subject recognition area, making it impossible to focus on the subject intended by the user. Furthermore, if manual AF is performed without using the conventional technology to avoid this, it is necessary to accurately and continuously focus the subject to be focused on in the AF frame. However, if the subject moves out of the AF frame due to reasons such as the subject being small in size or moving quickly, it is impossible to focus on the subject.

[0010] In view of the above, an object of the present invention is to provide a method for enabling a user to focus on a subject that the user desires to focus on, even if the user is unable to correctly capture the subject in an AF frame that the user has set. [Means for solving the problem]

[0011] In order to achieve the above object, the present invention provides a control method for an imaging device having a signal acquisition means for acquiring an image signal output from an imaging element that captures an image via an imaging optical system including a focus lens, the control method comprising: a focus detection process for outputting a focus detection result based on the image signal; a setting process for setting an AF frame area specified by a user within the image signal; an acquisition process for acquiring a subject area; and a control process for controlling movement of the focus lens based on the focus detection result, wherein the control process is characterized in that, when the subject area changes from being within the AF frame area to being no longer within the AF frame area, the focus lens is not driven based on the focus detection result corresponding to the AF frame. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide a method for focusing on a subject in an imaging device having an automatic focus adjustment function without the user having to accurately align the AF frame with the subject on which the user wants to focus. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram of an imaging apparatus according to an embodiment. [Figure 2] FIG. 2 is a schematic diagram of a pixel array in the embodiment. [Figure 3] 1A and 1B are a schematic plan view and a schematic cross-sectional view of a pixel in an embodiment. [Figure 4] FIG. 2 is a schematic diagram illustrating pixels and pupil division in the embodiment. [Figure 5] FIG. 2 is a schematic explanatory diagram of an image sensor and pupil division in the embodiment. [Figure 6] 5A and 5B are diagrams illustrating a schematic relationship between a defocus amount and an image shift amount in an embodiment. [Figure 7] 1 is an overall control flow in an embodiment. [Figure 8] FIG. 10 is a schematic diagram illustrating a defocus region setting in the embodiment. [Figure 9] 10 is a flowchart showing a main subject region selection flow in which a subject detection class is given priority in an embodiment. [Figure 10] This is a flow chart for selecting a main subject area with priority given to the subject recognition position. [Figure 11] FIG. 10 is a diagram illustrating a scene where the subject recognition area and the area within the AF frame overlap. [Figure 12] 10A and 10B are diagrams showing display examples of whether or not the subject recognition area overlaps with the area within the AF frame. [Figure 13] 10A and 10B are diagrams illustrating the relationship with the distance measurement frame in overlap determination. [Figure 14] FIG. 10 is a diagram illustrating a scene in which a part with a high priority is determined to be overlapped. [Figure 15] 10A and 10B are diagrams illustrating an example of display of scenes in which parts with high priority are determined to overlap. [Figure 16] FIG. 10 is a diagram for explaining class classification. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0015] Example 1 (Configuration of imaging device) First, with reference to FIG. 1, the configuration of a camera system according to a first embodiment of the present invention will be described. FIG. 1 is a block diagram of a camera system 10 (a single-lens reflex digital camera with interchangeable lenses) according to this embodiment. The camera system 10 is a camera system having a lens unit 100, which is an interchangeable lens, and a camera body 120, which is an imaging device. The lens unit 100 is detachably attached to the camera body 120 via a mount M, which is indicated by a dotted line in FIG. 1. However, this embodiment is not limited to this, and can also be applied to a digital camera in which the lens unit (imaging optical system) and the camera body are integrally configured. Furthermore, this embodiment is not limited to a digital camera, and can also be applied to other imaging devices, such as a video camera.

[0016] Lens unit 100 has an optical system consisting of first lens group 101, aperture 102, second lens group 103, focus lens group (hereinafter simply referred to as "focus lens") 104, and a drive / control system. Thus, lens unit 100 is a photographing lens (image pickup optical system) that includes focus lens 104 and forms a subject image.

[0017] The first lens group 101 is disposed at the tip of the lens unit 100 and is held so as to be able to move back and forth in the optical axis direction OA. The diaphragm 102 adjusts the amount of light during shooting by adjusting its aperture diameter, and also functions as a shutter for adjusting the exposure time during still image shooting. The diaphragm 102 and the second lens group 103 are movable together in the optical axis direction OA, and a zoom function is achieved in conjunction with the forward and backward movement of the first lens group 101. The focus lens 104 is movable in the optical axis direction OA, and the subject distance (focusing distance) at which the lens unit 100 focuses changes depending on its position. Controlling the position of the focus lens 104 in the optical axis direction OA enables focus adjustment (focus control) to adjust the focusing distance of the lens unit 100.

[0018] The drive / control system includes a zoom actuator 111, an aperture actuator 112, a focus actuator 113, a zoom drive circuit 114, an aperture drive circuit 115, a focus drive circuit 116, a lens MPU 117, and a lens memory 118. The zoom drive circuit 114 uses the zoom actuator 111 to drive the first lens group 101 and the third lens group 103 in the optical axis direction OA, thereby controlling the angle of view of the optical system of the lens unit 100 (performing a zoom operation). The aperture drive circuit 115 uses the aperture actuator 112 to drive the aperture 102, thereby controlling the aperture diameter and opening / closing operation of the aperture 102. The focus drive circuit 116 uses the focus actuator 113 to drive the focus lens 104 in the optical axis direction OA, thereby controlling the focal length of the optical system of the lens unit 100 (performing focus control). The focus drive circuit 116 also functions as a position detector that uses the focus actuator 113 to detect the current position (lens position) of the focus lens 104.

[0019] The lens MPU (processor) 117 performs all calculations and controls related to the lens unit 100 and controls the zoom drive circuit 114, aperture drive circuit 115, and focus drive circuit 116. The lens MPU 117 is also connected to the camera MPU 125 via a mount M to exchange commands and data. For example, the lens MPU 117 detects the position of the focus lens 104 and notifies the camera MPU 125 of lens position information in response to a request. This lens position information includes information such as the position of the focus lens 104 in the optical axis direction OA, the position and diameter of the exit pupil in the optical axis direction OA when the optical system is not moving, and the position and diameter of the lens frame that limits the light beam from the exit pupil in the optical axis direction OA. The lens MPU 117 also controls the zoom drive circuit 114, aperture drive circuit 115, and focus drive circuit 116 in response to a request from the camera MPU 125. The lens memory 118 stores optical information required for automatic focus adjustment (AF control). The camera MPU 125 controls the operation of the lens unit 100 by executing programs stored in, for example, an internal nonvolatile memory or the lens memory 118 .

[0020] The camera body 120 has an optical low-pass filter 121, an image sensor 122, and a drive / control system. The optical low-pass filter 121 and the image sensor 122 function as an image sensor (image capture means) that photoelectrically converts an object image (optical image) formed via the lens unit 100 and outputs image data. In this embodiment, the image sensor 122 photoelectrically converts an object image formed via the imaging optical system and outputs an imaging signal and a focus detection signal as image data. In this embodiment, the first lens group 101, the aperture 102, the second lens group 103, the focus lens 104, and the optical low-pass filter 121 constitute an imaging optical system.

[0021] The optical low-pass filter 121 reduces false colors and moiré in captured images. The image sensor 122 is composed of a CMOS image sensor and its peripheral circuits, and is arranged with m pixels horizontally and n pixels vertically (m and n are integers of 2 or greater). In this embodiment, the image sensor 122 also functions as a focus detection element, has a pupil-splitting function, and has pupil-splitting pixels that enable phase-difference detection focus detection (phase-difference AF) using image data (image signals). Based on the image data output from the image sensor 122, the image processing circuit 124 generates data for phase-difference AF and image data for display, recording, and subject detection.

[0022] The drive / control system includes an image sensor drive circuit 123, an image processing circuit 124, a camera MPU 125, a display 126, a group of operation switches (operation SW) 127, a memory 128, a phase-difference AF unit 129 (image plane phase-difference focus detection unit, control means), an AE unit 130 (control means), a white balance adjustment unit 131 (control means), and a subject detection unit 132 (detection means). The image sensor drive circuit 123 controls the operation of the image sensor 122 and performs A / D conversion on the image signal (image data) output from the image sensor 122 and sends the converted data to the camera MPU 125. The image processing circuit 124 performs typical image processing performed in digital cameras, such as gamma conversion, color interpolation, and compression encoding, on the image signal output from the image sensor 122. The image processing circuit 124 also generates signals for phase-difference AF, AE, white balance adjustment, and subject detection. In this embodiment, a signal for phase difference AF, a signal for AE, a signal for white balance adjustment, and a signal for subject detection are generated, but for example, the signal for AE, the signal for white balance adjustment, and the signal for subject detection may be generated as a common signal. Also, the combination of common signals is not limited to this.

[0023] The camera MPU 125 (processor, control device) performs all calculations and controls related to the camera body 120. That is, the camera MPU 125 controls the image sensor drive circuit 123, image processing circuit 124, display 126, operation switch group 127, memory 128, phase difference AF unit 129, AE unit 130, white balance adjustment unit 131, and subject detection unit 132. The camera MPU 125 is connected to the lens MPU 117 via a signal line of the mount M, and exchanges commands and data with the lens MPU 117. The camera MPU 125 issues requests to the lens MPU 117 to acquire the lens position and to drive the lens by a predetermined drive amount, and also issues requests from the lens MPU 117 to acquire optical information specific to the lens unit 100.

[0024] The camera MPU 125 incorporates a ROM 125a that stores a program for controlling the operation of the camera body 120, a RAM 125b (camera memory) that stores variables, and an EEPROM 125c that stores various parameters. The camera MPU 125 also executes focus detection processing based on the program stored in the ROM 125a. In the focus detection processing, a known correlation calculation process is executed using a pair of image signals obtained by photoelectrically converting optical images formed by light beams that have passed through different pupil regions (pupil partial regions) of the imaging optical system.

[0025] The display 126 is composed of an LCD or the like, and displays information about the shooting mode of the imaging device 10, a preview image before shooting and a confirmation image after shooting, an in-focus state display image when focus is detected, etc. The operation switch group 127 is composed of a power switch, a release (shooting trigger) switch, a zoom operation switch, a shooting mode selection switch, etc. The memory 128 (storage means) is a removable flash memory, and records shot images.

[0026] The phase-difference AF unit 129 performs focus detection processing using a phase-difference detection method based on image signals (signals for phase-difference AF) of focus detection image data obtained from the image sensor 122 and the image processing circuit 124. More specifically, the image processing circuit 124 generates a pair of image data formed by light beams passing through a pair of pupil regions of the imaging optical system as focus detection data, and the phase-difference AF unit 129 detects the amount of focus deviation based on the amount of deviation between the pair of image data. As described above, the phase-difference AF unit 129 of this embodiment performs phase-difference AF (image-surface phase-difference AF) based on the output of the image sensor 122 without using a dedicated AF sensor. In this embodiment, the phase-difference AF unit 129 includes an acquisition unit 129a and a calculation unit 129b. The operation of each of these units will be described later. At least some of the units of the phase-difference AF unit 129 (a part of the acquisition unit 129a or the calculation unit 129b) may be provided in the camera MPU 125. The operation of the phase-difference AF unit 129 will be described later in detail. The phase difference AF unit 129 functions as a focus control unit that controls the position of the focus lens 104 using the focus detection result.

[0027] The AE unit 130 performs exposure adjustment processing to optimize shooting conditions by performing photometry based on AE signals obtained from the image sensor 122 and image processing circuit 124. Specifically, it performs photometry based on the AE signals and calculates the exposure amount at the currently set aperture value, shutter speed, and ISO sensitivity. From the difference between the calculated exposure amount and a predetermined optimum exposure amount, it calculates the appropriate aperture value, shutter speed, and ISO sensitivity to be set during shooting and sets these as shooting conditions, thereby performing exposure adjustment processing. The AE unit 130 calculates the exposure conditions during shooting using the photometry results and functions as exposure adjustment means that controls the aperture value, shutter speed, and ISO sensitivity of the aperture 102.

[0028] The white balance adjustment unit 131 performs white balance adjustment processing based on a signal for white balance adjustment obtained from the image sensor 122 and the image processing circuit 124. Specifically, the white balance adjustment processing is performed by calculating the white balance of the signal for white balance adjustment and adjusting the weight of colors based on the difference from a predetermined appropriate white balance.

[0029] The subject detection unit 132 performs subject detection processing based on a subject detection signal generated by the image processing circuit 124. The subject detection processing detects the type and state of the subject (detection type), and the position and size of the subject (detection area). Details of the operation of the subject detection unit 132 will be described later.

[0030] In this way, the camera system 10 of this embodiment can perform a combination of phase-difference AF, photometry (exposure adjustment), white balance adjustment, and subject detection, and can select the position (image height range) for performing phase-difference AF, photometry, and white balance adjustment depending on the results of subject detection.

[0031] (image sensor) FIG. 2 shows a schematic diagram of the array of imaging pixels (and focus detection pixels) of the image sensor in this first embodiment. FIG. 2 shows the pixel (imaging pixel) array of the two-dimensional CMOS sensor (image sensor) in this first embodiment in an area of ​​4 columns x 4 rows, and the focus detection pixel array in an area of ​​8 columns x 4 rows. In this first embodiment, the 2-column x 2-row pixel group 200 shown in FIG. 2 has a pixel 200R having R (red) spectral sensitivity arranged in the upper left, pixels 200G having G (green) spectral sensitivity arranged in the upper right and lower left, and a pixel 200B having B (blue) spectral sensitivity arranged in the lower right. Furthermore, each pixel is composed of a first focus detection pixel 201 and a second focus detection pixel 202 arranged in a 2-column x 1-row array.

[0032] 2 are arranged on a surface, making it possible to acquire a captured image (focus detection signal). In this embodiment 1, the image sensor will be described as having a pixel pitch P of 4 μm, a pixel count N of 5,575 columns horizontally and 3,725 rows vertically = approximately 20.75 million pixels, a focus detection pixel column direction pitch PAF of 2 μm, and a focus detection pixel count NAF of 11,150 columns horizontally and 3,725 rows vertically = approximately 41.5 million pixels.

[0033] Figure 3(a) shows a plan view of one pixel 200G of the imaging element shown in Figure 2 as seen from the light receiving surface side (+z side) of the imaging element, and Figure 3(b) shows a cross-sectional view of the aa cross section of Figure 3(a) as seen from the -y side.

[0034] 3, in the pixel 200G of this first embodiment, a microlens 305 for collecting incident light is formed on the light-receiving side of each pixel, and a photoelectric conversion unit 301 and a photoelectric conversion unit 302 are formed that are divided into N-H divisions (two divisions) in the x direction and N-V divisions (one division) in the y direction. The photoelectric conversion unit 301 and the photoelectric conversion unit 302 correspond to the first focus detection pixel 201 and the second focus detection pixel 202, respectively.

[0035] The photoelectric conversion units 301 and 302 may be pin structure photodiodes with an intrinsic layer sandwiched between a p-type layer and an n-type layer, or may be pn junction photodiodes with the intrinsic layer omitted as needed. Each pixel has a color filter 306 formed between the microlens 305 and the photoelectric conversion units 301 and 302. Furthermore, the spectral transmittance of the color filter may be changed for each subpixel, or the color filter may be omitted as needed.

[0036] Light incident on pixel 200G shown in Figure 3 is collected by microlens 305, dispersed by color filter 306, and then received by photoelectric conversion unit 301 and photoelectric conversion unit 302. In photoelectric conversion unit 301 and photoelectric conversion unit 302, electron-hole pairs are generated according to the amount of received light, and after being separated by a depletion layer, the negatively charged electrons are accumulated in an n-type layer (not shown), while the holes are discharged to the outside of the image sensor through a p-type layer connected to a constant voltage source (not shown). The electrons accumulated in the n-type layers (not shown) of photoelectric conversion unit 301 and photoelectric conversion unit 302 are transferred to a capacitance unit (FD) via a transfer gate and converted into a voltage signal.

[0037] Fig. 4 is a schematic diagram illustrating the correspondence between the pixel structure of the first embodiment shown in Fig. 3 and pupil division. Fig. 4 shows a cross-sectional view of the aa cross section of the pixel structure of the first embodiment shown in Fig. 3(a) as viewed from the +y side, and the pupil plane (pupil distance DS) of the image sensor. In Fig. 4, the x-axis and y-axis of the cross-sectional view are reversed compared to Fig. 3 in order to correspond to the coordinate axes of the pupil plane of the image sensor.

[0038] In Fig. 4, the first partial pupil region 501 of the first focus detection pixel 201 is generally conjugate with the light receiving surface of the photoelectric conversion unit 301, whose center of gravity is decentered in the -x direction, via a microlens, and represents the pupil region that can receive light at the first focus detection pixel 201. The first partial pupil region 501 of the first focus detection pixel 201 has its center of gravity decentered on the +X side on the pupil plane. In Fig. 4, the second partial pupil region 502 of the second focus detection pixel 202 is generally conjugate with the light receiving surface of the photoelectric conversion unit 302, whose center of gravity is decentered in the +x direction, via a microlens, and represents the pupil region that can receive light at the second focus detection pixel 202. The second partial pupil region 502 of the second focus detection pixel 202 has its center of gravity decentered on the -X side on the pupil plane. In addition, in FIG. 4, pupil region 500 is the pupil region that can receive light in the entire pixel 200G when all of the photoelectric conversion units 301 and 302 (first focus detection pixel 201 and second focus detection pixel 202) are combined.

[0039] Image plane phase-difference AF is affected by diffraction because it uses microlenses on the image sensor to divide the pupil. In Figure 4, the pupil distance to the pupil plane of the image sensor is several tens of mm, while the diameter of the microlens is several microns. As a result, the aperture value of the microlens is several tens of thousands, causing diffraction blur on the level of several tens of mm. As a result, the image on the light-receiving surface of the photoelectric conversion unit does not become a clear pupil region or pupil subregion, but rather becomes a light-receiving sensitivity characteristic (incident angle distribution of light-receiving rate).

[0040] 5 is a schematic diagram showing the correspondence between the image sensor and pupil division in this first embodiment. Light beams that pass through different pupil partial regions, the first pupil partial region 501 and the second pupil partial region 502, are incident on each pixel of the image sensor at different angles and are received by the first focus detection pixel 201 and the second focus detection pixel 202, which are divided into 2×1 regions. This first embodiment is an example in which the pupil region is divided into two in the horizontal direction. If necessary, the pupil may also be divided vertically.

[0041] The image sensor of this first embodiment has an array of imaging pixels, each having a first focus detection pixel and a second focus detection pixel. The first focus detection pixel receives a light beam that passes through a first pupil partial area of ​​the photographing optical system. The second focus detection pixel receives a light beam that passes through a second pupil partial area of ​​the photographing optical system that is different from the first pupil partial area. The imaging pixel receives a light beam that passes through a pupil area that is a combination of the first and second pupil partial areas of the photographing optical system.

[0042] In the image sensor of this first embodiment, each imaging pixel is composed of a first focus detection pixel and a second focus detection pixel. If necessary, the imaging pixel, the first focus detection pixel, and the second focus detection pixel may be configured as separate pixels, and the first focus detection pixel and the second focus detection pixel may be partially arranged in a portion of the imaging pixel array.

[0043] In this first embodiment, focus detection is performed by collecting light reception signals from the first focus detection pixels 201 of each pixel of the image sensor to generate a first focus signal, and collecting light reception signals from the second focus detection pixels 202 of each pixel to generate a second focus signal. Furthermore, an image pickup signal (captured image) with a resolution of N effective pixels is generated by adding the signals from the first focus detection pixels 201 and second focus detection pixels 202 for each pixel of the image sensor. The method of generating each signal is not limited to the embodiment in this first embodiment, and for example, the second focus detection signal may be generated from the difference between the image pickup signal and the first focus signal.

[0044] (Relationship between defocus amount and image shift amount) The relationship between the defocus amount and the image shift amount of the first focus detection signal and the second focus detection signal acquired by the image sensor of this first embodiment will be described below.

[0045] FIG. 6 shows a schematic diagram of the relationship between the defocus amount of the first focus detection signal and the second focus detection signal and the image shift amount between the first focus detection signal and the second focus detection signal. An image sensor (not shown) according to the first embodiment is disposed on an image sensor surface 800. Similar to FIGS. 4 and 5, the pupil plane of the image sensor is divided into a first pupil partial region 501 and a second pupil partial region 502. The defocus amount d is defined as a negative sign (d<0) when the distance from the subject's imaging position to the image sensor is |d|, and a front-focus state in which the subject's imaging position is closer to the subject than the image sensor surface is defined as a front-focus state. A back-focus state in which the subject's imaging position is on the opposite side of the image sensor surface is defined as a back-focus state. The in-focus state in which the subject's imaging position is on the image sensor surface (focus position) is d=0. In FIG. 6, subject 801 shows an example of an in-focus state (d=0), and subject 802 shows an example of a front-focus state (d<0). The front focus state (d<0) and the back focus state (d>0) are combined to form a defocus state (|d|>0).

[0046] In a front-focus state (d<0), a light beam from the subject 802 that passes through the first pupil partial region 501 (second pupil partial region 502) is first focused and then spreads to a width Γ1 (Γ2) centered at the center of gravity G1 (G2) of the light beam, forming a blurred image on the imaging surface 800. The blurred image is received by the first focus detection pixels 201 (second focus detection pixels 202) that constitute the pixels arrayed on the image sensor, and a first focus detection signal (second focus detection signal) is generated. Therefore, the first focus detection signal (second focus detection signal) is recorded as a subject image in which the subject 802 is blurred to a width Γ1 (Γ2) at the center of gravity G1 (G2) on the imaging surface 800. The blur width Γ1 (Γ2) of the subject image increases roughly proportionally as the magnitude of the defocus amount d, |d|, increases. Similarly, the magnitude |p| of the image shift amount p (= the difference G1-G2 in the center of gravity positions of the light beams) of the subject image between the first focus detection signal and the second focus detection signal also increases roughly proportionally as the magnitude |d| of the defocus amount d increases. The same is true in the back-focus state (d>0), although the direction of the image shift of the subject image between the first focus detection signal and the second focus detection signal is opposite to that in the front-focus state.

[0047] As the magnitude of the defocus amount of the first focus detection signal and the second focus detection signal, or the imaging signal obtained by adding the first focus detection signal and the second focus detection signal, increases, the magnitude of the image shift amount between the first focus detection signal and the second focus detection signal also increases. Therefore, in this first embodiment, the phase difference AF section 129 converts the image shift amount into a detected defocus amount using a conversion coefficient calculated based on the base length, due to the relationship in which the magnitude of the image shift amount between the first focus detection signal and the second focus detection signal increases as the magnitude of the defocus amount of the imaging signal increases.

[0048] (Overall processing flow) The overall control method during photography according to this embodiment will be described below with reference to FIGS.

[0049] In S701, a defocus amount calculation area is set to be calculated by the phase difference AF unit 129. At this time, a calculation area 803 is set on a shooting screen 801 as shown in Fig. 8(a) so that defocus calculation can be performed at multiple points over a range wider than the AF frame, with an arbitrary AF frame 802 selected by the user at the center.

[0050] In S702, each defocus amount is calculated in the defocus amount calculation area set in S701.

[0051] In S703, each defocus amount calculated in S702 is classified. For this classification, a histogram of the defocus amount is created, and the class to which the subject belongs is identified by inter-frame matching, thereby obtaining an area where the defocus amount is within a predetermined range, such as the subject class area 804 in Figure 8(b). In S704, subject recognition detection of the specific subject is performed. These detection methods use known machine learning learning techniques or recognition processing using image processing means. In machine learning learning techniques, the feature amounts of each part (face, eyes, whole body) of the subject (person, animal, etc.) are learned in advance, the previously learned subject is recognized from the captured image, and its area (or position and size) is obtained.

[0052] For example, the types of machine learning include:

[0053] (1) Support Vector Machine (2) Convolutional Neural Network (3) Recurrent Neural Network As an example of recognition processing, a method is known in which a skin color area is extracted from the gradation color of each pixel represented by image data and a face is detected based on the degree of match with a prepared face outline plate. Another known method is to use well-known pattern recognition technology to extract facial feature points such as the eyes, nose, and mouth to perform face detection. Furthermore, the main area detection method applicable to the present invention is not limited to these methods, and other methods may also be used.

[0054] In S705, if the subject is recognized in the subject recognition detection in S704, the process proceeds to S706, and if the subject is not recognized, the process proceeds to S707.

[0055] In S706, it is confirmed whether or not the object detection class calculated in S703 exists. If the object detection class exists, the process proceeds to S708, and if the object detection class does not exist, the process proceeds to S709.

[0056] In S708, a main subject area is selected with priority given to the subject detection class calculated in S703, and a position on the shooting screen at which AF will be performed is determined. Details of this main subject area selection process will be described later.

[0057] In S709, a main subject area is selected with priority given to the subject recognition position detected in S704, and a position on the shooting screen at which AF will be performed is determined. Details of this main subject area selection process will be described later.

[0058] Also, in S707, it is confirmed whether or not there is a subject detection class calculated in S703. If there is a subject detection class, the process proceeds to S710, and if there is no subject detection class, the process proceeds to S711. The reason for prioritizing the subject detection class over subject recognition is that, for example, when photographing a basketball game, there are multiple subjects as shown in FIG. 8C, and in a scene where the focus has been set on subject 804 until then and subject 805 then enters the AF frame, even if subject 805 is in the AF frame, it is desirable to continue to focus on subject 804 while performing continuous focus adjustment. For this reason, if a subject detection class has been detected for subject 804, the subject detection class that includes subject 804 is prioritized as the main subject area selection.

[0059] In S710, a main subject area is selected with priority given to the subject detection class calculated in S703, and a position on the shooting screen at which AF will be performed is determined. Details of this main subject area selection process will be described later.

[0060] In S711, the center position of the AF frame optionally designated by the user is set to the main subject area.

[0061] In S712, a determination is made as to whether focus driving should be performed for the main subject region position selected by any of the means of S708 to S711. The image plane position at the timing of this defocus amount calculation is estimated using the least squares method or the like from the history of image plane positions in the optical axis direction of the subject obtained from the results of defocus amount calculations for the region selected in the main subject region selection. If the difference from the subject position selected as the main subject region is outside a predetermined range, it is determined that the user has lost framing or an obstacle has entered, and driving of the focus lens 104 is stopped. If it is within a predetermined threshold, the result of the defocus amount for the main subject region is used to drive the focus lens 104. Furthermore, if the position of the main subject region is outside the range of the AF frame, the user has lost framing, so lens driving is stopped and it is awaited for framing to be restored.

[0062] In S713, if it is determined in the focus drive determination made in S712 that focus drive is to be performed, the process proceeds to S714, and if it is determined that focus drive is not to be performed, the process proceeds to S715.

[0063] In S714, the lens drive amount is calculated based on the defocus amount of the main subject area position selected by any of the means of S708 to S711, and the focus actuator 113 moves the focus lens 104 to perform focus adjustment.

[0064] (Subject class) Here, the above-mentioned classification will be explained in more detail using FIG. 16. FIG. 16(a1)-(a2) shows the state of the subject in frames 1 and 2. FIG. 16(b1)-(b2) shows a histogram of the defocus amount. This is an example of a scene in which subject 804 moves toward the closer side (forward) from frame 1 to frame 2, while subject 805 is stationary. Using each defocus amount calculated in S702, the position of main subject 804 identified in frame 1 can also be identified in frames 2 and onward.

[0065] First, the histogram shown in FIG. 16(b1) is calculated for frame 1. The horizontal axis represents the defocus amount, with the left side representing the infinity side and the right side representing the close-up side. The vertical axis represents the frequency of the calculation area belonging to the corresponding class. The bin interval (the interval between each class) and range used in calculating the histogram are set according to the subject depth size on the image plane, calculated from the subject distance, subject size, and aperture value. By setting the bin interval and range according to the subject depth size, the resolution used to represent the subject on the histogram can be appropriately set regardless of the conditions. In this embodiment, the histogram bin interval and range are set according to the shooting conditions, but they may also be calculated using a predetermined bin interval and range. Class 841 in FIG. 16(b1) is the subject class identified in frame 1.

[0066] Next, in frame 2, a histogram (FIG. 16(b1)) of frame 1, which is the previous frame, and a histogram (FIG. 16(b2)) of frame 2, which is the current frame, are obtained. The peak on the left side of each histogram corresponds to subject 804, and the peak on the right side corresponds to subject 805. In frame 2, subject 804 is closer to the subject than in frame 1, and its size within the angle of view is also larger, so the peak on the left side corresponding to subject 804 has moved to the closer side (right side) and become larger. The subject class can be identified by utilizing the fact that what exists in front of and behind the subject does not change significantly between frames.

[0067] (Main subject area selection with priority given to subject detection class) Hereinafter, with reference to FIG. 9, a main subject region selection method that prioritizes the subject detection class according to this embodiment will be described.

[0068] In S901, an expected defocus amount is calculated from the history of the image plane position of the subject. Specifically, the past positions of the subject on the image plane in the optical axis direction of the main subject area are stored as history, and the subject position on the image plane at the time when the defocus amount is calculated is predicted using the least squares method or the like. Then, an expected defocus amount is calculated based on the past history from the predicted subject position and the lens position at the time when the defocus amount was calculated.

[0069] In S902, the defocus amount calculated for each calculation area within the AF frame is compared with the predicted defocus amount calculated in S901, and it is determined whether the percentage of calculation areas where the difference is equal to or greater than the threshold value TH1 is equal to or greater than the threshold value TH2. In Figure 8(b), there are a total of 20 calculation areas arranged in a 5x4 pattern within the AF frame, but most of the calculation areas do not correctly capture the subject. In such scenes, it is necessary to search for the correct subject position even outside the AF frame. Therefore, it is checked whether there are calculation areas within the AF frame that are not continuous with the history of past subject positions (i.e., have a defocus difference equal to or greater than the threshold value TH1) at a percentage equal to or greater than TH2. If there are, proceed to S903; if there are not, proceed to S907.

[0070] In S903, it is determined whether the object detection class calculated in S703 exists. If the object detection class exists, the process proceeds to S904, and if not, the process proceeds to S907.

[0071] In S904, it is determined whether the subject detection class calculated in S703 exists within the AF frame range. If it is outside the range, the process proceeds to S905, and if it is within the range, the process proceeds to S906.

[0072] In S905, the calculation area in the subject detection class that is closest to the predicted defocus amount is set as the main subject area.

[0073] In S906, the calculation area closest to the predicted defocus amount among the calculation areas in the subject detection class within the AF frame is set as the main subject area.

[0074] In S907, the calculation area closest to the estimated defocus amount within the AF frame is set as the main subject area.

[0075] (Main subject area selection with priority given to subject recognition position) With reference to the flowchart in FIG. 10, the process of selecting a main subject area that prioritizes the subject recognition position in S708 in FIG. 7 will be described. First, in S1001, an overlapping area between the area within the AF frame and the subject recognition area is identified. The overlapping area will be described with reference to FIG. 11. In FIGS. 11(a) and 11(b), reference numeral 1101 denotes the shooting area (shooting screen), the black frame 1102 denotes the AF frame, the double frame 1103 denotes the subject recognition frame that is an index indicating the subject recognition area, 1104 denotes the subject, and 1105 denotes a tree in the background. The shaded area 1106 in FIG. 11(b) is the overlapping area. The subject recognition frame 1103 is the subject recognition frame when the eye of the subject 1104 is detected, and in FIG. 11(b), the subject recognition frame 1103 and the AF frame 1102 overlap, so an overlapping area 1106 exists. Next, in S1102, it is determined whether the overlapping area is equal to or larger than the minimum overlap size. The minimum overlap size is a reference size for determining whether the AF frame and the subject recognition frame overlap. If the overlap area is equal to or larger than the minimum overlap size, it is determined that the AF frame and the subject recognition frame overlap, and the process proceeds to S1004. If the overlap area is smaller than the minimum overlap size in S1102, it is determined that the AF frame and the subject recognition frame do not overlap, and the process proceeds to S1003. This determination in S1102 is called overlap determination. In FIG. 11(a), the subject recognition frame 1103 and the AF frame 1102 do not overlap, and there is no overlap area, so it is determined that there is no overlap. In FIG. 11(b), there is an overlap area 1106, and if this overlap area is equal to or larger than the minimum overlap size, it is determined that there is overlap. If it is determined that there is overlap, a main subject area is selected from the area within the subject recognition frame in S1004. If there is no overlap, a main subject area is selected from the area within the AF frame in S1003. In S1005, if the difference between the defocus amount of the main subject area selected in S1004 and the defocus amount of the subject predicted from the history of the image plane position of the subject is greater than a predetermined value, the process transitions to S1003 and a main subject area is selected from the area within the AF frame. In S1008, it is determined whether the setting is to display the subject recognition frame. If the subject recognition frame is not to be displayed, the process of selecting a main subject area that prioritizes the subject recognition position in S708 of Figure 7 ends with the AF frame displayed.When a subject recognition frame is to be displayed and a main subject region is selected from the subject recognition region, the process proceeds to S1006, where the AF frame is made an inactive frame, and the subject recognition frame is made an active frame. When a subject recognition frame is to be displayed and a main subject region is selected from the AF frame region, the process proceeds to S1007, where the AF frame is made an active frame, and the subject recognition frame is made an inactive frame. Here, the active frame refers to an area used to select the main subject region, displayed on the shooting screen, and is displayed with a solid line, such as the active AF frame 1201 in FIG. 12(a) or the active subject recognition frame 1204 in FIG. 12(b). The inactive frame refers to an area not used to select the main subject region, displayed on the shooting screen, and is displayed with a gray line, such as the inactive AF frame 1203 in FIG. 12(a) or the inactive subject recognition frame 1202 in FIG. 12(b). The active and inactive frames are not limited to the solid and gray lines shown in FIG. 12, and may be displayed in other formats. As described above, the focus lens can be controlled using the defocus amount of the main subject area obtained from the selected calculation area by selecting either the area within the AF frame or the subject recognition area. If an area within the AF frame is selected, any area within the shooting area can be specified with the AF frame to focus on, and if the subject recognition area is selected, the subject can be focused on without having to be precisely aligned with the AF frame.

[0076] The overlap determination performed in S1002 of FIG. 10 is an important determination that determines whether the area the user wants to focus on is within the AF frame or the subject recognition area, and the minimum overlap size may be defined as follows. (a) of FIG. 13 shows the defocus amount calculation area set in S701 of FIG. 7 superimposed on (b) of FIG. 11. Here, the defocus amount calculation area set in S701 is simply referred to as the calculation area. The square 1301 in FIG. 13(a) is the area for one calculation area, the arc 1302 is the horizontal length of the calculation area, and the arc 1303 is the vertical length of the calculation area. (b) of FIG. 13 is an enlarged view of the dotted-line area surrounding the face in (a) of FIG. 13. The arc 1304 is the horizontal length of the overlap area between the area within the AF frame and the subject recognition area, and the arc 1305 is the vertical length of the overlap area. In overlap determination, the size of the overlap area, which is determined to be an overlap between an area within the AF frame and a subject recognition area, can be defined as the overlap area being longer in both the vertical and horizontal lengths of the overlap area and the calculated area. Referring to FIG. 13 , if the horizontal length 1304 of the overlap area is longer than the horizontal length 1302 of the calculated area and the vertical length 1305 of the overlap area is longer than the vertical length 1303 of the calculated area, it can be determined that they overlap. This definition of overlap determination ensures that there is at least one calculated area for detecting the defocus amount of the subject within the AF frame representing the area the user wants to focus on, and it can be estimated that the area the user wants to focus on is within the subject recognition frame. This allows the user to determine and select whether the area they want to focus on is within the AF frame or the subject recognition area.

[0077] If the minimum overlap size is small in the overlap determination performed in S1002 of FIG. 10 , the subject recognition area can be used even if there is even a small overlap between the area within the AF frame and the subject recognition area, thereby reducing the difficulty of keeping the AF frame focused on the subject. On the other hand, if the minimum overlap size is large, the area within the AF frame can be focused on even if there is a small overlap between the area within the AF frame and the subject recognition area, allowing the user to focus on the desired area using the AF frame. In other words, the minimum overlap size can be adjusted to strike a balance between reducing the difficulty of keeping the AF frame focused on the subject and allowing the user to focus on the desired area using the AF frame. Therefore, the minimum overlap size may be adjusted taking this balance into consideration. For example, if the focal length of the lens unit 100 is longer than a predetermined focal length, the minimum overlap size may be smaller than if the focal length is shorter. This is because a longer focal length is more susceptible to camera shake, making it more difficult to keep the AF frame focused on the subject. Furthermore, if the moving speed of the subject is faster than a predetermined speed, the minimum overlap size may be smaller than if the moving speed is slower. This is because the faster the subject moves, the more difficult it is to keep the subject focused on the AF frame. The subject's movement speed can be estimated from the speed at which the subject recognition area of ​​the subject, continuously recognized during shooting, moves vertically and horizontally within the shooting area. Furthermore, if the AF frame is smaller than a predetermined size, the minimum overlap size may be smaller than when the size is larger. This is because, when the AF frame size is small, even slight camera shake significantly changes the area included in the AF frame, making it difficult to keep the subject focused on the AF frame. Furthermore, if the subject size is smaller than a predetermined size, the minimum overlap size may be smaller than when the size is large. This is because, when the subject is far from the imaging device, even slight camera shake significantly changes the area included in the AF frame, making it difficult to keep the subject focused on the AF frame. The subject size can be estimated from the size of the subject's subject recognition area. Furthermore, if the distance between the subject and the imaging device is farther than a predetermined value, the minimum overlap size may be smaller than when the distance is closer. This is because, when the subject is far from the imaging device, even slight camera shake significantly changes the area included in the AF frame, making it difficult to keep the subject focused on the AF frame.Furthermore, when shooting video, the minimum overlap size may be smaller than when shooting still images. This is because frequent changes in focus position tend to be undesirable as video quality is poor, and stabilizing the position of the main subject area on the subject can stabilize the focus position. As described above, overlap determination is performed taking into consideration the size of the calculated area and the overlap area, and a balance between reducing the difficulty of keeping the AF frame on the subject and allowing the user to focus on the desired area using the AF frame, and it is possible to select whether to use the area within the AF frame or the subject recognition area to select the main subject area.

[0078] There may be a portion of the same subject that you want to prioritize focusing on over a calculated area within the subject recognition area determined to be overlapping in the overlap determination. For example, when a car is recognized and the subject recognition area determined to be overlapping in the overlap determination is the entire car, the head of the car is also recognized and you want to prioritize focusing on the head. In this case, if you can focus on the prioritized portion, you can further reduce the difficulty of keeping the AF frame on the subject. Therefore, the portion determined to be overlapping in the overlap determination may be changed based on the priority of the detected portion of the same subject. However, the overlapping area between the subject recognition area of ​​a higher priority portion and the area within the AF frame may be smaller than the minimum overlap size. Furthermore, even if there is no overlapping area between the subject recognition area of ​​a higher priority portion and the area within the AF frame, the portion determined to be overlapping may be changed to the subject recognition area of ​​the higher priority portion. In FIG. 14, 1401 denotes a car, which is the subject; 1402 denotes an AF frame; 1403 denotes a subject recognition frame that is an index indicating the subject recognition region for the entire car; 1404 denotes a subject recognition frame that is an index indicating the subject recognition region for the head portion; and 1405 denotes an overlapping region between the region within the AF frame and the subject recognition region for the entire car. In the example of FIG. 14, the entire car and the head portion are recognized, and overlapping region 1405 is larger than the lower limit overlap size. In this case, subject recognition region 1403 for the entire car overlaps with the region within the AF frame. However, if the head portion has a higher priority than the entire car, the subject recognition region determined to overlap through overlap determination can be the region corresponding to subject recognition region 1404 for the head portion. In this case, even if there is no overlapping region between region 1404 corresponding to the subject recognition frame for the head portion, which has a higher priority, and the region within the AF frame, as shown in FIG. 14, or even if the overlapping region is smaller than the lower limit overlap size, it can be determined that the region of subject recognition frame 1404 for the head portion overlaps.

[0079] Furthermore, regarding display, the subject recognition frame for a high-priority portion may be displayed as the active frame, and the overlapping subject recognition frames and AF frames for a low-priority portion may be displayed as inactive frames. FIG. 15 shows an example of the display of AF frames and subject recognition frames for the scene shown in FIG. 14. In FIG. 15, reference numeral 1501 denotes the subject car, reference numeral 1502 denotes the inactive AF frame, reference numeral 1503 denotes the inactive subject recognition frame for the entire car, and reference numeral 1504 denotes the subject recognition frame for the head portion of the active frame. FIG. 15 is a display example, and the active and inactive frames may be displayed in other formats. Furthermore, if the area used to select the main subject area corresponds to subject recognition frame 1504, subject recognition frame 1503 for a different portion of the same subject may be hidden. This makes it easier to focus on the desired portion of the subject by continuously aligning the AF frame with the subject.

[0080] The display processing before and after the user's shooting preparation instruction operation (SW1) when the subject recognition frame is not displayed and when it is displayed is as described below. First, when the subject recognition frame is not displayed, the subject recognition frame is hidden regardless of whether it overlaps with the AF frame, and only the AF frame is displayed with a white frame. When SW1 is pressed, if the AF frame and subject recognition frame overlap, the subject recognition frame is used; if they do not overlap, the AF frame is used to focus. This also applies when SW1 is held during servo AF. On the other hand, when the subject recognition frame is displayed, if the AF frame and subject recognition frame do not overlap, the AF frame is displayed with a white frame and the subject recognition frame is displayed with a gray frame. When SW1 is pressed at this time, the subject recognition frame is hidden and the subject within the AF frame is focused. In servo AF, when SW1 is held after focusing, the AF frame is hidden, and the subject recognition frame is attached to the focused object, and AF tracks it. When the AF frame and subject recognition frame overlap, the AF frame is displayed with a gray frame and the subject recognition frame is displayed with a white frame. If SW1 is pressed at this time, the AF frame will be hidden and the camera will focus on the subject within the subject recognition frame. In the case of servo AF, the subject recognition frame will be used to track the AF while SW1 is held down after focusing.

[0081] According to the above-described embodiment, it is possible to accurately focus on a subject without accurately framing the subject using optional AF.

Claims

1. a signal acquisition means for acquiring an image signal output from an image pickup element that captures an image via an image pickup optical system including a focus lens; a focus detection unit that outputs a focus detection result based on the image signal; a setting means for setting an AF frame area designated by a user within the image signal; an acquisition means for acquiring a region of a subject; a control unit that controls movement of the focus lens based on the focus detection result, The imaging device is characterized in that the control means does not drive the focus lens based on the focus detection result corresponding to the AF frame when the state changes from one in which the subject area is within the AF frame area to one in which the subject area is not within the AF frame area.

2. 2. The imaging device according to claim 1, wherein the subject area is an area in which the focus detection result obtained from a detection area including the AF frame area and an area other than the AF frame area is within a predetermined range.

3. The focus detection device further includes a prediction unit that predicts a focus detection result at a timing when a current focus detection result is calculated based on a history of focus detection results detected by the focus detection unit, The imaging device according to claim 2, characterized in that when there is a detection area in which the difference between the focus detection result predicted by the prediction means and the focus detection result in the detection area within the AF frame is equal to or greater than a first threshold value and the area of ​​the subject is not within the AF frame, the control means does not drive the focus lens based on the focus detection result corresponding to the AF frame.

4. The focus detection device further includes a prediction unit that predicts a focus detection result at a timing when a current focus detection result is calculated based on a history of focus detection results detected by the focus detection unit, The imaging device according to claim 2, characterized in that the control means drives the focus lens based on the focus detection result corresponding to the AF frame when a difference between the focus detection result predicted by the prediction means and the focus detection result in a detection area within the AF frame is equal to or greater than a first threshold value and is less than a second threshold value.

5. The focus detection device further includes a prediction unit that predicts a focus detection result at a timing when a current focus detection result is calculated based on a history of focus detection results detected by the focus detection unit, 3. The image pickup apparatus according to claim 1, wherein the control means sets the area in which the focus detection result closest to the predicted focus detection result is obtained as the main subject area.

6. further comprising an object recognition means for recognizing a specific object; The imaging device according to any one of claims 2 to 4, characterized in that when the subject recognition area recognized by the subject recognition means is within the area of ​​the AF frame, if the subject area is not acquired based on the focus detection result of the detection area, the control means performs focus control on the subject recognized by the subject recognition means, and if the subject area is acquired, performs focus control on the subject in the subject area.

7. A control method for an imaging device having a signal acquisition means for acquiring an image signal output from an imaging element that captures an image via an imaging optical system including a focus lens, comprising: a focus detection step of outputting a focus detection result based on the image signal; a setting step of setting an AF frame area designated by a user within the image signal; an acquisition step of acquiring a region of a subject; a control step of controlling movement of the focus lens based on the focus detection result, A control method for an imaging device, characterized in that in the control process, when a state in which the subject area is within the area of ​​the AF frame changes to a state in which the subject area is not within the area of ​​the AF frame, the focus lens is not driven based on the focus detection result corresponding to the AF frame.

8. 7. A program for causing a computer to execute each unit of the imaging device according to claim 1.

Images