Imaging device and control method thereof

Abstract

To stably perform highly accurate focus control using an image pickup device.SOLUTION: An imaging apparatus 120 has: an image pickup device 122 that can acquire, from an inside of an imaging surface picking up a subject image formed by an optical system 100, a first pair of signals pupil-divided in a first direction and a second pair of signals pupil-divided in a second direction different from the first direction; acquisition means 124 that acquires a first defocus amount from a phase difference between the first pair of signals, and acquires a second defocus amount from a phase difference between the second pair of signals; detection means 130 that detects a subject in the imaging surface; and determination means 125 that determines, according to the detected subject, at least one defocus amount of the first and second defocus amounts used for focus control of the optical system.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to focus control in an imaging device such as a digital camera.

Background Art

[0002] As disclosed in Patent Document 1, in the focus detection method, pupil division is performed by one microlens provided for each pixel of an imaging element and a pair of photoelectric conversion units, and the focus state (defocus amount) is obtained from the phase difference between a pair of image signals obtained from the pair of photoelectric conversion units. There is an imaging surface phase difference detection method.

[0003] Patent Document 2 discloses an imaging device that performs horizontal pupil division on pixels in which a pair of photoelectric conversion units are arranged horizontally and vertical pupil division on pixels in which a pair of photoelectric conversion units are arranged vertically, thereby obtaining defocus amounts in the horizontal and vertical directions. Further, Patent Document 3 discloses an imaging device that has a horizontal focus detection area and a vertical focus detection area that cross each other, and performs focus control by selecting the one with the smaller absolute value among the defocus amounts detected in each.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0006] The present invention provides an imaging device that enables stable, high-precision focus control using an image sensor. [Means for solving the problem]

[0007] An imaging device as one aspect of the present invention includes an image sensor capable of acquiring a first pair of signals divided into pupils in a first direction and a second pair of signals divided into pupils in a second direction different from the first direction from within the imaging surface that captures an image of a subject formed by an optical system; an acquisition means for acquiring a first defocus amount from the phase difference of the first pair of signals and a second defocus amount from the phase difference of the second pair of signals; a detection means for detecting a subject in the imaging screen; and, depending on the detected subject, the first and second defocus amounts used for focus control of the optical system. ru 1 It has a determination means for determining the amount of defocusing. When the detection means detects a subject behind an object extending in one direction, the determination means decides to use the defocus amount obtained from the first or second pair of signals, which are divided in the same direction as the one direction within the imaging screen, for focus control. It is characterized by doing so.

[0010] Another aspect of the present invention is a control method that is applied to an imaging device having an image sensor capable of acquiring a first pair of signals divided in a first direction from within the imaging plane that captures a subject image formed by an optical system, and a second pair of signals divided in a second direction different from the first direction. This control method acquires a first defocus amount from the phase difference of the first pair of signals and a second defocus amount from the phase difference of the second pair of signals. acquisition Step and detect the subject within the imaging screen. detection Depending on the step and the detected subject, determine at least one of the first and second defocus amounts to be used for optical system focus control. decisionSteps and Furthermore, if a subject behind an object extending in one direction is detected in the detection step, the determination step determines that the defocus amount obtained from the first or second pair of signals divided in the same direction as the one direction within the imaging screen will be used for focus control. It is characterized by doing so.

[0013] Furthermore, a program that causes the computer of the imaging device to execute processing according to the above control method also constitutes another aspect of the present invention. [Effects of the Invention]

[0014] According to the present invention, highly accurate focus detection using an image sensor can be performed stably. [Brief explanation of the drawing]

[0015] [Figure 1] A block diagram showing the configuration of the imaging device in Example 1. [Figure 2] A diagram showing the image sensors used in Examples 1-3 and 5-7. [Figure 3] A diagram showing the AF area used in focus detection processing. [Figure 4] A diagram showing a pair of image signals obtained from the AF region. [Figure 5] A diagram showing the relationship between the amount of shift and the amount of correlation of a pair of image signals. [Figure 6] A diagram showing the relationship between the amount of shift and the amount of correlation change of a pair of image signals. [Figure 7] A flowchart showing the imaging process in Examples 1 to 7. [Figure 8] A flowchart showing the auxiliary light irradiation determination process in Examples 1, 2, and 5-7. [Figure 9] A flowchart showing the process for determining the amount of defocus used in Examples 1, 6, and 7. [Figure 10] A flowchart showing the process for determining the amount of defocus used for stabilization in Examples 1 to 5. [Figure 11] A flowchart showing the process for determining the amount of defocus used for stabilization in Examples 1 to 5. [Figure 12] A flowchart showing the AF process in Examples 1-7. [Figure 13]A diagram showing an example of an imaging scene in the auxiliary light irradiation state and an example of an output signal of a photoelectric conversion unit. [Figure 14] Another diagram showing a light-receiving pixel. [Figure 15] A diagram showing an example of a cross-over imaging scene and an example of an output signal of a photoelectric conversion unit. [Figure 16] A diagram showing an example of an imaging scene including a horizontal line and an example of an output signal of a photoelectric conversion unit. [Figure 17] A diagram showing an example of a dark imaging scene and an example of an output signal of a photoelectric conversion unit. [Figure 18] A graph showing an example of a defocus amount detected when the signal amount is small. [Figure 19] A flowchart showing the defocus amount determination process used in Example 2. [Figure 20] A flowchart showing the auxiliary light irradiation determination process in Example 3. [Figure 21] A flowchart showing the defocus amount determination process used in Example 3. [Figure 22] A diagram showing a light-receiving pixel in Example 4. [Figure 23] A flowchart showing the auxiliary light irradiation determination process in Example 4. [Figure 24] A flowchart showing the defocus amount determination process used in Example 4. [Figure 25] A flowchart showing the defocus amount determination process used in Example 5. [Figure 26] A flowchart showing the defocus amount determination process using the xy priority direction map in Example 5. [Figure 27] A diagram showing an example of an xy priority direction map in a cross-over imaging scene. [Figure 28] A flowchart showing the stable defocus amount usage determination process in Examples 6 and 7. [Figure 29] A flowchart showing the defocus amount determination process used in Examples 6 and 7. [Figure 30] A flowchart showing the defocus amount determination process used in Example 6. [Figure 31] This diagram shows an example of AF area settings for dark imaging scenes. [Figure 32] A flowchart showing the process for determining the amount of defocus used in Example 7. [Figure 33] A diagram showing the image sensor in a modified example. [Modes for carrying out the invention]

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings. [Examples]

[0017] (Configuration of the imaging device) Figure 1 shows the configuration of a digital camera as an imaging device in Embodiment 1. The digital camera in this embodiment is an interchangeable-lens single-lens reflex camera and has an imaging optical system consisting of a lens unit 100 and a camera body 120. The lens unit 100 is mounted on the camera body 120 via a mount M shown by the dotted line in the figure.

[0018] The lens unit 100 has an optical system (first lens group 101, aperture) shutter The optical system comprises 102, a second lens group 103, a focusing lens group (hereinafter simply referred to as focusing lens) 104), and a drive control system. The optical system forms an optical image (subject image) of the subject.

[0019] The first lens group 101 is positioned closest to the subject in the optical system and is held so as to be movable in the optical axis direction. The aperture shutter 102 has an aperture function to adjust the amount of light and a shutter function to control the exposure time of the image sensor, which will be described later, when capturing still images. The aperture shutter 102 and the second lens group 103 are movable together in the optical axis direction and perform zooming by moving in conjunction with the first lens group 101. The focus lens 104 moves in the optical axis direction to adjust the focus (focusing).

[0020] The drive control system includes a zoom actuator 111, an aperture actuator 112, and a focus actuator 113. The drive control system also includes a zoom drive circuit 114, an aperture drive circuit 115, a focus drive circuit 116, a lens MPU 117, and a lens memory 118. The zoom drive circuit 114 drives the zoom actuator 111 to control the first lens group 101 and the first 2 Zooming is performed by moving the lens group 103 in the optical axis direction. The aperture drive circuit 115 drives the aperture actuator 112 to operate the aperture shutter 102 and control the amount of light and exposure time. The focus drive circuit 116 drives the focus actuator 113 to move the focus lens 104 in the optical axis direction to perform focusing. The focus drive circuit 116 also detects the position of the focus lens 104 from the amount of drive of the focus actuator 113.

[0021] The lens MPU 117 is a computer that performs calculations and processing in the lens unit 100 and controls the zoom drive circuit 114, aperture drive circuit 115, and focus drive circuit 116. The lens MPU 117 is also electrically connected to the camera MPU 125 via the mount M and sends and receives commands and data to and from the camera MPU 125. For example, the lens MPU 117 transmits information about the position of the detected focus lens 104 to the camera MPU 125 upon request from the camera MPU 125. This focus lens position information includes not only the position of the focus lens 104 in the optical axis direction, but also the position and diameter of the exit pupil in the optical axis direction when the optical system is not driven, as well as information such as the position and diameter of the lens frame in the optical axis direction that limits the light beam of the exit pupil. The lens MPU 117 also controls the zoom drive circuit 114, aperture drive circuit 115, and focus drive circuit 116 upon request from the camera MPU 125.

[0022] The lens memory 118 pre-stores optical information necessary for detecting the focus state (amount of defocus). The camera MPU 125 performs processing and control of the lens unit 100 by executing programs stored in its built-in non-volatile memory and the lens memory 118.

[0023] The camera body 120 includes an optical system (optical low-pass filter 121 and image sensor 122) and a drive control system. The optical low-pass filter 121 is positioned directly in front of the image sensor 122 to reduce false colors and moiré patterns in captured images.

[0024] The image sensor 122 consists of a two-dimensional CMOS area sensor and its peripheral circuitry. Multiple pixels are arranged within the imaging plane of the image sensor (area sensor) 122, both horizontally and vertically. The image sensor 122 has pupil division functions in the horizontal and vertical directions, enabling focus detection using an imaging plane phase-difference detection method. The detailed configuration of the image sensor 122 will be described later.

[0025] The drive control system includes an image sensor drive circuit 123, an image processing circuit 124, a camera MPU 125, a display 126, a group of operation switches 127, a memory 128, an image plane phase-detection focus unit 129, and a subject detection unit 130. The image sensor drive circuit 123 controls the drive of the image sensor 122, performs A / D conversion on the analog signal output from the image sensor 122, and transmits the digital signal to the camera MPU 125 and the image processing circuit 124.

[0026] The image processing circuit 124 generates a pair of image signals for phase difference detection and various image data from the digital signals from the image sensor drive circuit 123. The image processing circuit 124 also performs image processing on the display / recording image data, such as gamma conversion, white balance adjustment, color interpolation, and compression coding.

[0027] The camera MPU 125, acting as a computer, performs calculations and processing in the camera body 120 and controls the image sensor drive circuit 123, image processing circuit 124, display 126, operation switch group 127, memory 128, image plane phase-detection focus detection unit 129, and subject detection unit 130. The camera MPU 125 is electrically connected to the lens MPU 117 via the mount M and sends and receives commands and data to and from the lens MPU 117. The camera MPU 125 requests information from the lens MPU 117 such as focus lens position information, aperture, focus and zoom drive, and optical information of the lens unit 100. The camera MPU 125 functions as a control means and a decision means.

[0028] The camera MPU125 has a built-in ROM125a, RAM125b, and EEPROM125c. ROM125a stores the program that controls the operation of the camera body 120. RAM125b stores various variables. EEPROM125c stores various parameters and user-defined settings.

[0029] The display unit 126 is composed of an LCD or organic EL element and displays information related to the imaging mode, a preview image before imaging, a confirmation image after imaging, and the focus status during focus detection. The display unit 126 is equipped with a touch sensor, allowing the user to input or select information by touching the display unit 126.

[0030] The group of operation switches 127 includes a power switch, an AF (autofocus) start switch, a shutter release (still image capture trigger) switch, a zoom operation switch, an image capture mode selection switch, and a video capture switch, etc. The memory 128 is a flash memory that can be attached to and removed from the camera body 120 and records captured images.

[0031] The image plane phase-difference focus detection unit (acquisition means) 129 performs focus detection processing using a phase-difference detection method that utilizes one or more pairs of image signals obtained from the image processing circuit 124. Specifically, the image processing circuit 124 generates a pair of image signals from multiple pairs of photoelectric conversion signals obtained from multiple pixels that perform pupil division in the horizontal direction, and also generates a pair of image signals from multiple pairs of photoelectric conversion signals obtained from multiple pixels that perform pupil division in the vertical direction. Furthermore, the image plane phase-difference focus detection unit 129 detects the phase difference (amount of focus shift) between each pair of image signals generated by the image processing circuit 124, and calculates (acquires) the amount of defocus of the optical system from this phase difference. Details of the processing performed by the image plane phase-difference focus detection unit 129 will be described later.

[0032] The subject detection unit (detection means) 130 detects subjects within the image capture screen from display / recording image data obtained from the image processing circuit 124. Specifically, it uses a CNN (Convolutional Neural Network) to estimate the location of the subject to be detected within the image data. Any machine learning method for the CNN is acceptable. For example, the subject detection process may utilize a pre-trained model, which is an example of a neural network computation model. For example, the subject detection unit 130 reads image data from memory 128 and inputs it into the pre-trained model, obtaining the subject's position, such as the subject's outline or the region of the subject image, as the output result of the pre-trained model. In this case, for example, a computer such as a server may perform machine learning on the CNN, and the camera body 120 may obtain the trained CNN from the computer. The CNN of the subject detection unit 130 is trained by supervised learning, where the computer takes training image data as input and the subject's position corresponding to the training image data as training data. The subject detection result from the subject detection unit 130 is used to select the focus detection result of the image plane phase-difference focus detection unit 129, which is used by the camera MPU 125 to drive the focus lens 104 via the lens MPU 117.

[0033] The focus detection auxiliary light source 131 increases the amount of light incident on the image sensor 122 by illuminating the subject with auxiliary light (e.g., red light) in order to improve the accuracy of focus detection by the image plane phase-difference focus detection unit 129. The camera MPU 125 illuminates the focus detection auxiliary light source 131 in conjunction with the AF start switch in the operation switch group 127 or according to the settings stored in the EEPROM 125c. (Operation of the imaging plane phase-contrast focus detection unit 129) Figure 2(a) shows the pixel arrangement on the imaging surface of the image sensor 122 as seen from the lens unit 100 side, within a range of 8 horizontal (x direction: first direction) x 6 vertical (y direction: second direction) pixels. The imaging surface is provided with Bayer color filters, with red (R) and green (G) color filters alternately arranged from left to right for pixels in odd-numbered rows, and green (G) and blue (B) color filters alternately arranged from left to right for pixels in even-numbered rows.

[0034] Figure 2(b) shows pixel 211R. 211i represents an on-chip microlens. Inside the on-chip microlens 211i are a pair of photoelectric conversion units (first pair of photoelectric conversion units) 211A and 211B, which are divided in the x direction. Pixels 211Gr and 211B are configured similarly. Figure 2(c) shows pixel 211Gb. Inside the on-chip microlens 211i are a pair of photoelectric conversion units (second pair of photoelectric conversion units) 211C and 211D, which are divided in the y direction. Thus, the image sensor 122 of this embodiment has pixels 211R, 211Gr, and 211B, in which the photoelectric conversion unit is divided into two in the x direction, and pixel 211Gb, in which the photoelectric conversion unit is divided into two in the y direction. A pair of image signals and disparity image data as display / recording image data for 3D image observation are generated using the photoelectric conversion signals output from each of the pairs of photoelectric conversion units in the multiple pixels. Furthermore, image data for brightness determination and normal display / recording image data are generated using an imaging signal that is output by adding a pair of photoelectric conversion signals from each of multiple pixels.

[0035] Here, we will explain the focus detection method using image plane phase difference detection. In the pixel 211R (211Gr, 211B) shown in Figure 2(b), the microlens 211i performs pupil division in the x direction by imaging light beams from different regions in the x direction of the exit pupil of the optical system onto the photoelectric conversion unit 211A and the photoelectric conversion unit 211B. In the pixel 211Gb shown in Figure 2(c), the microlens 211i performs pupil division in the y direction by imaging light beams from different regions in the y direction of the exit pupil of the optical system onto the photoelectric conversion unit 211C and the photoelectric conversion unit 211D.

[0036] For each of the multiple pixels 211R within a predetermined range contained in the same pixel row, an A image signal is generated by combining the photoelectric conversion signals obtained from one of the pair of photoelectric conversion units 211A and 211B, and a B image signal is generated by combining the photoelectric conversion signals obtained from the other. The imaging plane phase difference focus detection unit 129 calculates the phase difference (image shift amount), which is the relative shift amount of these pair of image signals, the A image and the B image signal, using correlation calculation, and calculates (acquires) the amount of defocus in the x direction within a predetermined range from this phase difference.

[0037] Similarly, for each of the multiple pixels 211Gb within a predetermined range contained in the same pixel row, a C image signal is generated by combining the photoelectric conversion signals obtained from one of the pair of photoelectric conversion units 211C and 211D, and a D image signal is generated by combining the photoelectric conversion signals obtained from the other. The imaging plane phase-difference focus detection unit 129 calculates the phase difference between these pair of image signals (C image and D image signals) and calculates the amount of defocus in the y-direction within a predetermined range from this phase difference. The signal obtained by adding the photoelectric conversion signals from photoelectric conversion units 211A and 211B and the signal obtained by adding the photoelectric conversion signals from photoelectric conversion units 211C and 211D each form one pixel of the display / recording image data.

[0038] Figures 3(a) and 3(b) show examples of focus detection regions (hereinafter referred to as AF regions) 302 set on the imaging surface 301 of the image sensor 122. Figure 3(a) shows an example of correlation calculation in the x-direction. Shift regions 303 are set to the left and right of the AF region 302 to shift a pair of image signals during correlation calculation. Therefore, the pixel region 304, which is the sum of the AF region 302 and the left and right shift regions 303, becomes the pixel region required for correlation calculation in the x-direction. In the figures, p and q represent the x-coordinates of the start and end points of the pixel region 304, respectively, and s and t represent the x-coordinates of the start and end points of the AF region 302, respectively.

[0039] Figure 3(b) shows an example of performing correlation calculation in the y direction. Shift regions 305 are set above and below the AF region 302. Therefore, the pixel region 306, which is the sum of the AF region 302 and the upper and lower shift regions 305, becomes the pixel region required for correlation calculation in the y direction. In the figure, p' and q' represent the y coordinates of the start and end points of the pixel region 364, respectively, and s' and t' represent the y coordinates of the start and end points of the AF region 302, respectively. In the following, correlation calculation in the x direction will be explained using x coordinates p, q, s, and t, but correlation calculation in the y direction can be performed by replacing these with y coordinates p', q', s', and t', and by replacing the A and B image signals with the C and D image signals.

[0040] Figure 4(a) shows the A-image signal 401 and B-image signal 402 generated using photoelectric conversion signals acquired from multiple pixels (photoelectric conversion units 211A, 211B) included in the AF region 302 shown in Figure 3(a). Figures 4(b) and (c) show the A-image and B-image signals 401 and 402 shifted in the positive and negative directions, respectively, from the state shown in Figure 4(a). When calculating the correlation between the A-image and B-image signals 401 and 402, the A-image and B-image signals 401 and 402 are shifted by one bit in the direction of the arrows.

[0041] Then, for each shift, the sum of the absolute values ​​of the differences between the A image and B image signals 401 and 402 is calculated. When the shift amount is i, the maximum shift amount in the negative direction is ps, the maximum shift amount in the positive direction is qt, x is the start coordinate of the AF region 302, and y is the end coordinate of the AF region 302, the correlation amount COR can be calculated by the following equation (1).

[0042]

number

[0043] Figure 5(a) shows an example of the relationship between the shift amount and the correlation amount COR. The horizontal axis represents the shift amount, and the vertical axis represents the correlation amount COR. Among the regions 502 and 503 near the extreme values ​​of the correlation amount 501, which changes with the shift amount, the degree of agreement between the A image and B image signals is highest at the shift amount corresponding to the smaller correlation amount.

[0044] Next, the difference in correlation amounts at every shift in the waveform of correlation amount 501 shown in Figure 5(a) is calculated as the correlation change. When the shift amount is i, the maximum shift amount in the negative direction is ps, and the maximum shift amount in the positive direction is qt, the correlation change ΔCOR can be calculated by the following equation (2).

[0045]

number

[0046] Figure 6(a) shows an example of the relationship between the shift amount and the correlation change ΔCOR. The horizontal axis represents the shift amount, and the vertical axis represents the correlation change ΔCOR. The correlation change 601, which changes with the shift amount, changes from positive to negative in regions 602 and 603. The point where the correlation change becomes zero is called the zero-crossing, and the degree of agreement between the A image and B image signals is highest at this point. Therefore, the shift amount that gives rise to the zero-crossing is the image displacement amount.

[0047] Figure 6(b) shows an enlarged view of region 602 in Figure 6(a). The shaded area 604 is part of the correlation change 601. The shift amount (k-1+α) that gives zero crossing can be divided into an integer part β (=k-1) and a fractional part α. The fractional part α can be calculated by the following equation (3) based on the similarity relationship between triangle ABC and triangle ADE in the figure.

[0048]

number

[0049] The integer part β can be calculated from Figure 6(b) using the following equation (4). [Math 4] β=k-1 (4) Then, the image shift amount (prediction) can be obtained from the sum of α and β. Note that, as shown in Figure 6(a), if there are multiple zero-crossings of the correlation change amount ΔCOR, the one with the steeper change in the correlation change amount ΔCOR in that vicinity is designated as the first zero-crossing. This steepness is an indicator of how easy it is to perform autofocus (AF), and a larger value indicates that it is easier to perform high-precision AF at that point. The steepness maxder can be calculated by the following equation (5).

[0050]

number

[0051] Thus, in this embodiment, if there are multiple zero-crossings of the correlation change amount, the first zero-crossing is determined by its steepness, and the shift amount that gives this first zero-crossing is defined as the image displacement amount.

[0052] The reliability of the image misalignment amount (hereinafter also referred to as reliability) can be defined by the degree of agreement between the A image signal and the B image signal (hereinafter referred to as the two-image agreement) fnclvl and the steepness of the correlation change amount described above. The two-image agreement is an indicator of the accuracy of the image misalignment amount, and in the correlation calculation method in this embodiment, a smaller value indicates higher accuracy. Figure 5(b) shows an enlarged view of region 502 in Figure 5(a). Curve 504 is part of the correlation quantification 501. The two-image agreement score fnclvl can be calculated by the following equation (6).

[0053]

number

[0054] (Camera body processing 120) The camera MPU125 performs the following processes according to the program. First, the flowchart in Figure 7 shows the imaging process. S stands for step.

[0055] In S701, the camera MPU 125 determines whether or not a command to start video recording has been input via the operation switch group 127. A command to start video recording is notified when the video recording switch on the operation switch group 127 is pressed when video recording is not in progress. If a command to start video recording is notified, the camera MPU 125 proceeds to S702; otherwise, it proceeds to S709.

[0056] In S702, the camera MPU 125 performs video capture processing, records the video image data as recording image data to memory 128, and then proceeds to processing in S703.

[0057] In S703, the camera MPU125 cancels the illumination of the assist light from the focus detection assist light source 131 to the subject. This is because, as described later in S719, the assist light is only illuminated when an AF instruction (focus control instruction) is given, and the assist light is not illuminated when no AF instruction is given or when video is being recorded.

[0058] Next, in S704, the camera MPU125 sets the state to execute autofocus when no AF instruction is given (hereinafter referred to as the AF execution state when no AF instruction is given) and proceeds to S705.

[0059] In S705, the camera MPU 125 causes the imaging plane phase-detection unit 129 to perform the focus detection process described using Figures 3(a) to 6(b) to obtain the defocus amount in the x and y directions and the reliability of each defocus amount (reliability of image shift amount), and then proceeds to processing in S706.

[0060] In the S706, the camera MPU125 performs a process to determine the amount of defocus used for autofocus (AF) from the defocus amounts in the x and y directions. Details of this process for determining the amount of defocus to use will be described later.

[0061] Next, in S707, the camera MPU125 determines whether or not AF is in a state of no AF execution. If it is not in a state of no AF execution, that is, if it is set to AF execution when no AF instruction is given in S704, or if it is set to AF execution when an AF instruction is given in S720 (described later), the process proceeds to S708. Also, if it is set to AF no execution in S717 (described later), the imaging process ends immediately.

[0062] In S708, the camera MPU 125 performs AF processing to control the focus of the optical system and then terminates the imaging process. Details of the AF processing will be described later. Note that in optical system focus control, not only may the focus lens 104 be moved in the optical axis direction, but the image sensor 122 may also be moved in the optical axis direction.

[0063] Meanwhile, in S709, the camera MPU125 determines whether or not video recording is in progress. If video recording is in progress, processing proceeds to S710; otherwise, processing proceeds to S712.

[0064] In S710, the camera MPU 125 determines whether or not a command to stop video recording has been input via the operation switch group 127. A command to stop video recording is notified when the video recording switch on the operation switch group 127 is pressed during video recording. If a command to stop video recording is notified, the camera MPU 125 proceeds to S711, and if no command to stop video recording is notified, it proceeds to S702, thereby continuing the video recording process.

[0065] In S711, the camera MPU 125 stops video capture, stops recording video data to memory 128, and proceeds to S712.

[0066] In step S712, the camera MPU 125 determines whether an AF instruction, which is an instruction to start AF processing, has been input via the operation switch group 127. An AF instruction is input when the release switch of the operation switch group 127 is half-pressed or when the AF start switch is pressed. If no AF instruction has been input, the camera MPU 125 proceeds to step S713; if an AF instruction has been input, it proceeds to step S719.

[0067] In S713, the camera MPU 125 determines whether or not an imaging command has been input from the operation switch group 127. An imaging command is notified when the release switch of the operation switch group 127 is fully pressed. If no imaging command has been notified, the camera MPU 125 proceeds to S714; if an imaging command has been notified, it proceeds to S715.

[0068] In S714, the camera MPU125 releases the focus lock state. The focus lock state is a state in which AF is controlled by the AF processing in S708. When an imaging command is input in S713, if the focus lock state is in S715 (described later), still image capture processing is performed in S718. However, if no imaging command is input in S713, the focus lock state is released in preparation for the next image capture. Then the process proceeds to S703.

[0069] In S715, the camera MPU125 determines whether or not the camera is in a focus-stopped state. If it is in a focus-stopped state, it proceeds to S716; otherwise, it proceeds to S720.

[0070] In S716, the camera MPU125 disables the auxiliary light illumination and proceeds to processing in S717. Since still image capture processing will be performed in S718, which will be described later, the auxiliary light is not illuminated on the subject at this stage.

[0071] In S717, the camera MPU125 is set to a state where AF is not performed, and processing proceeds to S718.

[0072] In S718, the camera MPU 125 performs still image capture processing, records the image data for recording in memory 128, and then proceeds to processing in S705.

[0073] In S719, the camera MPU 125 performs a process to determine whether or not to illuminate with auxiliary light from the focus detection auxiliary light source 131. Details of this auxiliary light illumination determination process will be described later.

[0074] Next, in the S720, the camera MPU125 sets the AF execution state based on the AF instruction and proceeds to process the S705.

[0075] The flowchart in Figure 8 shows the auxiliary light irradiation determination process performed by the camera MPU125 at S719 in Figure 7.

[0076] In S801, the camera MPU 125 determines whether the camera body 120 is set to illuminate the subject with auxiliary light from the focus detection auxiliary light source 131. Specifically, it diagnoses whether the user has selected the setting to illuminate auxiliary light, which is provided as a user-selectable menu. If the setting to illuminate auxiliary light is enabled, the process proceeds to S802; otherwise, the auxiliary light illumination determination process ends.

[0077] In S802, the camera MPU 125 determines whether the brightness of the imaging environment is darker than a predetermined brightness based on brightness information obtained from brightness determination image data generated by the image processing circuit 124. Preferably, the criterion for determining whether it is dark is whether autofocus (AF) is difficult without illuminating with auxiliary light. If it is dark, the process proceeds to S803; otherwise, the auxiliary light illumination determination process ends.

[0078] In S803, the camera MPU 125 causes the focus detection auxiliary light source 131 to illuminate the subject with auxiliary light. Here, the focus detection auxiliary light source 131 illuminates with red auxiliary light. Then the auxiliary light illumination determination process ends.

[0079] The flowchart in Figure 9 shows the process of determining the amount of defocus used, which is performed by the camera MPU 125 at S706 in Figure 7.

[0080] In S901, the camera MPU125 determines whether or not the auxiliary light is illuminating the device. If the auxiliary light is illuminating the device, the process proceeds to S902; otherwise, the process proceeds to S903.

[0081] In S902, the camera MPU125 decides to use the defocus amount in the x-direction. Then, the process of determining the amount of defocus to use is terminated.

[0082] Using Figure 13, we will explain why the x-direction defocus amount is used in S902 when the auxiliary light is illuminating in S901. The left side of Figure 13 shows an example of an imaging scene in which a red auxiliary light 1304 is illuminating a subject 1302 in the imaging screen 1301, and an AF area 1303 is set on a part of the subject 1302 (face). The right side of Figure 13 shows examples of A and B image signals obtained from pixels (left and right photoelectric conversion units) 211R on the image sensor 122 shown in Figure 2(b), and pixels (upper and right) shown in Figure 2(c). under Examples of C and D image signals obtained from the side photoelectric conversion unit (211Gb) are shown.

[0083] When illuminated by the red auxiliary light 1304, the pixel 211R, which detects the phase difference in the x-direction, can acquire A and B image signals with a high degree of agreement, and the phase difference can be detected with high accuracy from the A and B image signals. On the other hand, the pixel 211Gb, which detects the phase difference in the y-direction, only yields C and D image signals with low signal levels and low agreement, and there is a risk that an incorrect phase difference will be detected from the C and D image signals. If autofocus is performed based on the amount of defocus calculated from the incorrectly detected phase difference, the focus state may not be achieved or the focus lens 104 may be driven in the wrong direction.

[0084] Therefore, in this embodiment, when the auxiliary light 1304 is irradiated, AF is performed using the defocus amount calculated from the phase difference detected by the pixel 211R and the pixel 211Gb of the same color as the auxiliary light. That is, when the red auxiliary light 1304 is irradiated, the y-direction defocus amount obtained from the green pixel 211Gb is not used, and the x-direction defocus amount obtained from the red pixel 211R is used.

[0085] The color of the auxiliary light may also be blue or green. Furthermore, the combination of pixel color and the division direction of the photoelectric conversion unit may be other than those shown in Figures 2(a) to (c). For example, as shown in Figure 14(a), the photoelectric conversion unit of the red pixel 212R may be divided in the y direction, and the photoelectric conversion units of the green pixels 212Gr, 212Gb and the blue pixel 212B may be divided in the x direction. In this case, if the color of the auxiliary light is red, the amount of defocus in the y direction obtained from pixel 212R will be used for AF. Also, as shown in Figure 14(b), the photoelectric conversion units of the red pixel 213R and the green pixels 213Gr, 213Gb may be divided in the x direction, and the photoelectric conversion unit of the blue pixel 213B may be divided in the y direction. Also, as shown in Figure 14(c), the photoelectric conversion unit of the red pixel 214R and the green pixel 214Gr may be divided in the x direction, and the photoelectric conversion unit of the other green pixel 214Gb and the blue pixel 214B may be divided in the y direction.

[0086] In S903, the camera MPU 125 determines whether a subject (hereinafter referred to as the horizontal cage subject) is detected by the subject detection unit 130 through a cage, which is an object extending horizontally in the screen direction (x direction: linear direction). If the horizontal cage subject is detected, the process proceeds to S902 to decide to use the defocus amount in the x direction; otherwise, the process proceeds to S904.

[0087] In S904, the camera MPU 125 determines whether a subject (hereinafter referred to as the "vertical cage subject") has been detected by the subject detection unit 130 through a cage extending in the vertical direction (y direction: linear direction) of the screen. If the vertical cage subject is detected, the process proceeds to S905; otherwise, the process proceeds to S906.

[0088] In S905, the camera MPU125 decides to use the defocus amount in the y-direction. Then, the process of determining the amount of defocus to use is terminated.

[0089] Using Figures 15(a) and (b), we will explain why the x-direction defocus amount is used in S902 when a horizontal cage subject is detected in S903, and why the y-direction defocus amount is used in S905 when a vertical cage subject is detected in S904.

[0090] The left side of Figure 15(a) shows an imaging scene in which a subject 1502 is captured behind a cage 1504 that extends vertically within the imaging screen 1501, and an AF area 1503 is set on a part of the subject 1502 (face). The right side of Figure 15(a) shows examples of A and B image signals obtained from pixels (left and right photoelectric conversion units) 211R on the image sensor 122 shown in Figure 2(b), and examples of C and D image signals obtained from pixels (upper and lower photoelectric conversion units) 211Gb shown in Figure 2(c).

[0091] In this imaging scene, the AF area 1503 includes not only the subject 1502 but also the vertically extending cage 1504. Therefore, the A and B image signals obtained from pixel 211R include not only the change component due to subject 1502 but also the change component due to cage 1504, resulting in a state where the distant subject 1502 and the nearby cage 1504 are competing. In this state, even if AF is performed using the defocus amount obtained from the phase difference of the A and B image signals, there is a risk that the subject 1502 will not be in focus and only the cage 1504 will be in focus, or that neither subject 1502 nor cage 1504 will be in focus. On the other hand, the C and D image signals obtained from pixel 211Gb include the change component due to subject 1502 but do not include the change component due to cage 1504. Therefore, the phase difference and defocus amount for subject 1502 can be obtained from the C and D image signals, enabling good AF. Therefore, if a vertical cage subject is detected in S904, it is decided in S905 to use the amount of defocus in the y direction.

[0092] The left side of Figure 15(b) shows an imaging scene in which a subject 1502 is captured behind a cage 1505 that extends horizontally within the imaging screen 1501, and an AF area 1503 is set on a part of the subject 1502 (face). The right side of Figure 15(b) shows examples of A and B image signals obtained from pixels (left and right photoelectric conversion units) 211R, and examples of C and D image signals obtained from pixels (upper and lower photoelectric conversion units) 211Gb.

[0093] In this imaging scene, the AF area 1503 includes not only the subject 1502 but also the horizontally extending cage 1505. Therefore, the C and D image signals obtained from pixel 211Gb include not only the change component due to subject 1502 but also the change component due to cage 1505, resulting in a state where the distant subject 1502 and the nearby cage 1505 are competing, causing the same problem as in Figure 15(a). On the other hand, the A and B image signals obtained from pixel 211R include the change component due to subject 1502 but not the change component due to cage 1505, so the phase difference and defocus amount for subject 1502 can be obtained from the A and B image signals. Therefore, when a horizontal cage subject is detected in S903, it is decided in S902 to use the defocus amount in the x direction.

[0094] In S906, the camera MPU 125 determines whether or not the horizon has been detected as a subject by the subject detection unit 130. If the horizon has been detected, the process proceeds to S907; otherwise, the process proceeds to S909.

[0095] In S907, the camera MPU125 determines whether the reliability of the defocus amount in the x-direction is above a predetermined level. If it is above the predetermined level, the process proceeds to S908; otherwise, the process proceeds to S905 to decide to use the defocus amount in the y-direction.

[0096] In S908, the camera MPU125 decides to use the weighted average value (hereinafter referred to as the weighted average value) of the defocus amount in the x-direction and the defocus amount in the y-direction at a predetermined ratio. Then, it terminates the process of determining the amount of defocus to use.

[0097] Using Figures 16(a) and (b), if a horizontal line is detected in S906, then S90 7 Depending on the reliability of the defocus amount in the x direction, S90 5 at y Explain why you should use the directional defocus amount or, in S908, use the weighted average of the x and y direction defocus amounts.

[0098] The left side of Figure 16(a) shows an example of an imaging scene in which a horizon line 1602 is captured within the imaging screen 1601, and the AF area 1603 is set on the horizon line 1602. The right side of Figure 16(a) shows examples of A and B image signals obtained from pixels (left and right photoelectric conversion units) 211R on the image sensor 122 shown in Figure 2(b), and examples of C and D image signals obtained from pixels (upper and lower photoelectric conversion units) 211Gb shown in Figure 2(c).

[0099] Because the horizontal line 1602 lacks contrast in the horizontal direction of the imaging screen 1601, the signal amount in the A and B image signals obtained from pixel 211R may be low, potentially preventing accurate detection of the phase difference. On the other hand, the horizontal line 1602 has contrast in the vertical direction of the imaging screen 1601, allowing for accurate detection of the phase difference in the C and D image signals obtained from pixel 211Gb. Therefore, by basically using the y-direction defocus amount in S905, accurate focusing on the horizontal line 1602 is possible.

[0100] However, in low-light conditions such as at night or on cloudy days, or when high shutter speeds or small apertures are set, the accuracy of defocus detection may decrease due to a decrease in signal strength and an increase in noise. For this reason, it is desirable to improve the accuracy of defocus detection by using the defocus amount in the x-direction as well. Therefore, if the reliability of the defocus amount in the x-direction is above a predetermined level in S907, the accuracy of focus detection is improved by using the weighted average value of the defocus amounts in the x and y directions in S908.

[0101] The predetermined level used as a reliability threshold in the S907 is preferably set to a level at which sufficient focusing accuracy can be obtained during autofocus. For example, as shown on the left side of Figure 16(b), when the imaging screen 1601 (AF area 1603) is tilted relative to the horizontal line 1602, it is possible to detect a phase difference from the A and B image signals obtained from the pixel 211R shown on the right side of Figure 16(b), although the signal amounts are small.

[0102] The S908 processing assumes such a case. Note that the weighting ratio performed in S908 can be 1:1, but it can also be changed according to the contrast level of the A-D image signals. For example, as shown in Figure 16(b), if the contrast of the C and D image signals is higher than that of the A and B image signals, the weight of the y-direction defocus amount obtained from the C and D image signals is changed to the weight of the x-direction defocus amount obtained from the A and B image signals. twist Increase the value. Additionally, the weighting ratio may be changed depending on information regarding the reliability of the defocus amount (such as the degree of two-image agreement and steepness).

[0103] In the S909, the camera MPU125 performs a determination process for using the stabilization defocus amount. Details of the stabilization defocus amount and the determination process for using the stabilization defocus amount will be described later.

[0104] Next, in S910, the camera MPU125 determines whether the conditions for using a stabilizing defocus amount as the defocus amount used for AF are met. If the conditions are met, the process proceeds to S911; otherwise, the process proceeds to S912.

[0105] In S911, the camera MPU125 performs a process to determine the amount of defocus used for stabilization. Details of this process will be described later. Then the process to determine the amount of defocus used ends.

[0106] In S912, the camera MPU125 determines whether the reliability of the defocus amount in the x-direction is above a predetermined level. If it is above the predetermined level, the process proceeds to S913; otherwise, the process proceeds to S914.

[0107] In S913, the camera MPU125 determines whether the reliability of the defocus amount in the y direction is above a predetermined level. If it is above the predetermined level, the process proceeds to S908; otherwise, the process proceeds to S902.

[0108] In S914, the camera MPU125, similar to S913, determines whether the reliability of the defocus amount in the y-direction is above a predetermined level. If it is above the predetermined level, the process proceeds to S905; otherwise, the process proceeds to S908.

[0109] In the S912, S913, and S914 processes, the amount of defocus to be used is determined based on the reliability of the defocus amounts in the x and y directions. The predetermined threshold level for reliability in S912, S913, and S914 is preferably set to a level that provides sufficient focusing accuracy during autofocus, similar to S907. In S912 and S913, if the reliability of the defocus amounts in both the x and y directions is above the predetermined level, i.e., if the defocus amounts in both directions meet the conditions for using them in autofocus, S908 determines that the defocus amounts in the x and y directions are weighted and added together for use. This allows for the use of highly accurate defocus amounts in autofocus.

[0110] If the reliability of the defocus amount in the x-direction is above a predetermined level in S912, but the reliability of the defocus amount in the y-direction is not above a predetermined level in S913, then in S902, it is decided to use the defocus amount in the x-direction. If the reliability of the defocus amount in the x-direction is not above a predetermined level in S912, but the reliability of the defocus amount in the y-direction is above a predetermined level in S914, then in S905, it is decided to use the defocus amount in the y-direction. If the reliability of the defocus amounts in both the x-direction and y-direction is not above a predetermined level in both S912 and S914, then in S908, it is decided to use the weighted average of the defocus amounts in the x-direction and y-direction. The weighting ratio in S908 may be 1:1, or it may be changed according to the contrast level of the A-D image signals.

[0111] The flowchart in Figure 10 illustrates the process performed by the camera MPU 125 in S909 to determine the use of the stabilization defocus amount.

[0112] In S1001, the camera MPU125 determines whether or not AF is running when no AF instruction is given. If AF is running when no AF instruction is given, the process proceeds to S1002; otherwise, the process proceeds to S1008.

[0113] In S1002, the camera MPU 125 determines whether the brightness of the imaging environment is darker than a predetermined brightness based on brightness information obtained from brightness determination image data generated by the image processing circuit 124. Preferably, the criterion for determining whether it is dark is whether the variability in the detection accuracy of the amount of defocus becomes greater than a predetermined amount. If it is dark, the process proceeds to S1004; otherwise, the process proceeds to S1003.

[0114] In S1003, the camera MPU 125 determines whether the camera body 120 is set to reduce the amount of the imaging signal. Settings that reduce the amount of the imaging signal include, for example, when the exposure compensation setting via the operation switch group 127 is set to be darker than the correct exposure, or when Log imaging is set. If the setting to reduce the amount of the imaging signal is set, the process proceeds to S1004; otherwise, the process proceeds to S1008.

[0115] In S1004, the camera MPU125 determines whether the absolute value of the defocus amount in the x direction is less than a predetermined value. If it is less than the predetermined value, the process proceeds to S1005; otherwise, the process proceeds to S1008.

[0116] In S1005, the camera MPU125 determines whether the absolute value of the defocus amount in the y direction is less than a predetermined value. If it is less than the predetermined value, the process proceeds to S1006; otherwise, the process proceeds to S1008.

[0117] In S1006, the camera MPU125 determines whether the difference in the amount of defocus in the x and y directions is less than a predetermined value. If it is less than the predetermined value, the process proceeds to S1007; otherwise, the process proceeds to S1008.

[0118] In S1007, the camera MPU125 determines to use the stabilization defocus amount. Then, the process of determining whether to use the stabilization defocus amount is terminated.

[0119] In S1008, the camera MPU125 decides not to use the stabilization defocus amount. Then, it terminates the process for determining whether to use the stabilization defocus amount.

[0120] Based on the results of the above stabilization defocus amount usage determination process, the camera MPU125 determines in S910 of Figure 9 whether or not the conditions for using the stabilization defocus amount are met. If the conditions for using the stabilization defocus amount are met, the stabilization defocus amount usage determination process is performed in S911. Details of the decision reason in the process of Figure 10 will be explained after the detailed explanation of the stabilization defocus amount usage determination process below.

[0121] The flowchart in Figure 11 shows the process performed by the camera MPU 125 in S911 to determine the amount of defocus used for stabilization.

[0122] In S1101, the camera MPU125 determines whether the reliability of the defocus amount in the x-direction is above a predetermined level. If it is above the predetermined level, the process proceeds to S1102; otherwise, the process proceeds to S1108.

[0123] In S1102, the camera MPU125 determines whether the reliability of the defocus amount in the y direction is above a predetermined level. If it is above the predetermined level, the process proceeds to S1103; otherwise, the process proceeds to S1106. The predetermined level used as the threshold for reliability in S1101 and S1102 is preferably set to a level that provides sufficient focusing accuracy during autofocus, similar to S907 and S912-S914.

[0124] In S1103, the camera MPU125 determines whether the direction of defocus in the x-direction (hereinafter simply referred to as the direction of the defocus amount) is in the near direction. If it is in the near direction, the process proceeds to S1104; if it is not in the near direction, i.e., in the infinity direction, the process proceeds to S1109.

[0125] In S1104, the camera MPU125 determines whether the direction of the defocus amount in the y-direction is the near direction or not. If it is the near direction, the process proceeds to S1105; if it is the infinity direction, the process proceeds to S1110.

[0126] In S1105, the camera MPU125 determines whether the absolute value of the defocus amount in the x direction is smaller than the absolute value of the defocus amount in the y direction. If it is smaller, the process proceeds to S1106; otherwise, the process proceeds to S1107.

[0127] In S1106, the camera MPU125 decides to use the defocus amount in the x-direction as the defocus amount for stabilization. Then, the process of determining the defocus amount to use for stabilization is terminated.

[0128] In S1107, the camera MPU125 decides to use the y-direction defocus amount as the defocus amount for stabilization. Then, the process of determining the defocus amount to use for stabilization is terminated.

[0129] In S1108, the camera MPU125, similar to S1102, determines whether the reliability of the defocus amount in the y-direction is above a predetermined level. If it is above the predetermined level, the process proceeds to S1107; otherwise, the process proceeds to S1105. The predetermined level used as the threshold for reliability in S1108 is preferably set to a level that provides sufficient focusing accuracy during autofocus, similar to S1102.

[0130] In S1109, the camera MPU125, similar to S1104, determines whether the direction of the defocus amount in the y-direction is the near direction or not. If it is the near direction, it proceeds to S1110; if it is the infinity direction, it proceeds to S1105.

[0131] In S1110, the camera MPU125 determines whether the absolute value of the weighted average of the defocus amounts in the x and y directions is smaller than the absolute values ​​of the defocus amounts in the x and y directions. If the absolute value of the weighted average is smaller than the absolute value of the defocus amounts in each direction, the process proceeds to S1111; otherwise, the process proceeds to S1105.

[0132] In S1111, the camera MPU125 decides to use the weighted average of the defocus amounts in the x and y directions as the defocus amount for stabilization. Then, the process of determining the defocus amount to use for stabilization is terminated.

[0133] Using Figures 17 and 18, the details of the process for determining the amount of defocus used for stabilization shown in Figure 10 and the process for determining the amount of defocus used for stabilization shown in Figure 11 will be explained. The left side of Figure 17 shows an example of an imaging scene in which a person 1702 as a subject is captured within the imaging screen 1701, and the AF area 1703 is set on a part of the person 1702 (face). The right side of Figure 17 shows examples of A and B image signals obtained from pixels (left and right photoelectric conversion units) 211R on the image sensor 122 shown in Figure 2(b), and pixels (upper and right) shown in Figure 2(c). under Examples of C and D image signals obtained from the side photoelectric conversion unit (211Gb) are shown.

[0134] In this imaging scene, light shines in from the background window, making the person 1702 appear dark due to backlighting, and the signal strength of the A-D image signals obtained in the AF area 1703 is reduced. Furthermore, if a setting that reduces signal strength, such as Log imaging, is used, the reduction in the signal strength of the A-D image signals becomes even more pronounced. In such situations, there is a risk that the phase difference cannot be accurately detected using either the A and B image signals or the C and D image signals.

[0135] Figure 18 shows the time variation of the defocus amount obtained under backlit conditions and other signal-reducing conditions shown in Figure 17. The vertical axis represents the defocus amount. 0 in the center corresponds to the in-focus state, with the area above 0 representing the defocus amount in the + direction (near direction) and the area below 0 representing the defocus amount in the - direction (infinity direction). The horizontal axis represents time, and here it shows the defocus amount from time A to time F. The dashed line 1801 connecting the circular markers at times A to F represents the defocus amount in the x direction, and the dotted line 1802 connecting the square markers represents the defocus amount in the y direction. Due to changes in the amount of light incident on the pixels and the contrast strength of the subject, both the defocus amounts in the x and y directions vary greatly in the near and infinity directions relative to the in-focus state as time progresses. As the defocus amount varies in the near and infinity directions in this way, the focus lens 104 continues to move to follow it, causing the image to repeatedly become in focus and then out of focus. In particular, if the focus changes between sharp and blurred during video recording, the resulting video will be of low quality.

[0136] The dashed line 1803 connecting the triangular markers represents the defocus amount as a weighted average of the defocus amounts in the x and y directions in a 1:1 ratio. The double-dotted line 1804 connecting the diamond markers represents the stabilization defocus amount determined in the stabilization defocus amount determination process shown in Figure 11. The stabilization defocus amount 1804 exhibits less variation in the focus state compared to the other defocus amounts 1801-1803. Therefore, using the stabilization defocus amount 1804 in AF can reduce focus fluctuations caused by the aforementioned variations in defocus amounts.

[0137] The stabilization defocus amount 1804 determined at each time point in Figure 18 by the stabilization defocus amount determination process in Figure 11 will be explained. However, the following explanation assumes that the reliability of the defocus amounts in the x and y directions is determined to be above a predetermined level at S1101 and S1102 in Figure 11.

[0138] First, in S1103 and S1104 of Figure 11, when the directions of the defocus amounts in the x and y directions are both in the near direction, that is, at time C in Figure 18, S1105 determines the defocus amount with the smaller absolute value among the defocus amounts in the x and y directions. At time C, the absolute value of the defocus amount in the y direction is smaller than the absolute value of the defocus amount in the x direction, so at S1107, the defocus amount 1802 in the y direction is determined as the stabilizing defocus amount 1804. At time C, among the defocus amounts 1801 to 1803, the absolute value of the defocus amount 1802 in the y direction is the smallest, and it is a defocus amount that is less likely to cause AF instability near the in-focus state.

[0139] Furthermore, in S1103 and S1109 of Figure 11, if the direction of the defocus amount in both the x and y directions is towards infinity, that is, at time E in Figure 18, the determination in S1105 is also made. At time E, the absolute value of the defocus amount in the x direction is smaller than the absolute value of the defocus amount in the y direction, so in S1106, the defocus amount 1801 in the x direction is determined as the stabilizing defocus amount 1804. At time E, the absolute value of the defocus amount 1801 in the x direction is the smallest among the defocus amounts 1801 to 1803, and it is a defocus amount that is less likely to cause AF instability near the focus state.

[0140] When the defocus amount in the x-direction and the defocus amount in the y-direction are in the same direction, as in times C and E, the AF instability can be reduced by determining the defocus amount with the smaller absolute value as the stabilizing defocus amount.

[0141] Furthermore, if the direction of the defocus amount in the x-direction is the near direction in S1103 of Figure 11, and the direction of the defocus amount in the y-direction is the infinity direction in S1104, that is, at times B and F in Figure 18, it is determined in S1110 whether the absolute value of the weighted average value 1803 of the defocus amounts in the x and y directions is smaller than the absolute values ​​of the defocus amounts 1801 and 1802 in the x and y directions. At times B and F, the absolute value of the weighted average value 1803 is smaller than the absolute values ​​of the defocus amounts 1801 and 1802 in the x and y directions, so in S1111 the weighted average value 1803 is determined as the stabilizing defocus amount 1804.

[0142] Furthermore, in Figure 11, if the defocus amount in the x-direction is in the direction of infinity at S1103 and the defocus amount in the y-direction is in the direction of near at S1109, that is, at times A and D in Figure 18, the same determination is made at S1110 as at times B and F. At time D, as at times B and F, the absolute value of the weighted average value 1803 is smaller than the absolute values ​​of the defocus amounts 1801 and 1802 in the x and y directions, so at S1111, the weighted average value 1803 is determined as the stabilizing defocus amount 1804. At times B, D, and F, the absolute value of the weighted average value 1803 is the smallest among the defocus amounts 1801 to 1803, and it is a defocus amount that is less likely to cause AF instability near the in-focus state.

[0143] On the other hand, at time A, the absolute value of the y-direction defocus amount 1802 is smaller than the absolute value of the weighted average value 1803, so a determination is made at S1105. At time A, the absolute value of the y-direction defocus amount is smaller than the absolute value of the x-direction defocus amount, so at S1107, the y-direction defocus amount 1802 is determined as the stabilizing defocus amount 1804. At time A, the absolute value of the y-direction defocus amount 1802 is the smallest among the defocus amounts 1801 to 1803, and it is a defocus amount that is less likely to cause AF instability near the in-focus state.

[0144] When the defocus amounts in the x-direction and y-direction are opposite in direction, as in times A, B, D, and F, not only the defocus amounts in the x-direction and y-direction, but also their weighted average value are considered as candidates for the stabilization defocus amount. By determining the defocus amount with the smallest absolute value among these defocus amounts as the stabilization defocus amount, the instability of the autofocus can be reduced.

[0145] Furthermore, if it is determined in steps S1101, S1102, and S1108 of Figure 11 that the reliability of the defocus amount in the x or y direction is not above a predetermined level, the following processing is performed. If the reliability of the defocus amount in either the x or y direction is above a predetermined level, the defocus amount with reliability above a predetermined level is used as the stabilization defocus amount in S1106 or S1107. If the reliability of neither the x nor y direction defocus amount is above a predetermined level, after determination in S1105, the defocus amount with the smaller absolute value of the x or y direction defocus amount is used as the stabilization defocus amount.

[0146] As shown in Figure 17, by determining the amount of defocus used for stabilization in the defocus amount determination process for stabilization, good autofocus with reduced instability can be achieved even when the signal levels of the A-D image signals are low. However, there are drawbacks to this method of determining the amount of defocus used for stabilization. For example, when it is desired to quickly focus using autofocus from a state where the subject is greatly blurred, if a defocus amount with a small absolute value is determined as the defocus amount for stabilization in the processing of S1105-S1107 in Figure 11, it may take longer to focus. Therefore, in the processing of S1004 and S1005 in Figure 10, if the absolute value of the defocus amount in the x or y direction is not less than a predetermined value, i.e., when the degree of blur is large, the amount of defocus used for stabilization is not used in S1008.

[0147] Furthermore, when the user issues an AF command, focusing should be completed as quickly as possible, so any process that may cause focusing to take a long time is undesirable. For this reason, if AF is not in operation when no AF command is issued in S1001 in Figure 10, the stabilizing defocus amount will not be used in S1008.

[0148] Furthermore, as shown in Figure 17, if the subject is not darkened by backlighting, or if settings that reduce signal intensity, such as Log imaging, are not applied, there is no need to determine the amount of defocus for stabilization. In these situations, if AF is performed using a defocus amount with a small absolute value as the amount of defocus for stabilization, there is a risk that tracking may be delayed, for example, when tracking a subject whose distance is changing. For this reason, in S1002 and S1003 of Figure 10, if the imaging environment is not dark and no settings that reduce signal intensity are applied, the amount of defocus for stabilization will not be used in S1008.

[0149] Furthermore, in steps S1110 and S1111 in Figure 11, a weighted average of the defocus amounts in the x and y directions is determined as the defocus amount for stabilization. However, the detection accuracy of the phase difference that forms the basis of one of the defocus amounts in the x and y directions is low. Become In some cases, the reliability of the weighted average value is low, and using this value for autofocus may actually increase blurring. For this reason, in S1006 of Figure 10, if the difference between the defocus amounts in the x and y directions is large (not less than a predetermined value), the stabilizing defocus amount will not be used in S1008.

[0150] The flowchart in Figure 12 shows the AF processing performed by the camera MPU 125 in S708 of Figure 7. In the AF processing, AF is performed using the defocus amount determined in the defocus amount determination process in Figure 9.

[0151] In S1201, the camera MPU125 determines whether or not the camera is in a state of autofocus stoppage. If it is not in a state of autofocus stoppage, it proceeds to S1202; if it is in a state of autofocus stoppage, it proceeds to S1209.

[0152] In S1202, the camera MPU 125 determines whether the reliability of the defocus amount is above a predetermined level. If it is above the predetermined level, the process proceeds to S1203; otherwise, the process proceeds to S1207. Preferably, the predetermined level used as the reliability threshold in S1202 is set to the lowest level of reliability that ensures not only the calculated defocus amount but also the direction of the defocus amount is reliable. The reliability of the defocus amount may be determined using both the degree of two-image agreement and the steepness of the image misalignment, or using only one of them. Other indicators such as signal intensity may also be used.

[0153] In S1203, the camera MPU125 determines whether the amount of defocus is within the depth of field. If it is within the depth of field, it proceeds to S1204; otherwise, it proceeds to S1205.

[0154] In S1204, the camera MPU125 determines that the amount of defocus is within the depth of field and enters a focus-stopped state. Then, it terminates the AF process.

[0155] In S1205, the camera MPU 125, assuming that focus has not yet been achieved, sets the lens drive settings to drive the focus lens 104 based on the amount of defocus. Then, it proceeds to S1206.

[0156] In S1206, the camera MPU 125 sends a drive command for the focus lens 104 to the lens MPU 117 based on the defocus amount and the lens drive settings set in S1205. This drives the focus lens 104 via the lens MPU 117, i.e., autofocus (AF) is performed. After this, the camera MPU 125 terminates the AF process.

[0157] A defocus amount determined to be unreliable in S1202 cannot be used for AF. Therefore, in S1207, the camera MPU125 performs a search drive to calculate the defocus amount while moving the focus lens 104 toward its movable end in order to detect the position of the focus lens 104 where a highly reliable defocus amount can be obtained. The camera MPU125 first sets the lens drive settings for the search drive. The lens drive settings for the search drive include settings such as the drive speed and drive start direction of the focus lens 104.

[0158] In S1208, the camera MPU125 controls the lens MPU117 to focus on the lens based on the lens drive settings for search drive set in S1207. Z1 A drive command 04 is sent. This initiates the search drive of the focus lens 104 via the lens MPU 117. After this, the camera MPU 125 terminates the AF process.

[0159] In S1209, the camera MPU125 first determines whether or not AF is being executed due to an AF command, in order to decide whether or not to maintain the focus-stopped state depending on whether or not an AF command has been issued. If AF is being executed due to an AF command, the process proceeds to S1210. If AF is not being executed due to an AF command, i.e., if AF is being executed during video recording or before an AF command is started, the process proceeds to S1211.

[0160] In the S1210, the camera MPU125 maintains the focus-locked state and terminates the AF process.

[0161] In S1211, the camera MPU125 determines whether the amount of defocus is within the depth of field. If it is within the depth of field, it proceeds to S1210 to maintain the focus-stopped state. If it is not within the depth of field, it proceeds to S1212.

[0162] In S1212, the camera MPU125 determines whether the state in which the amount of defocus is not within the depth of field has continued for a predetermined period of time. If so, it proceeds to S1213; otherwise, it proceeds to S1210.

[0163] During video recording or in AF before AF instruction is initiated, if the amount of defocus remains outside the depth of field for a predetermined period of time after the AF has achieved focus, as indicated in S1211 and S1212, the AF needs to track the increased amount of defocus. For this reason, the camera MPU125 releases the focus stop state in S1213 and terminates the AF process.

[0164] As explained above, in this embodiment, at least one defocus amount to be used from among the defocus amounts in the x and y directions is determined (selected) according to the illumination state and color of the auxiliary light. Since a sufficient image signal cannot be obtained in pixels of a different color from the color of the auxiliary light under the illumination state of the auxiliary light, and phase difference cannot be detected accurately, unstable AF can be prevented by not using the defocus amount obtained using pixels of a different color from the color of the auxiliary light. In addition, when the subject detection unit 130 detects a cage or the horizon, the change in contrast in one of the x and y directions is small, and phase difference cannot be detected accurately, so the use of that defocus amount is restricted. This prevents unstable AF.

[0165] Furthermore, in imaging environments where the amount of light received from the subject (i.e., the amount of signal from the pixels) is low, or in situations where the amount of defocus varies greatly, such as when the camera body 120 is set to reduce the signal amount, a stable defocus amount is used for autofocus. Specifically, if the directions of the defocus amounts in the x and y directions are the same, the defocus amount with the smaller absolute value is used as the stable defocus amount. If the directions of the defocus amounts in the x and y directions are different, the defocus amount with the smallest absolute value among the defocus amounts in the x and y directions and their weighted average value is used as the stable defocus amount. This improves the stability of autofocus.

[0166] In this embodiment, the image sensor 122 has a configuration in which the photoelectric conversion unit is divided into two pixels 211R, 211Gr, and 211B in the x-direction and two pixels 211Gb in the y-direction. However, other configurations are also possible. For example, as shown in Figure 33, a configuration is also possible in which pixels 211R, 211Gr, 211Gb, and 211B, each with a photoelectric conversion unit corresponding to one microlens divided into four parts vertically and horizontally, are used for focus detection. By dividing the photoelectric conversion unit below one microlens into four parts, each photoelectric conversion unit 211J, 211K, 211L, and 211M can receive light from the four divided exit pupil regions. Then, by using the signals from the four photoelectric conversion units that receive light from different exit pupil regions, focus detection can be performed. Here, the image signal generated from the signal acquired from the photoelectric conversion unit 211J of the divided pixels that divide the pupil from the light from the lens unit 100 is called the J image signal. . minutes The image signal generated from the signal acquired from the divided photoelectric conversion unit 211K is called the K image signal. Similarly, the image signals generated from the signals acquired from the photoelectric conversion units 211L and 211M, respectively, are called the L image signal and the M image signal. Furthermore, for each unit pixel, the photoelectric conversion units 211J and 211 L The image signal generated from the sum of the signals from J+ L The image signal is used, and for each unit pixel, the photoelectric conversion unit 211K, 211 M The image signal generated from the sum of the signals from K+ M This will be used as the image signal.

[0167] And J+ L Image signal and K+ M Focus detection is performed using the image signal. During focus detection, J+ L Image signal and K+ M The image signals are combined in the row direction to produce J+ as the output of a group of unit pixels of the same color. L Image signal and K+ M The image signal is generated and converted into data, and the shift between each corresponding point is determined by correlation calculation. Here, J+ L Image signal and K+ M By using the image signal, a horizontal correlation calculation is performed to detect the amount of horizontal image displacement. Also, J+K image signal and M+L image signal By performing a correlation operation on these values, the amount of image displacement in the vertical direction is detected. This allows us to obtain the defocus amounts in the x and y directions. From these values, as described above, we determine (select) at least one defocus amount to use from the x and y defocus amounts, depending on the illumination state and color of the auxiliary light. [Examples]

[0168] Next, Embodiment 2 of the present invention will be described. In this embodiment, the same configurations and processes as in Embodiment 1 will not be described. In this embodiment, the imaging device (camera body) is described as being equipped with the focus detection auxiliary light source 131 shown in Figure 1, but it is not necessary for the camera to be equipped with the focus detection auxiliary light source 131.

[0169] In this embodiment, the image sensor 122 having the pixel arrangement shown in Figure 2(a) is configured to output D-image signals for each color, red, blue, and green, from the photoelectric conversion unit 211D shown in Figure 2(c). This allows the camera MPU 125 to obtain information regarding the image signals for each color.

[0170] In this embodiment, the process for determining the amount of defocus used in S706 in Figure 7 is performed as shown in the flowchart in Figure 19. The processes S1901, S1902, and S1904-S1915 in Figure 19 are the same as the processes S901, S902, and S903-S914 in Figure 9, so their explanation is omitted.

[0171] In S1903, the camera MPU 125 determines whether the signal amount from the green pixel is less than or equal to a predetermined amount than the signal amount (signal level) of the red and blue pixels. If this is the case, the process proceeds to S1902; otherwise, the process proceeds to S1904. The predetermined amount used as the threshold is preferably set so that it can determine whether the signal amount from the green pixel is extremely small compared to the signal amounts from the red and blue pixels.

[0172] In Example 1, as shown in Figure 13, when a red auxiliary light is shone on the subject, the signal amount of the green pixels is low, making it impossible to accurately detect the phase difference. However, factors other than the auxiliary light, such as the color of the light source in the imaging environment or the color of the subject itself, can reduce the signal amount of a particular color, making it impossible to accurately detect the phase difference.

[0173] Therefore, in this embodiment, at S1903 in Figure 19, the signal amounts of the red, blue, and green pixels are compared to determine the imaging environment or subject in which the signal amount of the green pixel is less than that of the red and blue pixels. As shown in Figure 2(c), the phase difference in the y direction is detected by the G pixel, but if the signal amount of the green pixel is less than that of the red and blue pixels, it can be seen that the phase difference detection accuracy in the y direction is low, so the defocus amount in the x direction is used at S1902.

[0174] In this embodiment, the processing when using an image sensor 122 having the pixel arrangement shown in Figure 2(a) has been described, but the image sensor may have other pixel arrangements. For example, when using an image sensor having the pixel arrangement shown in Figure 14(a), if the signal amount of the red pixel 212R is less than the signal amounts of the green and blue pixels 212Gr, 212Gb, and 212B, the defocus amount in the x direction is used. Also, if the signal amounts of the green and blue pixels 212Gr, 212Gb, and 212B are less than the signal amount of the red pixel 212R, the defocus amount in the y direction is used. When using an image sensor having the pixel arrangement shown in Figure 14(b), if the signal amount of the blue pixel 213B is less than the signal amounts of the green and red pixels 213Gr, 213Gb, and 213R, the defocus amount in the x direction is used. Also, if the signal amounts of the green and red pixels 213Gr, 213Gb, and 213R are less than the signal amount of the blue pixel 213B, the defocus amount in the y direction is used. When using an image sensor having the pixel arrangement shown in Figure 14(c), if the signal amount of the red pixel 214R is less than the signal amount of the green and blue pixels 214Gb and 214B, the defocus amount in the y direction is used. Also, if the signal amounts of the green and blue pixels 214Gb and 214B are less than the signal amount of the red pixel 214R, the defocus amount in the x direction is used.

[0175] In this embodiment, depending on the illumination state of the auxiliary light and the signal amounts of the red, blue, and green pixels in S1901 and S1903 of Figure 19, the defocus amount in the x direction is used in S1902, or the process proceeds to S1906 and the defocus amount in the y direction is used. In addition, the process proceeds to S1909 and the weighted average of the defocus amounts in the x and y directions is used. In the weighted addition in S1909, the weights may be changed according to the signal amounts of the red, blue, and green pixels. In this case, the weights may be determined according to the maximum signal amounts of the red, blue, and green pixels, or the sum of the signal amounts of the red, blue, and green pixels, or other weight determination methods may be used.

[0176] Alternatively, the phase-difference focus detection unit 129 on the imaging surface in Figure 1 may calculate the defocus amounts obtained from each of the red, blue, and green pixels, and then calculate the defocus amounts in the x and y directions. In this case, the weighting for averaging the defocus amounts obtained for each pixel may be determined according to the signal amount of each of the red, blue, and green pixels. For example, the ratio of the maximum values ​​of the signals of the red, blue, and green pixels may be used as the weighting ratio for the defocus amounts obtained from the red, blue, and green pixels when calculating the defocus amounts in the x and y directions, or other weighting determination methods may be used. [Examples]

[0177] Next, Embodiment 3 of the present invention will be described. In this embodiment, the same configurations and processes as in Embodiment 1 will not be described. In this embodiment, the focus detection auxiliary light source 131 can emit not only red (first color) auxiliary light but also green (second color) auxiliary light. The camera MPU 125 functions as a selection means and a switching means.

[0178] In this embodiment, the auxiliary light irradiation determination process in S719 of Figure 7 is performed using the flowchart shown in Figure 20. The processes in S2001 and S2003 of Figure 20 are the same as the processes in S801 and S803 of Figure 8, so their explanations are omitted.

[0179] In S2002 of Figure 20, the camera MPU 125 determines whether the brightness of the imaging environment is darker than a first predetermined brightness (state A) based on brightness information obtained from brightness determination image data generated by the image processing circuit 124. The criteria for determining whether the environment is dark (state A) should preferably include not only whether AF is difficult without auxiliary light, but also whether sufficient accuracy can be obtained for detecting the phase difference in the y-direction in the image sensor 122 with the pixel arrangement shown in Figure 2(a). If the environment is dark (state A), the camera MPU 125 proceeds to S2003; otherwise, it proceeds to S2004.

[0180] In the image sensor 122 with the pixel arrangement shown in Figure 2(a), pixels 211R, 211Gr, and 211B for detecting the phase difference in the x-direction are arranged one pixel at a time. On the other hand, pixels 211Gr for detecting the phase difference in the y-direction are arranged every two pixels, and there are fewer pixels of them than those used to detect the phase difference in the x-direction. Therefore, the conversion factor multiplied by the phase difference when calculating the amount of defocus in the y-direction is larger than the conversion factor used to calculate the amount of defocus in the x-direction, and the accuracy of the amount of defocus in the y-direction is lower than the accuracy of the amount of defocus in the x-direction. In addition, the signal noise contained in the pair of image signals obtained from pixels 211Gr, which are fewer in number than pixels 211R, 211Gr, and 211B, tends to be greater, resulting in lower accuracy of the amount of defocus in the y-direction. Based on these characteristics, the criterion for determining dark state A in S2002 is also the presence or absence of sufficient detection accuracy for detecting the phase difference in the y-direction. In other words, if the detection accuracy of the phase difference in the y-direction is insufficient, the process proceeds to S2003; if the accuracy of the phase difference in the y-direction is sufficient, the process proceeds to S2004.

[0181] In S2003, the camera MPU 125, similar to S803 in Figure 8, causes the focus detection auxiliary light source 131 to illuminate the subject with red auxiliary light. Then, the auxiliary light illumination determination process is terminated.

[0182] In S2004, the camera MPU125 determines whether the brightness of the imaging environment is darker than a second predetermined brightness (state B) based on brightness information obtained from the brightness determination image data. The criterion for determining whether it is dark state B is preferably whether autofocus is difficult without illuminating with auxiliary light. The detection accuracy of the phase difference in the y-direction is already determined in the S2002 process, so it is not used as a criterion here. If the camera MPU125 determines it is dark state B, it proceeds to S2005; otherwise, it terminates the auxiliary light illumination determination process.

[0183] In S2005, the camera MPU 125 causes the focus detection auxiliary light source 131 to illuminate the subject with green auxiliary light. Then, the auxiliary light illumination determination process is terminated.

[0184] Thus, in this embodiment as in Embodiment 1, when the brightness of the imaging environment is darker than a predetermined brightness, auxiliary light is shone on the subject to improve AF accuracy. However, the color of the focus detection auxiliary light source is changed depending on whether sufficient accuracy can be obtained for the amount of defocus in the y direction. In dark conditions B where sufficient accuracy cannot be obtained for the phase difference in the y direction (i.e., the amount of defocus), red is used. Supplement Select a secondary light source, and in dark conditions A where sufficient accuracy can be obtained for defocusing in the y-direction, select a green auxiliary light.

[0185] Furthermore, in this embodiment, the process for determining the amount of defocus used in S706 of Figure 7 is performed as shown in the flowchart of Figure 21. The processes S2102 to S2114 in Figure 21 are the same as the processes S902 to S914 in Figure 9. in Therefore, these explanations will be omitted.

[0186] In S2101 of Figure 21, the camera MPU 125 determines whether the red auxiliary light selected in S2003 of Figure 20 is illuminating the subject. If the red auxiliary light is illuminating the subject, the amount of defocus in the x-direction is selected in S2102. If the green auxiliary light is illuminating the subject or if no auxiliary light is illuminating the subject, the process proceeds to S2103. If the green auxiliary light is illuminating the subject, both the x-direction and y-direction defocus amounts are used. The color of the auxiliary light may be selected considering not only the phase difference detection accuracy mentioned above but also other conditions such as the visibility of the subject, but in this embodiment, the phase difference detection accuracy is prioritized.

[0187] In this embodiment, the same color of auxiliary light as the green pixels 212Gr and 212Gb, which are used for detecting phase differences in both the x and y directions, is selected. However, if the detection accuracy of the phase difference in the y direction is insufficient, the color of the auxiliary light may be selected based on conditions other than detection accuracy. When using an image sensor with a pixel array different from the pixel array shown in Figure 2(a), for example, the pixel array shown in Figure 14(a), the color of the auxiliary light may be switched between green and red depending on whether sufficient detection accuracy of the phase difference in the y direction can be obtained. Also, when using an image sensor with the pixel array shown in Figure 14(b), the color of the auxiliary light may be switched between green and blue depending on whether sufficient detection accuracy of the phase difference in the y direction can be obtained. [Examples]

[0188] Next, Embodiment 4 of the present invention will be described. In this embodiment, the same configuration and processing as in Embodiment 1 will be omitted from the description. In this embodiment, the focus detection auxiliary light source 131 can emit not only red (first color) auxiliary light but also blue (second color) auxiliary light.

[0189] The operation of the image sensor phase-detection unit 129 in this embodiment will now be described. Figure 22(a) shows the pixel arrangement on the image sensor 122 in a range of 6 vertical (y-direction) rows and 8 horizontal (x-direction) columns on the image sensor surface as seen from the lens unit 100 side. Similar to Figure 2(a), a Bayer array of color filters is provided on the image sensor surface, with red (R) and green (G) color filters alternately arranged from left to right for pixels in odd-numbered rows, and green (G) and blue (B) color filters alternately arranged from left to right for pixels in even-numbered rows.

[0190] Pixel 211R, shown in Figure 22(b), is provided with a pair of photoelectric conversion units 211A and 211B, which are divided in the x-direction with respect to one on-chip microlens 211i, similar to Figure 2(b). Pixel 211B, shown in Figure 22(c), has a pair of light source conversion units 211C and 211D, which are divided in the y-direction with respect to one on-chip microlens 211i.

[0191] Furthermore, as shown in Figure 22(d), each pixel 211Gr is provided with a single photoelectric conversion unit 211E for each on-chip microlens 211i. The same applies to pixels 211Gb.

[0192] In this embodiment as well, the photoelectric conversion signals output from each of the pairs of photoelectric conversion units in multiple pixels are used to generate a pair of image signals and disparity image data as display / recording image data for 3D image observation. In addition, the imaging signal output by adding the pairs of photoelectric conversion signals from each of the multiple pixels, and the imaging signal output from a pixel having a single photoelectric conversion unit, are used to generate image data for brightness determination and normal display / recording image data. Note that an image sensor with a different pixel arrangement than that in Figure 22(a), such as the pixel arrangements shown in Figures 14(a) to (c), may also be used.

[0193] In this embodiment, the auxiliary light irradiation determination process in S719 of Figure 7 is performed using the flowchart shown in Figure 23. The processes S2301 and S2302 in Figure 23 are the same as the processes S801 and S802 in Figure 8, so their explanation is omitted.

[0194] In step S2303 of Figure 23, the camera MPU 125 determines whether or not a horizontal cage subject has been detected by the subject detection unit 130. If a horizontal cage subject has been detected, the process proceeds to S2304; otherwise, the process proceeds to S2305.

[0195] In S2304, the camera MPU 125 causes the focus detection auxiliary light source 131 to illuminate the subject with red auxiliary light. Then, the auxiliary light illumination determination process is terminated.

[0196] In S2305, the camera MPU 125 determines whether or not a vertical cage subject has been detected by the subject detection unit 130. If it has been detected, the process proceeds to S2306; otherwise, the process proceeds to S2307.

[0197] In S2306, the camera MPU 125 causes the focus detection auxiliary light source 131 to illuminate the subject with blue auxiliary light. Then, the auxiliary light illumination determination process ends.

[0198] In S2307, the camera MPU 125 determines whether the subject detection unit 130 has detected a horizontal subject. If it has detected a subject, it proceeds to S2306; otherwise, it proceeds to S2304.

[0199] In this embodiment, the color of the auxiliary light is switched according to the subject detection result by the subject detection unit 130. As shown in Figure 15(b), when a cage 1505 extending horizontally is detected, the x-direction defocus amount should be used for AF to prevent erroneous AF due to proximity competition between the cage 1505 and the subject 1502 behind it, and to focus on the subject 1502. In this embodiment, the x-direction defocus amount is detected by pixel 211R, so a red auxiliary light is shone on the subject in S2304.

[0200] On the other hand, as shown in Figure 15(a), when a vertically extending cage 1504 is detected, the y-direction defocus amount should be used for AF in order to focus on the subject 1502 behind the cage 1504. Furthermore, as shown in Figure 16(a), when a horizontal line 1602 is detected, it is preferable to use the y-direction defocus amount as the defocus amount in the direction that provides better AF accuracy. In this embodiment, since the y-direction defocus amount is detected by pixel 211B, blue auxiliary light is shone on the subject in S2306.

[0201] If a cage or horizon is not detected, S2304 will illuminate the subject with a red auxiliary light.

[0202] In this embodiment, the process for determining the amount of defocus used in S706 in Figure 7 is performed as shown in the flowchart in Figure 24. S2401 and S2403~S2 in Figure 24 4 Since the process in step 15 is the same as the processes in steps 901 and S902-S914 in Figure 9, their explanations will be omitted.

[0203] In step S2402 of Figure 24, the camera MPU 125 determines whether the blue auxiliary light selected in step S2306 of Figure 23 is illuminating the sensor. If the blue auxiliary light is illuminating the sensor, the process proceeds to step S2406, where it decides to use the y-direction defocus amount detected at pixel 211B. If the red auxiliary light selected in step S2304 of Figure 23 is illuminating the sensor, the process proceeds to step S2403, where it decides to use the x-direction defocus amount detected at pixel 211R.

[0204] In this embodiment, the amount of defocus in the x or y direction is switched depending on the color of the auxiliary light. However, as shown in Figure 16(b), if the imaging screen 1601 is tilted relative to the horizontal line 1602, either the x or y defocus amount can be used. In such cases, instead of using only one of the defocus amounts in the x or y direction, a weighted average of the x and y defocus amounts is used in S2409. In this case, red and blue auxiliary lights may be shone on the subject simultaneously.

[0205] In the embodiment described above, the accuracy of autofocus can be improved by changing the color of the auxiliary light according to the subject detection result. [Examples]

[0206] Next, we will describe Embodiment 5 of the present invention. In this embodiment, we will omit the explanation of the same configuration and processes as in Embodiment 1.

[0207] In this embodiment, the process for determining the amount of defocus used in S706 in Figure 7 is performed as shown in the flowchart in Figure 25. S2 in Figure 25 5 The processing in steps 01 to S2503 is the same as the processing in steps 901 to S903 in Figure 9, and the processing in steps S2505 and S2506 in Figure 25 is the same as the processing in steps S904 and S906 in Figure 9. Also, the processing in steps S2507 to S2511 in Figure 25 is the same as the processing in steps S909 to S913 in Figure 9, and the processing in steps S2512, S2513 and S2514 in Figure 25 is the same as the processing in steps S908, S914 and S905 in Figure 9. For this reason, the explanation of these processes will be omitted.

[0208] If the camera MPU125 detects a horizontal cage subject, a vertical cage subject, or a horizontal line by the subject detection unit 130 in S2503, S2505, or 2506, it performs a process to determine the amount of defocus to use using the xy priority direction map in S2504. The xy priority direction map is information that indicates which of the x-direction and y-direction defocus amounts should be prioritized (in other words, which is usable) in each of the multiple regions within the imaging screen, and details will be described later.

[0209] The flowchart in Figure 26 shows the process for determining the amount of defocus used using an xy-priority direction map.

[0210] In S2601, the camera MPU125 determines whether the reliability of the map area corresponding to the set AF area in the xy priority direction map (hereinafter referred to as the AF map area) is in an usable state. If the reliability is in an usable state, the process proceeds to S2602; otherwise, the process proceeds to S2606.

[0211] In the S2602, the camera MPU125 is AF mat P territory Determine whether the preferred direction of the region is the x-direction. If the preferred direction is the x-direction, proceed to S2603; otherwise, proceed to S2604.

[0212] In S2603, the camera MPU125 decides to use the defocus amount in the x-direction. Then, it terminates the process of determining the amount of defocus to use using the xy-priority direction map.

[0213] In the S2604, the camera MPU125 is AF mat P territory The system determines whether the preferred direction for the region is the y-direction or not. If the preferred direction is the y-direction, the process proceeds to S2605; otherwise, it proceeds to S2606. As will be explained in more detail later, if the reliability of the AF map region is usable and the preferred direction is neither the x-direction nor the y-direction, it means that both the x-direction and the y-direction are reliable.

[0214] In S2605, the camera MPU125 decides to use the defocus amount in the y direction. Then, it terminates the process of determining the amount of defocus to use using the xy-priority direction map.

[0215] In S2606, the camera MPU 125 decides to use a weighted average of the defocus amounts in the x and y directions. The weighting ratio at this time may be 1:1, or it may be determined according to the contrast level of the A to D image signals, etc. After that, the camera MPU 125 finishes the process of determining the amount of defocus to use using the xy-priority direction map.

[0216] As described above, in this embodiment, when a cage or the horizon is detected as a subject, the amount of defocus used for AF is determined using an xy-priority direction map. In Embodiment 1, the subject detection result was output as the output result of the CNN machine learning, but in this embodiment, the xy-priority direction map based on the subject detection result is output as the output result of the CNN machine learning. The xy-priority direction map is trained to show the direction in which the influence of the occlusion is small among the first and second defocus amounts. The xy-priority direction map may also be trained to show a binary or higher confidence level of whether the first and second defocus amounts are usable based on the degree of influence of the occlusion. Alternatively, a predetermined computer such as a server may perform CNN machine learning, and the camera body 120 may acquire the trained CNN from the predetermined computer. For example, the predetermined computer can perform supervised learning using training image data as input and the xy-priority directions corresponding to the training image data as training data to train the CNN of the xy-priority direction map.

[0217] The xy priority direction map includes priority direction information indicating which of the x-direction and y-direction defocus amounts should be prioritized for each of the multiple regions within the imaging screen, or whether both x-direction and y-direction defocus amounts are usable. Furthermore, the xy priority direction map also includes low-reliability information indicating that the reliability of the defocus amount for each divided region is not at a usable level.

[0218] Figure 27(a) shows an imaging scene in which a subject 2702 is captured behind a vertical cage 2704 extending vertically and a horizontal cage 2705 extending horizontally within the imaging screen 2701. An AF area 2703 is set on a portion (face) of the subject 2702. In this imaging scene as well, in order to prevent erroneous AF due to proximity competition between the subject 2702 and the cages 2704 and 2705, it should be possible to select either the defocus amount in the x direction or the y direction according to the direction in which the cages 2704 and 2705 extend.

[0219] Figure 27(b) shows an example in which the imaging screen 2701 is divided into 7x5 regions 2706, centered on the AF region 2703 shown in Figure 27(a). Note that the number of divided regions may be other than 7x5. Figure 27(c) shows the state in which the divided regions 2706 of Figure 27(b) are superimposed on the imaging screen 2701 of Figure 27(a). Figure 27(d) is an example of an xy priority direction map generated for the imaging scene of Figure 27(a), and each divided region contains priority direction information or low reliability information.

[0220] In Figure 27(d), the arrow → indicates priority direction information, showing that the x-direction defocus amount is reliable enough to be used and that the x-direction is the preferred direction. Specifically, as can be seen from Figure 27(c), the priority direction information → is placed in a divided region that contains only the horizontal cage 2705, where near and far competition occurs in the y-direction but not in the x-direction. When an AF area is set in this divided region, it is determined in S2603, via S2602 in Figure 26, that the x-direction defocus amount will be used.

[0221] In Figure 27(d), the arrow (↓) indicates priority direction information, showing that the y-direction defocus amount is reliable enough to be used and that the y-direction is the preferred direction. Specifically, as can be seen from Figure 27(c), the priority direction information (↓) is placed in a divided region that contains only the vertical cage 2704, where near and far competition occurs in the x-direction but not in the y-direction. When an AF area is set in this divided region, it is determined in S2605, via S2603 and S2604 in Figure 26, that the y-direction defocus amount will be used.

[0222] In Figure 27(d), the intersecting arrows → and ↓ represent priority direction information indicating that the defocus amount in either the x or y direction is reliable enough to be used. Specifically, as can be seen in Figure 27(c), this priority direction information is placed in a segmented region that does not include either the vertical or horizontal cages 2704 or 2705. When an AF area is set in this segmented region, it is determined in S2606, via S2603 and S2604 in Figure 26, that the weighted average of the defocus amounts in the x and y directions will be used.

[0223] Thus, in this embodiment, the amount of defocus used for AF is determined according to the priority direction information included in the xy priority direction map. Since the priority direction information for the divided region (AF map region) corresponding to the AF region 2703 shown in Figure 27(a) is ↓, the amount of defocus in the y direction is used.

[0224] In Figure 27(d), the × indicates low-reliability information, meaning that the defocus amounts in both the x and y directions are unreliable and unusable. Specifically, as can be seen from Figure 27(c), the low-reliability information × is placed in the divided region where both vertical and horizontal cages 2704 and 2705 are included and near / far competition occurs in both the x and y directions. When the AF area is set in this divided region, S2601 in Figure 26 determines that the reliability is unusable, and S2606 decides to use the weighted average value of the defocus amounts in the x and y directions. However, because the reliability of the defocus amounts in the x and y directions is low, search drive is performed in S1207 and S1208 of the AF processing in Figure 12.

[0225] Note that performing a search drive may increase the time required for autofocus. Therefore, if the AF area is set in a divided region where low reliability information × is placed in the xy priority direction map, the amount of defocus to use may be determined based on the priority direction information of the divided region that is as close as possible to the divided region and provides a usable level of reliability.

[0226] Furthermore, if multiple AF areas can be set as AF areas, the amount of defocus to use may be selected according to the priority direction information of multiple divided areas in the xy priority direction map that correspond to multiple AF areas. For example, in the cases shown in Figures 27(c) and (d), first, the divided areas with low reliability information × are excluded from the xy priority direction map. Then, among the divided areas with priority direction information (→, ↓, or crossing arrows), the defocus amount of the divided area from which the nearest defocus amount is obtained may be used, or the average value of the defocus amounts obtained from multiple divided areas may be used. Alternatively, the defocus amount of the divided area with a crossing arrow as priority direction information and the least influence of near / far competition may be prioritized. [Examples]

[0227] Next, we will describe Embodiment 6 of the present invention. In this embodiment, the same configurations and processes as in Embodiment 1 will not be described.

[0228] In this embodiment, the process shown in the flowchart of Figure 28 is performed as the process for determining the use of the stabilizing defocus amount in S909 in Figure 9. The processes S2801 to S2803 in Figure 28 are the same as the processes S1001 to S1003 in Figure 10, and the processes S2809 and S2810 in Figure 28 are the same as the processes S1007 and S1008 in Figure 10, so their explanations are omitted.

[0229] In S2804 of Figure 28, the camera MPU 125 sets the AF area to be used for determining whether or not to use the stabilization defocus amount, and proceeds to S2805. In this embodiment, the case in which multiple AF areas are set as the AF area will be described.

[0230] Figure 31 shows an imaging scene in which a person 3102 is captured as the subject within the imaging screen 3101, and multiple AF areas 3103 are set on a part of the person 3102 (face). The multiple AF areas 3103 here are 3x3 areas horizontally and vertically.

[0231] In S2805, the camera MPU125 determines whether or not it has performed the following determination for all AF areas 3103. If it has not yet performed the determination for all AF areas 3103, it proceeds to S2806 and performs the following determination while updating the AF areas that have not yet been determined in S2804 and S2805. On the other hand, if it has already performed the determination for all AF areas 3103, it proceeds to S2810.

[0232] In S2806, the camera MPU125 determines whether the absolute value of the defocus amount in the x direction is less than a predetermined value. If it is less than the predetermined value, the process proceeds to S2807; otherwise, the process proceeds to S2804.

[0233] In S2807, the camera MPU125 determines whether the absolute value of the defocus amount in the y direction is less than a predetermined value. If it is less than the predetermined value, the process proceeds to S2808; otherwise, the process proceeds to S2804.

[0234] In S2808, the camera MPU125 determines whether the difference in the amount of defocus in the x and y directions is less than a predetermined value. If it is less than the predetermined value, the process proceeds to S2809; otherwise, the process proceeds to S2804.

[0235] The processing in S2806, S2807, and S2808 is the same as S1004, S1005, and S1006 in Figure 10, but differs in that if the value is not below a predetermined value, the process returns to S2804. If one AF area is determined to be ineligible by any of the judgments in S2806, S2807, and S2808, instead of immediately disabling the use of the stabilizing defocus amount as in Figure 10, the process returns to S2804 and the same judgment is performed for each of the other target AF areas. If all AF areas are deemed ineligible by any of the judgments in S2806 to S2808, the stabilizing defocus amount is not used in S2810. Also, if any one AF area is deemed ineligible by all of the judgments in S2806 to S2808, the stabilizing defocus amount is used in S2809.

[0236] In this embodiment, the process for determining the amount of defocus used for stabilization, S911 in Figure 9, is performed as shown in the flowchart in Figure 29. The processes S2906 to S2911 in Figure 29 are the same as the processes S1101 to S1105 in Figure 11, and the processes S2914 to S2916 in Figure 29 are the same as the processes S1108 to S1110 in Figure 11. in Therefore, these explanations will be omitted.

[0237] In step S2901 of Figure 29, the camera MPU 125 sets multiple AF areas for which the amount of defocus used is to be determined. Then it proceeds to step S2902.

[0238] In S2902, the camera MPU125 determines whether or not a candidate for the amount of defocus to be used has been determined for all AF areas to be judged. If it has not been determined, the process proceeds to S2903; if it has been determined for all, the process proceeds to S2919. In S2901 and S2902, processing is performed for each AF area while updating the AF area, similar to S2804 and S2805 in Figure 28.

[0239] In S2903, the camera MPU125 determines whether the absolute value of the defocus amount in the x direction is less than a predetermined value. If it is less than the predetermined value, the process proceeds to S2904; otherwise, the process proceeds to S2918.

[0240] In S2904, the camera MPU125 determines whether the absolute value of the defocus amount in the y direction is less than a predetermined value. If it is less than the predetermined value, the process proceeds to S2905; otherwise, the process proceeds to S2918.

[0241] In S2905, the camera MPU125 determines whether the difference in the amount of defocus in the x and y directions is less than a predetermined value. If it is less than the predetermined value, the process proceeds to S2906; otherwise, the process proceeds to S2918.

[0242] The processes of S2903 to S2905 are the same as the processes of S2806 to S2808 in FIG. 28. If any of the determinations in S2903 to S2905 is not applicable, the process of S2918 described later is performed. Also, if the determinations in S2906 and S2914 are not applicable, that is, if the reliabilities of the defocus amounts in the x and y directions are both less than a predetermined level, the process proceeds to S2918.

[0243] In S2912, the camera MPU 125 determines the defocus amount in the x direction as a candidate for use as the stabilization defocus amount, and returns the process to S2901.

[0244] In S2913, the camera MPU 125 determines the defocus amount in the y direction as a candidate for use as the stabilization defocus amount, and returns the process to S2901.

[0245] In S2917, the camera MPU 125 determines the weighted average value of the defocus amounts in the x and y directions as a candidate for use as the stabilization defocus amount, and returns the process to S2901.

[0246] The processes of S2912, S2913, and S2917 are similar to S1106, S1107, and S1111 in FIG. 11. However, in this embodiment, instead of immediately determining the stabilization defocus amount to be used, candidates for use in each AF area are determined, and then the stabilization defocus amount to be finally used in a later process is determined.

[0247] In S2918, the camera MPU 125 excludes the defocus amount of the set AF area from the candidates for use and proceeds with the process to S2901. In this embodiment, when it is not preferable to use the stabilization defocus amount as it is excluded from the candidates for use.

[0248] In S2919, upon determining that the use defocus amount candidates have been determined for all AF regions in S2902, the camera MPU 125 performs a process (multiple AF region stabilization use defocus amount determination process) to finally determine the defocus amount for stabilization to be used. Then, this defocus amount for stabilization use determination process is terminated.

[0249] The flowchart of FIG. 30 shows the multiple AF region stabilization use defocus amount determination process performed by the camera MPU 125 in S2919 of FIG. 29.

[0250] In S3001, the camera MPU 125 determines whether there is one or more defocus amount candidates for use. If there is one or more, the process proceeds to S3002; if there is none, the process proceeds to S3003.

[0251] In S3002, the camera MPU 125 determines the defocus amount with the smallest absolute value among the defocus amount candidates for use as the defocus amount for use. Then, the multiple AF region stabilization use defocus amount determination process is terminated.

[0252] In S3003, the camera MPU 125 determines the defocus amount of the central AF region as the defocus amount for use. Then, the multiple AF region stabilization use defocus amount determination process is terminated.

[0253] In this embodiment, candidate defocus amounts for stabilization are determined in all AF regions, and the defocus amount with the smallest absolute value among them is used as the stabilization defocus amount. This suppresses the variation in the defocus amount in one AF region, as explained in Figure 18, and then further suppresses the variation in the defocus amount by using the defocus amounts obtained from multiple AF regions. As a result, the stability of AF near the focus state can be improved. In this embodiment, if there are no candidate defocus amounts for stabilization in all AF regions, the defocus amount of the central AF region is used as the stabilization defocus amount in S3003, but the average value of the defocus amounts of all AF regions may also be used as the stabilization defocus amount. [Examples]

[0254] Next, Example 7 of the present invention will be described. This example is a modification of Example 6.

[0255] The flowchart in Figure 32 shows the process performed by the camera MPU 125 at S2919 in Figure 29 to determine the amount of defocus used for stabilizing multiple AF areas. The processes at S3201 and S3203 in Figure 32 are the same as those at S3001 and S3003 in Figure 30, so their explanations are omitted.

[0256] In step S3202 of Figure 32, the camera MPU 125, after determining candidates for use as stabilizing defocus amounts for all AF areas in the process shown in Figure 29, calculates an average value of the sum of the defocus amounts of all candidates at a predetermined ratio. Then, it determines this average value as the stabilizing defocus amount to be used and terminates the process.

[0257] In S3202, the ratio for adding the defocus amount may be the same for all AF areas, or the ratio may be changed according to the magnitude of the signal for each AF area.

[0258] In this embodiment, as in Embodiment 6, after suppressing the variation in the amount of defocus in one AF region, the variation in the amount of defocus can be further suppressed by using the defocus amounts obtained from multiple AF regions. As a result, the stability of AF near the in-focus state can be improved. (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0259] The embodiments described above are merely representative examples, and various modifications and changes can be made to each embodiment when implementing the present invention. [Explanation of Symbols]

[0260] 100 Lens Unit 104 Focus Lens 120 Camera body 122 Image sensor 125 Camera MPU 129 Image plane phase-contrast focus detection unit 130 Subject detection unit 131 Focus detection auxiliary light source

Claims

1. An image sensor capable of acquiring a first pair of pupil-divided signals in a first direction and a second pair of pupil-divided signals in a second direction different from the first direction from within the imaging surface that captures a subject image formed by an optical system, An acquisition means for obtaining a first defocus amount from the phase difference of the first pair of signals and a second defocus amount from the phase difference of the second pair of signals, A detection means for detecting a subject within the imaging screen, The system includes a determination means for determining the amount of defocus used for focus control of the optical system from among the first and second defocus amounts, depending on the detected subject. When the detection means detects a subject located behind an object extending in one direction, The imaging apparatus is characterized in that the determination means determines to use the defocus amount obtained from the first pair or second pair of signals, which are divided in the same direction as the one direction within the imaging screen, for the focus control, from among the first and second defocus amounts.

2. The image sensor has a first pair of photoelectric conversion units divided into pupils in a first direction and a second pair of photoelectric conversion units divided into pupils in a second direction different from the first direction, within the imaging surface that captures the subject image formed by the optical system. The imaging apparatus according to claim 1, characterized in that the acquisition means acquires a first defocus amount from the phase difference of a first pair of signals generated using the outputs of the first pair of photoelectric conversion units, and acquires a second defocus amount from the phase difference of a second pair of signals generated using the outputs of the second pair of photoelectric conversion units.

3. The image sensor has pixels in which the photoelectric conversion unit corresponding to one microlens is divided into four pupils, vertically and horizontally. The imaging apparatus according to claim 1, characterized in that the acquisition means acquires the first and second defocus amounts from the phase difference of the first and second pairs of signals generated by adding them in different combinations.

4. The imaging apparatus according to any one of claims 1 to 3, characterized in that the determination means determines the first defocus amount, the second defocus amount, or the average sum of the first and second defocus amounts at a predetermined ratio as the defocus amount to be used for focus control.

5. When the detection means detects a horizontal line as the subject, The imaging apparatus according to any one of claims 1 to 4, characterized in that the determination means determines that at least the defocus amount obtained from the first pair or second pair of signals, which are pupil-divided in a direction different from the direction of the horizontal line in the imaging screen, is used for the focus control.

6. The aforementioned determination means is Depending on the detected subject, information is generated in each of the multiple regions within the imaging screen that indicates at least one of the first and second defocus amounts that can be used for focus control. The imaging apparatus according to any one of claims 1 to 5, characterized in that the focus control is performed using the amount of defocus shown in the information.

7. The decision means generates information indicating which of the first and second defocus amounts should be prioritized in each of the multiple regions within the imaging screen, using a learning model generated based on machine learning. The imaging apparatus according to claim 1, characterized in that the focus control is performed using the amount of defocus shown in the information above.

8. The imaging apparatus according to claim 7, characterized in that the determination means generates the information using machine learning that has learned the direction in which the influence of the obstruction is small among the first and second defocus amounts.

9. The imaging apparatus according to claim 7, characterized in that the determination means divides the imaging screen into the plurality of regions using machine learning that has learned a binary or more confidence level of whether the first and second defocus amounts are usable based on the degree of influence of an obstruction.

10. A control method for an imaging device having an image sensor capable of acquiring a first pair of signals divided into pupils in a first direction and a second pair of signals divided into pupils in a second direction different from the first direction, from within the imaging surface that captures an image of a subject formed by an optical system, An acquisition step of obtaining a first defocus amount from the phase difference of the first pair of signals and a second defocus amount from the phase difference of the second pair of signals, A detection step that detects a subject within the image capture screen, The process includes a determination step of determining the amount of defocus used for focus control of the optical system from among the first and second defocus amounts, depending on the detected subject. In the detection step described above, if a subject located behind an object extending in one direction is detected, The control method is characterized in that, in the determination step, it is determined that the amount of defocus obtained from the first pair or second pair of signals divided in the same direction as the one direction within the imaging screen is used for the focus control, among the first and second defocus amounts.

11. A program characterized by causing the computer of an imaging device to execute processing according to the control method described in claim 10.

Images