DETECTION APPARATUS AND DETECTION METHOD

Abstract

To provide a detection device and a detection method, enabling highly accurate and high-speed focus detection or depth detection.SOLUTION: A detection device includes: an imaging device that comprises a plurality of pixels having a plurality of light receiving units corresponding to a microlens, the plurality of light receiving units being provided according to a plurality of predetermined phase difference detection directions, and that adds the output signals of the plurality of light receiving units according to the phase difference detection direction and can output the added signal; and a control unit that performs focus detection or depth detection by phase difference detection which is based on the added signal of the imaging device. The imaging device makes heights of the potential barrier different between the light receiving units according to the plurality of phase difference detection directions (Ptb and Prl). The control unit instructs the imaging device in a specific phase difference detection direction to execute the storage operation of the imaging device, sets different storage determination levels according to the phase difference detection direction, and controls the storage operation of the imaging device on the basis of the added signal and the storage determination level.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a detection device that has an image sensor equipped with a plurality of pixels corresponding to microlenses, calculates a phase difference of an output signal using the outputs of the plurality of pixels, and can perform focus detection or depth detection based on this phase difference. [Background technology]

[0002] A focus detection device is known in which a plurality of microlenses and a plurality of pixels corresponding to each microlens are arranged on the imaging surface of an image sensor, a plurality of light beams passing through a region obtained by dividing an exit pupil of an imaging optical system are photoelectrically converted by each pixel, a phase difference of the photoelectrically converted output signals is detected, and the defocus amount of the imaging optical system is detected based on the phase difference. It is also known that the output signals of the divided pixels are added together to be used as an image signal for one pixel.

[0003] For example, the photoelectric conversion device shown in Patent Document 1 has a configuration in which a four-part photodiode (PD) is arranged corresponding to one microlens of an imaging element, and detects a phase difference in the left-right (RL) direction using a pixel output obtained by adding up the charges of the top and bottom (TB) PDs of the four-part PD. Also, detects a phase difference in the top-bottom (TB) direction using a pixel output obtained by adding up the charges of the left-right (RL) PDs. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2015-162658 A Summary of the Invention [Problem to be solved by the invention]

[0005] When light is obliquely incident on a pixel of an imaging element, the amount of charge generated by photoelectric conversion differs in each of a pair of pixels, and the charge of some pixels becomes saturated. In this case, the linearity of the image signal synthesized by adding the pair of pixel signals (charges) is lost. In Patent Document 1, when the amount of photoelectrically converted charge becomes saturated in any one of the four-segment PDs, both RL and TB become unable to detect the phase difference. Therefore, in Patent Document 1, the threshold voltage corresponding to the amount of charge (saturation) equivalent to the potential barrier of one PD needs to be set to 1 / 4 of the normal threshold voltage.

[0006] As shown in Patent Document 1, if the potential barrier between PDs is lowered, the combined image signal will not be saturated even if some pixels (PDs) are saturated. However, in Patent Document 1, the threshold voltage must be set to 1 / 4 of the normal threshold voltage, which results in an insufficient dynamic range of the focus detection (AF) signal and reduced detection accuracy.

[0007] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a detection device and a detection method capable of performing focus detection or depth detection with high accuracy and high speed. [Means for solving the problem]

[0008] In order to achieve the above object, a detection device according to a first invention includes an image sensor having a plurality of pixels each having a plurality of light receiving sections corresponding to a microlens, the plurality of light receiving sections being provided corresponding to a plurality of predetermined phase difference detection directions, capable of adding output signals of the plurality of light receiving sections in accordance with the phase difference detection direction to output an added signal, and a control section which performs focus detection or depth detection by phase difference detection based on the added signal of the image sensor, wherein the image sensor varies the height of a potential barrier between the light receiving sections in accordance with the plurality of phase difference detection directions, and the control section instructs the image sensor to a specific phase difference detection direction to cause the image sensor to execute an accumulation operation, sets different accumulation determination levels in accordance with the phase difference detection direction, and controls the accumulation operation of the image sensor based on the added signal and the accumulation determination level. When the specified specific phase difference detection direction is a first phase difference detection direction in which a height of a potential barrier between the light receiving sections is higher among the multiple phase difference detection directions, the image sensor adds together the output signals corresponding to phase difference detection in a second phase difference detection direction perpendicular to the first phase difference detection direction within the multiple pixels, and averages together the added output signals corresponding to a first side of the first phase difference detection direction and the added output signals corresponding to a second side opposite to the first side at a predetermined number of the multiple pixels in the second phase difference detection direction, and generates the added signal without averaging together for the first phase difference detection direction. do.

[0009] The detection device according to a second aspect of the present invention is the detection device according to the first aspect of the present invention, the control unit is capable of executing a live view operation for performing a live view display on a display unit based on the addition signal output by the imaging element, The imaging element is In the live view operation, the output signals corresponding to the phase difference detection in the second phase difference detection direction are added together in the plurality of pixels, and The added output signals corresponding to the same side of the plurality of pixels in both the first phase difference detection direction and the second phase difference detection direction are averaged for a predetermined number of the plurality of pixels to generate the added signal. A third aspect of the present invention relates to the detection device according to the second aspect of the present invention, wherein the control unit includes: In the above live view operation, The sum signal is input from the imaging element, and the sum signals that form a pair in the second phase difference detection direction are added together to generate image data for display. A detection device according to a fourth invention is any of the first to third inventions, wherein the control unit sets the accumulation determination level in accordance with a height of the potential barrier corresponding to the phase difference detection direction. A detection device according to a fifth aspect of the present invention is any one of the first to third aspects of the present invention, wherein the control unit compares the addition signal with the accumulation determination level, and controls the accumulation operation of the imaging element based on the comparison result.

[0010] No. 6 The detection device according to the present invention is the first ~ No. 5In the invention, the imaging element is capable of amplifying and outputting the sum signal, and the control unit sets an amplification factor of the imaging element and sets the accumulation judgment level according to the set amplification factor. No. 7 The detection device according to the present invention is the first ~ No. 6 In the invention, the control unit sets a saturation determination level according to the height of the potential barrier corresponding to the phase difference detection direction, and determines saturation by comparing the sum signal output by the imaging element with the saturation determination level. No. 8 The detection device according to the present invention is 7 In the invention, the control section performs focus detection or depth detection based on the result of the determination.

[0011] No. 9 The imaging device of the present invention has the detection device of the first invention and is capable of continuously capturing still images based on an additive signal obtained by adding together all of the output signals of the multiple light receiving sections of the imaging element, and when, during the continuous capturing of the still images, the control unit causes the imaging element to capture and acquire the additive signal of the multiple light receiving sections according to the phase difference detection direction, it reads out only the additive signal corresponding to the phase difference detection direction in which the potential barrier is set higher as the additive signal of the multiple light receiving sections according to the phase difference detection direction.

[0012] No. 10The detection method according to the present invention is a detection method for a detection device having an image sensor that includes a plurality of pixels having a plurality of light receiving sections corresponding to microlenses, the plurality of light receiving sections being provided corresponding to a plurality of predetermined phase difference detection directions, and that is capable of adding output signals of the plurality of light receiving sections in accordance with the phase difference detection direction to output an addition signal, the image sensor varies the height of a potential barrier between the light receiving sections in accordance with the plurality of phase difference detection directions, instructs the image sensor to a specific phase difference detection direction to execute an accumulation operation of the image sensor, sets an accumulation determination level that differs in accordance with the phase difference detection direction, controls the accumulation operation of the image sensor based on the addition signal and the accumulation determination level, and performs focus detection or depth detection by phase difference detection based on the addition signal of the image sensor. when the specified specific phase difference detection direction is a first phase difference detection direction in which a height of a potential barrier between the light receiving sections is higher among the multiple phase difference detection directions, the image sensor adds, within the multiple pixels, the output signals corresponding to phase difference detection in a second phase difference detection direction perpendicular to the first phase difference detection direction, and averages, at a predetermined number of the multiple pixels in the second phase difference detection direction, the added output signals corresponding to a first side of the first phase difference detection direction and the added output signals corresponding to a second side opposite to the first side, and generates the added signal without averaging for the first phase difference detection direction. . Effect of the Invention

[0013] According to the present invention, it is possible to provide a detection device and a detection method capable of performing focus detection or depth detection with high accuracy and high speed. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing mainly the electrical configuration of an imaging device according to an embodiment of the present invention; [Diagram 2] 1 is a block diagram showing mainly the electrical configuration of an imaging element of an imaging device according to an embodiment of the present invention. [Figure 3A] 1 is a diagram showing a configuration of a pixel of a 4PD pixel type in an imaging device according to an embodiment of the present invention. [Figure 3B] 1 is a cross-sectional view taken along the optical axis direction of a microlens in a pixel portion of a 4PD pixel type in an imaging device according to one embodiment of the present invention. [Figure 4] 11 is a diagram illustrating a potential barrier when RL pixel priority is given in a 4PD pixel type pixel section in an image pickup device according to one embodiment of the present invention. FIG. [Diagram 5] 4A to 4C are diagrams showing sequences during AF and LV in an image pickup apparatus according to an embodiment of the present invention. [Figure 6] FIG. 11 is a diagram illustrating pixel addition during RL readout in an imaging device according to an embodiment of the present invention. [Figure 7] 11 is a diagram illustrating pixel addition during TB readout in an image pickup device according to an embodiment of the present invention. FIG. [Figure 8] 4 is a diagram showing a sequence during continuous shooting in an imaging device according to an embodiment of the present invention. FIG. [Figure 9A] 4 is a flowchart showing the operation of the imaging apparatus according to the embodiment of the present invention. [Figure 9B] 4 is a flowchart showing the operation of the imaging apparatus according to the embodiment of the present invention. [Figure 9C] 4 is a flowchart showing the operation of the imaging apparatus according to the embodiment of the present invention. [Figure 10] FIG. 11 is a block diagram showing a circuit for setting an amplification factor of an image sensor in an image pickup apparatus according to a first modified example of one embodiment of the present invention. [Figure 11] 13 is a graph showing the relationship between the gain of an image sensor and a saturation threshold in an image pickup device according to a first modified example of one embodiment of the present invention. [Figure 12] 13A and 13B are graphs and a table showing the amount of correction for each pixel when a lens is attached in an imaging device according to a second modified example of one embodiment of the present invention. [Figure 13] FIG. 13 is a diagram showing a state in which the L pixel is saturated and charge overflows to the R pixel side in an imaging device according to a third modified example of an embodiment of the present invention. [Figure 14] 1 is a circuit diagram showing a configuration of a 4PD pixel in one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] An imaging device according to an embodiment of the present invention will be described below. This imaging device has an imaging section, which converts a subject image into image data, and based on the converted image data, displays the subject image in live view on a display section disposed on the rear of the main body or the like. The photographer determines the composition and shutter timing by observing the live view display. When the shutter is released, the image data is recorded on a recording medium. When a playback mode is selected, the image data recorded on the recording medium can be played back and displayed on the display section.

[0016] The imaging element 208 of the imaging device according to this embodiment includes a plurality of pixels each having a plurality of light receiving sections corresponding to a microlens, and the plurality of light receiving sections are provided corresponding to a plurality of predetermined phase difference detection directions (see Figs. 3A, 3B, 6, and 7). The imaging element 208 also varies the height of the potential barrier between the light receiving sections depending on the phase difference detection direction (see Fig. 4(a)).

[0017] In this embodiment, specifically, a four-segment photodiode is arranged corresponding to one microlens of the imaging element (see FIG. 3A), and the phase difference is detected in the horizontal direction (left-right direction) with priority over the vertical direction (up-down direction). The potential barrier between the R pixel (hereinafter R pixel) arranged on the right side and the L pixel (hereinafter L pixel) arranged on the left side and the T pixel (hereinafter T pixel) arranged on the upper side and the B pixel (hereinafter B pixel) arranged on the lower side and the B pixel (hereinafter B pixel) arranged on the lower side and the B pixel (hereinafter B pixel) arranged on the lower side and the R pixel is provided with a potential barrier suitable for the RL pixel priority (see FIG. 4(a)). Specifically, the potential barrier between the T pixel arranged on the upper side and the B pixel arranged on the lower side is set lower than the potential barrier between the R pixel and the L pixel (for example, 1 / 2, see FIG. 4(a)). In the case of reading out 2PDs (R / L pixel or T / B pixel) for detecting the horizontal (vertical) phase difference by top-bottom (left-right) addition, a threshold voltage (accumulation judgment level, saturation judgment level, described later) according to the height of the potential barrier of the pixel pair to be read out is set to perform the accumulation operation.

[0018] In the case of 2PD readout for phase difference detection, the threshold voltage for saturation judgment of pixel data when read out as R / L pixels is set to a threshold voltage corresponding to the charge amount of 2PD corresponding to the potential barrier of the R / L pixels (for example, twice the threshold voltage for 1PD (1 / 2 the threshold voltage for 4PD)). This makes it easy to increase the readout speed and increases the dynamic range of the R / L pixel signal, which has high readout efficiency, and enables improved AF accuracy and speed.

[0019] 1 is a block diagram showing an example of the configuration of an image capture device (specifically, for example, a digital camera) 1 including a focus detection device according to an embodiment of the present invention. In FIG. 1, solid lines with arrows indicate the flow of data, and dashed lines with arrows indicate the flow of control signals.

[0020] The imaging device 1 has an interchangeable lens 100 and a camera body 200. The interchangeable lens 100 is configured to be detachable from the camera body 200. When the interchangeable lens 100 is attached to the camera body 200, the interchangeable lens 100 and the camera body 200 are connected so as to be able to communicate with each other. Note that the imaging device 1 does not necessarily have to be a lens-interchangeable imaging device. For example, the imaging device 1 may be an imaging device with an integrated lens. It may also be an imaging device provided in a mobile device such as a smartphone.

[0021] The interchangeable lens 100 includes an imaging optical system 102, a drive unit 104, a lens CPU (Central Processing Unit) 106, and a lens side storage unit 108. Here, each block of the interchangeable lens 100 is configured, for example, by hardware. However, it is not necessarily required to be configured by hardware, and some may be configured by software. Furthermore, each block of the interchangeable lens 100 does not have to be configured by a single piece of hardware or software, and may be configured by a plurality of pieces of hardware or software. Furthermore, when the interchangeable lens and the camera body are configured integrally, the lens CPU 106 and the CPU 212 may be configured by a single CPU.

[0022] The imaging optical system 102 is an optical system for forming an image of a light beam from a subject on an image sensor 208 of the camera body 200. The imaging optical system 102 has a focus lens 102a and an aperture 102b. The focus lens 102a is configured to be able to adjust the focal position of the imaging optical system 102 by moving in the optical axis direction.

[0023] The aperture 102b is disposed on the optical axis of the focus lens 102a. The aperture diameter of the aperture 102b is variable. The aperture 102b adjusts the amount of light from the subject that passes through the focus lens 102a and enters the image sensor 208. The drive unit 104 has a drive motor, a drive circuit, and the like, and drives the focus lens 102a and the aperture 102b based on a control signal output from the lens CPU 106. Here, the imaging optical system 102 may be configured as a zoom lens. In this case, the drive unit 104 may also perform zoom driving, or the user may change the focal length by manual operation. The drive unit 104 functions as an aperture drive unit (actuator, driver) that drives the aperture included in the imaging optical system.

[0024] The lens CPU 106 is a processor including a CPU and its peripheral circuits, and operates according to a program stored in the lens side storage unit 108. The lens CPU 106 is configured to be able to communicate with the CPU 212 of the camera body 200 through an interface (I / F) 110. The lens CPU 106 controls the drive unit 104 according to a control signal from the CPU 212 of the camera body 200. In addition, the lens CPU 106 transmits various information such as the aperture value (F value) of the aperture 102b and lens information stored in the lens side storage unit 108 to the CPU 212 through the I / F 110. The lens CPU 106 functions as a focus lens control unit that controls the position of a focus lens included in the imaging optical system. This focus lens control unit communicates with the control unit and executes position control of the focus lens in synchronization with the first imaging operation and the second imaging operation of the imaging element.

[0025] It should be noted that the lens CPU 106 does not necessarily have to be configured as a CPU. In other words, functions similar to those of the lens CPU 106 may be realized by a processor such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). Furthermore, functions similar to those of the lens CPU 106 may be realized by software.

[0026] The lens side storage unit 108 has an electrically rewritable non-volatile memory, and stores, in addition to the above-mentioned programs, lens information and the like related to the interchangeable lens 100. The lens information includes, for example, information on the focal length and aberration of the imaging optical system 102.

[0027] The camera body 200 has a mechanical shutter 202, a drive unit 204, an operation unit 206, an image sensor 208, an image stabilization circuit 210, a CPU 212, an image processing circuit 214, an image compression / expansion unit 216, a focus detection circuit 218, an exposure control circuit 220, a display unit 222, a bus 224, a dynamic random access memory (DRAM) 226, a body side storage unit 228, and a recording medium 230. Here, each block of the camera body 200 is configured, for example, by hardware. However, it is not necessarily required to be configured by hardware, and some may be configured by software. Also, each block of the camera body 200 may not be configured by a single piece of hardware or software, but may be configured by multiple pieces of hardware or software.

[0028] Mechanical shutter 202 is configured to be able to open and close freely, and adjusts the time that a light beam from a subject is incident on image sensor 208 (exposure time of image sensor 208). As mechanical shutter 202, for example, a focal plane shutter is adopted. In addition to this focal plane shutter, a lens shutter may be provided on the lens barrel side. Driving unit 204 drives mechanical shutter 202 based on a control signal from CPU 212. Driving unit 204 has an actuator that drives mechanical shutter 202, a driving circuit for this actuator, etc., and performs the opening and closing operation of mechanical shutter 202.

[0029] The operation unit 206 is an interface for inputting user instructions to the imaging device 1, and includes various operation buttons such as a power button, a release button, a video button, a mode dial, a playback button, a menu button, etc., as well as various operation members such as a touch panel, etc. This operation unit 206 detects the operation state of the various operation members, and outputs a signal indicating the detection result to the CPU 212.

[0030] The image sensor 208 is disposed on the optical axis of the imaging optical system 102, behind the mechanical shutter 202, and in the vicinity of a position where a light beam from a subject is imaged by the imaging optical system 102. The image sensor 208 captures an image of the subject and generates a pixel signal related to the subject.

[0031] The image sensor 208 has a pixel section 22 (see FIG. 2) in which a plurality of image pixels (image capturing pixels) are arranged two-dimensionally. The image pixels correspond to microlenses L (see FIG. 3) and are configured to be divided into a plurality of focus detection pixels. The focus detection pixels photoelectrically convert each of the light beams passing through a region obtained by dividing the exit pupil of the image capturing lens 2, which is the image capturing optical system, into a plurality of pupils to generate a photoelectric conversion signal. The image sensor 208 is configured, for example, as a single-plate CMOS image sensor equipped with a primary-color Bayer array color filter, but is of course not limited to this configuration.

[0032] The image sensor 208 is an image sensor that includes a plurality of pixels each having a plurality of light receiving sections corresponding to a microlens, the plurality of light receiving sections being provided corresponding to a plurality of predetermined phase difference detection directions, and that can add output signals of the plurality of light receiving sections according to the phase difference detection direction to output a sum signal. The image sensor varies the height of the potential barrier between the light receiving sections according to the plurality of phase difference detection directions (see, for example, FIG. 4). The image sensor can amplify and output the sum signal (see analog processing section 23 in FIG. 2 and FIG. 10). A detailed configuration of the image sensor 208 will be described later with reference to FIG. 2 and FIG. 3.

[0033] The image stabilization circuit 210 moves the image sensor 208 in a direction parallel to its light receiving surface so as to suppress camera shake occurring in the camera body 200. By moving the image sensor 208 so as to cancel out the movement of the camera shake, blurring of the subject image occurring in the image data due to the camera shake is suppressed. The image stabilization circuit may be provided in the interchangeable lens 100. In this case, the image stabilization circuit is configured to move the image stabilization optical system included in the imaging optical system 102.

[0034] The CPU 212 is a processor including a CPU and its peripheral circuits, and performs overall control of the camera body 200 according to a program stored in the body side storage unit 228. The CPU 212 controls, for example, an imaging operation (including an imaging drive mode, a read mode, etc.) by the image sensor 208. The CPU 212 also outputs a control signal for driving the focus lens 102a to the lens CPU 106 according to the focus state of the focus lens 102a detected by the focus detection circuit 218. The CPU 212 also outputs an exposure setting value calculated by the exposure control circuit 220 to the lens CPU 106 and the image sensor 208. Here, the CPU 212 does not necessarily have to be configured as a CPU. That is, a function similar to that of the CPU 212 may be realized by an ASIC, an FPGA, or the like. A function similar to that of the CPU 212 may also be realized by software.

[0035] The image processing circuit 214 performs various types of image processing on the pixel data. For example, when shooting a still image (including continuous shooting), the image processing circuit 214 performs image processing for still image recording to generate still image data. Similarly, when shooting a moving image, the image processing circuit 214 performs image processing for moving image recording to generate moving image data. Furthermore, when displaying a live view, the image processing circuit 214 performs image processing for display to generate display image data.

[0036] The image compression / expansion unit 216 has an image compression circuit and an image expansion circuit. When recording image data, the image compression / expansion unit 216 compresses the image data (still image data or video data) generated by the image processing circuit 214. When reproducing the image data, the image compression / expansion unit 216 expands the image data recorded in a compressed state on the recording medium 230.

[0037] The focus detection circuit 218 performs focus detection of the focus lens 102a by a phase difference method using focus detection pixel data output from focus detection pixels of the image sensor 208. The focus detection circuit 218 can also detect the depth of an object by using the phase difference detection result.

[0038] The focus detection circuit 218 functions as a control unit that performs focus detection or depth detection by phase difference detection based on the addition signal of the image sensor. The function of this control unit is not limited to the focus detection circuit 218, and may be performed by other circuits, processors, etc., such as the CPU 212. The control unit instructs the image sensor to perform a specific phase difference detection direction to execute the accumulation operation of the image sensor (for example, see S5 in FIG. 9A and S23 in FIG. 9B), sets different accumulation determination levels according to the phase difference detection direction (for example, see S9 and S13 in FIG. 9A and S27 and S31 in FIG. 9B), and controls the accumulation operation of the image sensor based on the addition signal and the accumulation determination level.

[0039] The control unit sets an accumulation determination level according to the height of the potential barrier corresponding to the phase difference detection direction (see, for example, S9 and S13 in FIG. 9A and S27 and S31 in FIG. 9B). The control unit compares the addition signal with the accumulation determination level, and controls the accumulation operation of the imaging element based on the comparison result (see, for example, S9 and S13 in FIG. 9A and S27 and S31 in FIG. 9B).

[0040] The control unit also sets the amplification factor of the image sensor, and sets an accumulation judgment level according to the set amplification factor (see, for example, FIG. 11). The control unit sets a saturation judgment level according to the height of a potential barrier corresponding to the phase difference detection direction, and compares the sum signal output by the image sensor with the saturation judgment level to judge saturation (see, for example, FIG. 11). The control unit performs focus detection or depth detection based on the judgment result.

[0041] Furthermore, when the control unit described above acquires the sum signals of the multiple light receiving units according to the phase difference detection direction by capturing images using an imaging element during continuous shooting of still images, the control unit reads out only the sum signals corresponding to the phase difference detection direction in which the potential barrier is set higher as the sum signals of the multiple light receiving units according to the phase difference detection direction (e.g., see FIG. 8).

[0042] The exposure control circuit 220 functions as a photometry unit and calculates an exposure setting value based on pixel data of the image sensor 208. This exposure control circuit 220 measures the subject brightness from the pixel data of the image sensor 208, and calculates an exposure setting value required to make the brightness of the subject at the time of shooting an appropriate value from the measured subject brightness. The exposure setting value includes the opening amount (aperture value) of the diaphragm 102b and the exposure time (shutter speed) of the image sensor 208.

[0043] The display unit 222 has a display such as a liquid crystal display or an organic EL display, and is disposed on the rear surface of the camera body 200, or disposed as an electronic viewfinder. The display unit 222 displays images under the control of the CPU 212. The display unit 222 is used for live view display, playback display of recorded images, and the like.

[0044] The bus 224 is connected to the image sensor 208, the CPU 212, the image processing circuit 214, the image compression / decompression unit 216, the focus detection circuit 218, the exposure control circuit 220, the display unit 222, the DRAM 226, the main body memory unit 228, and the recording medium 230, and operates as a transfer path for transferring various data generated in these blocks.

[0045] The DRAM 226 is an electrically rewritable volatile memory, and temporarily stores various data such as pixel data output from the imaging element 208, still image data, video data, display image data, and data processed by the CPU 212. Note that a Synchronous Dynamic Random Access Memory (SDRAM) may be used for temporary storage.

[0046] The body-side storage unit 228 is an electrically rewritable non-volatile memory. The body-side storage unit 228 stores programs used by the CPU 212 and various data such as adjustment values ​​for the camera body 200. The recording medium 230 is an electrically rewritable non-volatile memory, and is configured to be built into or loaded into the camera body 200. The recording medium 230 records image data for recording as an image file of a predetermined format. Note that the DRAM 226, the body-side storage unit 228, and the recording medium 230 may each be configured as a single memory or the like, or may be configured as a combination of multiple memories or the like.

[0047] Next, the configuration of the image sensor 208 will be described with reference to Fig. 2. The image sensor 208 has image pixels, which are divided into a plurality of focus detection pixels. The image pixels generate image pixel signals and focus detection pixel signals based on photoelectric conversion signals generated by the focus detection pixels through photoelectric conversion of light beams. Note that the image pixels in this embodiment use photodiodes as light receiving sections.

[0048] In the example shown in Figure 2, the imaging element 208 includes a vertical scanning unit 21, a pixel unit 22, an analog processing unit 23, an ADC processing unit 24, a memory unit 25, a horizontal scanning unit 26, an output unit 27, an input unit 28, and an element control unit 29.

[0049] Image pixels and focus detection pixels are arranged in the pixel section 22. Image pixel signals and focus detection pixel signals are generated by photoelectrically converting a subject image, and the generated signals are read out by at least some of the components from the vertical scanning section 21 to the output section 27, the element control section 29, etc.

[0050] The vertical scanning unit 21 has a vertical scanning circuit and performs scanning in the vertical direction by sequentially selecting the horizontal arrangement (rows) of pixels in the pixel unit 22. The vertical scanning unit 21 selects a specific row and resets or transfers each pixel in the selected row, thereby controlling the charge accumulation time (exposure time) of the pixels.

[0051] The analog processing unit 23 has an analog processing circuit, and is a circuit that performs analog signal processing on the analog pixel signals read out from the pixel unit 22. The analog processing unit 23 includes, for example, a preamplifier that amplifies the pixel signals, and a correlated double sampling (CDS) circuit that reduces reset noise from the pixel signals.

[0052] The analog-to-digital conversion processing section (ADC processing section) 24 has an AD conversion circuit and converts the analog pixel signals output from the analog processing section 23 into digital pixel data. The ADC processing section 24 employs a configuration in which pixel signals read out from the pixel section 22 are AD-converted by an analog-to-digital converter (ADC) for each column, such as a column ADC.

[0053] The memory unit 25 has a memory and is configured with an electrically rewritable volatile memory circuit or the like that temporarily holds the pixel data converted by the ADC processing unit 24. The horizontal scanning unit 26 has a horizontal scanning circuit and reads out the pixel data (image pixel data and focus detection pixel data) from the memory unit 25 in column order.

[0054] The output unit 27 has an output circuit, arranges the pixel signals read by the horizontal scanning unit 26 to generate a pixel signal string, converts the signal into an output signal format such as a serial signal or a differential signal, and outputs the signal. Note that the output unit 27 or the ADC processing unit 24 also functions as a sensitizing unit that performs sensitizing processing (signal amplification processing according to the set ISO sensitivity).

[0055] The input unit 28 has an input circuit, and receives a synchronous signal, a reference clock, operation setting information, and the like relating to the control of the image sensor 208 from the CPU 212 and an image sensor driving unit (not shown).

[0056] The element control unit 29 has an imaging control circuit, controls each block in the imaging element 208 in accordance with a synchronization signal and a reference clock received via the input unit 28, and includes a readout method selection unit 30. The element control unit 29 also receives operation setting instructions, such as an instruction to switch the imaging drive mode, from the CPU 212 via the input unit 28, and controls each block in the imaging element 208.

[0057] The readout method selection unit 30 has a selection circuit, and selects and sets a readout method from the image sensor 208 based on information on the operation setting received through the input unit 28 (for example, a camera mode such as still image shooting, video shooting, live view, or AF). The readout method may include all or some of a 1PD simple readout method, a 2PD addition readout method, a 4PD addition readout method without phase difference information that reads out all the added values ​​of the focus detection pixels, etc. Furthermore, the AF readout (AF(rl)) described later is a 2PD addition readout method that generates and reads out both of a pair of focus detection pixel signals (R / L pixel signals) in the first pupil division direction based on a photoelectric conversion signal. The live view (LV) readout (LV+AF(tb)) is a 2PD addition readout method that generates and reads out both of a pair of focus detection pixel signals (T / B pixel signals) in the second pupil division direction based on a photoelectric conversion signal. This exposure and readout uses a 4PD addition readout method in which an image pixel signal is generated by adding up all the photoelectric conversion signals generated within one image pixel, and then only the generated image pixel signal is read out.

[0058] 3A and 3B, a description will be given of the structures of focus detection pixels and image pixels arranged in the pixel section 22. As described above, the pixel section 22 is a pixel array section in which image pixels and focus detection pixels are arranged two-dimensionally (vertically (column direction) and horizontally (row direction)).

[0059] FIG. 3A shows an example of a 4PD pixel, which has a pixel structure in which four photodiodes PD are arranged on one microlens Le. In the 4PD pixel shown in FIG. 3A, one color filter F and four photodiodes PDa, PDb, PDc, and PDd are arranged on one microlens Le. As shown in FIG. 3B, each pixel is configured such that a microlens Le, a color filter F, and photodiodes PDa to PDd are arranged in the stacking direction from the object side to the image side. The microlens Le collects light to increase the amount of light that reaches the 4PD pixel as an image pixel, thereby substantially increasing the aperture ratio of the image pixel. For example, in the case of a primary color Bayer array color filter, the color filter F is a red (R) filter, a green (G) filter, or a blue (B) filter arranged according to the pixel position.

[0060] In the 4PD pixel shown in Figures 3A and 3B, photodiodes PDa to PDd are arranged in the pupil division direction in the imaging range of one microlens Le, and the four photodiodes PDa to PDd are divided into four parts, top to bottom, left to right, so that phase differences in the horizontal and vertical directions can be detected. The four photodiodes PD are arranged at the upper left, lower left, upper right, and lower right positions, respectively. That is, one pixel has four photodiodes PDa, PDb, PDc, and PDd, and the pupil division directions are the horizontal direction and the vertical direction.

[0061] When the output of the photodiode PD is added vertically by 2 PDs, that is, when (PDa+PDb) and (PDc+PDd) in FIG. 3A are generated, a focus detection pixel signal for detecting a phase difference in the horizontal direction (vertical line detection) is obtained. As shown in the upper right side of FIG. 3A, a 2PD addition value L on the left side and a 2PD addition value R on the right side are obtained. When the output of the photodiode PD is added horizontally by 2 PDs, that is, when (PDa+PDc) and (PDb+PDd) are generated, a focus detection pixel signal T for detecting a phase difference in the vertical direction (horizontal line detection) is obtained. As shown in the lower right side of FIG. 3A, an upper 2PD addition value T and a lower 2PD addition value B are obtained. When the output of the photodiode PD is added by 4 PDs, that is, when (PDa+PDb+PDc+PDd) is generated, a pixel signal obtained by adding 4 PDs is an image pixel signal.

[0062] The configuration of a 4PD pixel and PD addition will be described using the circuit diagram shown in FIG. 14. Switching transistors Tr1 to Tr4 are connected to the four photodiodes PDa to PDd, respectively. When control signals TX1 to TX4 are applied from the vertical scanning unit 21 to the switching transistors Tr1 to Tr4, respectively, Tr1 to Tr4 are selectively turned on. When the switching transistors Tr1 to Tr4 are turned on, the photodiodes PDa to PDd are connected to the floating diffusion FD, and the signal charge of the photodiode PD corresponding to the turned-on transistor Tr is transferred to the floating diffusion FD. For example, by turning on Tr1 and Tr2, the signal charge obtained by adding up the signal charges of PDa and PDb can be transferred to the FD.

[0063] One end of the switching transistor Tr5 is connected to the connection point between the switching transistors Tr1 to Tr4 and the floating diffusion FD, and the other end is connected to the power supply voltage VDD. When a reset signal RES is applied to Tr5, the power supply voltage VDD and FD are connected to reset the FD. When the switching transistors Tr1 to Tr4 are turned on and the switching transistor Tr5 is turned on, the photodiodes PDa to PDd are reset. The floating diffusion FD is connected to the output OUT via the switching transistor Tr6 and an amplifying transistor Tr7 connected to the power supply voltage VDD. When a selection signal SEL is applied to the switching transistor Tr6, the voltage value of the floating diffusion FD is amplified by the transistor Tr7 and output to the output OUT, and this output voltage is input to the analog processing unit 23.

[0064] 4, a first boundary B2 exists between the pair of photodiodes PDa, PDb and the pair of photodiodes PDc, PDd, and a second boundary B1 exists between the pair of photodiodes PDa, PDc and the pair of photodiodes PDb, PDd. The potential barrier at the second boundary B1 is set to be higher than the potential barrier at the first boundary B2.

[0065] Next, the setting of the potential barrier of the 4PD pixels will be described with reference to Fig. 4. The height of the potential barrier between the light receiving sections (PD) is made different according to a plurality of phase difference detection directions, but since the readout efficiency is higher when the R / L pixels are given priority, the R / L pixels are used preferentially in this embodiment.

[0066] FIG. 4(a) shows the height of the potential barrier (hereinafter simply referred to as the barrier) between the pixel and the light receiving section PD in the case of RL pixel priority. In FIG. 4(a), LT and LB respectively indicate the upper left light receiving section LT (PDa) and the lower right light receiving section LB (PDb), and RT and RB respectively indicate the upper right light receiving section RT (PDc) and the lower right light receiving section RB (PDd). As described with reference to FIG. 3A, when the phase difference detection direction is the arrow A (also called the RL direction), the L pixel signal can be generated by adding the charges of LT and LB, and the R pixel signal can be generated by adding the charges of RT and RB. In the case of R / L pixel priority, the level of the barrier Ptb (corresponding to the boundary B2) located between LT and LB and between RT and RB is set to be lower than the level of the barrier Prl (corresponding to the boundary B1) located between LT and RT and between LB and RB.

[0067] In this way, since the level of the barrier Ptb is set to be lower than the level of the barrier Prl, for example, there may be cases where the amount of light incident on the LT is greater than that on the other light receiving parts, a large amount of charge is generated, and the accumulated charge overflows beyond the barrier Ptb. In this case, the overflowing charge flows into the LB. In this way, the case where the charge overflows beyond the barrier is referred to as charge saturation. As long as the charge generated in the L pixel (LT+LB) does not exceed the barrier Prl, the charge will not flow out to the R pixel (RT+RB). For this reason, when detecting a phase difference in the phase difference direction A, even if charge saturation occurs in relation to the barrier Ptb in any one of the light receiving parts LT, LB, RT, and RB, charge saturation does not occur in the R pixel (RT+RB) and the L pixel (LT+LB). Therefore, the pixel values ​​(pixel output voltages) of the R pixel and the L pixel are not saturated, and the phase difference in the phase difference direction A can be detected. Next, the effect of giving priority to the barriers will be described in comparison with the case where priority is not given to the barriers.

[0068] As shown in FIG. 4(b), in the case where the barriers are not prioritized, that is, the levels of the barrier Pa are equal in the RL and TB directions, if any one of the light receiving parts LT, LB, RT, and RB is saturated and exceeds the barrier Pa, causing the charge to flow out, the balance of the charge amount is lost in both the phase difference in the RL direction and the phase difference in the TB direction, making it impossible to detect the phase difference. In order to avoid such saturation of the charge, it is necessary to monitor the charge accumulation amount and adjust the accumulation time. In the image sensor 208, the charge accumulated in the PD is transferred to a capacitance (floating diffusion) and converted into a voltage, which is read out as a signal voltage. This capacitance is set to a capacitance value optimized as an image pixel signal, which is a pixel signal of 4PD addition. If the signal voltage corresponding to the saturation of the charge of the pixel signal of 4PD addition is 1, the signal voltage corresponding to the saturation of the charge of the light receiving parts LT, LB, RT, and RB is 1 / 4.

[0069] For this reason, the threshold voltage level of the pixel signal of 1PD must be set to a threshold voltage level equivalent to the charge amount (charge amount equivalent to charge saturation) equivalent to the level of the barrier Pa of 1PD, that is, 1 / 4 of the threshold voltage level equivalent to the charge saturation of the pixel signal of 4PD addition, and the accumulation time must be controlled so as not to exceed this threshold voltage level. In this way, if the level of the barrier Pa is the same and no priority is given, the threshold voltage level must be set to a charge saturation judgment level that is 1 / 4 times the threshold voltage level of the pixel signal of 4PD addition. The same is true for the pixel signal of 2PD addition, and in order to judge the charge saturation of 1PD, the threshold voltage level must be set to a charge saturation judgment level that is 1 / 4 times the threshold voltage level of the pixel signal of 4PD addition. Therefore, if no priority is given to the barrier, the dynamic range of the signal voltage of the phase difference detection pixels (R / L pixels, T / B pixels) is reduced to 1 / 4 of the dynamic range of the signal voltage of the image pixels (4PD addition), which causes a problem of reduced accuracy in phase difference detection.

[0070] On the other hand, when the barrier is given priority as in the above-mentioned RL pixel priority, when the R / L pixels are read out, even if the charge amount of LT (or LB) of one PD among 4PD exceeds the barrier Ptb, the signal voltage of each of the R / L pixels will not be saturated unless the charge amount of 2PD (LT, LB) exceeds the barrier Prl. If the level of the barrier Prl between the R / L pixels is the same as the level of the barrier Pa when the priority is not given, the charge amount of 2PD (LT, LB, or RT, RB) corresponding to the level of the barrier Pa of the R / L pixel is 1 / 2 of the charge amount of the 4PD addition corresponding to the barrier Pa level. Therefore, for the signal voltage of the R / L pixel, the charge saturation judgment level can be set to 1 / 2 of the charge saturation judgment level of the signal voltage of the 4PD addition. This makes it possible to set the dynamic range of the signal voltage of the phase difference detection pixel (R / L pixel) in the RL direction to 1 / 2 of the dynamic range of the signal voltage of the image pixel (4PD addition). The dynamic range can be expanded by two times compared to the case where the priority is not given. The potential barrier can be set by adjusting the impurity concentration of the semiconductor in the manufacturing process of the image sensor, and a detailed description thereof will be omitted.

[0071] Next, a method of setting the saturation determination level of pixel data will be described based on the barrier level setting described with reference to FIG. 4(a). When pixel data of R / L pixels is read out with RL pixel priority, the saturation determination level is set to a voltage level equivalent to the level of the barrier Prl between the R pixel and the L pixel, as described above. In general, the level of the barrier Prl is set to about 70% of the saturation level P of the charge of the 4PD sum pixel in order to ensure the linearity of the 4PD sum pixel. If the saturation level of the signal voltage when reading out the pixel signal of the 4PD sum pixel is, for example, 4096 (corresponding to 12-bit quantization), the saturation level of the signal voltage of the R / L pixel is equivalent to about 4096×70% / 2=1433 of the signal voltage corresponding to the charge amount of 2PD. Therefore, the saturation determination level of the signal voltage (pixel data) of the R / L pixel may be set to, for example, about 1300 (see S9 in FIG. 9A and S27 in FIG. 9B).

[0072] When reading out pixel data of the T / B pixel, the saturation determination level is set to a voltage level equivalent to the level of the barrier Ptb between the T pixel and the B pixel. The level of the barrier Ptb is set to about 70% of the level of the barrier Prl. When the saturation level of the signal voltage when reading out a 4PD addition pixel is set to 4096, the saturation level of the signal voltage of the T / B pixel is equivalent to about 4096×70%×70% / 4=501, which is a signal voltage equivalent to the charge amount of 2PD. Therefore, the saturation determination level of the signal voltage (pixel data) of the T / B pixel may be set to about 450, for example (see S13 in FIG. 9A and S31 in FIG. 9B).

[0073] On the other hand, if RL pixel priority is not given, charge saturation in any one of the four PDs will cause the balance to be lost, affecting the correlation calculation. In this case, under the same conditions, the saturation level of the signal voltage of the R / L pixel will be approximately 4096 x 70% / 4 = 716. Therefore, by giving priority to the RL pixel, the signal voltage (pixel data) of the R / L pixel can secure a dynamic range of approximately twice as much (1433 / 716), which is a great effect.

[0074] In a frame in which phase difference detection is performed as shown in Figure 8 described later, when the R / L pixel signal is exposed / read out, a saturation determination level for the R / L pixels is set to determine saturation, and when the T / B pixel signal is exposed / read out, a saturation determination level for the T / B pixels is set to determine saturation.

[0075] FIG. 5 is a timing chart for explaining the exposure and readout process for AF&LV (live view) described later (see, for example, S5 in FIG. 9A and S23 in FIG. 9B). "VD" indicates the timing of the synchronization signal (vertical synchronization signal) input to the element control unit 29. VD(1) is a vertical synchronization signal indicating the timing of AF operation including AF imaging and readout, and VD(2) is a vertical synchronization signal indicating the timing of LV operation including LV imaging and readout. VD(1) and VD(2) are input to the element control unit 29 once per frame, and the exposure and readout process for AF&LV is performed for each frame. The length of one frame is determined according to the update interval of the LV display screen.

[0076] In FIG. 5, the AF operation indicated by AF(rl) is an operation for generating and reading out focus detection pixel signals in the first pupil division direction. The first pupil division direction is, for example, the horizontal direction (RL direction) of the pixel unit 22. Prior to the AF operation, the element control unit 29 switches the setting of the pixel unit 22 so that the pixel unit 22 outputs R / L pixel signals, which are left aperture pixel signal l (Rl, Grl, Gbl, Bl) and right aperture pixel signal r (Rr, Grr, Gbr, Br). In order to shorten the readout time of the pixel signals, the element control unit 29 sets pixel addition to add (average) multiple R / L pixel signals of the same color and the same aperture output by the pixel unit 22. This pixel addition (average) is performed by the analog processing unit 23.

[0077] Next, pixel addition (average) of R / L pixels for detecting horizontal phase differences read out from the pixel section 22 during AF(rl) operation will be described with reference to Fig. 6. The upper diagram in Fig. 6 shows an image of the arrangement corresponding to each pixel. Rl located at (m1, n1) means the red pixel (R) of the left opening, and Rr located at (m2, n1) means the red pixel (R) of the right opening. Gbl located at (m1, n2) means the green pixel (Gb) of the left opening, and Gbr located at (m2, n2) means the green pixel (Gb) of the right opening. "B" means a blue pixel.

[0078] The pixel addition of the R / L pixels is set in the analog processing unit 23 so that addition is not performed in the horizontal direction, which is the first pupil division direction, but is performed only in the vertical direction, which is the second pupil division direction. In the horizontal direction, pixel signals of the same opening (left openings or right openings) are set to be added at 1 / 1 pixel (no addition), and in the vertical direction, pixel signals of the same opening (left openings or right openings) are set to be added at 5 / 9 pixel. The 5 / 9 pixel addition means adding 5 pixels of Rl corresponding to n1 to n9 and adding 5 pixels of Gbl corresponding to n2 to n10 out of 9 pixels in the vertical direction (in the example of m1 in FIG. 6, 9 pixels of Rl corresponding to n1 to n17 and 9 pixels of Gbl corresponding to n2 to n18). The number of pixel signals to be added is appropriately set according to, for example, the frame rate. This setting compresses the number of rows of pixel signals, reducing the number of rows of pixel signals, and shortens the readout time of pixel signals. On the other hand, since the number of columns of pixel signals is not compressed, the detection accuracy of the phase difference in the horizontal direction is ensured.

[0079] The CPU 212 outputs a control signal to the image sensor 208 (element control unit 29) to set the pixel addition mode as shown in Fig. 6, and then outputs a control signal to the image sensor 208 (element control unit 29) to perform imaging with an exposure time required to generate a focus detection pixel signal. This exposure time is set based on the subject brightness, the saturation state of pixel data, etc.

[0080] Upon receiving a control signal from the CPU 212, the element control unit 29 starts imaging (charge accumulation) for each row of the pixel unit 22 and controls the vertical scanning unit 21 to sequentially output RL pixel signals from the pixel units 22 in the rows for which imaging has been completed.

[0081] To increase speed and reduce power consumption, the detection of the phase difference in the horizontal direction is performed using the pair of Gr left aperture pixel signal Grl and right aperture pixel signal Grr, and the pair of Gb left aperture pixel signal Gbl and right aperture pixel signal Gbr as R / L pixel signals. The pair of R left aperture pixel signal Rl and right aperture pixel signal Rr, and the pair of B left aperture pixel signal Bl and right aperture pixel signal Br, do not necessarily need to be read out. During reading, only the pair of Grl and Grr and the pair of Gbl and Gbr may be read out in a packed manner, as shown in the lower part of Figure 6.

[0082] The lower diagram of Fig. 6 is an image diagram of data arrangement of pixel data A / D converted after pixel addition in the memory unit 25. In the m1 column in the lower part of Fig. 6, the sum value Gr_L (in Fig. 6, only "Gr" is written due to space constraints) of the Grl pixels n1, n3, n5, n7, and n9 rows on the left opening side of the m3 column in the upper diagram of Fig. 6 is arranged, and in the m2 column, the sum value Gr_R (in Fig. 6, only "Gr" is written due to space constraints) of the Grr pixels n1, n3, n5, n7, and n9 on the right opening side of the m4 column in the upper diagram of Fig. 6 is arranged. Similarly, in column m1, the sum Gb_L (in FIG. 6, due to space limitations, only "Gb" is written) of Gbl pixels n2, n4, n6, n8, and n10 rows on the left opening side of column m1 in the upper diagram of Fig. 6 is arranged, and in column m2, the sum Gb_R (in FIG. 6, due to space limitations, only "Gb" is written) of Gbr pixels n2, n4, n6, n8, and n10 rows on the right opening side of column m2 in the upper diagram of Fig. 6 is arranged. As shown in the lower diagram of Fig. 6, the other sum pixel data are arranged in a similar manner.

[0083] In this way, the sum values ​​Gr_L, Gr_R, Gb_L, and Gb_R of the Grl and Grr pixels or the Gbl and Gbr pixels of five pixels per column are stored as an R / L pixel signal in the memory unit 25 in the format shown in the lower diagram of Fig. 6. The image sensor 208 outputs the sum values ​​Gr_L and Gr_R, and Gb_L and Gb_R as an R / L pixel signal, and the focus detection circuit 218 uses this pixel data to perform correlation calculations, etc., which will be described later. Pairs of Rl and Rr and pairs of Bl and Br may also be read out.

[0084] Here, the relevant processing within the image sensor 208 will be described. The R / L pixel signals read out from the pixel unit 22 are subjected to analog processing such as CDS, gain processing, pixel addition processing, etc. in the analog processing unit 23. The analog-processed R / L pixel signals after pixel addition are converted into R / L pixel data after pixel addition, which is a digital signal, in the ADC processing unit 24 and stored in the memory unit 25. In response to a control signal from the element control unit 29, the horizontal scanning unit 26 transfers the R / L pixel data after pixel addition stored in the memory unit 25 (the lower diagram in FIG. 6) to the output unit 27 in column order.

[0085] The output unit 27 arranges the pixel data transferred by the horizontal scanning unit 26 to generate a pixel data string, and converts the generated pixel data string into a predetermined output signal format such as a serial signal or a differential signal, and outputs it. This pixel data string is stored in the DRAM 226 as post-pixel-addition R / L pixel data (focus detection pixel data rl) in the order of the array image shown in the lower diagram of Fig. 6. Note that, although it has been described that pixel addition is performed in the analog processing unit 23, pixel addition may be performed in the ADC processing unit 24 or memory unit 25 without being performed in the analog processing unit 23.

[0086] Next, the AF data processing related to the AF(rl) operation will be described. The pixel data string (R / L pixel data after pixel addition) stored in the DRAM 226 by the read operation in the AF(rl) operation is used for correlation calculation to calculate the amount of focus deviation. The focus detection circuit 218 performs correlation calculation using the pair of Grl and Grr and the pair of Gbl and Gbr, which are the R / L pixel data after pixel addition stored in the DRAM 226 (focus detection pixel data rl shown in the lower diagram of FIG. 6).

[0087] Next, the live view (LV) operation of FIG. 5 described above will be described. The LV operation indicated by LV+AF(tb) is an operation mainly involving the generation and readout (LV) of display pixel signals, but the generation and readout (AF(tb)) of focus detection pixel signals in the second pupil division direction are also performed in the LV operation. Prior to the LV operation, in accordance with an instruction from the CPU 212, the element control unit 29 switches the setting of the pixel unit 22 so that the upper opening pixel signal t (Rt, Grt, Gbt, Bt) and the lower opening pixel signal b (Rb, Grb, Gbb, Bb) which are T / B pixel signals are output from the pixel unit 22. In addition, the element control unit 29 sets pixel addition of the T / B pixel signals output from the pixel unit 22 in order to shorten the readout time of the pixel signals.

[0088] Next, pixel addition (average) in LV operation will be explained using Fig. 7. In LV operation, settings are made so that pixel addition is performed in both the horizontal direction, which is the first pupil division direction, and the vertical direction, which is the second pupil division direction. Fig. 7 is an image diagram of an arrangement similar to that in Fig. 6, but for convenience of drawing, the T pixel signals and B pixel signals are shown arranged left and right instead of top and bottom.

[0089] In the upper diagram of Figure 7, the pixel signals (T pixel signals or B pixel signals) of three identical openings (top openings or bottom openings) that are aligned horizontally are added together, and the pixel signals (T pixel signals or B pixel signals) of two identical openings (top openings or bottom openings) that are aligned vertically are added together.

[0090] Vertical 2 / 3 pixel addition adds two pixels out of three pixels of the same color in the vertical direction, and in the m1 column of Fig. 7, two pixels n1 and n5 out of three pixels of Rt corresponding to n1 to n5 are added, and two pixels n4 and n8 out of three pixels of Gbt corresponding to n4 to n8 are added. Horizontal 3 / 3 pixel addition adds three pixels of the same color that are adjacent in the horizontal direction, and in the n1 row of Fig. 7, three pixels m1, m5, and m7 adjacent to Rt (m2, m6, and m8 adjacent to Rb) are added. Also, three pixels m7, m11, and m15 adjacent to Grt (m8, m12, and m16 adjacent to Grb) are added.

[0091] For detecting phase differences in the vertical direction, the number of pixels added in the vertical direction is set to 2 in order to ensure detection resolution in the vertical direction. It is possible to further improve the detection resolution by eliminating pixel addition in the vertical direction, but this would cause a problem of an increase in the number of readout rows and a decrease in readout speed (low readout efficiency), so 2 / 3 pixel addition is set and the number of rows read out is reduced to 1 / 3 to shorten the readout time. In Figure 7, the second row and the third and fourth columns are shown as not being added, but this is simply due to the remainder that occurs when the number of rows and columns of the pixel section 22 are not a multiple of three.

[0092] The CPU 212 transmits a control signal to the image sensor 208 (element control unit 29) to set the pixel addition mode as shown in Fig. 7, and then outputs a control signal to the image sensor 208 (element control unit 29) to perform imaging with an exposure time required to generate a focus detection pixel signal. This exposure time is set based on the subject brightness, the saturation state of pixel data, and the like.

[0093] Here, a process related to the LV operation in the image sensor 208 will be described. The element control unit 29 receives a control signal input from the CPU 212 to start imaging (charge accumulation) for each row of the pixel unit 22, controls the vertical scanning unit 21, and causes the pixel unit 22 to sequentially output T / B pixel signals for rows for which imaging has been completed. The T / B pixel signals read out from the pixel unit 22 are analog-processed in the analog processing unit 23 and converted into pixel-added T / B pixel data, which is a digital signal, in the ADC processing unit 24, and stored in the memory circuit unit 25 in an arrangement as shown in the lower diagram of FIG. 7.

[0094] In the LV operation, R, Gr, Gb, and B pixel data are required to generate display image data, and the upper opening pixel data (T pixel data) Rt, Grt, Gbt, and Bt and the lower opening pixel data (B pixel data) Rb, Grb, Gbb, and Bb are stored in the memory circuit 25. The horizontal scanning unit 26 receives a control signal from the element control unit 29 and transfers the pixel-added T / B pixel data stored in the memory unit 25 to the output unit 27 in column order. The output unit 27 arranges the pixel-added T / B pixel data transferred by the horizontal scanning unit 26 to generate a pixel data string, and converts the generated pixel data string into a predetermined output signal format such as a serial signal or a differential signal and outputs it.

[0095] This pixel data string output from the image sensor 208 is stored in the DRAM 226 as T / B pixel data (focus detection pixel data tb) after pixel addition in a format as shown in the lower diagram of Fig. 7. Note that, although pixel addition has been described in Fig. 7 as being performed by the analog processing unit 23, it may be performed in the ADC processing unit 24 or memory unit 25 instead of by the analog processing unit 23.

[0096] Next, data processing of AF(tb) related to LV operation will be described. The pixel data string (T / B pixel data after pixel addition) stored in the DRAM 226 by the read operation by the LV operation is used for correlation calculation for calculating the amount of focus deviation. The focus detection circuit 218 performs correlation calculation using the pair of Grt and Grb and the pair of Gbt and Gbb, which are the T / B pixel data after pixel addition (focus detection pixel data tb shown in the lower diagram of FIG. 7) stored in the DRAM 226. Correlation calculation using the pair of Rt and Rb and the pair of Bt and Bb is also performed as necessary. In addition, LV data processing related to LV operation will be described. The pixel data string (T / B pixel data after pixel addition) stored in the DRAM 226 by the read operation by the LV operation is added by the image processing circuit 214 to the T pixel data and B pixel data of the same color and position to generate LV display image data having R, Gr, Gb, and B pixel data. The CPU 212 performs LV display on the display unit 222 using the LV display image data.

[0097] Next, the synchronous control of the main exposure (still image frame), AF exposure (phase difference frame), and lens in the high-speed continuous shooting mode will be described with reference to Fig. 8. Fig. 8 shows the operation executed during continuous shooting of still images by electronic shutter in continuous shooting mode and continuous AF (CAF) mode. For example, the user operates the operation unit 206 to set the above mode. In addition, the continuous shooting operation is performed with the mechanical shutter 202 in a fully open state when the second release switch is turned on by the user. The imaging device according to this embodiment is capable of continuous shooting of still images based on an added signal obtained by adding all output signals of a plurality of light receiving units of the imaging element.

[0098] 8, VD indicates the timing of the vertical synchronization signal for the imaging operation, and T1 to T13 on the horizontal axis indicate time. The first (first still image frame) main exposure starts at time T1, exposure of the first line ends at time T2, and then the main exposure for the second frame starts at time T4, the main exposure for the third frame starts at time T7, and the main exposure for the fourth frame starts at time T10.

[0099] In each main exposure, still image data consisting of only image pixel signals is obtained by reading out a 4PD sum value of pixel signals of photodiodes PDa to PDd of 4PD pixels (see FIG. 3A) from the image sensor 208. This image data is processed and recorded as image data for a still image, and is also processed as image data for live view display to perform live view display.

[0100] Between the main exposures, pixel data of focus detection pixels for phase difference detection is read out. At time T3, the exposure for the first (first frame) phase difference detection starts, and at time T4, the exposure for the first line ends. Thereafter, at time T6, the exposure for the second frame phase difference detection starts, and at time T9, the exposure for the third frame phase difference detection starts. In each exposure for phase difference detection, the sum values ​​PDa+PDb, PDc+PDd (for horizontal phase difference), which are the R / L pixel signals of the pixel signals of the photodiode divided into 4 PDs (FIG. 3A), and PDa+PDc, PDb+PDd (for vertical phase difference), which are the T / B pixel signals of the pixel signals of the photodiode, are read out as image data for phase difference detection.

[0101] In this embodiment, a potential barrier that prioritizes RL pixels is set in the image sensor 208, and a process of reading out the R / L pixel signals with a larger dynamic range is performed with priority. That is, in the exposure for phase difference detection, only the R / L pixel signals are read out and focus detection calculation is performed. In addition, as described below, exposure control (accumulation time control) is performed based on the R / L pixel signals to avoid data saturation. The AF(rl) operation shown in FIG. 5 may be executed as the exposure / readout operation for the R / L pixel signals. By using only the R / L pixel signals with a wider dynamic range, it is possible to shorten the time for the exposure / readout operation for phase difference detection and to improve the accuracy of the focus detection operation, enabling faster continuous shooting operations.

[0102] In addition, in each phase difference detection exposure, both exposure and readout of pixel signals for horizontal phase difference and vertical phase difference may be performed continuously. In addition, both the above-mentioned AF(rl) operation as exposure and readout of pixel signals for horizontal phase difference and the above-mentioned LV+AF(tb) operation as exposure and readout of pixel signals for vertical phase difference may be performed continuously. In addition, exposure and readout may be performed by alternately switching pixel signals for horizontal phase difference and vertical phase difference, such that phase difference detection exposure between time T3 and time T4 is exposure and readout of pixel signals for horizontal phase difference, and phase difference detection exposure between time T6 and time T7 is exposure and readout of pixel signals for vertical phase difference. In addition, exposure and readout for horizontal phase difference and vertical phase difference may be set for each pixel, and exposure and readout of both pixel signals may be performed simultaneously in one frame. In this way, image data for phase difference detection is acquired.

[0103] The camera body 200 performs communication by outputting a lens synchronization signal synchronized with the imaging synchronization signal VD to the interchangeable lens 100. The lens synchronization signal may have the same period as the imaging synchronization signal VD, or may have a period twice as long as the imaging synchronization signal VD, as shown in FIG. 8. In this example, VD is output at times T1, T2, T4, T5, T7, T8, T10, T11, and T13. On the other hand, the lens synchronization signal is output at times t1, t4, t7, t10, and t13, and has a period twice as long as VD. The lens synchronization signal may have a period other than the same period as VD or twice as long as VD. The phase of the lens synchronization signal may be completely the same as the phase of the imaging synchronization signal VD, as shown in FIG. 8, or may be shifted from the phase of the imaging synchronization signal VD.

[0104] Camera body 200 transmits the focus lens driving amount (direction) calculated based on the result of phase difference detection to lens 100 in synchronization with the lens synchronization signal. For example, camera body 200 (CPU 212) calculates the focus lens driving amount (direction) based on the result of phase difference detection of T3-T4 of VD, and transmits the focus lens driving amount (direction) after transmitting lens synchronization signal t4 to lens 100. Lens CPU 106 drives the focus lens based on the received focus lens driving amount (direction).

[0105] Next, the operation of the imaging device including the focus detection device will be described with reference to the flowcharts shown in Figures 9A to 9C. This flow shows the operation when the AF mode of the imaging device 1 is set to the continuous AF mode, which is a continuous shooting mode different from that shown in Figure 8 and is set to a low-speed continuous shooting mode in which still image shooting (main exposure) is performed by a mechanical shutter. The continuous AF mode is an AF mode suitable for a moving subject, and is an AF mode in which focusing is continued to follow the subject. For example, the user sets the above mode by operating the operation unit 206. This flowchart is realized by the CPU 212 in the camera body 200 controlling each part in the camera body 200 and the interchangeable lens 100 according to a program stored in the body side storage unit 228.

[0106] When it is detected that the user has turned on the power of the imaging device 1, the camera power-on flow shown in Fig. 9A starts. When the power-on operation is detected, first, it is determined whether or not the first release switch is on (S1). Here, the CPU 212 determines whether or not the first release switch of the release button in the operation unit 206 is on. When the user focuses on the subject and determines the exposure, he or she presses the release button halfway. When the user presses the release button halfway, the first release switch is turned on in response to this operation.

[0107] If the result of the determination in step S1 is that the first release switch is not on, a live view (LV) exposure readout is performed (S3). Here, the CPU 212 outputs a control signal to the drive unit 204 to fully open the mechanical shutter 202, and outputs a control signal to the lens CPU 106 to drive the aperture 102b by a predetermined amount (for example, the open aperture). After that, the CPU 212 outputs a control signal to the image sensor 208 at predetermined time intervals (times determined by the display frame rate) to perform imaging for LV display by the image sensor 208. Each time imaging for LV display is completed, the element control unit 29 reads out pixel signals from the pixel unit 22. Note that when reading out the pixel signals, the element control unit 29 adds pixel signals of the same aperture (same color) output from the pixel unit 22 and outputs the sum. The pixel data output from the image sensor 208 is stored in the DRAM 226 for display.

[0108] When the display pixel data is stored in step S3, next, the CPU 212 performs live view (LV) display and causes the image processing circuit 214 to generate display image data. The image processing circuit 214 performs necessary processing on the pixel data read out from the image sensor 208 to generate display image data for display. The display image data is obtained by averaging the pixel data of the photodiodes PDa to PDd belonging to the same pixel. This averaging process is not limited to being performed by the image processing circuit 214, but may be performed inside the image sensor 208. The CPU 212 causes the display unit 222 to display an LV image based on the display image data generated by the image processing circuit 214, and performs the LV display, and then returns to S1.

[0109] If the result of the determination in step S1 is that the first release switch is on, then AF&LV exposure & readout is performed (S5). The CPU 212 performs imaging and readout for autofocus (AF) and LV display described in FIG. 5. Imaging and readout for AF is performed by AF operation AF(rl), and imaging and readout for LV display is performed by LV operation (LV+AF(tb)). The focus detection pixel data rl and tb calculated in step S5 are stored in the DRAM 226 and used for correlation calculation by the phase difference method. In addition, the image processing circuit 214 adds the data that forms a pair of the focus detection pixel data tb stored in the DRAM 226 to generate display pixel data, and stores this generated display pixel data in the DRAM 226. The CPU 212 performs live view display based on the display pixel data stored in the DRAM 226.

[0110] Next, correlation calculation and reliability determination are performed (S7). Here, the CPU 212 causes the focus detection circuit 218 to perform focus detection calculation. The focus detection circuit 218 performs correlation calculation using paired focus detection pixel data rl, tb stored in the DRAM 226. The correlation calculation generates a two-image interval value corresponding to a phase difference for each AF area based on focus detection pixel data in a plurality of predetermined AF areas (focus detection regions). After the correlation calculation is completed, the focus detection circuit 218 then performs a reliability determination of the focus detection. The reliability determination is based on a reliability evaluation value corresponding to the contrast of the subject image obtained from the pixel data and a plurality of correlation values ​​calculated as a result of the correlation calculation.

[0111] After the reliability determination, next, saturation detection of the R / L pixels is performed (S9). It is detected whether the R / L pixel data read out as focus detection pixel data in the AF exposure readout (AF(rl)) is saturated or not. The pixel data is compared with a predetermined threshold TH (1300 as an example of a saturation determination level) set for the R / L pixels, and if it is greater than the predetermined threshold (corresponding to a large signal amount), it is determined to be saturated.

[0112] If the result of the determination in step S9 is that the R / L pixels are saturated, the exposure correction amount Tv_RL+ is set to 1 (S11). The set exposure correction amount Tv_RL+ is used to correct the exposure time in the next AF&LV exposure & readout in step S5 (FIG. 5: AF(rl)). The next exposure time corresponding to the R / L pixels is set to an exposure time increased by 1 Ev (1 / 2 the exposure time), and exposure is controlled so as not to saturate. 1 Ev is one example, and the change amount may be 0.5 Ev or 2 Ev. In addition, the positions of the R / L pixels detected as being saturated are stored in the DRAM 226. This position information of the saturated pixels is used to determine which of the multiple focus detection areas the saturated pixels are included in (area selection in step S17). In this way, the exposure time (accumulation time) is set so that the R / L pixel data is not saturated, and the saturation determination level is also called the accumulation determination level because it is used as a target accumulation level.

[0113] When the exposure compensation amount is set in step S11, or when the result of the determination in step S9 is that saturation of the R / L pixels is not detected, saturation detection of the T / B pixels is performed (S13). In the LV operation (LV+AF(tb)), it is detected whether the T / B pixel data read as focus detection pixel data is saturated or not. The pixel data is compared with a predetermined threshold TH (450 as an example of the saturation determination level) set for the T / B pixels, and if it is greater than the predetermined threshold (corresponding to a large signal amount), it is determined that the pixel data is saturated.

[0114] If the result of the determination in step S13 is that the TB pixel is saturated, the exposure correction amount Tv_TB+ is set to 1 (S15). The set exposure correction amount Tv_TB+ is used to correct the exposure time in the next AF&LV exposure & readout in step S5 (FIG. 5: LV+AF(tb)). That is, the exposure time corresponding to the next T / B pixel is set to an exposure time (1 / 2 the exposure time) increased by 1 Ev, and exposure is controlled so as not to saturate. 1 Ev is one example, and the change amount may be 0.5 Ev or 2 Ev. In addition, the position of the T / B pixel detected as being saturated is stored in the DRAM 226. This position information of the saturated pixel is used to determine which of the multiple focus detection areas the saturated pixel is included in (area selection in step S17). In this way, the exposure time (accumulation time) is controlled so that the T / B pixel data is not saturated, and the saturation determination level is also called the accumulation determination level because it is used as a target accumulation level.

[0115] When the exposure correction amount is set in step S15, or when the result of the determination in step S13 is that saturation of the T / B pixels is not detected, next, area selection is performed (S17). Here, the focus detection circuit 218 calculates the defocus amount of the focus lens 102a using the two-image interval value of the focus detection area determined to have high reliability by the reliability determination, and selects the focus detection area (and the corresponding defocus amount) based on this defocus amount. For example, a focus detection area showing a defocus amount corresponding to the closest object distance (closest) is selected. The area selection process is not limited to the closest object, and may be an area where a human face is present by face detection, or an area manually selected by the user. Note that the area selection may be performed by the CPU 212. In the area selection, a focus detection area that includes a predetermined number or more of either R / L pixels or T / B pixels detected as saturated is invalidated and excluded from the area selection. This is because focus detection calculation including pixels detected as saturated reduces detection accuracy. The predetermined number is appropriately changed according to the set conditions (for example, the number of pixels constituting the focus detection area, etc.).

[0116] After the area selection, the CPU 212 next judges whether the focus lens 102a is in focus (S19). Here, the CPU 212 judges whether the focus lens 102a is in focus. That is, the CPU 212 judges whether the defocus amount in the focus detection area selected in the area selection process is within a predetermined allowable range, and judges the focus state to be in focus if it is within the allowable range.

[0117] If the result of the determination in step S19 is that the focus lens is not in focus, focus lens driving is executed (S21). Here, the CPU 212 communicates with the lens CPU 106 and outputs a control signal so that the focus lens 102a is driven to the focus lens position calculated for the focus detection area selected in step S17. Upon receiving this control signal, the lens CPU 106 drives the focus lens 102a to the instructed position via the drive unit 104. After driving the focus lens 102a, the process returns to S1.

[0118] If the result of the determination in step S19 is that the focus is achieved, exposure and readout for AF & LV is performed (S23). Here, the CPU 212 performs imaging (exposure) and readout for autofocus (AF) and live view (LV) display, similar to step S5. Pixel signals are read out from the imaging element 208, and focus detection pixel data for AF is stored in the DRAM 226, and display pixel data for LV is stored in the DRAM 226. Also, live view (LV) display is performed using the display pixel data for LV.

[0119] Next, correlation calculation and reliability determination are performed (S25). Here, the CPU 212 causes the focus detection circuit 218 to perform focus detection calculation using the pixel data read out in step S23, similar to step S7. The focus detection circuit 218 performs correlation calculation using paired focus detection pixel data stored in the DRAM 226. After the correlation calculation, the focus detection circuit 218 performs focus detection reliability determination.

[0120] After the reliability determination, it is next detected whether the R / L pixel data is saturated (S27), and if the result of this determination is that the data is saturated, the exposure compensation amount Tv_RL+ is set to 1 (S29). After the exposure compensation amount is set, or if the result of the determination in step S27 is that the data is not saturated, it is next determined whether the T / B pixels are saturated (S31), and if the result of this determination is that the data is saturated, the exposure compensation amount Tv_TB+ is set to 1 (S33). The processes in these steps S27 to S33 are similar to the processes in steps S9 to S15 described above, and therefore detailed explanations will be omitted.

[0121] Once the exposure compensation amount is set in step S33, or if the result of the determination in step S31 is not saturated, the amount of focus deviation is then detected (S35) and an area is selected (S37). In steps S35 and S37, similar to step S17, the focus detection circuit 218 calculates the amount of focus deviation (defocus amount) of the focus lens 102a using the two-image interval value of the focus detection area determined to have high reliability in the reliability determination in step S25, and selects the focus detection area (and the corresponding defocus amount) based on this amount of focus deviation. An example of the selection of the focus detection area (area) is similar to step S17, so a detailed description will be omitted.

[0122] Once the area is selected, history information is then saved (S39). Here, the focus detection circuit 218 saves information related to focus detection as history information in, for example, the DRAM 226. The information related to focus detection includes, for example, information on the amount of focus deviation calculated in step S17, the time when pixel data was acquired, and information on the selected focus detection area. Note that the CPU 212 may save the information related to focus detection in the DRAM 226 as the history information.

[0123] After the history information is saved, it is next determined whether or not the second release switch is on (S41). Here, CPU 212 determines whether or not the second release switch in operation unit 206 is on. When a user takes a still image, he or she presses the release button all the way down. When the user presses the release button all the way down, the second release switch is turned on in response to this operation.

[0124] If the result of the determination in step S41 is that the second release switch is not on, it is determined whether or not the camera is in focus (S43). Here, the same process as in step S19 is performed, and if the result of the determination is that the camera is in focus, the process returns to S23.

[0125] On the other hand, if the result of the determination in step S43 is that the focus state is not achieved, the focus lens is driven (S45). Here, the CPU 212 moves the focus lens 102a to an appropriate focus lens position based on the amount of focus deviation, similar to step S21. After driving the focus lens, the process returns to step S23.

[0126] Returning to step S41, if the result of the determination in this step is that the second release switch is on, a moving object prediction calculation is performed (S47). Here, the CPU 212 causes the focus detection circuit 218 to execute the moving object prediction calculation. The moving object prediction calculation predicts the position of the focus lens 102a that will be in focus during the current exposure of the still image, based on the results of the past focus deviation amount calculations (focus lens positions) stored in step S39 and the history of detection times.

[0127] After the moving object prediction calculation is performed, the shutter operation is then started (S49). Here, the CPU 212 starts the operation of the mechanical shutter 202 to perform imaging (main exposure) for acquiring a still image. This operation of the mechanical shutter 202 includes opening and closing operations of the mechanical shutter 202 before and after the main exposure, and a full-open operation of the mechanical shutter 202 to start imaging for live view and AF after the main exposure (only the timing of the start of the shutter operation is described in the flow of FIG. 9C). When the CPU 212 starts the shutter operation, first, it switches the control signal of the drive unit 204 so as to bring the mechanical shutter 202 into a fully closed state. Then, in step S53, the mechanical shutter 202 is fully opened during the main exposure, and then fully closed once. Thereafter, the CPU 212 controls the drive unit 204 so as to bring the mechanical shutter 202 into a fully open state.

[0128] When the shutter operation starts, the aperture narrowing drive and the focus lens drive (LD) start simultaneously (S51). Here, the CPU 212 instructs the lens CPU 106 to drive the focus lens 102a and the aperture 102b simultaneously, and starts the operation of both. The drive position of the focus lens 102a is the position predicted in the moving object prediction calculation in step S45. The opening amount of the aperture 102b is the opening amount according to the aperture value calculated based on the subject brightness measured by the immediately preceding photometry calculation.

[0129] After the aperture and focus lens are driven, the main exposure is performed (S53). Here, the CPU 212 performs the main exposure, and controls the drive unit 204 so that the mechanical shutter 202 is exposed for only a predetermined exposure period, as described above. The main exposure is imaging for acquiring image data for still image recording, and in the exposure readout operation, a pixel value is generated and output by adding output values ​​of four photodiodes for each pixel unit (4PD addition). After the main exposure is completed, the CPU 212 executes an operation to read pixel signals from the image sensor 208, and after the pixel signals are read out, the image processing circuit 214 executes processing for generating still image data for recording. The image processing circuit 214 executes processing required for generating image data for recording to generate still image data for recording. After the image processing is completed, the CPU 212 compresses the still image data for recording by the image compression and expansion unit 216. After the compression is completed, the CPU 212 records the compressed still image data for recording as an image file on the recording medium 230.

[0130] Next, the aperture LD simultaneous drive (open) is executed (S55). Here, the CPU 212 instructs the lens CPU 106 to open the aperture 102b and drive the focus lens 102a to the target position. In step S51, the focus lens 102a is driven to the position predicted by the moving object prediction calculation in step S47. However, since the processing time until the main exposure is limited, it may not reach the target position and there may be a remainder in the drive. In step S55, an instruction is given to drive such a remaining amount of drive of the focus lens.

[0131] Next, it is determined whether the first release switch is on (S57). Here, CPU 212 determines whether the first release switch of the release button in operation unit 206 is on. If the result of this determination is that the first release switch is on, the process returns to step S23 and the above-mentioned process is executed.

[0132] On the other hand, if the result of the determination in step S57 is that the first release switch is not on, it is determined whether or not the camera power is off (S59). Here, CPU 212 determines whether or not to turn off the power to camera body 200. For example, if a user operates operation unit 206 to instruct turning off the power, or if the user does not operate operation unit 206 for a predetermined period of time, it is determined that the power is to be turned off. If the result of this determination is that the power to camera body 200 is not to be turned off, the process returns to step S1. On the other hand, if it is determined in step S59 that the power to camera body 200 is to be turned off, the process of this flow ends.

[0133] As described above, in one embodiment of the present invention, the level of the barrier Ptb located between LT and LB and between RT and RB of the 4PD is set lower than the level of the barrier Prl located between LT and RT and between LB and RB, giving RL priority. As a result, even if charge saturation occurs in the barrier Ptb in any one of the light receiving parts LT, LB, RT, and RB, charge saturation does not occur in the R pixel (RT+RB) and the L pixel (LT+LB). Therefore, compared to the case where the levels of the barriers Prl and Ptb are the same, it is possible to make the accumulation judgment level (saturation judgment level) of the signal voltage of the R / L pixel higher, and the dynamic range can be expanded. Note that TB priority may be used instead of RL priority. The level of the barrier Ptb located between LT and LB and between RT and RB of the 4PD may be set higher than the level of the barrier Prl located between LT and RT and between LB and RB, giving TB priority. In addition, a configuration may be adopted in which either an RL priority or a TB priority barrier is set for each 4PD pixel, and when an R / L pixel signal is used, an RL priority 4PD pixel is selected and used, and when a T / B pixel signal is used, a TB priority 4PD pixel is selected and used.

[0134] Next, a modified example of an embodiment will be described with reference to Figs. 10 and 11. Figs. 10 and 11 are block diagrams including an image sensor and a focus detection circuit. A method for setting a saturation determination level of digital pixel data (hereinafter, pixel data) after A / D conversion of a pixel signal in a case where the image sensor 208 can amplify and output the pixel signal will be described. In this example, a control unit (focus detection circuit 218) sets an amplification factor (also called gain) of the image sensor 208, and sets a saturation determination level of the pixel data after A / D conversion according to the set amplification factor. Note that the control unit is not limited to being within the focus detection circuit 218, and some of its functions may be realized in other circuits such as the CPU 212.

[0135] 10, the image sensor 208 is the same as the image sensor 208 shown in Fig. 2, and pixel signals read out from the pixel unit 22 are amplified in the analog processing unit 23, A / D converted in the ADC processing unit 24, and then output as pixel data. Note that the memory unit 25, horizontal scanning unit 26, and output unit 27 in Fig. 2 are omitted in Fig. 10.

[0136] The control unit (focus detection circuit 218) includes a phase difference pixel row generation unit 218a, a phase difference pixel data processing unit 218b, and a correlation calculation unit 218c. The phase difference pixel row generation unit 218a has a phase difference pixel row generation circuit, and is a block that outputs the addition signal output from the image sensor 208 as a pixel row in AF area (focus detection area) units. When there are 100 AF areas, for example, the output is performed for 100 pixel rows.

[0137] The phase difference pixel data processing unit 218b has a phase difference pixel data processing circuit, and performs pre-processing (various corrections such as illuminance correction, which will be described later) for correlation calculation. A saturation detection unit 218ba is provided in the phase difference pixel data processing unit 218b. This saturation detection unit 218ba has a saturation detection circuit, and when pixel data exceeds a saturation determination level for each AF area, it determines that saturation is NG and outputs the result. The saturation determination level will be described later with reference to FIG. 11. The correlation calculation unit 218c has a correlation calculation circuit, and performs correlation calculation using the output of the phase difference pixel data processing unit 218b.

[0138] When the gain of the image sensor 208 is changed, the saturation level of the pixel signal also changes, and it is necessary to change the saturation determination level of the pixel data, which is a digital output, accordingly. In the image sensor 1, the output value of the ADC processing unit 24 corresponding to the saturation determination level of the pixel signal when the gain of the image sensor 208 is 1x is stored as the saturation determination level of the pixel data. When the gain is increased, the focus detection circuit 218 increases the saturation determination level by the gain magnification. When the value of this saturation determination level exceeds the dynamic range of the digital data in the focus detection circuit 218, the saturation determination level is set to the maximum value of that dynamic range.

[0139] FIG. 11(b) shows the saturation judgment level versus the gain in the image sensor 208, and FIG. 11(a) shows the change in the RL saturation judgment level versus the analog gain in the image sensor 208 in a graph. As shown in FIG. 11(b), the saturation judgment level of the R / L pixel (see S27 in FIG. 9B) and the saturation judgment level of the T / B pixel (see S31 in FIG. 9B) are different (see FIG. 4(a)), and the R / L saturation judgment level and the T / B saturation judgment level are stored according to the gain. The focus detection circuit 218 (saturation detection unit 218ba) changes the saturation judgment level in steps S27 and S31 in FIG. 9B as the gain increases according to FIG. 11(b). The saturation detection unit 218ba sets the saturation judgment level according to the conditions of the pixel data for judging whether or not it is saturated, that is, whether it is R / L pixel data or T / B pixel data, and the gain. Then, when the pixel data exceeds the saturation judgment level, it is judged that the pixel data is saturated. In FIG. 11, the digital data in the focus detection circuit 218 is handled as 12 bits (4096 LSB), and the upper limit of the dynamic range is set to 3800 LSB, for example.

[0140] Next, as a second modified example, saturation detection of pixel data to be subjected to illuminance correction when performing illuminance correction to correct light quantity distribution characteristics caused by aberration of the optical system will be described with reference to Fig. 12. In this example, R / L pixel data output by the image sensor 208 is read, and illuminance correction processing is performed on this R / L pixel data to generate corrected data. The pixel data before this correction is compared with a saturation judgment level to judge whether the pre-correction data is saturated or not.

[0141] The light quantity distribution characteristic is corrected using an illuminance correction value (FIG. 12) that can be derived from the light receiving sensitivity characteristics of the R / L pixels (RT+RB, LT+LB) of the image sensor 208 and the optical characteristics of the photographing lens. The corrected pixel data PIaft can be calculated by the following formula (1) using the uncorrected pixel data PIpre and the correction value C. The phase difference pixel data processing unit 218b executes the calculation of the following formula (1). Note that i is the pixel number, and the correction value is usually about 1.0 to 5.0. PIaft(i)=PIpre(i)×C(i) ···(1)

[0142] Fig. 12 shows the correction value (illuminance correction value) C of the R / L pixel when a certain photographing lens is attached to the camera body. In Fig. 12, the horizontal axis indicates the pixel position on the image sensor 208, and the vertical axis indicates the correction value C. The correction values ​​of the pixel of the L aperture (L pixel (LT+LB)) and the pixel of the R aperture (R pixel (RT+RB)) are symmetrical with respect to the center of the image sensor 208 due to the characteristics of the optical system. The correction value C is obtained in advance by measurement or calculation and stored in memory. Illuminance correction can be performed by correcting each pixel value using the above formula (1).

[0143] The focus detection circuit 218 (saturation detection unit 218ba) determines that the pixel data PIpre before the illuminance correction is saturated when it exceeds the RL saturation determination level (FIG. 11) (FIG. 9A: S9, FIG. 9B: S27). As described above, performing correlation calculation including saturated pixel data may result in erroneous detection or a decrease in detection accuracy, so that an AF area (focus detection area) including a predetermined number or more of saturated pixel data is made undetectable. Alternatively, correlation calculation may be performed by performing a process such as replacing the saturated pixel data with other pixel data (for example, peripheral pixel data). The above description is about the R / L pixel data, but the same is true for the T / B pixel data (RT+LT / RB+LB), and saturation is determined using the TB saturation determination level (FIG. 11(b)) before performing illuminance correction using the illuminance correction value for the T / B pixel stored in the memory. In this way, by performing saturation detection on pixel data before the illuminance correction, only appropriate pixel data is used in the focus detection calculation, enabling high-precision focus detection.

[0144] Next, a third modified example will be described. In this embodiment, the case where R pixel data and L pixel data are read out respectively has been described. As another method for reading out pixel data, there is a method for reading out one line of R pixel (RT+RB) signals, and then reading out one line of (R+L) pixels (4PD addition) of the same pixel line. This reading method can obtain pixel data of L pixels by calculating the difference {(R+L)-R} between two pixel data, that is, pixel data (R) of only R pixels, and pixel data (R+L) corresponding to a signal obtained by adding together the R pixel signal and the L pixel signal. In this case, it is determined whether the {(R+L)-R} pixel data exceeds the saturation determination level.

[0145] FIG. 13 is an image diagram of the charges of the L / R pixels. For example, as shown in FIG. 13, in a state where the charge CL of the L pixel is saturated and exceeds the barrier, causing the charge CR to leak into the R pixel, the charge Cs overflowing from the L pixel is accumulated in the R pixel in addition to the charge CR photoelectrically converted by the R pixel. In this case, the charge of the R pixel is not saturated, so it is not saturated by the saturation determination of the R pixel data. However, it is determined to be saturated by the saturation determination of the L pixel data obtained by calculating the difference between the (R+L) pixel data and the R pixel data {(R+L)-R}. Therefore, the charge of the L pixel has become saturated, and it can be determined that saturation is NG. In this case, it can also be correctly determined that the correlation calculation result is inaccurate.

[0146] As described above, the detection device in one embodiment of the present invention includes an image sensor (see, for example, FIG. 3A and FIG. 3B) that includes a plurality of pixels each having a plurality of light receiving sections corresponding to a plurality of predetermined phase difference detection directions, and that can add output signals of the plurality of light receiving sections according to the phase difference detection direction to output an added signal, and a control section (see, for example, the focus detection circuit 218, CPU 212, and S35 in FIG. 9B) that performs focus detection or depth detection by phase difference detection based on the added signal of the image sensor. In addition, the image sensor has different heights of potential barriers between the light receiving sections according to the plurality of phase difference detection directions (see, for example, Ptb and Prl in FIG. 4(a)). The control section instructs the image sensor to perform a specific phase difference detection direction to execute the accumulation operation of the image sensor, sets different accumulation determination levels according to the phase difference detection direction, and controls the accumulation operation of the image sensor based on the added signal and the accumulation determination level (see, for example, S9 and S13 in FIG. 9A, and S27 and S31 in FIG. 9B). This makes it possible to increase the dynamic range of the signals from the phase difference detection pixels, enabling improved AF accuracy and speed.

[0147] In one embodiment of the present invention, the focus detection circuit 218, the image processing circuit 214, the image compression / expansion unit 216, the exposure control circuit 220, and the like may be integrated in whole or in part with the CPU 212 and its peripheral circuits. The focus detection circuit 218, the image processing circuit 214, the image compression / expansion unit 216, the exposure control circuit 220, and the like may be hardware configurations such as gate circuits generated based on a program language written in Verilog, or may utilize hardware configurations using software such as a DSP (Digital Signal Processor), or may be circuit parts in a processor configured with an integrated circuit such as an FPGA (Field Programmable Gate Array). Of course, these may be appropriately combined. Alternatively, a processor configured with one or more CPUs may execute the functions of each part by reading and executing a computer program recorded on a recording medium.

[0148] In addition, in one embodiment of the present invention, a digital camera has been used as the device for photographing, but the camera may be a digital single-lens reflex camera, a mirrorless camera, a compact digital camera, a video camera such as a video camera or a movie camera, or a camera built into a mobile phone, a smartphone, a personal digital assistant, a personal computer (PC), a tablet computer, a game device, etc., a medical camera (e.g., a medical endoscope), a camera for scientific equipment such as a microscope, an industrial endoscope, a camera mounted on an automobile, or a surveillance camera. In any case, the present invention can be applied to any device for photographing that employs phase difference AF on the imaging surface.

[0149] In addition, among the technologies described in this specification, the controls mainly described in the flowcharts can often be set by a program, and may be stored in a recording medium or a recording unit. The method of recording on this recording medium or recording unit may be recording at the time of product shipment, using a distributed recording medium, or downloading via the Internet.

[0150] In addition, in one embodiment of the present invention, the operation of this embodiment is explained using a flowchart, but the order of the processing procedure may be changed, any step may be omitted, steps may be added, and the specific processing content within each step may be changed.

[0151] Furthermore, even if the operational flow in the claims, specification, and drawings is described using words expressing an order such as "first" and "next" for convenience, this does not mean that it is necessary to carry out the operations in this order unless specifically explained.

[0152] The present invention is not limited to the above-described embodiment, and can be embodied by modifying the components without departing from the gist of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining the multiple components disclosed in the above-described embodiment. For example, some of the components shown in the embodiment may be deleted. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0153] 1 imaging device, 21 vertical scanning section, 22 pixel section, 23 analog processing section, 24 ADC processing section, 25 memory section, 26 horizontal scanning section, 27 output section, 28 input section, 29 element control section, 30 readout method selection section, 100 interchangeable lens, 102 imaging optical system, 102a focus lens, 102b aperture, 104 drive section, 106 lens CPU, 108 lens side memory section, 200 camera body, 202 mechanism a camera body, 204....drive section, 206...operation section, 208...imaging element, 210...image stabilization circuit, 212...CPU, 214...image processing circuit, 216...image compression / expansion section, 218...focus detection circuit, 218a...phase difference pixel row generation section, 218b...phase difference pixel data processing section, 218ba...saturation detection section, 218c...correlation calculation section, 220...exposure control circuit, 222...display section, 224...bus, 226...DRAM, 228...main body side storage section, 230...recording medium

Claims

1. an image sensor including a plurality of pixels each having a plurality of light receiving sections corresponding to a microlens, the plurality of light receiving sections being provided corresponding to a plurality of predetermined phase difference detection directions, and capable of adding output signals of the plurality of light receiving sections in accordance with the phase difference detection directions to output a sum signal; a control unit that performs focus detection or depth detection by phase difference detection based on the added signal of the image sensor; Equipped with the imaging element varies heights of potential barriers between the light receiving sections in accordance with the plurality of phase difference detection directions; the control unit instructs the imaging element to detect a specific phase difference in a direction to cause the imaging element to perform an accumulation operation, sets an accumulation determination level that differs according to the phase difference detection direction, and controls the accumulation operation of the imaging element based on the addition signal and the accumulation determination level; When the specified specific phase difference detection direction is a first phase difference detection direction in which a height of a potential barrier between the light receiving sections is higher among the plurality of phase difference detection directions, the image pickup element adding, within the plurality of pixels, the output signals corresponding to phase difference detection in a second phase difference detection direction perpendicular to the first phase difference detection direction; performing averaging of the added output signals corresponding to a first side in the first phase difference detection direction and the added output signals corresponding to a second side opposite to the first side at a predetermined number of the plurality of pixels in the second phase difference detection direction, and generating the added signal without performing averaging in the first phase difference detection direction; A detection device comprising:

2. The control unit is capable of performing a live view operation for performing a live view display on a display unit based on the sum signal output by the image sensor, In the live view operation, the image sensor adds the output signals corresponding to the phase difference detection in the second phase difference detection direction within the plurality of pixels, and 2. The detection device according to claim 1, wherein the summed output signals corresponding to the same side of the plurality of pixels in both the first phase difference detection direction and the second phase difference detection direction are averaged for a predetermined number of the plurality of pixels to generate the summed signal.

3. The detection device according to claim 2, characterized in that, in the live view operation, the control unit inputs the sum signal from the imaging element, and adds the sum signals that form a pair in the second phase difference detection direction to generate image data for display.

4. 4. The detection device according to claim 1, wherein the control unit sets the accumulation determination level in accordance with a height of the potential barrier corresponding to the phase difference detection direction.

5. 4. The detection device according to claim 1, wherein the control unit compares the sum signal with the accumulation determination level, and controls an accumulation operation of the imaging element based on a result of the comparison.

6. the imaging element is capable of amplifying and outputting the sum signal; 6. The detection device according to claim 1, wherein the control unit sets an amplification factor of the imaging element, and sets the accumulation determination level in accordance with the set amplification factor.

7. The detection device according to any one of claims 1 to 6, characterized in that the control unit sets a saturation judgment level according to a height of the potential barrier corresponding to the phase difference detection direction, and judges saturation by comparing the sum signal output by the imaging element with the saturation judgment level.

8. 8. The detection device according to claim 7, wherein the control unit performs focus detection or depth detection based on the result of the determination.

9. An imaging device having the detection device according to claim 1, a still image can be continuously captured based on a sum signal obtained by adding up all of the output signals of the plurality of light receiving sections of the image sensor; the control unit, when acquiring an image of the sum signal of the plurality of light receiving units according to the phase difference detection direction by capturing an image using the imaging element during continuous shooting of the still images, reads out only an addition signal corresponding to the phase difference detection direction in which the potential barrier is set higher as the addition signal of the plurality of light receiving units according to the phase difference detection direction.

10. A detection method for a detection device having an imaging element including a plurality of pixels each having a plurality of light receiving sections corresponding to a microlens, the plurality of light receiving sections being provided corresponding to a plurality of predetermined phase difference detection directions, and capable of adding output signals of the plurality of light receiving sections in accordance with the phase difference detection directions to output a sum signal, the imaging element varies heights of potential barriers between the light receiving sections in accordance with the plurality of phase difference detection directions; instructing the imaging element to perform an accumulation operation of the imaging element by specifying a specific phase difference detection direction, setting an accumulation determination level that differs according to the phase difference detection direction, and controlling the accumulation operation of the imaging element based on the addition signal and the accumulation determination level; performing focus detection or depth detection by phase difference detection based on the added signal of the image sensor; When the specified specific phase difference detection direction is a first phase difference detection direction in which a height of a potential barrier between the light receiving sections is higher among the plurality of phase difference detection directions, the image pickup element adding, within the plurality of pixels, the output signals corresponding to phase difference detection in a second phase difference detection direction perpendicular to the first phase difference detection direction; performing averaging of the added output signals corresponding to a first side in the first phase difference detection direction and the added output signals corresponding to a second side opposite to the first side at a predetermined number of the plurality of pixels in the second phase difference detection direction, and generating the added signal without performing averaging in the first phase difference detection direction; A detection method comprising:

Images