Imaging apparatus and method for controlling the same

Abstract

To read out an image signal necessary for focus detection and / or widening of a dynamic range in a shorter time according to a subject to be photographed and use of the read-out image signal.SOLUTION: An imaging apparatus has: an image pick-up device having a pixel region that has a plurality of pixels each including a microlens and a plurality of photoelectric conversion parts, amplification means that amplifies signals output from the pixel region, and scanning means that performs scanning so as to read out partial signals and addition signals from the plurality of photoelectric conversion parts; control means that controls the image pick-up device; processing means that widens a dynamic range; and focus detection means that performs focus detection of a phase difference system. The control means performs control in which the amplification means amplifies the partial signals and the addition signals by using a single gain; the processing means does not widen the dynamic range; and the focus detection means performs the focus detection, and control in which the amplification means amplifies the partial signals and the addition signals by using a plurality of different gains; the processing means widens the dynamic range; and the focus detection means performs the focus detection.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an imaging apparatus and a control method thereof. [Background technology]

[0002] In recent years, technologies have been proposed for image sensors that do not simply output image signals photoelectrically converted by pixels, but also expand the dynamic range or output distance information to the subject. Patent Document 1 proposes a technology that has a function for switching the input capacitance of an amplifier circuit provided for each column of the image sensor, and switches the gain according to the signal level. With a gain switching configuration such as that disclosed in Patent Document 1, image signals of low gain and high gain are output, and these are then combined in subsequent image processing, making it possible to create an image signal with a high dynamic range and low noise.

[0003] Meanwhile, a so-called image-surface phase-difference focus detection method has been proposed, in which a pair of images having parallax is read from an image sensor and focus detection is performed using a phase-difference detection method. One example of an image capture device that outputs a signal that can be used for the image-surface phase-difference focus detection method is one in which a pair of photoelectric conversion units is provided for each microlens that constitutes a two-dimensionally arranged microlens array. Patent Document 2 proposes an image capture device that can arbitrarily add or not add signals output by a pair of photoelectric conversion units to which light is incident via one microlens, for each pair of photoelectric conversion units. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-175517 [Patent Document 2] Japanese Patent Application Laid-Open No. 2001-83407 Summary of the Invention [Problem to be solved by the invention]

[0005] However, since the method of reading out the high-gain image signal and low-gain image signal for dynamic range expansion in Patent Document 1 is different from the method of reading out the image signal for phase difference detection in Patent Document 2, it was not possible to read out each in the same frame.

[0006] Furthermore, if the drive for reading out image signals for dynamic range expansion and the drive for reading out image signals for phase difference detection are switched within one frame in units of readout rows of the image sensor to maintain a high frame rate, the dynamic range cannot be expanded in the rows that read out image signals for phase difference detection.

[0007] The present invention has been made in consideration of the above problems, and aims to read out the image signals required for focus detection and / or dynamic range expansion in a shorter time depending on the subject being photographed and the use of the read-out image signals. [Means for solving the problem]

[0008] In order to achieve the above object, an imaging device of the present invention includes an imaging element having a pixel region in which a plurality of pixels, each having a microlens and a plurality of photoelectric conversion units, are arranged in a matrix, an amplification unit capable of amplifying a signal output from each pixel of the pixel region by a plurality of different gains, and a scanning unit capable of scanning the pixel region to read from each pixel a partial signal that is a signal accumulated in some of the photoelectric conversion units of the plurality of photoelectric conversion units in each pixel of the pixel region, and a sum signal obtained by adding together signals accumulated at the same timing in the plurality of photoelectric conversion units of each pixel of the pixel region; a control unit that controls the imaging element; a processing unit that expands a dynamic range using a plurality of signals obtained by amplifying the sum signal by the amplification unit by the plurality of different gains; and a focus detection unit that performs focus detection by a phase difference method using the partial signal and the sum signal. When the processing means does not expand the dynamic range and the focus detection means performs phase difference focus detection, the control means controls the scanning means to read out the partial signal and the sum signal from each pixel in the pixel area, and controls the amplifying means to amplify the partial signal and the sum signal read out from each pixel by a single gain and output the partial signal and the sum signal amplified by the single gain; when the processing means expands the dynamic range and the focus detection means does not perform phase difference focus detection, the control means controls the scanning means to read out the sum signal from each pixel in the pixel area but not read out the partial signal, and controls the amplifying means to amplify each of the sum signals read out from each pixel by the plurality of different gains and output the sum signals amplified by the plurality of different gains. [Effects of the Invention]

[0009] According to the present invention, it is possible to read out image signals necessary for focus detection and / or dynamic range expansion in a shorter time depending on the subject to be photographed and the use of the read-out image signals. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of an image sensor according to an embodiment of the present invention. [Figure 2] (a) A diagram showing details from the unit pixel of the image sensor to the AD circuit group, and (b) a circuit diagram showing the configuration of the column amplifier. [Figure 3] 6 is a timing chart showing control of a column amplifier when reading out a partial signal for phase difference detection and an image signal for dynamic range expansion according to an embodiment of the present invention. [Figure 4] 10 is a timing chart showing control of a column amplifier when a partial signal for phase difference detection and an image signal are read out when the dynamic range expansion according to the embodiment of the present invention is not performed. [Figure 5] 10 is a timing chart showing control of a column amplifier when an image signal for expanding a dynamic range is read out without reading out a partial signal for detecting a phase difference according to an embodiment of the present invention. [Figure 6] 4 is a diagram showing the timing of reading out an image signal from an image sensor according to the first embodiment. [Figure 7] 4 is a diagram showing the timing of reading out an image signal from an image sensor according to the first embodiment. [Figure 8] FIG. 1 is a block diagram showing a schematic configuration of an imaging apparatus according to a first embodiment. [Figure 9] 5 is a flowchart showing readout control of an image sensor in the first embodiment. [Figure 10] FIG. 10 is a diagram showing the timing of reading out an image signal from an image sensor according to the second embodiment. [Figure 11] FIG. 10 is a block diagram showing a schematic configuration of an imaging apparatus according to a second embodiment. [Figure 12] 10A and 10B are diagrams showing image data in each block in an imaging device according to a second embodiment. [Figure 13] FIG. 10 is a block diagram showing a schematic configuration of an imaging apparatus according to a third embodiment. [Figure 14] 10 is a flowchart showing processing in a third embodiment. [Figure 15] FIG. 10 is a block diagram showing a schematic configuration of an imaging apparatus according to a fourth embodiment. [Figure 16] 10 is a flowchart showing processing in the fourth embodiment. [Figure 17] FIG. 13 is a diagram showing details from a unit pixel to an AD circuit group of an image sensor according to a fifth embodiment. [Figure 18] 13 is a timing chart showing control of a column amplifier when partial signals are read out in parallel from two photoelectric conversion elements at high gain and low gain according to the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0012] First Embodiment 1(a) is a diagram showing an example of the configuration of an image sensor equipped with an AD converter according to an embodiment of the present invention. A plurality of unit pixels 101, each formed of a photodiode for photoelectric conversion or the like, are arranged in a matrix in a pixel region 100. For phase difference detection, each unit pixel 101 is configured with a photoelectric conversion unit A and a photoelectric conversion unit B for one microlens 111 (described later), and focus can be detected by determining the phase difference between image signals obtained from the photoelectric conversion unit A and the photoelectric conversion unit B.

[0013] 1(b) is a conceptual diagram showing a cross section of a unit pixel 101, and shows that two photoelectric conversion units A and B, each having a photodiode, are configured under one microlens 111. Each unit pixel 101 is also provided with a color filter 112. Generally, each pixel is provided with an RGB primary color filter in a Bayer array, with one of the three colors R (red), G (green), and B (blue) corresponding to each pixel, but this is not necessarily the case.

[0014] The vertical scanning circuit 102 controls the timing to sequentially read out pixel signals accumulated in the photoelectric conversion units A and B of the pixel region 100 during one frame period. Generally, pixel signals are sequentially read out row by row, from the top row to the bottom row, during one frame period. In this embodiment, the vertical scanning circuit 102 controls the reading out of each unit pixel 101 of a partial signal (signal A), which is the signal from photoelectric conversion unit A, and a sum signal (signal A+B), which is the sum of the signals from photoelectric conversion units A and B. By reading out the signal A+B in this manner, the signal A+B can be used as an image signal as is, and signal B can be obtained by subtracting signal A from signal A+B to perform focus detection using the image sensor phase difference method. However, if focus detection using the image sensor phase difference method is not performed, only the signal A+B can be read out.

[0015] The column amplifier group 103 is composed of multiple column amplifiers, one for each column of the pixel region 100, and is used to electrically amplify signals read from the pixel region 100. By amplifying signals in the column amplifier group 103, the signal level of the pixels is amplified to counter noise generated in the downstream AD circuit group 104, thereby equivalently improving the S / N ratio. Note that the column amplifier group 103 can amplify signals using multiple gains, and in this embodiment, the dynamic range is expanded by combining signals amplified with different gains. The detailed configuration of each column amplifier will be described later with reference to FIG. 2(b).

[0016] The AD circuit group 104 is made up of multiple circuits configured for each column of the pixel region 100, and converts the signals amplified by the column amplifier group 103 into digital signals. The pixel signals converted into digital signals are sequentially read out by a horizontal transfer circuit 105 and input to a signal processing unit 106. The signal processing unit 106 is a circuit that performs signal processing digitally, and in addition to performing offset correction such as FPN correction through digital processing, it can also perform simple gain calculations by performing shift calculations and multiplications. After each processing has been performed, the signals are output to the outside of the image sensor.

[0017] The memory 107 has a function of temporarily storing the A signal, A+B signal, and the like that are read from the pixel region 100 and processed by the column amplifier group 103, AD circuit group 104, and signal processing unit .

[0018] In the example shown in FIG. 1(b), each unit pixel 101 has two photoelectric conversion units A and B for one microlens 111. However, the number of photoelectric conversion units is not limited to two and may be more than two. The pupil division direction may be horizontal or vertical, or a mixture of these directions. Multiple pixels having different aperture positions for their light receiving units relative to the microlens 111 may be included. That is, any configuration may be used as long as two signals for phase difference detection, such as signals A and B, are obtained. The present invention is not limited to a configuration in which all pixels have multiple photoelectric conversion units, and may include a configuration in which pixels such as those shown in FIG. 2 are discretely provided within the normal pixels that make up the image sensor. Multiple types of pixels divided by different division methods may be included within the same image sensor.

[0019] Next, the circuit configuration and signal flow from the unit pixel 101 to the AD circuit group 104 will be described with reference to FIG. 2(a). A photoelectric conversion element 1101 corresponding to photoelectric conversion unit A in FIG. 1(b) and a photoelectric conversion element 1102 corresponding to photoelectric conversion unit B in FIG. 1(b) share a microlens and perform photoelectric conversion to convert light into electric charges. A transfer switch 1103 transfers the electric charges generated in the photoelectric conversion element 1101 to a subsequent circuit, and a transfer switch 1104 transfers the electric charges generated in the photoelectric conversion element 1102 to a subsequent circuit. A charge holding unit 1105 temporarily holds the electric charges transferred from the photoelectric conversion element 1101 and the photoelectric conversion element 1102 when the transfer switches 1103 and 1104 are ON. Therefore, the charge holding unit 1105 can hold only the charge of either the photoelectric conversion element 1101 or the photoelectric conversion element 1102, or the sum of the charges of both the photoelectric conversion element 1101 and the photoelectric conversion element 1102. The pixel amplifier 1106 converts the charge held in the charge holding unit 1105 into a voltage signal and transmits it to the subsequent column amplifier 103i through a vertical output line 1113. The current control unit 1107 controls the current of the vertical output line 1113.

[0020] 1 is made up of a plurality of column amplifiers 103i configured for each column, and amplifies signals output to each vertical output line 1113 and outputs the amplified signals to the downstream AD circuit group 104. Each AD circuit 104i constituting the AD circuit group 104 converts the analog signal output from the column amplifier 103i of the same column into a digital signal.

[0021] In the AD circuit 104i, the digital signals converted by the A / D conversion unit 1109 are temporarily stored in memories 1110 and 1111. The memory 1110 stores pixel signals read out from the photoelectric conversion element 1101 or the photoelectric conversion element 1102, and noise signals from the readout circuit unit (for convenience, this refers to the circuit from the charge holding unit 1105 to the A / D conversion unit 1109). On the other hand, the memory 1111 stores the noise signals from the readout circuit unit. Then, a subtraction unit 1112 subtracts the data held in the memory 1111 from the data held in the memory 1110, and the result is output to the horizontal transfer circuit 105 as a pixel signal.

[0022] 2(b) is a diagram showing the configuration of the column amplifier 103i. The column amplifier 103i is an inverting amplifier circuit made up of an operational amplifier 207, input-side capacitors 202 and 203, and feedback capacitors 205 and 206. In addition, the connections of the capacitors 202, 203, and 205 can be switched by switches 200, 201, and 204, respectively.

[0023] First, the signal input from the unit pixel 101 is stored in capacitors 202 and 203 by turning on switches 200 and 201. Then, in the case of an image signal with a proper exposure, switches 201 and 204 are turned off and switch 200 is turned on, thereby applying a high gain to the image signal and reading it out. Next, in the case of reading out an image signal of a high-brightness portion, switch 200 is turned off and switches 201 and 204 are turned on, thereby applying a low gain to the image signal and reading it out. In this way, by switching the capacitance of the capacitors with each switch, it is possible to read out image signals with different gains applied. It is expected that the dynamic range will be expanded by combining the image signals read out in this way.

[0024] FIG. 3 is a timing chart showing the control of the column amplifier 103i when reading out the partial signal for phase difference detection and the image signal for dynamic range expansion.

[0025] First, between times t1 and t4, the switch 200 is turned ON and the switches 201 and 204 are turned OFF, thereby setting the gain of the column amplifier 103i to high gain. In this state, at time t2, the transfer switch 1103 is turned ON and the A signal is read out. At this time, the A signal read out at high gain during the period from time t2 to t4 is A / D converted in the AD circuit 104i.

[0026] Next, from time t4 to t5, the switch 200 is turned OFF and the switches 201 and 204 are turned ON to set the gain of the column amplifier 103i to low gain. At this time, the A signal read out at low gain during the period from time t4 to t5 is A / D converted in the AD circuit 104i.

[0027] Between times t5 and t8, the switch 200 is turned ON again and the switches 201 and 204 are turned OFF, thereby setting the gain of the column amplifier 103i to high gain. In this state, at time t6, the transfer switch 1104 is turned ON and the B signal is read out. The A signal and the B signal are added in the charge holding unit 1105 and output as the A+B signal. At this time, the A+B signal read out at high gain during the period from time t6 to t8 is A / D converted in the AD circuit 104i.

[0028] Between times t8 and t9, the gain of the column amplifier 103i is set to low gain by turning off the switch 200 and turning on the switches 201 and 204. At this time, the A+B signals read out at low gain during the period from time t8 to t9 are A / D converted in the AD circuit 104i.

[0029] FIG. 4 is a timing chart showing the control of the column amplifier 103i when reading out a partial signal for phase difference detection and an image signal when the dynamic range is not expanded.

[0030] In this control, the gain of the column amplifier 103i is set to high gain by turning on the switch 200 and keeping the switches 201 and 204 off from time t11 to t16. In this state, the transfer switch 1103 is turned on at time t12 to read out the A signal. At this time, the A signal read out at high gain is A / D converted in the AD circuit 104i during the period from time t12 to t14.

[0031] Next, at time t14, the transfer switch 1104 is turned ON and the B signal is read out. The A signal and the B signal are added in the charge holding unit 1105 and output as the A+B signal. At this time, during the period from time t14 to t16, the A+B signal read out at high gain is A / D converted in the AD circuit 104i.

[0032] FIG. 5 is a timing chart showing the control of the column amplifier 103i when the partial signals for phase difference detection are not read out and the image signals for dynamic range expansion are read out.

[0033] In this control, the gain of the column amplifier 103i is set to high gain by turning on the switch 200 and keeping the switches 201 and 204 off from time t21 to t24. In this state, the transfer switches 1103 and 1104 are turned on at time t22 to read out the A+B signal. At this time, the A+B signal read out at high gain during the period from time t22 to t24 is A / D converted in the AD circuit 104i.

[0034] Next, from time t24 to t25, the switch 200 is turned OFF and the switches 201 and 204 are turned ON to set the gain of the column amplifier 103i to low gain. At this time, the A+B signal read out at low gain during the period from time t24 to t25 is A / D converted in the AD circuit 104i.

[0035] Fig. 6 is a diagram showing the timing of reading out image signals from the image sensor in the first embodiment, and illustrates the concept of signals read out in each frame under the control shown in Fig. 3. In Fig. 6, 1H transmission data refers to one row of data read out from the pixel area 100.

[0036] 6A is a diagram showing the readout timing when an image signal for phase difference detection, an image signal for phase difference detection when expanding the dynamic range, and an image signal for expanding the dynamic range are read out for all lines within one frame period. Specifically, an A signal (hereinafter referred to as a "high-gain A signal") is read out at a high gain, and then an A signal (hereinafter referred to as a "low-gain A signal") is read out at a low gain. Furthermore, an A+B signal (hereinafter referred to as a "high-gain A+B signal") is read out at a high gain, and then an A+B signal (hereinafter referred to as a "low-gain A+B signal") is read out at a low gain.

[0037] Phase difference detection can be performed using the high-gain A signal thus read out, a high-gain B signal obtained by subtracting the high-gain A signal from the high-gain A+B signal, the low-gain A signal, and a low-gain B signal obtained by subtracting the low-gain A signal from the low-gain A+B signal. Furthermore, the high-gain A+B signal and the low-gain A+B signal can be used as they are as image signals for dynamic range expansion.

[0038] As described above, with the configuration of the image sensor in this embodiment, the readout method from the image sensor when reading out one row of data can be changed, so that an image signal for phase difference detection and an image signal for dynamic range expansion can be read out within the same frame.

[0039] However, as shown in Figure 6(a), if you try to read out the image signal for phase difference detection and the image signal for dynamic range expansion in the same frame, the transmission time for one frame will be long. Therefore, if you want to quickly adjust the focus in a situation where you are not recording, such as when you half-press the shutter before taking a still image or when you enlarge the screen to adjust the focus, you should switch to the phase difference detection mode.

[0040] FIG. 6(b) is a diagram showing the readout timing when only image signals for phase difference detection are read out in the phase difference detection mode, and illustrates the concept of signals read out in each frame under the control shown in FIG. 4. Under this control, a high-gain A signal and a high-gain A+B signal are read out. In this way, by increasing the frame rate, it is possible to obtain information for phase difference detection with priority.

[0041] Furthermore, when the focus is predetermined and focus information is not required, such as when capturing a still image, the dynamic range expansion mode is used. Figure 7(a) is a diagram showing the readout timing when only the image signal for dynamic range expansion is read out in the dynamic range expansion mode, and illustrates the concept of the signals read out for each frame under the control shown in Figure 5. This control reads out a high-gain A+B signal and a low-gain A+B signal. This allows the shooting time for one frame to be shortened.

[0042] Furthermore, if phase difference detection is not required on the high-brightness side and only phase difference detection near the correct exposure is sufficient, a method can be considered in which the A signal, which is read out at low gain, is not output in order to increase the frame rate. Figure 7(b) shows the readout timing in such a case, and a high-gain A signal, a high-gain A+B signal, and a low-gain A+B signal are read out. In this way, an image signal for phase difference detection near the correct exposure and an image signal for expanding the dynamic range can be obtained.

[0043] In addition to the above, the mode may be changed depending on the subject. For example, even if the mode is for expanding the dynamic range as shown in Fig. 7(a), if there are no high-brightness subjects to be photographed, the mode may be changed to the phase difference detection mode (Fig. 6(b)), and phase difference detection may be performed while capturing a still image.

[0044] Furthermore, the order in which the high-gain A signal, low-gain A signal, high-gain A+B signal, and low-gain A+B signal are read out is not limited to the example described above. For example, the order in which the low-gain signal and the high-gain signal are read out can be reversed, or the high-gain signals can be read out first in succession and then the low-gain signals, or the low-gain signals can be read out first in succession and then the high-gain signals, or the like.

[0045] 8 is a block diagram showing the configuration of the imaging device of this embodiment, and shows only the components directly related to the present invention. Below, we will explain the signal flow when the image signal for phase difference detection and the image signal for dynamic range expansion are output from the image sensor 400 in the same frame as described in FIG. 6(a).

[0046] 1, and the low-gain A signal, high-gain A signal, low-gain A+B signal, and high-gain A+B signal output from the image sensor 400 are input to a distributor 401. The distributor 401 separates and outputs signals to be used as images (low-gain A+B signal, high-gain A+B signal) and signals to be used for phase difference detection (low-gain A signal, low-gain A+B signal, high-gain A signal, high-gain A+B signal).

[0047] The B signal generation unit 408 generates a high gain B signal by subtracting the high gain A from the high gain A+B signal output from the distributor 401. It also generates a low gain B signal by subtracting the low gain A signal from the low gain A+B signal.

[0048] The phase difference detection unit 403 detects the phase difference from the phase difference detection signals (high gain A signal, high gain B signal, low gain A signal, low gain B signal) output from the B signal generation unit 408. The focus calculation unit 404 performs focus calculation based on the phase difference information detected here and the focus position of the lens. Based on the obtained focus information, the imaging device notifies the user of focus information and controls the focus of the lens.

[0049] When the high-gain A+B signal is saturated, the image synthesis unit 402 synthesizes the dynamic range expansion signal output from the image sensor into an image with expanded dynamic range using an arbitrary synthesis method. For example, one method of synthesis uses a high-gain image for dark portions of the subject and a low-gain image for bright portions, but in this embodiment, the synthesis algorithm is not limited as long as it is a method of synthesis from two images with different gains. The control unit 405 controls switching of the shutter speed and gain of the image sensor 400, changes the readout drive, etc.

[0050] In the above example, the signal flow when the low gain A signal, high gain A signal, low gain A+B signal, and high gain A+B signal are read out has been described, but the signals may also be read out as described in Figures 6(b), 7(a), and 7(b). In this case, the image signal for phase difference detection and the image signal for dynamic range expansion may be distributed by the distributor 401 to the image synthesis unit 402 and the phase difference detection unit 403 as necessary, and then output.

[0051] 9 is a flowchart showing read control of the image sensor 400 by the control unit 405 in the first embodiment. First, in S100, the control unit 405 determines whether or not it is necessary to expand the dynamic range. Note that the determination of whether or not it is necessary to expand the dynamic range is made based on, for example, the half-press state of the shutter before capturing a still image, as described above, or the capture mode set by the user. Alternatively, the determination may be made using the result of the image composition unit 402 determining whether or not the high-gain A+B signal of the image signal of the previous frame is saturated.

[0052] If the dynamic range does not need to be expanded, the process proceeds to S112, where a high-gain A signal for phase difference detection is read out, and then the process proceeds to S113, where a high-gain A+B signal is read out. Then, in S121, it is determined whether readout from all rows has been completed, and if not, the process returns to S112, where readout continues. This corresponds to the readout order in the phase difference detection mode shown in FIG. 6(b).

[0053] On the other hand, if the dynamic range needs to be expanded, the process proceeds to S101, where it is determined whether phase difference detection is necessary. If it is determined that phase difference detection is not necessary, the process proceeds to S110, where the low-gain A+B signal is read out, and then the process proceeds to S111, where the high-gain A+B signal is read out. Then, in S122, it is determined whether readout from all rows has been completed, and if not, the process returns to S110 and continues readout. This corresponds to the readout order in the dynamic range expansion mode shown in FIG. 7(a).

[0054] On the other hand, if phase difference detection is required, the process proceeds to S102, where it is determined whether phase difference detection of high-brightness areas is required. If it is determined that phase difference detection is not required, the high-gain A signal is read out in S107, the high-gain A+B signal is read out in S108, and the low-gain A+B signal is read out in S109. Then, in S123, it is determined whether readout from all rows has been completed, and if not, the process returns to S107 and continues readout. This corresponds to the readout order shown in Figure 7(b).

[0055] If phase difference detection of high-brightness areas is also required, proceed to S103. Then, the high-gain A signal is read out in S103, the high-gain A+B signal is read out in S104, the low-gain A signal is read out in S105, and the low-gain A+B signal is read out in S106. Then, in S124, it is determined whether readout from all rows has been completed, and if not, return to S103 to continue readout. This corresponds to the readout order shown in Figure 6(a).

[0056] When reading of one frame is completed by any of the above readout methods, the process of FIG. 9 ends.

[0057] As described above, according to this embodiment, it is possible to obtain an image signal used for dynamic range expansion and an image signal used for phase difference detection in each frame. Furthermore, by controlling not to read out unnecessary image signals, it is possible to increase the frame rate compared to when all image signals used for dynamic range expansion and image signals used for phase difference detection are read out from each row.

[0058] <Second embodiment> Next, a second embodiment of the present invention will be described. Note that the image sensor in the second embodiment is the same as that described in the first embodiment, and therefore a description thereof will be omitted here.

[0059] 10 is a diagram showing the timing of reading out an image signal from an image sensor in the second embodiment. As in FIG.

[0060] FIG. 10(a) shows the readout timing when the image signals for phase difference detection and the image signals for dynamic range expansion are read out for all lines within one frame period without performing phase difference detection on the high-luminance side, which is the readout method shown in FIG. 7(b).

[0061] In this way, if image signals for phase difference detection and dynamic range expansion are read out for all lines in addition to the normal image signals, the amount of data becomes three times larger than normal readout without dynamic range expansion or phase difference detection, which puts strain on the transmission bandwidth and results in a slower frame rate than when only normal image signals are read out.

[0062] Therefore, in this embodiment, to increase the frame rate, an image signal for phase difference detection and an image signal for dynamic range expansion are output alternately for each row, as shown in FIG. 10(b). In this way, the frame rate can be increased by reducing the amount of data for one frame of image signal. Furthermore, the readout method shown in FIG. 10(b) enables phase difference detection over the entire screen area, making it possible to control the focus with high precision even when the user wants to focus on a specific location.

[0063] Fig. 11 is a block diagram showing a schematic configuration of an image pickup apparatus according to the second embodiment. The image pickup apparatus according to the second embodiment has a pixel interpolation processing unit 802 added to the configuration described in the first embodiment with reference to Fig. 8. Since the other configurations are the same as those in Fig. 8, the same reference numerals are used and descriptions thereof will be omitted where appropriate.

[0064] Hereinafter, a process will be described in which the image signals for phase difference detection and the image signals for dynamic range expansion are read out alternately for each row from the image sensor 400 as described with reference to FIG. 10(b).

[0065] The high-gain A signal read out at a high gain, the A+B signal read out at a high gain, and the A+B signal read out at a low gain output from the image sensor 400 are input to a distributor 401. The distributor 401 outputs signals to be used for images (low-gain A+B signal, high-gain A+B signal) and signals to be used for phase difference detection (high-gain A signal, high-gain A+B signal).

[0066] The B signal generation unit 408 subtracts high gain A from the high gain A+B signal output from the distributor 401 to generate a high gain B signal. The phase difference detection unit 403 detects the phase difference from the phase difference detection signals (high gain A signal, high gain B signal) output from the B signal generation unit 408. The focus calculation unit 404 performs focus calculation based on the phase difference information detected here and the focus position of the lens. Based on the obtained focus information, the imaging device notifies the user of focus information and controls the focus of the lens.

[0067] On the other hand, for a line from which the low-gain A+B signal has not been read, the pixel interpolation processing unit 802 interpolates pixel signals from the lines above and below, as will be described later. When the high-gain A+B signal is saturated, the image synthesis unit 402 synthesizes an expanded dynamic range image from the high-gain A+B signal and the low-gain A+B signal using an arbitrary synthesis method.

[0068] Fig. 12 shows an image of the image data in each block described in Fig. 11. Fig. 12(a) shows the image signal output from the image sensor 400. As shown in Fig. 12(a), the image signal output from the image sensor 400 is in a state in which the high-gain A signal and the low-gain A+B signal are alternately read out between the high-gain A+B signal.

[0069] 12(b) shows the image signal separated by the divider 401 and input to the pixel interpolation processing unit 802. The high-gain A signal used for phase difference detection is not necessary for expanding the dynamic range, and is therefore thinned out by the divider 401, and the line from which the high-gain A signal for phase difference detection is read out does not have the low-gain A+B signal for expanding the dynamic range.

[0070] 12(c) is a diagram showing the image signal output from the pixel interpolation processing unit 802. In the line where the high-gain A signal is read out, the low-gain A+B signal is interpolated using the adjacent low-gain A+B signals above and below.

[0071] 12(d) is a diagram showing the image signal input to the B signal generation unit 408. Since the phase difference detection process does not require a signal for expanding the dynamic range, the low gain A+B signal and the high gain A+B signal, which are image signals for expanding the dynamic range of a line from which the high gain A signal is not read, are thinned out by the distributor 401 and output.

[0072] As described above, according to the second embodiment, even when fewer image signals are read out, it is possible to expand the dynamic range and perform phase difference detection over the entire pixel area.

[0073] <Third embodiment> Next, a third embodiment of the present invention will be described. Note that the image sensor in the third embodiment is also the same as that described in the first embodiment, and therefore a description thereof will be omitted here.

[0074] In the second embodiment described above, for a line where a high-gain A signal for phase difference detection is read, an image signal for dynamic enlargement is generated by interpolating using adjacent low-gain A+B signals above and below. However, interpolating image signals from above and below reduces vertical resolution. Therefore, in this embodiment, the luminance level of the image signal for phase difference detection is detected, and if the luminance level is equal to or lower than a predetermined value and the phase difference is equal to or lower than a predetermined value, the image signal for phase difference detection is used as the image signal for dynamic range expansion instead of the interpolated low-gain A+B signal. This allows control to be performed so that vertical resolution is not reduced.

[0075] Fig. 13 is a block diagram showing a schematic configuration of an imaging device according to the third embodiment. The imaging device according to this embodiment is configured by adding a brightness detection unit 902 and a line selection processing unit 903 to the configuration described in the second embodiment with reference to Fig. 11. The processing of the distributor 401 is also different from that shown in Fig. 11. The other configurations are the same as those described in the first embodiment with reference to Fig. 8 and the second embodiment with reference to Fig. 11, and therefore the same reference numerals are used and descriptions thereof will be omitted where appropriate.

[0076] In the third embodiment, the distributor 401 outputs three signals to be used for the image: a high gain A signal, a low gain A+B signal, and a high gain A+B signal. These signals output from the distributor 401 are input to a luminance detection unit 902. The luminance detection unit 902 detects the luminance of the high gain A signal and outputs the detection result to a line selection processing unit 903. The line selection processing unit 903 selects whether to use the high gain A signal as an image signal for expanding the dynamic range based on information from the luminance detection unit 902 and the phase difference detection unit 403.

[0077] 14 is a flowchart showing processing in the third embodiment. First, in S300, the brightness detection unit 902 detects the brightness level of the high gain A signal of the input line. Next, in S301, the phase difference detection unit 403 detects the phase difference between the high gain A signal and the high gain B signal.

[0078] In S302, the line selection processing unit 903 determines whether the luminance level of the high gain A signal detected in S300 is equal to or less than a predetermined value Th1 (a threshold value). If it is equal to or less than the predetermined value Th1, the process proceeds to S303, and if it is greater than the predetermined value Th1, the process proceeds to S305.

[0079] In S303, it is determined whether the phase difference between the high gain A signal and the high gain B signal is equal to or less than a predetermined value Th2 based on the detection result of S301. If it is equal to or less than the predetermined value Th2, it is determined that the focus state is close to the in-focus state and the process proceeds to S304, and if it is greater than the predetermined value Th2, the process proceeds to S305.

[0080] In S304, the high-gain A signal is selected as the image signal for expanding the dynamic range. On the other hand, in S305, the low-gain A+B signal interpolated from above and below is selected as the image signal for expanding the dynamic range, and the pixel interpolation processing unit 802 generates upper and lower interpolated data in S306.

[0081] In S307, the image synthesis unit 402 performs dynamic range expansion processing using the high gain A signal or the low gain A+B signal.

[0082] As described above, according to the third embodiment, by using the high gain A signal when the brightness level of the high gain A signal is equal to or lower than a predetermined brightness and the image is in focus or close to in focus, the dynamic range can be expanded without reducing the vertical resolution.

[0083] <Fourth embodiment> Next, a fourth embodiment of the present invention will be described. Note that the image sensor in the fourth embodiment is also the same as that described in the first embodiment, and therefore a description thereof will be omitted here.

[0084] In the fourth embodiment, in addition to the third embodiment, the amount of motion of a subject is detected during video shooting, and if the amount of motion is less than a predetermined amount, the drive of the image sensor is changed and processing is performed to switch between readout for phase difference detection and readout for dynamic range expansion for each frame. Furthermore, when combining images, image signals from previous and following frames are used to prevent a decrease in vertical resolution.

[0085] Fig. 15 is a block diagram showing a schematic configuration of an imaging device according to the fourth embodiment. The configuration shown in Fig. 15 is obtained by adding a motion vector detection unit 909 and a memory 910 to the configuration described in the third embodiment with reference to Fig. 13. The other components are the same as those in Fig. 13, so the same reference numerals are used and descriptions thereof will be omitted where appropriate.

[0086] The motion vector detection unit 909 detects the amount of motion of the subject and outputs the detection result to the image synthesis unit 402 and the control unit 405. The memory 910 can temporarily store the image signal, making it possible to synthesize images using image signals from previous and subsequent frames.

[0087] Fig. 16 is a flowchart showing the processing in the fourth embodiment. Note that the same steps as those described with reference to Fig. 14 in the third embodiment are given the same step numbers, and the description thereof will be omitted as appropriate.

[0088] In S304, when the high gain A signal is selected as the image signal for dynamic range expansion, in S407 the image synthesis unit 402 performs the dynamic range expansion process using the high gain A signal as described in the third embodiment.

[0089] On the other hand, if the high-gain A signal is not selected, the process proceeds from S303 to S408, where it is determined whether the amount of motion of the subject is equal to or greater than a predetermined value Th3. If it is equal to or greater than the predetermined value Th3, the process proceeds to S305, where upper and lower interpolated data interpolated from the low-gain A+B signals of the upper and lower pixels is selected. In S306, the pixel interpolation processing unit 802 generates the upper and lower interpolated data, and in S410, the image synthesis unit 402 performs dynamic range expansion processing using the low-gain A+B signals.

[0090] On the other hand, if the amount of movement of the subject is less than a predetermined value Th3, the drive of the image sensor 400 is changed in S409. Here, a process is performed in which, for each frame, the high-gain A signal and the high-gain A+B signal are read out (the readout method of FIG. 6(b)), and the high-gain A+B signal and the low-gain A+B signal are read out (the readout method of FIG. 7(a)). Then, in S411, a signal interpolated from the low-gain A+B signals of the previous and next frames stored in memory 910 is used to perform a dynamic range expansion process.

[0091] As described above, according to the fourth embodiment, when there is little movement of the subject, dynamic range expansion processing is performed using a signal interpolated from the low-gain A+B signals of the previous and next frames, thereby suppressing a decrease in vertical resolution.

[0092] <Fifth embodiment> Next, a fifth embodiment of the present invention will be described. Note that the image sensor in the fifth embodiment is the same as that described in the first embodiment, and therefore a description thereof will be omitted here.

[0093] 17 is a diagram showing details from the unit pixel 101 to the AD circuit group 104 of an image sensor in which vertical output lines and column amplifiers are connected to photoelectric conversion units A and B, respectively. This configuration enables signals from a photoelectric conversion element 1101 corresponding to photoelectric conversion unit A in FIG. 1(b) and a photoelectric conversion element 1102 corresponding to photoelectric conversion unit B to be read out in parallel to vertical output lines 1113 and 1115, respectively.

[0094] 17, the same components as those in Fig. 2 are denoted by the same reference numerals, and their description will be omitted. In the configuration shown in Fig. 17, in addition to the configuration shown in Fig. 2, each pixel 101 has a charge holding unit 1108 and a pixel amplifier 1114 for independently reading out partial signals from the photoelectric conversion element 1102, and also has a current control unit 1116 for controlling the current of a vertical output line 1115.

[0095] The column amplifier 103i includes two amplifiers for amplifying signals output from the photoelectric conversion element 1101 and the photoelectric conversion element 1102 to the vertical output lines 1113 and 1115, respectively, and outputs the signals to the downstream AD circuit group 104. In addition to the configuration shown in Fig. 2, each AD circuit 104i also includes an A / D conversion unit 1118 for converting an analog signal from the photoelectric conversion element 1102 into a digital signal, and memories 1119 and 1120 for temporarily storing the digital signal.

[0096] With the above configuration, partial signals from the photoelectric conversion elements 1101 and 1102 can be read out in parallel, processed, and output, so that although the circuit scale increases, the readout time can be shortened.

[0097] Furthermore, an A+B signal can be obtained by adding the A signal and the B signal read out in the signal processing unit 106. In this case, however, the B signal generating unit 408 in FIGS. 8, 11, and 13 is not necessary, and an A+B signal generating unit is required between the distributor 401 and the image combining unit 402.

[0098] FIG. 18 is a timing chart showing the control of the column amplifier 103i when partial signals are read out in parallel from the photoelectric conversion elements 1101 and 1102 at high gain and low gain in the configuration shown in FIG.

[0099] First, between times t51 and t54, the switch 200 is turned ON and the switches 201 and 204 are turned OFF, thereby setting the gain of the column amplifier 103i to high gain. In this state, at time t52, the transfer switches 1103 and 1104 are turned ON, and the A signal and the B signal are read out. At this time, the A signal and the B signal read out at high gain during the period from time t52 to t54 are A / D converted in the AD circuit 104i.

[0100] Next, between times t54 and t55, the switch 200 is turned OFF and the switches 201 and 204 are turned ON to set the gain of the column amplifier 103i to low gain. At this time, the A signal and the B signal read out at low gain during the period from time t54 to t55 are A / D converted in the AD circuit 104i.

[0101] From the read high gain A signal, high gain B signal, low gain A signal, and low gain B signal, the high gain A signal and the high gain B signal are added in the signal processing unit 106 of Fig. 1 to generate and output a high gain A+B signal. Also, by adding the low gain A signal and the low gain B signal in the signal processing unit 106, a low gain A+B signal is generated and output.

[0102] 18, the case where signals are read out at high gain and low gain respectively has been described, but the present invention is not limited to this. For example, when a high-gain signal or a low-gain signal is not required, the readout speed can be increased by reading out the signal at the required gain.

[0103] As described above, according to the fifth embodiment, it is possible to obtain an image signal used for dynamic range expansion and an image signal used for phase difference detection in each frame without reducing the frame rate.

[0104] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0105] For example, even when a low-gain A+B signal is interpolated using image signals from upper and lower pixels, the interpolation ratio between the upper and lower pixels may be changed arbitrarily depending on the situation.

[0106] <Other embodiments> The present invention may be applied to a system made up of multiple devices (for example, a host computer, an interface device, a scanner, a video camera, etc.), or to an apparatus made up of a single device.

[0107] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0108] 100: pixel region, 101: unit pixel, 103: column amplifier group, 103i: column amplifier, 111: microlens, 1101, 1102: photoelectric conversion element, 400: image sensor, 401: distributor, 402: image synthesis unit, 403: phase difference detection unit, 404: focus calculation unit, 405: control unit, 408: B signal generation unit, 802: pixel interpolation processing unit, 902: luminance detection unit, 903: line selection processing unit, 909: motion vector detection unit, 910: memory

Claims

1. an imaging element having: a pixel region in which a plurality of pixels, each having a microlens and a plurality of photoelectric conversion units, are arranged in a matrix; an amplifier capable of amplifying a signal output from each pixel of the pixel region by a plurality of different gains; and a scanning unit capable of scanning the pixel region so as to read from each pixel a partial signal, which is a signal accumulated in some of the photoelectric conversion units of the plurality of photoelectric conversion units in each pixel of the pixel region, and a sum signal obtained by adding together signals accumulated at the same timing in the plurality of photoelectric conversion units of each pixel of the pixel region; a control means for controlling the imaging element; a processing means for expanding a dynamic range using a plurality of signals obtained by amplifying the sum signal with the plurality of different gains by the amplifying means; a focus detection unit that performs focus detection using the partial signal and the added signal by a phase difference method, The control means When the dynamic range is not expanded by the processing means and phase difference focus detection is performed by the focus detection means, the scanning means is controlled to read out the partial signals and the summed signals from each pixel of the pixel area, and the amplifying means is controlled to amplify the partial signals and the summed signals read out from each pixel by a single gain and output the partial signals and the summed signals amplified by the single gain, The processing means expands the dynamic range, and when phase-difference focus detection is not performed by the focus detection means, the scanning means reads out the sum signal from each pixel in the pixel area and controls not to read out the partial signal, and the amplifying means amplifies each of the sum signals read out from each pixel by the plurality of different gains and outputs the sum signals amplified by the plurality of different gains. An imaging device characterized by:

2. a pixel region in which a plurality of pixels, each having a microlens and a plurality of photoelectric conversion units, are arranged in a matrix; an amplifier capable of amplifying a signal output from each pixel of the pixel region by a plurality of different gains; and a scanning unit configured to scan the pixel region so as to read from each pixel a partial signal that is a signal accumulated in some of the photoelectric conversion units of the plurality of photoelectric conversion units in each pixel of the pixel region, and a sum signal obtained by adding together signals accumulated at the same timing in the plurality of photoelectric conversion units of each pixel of the pixel region, a control step in which a control means controls the imaging element; a dynamic range expansion step in which the processing means expands the dynamic range using a plurality of signals obtained by amplifying the added signal with the plurality of different gains by the amplification means; a focus detection step in which a focus detection unit performs focus detection using the partial signal and the added signal by a phase difference method, In the control step, When the dynamic range is not expanded by the processing means and phase difference focus detection is performed by the focus detection means, the scanning means is controlled to read out the partial signals and the summed signals from each pixel of the pixel area, and the amplifying means is controlled to amplify the partial signals and the summed signals read out from each pixel by a single gain and output the partial signals and the summed signals amplified by the single gain, The processing means expands the dynamic range, and when phase-difference focus detection is not performed by the focus detection means, the scanning means reads out the sum signal from each pixel in the pixel area and controls not to read out the partial signal, and the amplifying means amplifies each of the sum signals read out from each pixel by the plurality of different gains and outputs the sum signals amplified by the plurality of different gains. A control method comprising:

3. an imaging element having: a pixel region in which a plurality of pixels, each having a microlens and a plurality of photoelectric conversion units, are arranged in a matrix; an amplifier capable of amplifying a signal output from each pixel in the pixel region by a plurality of different gains; and a scanning unit that scans the pixel region so as to read out a signal from each pixel in the pixel region; a control means for controlling the imaging element; a processing means for expanding a dynamic range using signals obtained by amplifying signals read from each pixel in the pixel region with a plurality of different gains by the amplifying means; a focus detection unit that performs focus detection by a phase difference method using signals read out from the plurality of photoelectric conversion units of each pixel in the pixel area, the scanning means separately reads out signals from the plurality of photoelectric conversion units from each pixel; The control means when the processing means does not expand the dynamic range and the focus detection means performs phase difference focus detection, the amplifying means amplifies, by a single gain, each of the signals accumulated at the same timing in the plurality of photoelectric conversion units of each pixel in the pixel area, and controls the amplifying means to output the signals of the plurality of photoelectric conversion units of each pixel amplified by the single gain; When the dynamic range is expanded by the processing means and phase difference focus detection is not performed by the focus detection means, the amplification means amplifies, by the plurality of different gains, each of the signals accumulated at the same timing in the plurality of photoelectric conversion units of each pixel in the pixel area, and controls the amplification means to output the signals of the plurality of photoelectric conversion units of each pixel amplified by the plurality of different gains. An imaging device characterized by:

4. 4. The imaging apparatus according to claim 3, wherein the imaging element further comprises an adder that adds together signals read out from the plurality of photoelectric conversion units of each pixel in the pixel region.

5. A control method for an imaging device having an imaging element including: a pixel region in which a plurality of pixels, each having a microlens and a plurality of photoelectric conversion units, are arranged in a matrix; an amplifier capable of amplifying a signal output from each pixel of the pixel region by a plurality of different gains; and a scanning unit that scans the pixel region so as to read out a signal from each pixel of the pixel region, a control step in which a control means controls the imaging element; a processing step in which a processing unit expands a dynamic range by using signals obtained by amplifying signals read out from each pixel in the pixel region with the plurality of different gains by the amplification unit; a focus detection step in which a focus detection unit performs focus detection using a phase difference method using signals read out from the plurality of photoelectric conversion units of each pixel in the pixel area, the scanning means separately reads out signals from the plurality of photoelectric conversion units from each of the pixels; In the control step, when the processing means does not expand the dynamic range and the focus detection means performs phase difference focus detection, the amplification means amplifies, by a single gain, each of the signals accumulated at the same timing in the plurality of photoelectric conversion units of each pixel in the pixel area, and outputs the signals of the plurality of photoelectric conversion units of each pixel amplified by the single gain; When the processing means expands the dynamic range and the focus detection means does not perform phase difference focus detection, the amplification means amplifies, by the plurality of different gains, each of the signals accumulated at the same timing in the plurality of photoelectric conversion units of each pixel in the pixel area, and outputs the signals of the plurality of photoelectric conversion units of each pixel amplified by the plurality of different gains. The control method is characterized by controlling as follows.

6. A program for causing a computer to execute each step of the control method according to claim 2 or 5.

7. A computer-readable storage medium storing the program according to claim 6.

Images