Imaging device and its control method

Abstract

To read out an image signal necessary for focus detection and / or widening of a dynamic range in a shorter time according to a subject to be photographed and use of the read-out image signal.SOLUTION: An imaging apparatus has: an image pick-up device having a pixel region that has a plurality of pixels each including a microlens and a plurality of photoelectric conversion parts, amplification means that amplifies signals output from the pixel region, and scanning means that performs scanning so as to read out partial signals and addition signals from the plurality of photoelectric conversion parts; control means that controls the image pick-up device; processing means that widens a dynamic range; and focus detection means that performs focus detection of a phase difference system. The control means performs control in which the amplification means amplifies the partial signals and the addition signals by using a single gain; the processing means does not widen the dynamic range; and the focus detection means performs the focus detection, and control in which the amplification means amplifies the partial signals and the addition signals by using a plurality of different gains; the processing means widens the dynamic range; and the focus detection means performs the focus detection.SELECTED DRAWING: Figure 8

Description

Technical Field

[0005] , ,

[0001] The present invention relates to an imaging device and a control method thereof.

Background Art

[0002] In recent years, in imaging devices, not only outputting an image signal photoelectrically converted by pixels, but also technologies such as expanding the dynamic range and outputting distance information to a subject have been proposed. In Patent Document 1, a technology having a function of switching the input capacitance of an amplifier circuit provided for each column of an imaging device and switching the gain according to the signal level has been proposed. With a configuration for switching the gain as in Patent Document 1, by outputting image signals of a low-gain signal and a high-gain signal and synthesizing them in subsequent image processing, it becomes possible to create an image signal with a high dynamic range and low noise.

[0003] On the other hand, a so-called imaging surface phase difference method of focus detection has been proposed, in which an image having a pair of parallax is read out from an imaging device and phase difference detection type focus detection is performed. As an example of an imaging device that outputs a signal that can be used for the focus detection method of the imaging surface phase difference method, there is one in which a pair of photoelectric conversion units are provided for each micro lens constituting a two-dimensionally arranged micro lens array. Patent Document 2 proposes an imaging device capable of arbitrarily performing addition / non-addition of signals output from a pair of photoelectric conversion units through which light is incident via one micro lens for each pair of photoelectric conversion units.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the method for reading out high-gain and low-gain image signals for dynamic range expansion described in Patent Document 1 differs from the method for reading out image signals for phase difference detection described in Patent Document 2, making it impossible to read them in the same frame.

[0006] Furthermore, in order to maintain a high frame rate, suppose the drive for reading out image signals for dynamic range expansion and the drive for reading out image signals for phase difference detection are switched on a per-readout row basis of the image sensor within a single frame. In that case, dynamic range expansion cannot be performed in the rows where image signals for phase difference detection are read out.

[0007] This invention was made in view of the above-mentioned problems, and aims to read out the image signals necessary for focus detection and / or dynamic range expansion in a shorter time, depending on the subject being photographed and the intended use of the readout image signal. [Means for solving the problem]

[0008] To achieve the above objective, the imaging device of the present invention comprises an image sensor having: a pixel region in which a plurality of pixels, each having a microlens and a plurality of photoelectric conversion units, are arranged in a matrix; amplification means capable of amplifying the signal output from each pixel of the pixel region with a plurality of different gains; scanning means capable of scanning the pixel region so as to read out from each pixel a partial signal, which is a signal accumulated in some of the plurality of photoelectric conversion units in each pixel of the pixel region, and an added signal, which is obtained by adding the signals accumulated at the same timing in the plurality of photoelectric conversion units in each pixel of the pixel region; control means for controlling the image sensor; processing means for expanding the dynamic range using a plurality of signals obtained by amplifying the added signal with the plurality of different gains by the amplification means; and focus detection means for performing phase difference focus detection using the partial signal and the added signal. When the processing means does not expand the dynamic range and the focus detection means performs phase difference type focus detection, the control means controls the scanning means to read out the partial signal and the summation signal from each pixel in the pixel area, and controls the amplification means to amplify the partial signal and the summation signal read out from each pixel with a single gain and output the partial signal and the summation signal amplified with the single gain. When the processing means expands the dynamic range and the focus detection means does not perform phase difference type focus detection, the control means controls the scanning means to read out the summation signal from each pixel in the pixel area but not the partial signal, and controls the amplification means to amplify each of the summation signals read out from each pixel with the plurality of different gains and output the summation signal amplified with the plurality of different gains. [Effects of the Invention]

[0009] According to the present invention, depending on the subject being photographed and the intended use of the readout image signal, the image signals necessary for focus detection and / or dynamic range expansion can be read out in a shorter time. [Brief explanation of the drawing]

[0010] [Figure 1] A diagram showing a schematic configuration of an image sensor according to an embodiment of the present invention. [Figure 2] (a) A diagram showing the details from the unit pixel of the image sensor to the AD circuit group, (b) A circuit diagram showing the configuration of the column amplifier. [Figure 3] A timing chart showing the control of a column amplifier when reading out a partial signal for phase difference detection and an image signal for dynamic range expansion according to an embodiment of the present invention. [Figure 4] A timing chart showing the control of the column amplifier when reading out a partial signal for phase difference detection and an image signal, in the case where dynamic range expansion is not performed according to the embodiment of the present invention. [Figure 5] A timing chart showing the control of a column amplifier when reading out an image signal for dynamic range expansion without reading out a partial signal for phase difference detection according to an embodiment of the present invention. [Figure 6] A diagram showing the readout timing of the image signal from the image sensor in the first embodiment. [Figure 7] A diagram showing the readout timing of the image signal from the image sensor in the first embodiment. [Figure 8] A block diagram showing the schematic configuration of the imaging device in the first embodiment. [Figure 9] A flowchart showing the readout control of the image sensor in the first embodiment. [Figure 10] A diagram showing the readout timing of the image signal from the image sensor in the second embodiment. [Figure 11] A block diagram showing the schematic configuration of the imaging device in the second embodiment. [Figure 12] A diagram showing image data in each block within the imaging device in the second embodiment. [Figure 13] A block diagram showing the schematic configuration of the imaging device in the third embodiment. [Figure 14] A flowchart illustrating the process in the third embodiment. [Figure 15] A block diagram showing the schematic configuration of the imaging device in the fourth embodiment. [Figure 16] Flowchart showing the processing in the fourth embodiment. [Figure 17] Diagram showing details from a unit pixel of an image pickup device to an AD circuit group in the fifth embodiment. [Figure 18] Timing chart showing the control of a column amplifier when partial signals are read out in parallel from two photoelectric conversion elements according to the fifth embodiment, at high gain and low gain.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings.

[0012] <First Embodiment> FIG. 1(a) is a diagram showing a configuration example of an image pickup device equipped with an AD converter according to an embodiment of the present invention. In a pixel region 100, a plurality of unit pixels 101 formed of photodiodes for photoelectric conversion or the like are arranged in a matrix. Each unit pixel 101 is composed of a photoelectric conversion section A and a photoelectric conversion section B for one microlens 111 described later, and the focus can be detected by obtaining the phase difference of the image signals obtained from the photoelectric conversion section A and the photoelectric conversion section B.

[0013] FIG. 1(b) is a conceptual diagram showing a cross section of the unit pixel 101, showing that two photoelectric conversion sections A and B each having a photodiode are formed under one microlens 111. Each unit pixel 101 is provided with a color filter 112. Generally, it is often an RGB primary color filter in a Bayer array corresponding to one of the three colors R (red), G (green), and B for each pixel, but this is not necessarily the case.

[0014] The vertical scanning circuit 102 controls the timing for sequentially reading out the pixel signals stored in the photoelectric conversion unit A and the photoelectric conversion unit B of the pixel area 100 during one frame period. Generally, the pixel signals are read out row by row from the top row to the bottom row during one frame period. In this embodiment, the vertical scanning circuit 102 controls the reading out of each unit pixel 101 a partial signal (A signal), which is the signal from the photoelectric conversion unit A, and an added signal (A+B signal), which is the sum of the signals from the photoelectric conversion unit A and the photoelectric conversion unit B. By reading them out in this way, the A+B signal can be used directly as an image signal, and the B signal can be obtained by subtracting the A signal from the A+B signal, allowing for focus detection using the image plane phase difference method. However, if focus detection using the image plane phase difference method is not performed, only the A+B signal can be read out.

[0015] The column amplifier group 103 consists of multiple column amplifiers configured for each column of the pixel area 100, and is used to electrically amplify the signal read from the pixel area 100. By amplifying the signal with the column amplifier group 103, the signal level of the pixels can be amplified against noise generated in the subsequent AD circuit group 104, effectively improving the signal-to-noise ratio. The column amplifier group 103 can amplify the signal using multiple gains, and in this embodiment, the dynamic range is expanded by combining signals amplified with different gains. The detailed configuration of each column amplifier will be described later with reference to Figure 2(b).

[0016] The AD circuit group 104 consists of multiple circuits configured for each column of the pixel area 100, and converts the signal amplified by the column amplifier group 103 into a digital signal. The pixel signals converted into digital signals are sequentially read out by the horizontal transfer circuit 105 and input to the signal processing unit 106. The signal processing unit 106 is a circuit that performs signal processing digitally, and in addition to performing offset correction such as FPN correction through digital processing, it can also perform gain calculations in an easy manner by performing shift operations and multiplication. After each process is performed, the signal is output to the outside of the image sensor.

[0017] Memory 107 has the function of temporarily holding A signals, A+B signals, etc., that have been read from the pixel area 100 and processed by the column amplifier group 103, AD circuit group 104, and signal processing unit 106.

[0018] In the example shown in Figure 1(b), each unit pixel 101 has two photoelectric conversion units A and B for each microlens 111, but the number of photoelectric conversion units is not limited to two and may be more. Also, the pupil division direction may be horizontal, vertical, or a mixture of both. Furthermore, there may be multiple pixels with different aperture positions for the light-receiving part for each microlens 111. In other words, any configuration that results in two signals for phase difference detection, such as the A signal and the B signal, is acceptable. Moreover, the present invention is not limited to a configuration in which all pixels have multiple photoelectric conversion units, but may also be a configuration in which pixels as shown in Figure 2 are discretely provided within the normal pixels that make up the image sensor. Furthermore, multiple types of pixels divided by different division methods may be included within the same image sensor.

[0019] Next, the circuit configuration and signal flow from the unit pixel 101 to the AD circuit group 104 will be explained using Figure 2(a). The photoelectric conversion element 1101, which corresponds to the photoelectric conversion unit A in Figure 1(b), and the photoelectric conversion element 1102, which corresponds to the photoelectric conversion unit B in Figure 1(b), share a microlens and perform photoelectric conversion to convert light into electric charge. The transfer switch 1103 transfers the charge generated by the photoelectric conversion element 1101 to the subsequent circuit, and the transfer switch 1104 transfers the charge generated by the photoelectric conversion element 1102 to the subsequent circuit. The charge holding unit 1105 temporarily holds the charge transferred from the photoelectric conversion elements 1101 and 1102 when the transfer switches 1103 and 1104 are ON. Therefore, the charge holding unit 1105 can hold the charge of either the photoelectric conversion element 1101 or the photoelectric conversion element 1102, or the sum of the charges of both the photoelectric conversion element 1101 and the photoelectric conversion element 1102. The pixel amplifier 1106 converts the charge held in the charge holding unit 1105 into a voltage signal and transmits it to the subsequent row amplifier 103i through the vertical output line 1113. The current control unit 1107 controls the current of the vertical output line 1113.

[0020] As described above, the column amplifier group 103 shown in Figure 1 consists of multiple column amplifiers 103i configured for each column, which amplify the signals output to each vertical output line 1113 and output them to the subsequent AD circuit group 104. Each AD circuit 104i that makes up the AD circuit group 104 converts the analog signals output from the column amplifiers 103i of the same column into digital signals.

[0021] In the AD circuit 104i, the digital signal converted by the A / D conversion unit 1109 is temporarily held in memory 1110 and memory 1111. Memory 1110 holds the pixel signal read from the photoelectric conversion element 1101 or 1102 and the noise signal from the readout circuit (for convenience, this refers to the circuit from the charge holding unit 1105 to the A / D conversion unit 1109). Meanwhile, memory 1111 holds the noise signal from the readout circuit. Then, the subtraction unit 1112 subtracts the data held in memory 1111 from the data held in memory 1110, and the result is output to the horizontal transfer circuit 105 as the pixel signal.

[0022] Figure 2(b) shows the configuration of the column amplifier 103i. The column amplifier 103i is an inverting amplifier circuit consisting of the operational amplifier 207, input capacitors 202 and 203, and feedback capacitors 205 and 206. In addition, the connections of capacitors 202, 203, and 205 can be switched using switches 200, 201, and 204.

[0023] First, the signal input from unit pixel 101 is stored in capacitors 202 and 203 by turning on switches 200 and 201. Then, in the case of an image signal with proper exposure, switches 201 and 204 are turned OFF and switch 200 is turned ON to read out the image signal with high gain. Next, when reading out an image signal of a high-brightness area, switch 200 is turned OFF and switches 201 and 204 are turned ON to read out the image signal with low gain. In this way, by switching the capacitance of the capacitors using each switch, it is possible to read out the image signal with different gains. It is envisioned that the dynamic range will be expanded by combining the image signals read out in this way.

[0024] Figure 3 is a timing chart showing the control of the column amplifier 103i when reading out the partial signal for phase difference detection and the image signal for dynamic range expansion.

[0025] First, between times t1 and t4, switch 200 is turned ON and switches 201 and 204 are turned OFF to set the gain of the column amplifier 103i to high gain. In this state, at time t2, the transfer switch 1103 is turned ON and the A signal is read out. At this time, between times t2 and t4, the A signal read out at high gain is A / D converted in the AD circuit 104i.

[0026] Next, between time t4 and t5, switch 200 is turned OFF and switches 201 and 204 are turned ON to set the gain of the column amplifier 103i to a low gain. During this time, the A signal read out at the low gain is converted by the AD circuit 104i within the period from time t4 to t5.

[0027] Between times t5 and t8, switch 200 is turned ON again, and switches 201 and 204 are turned OFF to set the gain of the column amplifier 103i to high gain. In this state, at time t6, the transfer switch 1104 is turned ON and the B signal is read out. The A signal and the B signal are added in the charge holding unit 1105 and output as the A+B signal. At this time, between times t6 and t8, the A+B signal read out at high gain is A / D converted in the AD circuit 104i.

[0028] Between times t8 and t9, switch 200 is turned OFF and switches 201 and 204 are turned ON to set the gain of the column amplifier 103i to a low gain. During this time, the A+B signal read out at the low gain is converted by the AD circuit 104i within the period from time t8 to t9.

[0029] Figure 4 is a timing chart showing the control of the column amplifier 103i when reading out the partial signal for phase difference detection and the image signal, when dynamic range expansion is not performed.

[0030] In this control, the gain of the column amplifier 103i is set to high gain by turning switch 200 ON and keeping switches 201 and 204 OFF between times t11 and t16. In this state, the transfer switch 1103 is turned ON at time t12 to read out the A signal. At this time, between times t12 and t14, the A signal read out at high gain is converted to digital in the AD circuit 104i.

[0031] Next, at time t14, the transfer switch 1104 is turned ON and the B signal is read out. The A signal and the B signal are added together in the charge holding unit 1105 and output as the A+B signal. At this time, within the period from time t14 to t16, the A+B signal read out with high gain is A / D converted in the AD circuit 104i.

[0032] Figure 5 is a timing chart showing the control of the column amplifier 103i when reading out the image signal for dynamic range expansion without reading out the partial signal for phase difference detection.

[0033] In this control, the gain of the column amplifier 103i is set to high gain by turning switch 200 ON and keeping switches 201 and 204 OFF between times t21 and t24. In this state, at time t22, the transfer switches 1103 and 1104 are turned ON and the A+B signal is read out. At this time, between times t22 and t24, the A+B signal read out at high gain is converted A / D in the AD circuit 104i.

[0034] Next, between times t24 and t25, switch 200 is turned OFF, and switches 201 and 204 are turned ON, thereby setting the gain of the column amplifier 103i to a low gain. During this time, within the period from time t24 to t25, the A+B signal read out at the low gain is converted by the AD circuit 104i.

[0035] Figure 6 shows the timing of reading the image signal from the image sensor in the first embodiment, illustrating the concept of the signal read out in each frame by the control shown in Figure 3. In Figure 6, 1H transmission data refers to one line of data read out from pixel area 100.

[0036] Figure 6(a) shows the readout timing when the image signals for phase difference detection, the image signals for phase difference detection when expanding the dynamic range, and the image signals for dynamic range expansion are read out for all lines within one frame period. Specifically, the A signal (hereinafter referred to as the "high-gain A signal") is read out at high gain, and then the A signal (hereinafter referred to as the "low-gain A signal") is read out at low gain. Furthermore, the A+B signal (hereinafter referred to as the "high-gain A+B signal") is read out at high gain, and then the A+B signal (hereinafter referred to as the "low-gain A+B signal") is read out at low gain.

[0037] Phase difference detection can be performed using the high-gain A signal read out in this way, the high-gain B signal obtained by subtracting the high-gain A signal from the high-gain A+B signal, the low-gain A signal, and the low-gain B signal obtained by subtracting the low-gain A signal from the low-gain A+B signal. Furthermore, the high-gain A+B signal and the low-gain A+B signal can be used directly as image signals for dynamic range expansion.

[0038] In this embodiment, the image sensor configuration allows for changing the reading method from the image sensor when reading out one line of data, enabling the reading of both an image signal for phase difference detection and an image signal for dynamic range expansion within the same frame.

[0039] However, as shown in Figure 6(a), attempting to read out the image signal for phase difference detection and the image signal for dynamic range expansion in the same frame results in a longer transmission time per frame. Therefore, when you want to quickly focus in situations where you are not recording, such as when the shutter is half-pressed before taking a still image or when you zoom in on the screen to focus, you should switch to phase difference detection mode.

[0040] Figure 6(b) shows the readout timing when only the image signal for phase difference detection is read out in phase difference detection mode, illustrating the concept of the signals read out in each frame by the control shown in Figure 4. This control reads out the high-gain A signal and the high-gain A+B signal. In this way, by increasing the frame rate, it is possible to prioritize the acquisition of information for phase difference detection.

[0041] Furthermore, when the focus is predetermined and focus information is not needed, such as during still image shooting, the camera switches to dynamic range expansion mode. Figure 7(a) shows the readout timing when only the dynamic range expansion image signal is read out in dynamic range expansion mode, illustrating the concept of the signals read out in each frame by the control shown in Figure 5. This control reads out both the high-gain A+B signal and the low-gain A+B signal. In this way, the shooting time for one frame can be shortened.

[0042] Furthermore, if phase difference detection on the high-brightness side is not required, and only phase difference detection near the correct exposure is sufficient, then a method can be considered in which the A signal, which is read out at a low gain, is not output in order to increase the frame rate. Figure 7(b) shows the readout timing in such a case, where the high-gain A signal, the high-gain A+B signal, and the low-gain A+B signal are read out. In this way, an image signal for phase difference detection near the correct exposure and an image signal for dynamic range expansion can be obtained.

[0043] In addition, you may change the mode depending on the subject, other than those mentioned above. For example, even in the dynamic range expansion mode shown in Figure 7(a), if there are no high-brightness subjects to be photographed, you may switch to the phase difference detection mode (Figure 6(b)) and perform phase difference detection while taking a still image.

[0044] Furthermore, the order in which the high-gain A signal, low-gain A signal, high-gain A+B signal, and low-gain A+B signal are read is not limited to the examples described above. For example, the order in which the low-gain and high-gain signals are read can be reversed, or the high-gain signals can be read consecutively before the low-gain signals, or the low-gain signals can be read consecutively before the high-gain signals, and so on, as can be changed as appropriate.

[0045] Figure 8 is a block diagram showing the configuration of the imaging device in this embodiment, and only the components directly related to the present invention are shown. Below, the signal flow when the image sensor 400 outputs an image signal for phase difference detection and an image signal for dynamic range expansion in the same frame, as described in Figure 6(a), will be explained.

[0046] The image sensor 400 is the image sensor described in Figure 1, and the low-gain A signal, high-gain A signal, low-gain A+B signal, and high-gain A+B signal output from the image sensor 400 are input to the distributor 401. The distributor 401 separates and outputs the signals to be used as images (low-gain A+B signal, high-gain A+B signal) and the signals to be used for phase difference detection (low-gain A signal, low-gain A+B signal, high-gain A signal, high-gain A+B signal).

[0047] The B signal generation unit 408 generates a high-gain B signal by subtracting the high-gain A signal from the high-gain A+B signal output from the distributor 401. It also generates a low-gain B signal by subtracting the low-gain A signal from the low-gain A+B signal.

[0048] The phase difference detection unit 403 detects the phase difference from the phase difference detection signals (high-gain A signal, high-gain B signal, low-gain A signal, low-gain B signal) output from the B signal generation unit 408. The focus calculation unit 404 calculates the focus based on the phase difference information detected here and the focus position of the lens. Based on the obtained focus information, the imaging device notifies the user of the focus information or controls the focus of the lens.

[0049] The image synthesis unit 402 synthesizes a dynamic range-expanded image from the dynamic range expansion signal output from the image sensor using an arbitrary synthesis method when the high-gain A+B signal is saturated. For example, one method is to use a high-gain image for dark parts of the subject and a low-gain image for bright parts, but in this embodiment, the synthesis algorithm is not limited as long as it is a method of synthesizing from two images with different gains. The control unit 405 controls the switching of the shutter speed and gain of the image sensor 400, and changes the readout drive, etc.

[0050] In the example described above, the signal flow when reading out the low-gain A signal, high-gain A signal, low-gain A+B signal, and high-gain A+B signal was explained. However, the signals may also be read out as described in Figures 6(b), 7(a), and 7(b). In that case, the distributor 401 can distribute the image signal for phase difference detection and the image signal for dynamic range expansion to the image synthesis unit 402 and the phase difference detection unit 403 as needed.

[0051] Figure 9 is a flowchart showing the readout control of the image sensor 400 by the control unit 405 in this first embodiment. First, in S100, the control unit 405 determines whether or not it is necessary to expand the dynamic range. The determination of whether or not it is necessary to expand the dynamic range here is made, for example, according to the half-press state of the shutter before taking a still image, as described above, or according to the shooting mode set by the user. Alternatively, the determination may be made using the result of the determination by the image synthesis unit 402 as to whether or not the high-gain A+B signal of the image signal from the previous frame is saturated.

[0052] If dynamic range expansion is not required, the process proceeds to S112 to read the high-gain A signal for phase difference detection, and then to S113 to read the high-gain A+B signal. Then, in S121, it is determined whether reading from all rows has been completed. If not, the process returns to S112 to continue reading. This corresponds to the reading order in the phase difference detection mode shown in Figure 6(b).

[0053] On the other hand, if dynamic range expansion is required, the process proceeds to S101 to determine whether phase difference detection is necessary. If it is determined that it is not necessary, the process proceeds to S110 to read the low-gain A+B signal, and then to S111 to read the high-gain A+B signal. Then, in S122, it is determined whether reading from all rows has been completed, and if not, the process returns to S110 to continue reading. This corresponds to the reading order in the dynamic range expansion mode shown in Figure 7(a).

[0054] On the other hand, if phase difference detection is required, the process proceeds to S102 to determine whether phase difference detection is necessary for the high-brightness area. If it is determined that it is not necessary, the high-gain A signal is read out in S107, the high-gain A+B signal in S108, and the low-gain A+B signal in S109. Then, in S123, it is determined whether reading from all rows has been completed, and if not, the process returns to S107 to continue reading. This corresponds to the reading order shown in Figure 7(b).

[0055] If phase difference detection of high-brightness areas is also required, the process proceeds to S103. Then, in S103, the high-gain A signal is read out; in S104, the high-gain A+B signal is read out; in S105, the low-gain A signal is read out; and in S106, the low-gain A+B signal is read out. Finally, in S124, it is determined whether reading from all rows has been completed. If not, the process returns to S103 and continues reading. This corresponds to the reading order shown in Figure 6(a).

[0056] Once one frame has been read using one of the above reading methods, the process shown in Figure 9 is terminated.

[0057] As described above, according to this embodiment, in each frame, an image signal used for dynamic range expansion and an image signal used for phase difference detection can be obtained. Furthermore, by controlling the reading process to avoid reading unnecessary image signals, the frame rate can be increased compared to the case where all image signals used for dynamic range expansion and image signals used for phase difference detection are read from each row.

[0058] <Second Embodiment> Next, a second embodiment of the present invention will be described. Note that the image sensor in the second embodiment is the same as that described in the first embodiment, so its description will be omitted here.

[0059] Figure 10 shows the timing of reading the image signal from the image sensor in the second embodiment. Similar to Figure 6, 1H transmission data refers to one line of data read from pixel area 100.

[0060] Figure 10(a) shows the readout timing when the image signal for phase difference detection and the image signal for dynamic range expansion are read out for all lines within one frame period without performing phase difference detection on the high-brightness side, and the readout method is shown in Figure 7(b).

[0061] If, in addition to the normal image signal, image signals for phase difference detection and dynamic range expansion are read out for all lines in this manner, the amount of data becomes three times greater than with normal readout without dynamic range expansion and phase difference detection, putting pressure on the transmission bandwidth. As a result, the frame rate becomes slower than when only the normal image signal is read out.

[0062] Therefore, in this embodiment, in order to increase the frame rate, as shown in Figure 10(b), an image signal for phase difference detection and an image signal for dynamic range expansion are output alternately for each row. By reducing the amount of data in the image signal for one frame in this way, the frame rate can be increased. Furthermore, with the readout method shown in Figure 10(b), phase difference detection is possible in the entire area of ​​the screen, so even if the user wants to focus on an arbitrary location, it is possible to control the focus with high precision.

[0063] Figure 11 is a block diagram showing the schematic configuration of the imaging device in the second embodiment. The imaging device in this second embodiment is the same as the configuration described in the first embodiment with reference to Figure 8, with the addition of a pixel interpolation processing unit 802. The other configurations are the same as in Figure 8, so the same reference numerals are used and explanations are omitted as appropriate.

[0064] The following describes the processing when the image signal for phase difference detection and the image signal for dynamic range expansion are read alternately from the image sensor 400 row by row, as explained in Figure 10(b).

[0065] The high-gain A signal, the high-gain A+B signal, and the low-gain A+B signal, all read out at high gain from the image sensor 400, are input to the distributor 401. The distributor 401 separates and outputs the signals used for image processing (low-gain A+B signal, high-gain A+B signal) and the signals used for phase difference detection (high-gain A signal, high-gain A+B signal).

[0066] The B signal generation unit 408 subtracts high-gain A from the high-gain A+B signal output from the distributor 401 to generate a high-gain B signal. The phase difference detection unit 403 detects the phase difference from the phase difference detection signals (high-gain A signal, high-gain B signal) output from the B signal generation unit 408. The focus calculation unit 404 performs focus calculations based on the phase difference information detected here and the focus position of the lens. Based on the obtained focus information, the imaging device notifies the user of the focus information or controls the focus of the lens.

[0067] Meanwhile, the pixel interpolation processing unit 802 interpolates the pixel signal from the upper and lower lines for lines where the low-gain A+B signal has not been read out, as will be described later. The image synthesis unit 402 synthesizes a dynamic range-expanded image from the high-gain A+B signal and the low-gain A+B signal using an arbitrary synthesis method when the high-gain A+B signal is saturated.

[0068] Here, Figure 12 shows an image of the image data in each block described in Figure 11. Figure 12(a) shows the image signal output from the image sensor 400. As shown in Figure 12(a), the image signal output from the image sensor 400 is in a state where high-gain A signals and low-gain A+B signals are read out alternately between high-gain A+B signals.

[0069] Figure 12(b) shows the image signal separated by the distributor 401 and input to the pixel interpolation processing unit 802. The high-gain A signal used for phase difference detection is not necessary for expanding the dynamic range, so it is thinned out by the distributor 401. On the line from which the high-gain A signal is read out for phase difference detection, the low-gain A+B signal for expanding the dynamic range is absent.

[0070] Figure 12(c) shows the image signal output from the pixel interpolation processing unit 802. In the line where the high-gain A signal is read out, it shows how the low-gain A+B signal is interpolated using the adjacent upper and lower low-gain A+B signals.

[0071] Finally, Figure 12(d) shows the image signal input to the B signal generation unit 408. Since a signal for dynamic range expansion is not required for the phase difference detection process, the low-gain A+B signal and high-gain A+B signal, which are image signals for dynamic range expansion for lines where the high-gain A signal has not been read out, are thinned out by the distributor 401 and output.

[0072] As described above, according to this second embodiment, even when reading out fewer image signals, it becomes possible to perform dynamic range expansion and phase difference detection across the entire pixel area.

[0073] <Third Embodiment> Next, a third embodiment of the present invention will be described. Note that the image sensor in the third embodiment is the same as that described in the first embodiment, so its description will be omitted here.

[0074] In the second embodiment described above, on the line from which the high-gain A signal was read out for phase difference detection, the image signal for dynamic range expansion was generated by interpolation using adjacent low-gain A+B signals above and below. However, interpolating the image signal from above and below reduces the vertical resolution. Therefore, in this embodiment, the brightness level of the image signal for phase difference detection is detected, and if the brightness level is below a predetermined value and the phase difference is below a predetermined value, the image signal for phase difference detection is used as the image signal for dynamic range expansion instead of the interpolated low-gain A+B signal. This enables control that does not reduce the vertical resolution.

[0075] Figure 13 is a block diagram showing the schematic configuration of the imaging device in the third embodiment. The imaging device in this embodiment is the same as the configuration described in the second embodiment with reference to Figure 11, with the addition of a brightness detection unit 902 and a line selection processing unit 903. Also, the processing of the distributor 401 differs from that shown in Figure 11. The other configurations are the same as those described in the first embodiment with reference to Figure 8 and the second embodiment with reference to Figure 11, so the same reference numerals are used and explanations are omitted as appropriate.

[0076] In the third embodiment, the distributor 401 outputs three signals for use in the image: a high-gain A signal, a low-gain A+B signal, and a high-gain A+B signal. These signals output from the distributor 401 are input to the luminance detection unit 902. The luminance detection unit 902 detects the luminance of the high-gain A signal and outputs the detection result to the line selection processing unit 903. The line selection processing unit 903 selects whether to use the high-gain A signal as the image signal for dynamic range expansion based on the information from the luminance detection unit 902 and the phase difference detection unit 403.

[0077] Figure 14 is a flowchart showing the process in the third embodiment. First, in S300, the brightness detection unit 902 detects the brightness level of the high-gain A signal of the input line. Next, in S301, the phase difference detection unit 403 detects the phase difference between the high-gain A signal and the high-gain B signal.

[0078] In S302, the line selection processing unit 903 determines whether the brightness level of the high-gain A signal detected in S300 is less than or equal to a predetermined value Th1 (below the threshold). If it is less than or equal to the predetermined value Th1, the process proceeds to S303; otherwise, it proceeds to S305.

[0079] In S303, based on the detection results from S301, it is determined whether the phase difference between the high-gain A signal and the high-gain B signal is less than or equal to a predetermined value Th2. If it is less than or equal to the predetermined value Th2, it is determined that the focus state is close to the focus state and the process proceeds to S304. If it is greater than the predetermined value Th2, the process proceeds to S305.

[0080] In S304, a high-gain A signal is selected as the image signal for expanding the dynamic range. On the other hand, in S305, it is selected to use a low-gain A+B signal interpolated from above and below as the image signal for expanding the dynamic range, and the pixel interpolation processing unit 802 generates the upper and lower interpolated data in S306.

[0081] In S307, the image synthesis unit 402 performs dynamic range expansion processing using a high-gain A signal or a low-gain A+B signal.

[0082] As described above, according to this third embodiment, by using the high-gain A signal when the brightness level of the high-gain A signal is below a predetermined brightness level and the image is in focus or close to focus, the dynamic range can be expanded without reducing the vertical resolution.

[0083] <Fourth Embodiment> Next, a fourth embodiment of the present invention will be described. Note that the image sensor in the fourth embodiment is the same as that described in the first embodiment, so its description will be omitted here.

[0084] In the fourth embodiment, in addition to the third embodiment, the amount of motion of the subject is detected during video recording, and if it is below a predetermined amount of motion, the drive of the image sensor is changed, and the readout for phase difference detection and the readout for dynamic range expansion are swapped frame by frame. Furthermore, when combining images, the image signals of the preceding and succeeding frames are used to prevent a decrease in vertical resolution.

[0085] Figure 15 is a block diagram showing the schematic configuration of the imaging device in the fourth embodiment. The configuration shown in Figure 15 is the same as the configuration described in the third embodiment with reference to Figure 13, with the addition of a motion vector detection unit 909 and a memory 910. The other configurations are the same as in Figure 13, so the same reference numerals are used and explanations are omitted as appropriate.

[0086] The motion vector detection unit 909 detects the amount of motion of the subject and outputs the detected result to the image synthesis unit 402 and the control unit 405. The memory 910 can temporarily store the image signal, making it possible to synthesize an image using the image signals of the preceding and succeeding frames.

[0087] Figure 16 is a flowchart showing the process in the fourth embodiment. Note that steps similar to those described with reference to Figure 14 of the third embodiment are given the same step numbers, and their explanations are omitted as appropriate.

[0088] In S304, if a high-gain A signal is selected as the image signal for dynamic range expansion, in S407, the image synthesis unit 402 performs dynamic range expansion processing using the high-gain A signal as described in the third embodiment.

[0089] On the other hand, if a high-gain A signal is not selected, the process proceeds from S303 to S408, where it is determined whether the amount of motion of the subject is greater than or equal to a predetermined value Th3. If it is greater than or equal to the predetermined value Th3, the process proceeds to S305, where upper and lower interpolated data is selected from the low-gain A+B signals of the upper and lower pixels. In S306, the pixel interpolation processing unit 802 generates the upper and lower interpolated data, and in S410, the image synthesis unit 402 performs dynamic range expansion processing using the low-gain A+B signals.

[0090] On the other hand, if the amount of motion of the subject is less than a predetermined value Th3, the drive of the image sensor 400 is changed in S409. Here, for each frame, the process alternates between reading out the high-gain A signal and the high-gain A+B signal (readout method in Figure 6(b)) and reading out the high-gain A+B signal and the low-gain A+B signal (readout method in Figure 7(a)). Then, in S411, the dynamic range expansion process is performed using signals interpolated from the low-gain A+B signals of the preceding and succeeding frames that have been stored in memory 910.

[0091] As described above, according to this fourth embodiment, when there is little movement of the subject, dynamic range expansion processing is performed using a signal interpolated from the low-gain A+B signals of the preceding and succeeding frames, thereby suppressing a decrease in vertical resolution.

[0092] <Fifth Embodiment> Next, a fifth embodiment of the present invention will be described. Note that the image sensor in the fifth embodiment is the same as that described in the first embodiment, so its description will be omitted here.

[0093] Figure 17 shows the details of the image sensor from unit pixel 101 to AD circuit group 104, with vertical output lines and column amplifiers connected to photoelectric conversion units A and B, respectively. The configuration allows the signals from photoelectric conversion element 1101, corresponding to photoelectric conversion unit A in Figure 1(b), and photoelectric conversion element 1102, corresponding to photoelectric conversion unit B, to be read out in parallel to vertical output lines 1113 and 1115, respectively.

[0094] In Figure 17, the same reference numerals are used for components similar to those in Figure 2, and their explanations are omitted. In the configuration shown in Figure 17, in addition to the configuration shown in Figure 2, each pixel 101 is equipped with a charge holding unit 1108 and a pixel amplifier 1114 in order to independently read out partial signals from the photoelectric conversion element 1102, and also has a current control unit 1116 for controlling the current of the vertical output line 1115.

[0095] Furthermore, the column amplifier 103i includes two amplifiers for amplifying the signals output from the photoelectric conversion elements 1101 and 1102 to the respective vertical output lines 1113 and 1115, respectively, and outputs them to the subsequent AD circuit group 104. In addition, each AD circuit 104i has, in addition to the configuration shown in Figure 2, an A / D conversion unit 1118 for converting the analog signal from the photoelectric conversion element 1102 into a digital signal, and memories 1119 and 1120 for temporarily holding the digital signal.

[0096] With the above configuration, partial signals from the photoelectric conversion elements 1101 and 1102 can be read out in parallel, processed, and output. Although the circuit size increases, the readout time can be shortened.

[0097] Furthermore, the A+B signal can be obtained by adding the A signal and B signal read out in the signal processing unit 106. However, in that case, the B signal generation unit 408 in Figures 8, 11, and 13 is not necessary, and the A+B signal generation unit is required between the distributor 401 and the image synthesis unit 402.

[0098] Figure 18 is a timing chart showing the control of the column amplifier 103i when reading out partial signals in parallel from the photoelectric conversion elements 1101 and 1102 with high gain and low gain, in the configuration shown in Figure 17.

[0099] First, between time t51 and t54, switch 200 is turned ON and switches 201 and 204 are turned OFF to set the gain of the column amplifier 103i to high gain. In this state, at time t52, transfer switches 1103 and 1104 are turned ON and the A signal and B signal are read out, respectively. At this time, between time t52 and t54, the A signal and B signal read out at high gain are A / D converted in the AD circuit 104i.

[0100] Next, between time t54 and t55, switch 200 is turned OFF and switches 201 and 204 are turned ON to set the gain of the column amplifier 103i to a low gain. During this time, within the period from time t54 to t55, the A signal and B signal read out at the low gain are converted by the AD circuit 104i.

[0101] From the read-out high-gain A signal, high-gain B signal, low-gain A signal, and low-gain B signal, the signal processing unit 106 in Figure 1 adds the high-gain A signal and the high-gain B signal to generate and output a high-gain A+B signal. Furthermore, the signal processing unit 106 also adds the low-gain A signal and the low-gain B signal to generate and output a low-gain A+B signal.

[0102] While the timing chart in Figure 18 illustrates the case of reading signals with high gain and low gain respectively, the present invention is not limited to these cases. For example, if a high-gain signal or a low-gain signal is not required, the reading speed can be increased by reading the signal with the required gain.

[0103] As described above, according to this fifth embodiment, an image signal used for dynamic range expansion and an image signal used for phase difference detection can be obtained in each frame without reducing the frame rate.

[0104] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate.

[0105] For example, when interpolating a low-gain A+B signal using the image signals of the upper and lower pixels, the interpolation ratio of the upper and lower pixels can be arbitrarily changed depending on the situation.

[0106] <Other Embodiments> Furthermore, the present invention may be applied to a system consisting of multiple devices (for example, a host computer, interface devices, scanners, video cameras, etc.) or to a device consisting of a single device.

[0107] Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of Symbols]

[0108] 100: Pixel area, 101: Unit pixel, 103: Column amplifier group, 103i: Column amplifier, 111: Microlens, 1101, 1102: Photoelectric conversion element, 400: Image sensor, 401: Distributor, 402: Image synthesis unit, 403: Phase difference detection unit, 404: Focus calculation unit, 405: Control unit, 408: B signal generation unit, 802: Pixel interpolation processing unit, 902: Brightness detection unit, 903: Line selection processing unit, 909: Motion vector detection unit, 910: Memory

Claims

1. An image sensor comprising: a pixel region in which a plurality of pixels, each having a microlens and a plurality of photoelectric conversion units, are arranged in a matrix; amplification means capable of amplifying the signals output from each pixel of the pixel region with a plurality of different gains; scanning means capable of scanning the pixel region so as to read out from each pixel a partial signal, which is a signal accumulated in some of the plurality of photoelectric conversion units in each pixel of the pixel region, and an added signal, which is the sum of the signals accumulated at the same timing in the plurality of photoelectric conversion units in each pixel of the pixel region; Control means for controlling the image sensor, A processing means for expanding the dynamic range using a plurality of signals obtained by amplifying the summed signal with a plurality of different gains using the amplification means, The system includes a focus detection means that performs phase difference type focus detection using the aforementioned partial signal and the aforementioned summation signal, The control means is When dynamic range expansion by the processing means is not performed and phase difference focus detection by the focus detection means is performed, the scanning means is controlled to read out the partial signal and the summation signal from each pixel in the pixel region, and the amplification means is controlled to amplify the partial signal and the summation signal read out from each pixel with a single gain, and to output the partial signal and the summation signal amplified with the single gain. When the processing means expands the dynamic range and the focus detection means does not perform phase difference type focus detection, the scanning means is controlled to read the sum signal from each pixel in the pixel region and not read the partial signal, and the amplification means amplifies each of the sum signals read from each pixel with the plurality of different gains and outputs the sum signal amplified with the plurality of different gains. An imaging device characterized by the following features.

2. A control method for an imaging device having an image sensor, comprising: a pixel region in which a plurality of pixels are arranged in a matrix, each having a microlens and a plurality of photoelectric conversion units; amplification means capable of amplifying the signals output from each pixel of the pixel region with a plurality of different gains; a scanning means for scanning the pixel region so as to read out from each pixel a partial signal, which is a signal accumulated in some of the plurality of photoelectric conversion units in each pixel of the pixel region, and an added signal, which is the sum of the signals accumulated at the same timing in the plurality of photoelectric conversion units in each pixel of the pixel region; The control means includes a control step for controlling the image sensor, The processing means includes a dynamic range expansion step in which the summation signal is amplified by the amplification means with a plurality of different gains to expand the dynamic range of a plurality of signals obtained, The focus detection means includes a focus detection step that performs phase difference type focus detection using the partial signal and the summation signal, In the control process described above, When dynamic range expansion by the processing means is not performed and phase difference focus detection by the focus detection means is performed, the scanning means is controlled to read out the partial signal and the summation signal from each pixel in the pixel region, and the amplification means is controlled to amplify the partial signal and the summation signal read out from each pixel with a single gain, and to output the partial signal and the summation signal amplified with the single gain. When the processing means expands the dynamic range and the focus detection means does not perform phase difference type focus detection, the scanning means is controlled to read the sum signal from each pixel in the pixel region and not read the partial signal, and the amplification means amplifies each of the sum signals read from each pixel with the plurality of different gains and outputs the sum signal amplified with the plurality of different gains. A control method characterized by the following:

3. An image sensor having a pixel region in which multiple pixels, each having a microlens and multiple photoelectric conversion units, are arranged in a matrix, amplification means capable of amplifying the signals output from each pixel of the pixel region with multiple different gains, and scanning means for scanning the pixel region to read signals from each pixel of the pixel region, Control means for controlling the image sensor, A processing means for expanding the dynamic range using a signal obtained by amplifying the signals read from each pixel of the aforementioned pixel region with multiple different gains using the amplification means, The system includes a focus detection means that performs phase-difference type focus detection using signals read out from each of the plurality of photoelectric conversion units of each pixel in the aforementioned pixel region, The scanning means reads out the signals from the plurality of photoelectric conversion units separately from each pixel. The control means is When dynamic range expansion by the processing means is not performed and phase difference focus detection by the focus detection means is performed, the amplification means amplifies each of the signals accumulated at the same timing in the plurality of photoelectric conversion units of each pixel in the pixel region with a single gain, and controls the output of the signal from the plurality of photoelectric conversion units of each pixel amplified with the single gain. The processing means expands the dynamic range, and when the focus detection means does not perform phase-difference focus detection, the amplification means amplifies each of the signals accumulated at the same timing in the plurality of photoelectric conversion units of each pixel in the pixel region by the plurality of different gains, and controls the output of the signals from the plurality of photoelectric conversion units of each pixel amplified by the plurality of different gains. An imaging device characterized by the following features.

4. The imaging apparatus according to claim 3, further comprising an adding means for adding signals read out from each of the plurality of photoelectric conversion units of each pixel in the pixel region.

5. A control method for an imaging device having an image sensor comprising: a pixel region in which a plurality of pixels are arranged in a matrix, each having a microlens and a plurality of photoelectric conversion units; amplification means capable of amplifying the signals output from each pixel of the pixel region with a plurality of different gains; and scanning means for scanning the pixel region to read signals from each pixel of the pixel region, The control means includes a control step for controlling the image sensor, The processing means includes a process of expanding the dynamic range using a signal obtained by amplifying the signals read from each pixel of the pixel region with the amplification means at a plurality of different gains, The focus detection means includes a focus detection step that performs phase difference type focus detection using signals read out from each of the plurality of photoelectric conversion units of each pixel in the pixel region, The scanning means reads out the signals from the plurality of photoelectric conversion units separately from each pixel. In the control process described above, When dynamic range expansion by the processing means is not performed and phase difference type focus detection by the focus detection means is performed, the amplification means amplifies each of the signals accumulated at the same timing in the plurality of photoelectric conversion units of each pixel in the pixel region with a single gain, and outputs the signals from the plurality of photoelectric conversion units of each pixel amplified with the single gain. When the dynamic range is expanded by the processing means and phase-difference focus detection is not performed by the focus detection means, the amplification means amplifies each of the signals accumulated at the same timing in the plurality of photoelectric conversion units of each pixel in the pixel region by the plurality of different gains, and outputs the signals from the plurality of photoelectric conversion units of each pixel amplified by the plurality of different gains. A control method characterized by controlling in such a manner.

6. A program for causing a computer to perform each step of the control method described in claim 2 or 5.

7. A computer-readable storage medium storing the program described in claim 6.

Images