Imaging apparatus

Abstract

To acquire a HDR image of which a dynamic range is expanded while suppressing a subject shake.SOLUTION: Each of pixels comprises: a photoelectric conversion part; and two electric charging holding parts that accumulates an electric charging that was converted by the photoelectric conversion part, and performs a HDR composition on the basis of a plurality of signals of the same frame obtained by amplifying a signal based on the electric charging accumulated to the two electric charging holding parts with a plurality of different gains.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an imaging device. [Background technology]

[0002] Conventionally, techniques have been proposed to expand the dynamic range of gradation using imaging elements such as CCD image sensors and CMOS image sensors used in general digital cameras. One technique for expanding the dynamic range is to synthesize images taken with multiple different exposures, but this requires taking multiple exposure images in sequence, which results in differences in the timing of each exposure. Therefore, when taking pictures of a moving object, subject blur may be noticeable when the images are synthesized.

[0003] Patent Document 1 also describes a method for expanding the dynamic range in a pixel configuration having two charge storage units for each unit pixel. That is, the magnitude relationship between the charge signal accumulated in the photoelectric conversion unit of each pixel and a reference voltage is determined. If the charge signal is smaller than the reference voltage, the first charge storage unit is used, and if the charge signal is larger than the reference voltage, the second charge storage unit is used, thereby expanding the dynamic range. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2020-178163 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology disclosed in Patent Document 1, the dynamic range of the image data is determined by the capacitance of the charge retention section, so the effect of expanding the dynamic range is limited compared to a method of HDR compositing images taken with multiple different exposures.

[0006] An object of the present invention is to provide an imaging device capable of capturing HDR images with little subject blur and a sufficiently expanded dynamic range. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the imaging device of the present invention comprises a plurality of pixels, each having a photoelectric conversion unit, a first charge holding unit that holds the charge converted by the photoelectric conversion unit, and a second charge holding unit that accumulates the charge converted by the photoelectric conversion unit, an amplification means capable of amplifying a signal based on the charge at a plurality of different gains, and an image processing means that performs HDR synthesis processing of the plurality of signals amplified at the different gains by the amplification means, wherein the amounts of charge accumulated in the first charge holding unit and the second charge holding unit during one frame period are different, and at least one of the signals based on the charge accumulated in the first charge holding unit and the second charge holding unit are amplified at the different gains by the amplification means. Effect of the Invention

[0008] According to the present invention, it is possible to provide an imaging device capable of capturing an HDR image with little subject blur and a sufficiently expanded dynamic range. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing a configuration of an imaging apparatus according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing the configuration of an image sensor according to an embodiment. [Diagram 3] FIG. 2 is a diagram showing the configuration of an image sensor according to an embodiment. [Figure 4] FIG. 2 is a diagram showing the configuration of an image sensor according to an embodiment. [Diagram 5] FIG. 2 is a diagram showing the configuration of an image sensor according to an embodiment. [Figure 6] 4 is a flowchart showing the operation of the imaging apparatus according to the first embodiment. [Figure 7] FIG. 4 is a diagram showing accumulation time control according to the embodiment. [Figure 8]4 is a flowchart showing the operation of the imaging apparatus according to the first embodiment. [Figure 9] 10 is a flowchart showing the operation of the imaging apparatus according to the second embodiment. [Figure 10] 11 is a flowchart showing the operation of an imaging apparatus according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, descriptions of identical or similar configurations are omitted.

[0011] Example 1 FIG. 1 is a block diagram showing a configuration of an imaging device according to a first embodiment of the present invention.

[0012] In FIG. 1, the photographing optical system 120 includes a first lens 100 , an aperture 101 , a second lens 102 , a third lens 103 , a focal plane shutter 104 , and an optical low-pass filter 105 .

[0013] The aperture 101 adjusts the amount of light incident from the first lens 100 during shooting by adjusting the aperture diameter. The second lens 102 and the third lens 103 are driven by a focus actuator 116 (described later) and move forward and backward in the optical axis direction to adjust the focal position of the shooting optical system 120.

[0014] The focal plane shutter 104 adjusts the exposure time of the image sensor 106 by opening and closing when capturing a still image. The exposure time may be adjusted by an electronic shutter function of the image sensor 106. The optical low pass filter 105 is provided to reduce false colors and moire in the captured image. The image sensor 106 photoelectrically converts an optical image of a subject formed by the photographing optical system 120 and outputs an electrical signal. The image sensor 106 is controlled by a CPU 109, which will be described later.

[0015] A DSP (Digital Signal Processor) 107 performs image processing such as various corrections and HDR composition of image data captured by the imaging element 106. A RAM 108 has a function of temporarily holding output data from the imaging element 106, storing image data processed by the DSP 107, and serving as a work memory when a CPU 109 (described later) operates.

[0016] In this embodiment, these functions are realized using RAM 108, but other types of memory may be used as long as the access speed is fast enough to cause no operational problems. In this embodiment, RAM 108 is disposed outside DSP 107 and CPU 109, but some or all of the functions may be built into DSP 107 and CPU 109.

[0017] The CPU 109 executes a program for controlling each unit of the imaging apparatus and performs overall control of the operation of the imaging apparatus. The CPU 109 also has a function of controlling the focus driving circuit 115 using the result of correlation calculation output from the DSP 107 and adjusting the focal position of the imaging optical system 120.

[0018] The display unit 110 displays captured still images, moving images, menus, etc. The operation unit 111 allows the user to set shooting instructions, shooting conditions, etc., to the CPU 109. The recording medium 112 is a removable recording medium, and records captured still image data and moving image data. The ROM 113 stores programs that the CPU 109 loads and executes to control the operation of each unit.

[0019] The shutter driving circuit 114 drives and controls the focal plane shutter 104 based on the output of the CPU 109. The focus driving circuit 115 controls a focus actuator 116 based on the output of the CPU 109, and drives the second lens 102 and the third lens 103 forward and backward in the optical axis direction to change the focal position of the photographing optical system 120, thereby performing focus adjustment. The iris driving circuit 117 controls an iris actuator 118 based on the output of the CPU 109 to control the opening of the iris 101.

[0020] 2 is a diagram showing an example of a circuit configuration of the image sensor 106. In the pixel group 250, a plurality of pixels 200 are arranged in a matrix such as pixels R1_1 to Bm_n (m and n are arbitrary integers). The symbol R attached to the pixel 200 means that a red color filter, a symbol G means that a green color filter, and a symbol B means that a blue color filter are arranged on the pixel.

[0021] Furthermore, R(G,B)i_j indicates a pixel located in the i-th row and j-th column in the pixel section 250. Furthermore, among the pixels 200 in the same column, pixels in odd-numbered rows are connected to a vertical output line 202, and pixels in even-numbered rows are connected to a vertical output line 206. Note that, although one vertical output line is connected to one pixel section 200 in this embodiment, multiple vertical output lines may be connected to one pixel section 200.

[0022] When the column circuit switch 205 is off, pixel signals from pixels in odd-numbered rows are input only to the column circuit 203 via the vertical output line 202, and when the column circuit switch 205 is on, the pixel signals are input to both the column circuit 203 and the column circuit 207. Similarly, when the column circuit switch 205 is off, pixel signals from pixels in even-numbered rows are input only to the column circuit 207 via the vertical output line 206.

[0023] Furthermore, when the column circuit switch 205 is on, the pixel signals are input to the column circuit 203 and the column circuit 207. When the column circuit switch 205 is off, pixel signals from even and odd rows can be read out simultaneously, and when the column circuit switch 205 is on, pixel signals from even and odd rows are read out sequentially. By turning on the column circuit switch 205, two column circuits 203 and 207 are connected to one vertical output line, and pixel signals output from one pixel unit 200 can be amplified by the column circuits 203 and 207 with a plurality of different gains.

[0024] The drive signals RES, TX, SEL, FDINC, and GS are respectively supplied from a vertical scanning circuit 201 to the pixel unit 200. These drive signals are denoted as RES_m, TX_m, SEL_m, FDINC_m, and GS_m, with a natural number variable m (=1, 2, . . . , n) added to indicate a number corresponding to a row.

[0025] A timing generator (hereinafter, TG) 210 supplies driving signals to the vertical scanning circuit 201, the column circuits 203 and 207, and the memory 211. The memory 211 holds image data that has been analog-to-digital converted in the column circuits 203 and 207, as will be described later, in accordance with a write signal memwr output from the TG 210.

[0026] The image data held in the memory 211 is sequentially transferred to a data output unit 213 by scanning of a horizontal scanning circuit 212. The data output unit 213 outputs the image data to the outside of the image sensor 106 by a transmission method such as LVDS (Low Voltage Differential Signaling).

[0027] Fig. 3 is a diagram showing a configuration of a pixel 200 in an image sensor. The configuration of the pixel 200 will be described with reference to Fig. 3. A photodiode (hereinafter, PD) 300 photoelectrically converts incident light and accumulates charges according to the amount of incident light. When a signal gs1 is at a high level, a transfer switch 301a transfers the charges accumulated in the PD 300 to a first holding unit 310a. When a signal gs2 is at a high level, a transfer switch 301b transfers the charges accumulated in the PD 300 to a second holding unit 310b.

[0028] When the level of the signal tx1 is at a high level, the transfer switch 302a transfers the charge stored in the first holding unit 310a to the floating diffusion unit (hereinafter, FD unit) 303. When the level of the signal tx2 is at a high level, the transfer switch 302b transfers the charge stored in the second holding unit 310b to the FD unit 303.

[0029] The FD unit 303 is connected to the gate of a transistor that constitutes the FD amplifier 304. The FD amplifier 304 outputs a voltage value according to the amount of charge transferred from the PD 301 to the FD unit 303. The FD reset switch 305 is a switch element for resetting the FD unit 303, and resets the FD unit 303 when the level of a signal res is at a high level.

[0030] The FDinc switch 307 is turned on when the level of the signal fdinc is at a high level, and connects the additional capacitance 308 to the FD unit 303. On the other hand, when the level of the signal fdinc is at a low level, the FDinc switch 307 is turned off, and the additional capacitance 308 and the FD unit 303 are not connected to each other. This changes the capacitance generated in the FD unit 303. In other words, the FDinc switch 307 and the additional capacitance 308 make the capacitance value of the input node variable.

[0031] By turning on and off the FDinc switch 307, it is possible to switch the conversion ratio of the output voltage of the FD amplifier 304 to the charge transferred to the FD section 303, that is, the gain (FD gain) which is the signal amplification factor in the pixel. In a connected state where the additional capacitance 308 is connected to the FD section 303, the capacitance value increases and the FD gain becomes smaller compared to the disconnected state. Conversely, when the additional capacitance 308 and the FD section 303 are disconnected, the capacitance value decreases and the FD gain becomes larger compared to the connected state.

[0032] In this way, the imaging device of this embodiment can output pixel signals at a plurality of different gains because it can vary the amount of charge held in the FD unit 303 according to the signal fdinc and switch the FD gain of the pixel unit 200. When the level of the signal sel is at a High level, the pixel selection switch 306 outputs the pixel signal vout converted into a voltage by the FD amplifier 304.

[0033] The PD reset transistor 309 is a switch for resetting the charge accumulated in the PD, and when the level of the signal ofd is at a high level, the charge accumulated in the PD 300 is reset.

[0034] Next, Fig. 4 is a diagram showing the configuration of the column circuit 203 in the image sensor. The configuration of the column circuit 203 will be described with reference to Fig. 4. The basic configuration of the column circuit 207 is the same as that of the column circuit 203, so the description will be omitted here. The output vout of the pixel section 200 is input to the column circuit 203 via a vertical output line 202 connected to the output terminal of the pixel 200. The vertical output line 202 provided for each column is grounded via a current source 204. The current source 204 and the FD amplifier 304 of the pixel section 200 connected to the vertical output line 202 form a source follower circuit.

[0035] In the column circuit 203, the clamp capacitor 401 has a capacitance C1, and the feedback capacitor 402 has a capacitance C2. The output vout of the pixel unit 200 is input to the inverting input terminal of the operational amplifier 403 via the clamp capacitor 401. A reference voltage Vref is supplied to the non-inverting input terminal of the operational amplifier 403. A switch 404 is connected in parallel with the feedback capacitor 402, which is connected to the inverting input terminal and output terminal of the operational amplifier 403.

[0036] The switch 404 is a switch element for shorting both ends of the feedback capacitance 402, and is controlled by a drive signal cfs. The gain setting, which is the signal amplification factor, can be changed by controlling the on / off of the switch 404, making it possible to output signals amplified with a number of different gains (column amplifier gains).

[0037] The S signal transfer switch 405 is a switch element for transferring a photoelectrically converted optical signal (S signal) read out from the pixel unit 200 to an S signal holding capacitance 407, and is on / off controlled by a drive signal ts. By setting the level of the drive signal ts to a High level, the S signal amplified by the operational amplifier 403 is held in the S signal holding capacitance 407 via the S signal transfer switch 405.

[0038] The N signal transfer switch 406 is a switch element for transferring a reset release signal (N signal) read out from the pixel unit 200 immediately after the pixel unit 200 is reset to an N signal holding capacitor 408, and is on / off controlled by a drive signal tn. By setting the level of the drive signal tn to a High level, the N signal amplified by the operational amplifier 403 is held in the N signal holding capacitor 408 via the N signal transfer switch 406. The drive signals cfs, ts, tn, etc. are supplied from the TG 210 under the control of the CPU 109.

[0039] Next, Fig. 5 is a diagram showing the configuration of an analog-to-digital conversion circuit (AD conversion circuit) built into the column circuits 203 and 207 in the image sensor. The AD conversion circuit will be described with reference to Fig. 5. The AD conversion circuit performs analog-to-digital conversion on the difference between the S signal vs and the N signal vn held in the S signal holding capacitance 407 and the N signal holding capacitance 408, respectively.

[0040] The AD conversion circuit includes a comparator 501, a counter circuit 502, a latch circuit 503, and an arithmetic circuit 504. A ramp signal generating unit 506 (RAMP circuit) is a circuit that generates a ramp signal whose signal level changes over time.

[0041] The comparator 501 compares the pixel signal amplified by the column amplifier gain described in FIG. 4 with the output of a ramp signal, and outputs an inverted signal at the timing when the time-varying ramp signal matches the pixel output.

[0042] The counter circuit 502 performs a counting operation based on a clock signal pCNT supplied from a connected control line. That is, the counter circuit 502 starts a counting operation from the timing when the comparator 501 starts comparing the pixel signal with the ramp signal, and outputs a count value when the output of the comparator 501 is inverted.

[0043] In other words, if the time change of the ramp signal is slow, the timing at which the output of the comparator 501 is inverted is also delayed, and the count value increases. Since the speed of the time change of the ramp signal corresponds to the amplification factor of the output signal of the pixel unit 200, the gain (RAMP gain), which is the signal amplification factor in the AD conversion circuit, can be switched by changing the degree of the time change.

[0044] The latch circuit 503 temporarily holds the count value output by the counter circuit 502, and outputs the held count value under the control of a latch control signal pLTC.

[0045] The arithmetic circuit 504 temporarily stores the count value output by the latch circuit 503 as a digital pixel signal under the control of a connected arithmetic control signal pCAL, and outputs the stored digital pixel signal.

[0046] Next, Fig. 6 is a flowchart for explaining the shooting operation of the imaging device in the embodiment 1. The shooting operation of the imaging device in the embodiment 1 will be explained with reference to Fig. 6. The operation of this flowchart is realized by the CPU 109 executing a program stored in the ROM 113.

[0047] In step S600, the CPU 109 controls the shutter drive circuit 114 in accordance with a shooting instruction from the operation unit 111, exposes the image sensor 106, and starts shooting.

[0048] In step S601, charge accumulation in the PD 300 of the image sensor 106 starts.

[0049] In step S602, the charge amount M stored in the PD 300 is transferred to the first holding unit 310a. This charge amount M is the amount of charge transferred in one transfer.

[0050] In step S603, the charge amount N stored in the PD 300 is transferred to the second holding unit 310b. This charge amount N is the amount of charge transferred in one transfer.

[0051] In step S604, it is determined whether or not the transfer of all the charges accumulated in the PD 300 during one frame period to the first holding unit 310a or the second holding unit 310b is completed. If the transfer is completed, the process proceeds to step S605, and if not, the process returns to step S602.

[0052] In step S605, the charge stored in the first holding unit 310a is transferred to the FD unit 303.

[0053] In step S606, a signal based on the charge stored in the first holding unit 310a is amplified by a first gain and a second gain and read out. In this embodiment, the first gain is described as a high gain and the second gain is described as a low gain, but the first gain may be a low gain and the second gain may be a high gain.

[0054] That is, a high gain image and a low gain image can be read out by turning on the column circuit switch 205 of the image sensor 106 and amplifying a pixel signal output from one pixel unit 200 with different gains in the column circuits 203 and 207. Note that, as described above, a high gain image and a low gain image may be read out by switching the RAMP gain in the AD conversion circuit.

[0055] Furthermore, it is possible to switch the FD gain described above to read out a high-gain image and a low-gain image. That is, charges may be accumulated in the first holding unit 310a and read out while the FDinc switch 307 is turned on, and charges may be accumulated in the first holding unit 310a and read out while the FDinc switch 307 is turned off.

[0056] In step S607, the charge stored in the second holding unit 310b is transferred to the FD unit 303.

[0057] In step S608, a signal based on the charge stored in the second holding unit 310b is amplified with a high gain and a low gain and read out.

[0058] In step S609, an HDR image is obtained from the four images read out in steps S606 and S608, as described below.

[0059] In step S610, it is determined whether or not shooting is to continue. If HDR shooting is not to be performed, the process ends. If shooting is to be continued, the process returns to step S601.

[0060] FIG. 7 is a diagram showing the state of charge storage in each of the first holding unit 310a and the second holding unit 310b during one frame period.

[0061] First, FIG. 7(a) shows an example in which, during one frame period, first, the transfer switch 301a is turned on to transfer charge of amount M to the first holding unit 310a, and then the transfer switch 301b is turned on to transfer charge of amount N to the second holding unit 310b.

[0062] For example, images with different exposures (amounts of accumulated charge) can be acquired by controlling so that the amount of charge M held in the first holding unit 310a is smaller than the amount of charge N held in the second holding unit 310b. Note that the transfer switch 301a is controlled to be turned off after the charge transfer to the first holding unit 310a is completed, and the transfer switch 301b is controlled to be turned off after the charge transfer to the second holding unit 310b is completed.

[0063] 7(b) shows an example in which the start and end timings of charge transfer to the first holding unit 310a and the second holding unit 310b are the same in one frame period. That is, the transfer switch 301a and the transfer switch 301b are simultaneously turned on to start the charge transfer to the first holding unit 310a and the second holding unit 310b in one frame period. After that, the charges are transferred to the first holding unit 310a and the second holding unit 310b in multiple divided times.

[0064] Then, at the end of one frame period, transfer switch 301a and transfer switch 301b are simultaneously turned off, thereby ending the charge transfer to first holding unit 310a and second holding unit 310b during one frame period. By such control, the periods during which charge is held in first holding unit 310a and second holding unit 310b during one frame period (the periods from the start of charge holding to the end of charge holding) can be made to coincide, making it possible to obtain an image with less motion blur of the subject.

[0065] 7(c) shows an example in which the exposure center of the charge held in the first holding unit 310a and the exposure center of the charge held in the second holding unit 310b are made equal. Here too, in one frame period, the charge is transferred in multiple parts to the first holding unit 310a and the second holding unit 310b. Although the start and end timings of the charge transfer to the first holding unit 310a and the second holding unit 310b are different, by aligning the exposure centers of the accumulated charges, it is possible to obtain an image with less motion blur of the subject even when there is a large accumulation time difference.

[0066] 7(b) and 7(c), images with different exposures (amounts of accumulated charge) can be obtained by controlling the amount of charge M held in the first holding unit 310a to be less than the amount of charge N held in the second holding unit 310b. In this way, images with different exposures can be obtained by accumulating different amounts of charge in the first holding unit 310a and the second holding unit 310b.

[0067] 8 is a flowchart for explaining the operation of acquiring an HDR image by performing HDR blending processing by inputting four images with different exposures to the DSP 107. The operation of this flowchart is realized by the CPU 109 executing a program stored in the ROM 113.

[0068] In step S606a, a first image stored in the first storage unit 310a and amplified with a low gain is read out, and in step S606b, a second image stored in the first storage unit 310a and amplified with a high gain is read out.

[0069] In step S701, the DSP 107 performs HDR synthesis processing on the first image and the second image. Here, the synthesized image is defined as a fifth image. At this time, since the accumulation times of the first image and the second image are the same, the gain difference when amplifying with high gain and low gain contributes to the expansion of the dynamic range after synthesis. However, as described above, when HDR synthesis processing is performed using only the gain difference, the dynamic range expansion effect may be limited compared to HDR synthesis processing in which multiple images are taken with different exposures.

[0070] In step S608a, the third image stored in the second storage unit 310b and amplified with a high gain is read out, and in step S608b, the fourth image stored in the second storage unit and amplified with a low gain is read out.

[0071] In step S702, the DSP 107 performs HDR synthesis processing of the third image and the fourth image. Here, the synthesized image is set as a sixth image. At this time, since the accumulation times of the third image and the fourth image are the same, the gain difference when amplifying with high gain and low gain contributes to the expansion of the dynamic range after synthesis.

[0072] In step S703, the DSP 107 performs HDR synthesis processing on the fifth image and the sixth image. As described above, the fifth image is an image obtained by performing HDR synthesis processing on an image signal based on the charge accumulated in the first holding unit 310a, and the sixth image is an image obtained by performing HDR synthesis processing on an image signal based on the charge accumulated in the second holding unit. In other words, these two images have different accumulation times. Therefore, by performing HDR synthesis processing on the fifth image and the sixth image, it is possible to obtain an HDR image having a gain difference and an accumulation time difference in one frame period.

[0073] As described above, according to the imaging device of the first embodiment, when performing HDR shooting, the electric charge generated in the photoelectric conversion unit during one frame period is divided and transferred to the first holding unit 310a and the second holding unit 310b, so that images with different accumulation times can be acquired. Furthermore, by amplifying the images with different accumulation times with different gains, it is possible to perform HDR synthesis processing based on four images with different accumulation times and gains. Therefore, it is possible to generate an image with a sufficiently expanded dynamic range while suppressing blurring of the subject.

[0074] Example 2 9 is a flowchart for explaining the shooting operation of the imaging device in the second embodiment of the present invention. The shooting operation of the imaging device in the second embodiment will be explained with reference to FIG. 9, focusing on the difference from the first embodiment. The operation of this flowchart is realized by the CPU 109 executing a program stored in the ROM 113.

[0075] Steps S900 to S910 are the same as steps S600 to S610 described with reference to FIG. 6 in the first embodiment, and therefore a description thereof will be omitted.

[0076] In step S911, it is determined whether or not it is necessary to expand the dynamic range of the signal based on the charge transferred in step S905 from the first holding unit 310a to the FD unit 303. For example, if the amount of charge transferred from the first holding unit 310a to the FD unit 303 is sufficiently large and the range on the high luminance side of the subject is secured, the process proceeds to step S912.

[0077] In step S912, the signal based on the charge held in the first holding unit 310a is amplified and read out using only the first gain (high gain) or the second gain (low gain). As described above, if the high luminance range has been secured, there is no need to widen the high luminance range, so the signal is amplified and read out using the second gain (low gain). This makes it possible to shorten the time it takes to read out the image signal from the image sensor while still securing the high luminance range.

[0078] In addition, the control may be changed not only according to the subject brightness but also according to the shooting mode. For example, when it is desired to acquire an HDR image at high speed, in order to shorten the processing time, the signal based on the charge of the first holding unit 310a may be amplified and read out by only one gain, and the signal based on the charge of the second holding unit 310b may be amplified and read out by both gains.

[0079] In this embodiment, an example has been described in which the signal based on the charges held in the first holding unit is amplified and read out by only one gain, and the signal based on the charges held in the second holding unit is amplified and read out by both a high gain and a low gain, but the present invention is not limited to this. That is, the signal based on the charges held in the first holding unit may be amplified and read out by both a high gain and a low gain, and the signal based on the charges held in the second holding unit may be amplified and read out by only one gain.

[0080] According to the second embodiment described above, in HDR imaging, a signal based on the charge in the first storage unit or the charge in the second storage unit is amplified and read out using only one of the gains depending on the subject and imaging mode, thereby making it possible to shorten the readout time of the image signal. Therefore, it is possible to obtain an HDR image with a sufficiently expanded dynamic range while shortening the readout time of the image signal.

[0081] Example 3 10 is a flowchart for explaining an operation of acquiring an HDR image in the third embodiment of the present invention. The operation in the third embodiment will be explained with reference to FIG. 10, focusing on the difference from the first embodiment. The operation of this flowchart is realized by the CPU 109 executing a program stored in the ROM 113.

[0082] Steps S606a, S606b, S608a, and S608b are similar to steps S606a, S606b, S608a, and S608b described in Fig. 8 of the first embodiment, and therefore their description will be omitted. Also, steps S1001 to S1003 are similar to steps S701 to S703 described in Fig. 8 of the first embodiment, and therefore their description will be omitted.

[0083] In step S1010, the first image and the second image are compared to determine whether or not the dynamic range needs to be expanded. For example, since the first image and the second image are images accumulated over a short period of time, they may be used to expand the range on the high luminance side, but if the range on the high luminance side can be secured only with the image amplified with a low gain, the process proceeds to step S1011.

[0084] In step S1011, only the first image is output without performing HDR blending processing. Here, the second image not used in the HDR blending processing may be used for a predetermined detection related to the subject (for example, vector detection of the subject movement direction, etc.).

[0085] In step S1012, the third image and the fourth image are compared to determine whether or not the dynamic range needs to be expanded. For example, since the third image and the fourth image are images obtained during long-time accumulation, they may be used to expand the range on the low-luminance side. However, if the range on the low-luminance side can be secured only by the image amplified with high gain, the process proceeds to step S1013.

[0086] In step S1013, only the fourth image is output without performing HDR blending processing. Here, the third image not used in the HDR blending processing may be used for a predetermined detection related to the subject (for example, vector detection of the subject movement direction, etc.).

[0087] In step S1014, HDR blending processing of the first image and the fourth image is performed by the DSP 107. In Fig. 8 of the first embodiment, the HDR blending processing is performed three times, but the method of this embodiment makes it possible to acquire an image with an expanded dynamic range while reducing the number of times the HDR blending processing is performed.

[0088] In addition, the control may be changed not only according to the subject brightness but also according to the shooting mode. For example, when it is desired to obtain an HDR image at high speed, the number of times of HDR synthesis processing or the synthesis pattern may be changed as described above in order to shorten the processing time, thereby making it possible to obtain an image with an expanded dynamic range. Note that, in this embodiment, an example of performing HDR synthesis processing on the first image and the fourth image has been described, but the present invention is not limited to this.

[0089] According to the third embodiment described above, in HDR photography, it is possible to acquire an HDR image with a sufficiently expanded dynamic range while shortening the processing time by changing the number of times and the pattern of HDR synthesis processing depending on the subject and the photography mode.

[0090] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0091] 106 Image sensor 107 DSP 109 CPU 203, 207 column circuit 205 column circuit switch 301a First holding part 301b Second holding part 308 Additional Capacity

Claims

1. a plurality of pixels each having a photoelectric conversion unit, a first charge holding unit that holds the charge converted by the photoelectric conversion unit, and a second charge holding unit that accumulates the charge converted by the photoelectric conversion unit; an amplifier capable of amplifying a signal based on the charge at a plurality of different gains; an image processing unit that performs HDR synthesis processing of a plurality of signals amplified by the amplifier unit at different gains, an amount of charge stored in the first charge retention unit and an amount of charge stored in the second charge retention unit during one frame period differ from each other, and at least one of signals based on the charges stored in the first charge retention unit and the second charge retention unit is amplified with a different gain by the amplifier means.

2. 2. The image pickup device according to claim 1, wherein the first charge holding portion and the second charge holding portion start charge accumulation at different timings.

3. 2. The image pickup device according to claim 1, wherein the first charge holding portion and the second charge holding portion start holding electric charges at the same timing.

4. 2. The imaging device according to claim 1, wherein one of a signal based on the charges stored in the first charge storage portion and a signal based on the charges stored in the second charge storage portion is amplified by the amplifier with a different gain.

5. 2. The imaging device according to claim 1, wherein both a signal based on the charges stored in the first charge storage portion and a signal based on the charges stored in the second charge storage portion are amplified by the amplifier with different gains.

6. 2. The imaging device according to claim 1, wherein the number of times or combination of HDR synthesis processes performed by the image processing means is changed depending on the subject brightness or the shooting mode.

7. The imaging apparatus according to claim 1 , wherein a signal that is not used in the HDR synthesis process by the image processing means is used for a predetermined detection related to a subject.

Images