Imaging device, computer program, and storage medium

Abstract

To improve linearity characteristics in composition processing for expanding a dynamic range.SOLUTION: An imaging apparatus has: an AD conversion unit that performs AD conversion of an input signal through comparison between a first lamp signal and a second lamp signal different in inclination from the first lamp signal; a first composition unit that composes a signal analog digital converted with the first lamp signal and a signal analog digital converted with the second lamp signal with each other; a first correction unit that corrects a step during the composition in the first composition unit; a reading unit that amplifies pixel signals with different amplification factors and outputs the pixel signals; a second composition unit that composes the signals with each other which are amplified with the different amplification factors and read out from a read-out circuit; and a second correction unit for correcting a step during the composition in the second composition unit.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present invention relates to an imaging device suitable for a digital camera, etc. [Background technology]

[0002] In recent years, television standards have changed from Full HD, with a pixel count of 1920 pixels horizontally and 1080 pixels vertically (hereafter referred to as 1920 x 1080), to 4K2K, with a pixel count of 3840 x 2160 pixels, four times that of HD. There is also a standard called 8K4K or Super Hi-Vision, which is the next generation after 4K2K and has a pixel count of 7680 x 4320 pixels. In addition to the increase in pixel count, the frame rate is also steadily increasing.

[0003] Along with these changes, imaging devices for capturing television images are also required to record images with higher pixel counts and higher frame rates. In order to satisfy the above demands, one of the important issues is how to speed up the readout time of image signals even in image sensors that convert light into electrical signals, and there is a particular demand for faster AD converters within the image sensors.

[0004] In addition to increasing the frame rate, expanding the dynamic range is also an important factor for image expression. In Patent Document 1, a comparison unit that compares an analog signal with a predetermined voltage is provided, and a reference voltage with a different gradation precision is selected, thereby shortening the time required for AD conversion without reducing the number of gradations.

[0005] Patent Document 2 proposes a technology in which signals are read out from one pixel column using multiple vertical signal lines and the same pixel is read out at multiple different gains. With a configuration like that of Patent Document 2, it is possible to create an image signal with a high dynamic range and low noise by outputting image signals of a low gain signal and a high gain signal and synthesizing them in a subsequent image processing stage. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2013-251677 A [Patent Document 2] JP 2015-128253 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the dynamic range expansion by image processing, if the linearity characteristics of the two image signals to be synthesized are poor, the synthesis process cannot be performed correctly. Also, in the control of Patent Document 1, if the linearity characteristics deteriorate due to switching of the reference voltage, this will have an adverse effect on the dynamic range expansion by synthesis. An object of the present invention is to provide an image pickup apparatus capable of image synthesis processing with good linearity characteristics. [Means for solving the problem]

[0008] an AD conversion unit that performs AD conversion on an input signal by comparing a first ramp signal with a second ramp signal having a slope different from that of the first ramp signal; The first ramp signal AD The converted signal and the second ramp signal AD a first synthesis unit that synthesizes the converted signals; a first correction unit that corrects a step occurring during synthesis in the first synthesis unit; a readout unit that amplifies pixel signals at different amplification rates and outputs the amplified pixel signals; a second combining unit that combines the signals amplified at different amplification factors read from the readout circuit; A second correction unit for correcting a step occurring during synthesis in the second synthesis unit. death, The second correction unit corrects a step at the time of synthesis in the second synthesis unit based on an AD conversion value using either the first ramp signal or the second ramp signal by the AD conversion unit. The present invention is characterized by the above. Effect of the Invention

[0009] According to the present invention, it is possible to obtain an imaging device capable of performing image synthesis processing with good linearity characteristics. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram of an imaging apparatus according to a first embodiment. [Diagram 2] 4 is a diagram illustrating a typical configuration and operation timing of a comparison unit in the image sensor of the first embodiment. FIG. [Diagram 3] FIG. 4 is a diagram illustrating an operation of AD conversion using multiple slopes in the first embodiment. [Figure 4] 10A and 10B are diagrams relating to correction of AD conversion results using multiple slopes in the first embodiment. [Diagram 5] 1 is a schematic front view of a pixel section 110 according to a first embodiment. [Figure 6] FIG. 4 is a diagram illustrating an operation of AD conversion of a dummy pixel according to the first embodiment. [Figure 7] 1 is a block diagram of a portion of an image sensor according to a first embodiment. [Figure 8] 2 is a block diagram showing a configuration inside a column amplifier according to a first embodiment. FIG. [Figure 9] FIG. 4 is a diagram showing pixel signal readout drive timing in the first embodiment. [Figure 10] FIG. 4 is an explanatory diagram of linearity in dynamic range expansion synthesis in the first embodiment. [Figure 11] 4 is a flowchart showing a control flow of the imaging apparatus of the first embodiment. [Figure 12] 13A to 13C are diagrams illustrating AD conversion of dummy pixels outputted at different amplification factors in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. In each drawing, the same members or elements are designated by the same reference numerals, and duplicated descriptions will be omitted or simplified. In the embodiment, the imaging device is a digital still camera, but the imaging device may include electronic devices having an imaging function, such as digital movie cameras, smartphones with cameras, tablet computers with cameras, vehicle-mounted cameras, and network cameras. <Example 1>

[0012] A first embodiment of the present invention will be described below. FIG. 1 is a block diagram of an image pickup device 1000 according to a first embodiment, which includes an image pickup element (CMOS image sensor) 1 equipped with a parallel AD converter and an image processing LSI 2. The image processing LSI 2 performs processes on the image data output by the image sensor 1, such as white balance processing, gamma processing, HDR synthesis processing (described later), and processing to correct the ratio of two pixel signals with different gains, and finally records the image data on a recording medium.

[0013] The image processing LSI 2 also has a built-in CPU as a computer, and communicates (eg, serial communication) with the image sensor 1 and the like based on a computer program stored in the memory, thereby controlling the entire image pickup apparatus 1000. The imaging element 1 includes a pixel section 110, a vertical scanning circuit 120, a column amplifier (column AMP) 130, a ramp circuit 140, a column ADC (Analog To Digital Converter) 150, a horizontal transfer circuit 160, a signal processing circuit 170, an external output circuit 180, a controller circuit 300, and the like.

[0014] The controller circuit 300 is an I / F section with the image processing LSI 2, and receives control signals from the CPU of the image processing LSI 2 to the image sensor 1 using serial communication means or the like, and controls each circuit within the image sensor 1. A plurality of pixels, each having a photoelectric conversion element (photodiode) that performs photoelectric conversion according to the amount of incident light and outputs a voltage signal, are arranged two-dimensionally in the pixel section 110. A color filter and a microlens are mounted on the surface of each photoelectric conversion element.

[0015] Each photoelectric conversion element has one of three color filters of R (red), G (green), or B (blue) periodically arranged, and the pixel section 110 as a whole has a Bayer array, but this is not necessarily limited to this. A timing control section 100 supplies an operating clock CLK or a timing signal to each block of this image sensor, and controls the operation of each block. The vertical scanning circuit 120 performs timing control to sequentially read out output signals from each pixel arranged two-dimensionally in the pixel section 110 row by row during one frame. That is, image signals are sequentially read out row by row from the top row to the bottom row in the pixel section 110 during one frame.

[0016] The column amplifier 130 electrically amplifies the image signal read out row by row from the pixel unit 110 for each column, and includes an operational amplifier as described below. By amplifying the image signal with the column amplifier 130, the S / N ratio is equivalently improved with respect to the noise generated by the column ADC 150 provided for each subsequent column.

[0017] 1, for example, output signals of pixels in odd-numbered columns of the pixel unit 110 are amplified and read out by a column amplifier 130 above the pixel unit 110, and signals of pixels in even-numbered columns are amplified and read out by a column amplifier 130 below the pixel unit 110. In addition, the gain of the column amplifier 130 can be changed by a control signal from the timing control unit 100.

[0018] The constant voltage circuit 400 can supply a predetermined voltage to each of the signal output lines before inputting the output signal of each pixel to the column amplifier 130. Note that instead of the constant voltage circuit 400, a clipping circuit that clips the signal at a predetermined constant voltage may be used. The ramp circuit 140 is a signal generator for generating a ramp-shaped voltage signal (ramp signal) whose voltage changes with the passage of time at a constant slope.

[0019] The column ADC 150 has a comparison unit 151 that compares the signal amplified by the column amplifier 130 with a ramp signal supplied from the ramp circuit 140. Also, the column ADC 150 has a counter latch circuit 152 for each column that counts the time until the voltage level of the signal amplified by the column amplifier 130 matches the voltage level of the ramp signal by the comparison unit 151 and holds the count value. A detailed operation will be described later. In addition, as described below, the column ADC 150 performs AD conversion on the first image signal and the second image signal amplified with different gains in the column amplifier 130, and the first noise signal and the second noise signal amplified with different gains in the column amplifier 130.

[0020] The count values ​​of each column for one row held in the counter latch circuit 152 are sequentially read out by the horizontal transfer circuit 160 from the rightmost column as AD converted image data. The image data output from the horizontal transfer circuit 160 is input to a signal processing circuit 170. The signal processing circuit 170 is a circuit that performs signal processing digitally, and can perform digital gain calculations by adding a certain amount of offset value through digital processing, or by performing shift calculations and multiplications.

[0021] Moreover, by providing a light-shielded pixel region (optical black) in the pixel section 110, a digital black level clamp operation may be performed using the output signal thereof. The image data output from the signal processing circuit 170 is supplied to the external output circuit 180. The external output circuit 180 has a serializer function and converts the multi-bit parallel signal input from the signal processing circuit 170 into a serial signal. The external output circuit 180 also converts this serial signal into, for example, a signal conforming to the LVDS (Low Voltage Differential Signaling) standard, and outputs it to an external device (in this embodiment, the image processing LSI 2) as image data.

[0022] Next, AD conversion using the column ADC 150 of the image sensor 1 will be described with reference to Fig. 2(A). Fig. 2 is a diagram showing a typical configuration and operation timing of a comparison unit in the image sensor of the first embodiment. The column ADC 150 has therein a comparison unit 151 and a counter / latch circuit 152. The comparison unit 151 compares the signal VAMP amplified by the column amplifier 130 with the ramp signal VRAMP supplied from the ramp circuit 140, and outputs the comparison result.

[0023] The counter latch circuit 152 performs a counting operation from when the count value is reset until the voltage level of the signal amplified by the column amplifier 130 matches the voltage level of the ramp signal and the comparison output of the comparator 151 is inverted. This operation provides a count value proportional to the signal level from the column amplifier 130, and this count value becomes the AD conversion result. FIG. 2B is a diagram for explaining a case where the slope of the ramp signal VRAMP, which is the output signal of the ramp circuit 140, is changed.

[0024] By changing the slope of the voltage change over time, the timing at which the comparison output of the comparator is inverted changes, and the count value changes according to the count time. This makes it possible to change the digital gain of the AD converter. For example, making the slope of the voltage change gentler increases the count value. In other words, the AD conversion gain increases. Here, the column ADC 150 functions as an AD conversion unit that performs AD conversion of an input signal by comparing a first ramp signal VRAMP1 with a second ramp signal VRAMP2 having a different slope from the first ramp signal.

[0025] 3 is a diagram showing the operation of AD conversion using multiple ramp signals with different slopes according to the first embodiment, and shows the operation timing of the column ADC 150 when the horizontal axis represents time. Using this diagram, an example of changing the slope of the ramp signal VRAMP from the ramp circuit 140 according to the voltage level of the output signal of the column amplifier 130 will be described.

[0026] In this embodiment, when AD converting the signal of each pixel, first, an N signal (noise) is read from each pixel and AD converted, and then an S signal (noise + photoelectric conversion signal) is read from each pixel and AD converted. For these two digital signals, a subtraction process is performed in the signal processing circuit 170 to subtract the N signal from the S signal, and the noise component is cancelled out to produce a signal with a good S / N ratio.

[0027] First, during the AD conversion period of the N signal, the N signal accumulated in each pixel of the pixel unit 110 is read out and amplified by the column amplifier 130. The ramp circuit 140 outputs a first ramp signal VRAMP1 with a relatively gentle slope to the comparison unit 151, and the counter latch circuit 152 performs a counting operation based on the comparison result between the first ramp signal VRAMP1 and the N signal. Since the N signal has a smaller voltage level than the S signal, it is AD converted using the first ramp signal with a relatively gentle slope. In other words, the AD conversion gain is increased.

[0028] Next, during a level determination period, the S signal accumulated in each pixel of the pixel section 110 is read out and amplified by the column amplifier 130. The ramp circuit 140 outputs a predetermined determination level Vth to the comparator 151, which compares the determination level Vth with the S signal. The counter / latch circuit 152 receives the result and determines whether the S signal is greater or smaller than the determination level Vth.

[0029] If the S signal is smaller than the judgment level Vth (Sout1 in FIG. 3), the luminance of the subject is judged to be appropriate (normal). Then, the ramp circuit 140 outputs a first ramp signal VRAMP1, which is the same as the N signal, to the comparison unit 151, and performs AD conversion by performing a counting operation based on the comparison result of the first ramp signal VRAMP1 and the S signal in the counter latch circuit 152. That is, the AD conversion gain is kept high. If the S signal is larger than the judgment level Vth (Sout2 in FIG. 3), the luminance of the subject is judged to be high. Then, the ramp circuit 140 outputs a second ramp signal VRAMP2, which has a slope n times larger than that of the first ramp signal, to the comparison unit 151, and performs AD conversion by performing a comparison and counting operation of the second ramp signal VRAMP2 and the S signal in the counter latch circuit 152. That is, the AD conversion gain is reduced.

[0030] In FIG. 4, the horizontal axis indicates the amount of light incident on the pixel unit 110 or the output of the column amplifier 130, and the vertical axis indicates the digital value after AD conversion of the S signal. The signal processing circuit 170 (first combining unit) combines the signal obtained by analog-to-digital conversion using the first ramp signal and the signal obtained by analog-to-digital conversion using the second ramp signal using the method shown in FIG.

[0031] The solid lines in the figure represent digital values ​​that are AD converted by the comparator 151 and counter / latch circuit 152, passed through the horizontal transfer circuit 160, and input to the signal processing circuit 170. Because signals smaller than the judgment level Vth of the S signal are AD converted by the first ramp signal VRAMP1, and signals larger than the judgment level Vth are AD converted by the second ramp signal VRAMP2, a step occurs in the AD converted value relative to the amount of light before and after the judgment level Vth.

[0032] Therefore, in this embodiment, for example, the signal processing circuit 170 multiplies the S signal that is larger than the judgment level Vth by the ratio n of the slopes of the first ramp signal and the second ramp signal. Furthermore, by adding a predetermined offset so that no step occurs at the judgment level Vth, the step or the like is corrected so that the amount of incident light and the AD conversion value are in a straight line. That is, in this embodiment, A step correction during synthesis is performed with high accuracy by performing gain correction and offset correction on the signal obtained by AD conversion using the first ramp signal or the second ramp signal.

[0033] If an image signal of an effective pixel is output without the correction as in this embodiment, a step may occur in the image signal level at a certain brightness, resulting in an unnatural image; however, this embodiment can suppress such a problem. This correction value changes depending on the temperature of the image sensor, the drive timing of the image sensor (such as the gain and operating state of the column amplifier 130) or the drive settings (such as the power supply settings), so in this embodiment, the correction value is acquired, for example, periodically. Next, an example of a method for calculating a correction value such as an offset value to be added to the slope ratio n in this embodiment will be described.

[0034] Fig. 5 shows a schematic front view of the pixel unit 110. As a pixel configuration, a dummy pixel area (correction value output area) 1101 is arranged at the top of Fig. 5, an optical black area 1102 is arranged below that, and an effective pixel area 1103 for acquiring an actual image signal is arranged further below that. In this embodiment, the dummy pixel area 1101 is used to calculate the slope ratio n and the offset amount.

[0035] Compared to normal pixels, the pixels in the dummy pixel region 1101 have a structure without a photodiode, and the output terminals of the dummy pixels are connected to the inputs of the column amplifier 130. However, during a period in which the output signals of the dummy pixels are read out, a fixed voltage is input from the constant voltage circuit 400 to the column amplifier 130. Therefore, the voltage input from the column amplifier 130 to the comparison unit 151 is fixed to a predetermined voltage (voltage V1 or voltage V2 in this embodiment). Next, AD conversion when reading out the output signals of the dummy pixel regions will be described with reference to Fig. 6. Fig. 6 is a diagram showing the operation of AD conversion of the output signals of the dummy pixels in the first embodiment.

[0036] 6A and 6B, when the output signal of the dummy pixel region 1101 is read out, a voltage V1 is input to the column amplifier 130 and is AD converted. Unlike in FIG. 3, there is no need to provide an AD conversion period for the N signal. In Fig. 6(A), the VRAMP level during the level determination period is set to a maximum value, which is a predetermined fixed value, so that the input signal level is smaller than the determination level. As a result, the voltage V1 is AD converted using the first ramp signal VRAMP1, which always has a small slope (i.e., a large AD conversion gain). The result of the AD conversion is V1H.

[0037] Next, in Fig. 6B, the input signal level is made higher than the judgment level by setting VRAMP during the level judgment period to a predetermined fixed minimum value. As a result, the voltage V1 is AD converted with the second ramp signal VAMP2, which always has a steeper slope (i.e., a smaller AD conversion gain). The result of the AD conversion is V1L. In Figures 6(C) and 6(D), the fixed voltage is changed to voltage V2, and AD conversion is performed in the same manner as in Figures 6(A) and 6(B). The results are V2H and V2L.

[0038] These are shown in Fig. 4(B) with the horizontal axis representing the amount of incident light and the vertical axis representing the AD conversion value. Fig. 4(B) is an enlarged view of the portion of Fig. 4(A) that is smaller than the decision level Vth. In this way, fixed voltages V1 and V2 are input during the output signal readout period of the dummy pixel region 1101, and AD conversion values ​​are obtained by the first ramp signal and the second ramp signal for each fixed voltage.

[0039] Based on the coordinates of these four points (V1L, V1H, V2L, V2H) and the actual result of AD conversion by the first ramp signal VAMP1 and the actual result of AD conversion by the second ramp signal VAMP2, the slope ratio n of the AD conversion results is calculated. Furthermore, the actual step (offset value) at Vth is calculated. For example, the slope ratio n can be calculated using the following equation (1). n=(V2H-V1H) / (V2L-V1L)...(1)

[0040] After determining the slope ratio n, the value that the line connecting V2L and V1L takes at the judgment level Vth is calculated to obtain VL1, and VL1 is multiplied by n using the ratio n to obtain nVL1. On the other hand, the value that the line connecting V2H and V1H takes at the judgment level Vth is calculated to obtain VH1, α = VH1 − nVL1 (2) The offset value α that satisfies the above formula (2) can be obtained.

[0041] In addition, in Figures 6 and 4(B), only two voltages, V1 and V2, were used when reading out the output signals of the dummy pixels, so in Figure 4(B), the line connecting V2L and V1L and the line connecting V2H and V1H are both straight lines. Therefore, in the above formula (2), α = 0, but if three or more voltages are used when reading out the output signals of the dummy pixels, the line will not necessarily be a straight line, and in that case, an offset value α other than 0 will be obtained. Using the offset value α allows for more accurate correction.

[0042] The above calculations may be performed by a circuit inside the image sensor, or may be performed by two image processing LSIs. Moreover, the circuit that performs the above calculation functions as a first correction section that corrects a step that occurs during synthesis in the first synthesis section.

[0043] Fig. 7 is a block diagram of a portion of the image sensor of the first embodiment, and will be used to explain signal processing from the pixel unit 110 to the horizontal transfer circuit 160. The photoelectric conversion element (photodiode) 201 receives light through a microlens (not shown) and converts the light into an electric charge by photoelectric conversion. The transfer switch 202 transfers the electric charge generated by the photoelectric conversion element 201 to the electric charge holding unit 203 at the subsequent stage.

[0044] The charge holding unit 203 temporarily holds the charge generated in the photoelectric conversion element 201. The reset MOS transistor 200 resets the charge in the charge holding unit 203 etc., the source of the reset MOS transistor 200 is connected to the charge holding unit 203, and the drain of the reset MOS transistor 200 is connected to a power supply voltage Vcc.

[0045] The pixel amplifier 204 amplifies the signal of the charge holding unit 203 and outputs the signal to the downstream column amplifier 130 via a signal output line 205. A constant current source 206 supplies a constant current to the signal output line 205. The signal amplified by the column amplifier 130 is output to the horizontal transfer circuit 160 via the column ADC 150 .

[0046] FIG. 8 is a block diagram showing the internal configuration of the column amplifier 130, and a method of reading out the output signal of each pixel for expanding the dynamic range will be described with reference to FIG. In this embodiment, the dynamic range is expanded by synthesizing two image signals with different gains in the subsequent signal processing, such as using a high-gain image signal for an image of a low-luminance portion and a low-gain image signal for an image of a high-luminance portion. This will be described later with reference to FIG. 10.

[0047] The column amplifier 130 is configured as an inverting amplifier circuit consisting of an operational amplifier 306, an input-side capacitor 301, and feedback capacitors 302 and 303. Also, the gain can be switched by switches 304 and 305. First, when reading out a high-gain pixel signal for an image with proper exposure (low brightness area), for example, the switch 304 is turned OFF and the switch 305 is turned ON, thereby reading out a pixel signal with a high gain GainH.

[0048] Next, when reading out a low-gain pixel signal for an image of a high-luminance portion, a pixel signal with a low gain GainL is read out by turning on both switches 304 and 305. In this way, by switching the connection of the capacitor with both switches, it becomes possible to switch the feedback capacitance of the operational amplifier 306 and read out pixel signals with different gains (GainH, GainL). Here, the column amplifier 130 (readout unit) amplifies the pixel signals with different amplification factors and outputs the amplified signals.

[0049] FIG. 9 is a diagram showing pixel signal readout drive timing according to the first embodiment of the present invention, and a general method for reading out pixel signals will be described with reference to FIG. First, the switch 304 is turned OFF and the switch 305 is turned ON to set the amplification factor of the column amplifier 130 to the high gain side (time t1).

[0050] At this time, the charge in the pixel is reset by turning on the reset MOS transistor 200. In this state (before turning on the transfer switch 202 of the photoelectric conversion element), the noise component (N signal) of the pixel is read out and amplified by the column amplifier 130 with a high gain GainH, and further AD converted to obtain the N1 signal with the high gain GainH (time t2).

[0051] At time t3, the transfer switch 202 of the photoelectric conversion element is turned ON to read out the image signal component (S signal). After that, the read S signal is amplified by the column amplifier 130 at a high gain GainH, and further AD converted to output as an S1 signal with a high gain GainH (time t4). Next, at time t5, the switch 304 is turned ON to set the amplification factor of the column amplifier 130 to the low gain side. After that, the read S signal is amplified by the column amplifier 130 at a low gain GainL, and further AD converted to obtain an S2 signal with a low gain GainL (time t6).

[0052] After that, at time t7, the charge in the pixel is reset again by the reset MOS transistor 200, the N signal is read out, amplified by the low gain GainL, and AD converted to obtain the N2 signal of the low gain GainL (time t8). Note that in the subsequent circuit, the above S1 signal and N1 signal are used to perform subtraction processing of S1-N1, and the above S2 signal and N2 signal are used to perform subtraction processing of S2-N2. In this way, by changing the gain of the column amplifier 130 when reading out the image signals, it is possible to obtain image signals with two types of gain (GainH, GainL) for use in dynamic range expansion synthesis while maintaining the synchronicity of the image signals.

[0053] Next, the dynamic range expansion synthesis performed in, for example, the image processing LSI2 (second synthesis unit) will be described with reference to Fig. 10. Fig. 10 is an explanatory diagram of linearity in the dynamic range expansion synthesis in the first embodiment. Here, the image processing LSI 2 combines the image signals that have been read out from the readout circuit and amplified with different amplification factors.

[0054] In the first embodiment, an image with a normal luminance level is amplified by an image signal with a high gain GainH by the column amplifier 130, and an image signal with a low gain GainL is amplified by the column amplifier 130 for a bright, blown-out high luminance portion, and both image signals are combined and synthesized. In the synthesis process, alpha blending (weighted addition) or the like is used to smoothly connect the images so that the switching portions are not noticeable.

[0055] In this embodiment, before performing the synthesis process, the luminance of the image with the low gain GainL is adjusted to the luminance of the image with the high gain GainH. Since the amplification factor of the column amplifier 130 is known in advance, the synthesis process can be performed correctly in theory by applying the amplification factor ratio (GainH / GainL) to the low gain image (FIG. 10(A)). Therefore, the exact amplification factor of the column amplifier 130 is determined in advance.

[0056] However, when there is variation in the column amplifier 130 or when the linearity characteristics of the image sensor 1 are poor to begin with, a step occurs at the joint of images in dynamic range expansion synthesis, resulting in poor linearity (FIG. 10B). Therefore, in this embodiment, even when combining image signals output with different amplification factors (GainH, GainL) of the column amplifiers, a correction process is performed to match the levels of the switching parts of each image.

[0057] That is, in this embodiment, not only is the step correction performed for the joining of signals with different AD conversion gains as shown in Fig. 4, but step correction is also performed for the joining of signals with different column amplifier gains. In this case, step correction is performed by at least adjusting the gain. Here, the image processing LSI 2 functions as a second correction section for correcting a step that occurs during synthesis in the second synthesis section.

[0058] Furthermore, information on the linearity characteristics (non-linear characteristics) of the image sensor, such as that shown in FIG. 10B, is acquired in advance, and the image processing LSI 2 corrects the gain and offset based on that information so that the steps become less noticeable. Although the dynamic range expansion synthesis process has been described above, the synthesis algorithm is not limited to this as long as it is a method for synthesizing two images with different gains. Furthermore, the method is not limited as long as it is a process for correcting the switching portion of the image in accordance with the linearity.

[0059] Fig. 11 is a flowchart showing a control flow of the imaging device 1000 of the first embodiment, and the operation performed in, for example, the image processing LSI 2 of this embodiment will be described with reference to Fig. 11. Note that the image processing LSI 2 has a built-in CPU as a computer as described above, and performs the operation shown in Fig. 11 by executing a computer program stored in a memory (not shown). In step S1, in the first frame, the image sensor 1 starts outputting two images amplified with different amplification factors (GainH, GainL).

[0060] Here, assuming that the number of frames is N, in step S1, the image signal of the first frame is output, so N=1 is set. Note that the first stage of each frame is the readout period of the output signal from the dummy pixel region 1101 in Fig. 5, and as described above, fixed voltages V1 and V2 are supplied to the column amplifier 130. Then, in the column amplifier 130, the signal is amplified by at least one of the gains GainH and GainL.

[0061] In step S2, the output signal from the dummy pixel region 1101 in the first frame is supplied with voltages V1 and V2 to the column amplifier 130, and the AD conversion correction values ​​(V1L, V1H, V2L, V2H) for the multiple slopes shown in Figure 4(B) are calculated using the AD conversion signal. In step S3, the aforementioned slope ratio n and offset value are calculated as correction coefficients based on the correction value calculated in step S2. This correction coefficient is sent to the imaging element and set in the correction circuit in the signal processing circuit 170.

[0062] In step S4, with the correction coefficient set, the image signal of the second frame (N=N+1) is read out. That is, the N signal and the S signal are read out from the effective pixel area 1103 in FIG. 5 while changing the gain (GainH, GainL) at the timing as shown in FIG. 9, and then the SN calculation is performed. Then, for the SN signal of each gain, as shown in FIG. 4, for the SN signal having a voltage level lower than the judgment level Vth, an image signal that has been AD converted with the first ramp signal VRAMP1 is output. Also, for the SN signal having a voltage level higher than Vth, an image signal that has been AD converted with the second ramp signal VRAMP2 is output. This operation is repeated for each row in sequence.

[0063] In step S5, linearity correction is performed on the image signal AD converted by the first ramp signal VAMP1 and the image signal AD converted by the second ramp signal VAMP2 based on the slope ratio n and the offset value. By performing such linearity correction, the signals can be joined together without noticeable steps at the joints. Such linearity correction and joining are performed on the signal amplified by the high gain GainH and the signal amplified by the low gain GainL, respectively.

[0064] In step S6, the dynamic range expansion synthesis process is performed by synthesizing two image signals output with different amplification factors (GainH, GainL) and AD converted with different ramp voltages according to Vth, as shown in Fig. 10. At this time, the linearity characteristics of each of the image signals amplified and output with different amplification factors (GainH, GainL) in the image sensor 1 are acquired in advance and stored in memory as linearity characteristic information. Then, based on the linearity characteristic information stored in the memory, the step at the joint in the dynamic range expansion synthesis as shown in Fig. 10 is corrected.

[0065] Steps S4 to S7 are repeated until it is determined in step S7 that the output of the image has ended, for example when the power of the imaging device is turned off, etc. If it is determined in step S7 that the output of the image has ended, this flow ends. <Example 2>

[0066] In the first embodiment, linearity characteristic information of each of the image signals outputted with different amplification factors (GainH, GainL) as described above is stored in advance in a memory. Then, based on the linearity characteristic information stored in the memory, the step at the joint in the dynamic range expansion synthesis is corrected. However, the linearity characteristic may change due to the temperature of the image sensor, aging, etc. Therefore, in the second embodiment, a control for acquiring the above correction value in real time and performing correction will be described.

[0067] 12 is a diagram for explaining AD conversion of voltages V1 and V2 during a readout period of an output signal from a dummy pixel region 1101. The output signal of the dummy pixel is amplified by the column amplifier 130 to which the voltages V1 and V2 are supplied, and then AD converted with two types of slopes. Also, for the voltages V1 and V2 of the high gain GainH (FIG. 12(A)), AD converted values ​​are output by a first ramp signal VRAMP1 and a second ramp signal VRAMP2, respectively.

[0068] Furthermore, for the voltages V1 and V2 of the low gain GainL (FIG. 12(B)), AD converted values ​​are output by the first ramp signal VRAMP1 and the second ramp signal VRAMP2, respectively. The slope ratio n and offset amount can be calculated from the AD converted values ​​at the high gain GainH and the low gain GainL. For example, the ratio n of the amplification factor after AD conversion of the high gain GainH signal to the low gain GainL signal can be calculated by the following formula (3).

[0069] n=(V2HH-V1HH) / (V2HL-V1HL)...(3) After determining the slope ratio n, an offset value can be determined so that the two straight lines in FIG. 12(A) intersect at the judgment level Vth, and an offset value can also be determined so that the two straight lines in FIG. 12(B) intersect at the judgment level Vth. That is, for each amplification factor (GainH, GainL), the offset value can be determined so that the value obtained by multiplying the slope of the second ramp signal by n and performing AD conversion at the judgment level Vth is the same as the value obtained by AD converting the first ramp signal.

[0070] The above calculation may be performed by a circuit inside the image sensor, or may be performed by the image processing LSI 2. By using the correction values ​​of the slope and offset obtained here, it is possible to further improve the linearity characteristics when, for example, an image with a low gain GainL and an image with a high gain GainH are joined together. <Example 3>

[0071] In the second embodiment, the control of acquiring and correcting the correction value in real time has been described. However, when the offset value is acquired using AD conversion values ​​of multiple slopes (for example, when V2HL and V2LL are used), the correct offset value may not be obtained due to the correction error of AD conversion when there are multiple slopes. Therefore, in the third embodiment, when obtaining the linearity characteristic, an offset value for correcting the linearity characteristic may be obtained only from information on the AD conversion value using one of the ramp signals.

[0072] The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention, and these modifications are not excluded from the scope of the present invention. A computer program for implementing all or part of the control in this embodiment may be supplied to the imaging device via a network or various storage media. Then, a computer (or a CPU, MPU, etc.) in the imaging device may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0073] 1000: Imaging device 110: Pixel section 130: Column amplifier 150: Column ADC

Claims

1. an AD conversion unit that performs AD conversion of an input signal by comparing a first ramp signal with a second ramp signal having a slope different from that of the first ramp signal; a first combining unit that combines a signal obtained by AD conversion using the first ramp signal and a signal obtained by AD conversion using the second ramp signal; a first correction unit that corrects a step occurring during synthesis in the first synthesis unit; a readout unit that amplifies pixel signals at different amplification rates and outputs the amplified pixel signals; a second combining unit that combines the signals amplified by the readout unit at different amplification factors and read out; a second correction unit for correcting a step occurring during synthesis in the second synthesis unit, the second correction unit corrects a step occurring during synthesis in the second synthesis unit based on an AD conversion value obtained by the AD conversion unit using either the first ramp signal or the second ramp signal.

2. 2. The imaging device according to claim 1, wherein the AD conversion section performs AD conversion on the signal read out by the readout section.

3. 2 . The imaging device according to claim 1 , wherein the first correction unit performs gain correction and offset correction on a signal obtained by AD conversion using the first ramp signal or the second ramp signal.

4. The imaging device according to claim 1 , wherein the second correction section corrects at least a gain of the signal amplified with the different amplification factor.

5. 2. The imaging apparatus according to claim 1, wherein the second correction section includes a memory for storing in advance linearity characteristics of the signals amplified at the different amplification factors.

6. A computer program for controlling each unit of the imaging device according to any one of claims 1 to 5 by a computer.

7. A computer-readable storage medium storing the computer program according to claim 6.

Images