Photoelectric conversion device, image processing method, imaging system, movable body and instrument

Abstract

To realize highly accurate correction processing.SOLUTION: A photoelectric conversion device according to one embodiment comprises: a calculation unit which calculates a correction value based on pixel values read from a first region and a second region; and a correction unit which corrects offset of the pixel value according to incident light based on the correction value. The calculation unit calculates an initial value of each of a plurality of first correction components based on the first pixel value read from the first region, updates each of the plurality of first correction components based on the second pixel value read from the second region and a prescribed second correction component, and calculates a correction value by using the updated first correction component and the second correction component.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a photoelectric conversion device, an image processing method, an imaging system, a mobile device, and equipment. [Background technology]

[0002] In recent years, development has been underway on photoelectric conversion devices for use in digital still cameras, video cameras, and mobile phones. For example, CMOS (Complementary Metal Oxide Semiconductor) sensors are widely known as photoelectric conversion devices.

[0003] In a CMOS sensor, a row containing pixels arranged in a matrix in the row and column directions is selected, and the pixel signals of all pixels in the selected row are read out simultaneously. It is known that the image signal read out by a CMOS sensor contains an offset. Patent documents 1 and 2 describe circuits that use the pixel signals of the optical black (hereinafter referred to as "OB") region to correct the offset of the pixel signals of the effective pixels. However, with the recent increase in image quality of CMOS sensors, the number of rows and vertical signal lines has increased, making it difficult to perform high-precision correction processing with the configurations described in Patent documents 1 and 2. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2009-005329 [Patent Document 2] Japanese Patent Publication No. 2013-106186 [Overview of the project] [Problems that the invention aims to solve]

[0005] The present invention aims to provide a photoelectric conversion device, an image processing method, a photoelectric conversion system, a mobile device, and equipment capable of performing high-precision correction processing. [Means for solving the problem]

[0006] A photoelectric converter in one disclosure of this specification includes a pixel section including a plurality of pixels arranged in a matrix, a drive section that drives the pixels in a first region in a first way and the pixels in a second region in a second way, a calculation section that calculates a correction value based on pixel values ​​read from the first region and the second region, and a correction section that corrects the offset of the pixel values ​​according to the incident light based on the correction value, wherein the calculation section calculates an initial value for each of a plurality of first correction components based on a first pixel value read from the first region, updates each of the plurality of first correction components based on a second pixel value read from the second region and a predetermined second correction component, and calculates the correction value using the updated first correction component and the second correction component.

[0007] Image processing methods in other disclosures of this specification include the steps of: calculating a first correction component based on first data in a first region of image data; calculating a second correction component using second data in a second region of the image data and the first correction component; updating the first correction component using the second data and the second correction component; and correcting the second data using the updated first correction component and the second correction value. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a photoelectric conversion device, an image processing method, and a photoelectric conversion system capable of performing high-precision correction processing. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram of a photoelectric converter according to the first embodiment. [Figure 2] This is an equivalent circuit diagram of a pixel according to the first embodiment. [Figure 3A] This is a diagram showing the pixel section according to the first embodiment. [Figure 3B] It is a diagram showing a pixel portion according to the first embodiment. [Figure 4A] It is a diagram showing the driving of the pixel portion by the driving portion according to the first embodiment. [Figure 4B] It is a diagram showing the driving of the pixel portion by the driving portion according to the first embodiment. [Figure 4C] It is a diagram showing the driving of the pixel portion by the driving portion according to the first embodiment. [Figure 5] It is a block diagram of a signal processing portion in an arithmetic portion according to the first embodiment. [Figure 6] It is a block diagram of an OB clamp arithmetic portion according to the first embodiment. [Figure 7A] It is a block diagram of a first processing portion and a second processing portion according to the first embodiment. [Figure 7B] It is a block diagram of a first processing portion and a second processing portion according to the first embodiment. [Figure 7C] It is a block diagram of a first processing portion and a second processing portion according to the first embodiment. [Figure 7D] It is a block diagram of a first processing portion and a second processing portion according to the first embodiment. [Figure 8] It is a diagram showing a configuration example of a photoelectric conversion system according to the second embodiment. [Figure 9] It is a flowchart of correction arithmetic according to the second embodiment. [Figure 10] It is a diagram showing an example of curve approximation according to the second embodiment. [Figure 11] It is a block diagram showing a schematic configuration of an imaging system according to the third embodiment. [Figure 12] It is a diagram showing a configuration example of an imaging system and a moving body according to the fourth embodiment. [Figure 13] It is a block diagram showing a schematic configuration of a device according to the fifth embodiment.

Modes for Carrying Out the Invention

[0010] Embodiments of the present invention will be described below with reference to the drawings. Note that the embodiments described above can be combined in any way as long as they do not contradict each other. Furthermore, in the embodiments described below, the imaging device will be the main focus as an example of a photoelectric conversion device. However, each embodiment is not limited to an imaging device and can be applied to other examples of photoelectric conversion devices. Examples include distance measuring devices (devices for distance measurement using focus detection or TOF (Time of Flight)), and photometric devices (devices for measuring the amount of incident light, etc.).

[0011] [First Embodiment] (Overall structure) Referring to Figure 1, the circuit configuration of the photoelectric converter 10 according to this embodiment will be described. The photoelectric converter 10 is, for example, a CMOS image sensor and includes a pixel unit 11, a drive unit 12, a readout unit 14, a signal processing unit 15, and a timing generator (TG) 16.

[0012] The pixel unit 11 comprises a plurality of pixels 110 arranged in a two-dimensional array, and each pixel 110 is equipped with a photoelectric conversion unit that generates and stores a signal charge according to the amount of light received. In this specification, the row direction refers to the horizontal direction in the drawing, and the column direction refers to the vertical direction in the drawing. Microlenses and color filters may be placed on the pixels 110. The color filters are, for example, primary color filters of red (R), blue (B), and green (Gr, Gb), and are provided on each pixel 110 according to the Bayer array. Some pixels 110 are shielded from light as OB pixels (optical black pixels). In addition, the pixel unit 11 may be provided with NULL pixels that are not connected to the vertical signal line VL. Furthermore, the pixel unit 11 may include a distance measuring row on which focus detection pixels that output a pixel signal for focus detection are arranged, and a plurality of imaging rows on which imaging pixels that output pixel signals for generating an image are arranged. A vertical signal line VL is provided for each column of pixels 110, and multiple pixels 110 in the same column output pixel signals to a common vertical signal line VL. The pixel section 11 may include NULL pixels that are not connected to the vertical signal line VL. Although not shown, a constant current circuit is connected to each of the vertical signal lines VL, and the constant current circuit functions as a load circuit for the pixel 110.

[0013] The drive unit 12 consists of a shift register, a gate circuit, a buffer circuit, etc., and outputs control signals to the pixels 110 based on a vertical synchronization signal, a horizontal synchronization signal, a clock signal, etc., driving the pixels 110 row by row. In this embodiment, the drive unit 12 is capable of performing vertical scanning, sequentially selecting four rows of pixels 11 in one horizontal scanning period. The selected four rows of pixels 110 are read out by the readout unit 14 via the vertical signal line VL. In the following description, the four vertical signal lines VL that can be read out simultaneously will be referred to as vertical signal lines VL1 to VL4. For each horizontal scanning period (hereinafter sometimes referred to as "1H"), pixel signals are read out from vertical signal lines VL1 to VL4, and the OB correction value calculation process is performed.

[0014] The readout unit 14 is capable of reading pixel signals from pixels 110 via vertical signal lines VL and includes an amplification circuit, an ADC (Analog to Digital Converter) circuit, a column memory, a horizontal scanning circuit, and the like. The readout unit 14 can simultaneously read pixel signals from multiple rows, for example, four rows of vertical signal lines VL, and output them as digital data (pixel values).

[0015] The signal processing unit 15 performs digital signal processing on the pixel data output from the readout unit, including OB clamping, digital gain, digital correlation double sampling, digital offset, and linearity correction. In this embodiment, the signal processing unit 15 can perform digital signal processing such as CDS (Correlated Double Sampling) processing and OB clamping on four rows of pixel signals per horizontal scanning period. The signal processing unit 15 also includes an LVDS (Low Voltage Differential Signaling) serial output circuit, which outputs the processed digital signal to the outside of the photoelectric converter at high speed and with low power consumption.

[0016] (Pixel section) Figure 2 shows an example of the configuration of a pixel 110 according to this embodiment. The multiple pixels 110 are arranged in a matrix across multiple rows and multiple columns, as shown in Figure 1. The pixel 110 may include, for example, a photoelectric conversion unit 111, a transfer transistor 112, a floating diffusion (FD) 113, an amplification transistor 114, a selection transistor 115, and a reset transistor 116.

[0017] The photoelectric conversion unit 111 is composed of, for example, a photodiode. The anode of the photoelectric conversion unit 111 is connected to the ground node (GND), and the cathode is connected to the source of the transfer transistor 112. The drain of the transfer transistor 112 is connected to the source of the reset transistor 116 and the gate of the amplification transistor 114. The connection nodes of the drain of the transfer transistor 112, the source of the reset transistor 116, and the gate of the amplification transistor 114 form a so-called floating diffusion (FD) 113. The FD 113 contains a capacitive component and functions as a charge holding unit and a charge-voltage conversion unit. The drains of the reset transistor 116 and the amplifier transistor 114 are connected to the power supply node (voltage VDD). The source of the amplifier transistor 114 is connected to the drain of the selection transistor 115. The source of the selection transistor 115 is connected to the vertical signal line VL. Note that the names of the source and drain of a transistor may differ depending on the conductivity type and application of the transistor, and the source and drain may also be referred to in reverse.

[0018] Each row selection line includes a signal line connected to the gate of the transfer transistor 112, a signal line connected to the gate of the reset transistor 116, and a signal line connected to the gate of the selection transistor 115. The signal line connected to the gate of the transfer transistor 112 is supplied with a control signal PTX from the drive unit 12. The signal line connected to the gate of the reset transistor 116 is supplied with a control signal PRES from the drive unit 12. The signal line connected to the gate of the selection transistor 115 is supplied with a control signal PSEL from the drive unit 12. If each transistor is an N-type transistor, the transistor is turned on by a high-level control signal and turned off by a low-level control signal.

[0019] When light is incident on the pixel section 11, the photoelectric conversion section 111 of each pixel 110 converts the incident light into an electric charge corresponding to the amount of light (photoelectric conversion) and stores the generated charge. The transfer transistor 112 turns on, transferring the charge from the photoelectric conversion section 111 to the FD 113. The FD 113 holds the charge transferred from the photoelectric conversion section 111. The charge transferred from the photoelectric conversion section 111 is converted into a voltage by the capacitive component of the FD 113.

[0020] Voltage VDD is supplied to the drain of the amplification transistor 114, and bias current is supplied to the source from the current source 118 via the selection transistor 115. The amplification transistor 114 constitutes an amplification section (source follower circuit) with its gate as the input node. As a result, the amplification transistor 114 outputs a signal based on the voltage of FD113 to the vertical signal line VL via the selection transistor 115. The reset transistor 116 resets FD113 to a voltage corresponding to the voltage VDD when it is turned on.

[0021] The transfer transistor 112, reset transistor 116, and selection transistor 115 of the pixel 110 are controlled row by row by control signals PTX, PRES, and PSEL supplied from the drive unit 12. The normal readout of the pixel 110 is described below. Except during the storage period (the period from shutter scanning to readout scanning, which will be described later), the pixel 110 is reset. During shutter scanning, when the reset by the reset transistor 116 is released, the photoelectric conversion unit 111 starts to store charge. Next, after a predetermined storage time, a readout scan is performed. During the readout scan, the row is selected, and the pixel signal (N signal) of the reset level of the FD113 is read out. Then, the charge of the photoelectric conversion unit 111 is transferred to the FD113, and the pixel signal (S signal) based on the charge is read out. After that, the photoelectric conversion unit 111 and the FD113 are reset. The signal processing unit 15 calculates the difference between the S signal and the N signal (CDS processing), thereby obtaining the pixel signal based on the charge stored in the photoelectric conversion unit 111.

[0022] Figures 3A and 3B show the pixel section according to this embodiment. The pixel section 11 shown in Figure 3A comprises an aperture region 11a, and OB regions 11b and 11c, while the pixel section 11 shown in Figure 3B comprises an aperture region 11a, and OB regions 11b, 11c, and 11d. The aperture region 11a includes unshielded pixels 110 and can output a pixel signal corresponding to incident light. The OB regions 11b, 11c, and 11d include shattered pixels 110 or NULL pixels not connected to the vertical signal line VL, and are used for offset correction and the like. In Figures 3A and 3B, the OB region 11b is located in the row above the aperture region 11a, and the OB region 11c is located in the left column of the aperture region 11a. Also, in Figure 3B, the OB region 11d is located in the row below the aperture region 11a. Note that the position of the OB region is not limited to Figures 3A and 3B, and may be provided in the right column of the aperture region 11a.

[0023] As will be described later, the photoelectric conversion device in this embodiment is capable of performing cyclic readout, which reads out multiple rows of pixels 110 multiple times in succession, and normal readout other than cyclic readout. Cyclic readout may be performed, for example, in the upper row of the two-part OB region 11b as shown in Figure 3A, or in the OB region 11d provided below the aperture region 11a as shown in Figure 3B. In either case, normal readout may typically be performed after cyclic readout.

[0024] (Drive unit) In this embodiment, the drive unit 12 can perform various drives in the pixel unit 11 for purposes such as shutter drive and OB value calculation.

[0025] Figures 4A, 4B, and 4C show the driving of the pixel unit 11 by the drive unit 12 according to this embodiment. In the figures, the horizontal axis represents time, and the vertical axis represents the row direction (vertical direction) of the pixel device. Hatched rectangles represent shutter scanning, and white rectangles represent readout scanning. The numbers within the rectangles in readout scanning correspond to vertical signal lines VL1 to VL4.

[0026] Figure 4A shows the driving of the pixel unit 11 by the drive unit 12 according to this embodiment, and shows shutter driving. At time t101, the drive unit 12 performs shutter scanning by setting the control signals PTX and PRES for the first to fourth rows to high levels, and resets the charge of the photoelectric conversion unit 111 and FD113 of the pixel 110 to voltage VDD. Subsequently, the drive unit 12 sets the control signals PTX and PRES for the first to fourth rows from high levels to low levels, and the photoelectric conversion unit 111 begins accumulating charge according to the incident light. At time t102, the drive unit 12 sets the control signals PTX and PRES for the fifth to eighth rows to high levels, and resets the charge of the photoelectric conversion unit 111 and FD113 of the pixel 110 to voltage VDD. Subsequently, the drive unit 12 sets the control signals PTX and PRES from high levels to low levels, and the photoelectric conversion unit 111 begins accumulating charge according to the incident light. Similarly, after time t103, the drive unit 12 performs shutter scanning every four rows.

[0027] At times t111 to t112, after a predetermined storage time (exposure time) following the shutter scan, the drive unit 12 performs a readout scan of the first to fourth rows. Specifically, the drive unit 12 sets the control signal PSEL for the first to fourth rows to a high level and turns on the selection transistor 115. As a result, the reset N signal is output from the pixel 110 to the vertical signal lines VL1 to VL4. Subsequently, the drive unit 12 sets the control signal PTX for the first to fourth rows to a high level and turns on the transfer transistor 112. As a result, the charge stored in the photoelectric conversion unit 111 is transferred to the FD113. The voltage of the FD113 decreases according to the transferred charge, and the S signal is output from the source of the amplification transistor 114 through the selection transistor 115 to the vertical signal lines VL1 to VL4, respectively. That is, the pixel signals of the first to fourth rows of pixels 110 are simultaneously read out via the vertical signal lines VL1 to VL4.

[0028] Between times t112 and t113, the drive unit 12 sets the control signal PSEL for rows 5 through 8 to a high level, and then sets the control signal PTX to a high level. As a result, pixel signals from pixels 110 in rows 5 through 8 are output sequentially to vertical signal lines VL1 through VL4. In this way, a readout scan is performed in the pixel unit 11 every four rows. The numbers within the rectangle of the readout scan correspond to vertical signal lines VL1 through VL4. That is, pixels 110 in rows 1, 5, 9, ..., and the (4×N-3) are read out via vertical signal line VL1, and pixels 110 in rows 2, 6, 10, ..., and the (4×N-2) are read out via vertical signal line VL2. Additionally, pixels 110 in the 3rd, 7th, 11th, ..., and the (4×N-1)th row are read out via the vertical signal line VL3, and pixels 110 in the 4th, 8th, 12th, ..., and the (4×N)th row are read out via the vertical signal line VL4.

[0029] The drive unit 12 may perform various types of drives in addition to normal readout drives, for example, it may read out NULL pixels. The drive unit 12 may also perform NN readout drives, which read out the S signal after reading out the N signal while the transfer transistor is off. Furthermore, the drive unit 12 may perform cyclic readout drives, which repeatedly read out a predetermined region.

[0030] Figure 4B is a diagram showing the driving of the pixel section 11 by the drive unit 12 according to this embodiment, and shows the driving of cyclic readout. As shown in Figures 3A and 3B, cyclic readout may be performed in OB area 11b or OB area 11d.

[0031] Between times t201 and t202, the drive unit 12 sequentially turns on the selection transistors 115 and transfer transistors 112 of the first to fourth rows, causing the pixel signals of the N and S signals from the pixels 110 to be output to the vertical signal lines VL1 to VL4. Similarly, between times t202 and t203, the drive unit 12 causes the pixel signals from the pixels 110 of the fifth to eighth rows to be output to the vertical signal lines VL1 to VL4, and between times t203 and t204, the drive unit 12 causes the pixel signals from the pixels 110 of the ninth to twelfth rows to be output to the vertical signal lines VL1 to VL4. Between times t204 and t205, the drive unit 12 again outputs pixel signals from the first to fourth rows of pixels 110 to the vertical signal lines VL1 to VL4, and between times t205 and t206, the drive unit 12 outputs pixel signals from the fifth to eighth rows of pixels 110 to the vertical signal lines VL1 to VL4. Similarly, the readout scan is repeated every four rows for the first to twelfth rows of pixels 110.

[0032] In the cyclic readout described above, shutter scanning is not necessarily required; in this case, the readout scan cycle corresponds to the storage time. Shutter scanning may be performed between readout scans, but the readout scan cycle becomes the upper limit of the storage time. Although there are these constraints on storage time, cyclic readout makes it possible to acquire a large amount of data from a small area in a short time.

[0033] Figure 4C shows the driving of the pixel section 11 by the drive unit 12 according to this embodiment, and illustrates the driving that performs normal reading after cyclic reading.

[0034] In the following description of this embodiment, it is assumed that a normal read operation is performed after a portion of the OB region 11b has been cyclically read. The normal read operation may be performed in the remaining area of ​​the OB region 11b where cyclic reading was not performed, such as the OB region 11c and the opening region 11a. As shown in Figure 3A, the upper area of ​​the OB region 11b may be allocated to cyclic reading, and the lower area of ​​the OB region 11b may be allocated to normal reading. Alternatively, as shown in Figure 3B, the OB region 11d located below the opening region 11a may be allocated to cyclic reading, and the OB region 11b located above the opening region 11a may be allocated to normal reading. When cyclic reading is performed in the opening region 11a, it is preferable that NN reading is performed. In any case, normal reading may be performed after cyclic reading.

[0035] In the following explanation, cyclic reading is assumed to be performed, for example, on 12 rows of the OB area 11b. The reading for 3 horizontal scan periods is repeated 8 times, yielding cyclic reading data for 24 horizontal scan periods. Note that the cyclically read data may be referred to as pre-data or reference data, while the normally read data may be referred to as the main data.

[0036] During the cyclic readout period T1, cyclic readout is performed in groups of four in the 12 rows of the OB region 11b. That is, at times t301 to t302, the drive unit 12 performs a readout scan of the 1st to 4th rows, outputting the N and S signal pixel signals from the pixels 110 to the vertical signal lines VL1 to VL4. Similarly, the drive unit 12 performs readout scans of the 5th to 8th rows (times t302 to t303), the 9th to 12th rows (times t303 to t304), and the 1st to 4th rows (times t304 to t305). Furthermore, at times t305 to t306, the drive unit 12 performs a readout scan of the 5th to 8th rows, as well as a shutter scan of the 13th to 16th rows of the aperture region 11a. Subsequently, the shutter scan is performed following the cyclic readout in the same manner. The pixel signal during cyclic readout is used as the initial value for the first correction component, which will be described later.

[0037] During the normal readout period T2 following the cyclic readout period T1, normal readout is performed every four rows in the remaining area in the OB region 11b where cyclic readout was not performed, the OB region 11c, and the aperture region 11a. Between times t310 and t311, the drive unit 12 outputs pixel signals of the N signal and S signal from pixels 110 in rows 13 to 16 to vertical signal lines VL1 to VL4. Similarly, between times t311 and t312, the drive unit 12 outputs pixel signals from pixels 110 in rows 17 to 20 to vertical signal lines VL1 to VL4. The pixel signals during period T20 (times t310 to t312) are used as initial values ​​for the filter section of the signal processing unit 15, as will be described later. During period T21 after time t312, the signal processing unit 15 uses the pixel signals from the normal readout to determine a second correction component that depends on the vertical position. Furthermore, the signal processing unit 15 continuously updates the first correction component sequentially using the pixel signal and the second correction component. Details will be described later.

[0038] (Signal processing unit) Figure 5 is a block diagram of the signal processing unit 15 according to this embodiment. The signal processing unit 15 includes calculation units 15R, 15B, 15Gr, and 15Gb for each color of the Bayer-arranged color pixels 110. Specifically, the signal processing unit 15 comprises a calculation unit 15R for processing the signal of the R pixel 110, a calculation unit 15B for processing the signal of the B pixel 110, a calculation unit 15Gr for processing the signal of the Gr pixel 110, and a calculation unit 15Gb for processing the signal of the Gb pixel 110. The calculation unit 15R processes the signal of the R pixel 110 read out via vertical signal lines VL1 and VL3, and the calculation unit 15B processes the signal of the B pixel 110 read out via vertical signal lines VL2 and VL4. The calculation unit 15Gr processes the signal of the Gr pixel 110 read out via vertical signal lines VL1 and VL3, and the calculation unit 15Gb processes the signal of the Gb pixel 110 read out via vertical signal lines VL2 and VL4.

[0039] The calculation unit 15R includes CDS calculation units 151R and 152R, a memory unit 153R, an OB clamp calculation unit 150R, and a data output unit 154R. The CDS calculation units 151R and 152R calculate the difference between the S signal and the N signal read from the column memory in the reading unit 14 via the vertical signal line VL1 and output it to the memory unit 153R. Specifically, the CDS calculation unit 151R calculates the difference between the S signal R_VL1S and the N signal R_VL1N read from the R pixel 110 via the vertical signal line VL1. Similarly, the CDS calculation unit 152R calculates the difference between the S signal R_VL3S and the N signal R_VL3N read from the R pixel 110 via the vertical signal line VL3.

[0040] The memory unit 153R can store data for two rows (one horizontal scanning period) of R pixels 110 read via vertical signal lines VL1 and VL3. The memory unit 153R alternately outputs the data from vertical signal line VL1 and the data from vertical signal line VL3 to the OB clamp calculation unit 150R every 1 / 2 horizontal scanning period. At this time, the memory unit 153R adds an identifier (vertical signal line identifier) ​​to the data indicating the vertical signal line VL from which each data was read. The OB clamp calculation unit 150R mainly calculates the OB value using the data of the OB region 11b and realizes the OB clamp function (OB value correction function) by subtracting the OB value from the data of the aperture region 11a. The OB value includes dark current, circuit-dependent components, etc., and the OB value is different for each vertical signal line VL, i.e., it includes a vertical line difference. For this reason, the OB clamp calculation unit 150R calculates the OB value of vertical signal line VL1 and the OB value of vertical signal line VL3 separately. The OB clamp calculation unit 150R outputs the OB-clamped data to the data output unit 154R. The data output unit 154R converts the input data into a predetermined format and outputs it outside the device.

[0041] The other arithmetic units 15B, 15Gr, and 15Gb are configured similarly to arithmetic unit 15R. Specifically, arithmetic unit 15B can perform CDS processing and OB clamping on B pixel 110 data read from vertical signal lines VL2 and VL4. Arithmetic unit Gr performs CDS processing and OB clamping on Gr pixel 110 data read from vertical signal lines VL1 and VL3, and arithmetic unit 15Gb performs CDS processing and OB clamping on Gb pixel 110 data read from vertical signal lines VL2 and VL4. In this way, arithmetic units 15R, 15B, 15Gr, and 15Gb can calculate color-specific OB correction values ​​and perform OB clamping on color pixels 110.

[0042] (OB clamp calculation unit) Figure 6 is a block diagram of the OB clamp calculation unit 150R in the calculation unit 15R. The other OB clamp calculation units 150B, 150Gr, and 150Gb are configured similarly, so the OB clamp calculation unit 150R will be described below. As shown in Figure 6, the OB clamp calculation unit 150R includes a scratch correction unit 1501, an averaging unit 1502, a first processing unit 1503, a second processing unit 1504, a subtraction unit 1505, and a control unit 1506. When an OB value is calculated using data input to the OB clamp calculation unit 150R, that data is input to the scratch correction unit 1501. Also, when an OB clamp is performed using data input to the OB clamp calculation unit 150R, that data is input to the subtraction unit 1505.

[0043] The scratch correction unit 1501 performs scratch correction processing using a predetermined value or an OB value calculated by the second processing unit 1504 as a reference value. Specifically, the scratch correction unit 1501 determines the normal range of the OB value by performing a predetermined calculation on the reference value and determines data outside the normal range as scratch data. Data determined to be scratch data is replaced with the reference value or deleted. This prevents data outside the normal range from affecting the calculation of the OB value.

[0044] The averaging unit 1502 calculates the average value by dividing the accumulated data over a 1 / 2 horizontal scanning period, i.e., one horizontal scanning period for each vertical signal line VL, by the number of data points. The first processing unit 1503 calculates the first correction component for each vertical signal line VL using the average value calculated by the averaging unit 1502 and the second correction component calculated by the second processing unit 1504. The second processing unit 1504 calculates the second correction component using the average value calculated by the averaging unit 1502 and the first correction component calculated by the first processing unit 1503. The second processing unit 1504 also calculates the OB value from the first and second correction components and outputs the OB value to the subtraction unit 1505 and the scratch correction unit 1501.

[0045] The subtraction unit 1505 subtracts the OB value from the input data and performs OB clamping. The control unit 1506 controls the operation of the entire OB clamping calculation unit 150R. For example, the control unit 1506 determines whether the input data is data from the area to be used for OB value calculation. Furthermore, the control unit 1506 controls the switching of the operating modes of the first processing unit 1503 and the second processing unit 1504.

[0046] Figures 7A, 7B, 7C, and 7D are block diagrams of the first processing unit 1503 and the second processing unit 1504 according to this embodiment. Figure 7A shows the overall configuration of the first processing unit 1503 and the second processing unit 1504, and in Figures 7B, 7C, and D, blocks shown with solid lines represent an operating state, and blocks shown with dashed lines represent a non-operating state.

[0047] The first processing unit 1503 includes a subtraction unit 1530, a filter unit 1531, a filter unit 1532, an average value calculation unit 1533, and multiplexers 1534 and 1535. The multiplexers 1534 and 1535 each have multiple switch circuits and switch between them according to the operating state of the signal processing unit 15. The first processing unit 1503 performs predetermined signal processing on the average values ​​R1 and R3 output from the average unit 1502 in Figure 6 and calculates a first correction component Va that depends on the vertical signal line VL. The second processing unit 1504 includes a subtraction unit 1540, a filter unit 1541, and an addition unit 1542, and updates a second correction component Vb according to the vertical position using the first correction component Va. The addition unit 1542 outputs the sum of the first correction component Va and the second correction component Vb as the OB value.

[0048] In the above configuration, each part may share hardware configuration as long as equivalent functionality is achieved. For example, since filter sections 1531 and 1532 operate alternately, filter sections 1531 and 1532 may be configured with a common circuit.

[0049] Figure 7B is a block diagram of the first processing unit 1503 and the second processing unit 1504 according to this embodiment, showing the operating state during the cyclic readout period T1 (time t301~t310) in Figure 4C. The filter unit 1531 calculates the average value Va1 by dividing the cumulative value of the average data R1 from the averaging unit 1502 in Figure 6 by the number of horizontal scanning periods. That is, the filter unit 1531 calculates the average value Va1 of the data R1 of the vertical signal line VL1 during the same period. Similarly, the filter unit 1532 calculates the average value Va3 of the data R3 of the vertical signal line VL3 during the same period. In this way, the filter units 1531 and 1532 calculate the average values ​​Va1 and Va3 for each vertical signal line VL based on the cyclically read data R1 and R3. Furthermore, the average value calculation unit 1533 calculates the average value Va0 of the average values ​​Va1 and Va3. The average value Va0 represents the average value of the data of all vertical signal lines VL during the cyclic readout period. The calculated average values ​​Va1, Va3, and Va0 are classified according to the vertical signal line identifier and stored in the filter unit 1531, filter unit 1532, and average value calculation unit 1533 as initial values ​​for the first correction component described later. At the end of the cyclic readout (time t310), the first correction components Va1, Va3, and Va0 stored in the filter units 1531, 1532, and average value calculation unit 1533 are expressed by the following formulas, respectively.

[0050]

number

number

number

[0051] In equations 1 to 3 above, "R 1,k "R 3,k " indicates the data R1 and R3 during the cyclic readout of vertical signal lines VL1 and VL3 during the kth horizontal scan period. "N1" indicates the number of horizontal scan periods of the cyclic readout, and "k" indicates the kth horizontal scan period out of the 1st to N1th.

[0052] Generally, the vertical line difference does not depend on the accumulation time, so the difference between the first correction component Va1 and the first correction component Va3 corresponds to the vertical line difference. By averaging the data R1 and R3 from the cyclic readout of each horizontal scanning period over the horizontal scanning period of N1, the random noise contained in the data R1 and R3 is reduced. Note that the initial values ​​of the first correction components Va1, Va3, and Va0 can be representative values ​​of their respective target regions (vertical signal lines VL), such as the median.

[0053] Figure 7C is a block diagram of the first processing unit 1503 and the second processing unit 1504 according to this embodiment, showing the operating state during the normal readout period T20 (time t310~t312) in Figure 4C. As described above, period T20 includes the data S1 and S3 from the normal readout of the first few horizontal scanning periods of the normal readout period T2. The data S1 and S3 in period T20 are used to calculate the initial value of the filter unit 1541. The averaged data S1 and S3 by the averaging unit 1502 are input to the second processing unit 1504. The first correction component Va calculated by the average value calculation unit 1533 is output to the second processing unit 1504, and the subtraction unit 1540 calculates the difference between the data S1 and S3 and the first correction component Va. The filter unit 1541 obtains the average value Vb0 by dividing the accumulated value of this difference by the number of horizontal scanning periods N2. The average value Vb0 is used as the initial value of the second correction component Vb, which will be described later. At the end of the first period T20 of normal reading (time t312), the second correction component Vb0 held in the filter unit 1541 is expressed by Equation 4.

number

[0054] In equation 4 above, "N2" represents the number of horizontal scanning periods in period T20, and "S 1,k "S 3,k" represents the data S1 and S3 during normal reading of the vertical signal lines VL1 and VL3 in the k-th horizontal scanning period of the normal reading area. By subtracting the first correction component Va0 calculated by Equation 3 from the sum of the data S1 and S3, the level difference (DC difference) between the data R1 and R3 during cyclic reading and the data S1 and S3 during normal reading is reduced. This ensures the continuity of the smoothing process described later and suppresses sudden value changes. Note that the level difference between the data R1, R3 and the data S1, S3 is mainly due to the difference in the accumulation time.

[0055] FIG. 7D is a block diagram of the first processing unit 1503 and the second processing unit 1504 according to the present embodiment, showing the operating state during the normal reading period T21 (after time t312) in FIG. 4C. During the period T21, normal reading after the N2-th horizontal scanning period is performed. The subtraction unit 1530 sequentially updates the first correction components Va1 and Va3 based on the difference between the data S1 and S3 output from the averaging unit 1502 and the second correction component Vb. The calculated difference is classified according to the vertical signal line identifier and input to the filter unit 1531 or the filter unit 1532. The filter units 1531 and 1532 perform a smoothing process (low-pass filter process) using the input difference and the operation results held by the filter units 1531 and 1532 up to now. The smoothing process can be executed, for example, by the following first-order IIR (Infinite Impulse Response) process. Va 1,N ={Va 1,(N-1) ×A1+(S 1,N -Vb (N-1) )×(1 - A1)} (Equation 5) Va 3,N ={Va 3,(N-1) ×A1+(S 3,N -Vb (N-1) )×(1 - A1)} (Equation 6)

[0056] In the above Equations 5 and 6, "A1" represents the attenuation coefficient of the first-order IIR process and is a value in the range of 0 < A1 < 1. Also, "Va 1,N ", "Va 3,N" represents the first correction component Va updated during the Nth horizontal scanning period of the vertical signal lines VL1 and VL3, respectively, and "Vb (N-1) This represents the second correction component Vb during the (N-1)th horizontal scanning period. That is, the first correction component Va for each vertical signal line VL is updated using the second correction component Vb, which depends on the vertical position, and the data S.

[0057] Note that the subtraction unit 1530 is not mandatory; the input data "S 1,N "S 3,N The following inputs may be provided to the filter units 1531 and 1532. However, if the image has steep vertical shading, it is preferable to provide a subtraction unit 1530. The reasons are explained in detail below. The calculation results of equations 5 and 6 are updated only once per horizontal scanning period. For this reason, if the image has steep vertical shading, a relatively large attenuation coefficient A1 is required to smooth the vertical shading. That is, in order to follow steep vertical shading, it is necessary to set a high cutoff frequency for the low-pass filter. On the other hand, in order to avoid the effects of noise, a small attenuation coefficient A1 is desirable, such as when shooting at high sensitivity. For this reason, the attenuation coefficient A1 can be determined by a trade-off between conflicting factors. If a subtraction unit 1530 is provided, the second correction component Vb is subtracted from the data S1 and S3, so the effect of steep vertical shading can be reduced. Since the calculation results of the filter unit 1541 are updated twice per horizontal scanning period, it can more easily track steep vertical shading compared to the filter units 1531 and 1532. Thus, the subtraction unit 1530 is particularly effective when vertical shading is large.

[0058] The subtraction unit 1540 calculates the difference between the input data S1, S3 and the first correction components Va1, Va3 respectively. Since the first correction components Va1, Va3 include the vertical line difference, the vertical line difference is reduced or not included in the difference after the first correction components Va1, Va3 are subtracted. The difference by the subtraction unit 1540 is input to the filter unit 1541. The data S1 of the vertical signal line VL1 and the data S3 of the vertical signal line VL3 are alternately input to the filter unit 1541 every half horizontal scanning period. The filter unit 1541 performs a smoothing process using the input difference and the calculation result held in the filter unit 1541. The smoothing process can be implemented by, for example, the following first-order IIR process. Vb N ={Vb (N-1) / 2 ×A2+(S i,N -Va i,N )×(1 - A2)} (Equation 7)

[0059] In the above Equation 7, "i" indicates the identification number of the vertical signal line. "A2" indicates the attenuation coefficient of the first-order IIR process and is a value in the range of 0 < A2 < 1. "Vb N " represents the second correction component in the Nth horizontal scanning period. The second correction component Vb N may vary depending on the vertical position but is a common value for the vertical signal lines VL1, VL3. In Equation 7, for the data S1 of the vertical signal line VL1, the first correction component Va1 which is the calculation result of the filter unit 1531 is referred to, and for the data S3 of the vertical signal line VL3, the first correction component Va3 which is the calculation result of the filter unit 1532 is referred to. The second correction component Vb is common to the vertical signal lines VL1, VL3 and does not depend on the vertical signal line VL. Therefore, the second correction component Vb obtained from the data S1 of the vertical signal line VL1 can be referred to for the second correction component Vb of the vertical signal line VL3, and the second correction component Vb obtained from the data S3 of the vertical signal line VL3 can be referred to for the second correction component Vb of the vertical signal line VL1. "Vb (N-1) / 2This indicates that the next second correction component Vb is obtained from the second correction component Vb from 1 / 2 of the horizontal scanning period prior. Even if vertical shading is present, the filter section 1532 can output a second correction component Vb with reduced noise while tracking the vertical shading.

[0060] The addition unit 1542 adds the updated first correction component Va and second correction component Vb, respectively, and outputs the added value as the OB value (offset value). The subtraction unit 1505 of the OB clamp calculation unit 150 subtracts the OB value from the pixel data in the aperture region 11a, thereby enabling OB value correction that reduces vertical line difference and vertical shading.

[0061] As described above, the photoelectric converter in this embodiment decomposes the OB value into a first correction component and a second correction component, and sequentially updates the first correction component using the second correction component. First, multiple first correction components Va, including the vertical line difference, are obtained from the data R during cyclic readout. Each of the multiple first correction components Va of the multiple vertical signal lines VL is sequentially updated using a second correction component common to the multiple first correction components, and the data S during normal readout. Therefore, according to this embodiment, it is possible to achieve high-precision OB correction without providing many OB regions.

[0062] In particular, the photoelectric converter in this embodiment subtracts a first correction component Va from the data S during normal readout to obtain a second correction component Vb that is independent of the vertical signal line VL. Therefore, when calculating the second correction component Vb, it is not necessary to separate and obtain correction values ​​for each vertical signal line VL, enabling high-precision OB correction with less data and in the OB area.

[0063] Furthermore, the initial value of the first correction component Va can be calculated from a smaller area of ​​data R by cyclic readout. By setting the number of iterations of cyclic readout according to the amount of noise, the vertical line difference can be determined with high accuracy from a small area.

[0064] Here, the effects of this embodiment will be explained in comparison with other configurations. Generally, the OB value includes components that depend on the horizontal position, components that depend on the vertical position, and components that do not depend on the position. The OB clamp in this embodiment targets the correction of the vertical position-dependent component and the position-independent component, while the horizontal position-independent component may be corrected by a different configuration. The vertical position-dependent component and the position-independent component each include a component that depends on the vertical signal line VL and other components such as dark current. Here, it is assumed that the component that depends on the vertical signal line among the vertical position-dependent components is small.

[0065] Another method for OB correction involves obtaining data for each row and independently smoothing the vertical data for each vertical signal line to determine the OB value. In this method, a large OB area is required depending on the number of vertical signal lines to reduce random noise. Also, the sampling interval widens, making it difficult to track vertical shading.

[0066] On the other hand, in this embodiment, as described above, the OB value is decomposed into a first correction component Va and a second correction component Vb, and the first correction component Va is sequentially updated using the second correction component Vb. Therefore, when calculating the second correction component Vb, there is no need to separate and determine the correction value for each vertical signal line VL, nor is there a need to provide many OB regions corresponding to the number of vertical signal lines VL. Furthermore, since the first correction component Va is sequentially updated in the data S during normal reading, the first correction component Va can follow vertical shading. Accordingly, according to this embodiment, it is possible to determine the OB value with high accuracy without requiring many OB regions.

[0067] In the above explanation, the number of vertical signal lines VL was assumed to be 4, but the number of vertical signal lines VL can be any number greater than or equal to 2. Also, although the first correction component Va is classified (calculated) for each vertical signal line VL, it may also be classified for each group of other circuit elements. For example, if a floating diffusion is shared by multiple pixels, differences in OB values ​​may occur depending on the order in which pixels transfer charge to the floating diffusion. In this case, the first correction component Va may be classified according to the order in which pixels use the floating diffusion FD. Furthermore, if a pixel is a ranging pixel equipped with multiple photoelectric conversion units, the first correction component Va may be classified according to the pixel driving method. For example, the first correction component Va may be classified according to the readout method for each of the A image, B image, and A+B image. In addition, if the A image is readout intermittently, the first correction component Va may be classified according to whether or not the row reads the A image. Furthermore, multiple classifications may be combined. The more classifications there are, the more effective it is to determine the OB value with high accuracy.

[0068] Furthermore, although the filter sections 1531, 1532, and 1541 were described as first-order IIRs, other IIRs such as second-order IIRs, or even FIRs (Finite Impulse Responses) could be used.

[0069] Furthermore, in cyclic reading, NN reading and NULL reading may be combined. Alternatively, if vertical scanning is decimated during normal reading and not all rows are read, the unread rows may be used to calculate the OB value. Note that if the pixel 110 used to calculate the OB value is included in the aperture region 11a, NN reading is preferable. In this way, OB values ​​such as vertical line difference can be determined using various reading methods.

[0070] The number of cycles (horizontal scanning period N1) for cyclic readout can be set arbitrarily. A higher number of cycles reduces the noise included in the initial value of the first correction component Va. On the other hand, from the viewpoint of power consumption and frame rate, a lower number of cycles is preferable. The number of cycles can be determined by trade-offs between several factors. For example, the number of cycles may be dynamically changed depending on the vertical scanning mode, ISO sensitivity, temperature, etc.

[0071] A vertical signal line identifier was used to classify the first correction component Va, but the vertical signal line identifier switches every 1 / 2 horizontal scanning period. Therefore, the vertical signal line identifier is not essential, and the vertical signal line can be identified from the timing of the horizontal scan.

[0072] [Second Embodiment] Next, the photoelectric conversion system in this embodiment will be described. In the first embodiment, the OB clamp calculation was performed inside the photoelectric conversion device, but in this embodiment, the OB clamp calculation may be performed outside the photoelectric conversion device.

[0073] Figure 8 shows an example of the configuration of the photoelectric conversion system in this embodiment. The photoelectric conversion system includes a photoelectric conversion device 10 and a computing device 80. The computing device 80 receives data from the photoelectric conversion device 10 via a communication line 20 and can perform predetermined correction calculations by executing a computer program. The communication line 20 may be wired or wireless, and may be public wireless communication or short-range wireless communication. The computing device 80 can perform the functions of the OB clamp calculation unit 150 in the first embodiment and receives data input to the OB clamp calculation unit 150. The computing device 80 stores data for one frame, including cyclic read data and normal read data, in its memory and can repeatedly refer to the data stored in memory during calculation processing. The computing device 80 may be a standalone personal computer, an edge terminal, or a server on the cloud.

[0074] Figure 9 is a flowchart showing the operation of the arithmetic unit 80 in this embodiment. In step S101, the arithmetic unit 80 calculates a first correction component Va for each vertical signal line from a first region of the data of one frame stored in memory. Here, the first region may be a data region corresponding to the OB region 11b, etc. The arithmetic unit 80 classifies the data R during cyclic reading for each vertical signal line VL and calculates the average value of each classified data R. The method for calculating the average value is the same as in equations 1 and 2 of the first embodiment. The average value calculated here is used as the initial value of the first correction component Va.

[0075] In step S102, the arithmetic unit 80 calculates the average value S for each row in the second region, which is the correction value calculation region for the data of one frame. Note that since the processes in steps S101 and S102 are independent, the process in step S102 may be executed before the process in step S101.

[0076] In step S103, the arithmetic unit 80 calculates the difference obtained by subtracting the first correction component Va of the vertical signal line VL from the average value S calculated in step S102.

[0077] In step S104, the arithmetic unit 80 approximates the difference value calculated in step S103 with a curve in the vertical direction. Figure 10 shows an example of curve approximation according to this embodiment. In Figure 10, the horizontal axis represents the difference, and the vertical axis represents the vertical position (Y coordinate of the image). Each point corresponds to the difference for each row, and the solid line shows the curve obtained by approximating the difference with a function. In this way, a correction value corresponding to the vertical shading component can be obtained by curve approximation. The curve may be approximated by a polynomial for the Y coordinate, for example, or by any function such as a spline curve. The curve obtained in this way becomes a second correction component Vb which has different values ​​depending on the vertical position N.

[0078] In step S105, the arithmetic unit 80 calculates the difference by subtracting the second correction component Vb from the average value S of each row calculated in step S102.

[0079] In step S106, the arithmetic unit 80 performs curve approximation on the difference in step S105, similar to step S104. A curve is obtained for each vertical signal line VL. The arithmetic unit 80 updates the obtained curve as the first correction component Va.

[0080] In step S107, the arithmetic unit 80 determines whether the update has converged. That is, in the update process in steps S104 and S106, the arithmetic unit 80 calculates the difference between the first correction component Va and the second correction component Vb before the update and the first correction component Va and the second correction component Vb after the update, and determines whether the difference is less than or equal to a predetermined value. If the update has not converged, i.e., the difference exceeds the predetermined value (NO in step S107), the arithmetic unit 80 repeats the process in steps S103 to S107. If the update has converged, i.e., the difference is less than or equal to the predetermined value (YES in step S107), the arithmetic unit 80 executes the process in step S108. The arithmetic unit 80 continues to update the first correction component Va and the second correction component Vb by repeating the process in steps S103 to S107 until the difference converges. This reduces the error in the first correction component Va and the second correction component Vb.

[0081] In step S108, the arithmetic unit 80 calculates an OB value by adding the second correction component Vb from step S104 and the first correction component Va from step S106. The arithmetic unit 80 subtracts the OB value from the pixel data in the aperture region 11a, enabling high-precision OB correction.

[0082] Based on the above, the correction calculation is performed in the arithmetic unit 80. In this embodiment, since the data is repeatedly referenced, a relatively large memory is required, but high-precision correction becomes possible.

[0083] In this embodiment, a vertical curve was determined to accommodate vertical shading; however, to accommodate horizontal shading as well, surface approximation may be performed in the vertical, horizontal, and three-dimensional space of the pixel values.

[0084] [Third Embodiment] The imaging system in this embodiment will now be described. Figure 11 is a block diagram showing the schematic configuration of the imaging system according to this embodiment. The imaging system 200 illustrated in Figure 11 includes an imaging device 201, a lens 202 that forms an optical image of a subject onto the imaging device 201, an aperture 204 for varying the amount of light passing through the lens 202, and a barrier 206 for protecting the lens 202. The lens 202 and aperture 204 are an optical system that focuses light onto the imaging device 201. The imaging device 201 is a photoelectric conversion device 10 described in any of the first to sixth embodiments, which converts the optical image formed by the lens 202 into image data.

[0085] The imaging system 200 also includes a signal processing unit 208 that processes the output signal from the imaging device 201. The signal processing unit 208 generates image data from the digital signal output by the imaging device 201. The signal processing unit 208 also performs various corrections and compressions as needed before outputting the image data. The imaging device 201 may include an AD conversion unit that generates the digital signal processed by the signal processing unit 208. The AD conversion unit may be formed on the semiconductor layer (semiconductor substrate) on which the photoelectric conversion unit of the imaging device 201 is formed, or on a semiconductor substrate separate from the semiconductor layer on which the photoelectric conversion unit of the imaging device 201 is formed. The signal processing unit 208 may also be formed on the same semiconductor substrate as the imaging device 201.

[0086] The imaging system 200 further includes a memory unit 210 for temporarily storing image data, and an external interface unit (external I / F unit) 212 for communicating with an external computer or the like. Furthermore, the imaging system 200 includes a recording medium 214 such as a semiconductor memory for recording or reading imaging data, and a recording medium control interface unit (recording medium control I / F unit) 216 for recording or reading data from the recording medium 214. The recording medium 214 may be built into the imaging system 200 or it may be detachable.

[0087] Furthermore, the imaging system 200 includes an overall control and calculation unit 218 that performs various calculations and controls the entire digital still camera, and a timing generation unit 220 that outputs various timing signals to the imaging device 201 and the signal processing unit 208. Here, the timing signals and the like may be input from an external source, and the imaging system 200 only needs to include at least an imaging device 201 and a signal processing unit 208 that processes the output signals output from the imaging device 201.

[0088] The imaging device 201 outputs an imaging signal to the signal processing unit 208. The signal processing unit 208 performs predetermined signal processing on the imaging signal output from the imaging device 201 and outputs image data. The signal processing unit 208 generates an image using the imaging signal.

[0089] Thus, according to this embodiment, an imaging system can be realized that applies the photoelectric converter 10 according to the first to fifth embodiments.

[0090] [Fourth Embodiment] The imaging system and mobile body in this embodiment will now be described. Figure 12 is a diagram showing the configuration of the imaging system and mobile body according to this embodiment.

[0091] Figure 12(a) shows an example of an imaging system for an in-vehicle camera. The imaging system 300 includes an imaging device 310. The imaging device 310 is the photoelectric converter 10 described in any of the first to sixth embodiments above. The imaging system 300 includes an image processing unit 312 that performs image processing on a plurality of image data acquired by the imaging device 310, and a parallax acquisition unit 314 that calculates parallax (phase difference of parallax images) from the plurality of image data acquired by the imaging system 300. The imaging system 300 also includes a distance acquisition unit 316 that calculates the distance to an object based on the calculated parallax, and a collision determination unit 318 that determines whether or not there is a possibility of collision based on the calculated distance. Here, the parallax acquisition unit 314 and the distance acquisition unit 316 are examples of distance information acquisition means that acquire distance information to an object. That is, distance information is information related to parallax, defocus amount, distance to an object, etc. The collision determination unit 318 may use any of this distance information to determine the possibility of collision. The means for acquiring distance information may be implemented by specially designed hardware, or by a software module. It may also be implemented by an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a combination thereof.

[0092] The imaging system 300 is connected to the vehicle information acquisition device 320 and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. The imaging system 300 is also connected to a control ECU 330, which is a control means that outputs a control signal to generate braking force on the vehicle based on the judgment result of the collision judgment unit 318. The imaging system 300 is also connected to a warning device 340 that issues a warning to the driver based on the judgment result of the collision judgment unit 318. For example, if the collision judgment result of the collision judgment unit 318 indicates a high probability of collision, the control ECU 330 performs vehicle control to avoid a collision or mitigate damage by applying the brakes, releasing the accelerator, or suppressing engine output. The warning device 340 warns the user by sounding an alarm, displaying warning information on a screen such as a car navigation system, or vibrating the seat belt or steering wheel.

[0093] In this embodiment, the imaging system 300 captures images of the area around the vehicle, for example, the front or rear. Figure 12(b) shows the imaging system when capturing images of the area in front of the vehicle (imaging range 350). The vehicle information acquisition device 320 sends instructions to the imaging system 300 or the imaging device 310. This configuration allows for further improvement of the accuracy of distance measurement.

[0094] The above example illustrates control to prevent collisions with other vehicles, but it can also be applied to control systems that automatically follow other vehicles or automatically prevent vehicles from straying from their lanes. Furthermore, the imaging system can be applied not only to vehicles such as the vehicle itself, but also to moving objects (mobile devices) such as ships, aircraft, or industrial robots. In addition, it can be applied not only to moving objects but also to a wide range of devices that utilize object recognition, such as intelligent transportation systems (ITS).

[0095] [Fifth Embodiment] The equipment in this embodiment will now be described. Figure 13 is a block diagram showing the schematic configuration of the equipment according to this embodiment.

[0096] Figure 13 is a schematic diagram showing an instrument EQP including a photoelectric converter APR. The photoelectric converter APR has the functions of the photoelectric converter 10 of the first embodiment. All or part of the photoelectric converter APR is a semiconductor device IC. The photoelectric converter APR in this example can be used, for example, as an image sensor, an AF (Auto Focus) sensor, a photometering sensor, or a distance measuring sensor. The semiconductor device IC has a pixel area PX in which pixel circuits PXC including a photoelectric conversion unit are arranged in a matrix. The semiconductor device IC may have a peripheral area PR around the pixel area PX. Circuits other than pixel circuits can be arranged in the peripheral area PR.

[0097] The photoelectric converter APR may have a stacked structure (chip stacking structure) comprising a first semiconductor chip equipped with multiple photoelectric conversion units and a second semiconductor chip equipped with peripheral circuits. The peripheral circuits on the second semiconductor chip can each be a column circuit corresponding to a pixel row of the first semiconductor chip. Alternatively, the peripheral circuits on the second semiconductor chip can each be a matrix circuit corresponding to a pixel or pixel block of the first semiconductor chip. Connections between the first and second semiconductor chips can be made using through-silicon vias (TSVs), direct bonding of conductors such as copper for inter-chip wiring, microbump connections between chips, or wire bonding.

[0098] The photoelectric converter APR may include a semiconductor device IC as well as a package PKG that houses the semiconductor device IC. The package PKG may include a substrate on which the semiconductor device IC is fixed, a cover made of glass or the like that faces the semiconductor device IC, and connecting members such as bonding wires or bumps that connect terminals provided on the substrate to terminals provided on the semiconductor device IC.

[0099] The EQP device may further comprise at least one of the following: an optical device (OPT), a control unit (CTRL), a processing unit (PRCS), a display device (DSPL), a memory device (MMRY), and a mechanical device (MCHN). The optical device (OPT) corresponds to the photoelectric converter (APR) as a photoelectric converter, and is, for example, a lens, shutter, or mirror. The control unit (CTRL) controls the photoelectric converter (APR) and is, for example, a semiconductor device such as an ASIC. The processing unit (PRCS) processes the signals output from the photoelectric converter (APR) and constitutes an AFE (analog front end) or a DFE (digital front end). The processing unit (PRCS) is a semiconductor device such as a CPU (central processing unit) or an ASIC (application-specific integrated circuit). The display device (DSPL) is an EL display device or liquid crystal display device that displays information (images) obtained from the photoelectric converter (APR). The memory device (MMRY) is a magnetic device or semiconductor device that stores information (images) obtained from the photoelectric converter (APR). The memory device (MMRY) is a volatile memory such as SRAM or DRAM, or a non-volatile memory such as flash memory or a hard disk drive. The mechanical device MCHN has movable parts or propulsion parts such as motors and engines. The device EQP displays signals output from the photoelectric converter APR on the display device DSPL, or transmits them to the outside using a communication device (not shown) provided by the device EQP. For this purpose, it is preferable that the device EQP further includes a memory device MMRY and a processing device PRCS, separate from the memory circuit and arithmetic circuit of the photoelectric converter APR.

[0100] The EQP (Equipment Equipment) shown in Figure 13 can be electronic devices such as information terminals with imaging capabilities (e.g., smartphones and wearable devices) or cameras (e.g., interchangeable lens cameras, compact cameras, video cameras, surveillance cameras). In cameras, the mechanical device MCHN can drive components of the optical device OPT for zooming, focusing, and shutter operation. Furthermore, the EQP can be transportation equipment (mobile devices) such as vehicles, ships, and aircraft. Also, the EQP can be medical devices such as endoscopes and CT scanners.

[0101] The mechanical device MCHN in transport equipment can be used as a mobile device. The device EQP as transport equipment is suitable for transporting the photoelectric converter APR, or for assisting and / or automating driving (operation) through its imaging function. The processing device PRCS for assisting and / or automating driving (operation) can perform processing to operate the mechanical device MCHN as a mobile device based on information obtained from the photoelectric converter APR.

[0102] The photoelectric converter APR according to this embodiment can provide high value to its designers, manufacturers, distributors, buyers, and / or users. Therefore, by installing the photoelectric converter APR in the EQP (Equipment Equipment), the value of the EQP can also be increased. Thus, when manufacturing and selling the EQP, deciding to install the photoelectric converter APR of this embodiment in the EQP is advantageous in increasing the value of the EQP.

[0103] [Other embodiments] The present invention is not limited to the embodiments described above and can be modified in various ways. For example, an example in which a part of the configuration of one embodiment is added to another embodiment, or in which a part of the configuration of another embodiment is replaced, is also an embodiment of the present invention.

[0104] The embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various ways without departing from its technical concept or its main features. [Explanation of symbols]

[0105] 10 Photoelectric converter 11 pixel section 11a Opening area 11b~11d OB area 110 pixels 12 Drive unit 14. Readout section 15 Signal Processing Unit 151, 152 CDS Department 153 memory 150 OB clamp section 1503 First Processing Unit 1504 Second Processing Unit

Claims

1. A pixel section including a plurality of pixels arranged in a matrix; a drive unit that drives the pixels in a first region by a first method and drives the pixels in a second region by a second method; a calculation unit that calculates a correction value based on pixel values ​​read from the first region and the second region; a correction unit that corrects an offset of a pixel value according to incident light based on the correction value, The calculation unit is Calculating initial values ​​of a plurality of first correction components based on first pixel values ​​read from the first region; updating each of the plurality of first correction components based on a second pixel value read from the second region and a predetermined second correction component; A photoelectric conversion device, comprising: a first correction component and a second correction component that are updated to calculate the correction value.

2. The photoelectric conversion device according to claim 1 , wherein the second correction component is calculated from the second pixel value and one of the plurality of first correction components.

3. 3. The photoelectric conversion device according to claim 1, wherein the second correction component is calculated from a difference between the second pixel value and one of the plurality of first correction components.

4. 4. The photoelectric conversion device according to claim 1, wherein the plurality of first correction components are representative values ​​of a plurality of the first pixel values.

5. 5. The photoelectric conversion device according to claim 1, wherein the plurality of first correction components are an average value of a plurality of the first pixel values.

6. a plurality of signal lines provided corresponding to columns of the pixel unit; the first correction component is calculated for each of the signal lines; 6. The photoelectric conversion device according to claim 1, wherein the second correction component is common to the plurality of signal lines and has a value corresponding to a row of the pixel unit.

7. The pixel includes a plurality of color pixels, The photoelectric conversion device according to claim 6 , wherein the first correction component is further calculated for each color of the color pixels.

8. The pixel includes a first photoelectric conversion unit and a second photoelectric conversion unit, 8. The photoelectric conversion device according to claim 6, wherein the first correction component is further calculated for each method of reading out charges from the first photoelectric conversion unit and the second photoelectric conversion unit.

9. 2. The photoelectric conversion device according to claim 1, wherein the first method and the second method differ in accumulation time.

10. 10. The photoelectric conversion device according to claim 1, wherein the first method is a cyclic readout method in which a predetermined area is repeatedly read out.

11. 11. The photoelectric conversion device according to claim 10, wherein the number of times the cyclic readout is performed is changed depending on at least one of ISO sensitivity, temperature, and a driving method used by the driving section.

12. When the identifier of the signal line is represented by "i", the second pixel value is represented by "S", the first correction component is represented by "Va", and the second correction component is represented by "Vb", The first correction component (V a ｉ，Ｎ ) is the difference (S ｉ，Ｎ -Vb (Ｎ－１) 7. The photoelectric conversion device according to claim 6, wherein the calculation is performed for each horizontal scanning period based on the following:

13. When the coefficient A1 of the IIR (Infinite Impulse Response) processing is 0<A1<1, the first correction component (Va ｉ，Ｎ )teeth, Wow ｉ，Ｎ ={ｶa ｉ，(Ｎ－１) ×A1+(S ｉ，Ｎ -Vb (Ｎ－１) )×(1-A1)} 13. The photoelectric conversion device according to claim 12, wherein the photoelectric conversion efficiency is calculated according to the following formula:

14. The second correction component (Vb Ｎ ) is the difference (S ｉ，Ｎ -Va ｉ，Ｎ 14. The photoelectric conversion device according to claim 12, wherein the calculation is based on a.

15. When the coefficient A2 of the IIR (Infinite Impulse Response) processing is 0<A2<1, the second correction component (Vb Ｎ )teeth, <h2 style=";text-align:left;direction:ltr">Vb<h2 style=";text-align:left;direction:ltr"> Ｎ <h2 style=";text-align:left;direction:ltr"> ={Vb<h2 style=";text-align:left;direction:ltr"> (Ｎ－１) <h2 style=";text-align:left;direction:ltr"> ×A2+(S<h2 style=";text-align:left;direction:ltr"> ｉ，Ｎ <h2 style=";text-align:left;direction:ltr"> -Va<h2 style=";text-align:left;direction:ltr"> ｉ，Ｎ <h2 style=";text-align:left;direction:ltr"> )×(11-A2)} 15. The photoelectric conversion device according to claim 12, wherein the calculation is performed according to the following formula:

16. When multiple rows are read out during one horizontal scanning period and the coefficient A2 of the IIR (Infinite Impulse Response) processing is 0<A2<1, the second correction component (Vb Ｎ )teeth, <h2 style=";text-align:left;direction:ltr">Vb<h2 style=";text-align:left;direction:ltr"> Ｎ <h2 style=";text-align:left;direction:ltr"> ={Vb<h2 style=";text-align:left;direction:ltr"> (Ｎ－１)／２ <h2 style=";text-align:left;direction:ltr"> ×A2+(S<h2 style=";text-align:left;direction:ltr"> ｉ，Ｎ <h2 style=";text-align:left;direction:ltr"> -Va<h2 style=";text-align:left;direction:ltr"> ｉ，Ｎ <h2 style=";text-align:left;direction:ltr"> )×(11-A2)} 15. The photoelectric conversion device according to claim 12, wherein the photoelectric conversion efficiency is calculated according to the following formula:

17. When the number of horizontal scanning periods in the first region is "N1", the first pixel value is "R", and the kth horizontal scanning period among the first to N1 is "k", the first correction component (Va ｉ The initial value of [0010] 17. The photoelectric conversion device according to claim 12, wherein the photoelectric conversion efficiency is calculated according to the following formula:

18. calculating a first correction component based on first data in a first region of the image data; calculating a second correction component using second data in a second region of the image data and the first correction component; updating the first correction component using the second data and the second correction component; and correcting the second data using the updated first correction component and the updated second correction component.

19. The photoelectric conversion device according to any one of claims 1 to 17, a signal processing unit that processes a signal output from the photoelectric conversion device; An imaging system comprising:

20. A mobile object, The photoelectric conversion device according to any one of claims 1 to 17, a distance information acquiring means for acquiring distance information to an object from a parallax image based on a signal from the photoelectric conversion device; a control means for controlling the moving object based on the distance information; A moving object comprising:

21. The photoelectric conversion device according to any one of claims 1 to 17, an optical device corresponding to the photoelectric conversion device; A control device for controlling the photoelectric conversion device; a processing device that processes a signal output from the photoelectric conversion device; a mechanical device controlled based on information obtained by the photoelectric conversion device; A display device that displays information obtained by the photoelectric conversion device; and a storage device for storing information obtained by the photoelectric conversion device; An apparatus comprising:

Images