Photoelectric conversion device, photoelectric conversion system, and mobile body

Abstract

To provide a technology advantageous for obtaining a high quality image in a photoelectric conversion device having a plurality of pixels for counting photons and outputting a count value.SOLUTION: According to the present invention, a photoelectric conversion device having a plurality of pixels for counting photons and outputting a count value includes: a memory; a first arithmetic unit for generating a first arithmetic value by multiplying the count value generated in each pixel by a first coefficient; a second arithmetic unit for generating a second arithmetic value by multiplying data provided from the memory by a second coefficient; an adder for adding the first arithmetic value and the second arithmetic value to generate an addition value; and a memory controller for updating the data stored in the memory according to the addition value. The number of bits of the first arithmetic value generated by the first arithmetic unit is larger than the number of bits of the count value.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a photoelectric conversion device, a photoelectric conversion system, and a mobile object. [Background technology]

[0002] Patent Document 1 describes a configuration in which each pixel has an avalanche photodiode (APD), a waveform shaping unit that shapes the output signal of the APD to generate a detection signal, a counting unit that counts the detection signals, and an IIR filter that processes the counting result by the counting unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-044636 Summary of the Invention [Problem to be solved by the invention]

[0004] In an IIR filter, if the number of bits of the data indicating the counting result output from the counting unit is the same as the number of bits of the adder, some of the image information (e.g., information after the decimal point) will be lost, making it difficult to obtain a high-quality image.

[0005] An object of the present invention is to provide an advantageous technique for obtaining high-quality images in a photoelectric conversion device having a plurality of pixels that count photons and output count values. [Means for solving the problem]

[0006] One aspect of the present invention relates to a photoelectric conversion device having a plurality of pixels that count photons and output count values, the photoelectric conversion device comprising: a memory; a first calculator that multiplies the count value generated in each pixel by a first coefficient to generate a first calculation value; a second calculator that multiplies data provided from the memory by a second coefficient to generate a second calculation value; an adder that adds the first calculation value and the second calculation value to generate a sum; and a memory controller that updates data stored in the memory according to the sum, wherein the number of bits of the first calculation value generated by the first calculator is greater than the number of bits of the count value. [Effects of the Invention]

[0007] According to the present invention, an advantageous technique is provided for obtaining high-quality images in a photoelectric conversion device having a plurality of pixels that count photons and output count values. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing a basic configuration of a photoelectric conversion device according to an embodiment. [Figure 2] FIG. 2 is a diagram showing an example of the configuration of a sensor substrate. [Figure 3] FIG. 2 is a diagram showing an example of the configuration of a circuit board. [Figure 4] FIG. 2 is an equivalent circuit diagram of one pixel and its signal processing unit. [Figure 5] 3A to 3C are diagrams illustrating the operation of a pixel. [Figure 6] FIG. 2 is a diagram showing the configuration of an image processing unit in the photoelectric conversion device according to the first embodiment. [Figure 7] FIG. 2 is a diagram illustrating the operation of the photoelectric conversion device according to the first embodiment. [Figure 8] FIG. 3 is a diagram illustrating the operation of an image processing unit in the photoelectric conversion device according to the first embodiment. [Figure 9] FIG. 10 is a diagram showing the configuration of an image processing unit in a photoelectric conversion device according to a second embodiment. [Figure 10] FIG. 10 is a diagram showing the configuration of an image processing unit in a photoelectric conversion device according to a third embodiment. [Figure 11]FIG. 10 is a diagram illustrating the operation of the photoelectric conversion device according to the third embodiment. [Figure 12] FIG. 10 is a diagram illustrating the operation of an image processing unit in the photoelectric conversion device according to the fourth embodiment. [Figure 13] FIG. 11 is a diagram illustrating the operation of an image processing unit in the photoelectric conversion device according to the fifth embodiment. [Figure 14] FIG. 13 is a diagram illustrating the operation of an image processing unit in the photoelectric conversion device according to the sixth embodiment. [Figure 15] FIG. 10 is a diagram showing a modified example of the first computing unit. [Figure 16] FIG. 4 is a diagram showing a specific example of a data converter in the first computing unit. [Figure 17] FIG. 4 is a diagram schematically illustrating the operation of a bit width expanding unit. [Figure 18] FIG. 4 is a diagram illustrating the operation of a nonlinear correction unit. [Figure 19] FIG. 4 is a diagram illustrating the operation of a smoothing processing unit. [Figure 20] FIG. 13 is a diagram showing the configuration of a photoelectric conversion system according to a seventh embodiment. [Figure 21] FIG. 10 is a diagram showing a first application example. [Figure 22] FIG. 10 is a diagram showing a second application example. [Figure 23] FIG. 10 is a diagram showing a third application example. [Figure 24] FIG. 10 is a diagram showing a fourth application example. [Figure 25] FIG. 10 is a diagram showing a fifth application example. [Figure 26] FIG. 10 is a diagram showing a sixth application example. [Figure 27] FIG. 10 is a diagram showing a seventh application example. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] In this specification, the term "planar view" refers to viewing the photoelectric conversion device from a direction perpendicular to the light incident surface of the semiconductor layer, and is synonymous with an orthogonal projection onto the light incident surface. Furthermore, the term "cross-sectional view" refers to a surface in a direction perpendicular to the light incident surface of the semiconductor layer. When the light incident surface of the semiconductor layer is a rough surface when viewed microscopically, the planar view is defined based on the light incident surface of the semiconductor layer when viewed macroscopically.

[0011] The present invention can be applied to a photoelectric conversion device having a plurality of pixels that count photons and output count values. As an example of such a photoelectric conversion device, a photoelectric conversion device in which each pixel has an avalanche photodiode (hereinafter also referred to as an APD) will be described below. However, the present invention can also be applied to other photoelectric conversion devices, such as a photoelectric conversion device called an EMCCD or a photoelectric conversion device equipped with a scintillator.

[0012] In the following description of exemplary embodiments, the anode of an avalanche photodiode (APD) is set to a fixed potential, and a signal is extracted from the cathode side. Therefore, the first conductivity type semiconductor region having majority carriers of charges of a first polarity, the same as the signal charge, is an N-type semiconductor region, and the second conductivity type semiconductor region having majority carriers of charges of a second polarity, different from the signal charge, is a P-type semiconductor region. The present invention also applies when the cathode of the APD is set to a fixed potential and a signal is extracted from the anode side. In this case, the first conductivity type semiconductor region having majority carriers of charges of a first polarity, the same as the signal charge, is a P-type semiconductor region, and the second conductivity type semiconductor region having majority carriers of charges of a second polarity, different from the signal charge, is an N-type semiconductor region. While the following description focuses on a case where one node of the APD is set to a fixed potential, the potentials of both nodes may fluctuate.

[0013] First, a basic configuration and driving method common to photoelectric conversion devices and driving methods thereof according to a plurality of embodiments to be described later will be described with reference to FIGS. 1, 2, 3, 4, and 5. FIG.

[0014] FIG. 1 illustrates a basic configuration of a photoelectric conversion device 100 according to an embodiment. Here, an example in which the photoelectric conversion device 100 is configured as a stacked-type photoelectric conversion device is described. However, the present invention is also applicable to photoelectric conversion devices other than stacked-type photoelectric conversion devices. The photoelectric conversion device 100 may be configured by stacking multiple substrates, including a sensor substrate 11 and a circuit substrate 21, and electrically connecting the multiple substrates. The sensor substrate 11 may have a first semiconductor layer having a photoelectric conversion element 102 (described later) and a first wiring structure. The circuit substrate 21 may have a second semiconductor layer having circuits such as an intra-pixel processing unit 103 (described later) and a second wiring structure. The photoelectric conversion device 100 may be configured by stacking the second semiconductor layer, the second wiring structure, the first wiring structure, and the first semiconductor layer in this order, for example. The photoelectric conversion device 100 described as the following embodiment may be, for example, a back-illuminated photoelectric conversion device. However, the photoelectric conversion device according to the present invention may also be configured as a front-illuminated photoelectric conversion device.

[0015] The sensor substrate 11 may include a semiconductor layer including a pixel array region 12 having a plurality of pixels and a peripheral region 13 arranged around the pixel array region 12. The region between the outer edge of the pixel array region 12 and the outer edge of the sensor substrate 11 (the scribe region including the outer edge) may be the peripheral region 13. Circuit elements such as active elements may or may not be arranged in the peripheral region 13. The circuit substrate 21 may include a semiconductor layer including a circuit region 22 that processes signals detected by pixels in the pixel array region 12.

[0016] 2 is a diagram showing an example of the configuration of the sensor substrate 11. In the pixel array region 12, a plurality of pixels 101 can be arranged in a two-dimensional array to form a plurality of rows and a plurality of columns. Each pixel 101 can include a photoelectric conversion element 102 including an avalanche photodiode (APD).

[0017] The pixels 101 arranged in the pixel array region 12 may be pixels for forming an image. However, when the sensor substrate 11 or the photoelectric conversion device 100 is applied to TOF (Time of Flight), each pixel 101 does not have to be a pixel for forming an image. In other words, the pixel 101 may be a pixel for measuring the time and amount of light that arrives.

[0018] 3 is a diagram showing the configuration of the circuit board 21. The circuit board 21 may include, for example, an in-pixel processing unit 103 that processes a signal (electrical signal) generated in response to an electric charge (signal charge) generated by photoelectric conversion in the photoelectric conversion element 102. The circuit board 21 may also include a readout circuit 112, a control pulse generating unit 115, a horizontal scanning circuit 111, signal lines 113, a vertical scanning circuit 110, and an image processing unit 120. One in-pixel processing unit 103 may be provided for one pixel 101. The photoelectric conversion element 102 in FIG. 2 and the in-pixel processing unit 103 in FIG. 3 may be electrically connected via connection wiring provided for each pixel 101.

[0019] The vertical scanning circuit 110 may be configured, for example, to receive a first control pulse supplied from the control pulse generating unit 115, generate a second control pulse, and supply the second control pulse to each pixel 101. The vertical scanning circuit 110 may include, for example, logic circuits such as a shift register and an address decoder. A signal output from the photoelectric conversion element 102 of each pixel 101 may be processed by an in-pixel processing unit 103 provided corresponding to that pixel 101. The in-pixel processing unit 103 may include a counter, a memory, etc., and the memory may hold digital values.

[0020] The horizontal scanning circuit 111 may be configured to supply the in-pixel processing unit 103 with a third control pulse that sequentially selects each column in order to read out a signal from the memory of each pixel 101 that holds the digital signal. The circuit board 21 may have a plurality of signal lines 113. Pixel signals are output to the plurality of signal lines 113 from the in-pixel processing unit 103 assigned to the pixels 101 in a row selected by the vertical scanning circuit 110. The pixel signals output to the plurality of signal lines 113 may be read out by the readout circuit 112 and sequentially supplied from the readout circuit 112 to the image processing unit 120 under the control of the horizontal scanning circuit 111. The image processing unit 120 performs TIIR filtering on the count values (pixel signals) sequentially supplied from the readout circuit 112 and provides the data obtained thereby to the output unit 114. The output unit 114 may output the data provided from the image processing unit 120 to a recording unit or signal processing unit external to the photoelectric conversion device 100.

[0021] 2, the photoelectric conversion elements 102 or pixels 101 may be arranged one-dimensionally in the pixel array region 12. Each intra-pixel processing unit 103 may be assigned to at least two photoelectric conversion elements 102 or pixels 101.

[0022] 2 and 3, a plurality of intra-pixel processing units 103 may be arranged in a region overlapping the pixel array region 12 in a planar view. Then, a vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output unit 114, a control pulse generation unit 115, and an image processing unit 120 may be arranged so as to overlap in a region between the outer edge of the sensor substrate 11 and the outer edge of the pixel array region 12 in a planar view. In other words, the vertical scanning circuit 110, the horizontal scanning circuit 111, the readout circuit 112, the output unit 114, the control pulse generation unit 115, and the image processing unit 120 may be arranged in a region overlapping with the peripheral region 13 of the sensor substrate 11 in a planar view.

[0023] FIG. 4 illustrates an equivalent circuit of one pixel 101 in FIG. 2 and one in-pixel processing unit 103 in FIG. 3. The photoelectric conversion element 102 includes an APD 201. The APD 201 generates charge pairs in response to incident light through photoelectric conversion. A voltage VL (first voltage) is supplied to the anode of the APD 201. A voltage VH (second voltage) higher than the voltage VL supplied to the anode may be supplied to the cathode of the APD 201. A reverse bias voltage (predetermined voltage) that can cause the APD 201 to perform avalanche multiplication may be supplied between the anode and cathode. With such a reverse bias voltage supplied between the anode and cathode, charges generated by incident light may cause avalanche multiplication, generating an avalanche current.

[0024] A mode in which an APD is operated with a voltage between the anode and cathode greater than the breakdown voltage is called Geiger mode. A mode in which an APD is operated with a voltage between the anode and cathode close to or less than the breakdown voltage is called linear mode. An APD operated in Geiger mode is called a SPAD. For example, the voltage VL (first voltage) is −30 V and the voltage VH (second voltage) is 1 V. The APD 201 may be operated in either linear mode or Geiger mode.

[0025] The quench element 202 can be arranged to connect a power supply that supplies voltage VH to the APD 201. The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, and has the function of suppressing avalanche multiplication by suppressing the voltage supplied to the APD 201 (quench operation). The quench element 202 also has the function of returning the voltage supplied to the APD 201 to voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0026] The intra-pixel processing unit 103 may include a waveform shaping unit 210, a counter circuit 211, and a selection circuit 212. The intra-pixel processing unit 103 may be a circuit including at least one of the waveform shaping unit 210, the counter circuit 211, and the selection circuit 212. The waveform shaping unit 210 may shape the potential change of the cathode of the APD 201 obtained upon photon detection and output a pulse signal. For example, an inverter circuit may be used as the waveform shaping unit 210. In FIG. 4, the waveform shaping unit 210 is configured with one inverter, but the waveform shaping unit 210 may include a series connection of multiple inverters or may include another circuit that has a waveform shaping effect.

[0027] The counter circuit 211 can count the pulse signals output from the waveform shaping unit 210 and hold the count value. The counter circuit 211 can be configured to reset the count value held in the counter circuit 211 by receiving a control pulse pRES via a drive line 213. The selection circuit 212 can receive a control pulse pSEL from the vertical scanning circuit 110 in FIG. 3 via a drive line 214 in FIG. 4 (not shown in FIG. 3), and can switch between electrical connection and disconnection between the counter circuit 211 and the signal line 113. The selection circuit 212 can include, for example, a buffer circuit for outputting a signal.

[0028] A switch such as a transistor may be disposed between the quench element 202 and the APD 201 and / or between the photoelectric conversion element 102 and the intra-pixel processing unit 103 to control the electrical connection. Similarly, the supply of the voltage VH and / or the voltage VL to the photoelectric conversion element 102 may be controlled by a switch such as a transistor.

[0029] The photoelectric conversion device 100 may be configured to acquire pulse detection timing using a time-to-digital converter (hereinafter referred to as TDC) and a memory instead of the counter circuit 211. The generation timing of the pulse signal output from the waveform shaping unit 210 may be converted into a digital signal by the TDC. To measure the timing of the pulse signal, a control pulse pREF (reference signal) may be supplied to the TDC from the vertical scanning circuit 110 in FIG. 1 via a drive line. The TDC may acquire, as a digital signal, a signal obtained by converting the input timing of the signal output from each pixel via the waveform shaping unit 210 into a relative time based on the control pulse pREF.

[0030] 5 is a diagram schematically illustrating the relationship between the operation of the APD 201 and the output signal. FIG. 5(a) is a diagram illustrating the APD 201, the quench element 202, and the waveform shaping unit 210 excerpted from FIG. 4. Here, the input side of the waveform shaping unit 210 is referred to as node A, and the output side is referred to as node B. FIG. 5(b) shows the waveform change at node A in FIG. 5(a), and FIG. 5(c) shows the waveform change at node B in FIG. 5(a).

[0031] Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201 in FIG. 5(a). When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at node A drops. As the voltage drop increases and the potential difference applied to the APD 201 decreases, the avalanche multiplication operation of the APD 201 stops, as shown at time t2, and the voltage level at node A no longer drops below a certain value. After that, between time t2 and time t3, a current flows through node A from voltage VL to compensate for the voltage drop, and at time t3, node A returns to its original potential level. At this time, the portion of the output waveform at node A that exceeds a certain threshold is shaped by the waveform shaping unit 210 and output as a signal at node B.

[0032] The arrangement of the signal lines 113, the readout circuits 112, the output units 114, the image processing units 120, etc. is not limited to the arrangement shown in Fig. 3. For example, the signal lines 113 may be arranged to extend in the row direction, and the readout circuits 112 may be arranged at the ends of the signal lines 113.

[0033] Several embodiments of the photoelectric conversion device 100 will be described below. First, a photoelectric conversion device 100 according to a first embodiment will be described. FIG. 6 is a diagram showing the configuration of the image processing unit 120 in the photoelectric conversion device 100 according to the first embodiment. A count value can be provided to the image processing unit 120 from each pixel 101 via a signal line 113 and a readout circuit 112. As described above, this count value is obtained by counting photons in the pixel 101. The image processing unit 120 can include, for example, a memory 403, a first computing unit 410, a second computing unit 420, an adder 401, and a memory controller 406. However, at least one of the memory 403, the first computing unit 410, the second computing unit 420, the adder 401, and the memory controller 406 may be disposed in the pixel 101.

[0034] The memory 403 may be configured, for example, as any one of DRAM, SRAM, MRAM, FeRAM, ReRAM, and EEPROM, but may also be configured, for example, as a collection of flip-flops or latch circuits. The memory 403 may constitute a frame memory capable of holding one frame's worth of data. One frame may be composed of data from a plurality of pixels 101 arranged in the pixel array region 12. The first calculator 410 may be configured to multiply a count value generated in each pixel 101 by a first coefficient (here, (1-α)) to generate a first calculation value and provide the first calculation value to the adder 401. The second calculator 420 may be configured to multiply data provided from the memory 403 (a value held by the memory 403) by a second coefficient (here, α) to generate a second calculation value and provide the second calculation value to the adder 401. In this example, the first coefficient is (1-α), the second coefficient is α, and the sum of the first coefficient and the second coefficient is 1. However, the first coefficient and the second coefficient may be independent coefficients that are not correlated with each other. The first coefficient may be a value greater than 0 and less than 1. The second coefficient may be a value greater than 0 and less than 1.

[0035] The adder 401 may be configured to generate an addition value by adding a first operation value provided from the first operation unit 410 and a second operation value provided from the second operation unit 420. The memory controller 406 may be configured to update data stored in the memory 403 (a value held by the memory 403) according to the addition value generated by the adder 401. The memory controller 406 may be configured to, for example, overwrite data stored in the memory 403 with the addition value generated by the adder 401. The memory controller 406 and the memory 403 may be connected by a data bus DATA and an address bus ADD so that the memory controller 406 can write data to the memory 403 or read data from the memory 403.

[0036] The number of bits of the first calculation value generated by the first calculation unit 410 is larger than the number of bits of the count value provided from each pixel 101 via the signal line 113 and the readout circuit 112. In one example, the number of bits of the count value provided from the pixel 101 via the signal line 113 and the readout circuit 112 is 4, and the number of bits of the first calculation value generated by the first calculation unit 410 is 8. The number of bits of the second calculation value generated by the second calculation unit 420 and the number of bits of the sum value generated by the adder 401 are larger than the number of bits of the count value provided from each pixel 101 via the signal line 113 and the readout circuit 112. In one example, the number of bits of the second calculation value generated by the second calculation unit 420 is 8, and the number of bits of the sum value generated by the adder 401 is 8.

[0037] The first computing unit 410 may include, for example, a data converter 402 and a first multiplier 404. The data converter 402 may be configured to expand the number of bits (bit width) of a count value provided from each pixel 101 via the signal line 113 and the readout circuit 112. The first multiplier 404 may include a first multiplier that multiplies the count value, the number of bits (bit width) of which has been expanded by the data converter 402, by a first coefficient (1-α) to generate a first computed value. The second computing unit 405 may include a second multiplier that multiplies data provided from the memory 403 (a value held by the memory 403) by a second coefficient (α) to generate a second computed value. The memory controller 406 may be configured to read one frame's worth of data (image data) from the memory 403 for each frame period and output the data to the outside of the photoelectric conversion device 100 via the output unit 407. Note that output section 407 may constitute all or part of output section 114, or output section 407 may be provided at a subsequent stage of output section 407.

[0038] The memory controller 406 may determine (correct) the address of the memory 403 to be updated according to the added value based on the correction signal MBC for correcting image blur (camera shake) caused by vibration of the imaging device in which the photoelectric conversion device 100 is installed.

[0039] The plurality of pixels 101, the memory 403, the first computing unit 410, the second computing unit 420, the adder 401, and the memory controller 406 are arranged on one semiconductor chip (packaged semiconductor device). Also, in one semiconductor chip, the plurality of pixels 101 may be arranged on one semiconductor layer, and at least one of the memory 403, the first computing unit 410, the second computing unit 420, the adder 401, and the memory controller 406 may be arranged on another semiconductor layer. However, the memory 403, the first computing unit 410, the second computing unit 420, the adder 401, and the memory controller 406 may be distributed and arranged on two or more semiconductor layers.

[0040] 7 exemplifies the operation of the photoelectric conversion device 100 of the first embodiment. In one example, a cycle in which one frame's worth of image data is output from the photoelectric conversion device 100 to the outside is called one frame period. One frame period can be made up of M subframe periods. In one subframe period, the count values of all the pixels 101 that make up the pixel array region 12 can be read out by the readout circuit 112 and provided to the image processing unit 120. The image processing unit 120 can generate one frame of image data by performing TIIR (Temporal Infinite Impulse Response) filter processing on the image data of the M subframes in one frame period, and output the image data to the outside.

[0041] In FIG. 7, "frame" refers to a "frame period," and "subframe" refers to a "subframe period." Furthermore, "row selection" refers to a "row selection period" during which one row is selected, and "memory update" refers to a period during which the data in the memory 403 is updated with the sum generated by the adder 401. Furthermore, "output" refers to a period during which one frame of image data is output from the photoelectric conversion device 100. In the example of FIG. 7, the multiple pixels 101 in the pixel array region 12 are reset by a rolling shutter method, and count values are read out from the multiple pixels 101 in row sequence. However, the multiple pixels 101 in the pixel array region 12 may also be driven by a global shutter method.

[0042] FIG. 8 shows an example in which α is 0.5 (i.e., the first coefficient and the second coefficient are 0.5), the count value provided by the pixel 101 is 4 bits, and the data converter 402 converts the count value from 4 bits to 8 bits. In FIG. 8, "count value (decimal)" is the count value provided by the pixel 101 at an arbitrary position in the pixel array area 12 expressed in decimal, and "count value (binary)" is the count value provided by the pixel 101 expressed in binary. In the example of FIG. 8, the count value data provided by the pixel 101 consists of 4 bits (i.e., the bit width is 4). In FIG. 8, "data after conversion C" is the count value C after the 4-bit count value provided by the pixel 101 at the arbitrary position is converted to 8 bits by the data converter 402. In the example of FIG. 8, the data converter 402 expands the number of bits (bit width) of the count value provided by the pixel 101 at the arbitrary position by shifting it 4 bits to the left. In FIG. 8, "retained value M" is data (retained value) M read from a corresponding address in the memory 403 (an arbitrary corresponding address in the pixel array region 12).

[0043] 8, "(1-α)×C" indicates the first calculation value generated by the first multiplier 404, and "α×M" indicates the second calculation value generated by the second multiplier 405. Also in Fig. 8, "(1-α)×C+α×M" indicates the addition value generated by the adder 401 expressed in binary notation, and "(decimal notation)" indicates the addition value generated by the adder 401 expressed in decimal notation.

[0044] During the first subframe (subframe=1), the bit width of "1101", which is the count value provided from pixel 101 at an arbitrary position, is expanded to "11010000" by the data converter 402. Also during the first subframe, the first multiplier 404 multiplies "11010000" by a first coefficient (1-α)=0.5 to generate "01101000" as a first calculation value. Also during the first subframe, the second multiplier 405 multiplies the held value "00000000" stored in a corresponding address of the memory 403 by a second coefficient α=0.5 to generate "00000000" as a second calculation value. Furthermore, during the first subframe, the adder 401 adds the first operation value and the second operation value to generate the sum "01101000", and this sum is overwritten at the corresponding address in the memory 403 as the stored value M. That is, the data "00000000" at the corresponding address in the memory 403 is updated with "01101000".

[0045] During the second subframe (subframe=2), the bit width of "1010", which is the count value provided from the pixel 101 at the arbitrary position, is expanded to "10100000" by the data converter 402. Also during the first subframe, the first multiplier 404 multiplies "10100000" by a first coefficient (1-α)=0.5 to generate "01010000" as a first operation value. Also during the second subframe, the second multiplier 405 multiplies the held value "01101000" stored in a corresponding address of the memory 403 by a second coefficient α=0.5 to generate "00110100" as a second operation value. Furthermore, during the second subframe, the adder 401 adds the first operation value and the second operation value to generate the sum "10000100", and this sum is overwritten at the corresponding address in the memory 403 as the stored value M. That is, the data "01101000" at the corresponding address in the memory 403 is updated with "10000100".

[0046] Thereafter, during the periods from the second subframe to the Mth subframe, the same processing is repeated based on the count value provided from the pixel 101 at the arbitrary position and the data (retained value) stored at the corresponding address in the memory 403. This type of filtering processing is called TIIR filtering processing. By expanding the bit width in the first computing unit 410 as in this embodiment, the data overwritten at the corresponding address in the memory 403 is smoothed with high resolution on the time axis, resulting in data with an improved SNR (signal-to-noise ratio), and a high-quality image can be obtained.

[0047] On the other hand, if the bit width is not expanded in the first computing unit 410, the lower-order bits of data are lost in the calculations by the first computing unit 410, the second computing unit 420, and the adder 401, and therefore, improvement in image quality due to TIIR filter processing cannot be expected.

[0048] The memory controller 406 may compress the sum generated by the adder 401 to generate compressed data, and update the value stored in the memory 403 based on the compressed data. In this case, the second calculator 429 multiplies the data obtained by decompressing the compressed data provided by the memory 403 by a second coefficient to generate a second calculation value.

[0049] Next, a photoelectric conversion device 100 according to a second embodiment will be described. Matters not mentioned in the second embodiment may follow the first embodiment. FIG. 9 is a diagram illustrating the configuration of the image processing unit 120 in the photoelectric conversion device 100 according to the second embodiment. In the second embodiment, the image processing unit 120 includes a coefficient determination unit 408. The coefficient determination unit 408 may be configured to determine a first coefficient and a second coefficient based on data stored in the memory 403. In the example illustrated in FIG. 9, the first coefficient is 1-α, the second coefficient is α, and the sum of the first coefficient and the second coefficient is 1. However, the first coefficient and the second coefficient may be independent coefficients that are not correlated with each other. The first coefficient may be a value greater than 0 and less than 1. The second coefficient may be a value greater than 0 and less than 1.

[0050] In an example where the first coefficient is 1-α and the second coefficient is α, the larger the value of α, the stronger the smoothing effect in the time direction, improving the SNR but increasing the likelihood of subject blur. For example, when processing a count value generated at a pixel at an arbitrary position in the pixel array region 12, α may be determined based on the value of data at a corresponding address in memory 403 or the average value of data at addresses corresponding to a group of pixels within an area including the pixel at the arbitrary position. For example, by reducing α under high light levels (high count values) and increasing α under low light levels (low count values), it is possible to improve the SNR and suppress subject blur at the same time.

[0051] Next, a photoelectric conversion device 100 according to a third embodiment will be described. Matters not mentioned in the third embodiment may follow the first and second embodiments. FIG. 10 is a diagram illustrating the configuration of an image processing unit 120 in a photoelectric conversion device 100 according to the third embodiment. In the third embodiment, the photoelectric conversion device 100 includes a receiving unit 430 that receives a trigger signal TRIG from outside the photoelectric conversion device 100. The output unit 407 of the image processing unit 120 outputs one frame of data stored in the memory 403 in response to activation of the trigger signal TRIG. In one example, the memory controller 406 may operate to read one frame of data from the memory 403 in response to activation of the trigger signal TRIG and output the data to the outside via the output unit 407. The memory 403 and the output unit 407 may be connected by a data bus. In this case, the data output from the memory 403 may be provided to the output unit 407 without passing through the memory controller 406, and may be output from the output unit 407 to the outside.

[0052] FIG. 11 exemplifies the operation of the photoelectric conversion device 100 of the second embodiment. In response to activation of the trigger signal TRIG (transition to high level in the example of FIG. 11), the memory controller 406 can operate to read one frame of data from the memory 403 and output it to the outside via the output unit 407. In one example, the memory controller 406 can start an operation to read one frame of data from the memory 403 and output it to the outside via the output unit 407 during the subframe period following the subframe in which the trigger signal TRIG is activated. The memory controller 406 can operate such that the data to be read is not updated during the period in which one frame of data is read from the memory 403 and output to the outside. Alternatively, depending on the application, the memory controller 406 may operate such that the data to be read is updated during the period in which one frame of data is read from the memory 403 and output to the outside.

[0053] Next, a photoelectric conversion device 100 according to a fourth embodiment will be described. Matters not mentioned in the fourth embodiment may follow those of the first to third embodiments. FIG. 12 is a diagram showing the configuration of an image processing unit 120 in a photoelectric conversion device 100 according to the fourth embodiment. In the fourth embodiment, the image processing unit 120 includes a coefficient determination unit 408, but the coefficient determination unit 408 of the fourth embodiment is different from the coefficient determination unit 408 of the second embodiment.

[0054] The coefficient determination unit 408 may be configured to determine the first coefficient and the second coefficient based on a comparison between a count value obtained by expanding the number of bits of the count value of a pixel at an arbitrary position in the pixel array region 12 by the data converter 402 and data stored at a corresponding address in the memory 403. In the example shown in FIG. 12, the first coefficient is 1-α, the second coefficient is α, and the sum of the first coefficient and the second coefficient is 1. However, the first coefficient and the second coefficient may be independent coefficients that are not correlated with each other. The first coefficient may be a value greater than 0 and less than 1. The second coefficient may be a value greater than 0 and less than 1.

[0055] For example, when the difference between the count value provided by the data converter 402 and the value of the data provided by the memory 403 is within a predetermined value, the coefficient determination unit 408 increases the value of α above the reference value to strengthen the smoothing effect. On the other hand, when this difference is greater than the predetermined value (when the change in light amount is deemed to be large), the coefficient determination unit 408 decreases the value of α below the reference value to prioritize tracking of the change in light amount. With this operation, α (i.e., the first coefficient and the second coefficient) is set according to the magnitude of the change in light amount in the scene, making it possible to achieve both an improvement in SNR and suppression of subject blur.

[0056] The image processing unit 120 may include a control unit 412. The control unit 412 may, for example, detect an abnormality in the data (image) held by the memory 403, and accordingly reset all or part of the multiple pixels 101 in the pixel array region (e.g., the counter circuit 211), the readout circuit 112, and the memory 403. The control unit 412 may control the reset operation so that the multiple pixels 101 in the pixel array region 12 are reset more frequently than the memory 403. When the control unit 412 detects an abnormality in the data (image) held by the memory 403, it may execute other processing, for example, noise removal processing.

[0057] Next, a photoelectric conversion device 100 according to a fifth embodiment will be described. Matters not mentioned in the fifth embodiment may follow those of the first to fourth embodiments. Fig. 13 is a diagram showing the configuration of an image processing unit 120 in the photoelectric conversion device 100 according to the fifth embodiment. The fifth embodiment provides a modified example of the coefficient determination unit 408 of the fourth embodiment.

[0058] As in the fourth embodiment, the coefficient determination unit 408 may be configured to determine the first coefficient and the second coefficient based on a comparison between the count value provided by the data converter 402 and the data provided by the memory 403. The coefficient determination unit 408 may also generate information corresponding to the difference (i.e., the change in light amount) between the count value obtained by expanding the number of bits of the count value of a pixel at an arbitrary position in the pixel array region 12 by the data converter 402 and the data stored at a corresponding address in the memory 403. The information may be output to the outside via the output unit 407. The coefficient determination unit 408 may generate binary information indicating whether the difference is greater than a reference value.

[0059] Next, a photoelectric conversion device 100 according to a sixth embodiment will be described. Matters not mentioned in the sixth embodiment may follow the first to fifth embodiments. FIG. 14 is a diagram illustrating the configuration of an image processing unit 120 in a photoelectric conversion device 100 according to the sixth embodiment. In the sixth embodiment, the image processing unit 120 includes a third computing unit (third multiplier) 410. The first computing unit 410 may be configured to multiply a count value generated in each pixel 101 by a first coefficient (γ here) to generate a first computation value. The third computing unit 410 may be configured to multiply data stored in the memory 403 during a first period by a third coefficient (β here) to generate a third computation value. The second computing unit 420 may be configured to multiply data stored in the memory 403 during a second period that precedes the first period by a second coefficient (α here) to generate a second computation value. The adder 401 may be configured to add the first computation value, the second computation value, and the third computation value to generate a sum.

[0060] Here, during the first period, data is written to a first region 403a of the memory 403, and during the second period, data is written to a second region 403b of the memory 403, and the data in the first region 403a may be transferred to the second region 403b between the first and second periods. From another perspective, the latest version of the sum generated by the adder 401 may be written to the first region 403a, and before that, the data in the first region 403a may be transferred to the second region 403b. The memory constituting the first region 403a and the memory constituting the second region 403b may be separate. In another control method, the first region 403a and the second region 403b may be swapped for each subframe, in which case the transfer of data from the first region 403a to the second region 403b is not necessary.

[0061] According to the sixth embodiment, data processed by the TIIR filter can be obtained using data of the current subframe and data of two subframes before the current subframe, thereby obtaining data with an improved SNR.

[0062] Modifications or specific examples of the first to sixth embodiments will be described below. FIG. 15 is a diagram showing a modification of the first computing unit 410. a[3:0] is data (4 bits in this example) indicating a count value provided from a pixel 101 at an arbitrary position via the readout circuit 112, and b[3:0] is data (4 bits in this example) indicating a first coefficient (here, (1-α) in the above example). The first computing unit 410 generates the result (shown as a×b) of multiplying a[3:0] and b[3:0] as a first computation value. In this example, the number of bits of the first computation value generated by the first computing unit 410 is 8 bits, which is larger than the number of bits of the count value provided from each pixel 101 via the signal line 113 and the readout circuit 112, which is 4 bits.

[0063] 16 is a diagram showing a specific example of the data converter 402 in the first computing unit 410. The data converter 402 may include at least one of a nonlinear correction unit 422 and a smoothing processing unit 423 in addition to a bit width expansion unit 421, for example.

[0064] FIG. 17 schematically illustrates the bit-width expansion process performed by the bit-width expansion unit 421. The bit-width expansion unit 421 adds one or more bits (four bits in this example) to the lower-order side of the count value (four bits in this example) provided by the pixel 101. The bit expansion illustrated in FIG. 17 may be achieved by providing an n-bit count value to the upper-order side of an m-bit input terminal of a downstream circuit and providing (mn) bits of "0" to the lower-order side. Note that m and n have a relationship of m>n. FIG. 18 schematically illustrates the nonlinear correction process performed by the nonlinear correction unit 422. As illustrated in FIG. 18, the pixel 101 may have a nonlinear characteristic in which the number of input photons and the output level (count value) are nonlinear. The nonlinear correction unit 422 may correct the count value provided by the pixel 101 so as to obtain an output level (count value) proportional to the number of input photons. FIG. 19 schematically illustrates the spatial smoothing process performed by the smoothing processing unit 423. Examples of smoothing processing include convolution operations including a moving average filter or a Gaussian filter, nonlinear operations including a median filter, and noise removal processing using a Fourier transform or a wavelet transform, but other methods may also be used.

[0065] Fig. 20 shows the configuration of a photoelectric conversion system 1000 according to a seventh embodiment. In the seventh embodiment, a signal processing device 120' corresponding to the image processing unit 120 described above is arranged outside the photoelectric conversion device 100. Fig. 7 is used as a diagram showing the operation of the photoelectric conversion system 1000 shown in Fig. 20, and "output" indicates the output of the signal processing device 120'. Furthermore, signals from pixels in a row selected in "row selection" are read out by the readout circuit 112 and output from the photoelectric conversion device 100 to the signal processing device 120' via the output unit 114.

[0066] The photoelectric conversion device 100 may include a plurality of pixels 101 that count photons and output count values, a readout circuit 112 that reads out the count values generated in the plurality of pixels 101, and an output unit 114 that outputs the count values read out by the readout circuit 112. The photoelectric conversion device 100 may also include a control pulse generation unit 115, a horizontal scanning circuit 111, a signal line 113, and a vertical scanning circuit 110.

[0067] The signal processing device 120′ may include a memory 403, a first computing unit 410, a second computing unit 420, an adder 401, and a memory controller 406. The configurations and functions of the memory 403, the first computing unit 410, the second computing unit 420, the adder 401, and the memory controller 406 may be similar to those described above.

[0068] In the above-described embodiments, the count value generated by the pixel 101 has been described as a signal having a resolution corresponding to the incidence of one photon. However, this is not limiting. For example, the pixel 101 may output a count value whose signal value increases by one when multiple photons are incident. An example of such a configuration is a configuration in which a detection circuit for detecting a pulse signal output by the waveform shaping unit 210 is provided between the waveform shaping unit 210 and the counter circuit 211 in the pixel 101 shown in FIG. 4 . In this configuration, when multiple pulse signals are input to the detection circuit, the detection circuit outputs a signal to the counter circuit 211. The counter circuit 211 counts this signal. As a result, 1 LSB (Lowest Significant Bit) of the count value generated by the counter circuit 211 corresponds to the incidence of multiple photons. In this case, the signal generated by the data converter 402 has a resolution smaller than the multiple photons corresponding to 1 LSB of the count value generated by the counter circuit 211.

[0069] Hereinafter, application examples of the photoelectric conversion device 100 according to the first to seventh embodiments will be described. When the seventh embodiment is applied, the signal processing device 120′ can constitute a part of a signal processing device that processes a signal output from an imaging device (photoelectric conversion device), such as the signal processing unit 1007, the image processing unit 1312, the image processing circuit 1404, the signal processing unit 352, etc.

[0070] FIG. 21 is a block diagram showing a schematic configuration of a photoelectric conversion system according to the first application example.

[0071] The above-described photoelectric conversion device 100 can be applied to various photoelectric conversion systems. Examples of applicable photoelectric conversion systems include digital still cameras, digital camcorders, surveillance cameras, copiers, fax machines, mobile phones, vehicle-mounted cameras, and observation satellites. Camera modules equipped with an optical system such as a lens and an imaging device are also included in photoelectric conversion systems. Fig. 21 illustrates a block diagram of a digital still camera as an example of such systems.

[0072] 21 includes an imaging device 1004, which is an example of a photoelectric conversion device. The photoelectric conversion system 1000 also includes a lens 1002 that forms an optical image of a subject on the imaging device 1004, an aperture 1003 that adjusts the amount of light passing through the lens 1002, and a barrier 1001 that protects the lens 1002. The lens 1002 and the aperture 1003 form an optical system (optical device) that focuses light on the imaging device 1004. The imaging device 1004 is the photoelectric conversion device 100 (imaging device) of any of the above embodiments, and converts the optical image formed by the lens 1002 into an electrical signal.

[0073] The photoelectric conversion system 1000 also has a signal processing unit 1007, which is an image generation unit that generates an image by processing an output signal output by the imaging device 1004. The signal processing unit 1007 functions as a processing device that performs various corrections and compressions as necessary and outputs image data. The signal processing unit 1007 may be formed on the same semiconductor substrate on which the imaging device 1004 is provided, or may be formed on a semiconductor substrate separate from the imaging device 1004. Furthermore, the imaging device 1004 and the signal processing unit 1007 may be formed on the same semiconductor substrate.

[0074] The photoelectric conversion system 1000 further includes a memory unit 1010 for temporarily storing image data, and an external interface unit (external I / F unit) 1013 for communicating with an external computer or the like. The photoelectric conversion system 1000 also includes a recording medium 1012 such as a semiconductor memory for recording or reading out imaging data, and a recording medium control interface unit (recording medium control I / F unit) 1011 for recording or reading out data from the recording medium 1012. The recording medium control I / F unit 1011 and the recording medium 1012 may form part of a storage device. The recording medium 1012 may be built into the photoelectric conversion system 1000, or may be removable.

[0075] The photoelectric conversion system 1000 further includes an overall control and calculation unit 1009 that performs various calculations and controls the entire digital still camera, and a timing generation unit 1008 that outputs various timing signals to the image capture device 1004 and the signal processing unit 1007. The overall control and calculation unit 1009 and the timing generation unit 1008 may form part of a control device for controlling the operation of the photoelectric conversion system 1000. Here, timing signals and the like may be input from outside, and the photoelectric conversion system 1000 only needs to include at least the image capture device 1004 and the signal processing unit 1007 that processes the output signal output from the image capture device 1004.

[0076] The imaging device 1004 outputs an imaging signal to a signal processing unit 1007. The signal processing unit 1007 performs predetermined signal processing on the imaging signal output from the imaging device 1004 and outputs image data. The signal processing unit 1007 generates an image using the imaging signal. Although not shown in FIG. 21 , a display device such as a display for displaying the generated image may be provided in the photoelectric conversion system 1000. As described above, according to this embodiment, it is possible to realize a photoelectric conversion system 1000 to which the photoelectric conversion device 100 (imaging device) of any of the above embodiments is applied.

[0077] 22(a) and 22(b) are diagrams showing the configurations of a photoelectric conversion system 1300 and a moving object 1301 according to the second application example.

[0078] FIG. 21(a) shows an example of a photoelectric conversion system related to an in-vehicle camera. The photoelectric conversion system 1300 includes an image capturing device 1310. The image capturing device 1310 is the photoelectric conversion device 100 (image capturing device) described in any of the above embodiments. The photoelectric conversion system 1300 includes an image processing unit 1312 that performs image processing on multiple pieces of image data acquired by the image capturing device 1310. The photoelectric conversion system 1300 also includes a distance acquisition unit 1316 that calculates the distance to an object, and a collision determination unit 1318 that determines whether or not there is a possibility of a collision based on the calculated distance. Here, the distance acquisition unit 1316 may acquire distance information to the object using a Time of Flight (ToF) method, or may acquire distance information using parallax information or the like. That is, the distance information is information related to parallax, defocus amount, distance to the object, etc. The collision determination unit 1318 may determine the possibility of a collision using any of this distance information. The distance acquisition unit 1316 may be realized by dedicated hardware or a software module. The distance acquisition unit 1316 may also be realized by a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or a combination thereof.

[0079] The photoelectric conversion system 1300 is connected to a vehicle information acquisition device 1320 and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. The photoelectric conversion system 1300 is also connected to an ECU 1330, which is a control device that outputs a control signal to generate a braking force for the vehicle based on the determination result of a collision determination unit 1318. The photoelectric conversion system 1300 is also connected to an alarm device 1340 that issues an alarm to the driver based on the determination result of the collision determination unit 1318. For example, if the determination result of the collision determination unit 1318 indicates a high collision possibility, the ECU 1330 controls the drive device (mechanical device) 1360 by applying the brakes, releasing the accelerator, suppressing engine output, or other vehicle control to avoid the collision and mitigate damage. The alarm device 1340 warns the user by sounding an alarm, displaying alarm information on a screen of a car navigation system, or vibrating a seat belt or steering wheel.

[0080] In this embodiment, the surroundings of a vehicle (mobile body 1301), for example, the front or rear, are imaged by a photoelectric conversion system 1300. FIG. 21(b) shows a photoelectric conversion system configured to image the area in front of the vehicle (image capture range 1350). A vehicle information acquisition device 1320 sends instructions to the photoelectric conversion system 1300 or the image capture device 1310. This configuration can further improve the accuracy of distance measurement.

[0081] While the above describes an example of control to prevent collisions with other vehicles, the photoelectric conversion system 1300 can also be applied to other autonomous driving control systems, such as control systems that automatically follow other vehicles and automatically drive vehicles to avoid drifting out of their lanes. Furthermore, the photoelectric conversion system 1300 can be applied not only to vehicles such as automobiles, but also to moving bodies (mobile devices) such as ships, aircraft, and industrial robots. The moving body includes one or both of a driving force generating unit that generates a driving force primarily used to move the moving body and a rotating body primarily used to move the moving body. The driving force generating unit can be an engine, a motor, or the like. The rotating body can be a tire, a wheel, a ship's screw, an aircraft's propeller, or the like. In addition to moving bodies, the photoelectric conversion system 1300 can be applied to a wide range of devices that use object recognition, such as intelligent transport systems (ITS).

[0082] FIG. 23 is a block diagram showing an example of the configuration of a range image sensor 1401 which is a photoelectric conversion system of the third application example.

[0083] 23, the range image sensor 1401 is configured to include an optical system 1407, a photoelectric conversion device 1408, an image processing circuit 1404, a monitor 1405, and a memory 1406. The range image sensor 1401 can obtain a range image according to the distance to the subject by receiving light (modulated light or pulsed light) that is projected toward the subject from a light source device 1409 and reflected from the surface of the subject.

[0084] The optical system 1407 is configured to have one or more lenses, and guides image light (incident light) from a subject to the photoelectric conversion device 1408, forming an image on the light receiving surface (sensor section) of the photoelectric conversion device 1408.

[0085] The photoelectric conversion device 1408 is the photoelectric conversion device 100 according to each of the above-described embodiments, and a distance signal indicating a distance determined from a light receiving signal output from the photoelectric conversion device 1408 is supplied to the image processing circuit 1404.

[0086] The image processing circuit 1404 performs image processing to construct a distance image based on the distance signal supplied from the photoelectric conversion device 1408. The distance image (image data) obtained by this image processing is then supplied to a monitor 1405 for display, or supplied to a memory 1406 for storage (recording).

[0087] In the range image sensor 1401 configured in this way, by applying the above-described photoelectric conversion device 100, it is possible to obtain, for example, a more accurate range image as the pixel characteristics improve.

[0088] FIG. 24 is a diagram showing an example of a schematic configuration of an endoscopic surgery system 1250 which is a photoelectric conversion system according to the fourth application example.

[0089] 24 shows a state in which an operator (doctor) 1231 is performing surgery on a patient 1232 on a patient bed 1233 using an endoscopic surgery system 1250. As shown in the figure, the endoscopic surgery system 1250 includes an endoscope 1200, a surgical tool 1210, and a cart 1234 on which various devices for endoscopic surgery are mounted.

[0090] The endoscope 1200 includes a lens barrel 1201, a region of which a predetermined length from the tip is inserted into a body cavity of a patient 1232, and a camera head 1202 connected to the base end of the lens barrel 1201. In the illustrated example, the endoscope 1200 is configured as a so-called rigid lens barrel having a rigid lens barrel 1201, but the endoscope 1200 may also be configured as a so-called flexible lens barrel having a flexible lens barrel.

[0091] An opening into which an objective lens is fitted is provided at the tip of the lens barrel 1201. A light source device 1203 is connected to the endoscope 1200, and light generated by the light source device 1203 is guided to the tip of the lens barrel by a light guide extending inside the lens barrel 1201, and is irradiated via the objective lens towards an observation target inside a body cavity of a patient 1232. The endoscope 1200 may be a direct-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0092] An optical system and a photoelectric conversion device are provided inside the camera head 1202, and light reflected from an object to be observed (observation light) is focused onto the photoelectric conversion device by the optical system. The observation light is photoelectrically converted by the photoelectric conversion device to generate an electrical signal corresponding to the observation light, i.e., an image signal corresponding to an observation image. The photoelectric conversion device may be the photoelectric conversion device 100 (image capture device) described in each of the above-described embodiments. The image signal is transmitted as RAW data to a camera control unit (CCU) 1235.

[0093] The CCU 1235 is configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and comprehensively controls the operations of the endoscope 1200 and the display device 1236. Furthermore, the CCU 1235 receives an image signal from the camera head 1202 and performs various image processing on the image signal, such as development processing (demosaic processing), to display an image based on the image signal.

[0094] Under the control of the CCU 1235 , the display device 1236 displays an image based on the image signal that has been subjected to image processing by the CCU 1235 .

[0095] The light source device 1203 is configured from a light source such as an LED (Light Emitting Diode), and supplies the endoscope 1200 with irradiation light when photographing an operation site or the like.

[0096] The input device 1237 is an input interface for the endoscopic surgery system 1250. A user can input various information and instructions to the endoscopic surgery system 1250 via the input device 1237.

[0097] The treatment tool control device 1238 controls the driving of the energy treatment tool 1212 for cauterizing tissue, incising, sealing blood vessels, or the like.

[0098] The light source device 1203, which supplies illumination light to the endoscope 1200 when photographing the surgical site, can be configured from a white light source configured from, for example, an LED, a laser light source, or a combination of these. When the white light source is configured from a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, making it possible to adjust the white balance of the captured image in the light source device 1203. In this case, it is also possible to capture images corresponding to each RGB color in a time-division manner by irradiating the object of observation with laser light from each RGB laser light source in a time-division manner and controlling the drive of the image sensor of the camera head 1202 in synchronization with the irradiation timing. According to this method, a color image can be obtained without providing a color filter to the image sensor.

[0099] Furthermore, the light source device 1203 may be controlled to change the intensity of light it outputs at predetermined time intervals. By controlling the driving of the image sensor of the camera head 1202 in synchronization with the timing of the change in light intensity to acquire images in a time-division manner and combining the images, it is possible to generate an image with a high dynamic range that is free of so-called blocked-up shadows and blown-out highlights.

[0100] The light source device 1203 may also be configured to supply light in a predetermined wavelength band corresponding to special light observation. Special light observation utilizes, for example, the wavelength dependency of light absorption in body tissue. Specifically, specific tissue, such as blood vessels on the surface of the mucosa, can be photographed with high contrast by irradiating light with a narrower band than the light (i.e., white light) used in normal observation. Alternatively, special light observation may involve fluorescence observation, in which an image is obtained using fluorescence generated by irradiating excitation light. Fluorescence observation may involve irradiating excitation light onto body tissue and observing the fluorescence from the body tissue, or locally injecting a reagent such as indocyanine green (ICG) into the body tissue and irradiating the body tissue with excitation light corresponding to the fluorescent wavelength of the reagent to obtain a fluorescent image. The light source device 1203 may be configured to supply narrow band light and / or excitation light corresponding to such special light observation.

[0101] 25(a) and 25(b) illustrate glasses 1600 (smart glasses) that are a photoelectric conversion system according to a fifth application example. The glasses 1600 include a photoelectric conversion device 1602. The photoelectric conversion device 1602 is the photoelectric conversion device 100 (imaging device) described in each of the above embodiments. A display device including a light-emitting device such as an OLED or LED may be provided on the rear surface of the lens 1601. The photoelectric conversion device 1602 may be one or more. A combination of multiple types of photoelectric conversion devices may also be used. The arrangement of the photoelectric conversion device 1602 is not limited to that shown in FIG. 25(a).

[0102] The glasses 1600 further include a control device 1603. The control device 1603 functions as a power source that supplies power to the photoelectric conversion device 1602 and the display device. The control device 1603 also controls the operations of the photoelectric conversion device 1602 and the display device. The lens 1601 is formed with an optical system for focusing light onto the photoelectric conversion device 1602.

[0103] FIG. 25(b) illustrates glasses 1610 (smart glasses) according to a fifth application example. The glasses 1610 include a control device 1612, which includes a photoelectric conversion device equivalent to the photoelectric conversion device 1602 and a display device. A lens 1611 includes an optical system for projecting light emitted from the photoelectric conversion device and the display device, and an image is projected onto the lens 1611. The control device 1612 functions as a power source for supplying power to the photoelectric conversion device and the display device, and controls the operation of the photoelectric conversion device and the display device. The control device may include a gaze detection unit for detecting the gaze of the wearer. Infrared light may be used for gaze detection. The infrared light emitter emits infrared light toward the eyeball of a user gazing at a displayed image. An imaging unit with a light-receiving element detects the reflected infrared light from the eyeball, thereby obtaining an image of the eyeball. A reduction unit for reducing light from the infrared light emitter to the display unit in a planar view reduces degradation of image quality.

[0104] The gaze of the user relative to the displayed image is detected from an image of the eyeball captured using infrared light. Any known method can be used for gaze detection using an image of the eyeball. As an example, a gaze detection method based on a Purkinje image formed by reflection of irradiated light on the cornea can be used.

[0105] More specifically, gaze detection processing is performed based on the pupil-corneal reflex method, which calculates a gaze vector representing the direction (rotation angle) of the eyeball based on the pupil image and Purkinje image included in the captured image of the eyeball, thereby detecting the user's gaze.

[0106] The display device of this embodiment may have a photoelectric conversion device having a light receiving element, and may control the image displayed on the display device based on information on the user's line of sight from the photoelectric conversion device.

[0107] Specifically, the display device determines a first field of view area where the user gazes and a second field of view area other than the first field of view area based on the line-of-sight information. The first field of view area and the second field of view area may be determined by a control device of the display device, or may be determined by an external control device and received. In the display area of the display device, the display resolution of the first field of view area may be controlled to be higher than the display resolution of the second field of view area. In other words, the resolution of the second field of view area may be lower than that of the first field of view area.

[0108] The display area may include a first display area and a second display area different from the first display area, and a high-priority area may be determined from the first display area and the second display area based on line-of-sight information. The first field of view area and the second field of view area may be determined by a control device of the display device, or may be determined by an external control device and received. The resolution of the high-priority area may be controlled to be higher than the resolution of areas other than the high-priority area. In other words, the resolution of an area with a relatively low priority may be lowered.

[0109] Note that AI may be used to determine the first field of view area and areas with high priority. The AI may be a model configured to estimate the angle of gaze and the distance to an object in the line of sight from an image of the eyeball, using as training data an image of the eyeball and the direction in which the eyeball in the image was actually looking. The AI program may be included in the display device, the photoelectric conversion device, or an external device. If included in an external device, it is transmitted to the display device via communication.

[0110] When display control is performed based on visual recognition detection, the present invention is preferably applied to smart glasses that further include a photoelectric conversion device for capturing images of the outside world. The smart glasses can display captured external information in real time.

[0111] 26(a) and 26(b) are diagrams showing an electronic device of Application Example 6. The electronic device may be configured as, for example, a smartphone or a tablet.

[0112] 26(a) shows the front side of electronic device 1500, and FIG. 26(b) shows the rear side of electronic device 1500. In FIG.

[0113] 26(a), a display 1510 for displaying an image is disposed in the center of the surface of the electronic device 1500. Then, along the upper side of the surface of the electronic device 1500, front cameras 1521 and 1522 using the above-described photoelectric conversion device 100, an IR light source 1530 for emitting infrared light, and a visible light source 1540 for emitting visible light are disposed.

[0114] Also, as shown in Figure 26(b), along the upper edge of the back of the electronic device 1500, rear cameras 1551 and 1552 using the above-mentioned photoelectric conversion device 100, an IR light source 1560 that emits infrared light, and a visible light source 1570 that emits visible light are arranged.

[0115] In the electronic device 1500 configured as described above, by applying the above-described photoelectric conversion device 100, it is possible to capture, for example, higher quality images. Note that the photoelectric conversion device can also be applied to other electronic devices such as infrared sensors, distance measuring sensors using active infrared light sources, security cameras, and personal or biometric authentication cameras. This can improve the accuracy and performance of these electronic devices.

[0116] 27 is a block diagram of an X-ray CT apparatus according to a seventh application example. The photoelectric conversion device 100 described above is applicable to the detector of the X-ray CT apparatus. The X-ray CT apparatus 30 according to this embodiment includes an X-ray generation unit 310, a wedge 316, a collimator 318, an X-ray detection unit 320, a top plate 330, a rotating frame 340, and a high-voltage generation device 350. The X-ray CT apparatus 30 also includes a data acquisition system (DAS) 351, a signal processing unit 352, a display unit 353, and a control unit 354.

[0117] The X-ray generating unit 310 is composed of, for example, a vacuum tube that generates X-rays. A high voltage and a filament current are supplied to the vacuum tube of the X-ray generating unit 310 from a high voltage generator 350. X-rays are generated by irradiating the anode (target) with thermoelectrons from the cathode (filament).

[0118] The wedge 316 is a filter that adjusts the amount of X-rays irradiated from the X-ray generation unit 310. The wedge 316 attenuates the amount of X-rays so that the X-rays irradiated from the X-ray generation unit 310 to the subject have a predetermined distribution. The collimator 318 is made of a lead plate or the like that narrows the irradiation range of the X-rays that have passed through the wedge 316. The X-rays generated by the X-ray generation unit 310 are shaped into a cone beam via the collimator 318 and are irradiated onto the subject on the tabletop 330.

[0119] The X-ray detection unit 320 is configured using the above-mentioned photoelectric conversion device 100. The X-ray detection unit 320 detects X-rays emitted from the X-ray generation unit 310 and passed through the subject, and outputs a signal corresponding to the X-ray dose to the DAS 351.

[0120] The rotating frame 340 has an annular shape and is configured to be rotatable. An X-ray generation unit 310 (wedge 316, collimator 318) and an X-ray detection unit 320 are arranged facing each other inside the rotating frame 340. The X-ray generation unit 310 and the X-ray detection unit 320 can rotate together with the rotating frame 340.

[0121] The high voltage generator 350 includes a booster circuit and outputs a high voltage to the X-ray generation unit 310. The DAS 351 includes an amplifier circuit and an A / D conversion circuit and outputs a signal from the X-ray detection unit 320 to the signal processing unit 352 as digital data.

[0122] The signal processing unit 352 includes a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory), and is capable of performing image processing on digital data. The display unit 353 includes a flat display device and is capable of displaying X-ray images. The control unit 354 includes a CPU, ROM, RAM, and the like, and controls the overall operation of the X-ray CT device 30.

[0123] The above-described embodiments can be modified as appropriate without departing from the spirit of the present invention. The disclosure of this specification includes not only what is described herein but also all matters that can be understood from the specification and the accompanying drawings. The disclosure of this specification also includes the complement of the concepts described herein. In other words, if the specification contains a statement that "A is greater than B," even if the statement that "A is not greater than B" is omitted, the specification can still be said to disclose that "A is not greater than B." This is because the statement that "A is greater than B" presupposes that the case in which "A is not greater than B" is taken into consideration.

[0124] The disclosure of the present specification includes the following configurations. (Item 1) A photoelectric conversion device having a plurality of pixels that count photons and output count values, Memory and a first calculator that multiplies a count value generated for each pixel by a first coefficient to generate a first calculation value; a second calculator that multiplies the data provided from the memory by a second coefficient to generate a second calculation value; an adder that adds the first calculation value and the second calculation value to generate a sum; a memory controller that updates data stored in the memory in accordance with the added value; the number of bits of the first calculation value generated by the first calculation unit is greater than the number of bits of the count value; A photoelectric conversion device characterized by: (Item 2) the number of bits of the second operation value generated by the second operation unit and the number of bits of the sum value generated by the adder are greater than the number of bits of the count value; 2. The photoelectric conversion device according to item 1, (Item 3) the first computing unit includes: a data converter that expands the number of bits of the count value provided from each pixel; and a multiplier that multiplies the count value, the number of bits of which has been expanded by the data converter, by the first coefficient to generate the first computed value. 3. The photoelectric conversion device according to item 1 or 2, (Item 4) the data converter performs nonlinear correction processing after expanding the number of bits of the count value provided from each pixel; 4. The photoelectric conversion device according to item 3, (Item 5) the data converter performs nonlinear correction processing and smoothing processing after expanding the number of bits of the count value provided from each pixel; 4. The photoelectric conversion device according to item 3, (Item 6) the memory controller compresses the sum generated by the adder to generate compressed data, and updates the value stored in the memory based on the compressed data; 6. The photoelectric conversion device according to any one of items 1 to 5, characterized in that: (Item 7) the second computing unit multiplies data obtained by decompressing the compressed data provided from the memory by the second coefficient to generate the second computation value. 7. The photoelectric conversion device according to item 6, (Item 8) the first coefficient is a value greater than 0 and less than 1, The second coefficient is a value greater than 0 and less than 1. 8. The photoelectric conversion device according to any one of items 1 to 7, characterized in that: (Item 9) the sum of the first coefficient and the second coefficient is 1; 9. The photoelectric conversion device according to any one of items 1 to 8, characterized in that: (Item 10) further comprising a coefficient determination unit that determines the first coefficient and the second coefficient based on data stored in the memory; 10. The photoelectric conversion device according to any one of items 1 to 9, characterized in that: (Item 11) further comprising a coefficient determination unit that determines the first coefficient and the second coefficient based on the count value provided from each pixel and data stored in the memory, 10. The photoelectric conversion device according to any one of items 1 to 9, characterized in that: (Item 12) further comprising a coefficient determination unit that determines the first coefficient and the second coefficient based on a difference between the count value provided from each pixel and the data stored in the memory; 10. The photoelectric conversion device according to any one of items 1 to 9, characterized in that: (Item 13) a receiving unit for receiving a trigger signal; an output unit that outputs one frame of data stored in the memory in response to activation of the trigger signal by the receiving unit; 13. The photoelectric conversion device according to any one of items 1 to 12, further comprising: (Item 14) 13. The photoelectric conversion device according to any one of items 1 to 12, further comprising an output unit that outputs information corresponding to the difference between the count value provided from each pixel and the data stored in the memory. (Item 15) Further comprising a third computing unit, the third computing unit multiplies the data stored in the memory during the first period by a third coefficient to generate a third computation value; the second computing unit multiplies data stored in the memory for a second period earlier than the first period by a second coefficient to generate the second computed value; the adder adds the first calculation value, the second calculation value, and the third calculation value to generate the sum; 15. The photoelectric conversion device according to any one of items 1 to 14, (Item 16) The frequency of resetting the plurality of pixels is higher than the frequency of resetting the memory. 16. The photoelectric conversion device according to any one of items 1 to 15, (Item 17) the memory controller determines an address of the memory to be updated according to the added value based on a correction signal for correcting image blur caused by vibration of the imaging device; 17. The photoelectric conversion device according to any one of items 1 to 16, (Item 18) Each pixel includes an avalanche photodiode. 8. The photoelectric conversion device according to any one of items 1 to 7, characterized in that: (Item 19) Each pixel further includes a waveform shaping unit connected to the avalanche photodiode and a counter circuit that generates a first count value by counting the pulse signal output by the waveform shaping unit; the first computing unit includes: a data converter that expands the number of bits of the first count value provided from each pixel; and a multiplier that multiplies the count value, the number of bits of which has been expanded by the data converter, by the first coefficient to generate the first computation value. Item 19. The photoelectric conversion device according to item 18, (Item 20) the plurality of pixels, the memory, the first computing unit, the second computing unit, the adder, and the memory controller are arranged on one semiconductor chip. 20. The photoelectric conversion device according to any one of items 1 to 19, characterized in that: (Item 21) In the one semiconductor chip, the plurality of pixels are arranged in one semiconductor layer, and at least one of the memory, the first computing unit, the second computing unit, the adder, and the memory controller is arranged in another semiconductor layer. 21. The photoelectric conversion device according to item 20, (Item 22) The photoelectric conversion device according to any one of items 1 to 21, a signal processing unit that processes a signal output from the photoelectric conversion device; A photoelectric conversion system comprising: (Item 23) A mobile object including the photoelectric conversion device according to any one of items 1 to 21, A moving body comprising a control unit that controls the movement of the moving body using a signal output from the photoelectric conversion device. (Item 24) A photoelectric conversion system including a photoelectric conversion device and a signal processing device that processes a signal output from the photoelectric conversion device, the photoelectric conversion device includes a plurality of pixels that count photons and output count values, a readout circuit that reads out the count values generated in the plurality of pixels, and an output unit that outputs the count values read out by the readout circuit; The signal processing device includes: Memory and a first calculator that multiplies a count value generated for each pixel by a first coefficient to generate a first calculation value; a second calculator that multiplies the data provided from the memory by a second coefficient to generate a second calculation value; an adder that adds the first calculation value and the second calculation value to generate a sum; a memory controller that updates data stored in the memory in accordance with the added value; the number of bits of the first calculation value generated by the first calculation unit is greater than the number of bits of the count value; A photoelectric conversion system comprising: (Item 25) A mobile object equipped with the photoelectric conversion system according to item 24, A moving body comprising a control unit that controls the movement of the moving body using a signal output by the photoelectric conversion system.

[0125] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0126] 100: Photoelectric conversion device, 401: Adder, 402: Data converter, 403: Memory, 404: First multiplier, 405: Second multiplier, 406: Memory controller, 407: Output unit, 410: First arithmetic unit, 420: Second arithmetic unit,

Claims

1. A photoelectric conversion device having a plurality of pixels that count photons and output count values, Memory and a first calculator that multiplies a count value generated for each pixel by a first coefficient to generate a first calculated value; a second calculator that multiplies the data provided from the memory by a second coefficient to generate a second calculation value; an adder that adds the first calculation value and the second calculation value to generate a sum; a memory controller that updates data stored in the memory in accordance with the added value; the number of bits of the first calculation value generated by the first calculation unit is greater than the number of bits of the count value; A photoelectric conversion device characterized by:

2. the number of bits of the second operation value generated by the second operation unit and the number of bits of the sum value generated by the adder are greater than the number of bits of the count value; 2. The photoelectric conversion device according to claim 1.

3. the first computing unit includes: a data converter that expands the number of bits of the count value provided from each pixel; and a multiplier that multiplies the count value, the number of bits of which has been expanded by the data converter, by the first coefficient to generate the first computed value.

2. The photoelectric conversion device according to claim 1.

4. the data converter performs nonlinear correction processing after expanding the number of bits of the count value provided from each pixel; 4. The photoelectric conversion device according to claim 3.

5. the data converter performs nonlinear correction processing and smoothing processing after expanding the number of bits of the count value provided from each pixel; 4. The photoelectric conversion device according to claim 3.

6. the memory controller compresses the sum generated by the adder to generate compressed data, and updates the value stored in the memory based on the compressed data; 2. The photoelectric conversion device according to claim 1.

7. the second computing unit multiplies data obtained by decompressing the compressed data provided from the memory by the second coefficient to generate the second computation value; 7. The photoelectric conversion device according to claim 6.

8. the first coefficient is a value greater than 0 and less than 1, the second coefficient is a value greater than 0 and less than 1; 2. The photoelectric conversion device according to claim 1.

9. the sum of the first coefficient and the second coefficient is 1; 2. The photoelectric conversion device according to claim 1.

10. further comprising a coefficient determination unit that determines the first coefficient and the second coefficient based on data stored in the memory; 2. The photoelectric conversion device according to claim 1.

11. a coefficient determination unit that determines the first coefficient and the second coefficient based on the count value provided from each pixel and data stored in the memory, 2. The photoelectric conversion device according to claim 1.

12. a coefficient determination unit that determines the first coefficient and the second coefficient based on a difference between the count value provided from each pixel and the data stored in the memory, 2. The photoelectric conversion device according to claim 1.

13. a receiving unit for receiving a trigger signal; an output unit that outputs one frame of data stored in the memory in response to activation of the trigger signal by the receiving unit; The photoelectric conversion device according to claim 1 , further comprising:

14. 2. The photoelectric conversion device according to claim 1, further comprising an output section that outputs information corresponding to a difference between the count value provided from each pixel and the data stored in the memory.

15. further comprising a third computing unit; the third computing unit multiplies the data stored in the memory during the first period by a third coefficient to generate a third computation value; the second computing unit multiplies data stored in the memory for a second period earlier than the first period by a second coefficient to generate the second computed value; the adder adds the first calculation value, the second calculation value, and the third calculation value to generate the sum; 2. The photoelectric conversion device according to claim 1.

16. The frequency of resetting the plurality of pixels is higher than the frequency of resetting the memory.

2. The photoelectric conversion device according to claim 1.

17. the memory controller determines an address of the memory to be updated according to the added value based on a correction signal for correcting image blur caused by vibration of the imaging device; 2. The photoelectric conversion device according to claim 1.

18. Each pixel includes an avalanche photodiode.

2. The photoelectric conversion device according to claim 1.

19. Each pixel further includes a waveform shaping unit connected to the avalanche photodiode and a counter circuit that generates a first count value by counting pulse signals output by the waveform shaping unit; the first calculator includes: a data converter that expands the number of bits of the first count value provided from each pixel; and a multiplier that multiplies the count value, the number of bits of which has been expanded by the data converter, by the first coefficient to generate the first calculation value.

19. The photoelectric conversion device according to claim 18.

20. the plurality of pixels, the memory, the first computing unit, the second computing unit, the adder, and the memory controller are arranged on one semiconductor chip.

2. The photoelectric conversion device according to claim 1.

21. In the one semiconductor chip, the plurality of pixels are arranged on one semiconductor layer, and at least one of the memory, the first computing unit, the second computing unit, the adder, and the memory controller is arranged on another semiconductor layer.

21. The photoelectric conversion device according to claim 20.

22. The photoelectric conversion device according to any one of claims 1 to 21, a signal processing unit that processes a signal output from the photoelectric conversion device; A photoelectric conversion system comprising:

23. A moving object comprising the photoelectric conversion device according to any one of claims 1 to 21, A moving body comprising a control unit that controls the movement of the moving body using a signal output from the photoelectric conversion device.

24. A photoelectric conversion system including a photoelectric conversion device and a signal processing device that processes a signal output from the photoelectric conversion device, the photoelectric conversion device includes a plurality of pixels that count photons and output count values, a readout circuit that reads out the count values generated in the plurality of pixels, and an output unit that outputs the count values read out by the readout circuit; The signal processing device includes: Memory and a first calculator that multiplies a count value generated for each pixel by a first coefficient to generate a first calculated value; a second calculator that multiplies the data provided from the memory by a second coefficient to generate a second calculation value; an adder that adds the first calculation value and the second calculation value to generate a sum; a memory controller that updates data stored in the memory in accordance with the added value; the number of bits of the first calculation value generated by the first calculation unit is greater than the number of bits of the count value; A photoelectric conversion system comprising:

25. A moving object comprising the photoelectric conversion system according to claim 24, A moving body comprising a control unit that controls the movement of the moving body using a signal output by the photoelectric conversion system.

Images