Photoelectric conversion device

Abstract

To provide a photoelectric conversion device capable of solving a problem that may occur due to a difference in structure between a detection pixel and a counting pixel.SOLUTION: A photoelectric conversion device according to one aspect of the present disclosure includes: a pixel array which includes a plurality of pixel units that are arranged in a matrix, each pixel unit including an avalanche photodiode and a counter that generates a count value based on photons incident to the avalanche photodiode; a plurality of tile regions which are obtained by dividing the pixel array in a matrix; and a storage unit which is provided in each tile region and which holds a reference value used for detecting an event based on the count value.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a photoelectric conversion device. [Background technology]

[0002] Patent Document 1 discloses a semiconductor device including pixel blocks arranged in a matrix, each of which includes a detection pixel and a counting pixel. The detection pixel detects the occurrence of an event based on a change in the amount of incident light. The counting pixel includes a single-photon avalanche diode (SPAD) and counts the number of photons incident on the SPAD. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-96347 A Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, the change in the amount of light in the detection pixel is detected based on the photocurrent generated by the photodiode. Meanwhile, in the counting pixel, the number of photons incident on the SPAD is counted. That is, the detection pixel and the counting pixel generate and process different signals. Therefore, in terms of the noise resistance of the signal, the circuit scale of each pixel, and the like, differences may occur between the detection pixel and the counting pixel included in the same pixel block.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and has an object to provide a photoelectric conversion device that can solve problems that may arise due to differences in the configurations of detection pixels and counting pixels. [Means for solving the problem]

[0006] A photoelectric conversion device according to one aspect of the present disclosure includes a pixel array including a plurality of pixel units arranged in a matrix, each pixel unit having an avalanche photodiode and a counter that generates a count value based on photons incident on the avalanche photodiode, a plurality of tile regions into which the pixel array is divided in a matrix, and a memory unit provided for each of the tile regions, for holding a reference value used to detect an event based on the count value. Effect of the Invention

[0007] According to the present disclosure, it is possible to solve problems caused by differences in the configurations of detection pixels and counting pixels. [Brief description of the drawings]

[0008] [Figure 1] 1 is a schematic diagram of a photoelectric conversion device according to a first embodiment. [Diagram 2] 3A to 3C are diagrams illustrating an example of an arrangement of a sensor substrate in the first embodiment. [Diagram 3] 1 is a block diagram showing a schematic configuration of a photoelectric conversion device according to a first embodiment. [Figure 4] FIG. 2 is a block diagram showing a schematic configuration of a tile region in the first embodiment. [Diagram 5] FIG. 2 is a diagram showing a schematic configuration of a pixel unit in the first embodiment. [Figure 6] 4 is a timing chart showing the operation of a pixel unit in the first embodiment. FIG. [Figure 7] FIG. 2 is a diagram showing an example of a layout within a tile region in the first embodiment. [Figure 8A] 4A to 4C are diagrams illustrating an example of an arrangement of event detection pixels and imaging pixels in a tile region in the first embodiment. [Figure 8B] 4A to 4C are diagrams illustrating an example of an arrangement of event detection pixels and imaging pixels in a tile region in the first embodiment. [Figure 9] 4 is a diagram showing an example of an arrangement of color filters arranged in a tile region in the first embodiment. FIG. [Figure 10]FIG. 11 is a block diagram showing a schematic configuration of a photoelectric conversion device according to a second embodiment. [Figure 11] FIG. 13 is a block diagram showing a schematic configuration of a tile region in a second embodiment. [Figure 12] FIG. 13 is a diagram showing a modified example of the pixel layout in a tile region in the second embodiment. [Figure 13] 13A and 13B are diagrams illustrating modified examples of patterns for connecting tile regions in the second embodiment. [Figure 14] FIG. 13 is a diagram showing a schematic configuration of a pixel unit in a third embodiment. [Figure 15] FIG. 11 is a timing chart showing the operation of a pixel unit in the third embodiment. [Figure 16] FIG. 13 is a block diagram of an imaging system according to a fourth embodiment. [Figure 17] FIG. 13 is a block diagram of a light detection system according to a fifth embodiment. [Figure 18] FIG. 13 is a schematic diagram of an endoscopic surgery system according to a sixth embodiment. [Figure 19A] FIG. 13 is a schematic diagram of a light detection system according to a seventh embodiment. [Figure 19B] FIG. 13 is a schematic diagram of a moving body in a seventh embodiment. [Figure 20] 20 is a flowchart showing the operation of the light detection system in the seventh embodiment. [Figure 21] FIG. 13 is a diagram showing a specific example of an electronic device in the eighth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The embodiments shown below are intended to embody the technical ideas of the present invention, and are not intended to limit the present invention. The size and positional relationship of the members shown in each drawing may be exaggerated for clarity of explanation. In the following explanation, elements having common functions throughout each drawing may be given the same reference numerals, and duplicate explanations may be omitted or simplified.

[0010] [First embodiment] The configuration of the photoelectric conversion device according to this embodiment will be described with reference to Figs. 1 to 9. The photoelectric conversion device has SPAD type pixels including avalanche photodiodes (hereinafter referred to as "APDs"). The conductivity type of the charge used as the signal charge among the charge pairs generated in the APD is called the first conductivity type. The first conductivity type refers to a conductivity type in which the charge of the same polarity as the signal charge is the majority carrier. The conductivity type opposite to the first conductivity type is called the second conductivity type. In the following, an example will be described in which the signal charge is an electron, the first conductivity type is an N type, and the second conductivity type is a P type, but the signal charge may be a hole, the first conductivity type is a P type, and the second conductivity type is an N type.

[0011] FIG. 1 is a schematic diagram of a photoelectric conversion device according to the present embodiment, showing the configuration of a stacked photoelectric conversion device 100. The photoelectric conversion device 100 includes a sensor substrate (first substrate) 1 and a circuit substrate (second substrate) 2 stacked on each other, and the sensor substrate 1 and the circuit substrate 2 are electrically connected to each other. The photoelectric conversion device in this embodiment is a back-illuminated photoelectric conversion device in which light is incident from a first surface of the sensor substrate 1 and the circuit substrate 2 is disposed on a second surface of the sensor substrate 1. The sensor substrate 1 has a first semiconductor layer having a photoelectric conversion element described later and a first wiring structure. The circuit substrate 2 has a second semiconductor layer having a circuit such as a signal processing unit described later and a second wiring structure. The second semiconductor layer, the second wiring structure, the first wiring structure, and the first semiconductor layer are stacked in this order to form the photoelectric conversion device 100.

[0012] In the following, the sensor substrate 1 and the circuit substrate 2 may be diced chips, but are not limited to chips. For example, each substrate may be a wafer. Also, each substrate may be stacked in a wafer state and then diced, or may be chipped and then stacked and bonded. A pixel array 1a is disposed on the sensor substrate 1, and a circuit region 2a that processes signals detected in the pixel array 1a is disposed on the circuit substrate 2.

[0013] 2 is a diagram showing an example of the arrangement of the sensor substrate 1. A plurality of pixels 10 each include an APD 11, and are arranged in a two-dimensional array in a plan view to form a pixel array 1a.

[0014] The pixel 10 is typically a pixel for forming an image, but when used for TOF (Time of Flight), it does not necessarily have to form an image. That is, the pixel 10 may be a pixel for measuring the time and amount of light that arrives.

[0015] 3 is a block diagram showing a schematic configuration of a photoelectric conversion device according to this embodiment. The photoelectric conversion device 100 according to this embodiment includes a pixel array 1a, a control unit 202, a first processing unit 203, a second processing unit 204, and an input / output unit 205.

[0016] The pixel array 1a includes a number of tile regions 201 partitioned in a matrix shape across a number of rows and a number of columns. In Fig. 3, the tile regions 201 arranged in the first and second rows and the first and second columns are shown with reference characters indicating the row and column numbers. For example, the tile region 201 arranged in the second row and first column is given the reference character "T21".

[0017] Each of the multiple tile regions 201 includes multiple pixel units 301. The multiple pixel units 301 include multiple pixels 10 arranged in a matrix of M rows and N columns. M and N are integers equal to or greater than 1, and at least one of M and N is an integer equal to or greater than 2. In FIG. 3, the multiple pixels 10 arranged in the first pixel row to the eighth pixel row and the first pixel column to the eighth pixel column are shown with reference characters indicating the row number and column number. For example, the pixel 10 arranged in the second pixel row and the first pixel column is given the reference character "P21."

[0018] In the tile area 201, pixel signals output from the pixel unit 301 are processed. The pixel signal processing will be described later. The tile area 201 is connected to a first processing unit 203 through a vertical output line, and is connected to a second processing unit 204 through a horizontal output line. The signal output from the tile area 201 is transmitted to the first processing unit 203 or the second processing unit 204.

[0019] The control unit 202 controls the operation of the tile region 201, the pixel unit 301, the first processing unit 203, and the second processing unit 204. The control unit 202 is electrically connected to each of the tile region 201, each of the pixel units 301, the first processing unit 203, and the second processing unit 204. The control unit 202 transmits control signals for controlling the tile region 201, the pixel unit 301, the first processing unit 203, and the second processing unit 204. The tile region 201, the pixel unit 301, the first processing unit 203, and the second processing unit 204 execute signal processing, signal transmission, and the like based on the received control signals. In this embodiment, the tile region 201 transmits signals output from some pixel units among the plurality of pixel units 301 to the second processing unit 204 based on the received control signals, and transmits signals output from pixel units other than the some pixel units to the first processing unit 203. In the following description, the some pixel units are referred to as event detection pixel units or simply detection pixel units. Moreover, the pixel portion other than the detection pixel portion is called an imaging pixel portion or a frame pixel portion.

[0020] First processing unit 203 and second processing unit 204 perform predetermined arithmetic processing on the signals received from tile region 201, and output the results of the arithmetic processing to input / output unit 205. Input / output unit 205 outputs the signals received from first processing unit 203 and second processing unit 204 to an external device such as an image processing circuit.

[0021] In this embodiment, the first processing unit 203 processes pixel signals transmitted from the imaging pixel unit, and the second processing unit 204 processes pixel signals transmitted from the event detection pixel unit. However, the photoelectric conversion device 100 according to the present disclosure may be configured to include a single processing unit. For example, the first processing unit 203 may be configured to process pixel signals transmitted from the imaging pixel unit and pixel signals transmitted from the event detection pixel unit.

[0022] 4 is a block diagram showing a schematic configuration of the tile area 201 in this embodiment. The tile area 201 includes a pixel unit 301, a storage unit 302, a calculation unit 303, a threshold control unit 304, a comparison unit 305, and an input / output unit 306.

[0023] The pixel unit 301 detects photons incident on the pixel 10, counts the incident photons, and processes and outputs the number of counted photons as a count value. The pixel unit 301 is connected to a storage unit 302, a calculation unit 303, and an input / output unit 306. The pixel unit 301 includes an event detection pixel unit used to detect an event and an imaging pixel unit used to output imaging data. The pixel unit 301 receives an event synchronization signal SyncE and a frame synchronization signal SyncF transmitted from the control unit 202. The event synchronization signal SyncE and the frame synchronization signal SyncF control the operation of the pixel unit 301. For example, the event synchronization signal SyncE can operate the event detection pixel unit in the pixel unit 301 at a frame rate of 1000 fps. Also, the frame synchronization signal SyncF can operate the imaging pixel unit in the pixel unit 301 at a frame rate of 60 fps. In this embodiment, the pixel unit 301 that operates based on the event synchronization signal SyncE is an event detection pixel unit, and the pixel unit 301 that operates based on the frame synchronization signal SyncF is an imaging pixel unit. The number of event detection pixel units and imaging pixel units can be selected arbitrarily. For example, the tile region 201 includes a selection switch (not shown) connected to each of the control unit 202 and the pixel units 301. The control unit 202 can operate each of the pixel units 301 as either an event detection pixel unit or an imaging pixel unit via the selection switch.

[0024] The memory unit 302 holds a reference value used to detect an event. The reference value may be a count value output from the event detection pixel unit (pixel unit 301) when an event occurred in the past. The reference value may be updated when an event occurs. The memory unit 302 is connected to the calculation unit 303 and the comparison unit 305. The memory unit 302 transmits the reference value to the calculation unit 303. The memory unit 302 also receives a signal transmitted from the comparison unit 305. The signal transmitted from the comparison unit 305 to the memory unit 302 will be described later.

[0025] The storage unit 302 may be configured arbitrarily according to the number of event detection pixels. For example, the tile region 201 may include the same number of storage units 302 as the number of event detection pixels, and each storage unit 302 may hold a reference value of the corresponding event detection pixel unit. As another example, a single storage unit 302 may be configured to have a capacity capable of storing all reference values ​​corresponding to multiple event detection pixels.

[0026] The calculation unit 303 performs calculations on the signals received from the pixel unit 301 and the memory unit 302. The calculation unit 303 is connected to the pixel unit 301, the memory unit 302, and the comparison unit 305. The calculation unit 303 receives a count value from the event detection pixel unit in the pixel unit 301. The count value indicates the number of photons that have been incident on the pixel 10 of the event detection pixel unit in a predetermined period. The calculation unit 303 also receives a reference value from the memory unit 302. The calculation unit 303 performs a predetermined calculation based on the count value and the reference value. Specifically, the calculation unit 303 subtracts the reference value from the count value. If the result of the subtraction (difference) is a positive value, it indicates that a greater number of photons than the reference value have been incident on the event detection pixel unit in a predetermined period. On the other hand, if the difference is a negative value, it indicates that a smaller number of photons than the reference value have been incident on the event detection pixel unit in a predetermined period. The calculation unit 303 transmits the result of the calculation to the comparison unit 305.

[0027] The threshold control unit 304 is connected to the comparison unit 305 and transmits a predetermined threshold to the comparison unit 305. The threshold is used in the comparison unit 305 to determine whether an event has occurred in the event detection pixel unit. The threshold control unit 304 may supply two or more thresholds to the comparison unit 305. For example, the threshold control unit 304 may transmit a first threshold Th1 and a second threshold Th2 to the comparison unit 305. The first threshold Th1 is used in the comparison unit 305 when the result of the subtraction in the calculation unit 303 is a positive value. The second threshold Th2 is used in the comparison unit 305 when the result of the subtraction in the calculation unit 303 is a negative value. The first threshold Th1 and the second threshold Th2 are different values. The first threshold Th1 and the second threshold Th2 may be fixed values ​​or may be changed as appropriate. For example, the first threshold Th1 and the second threshold Th2 may be changed according to a change in a reference value stored in the storage unit 302. For example, when the reference value stored in the storage unit 302 is A, the threshold control unit 304 can change the first threshold to Th1×α(A) and the second threshold to Th2×β(A). Here, the parameters α and β are both values ​​that change according to the change in the reference value A. The parameters α and β can be different from each other. The amount of change in the parameters α and β due to the change in the reference value A can be different from each other. Thus, the amount of change in the first threshold due to the change in the reference value A can be different from the amount of change in the second threshold due to the change in the reference value A. Furthermore, any offset value may be added to the first threshold and the second threshold. For example, the threshold control unit 304 can change the first threshold to Th1×α(A)+X. (X is any value). Similarly, the threshold control unit 304 can change the second threshold to Th2×β(A)+Y (Y is any value). When the threshold is changed, the threshold control unit 304 can be configured to appropriately acquire the reference value A stored in the storage unit 302. The threshold control unit 304 appropriately changes the threshold. Therefore, the photoelectric conversion device 100 according to the present disclosure makes it possible to accurately detect an event according to the environment, conditions, etc. of the scene to be imaged.

[0028] The comparison unit 305 compares the signal received from the calculation unit 303 with the signal received from the threshold control unit. The comparison unit 305 is connected to the storage unit 302, the calculation unit 303, the threshold control unit 304, and the input / output unit 306. The comparison unit 305 receives the result of the calculation from the calculation unit 303 and the threshold from the threshold control unit 304. The comparison unit 305 compares the result of the calculation with the threshold and determines whether or not a first event has occurred in the event detection pixel unit. For example, if the result of the calculation in the calculation unit 303 is equal to or greater than the first threshold, the comparison unit 305 determines that the first event has occurred. The occurrence of the first event indicates that the increase in the number of photons incident on the event detection pixel unit in a predetermined period of time has reached a predetermined condition. Also, if the result of the calculation in the calculation unit 303 is equal to or less than the second threshold, the comparison unit 305 determines that the second event has occurred. The occurrence of the second event indicates that the decrease in the number of photons incident on the event detection pixel unit in a predetermined period of time has reached a predetermined condition. If the result of the calculation in the calculation unit 303 is within the range of the set thresholds, the comparison unit 305 determines that no event has occurred. The comparison unit 305 transmits the result of the determination to the input / output unit 306. For example, the comparison unit 305 outputs +1 when it determines that the result of the calculation exceeds a first threshold, outputs -1 when it determines that the result of the calculation exceeds a second threshold, and outputs 0 when the result of the calculation is within the range of the set thresholds. In addition, if it is determined that an event has occurred, the comparison unit 305 updates the reference value stored in the storage unit 302. Specifically, the reference value of the detection pixel unit that detected the event is updated to the count value output from the detection pixel unit. In order to update the count value, the comparison unit 305 transmits a signal for updating the reference value to the storage unit 302 or the control unit 202.

[0029] The input / output unit 306 is an interface that receives a signal processed in the tile region 201 and transmits the signal to the outside of the tile region 201. The input / output unit 306 receives a count value (S203) transmitted from an imaging pixel unit in the pixel unit 301, and outputs the count value to the first processing unit 203. The transmitted count value corresponds to the number of photons incident on a pixel 10 of the imaging pixel unit within a predetermined period. The input / output unit 306 also receives an event detection result (S204) from the comparison unit 305, and outputs the detection result to the second processing unit 204. The input / output unit 306 can be configured to output the event detection result to the second processing unit 204 only when the comparison unit 305 detects the occurrence of an event.

[0030] FIG. 5 is a diagram showing a schematic configuration of the pixel unit 301 in this embodiment. The pixel unit 301 has a pixel 10 including an APD 11, a PMOS transistor 401, an inverter 402, and a counter 403. A pulse signal ENB is applied to the gate of the PMOS transistor 401 from the control unit 202. A voltage VH is applied to one of the source and drain of the PMOS transistor 401, and a cathode of the APD 11 and one end of the inverter 402 are connected to the other of the source and drain. A voltage VL is applied to the anode of the APD 11. In this embodiment, the voltage VH is set to about 3.3 V, and the voltage VL is set to about −20 V. The other end of the inverter 402 is connected to the counter 403. A signal RES is input to the counter 403 from the control unit 202. The control unit 202 controls the counter 403 via the signal RES.

[0031] When the source-drain of the PMOS transistor 401 is brought into a conductive state (on state) in response to the signal ENB, a reverse bias voltage due to the potential difference between the voltages VH and VL is applied to the APD 11. The reverse bias voltage is set to be larger than the breakdown voltage of the APD 11. Therefore, the APD 11 operates as a Geiger-mode avalanche photodiode.

[0032] The inverter 402 receives a signal Vc generated in response to the operation of the APD 11 , and outputs a waveform-shaped pulse signal Vpulse to the counter 403 .

[0033] The counter 403 receives the pulse signal output from the inverter 402, counts the rising edges of the pulse signal, and stores the counted number as a count value. The count value stored in the counter 403 is a digital value indicating the number of photons incident on the pixel 10 in a predetermined period. In order for the counter 403 to count the rising edges, the inverter 402 is used as a buffer. In addition, when a buffer that does not invert the polarity is used, the counter 403 may be configured to count the falling edges of the pulse signal output from the buffer. The counter 403 is connected to the storage unit 302, the calculation unit 303, and the input / output unit 306. The count value stored in the counter 403 is output to the storage unit 302, the calculation unit 303, and the input / output unit 306.

[0034] Fig. 6 is a timing diagram showing the operation of the pixel unit 301 in this embodiment. Fig. 6 shows a timing diagram of the incidence of a photon on the pixel 10, the signal ENB input to the gate of the PMOS transistor 401, the signal Vc generated based on the operation of the APD 11, the pulse signal Vpulse waveform-shaped by the inverter 402, and the count value of the counter 403.

[0035] In the period before time t1, the signal ENB is at a high level, the signal Vc is at a low level, the signal Vpulse is at a high level, and the count value is n. The signal ENB at a high level is applied to the gate of the PMOS transistor 401, and there is no conduction between the source and drain of the PMOS transistor 401 (off state). The signal Vc is at a low level, and the APD 11 is not charged.

[0036] At time t1, the signal ENB transitions from high level to low level, and the PMOS transistor 401 switches from an off state to an on state. During the period from time t1 to t2, the signal ENB is maintained at low level. The APD 11 is charged via the PMOS transistor 401, and the signal Vc transitions from low level to high level. That is, the APD 11 becomes capable of generating avalanche multiplication by the incidence of photons. In response to the signal Vc transitioning to high level, the signal Vpulse transitions from high level to low level.

[0037] At time t2, a photon is incident on pixel 10, and APD 11 generates avalanche multiplication. The signal Vc transitions from high level to low level due to the occurrence of avalanche multiplication. The signal Vc exceeds the threshold of inverter 402, and the signal Vpulse transitions from low level to high level. The transition of signal Vpulse to high level causes counter 403 to increase the stored count value by 1 LSB from n to n+1.

[0038] During the period from time t2 to t3, the signal ENB is maintained at a low level. The APD 11 is recharged via the PMOS transistor 401. At time t3, the signal Vc falls below the threshold of the inverter 402, and the signal Vpulse transitions from a high level to a low level. During the period from time t3 to t4, the signal ENB is maintained at a low level, the signal Vc is maintained at a high level, and the signal Vpulse is maintained at a low level.

[0039] At time t4, a photon is incident on pixel 10, and APD 11 generates avalanche multiplication. The signal Vc transitions from high level to low level due to the occurrence of avalanche multiplication. The signal Vc exceeds the threshold of inverter 402, and the signal Vpulse transitions from low level to high level. The transition of signal Vpulse to high level causes counter 403 to increase the stored count value by 1 LSB from n+1 to N+2.

[0040] During the period from time t4 to t5, the signal ENB is maintained at a low level. The APD 11 is recharged via the PMOS transistor 401. At time t5, the signal Vc falls below the threshold of the inverter 402, and the signal Vpulse transitions from a high level to a low level. During the period from time t5 to t6, the signal ENB is maintained at a low level, the signal Vc is maintained at a high level, and the signal Vpulse is maintained at a low level.

[0041] At time t6, the signal ENB transitions from low to high, and the PMOS transistor 401 switches from the on state to the off state. The voltage VH is not supplied to the APD 11 via the PMOS transistor 401. That is, the APD 11 is not recharged.

[0042] The period for counting photons (counting period) can be adjusted by changing the period during which the signal ENB is at a low level. Also, the counting period of the event detection pixel unit can be different from the counting period of the imaging pixel unit. That is, the control unit 202 can supply separate signals ENB to the event detection pixel unit and the imaging pixel unit according to the frame rate applied to the pixel unit 301. For example, the event detection pixel unit can be driven at a relatively high frame rate, and the imaging pixel unit can be driven at a relatively low frame rate. According to this configuration, it is possible to detect the occurrence of an event with high accuracy via the event detection pixel unit, and at the same time capture a high-definition image via the imaging pixel unit.

[0043] FIG. 7 is a diagram showing an example of a layout in the tile region 201 of this embodiment. FIG. 7 shows the counter 403 and the storage unit 302 of the pixel unit 301 arranged in the tile region 201. In this example, the counter 403 is arranged outside (periphery) of the tile region 201, and the storage unit 302 is arranged inside (center) of the tile region 201 so as to be surrounded by the counter 403. Furthermore, the calculation unit 303, the threshold control unit 304, the comparison unit 305, and the input / output unit 306 are arranged in the center of the tile region 201 so as to be surrounded by the counter 403. The multiple counters 403 share the storage unit 302, the calculation unit 303, the threshold control unit 304, the comparison unit 305, and the input / output unit 306. The count values ​​output from the counters 403 are transmitted to the shared storage unit 302, the calculation unit 303, and the input / output unit 306, and are processed and calculated by the shared threshold control unit 304 and the comparison unit 305. By adopting a layout as shown in this example, it is possible to equalize the transmission distance of data from each of the multiple pixel units 301 arranged in the tile region 201. In this example, in the tile region 201, 16 counters 403 (pixel units 301) share four storage units 302, one calculation unit 303, one threshold control unit 304, one comparison unit 305, and one input / output unit 306. However, the configuration of the tile region 201 according to the present disclosure is not limited to the example shown in FIG. 7. For example, the tile region 201 may be configured to include one storage unit 302, and the storage unit 302 may be configured to have a capacity capable of storing a number of reference values ​​corresponding to the number of event detection pixel units (pixel units 301).

[0044] 8A and 8B are diagrams showing an example of the arrangement of event detection pixels and imaging pixels in a tile region 201 in this embodiment. In this example, pixels 10 are arranged in a 4×4 matrix in the tile region 201. The control unit 202 can operate any part of the pixels 10 in the tile region 201 as event detection pixels 10a via a selection switch (not shown). In addition, the control unit 202 can operate pixels other than the event detection pixels 10a as imaging pixels 10b via the selection switch. For example, as shown in FIG. 8A, the control unit 202 can operate pixels P22, P24, P42, and P44 in the tile region T11 as event detection pixels 10a and the other pixels as imaging pixels 10b. As another example, as shown in FIG. 8B, the control unit 202 can operate pixels P51, P61, P71, and P81 in the tile region T21 as event detection pixels 10a and the other pixels as imaging pixels 10b. The control unit 202 can arbitrarily and separately select the number and arrangement of the event detection pixels 10a and the imaging pixels 10b for each of the tile regions 201 via a selection switch. That is, the number of the event detection pixels 10a and the imaging pixels 10b according to the present disclosure is variable. The control unit 202 can increase or decrease the number of the event detection pixels 10a and the imaging pixels 10b. Also, as shown in FIG. 8 and FIG. 8B, the control unit 202 can change the arrangement of the event detection pixels 10a and the imaging pixels 10b.

[0045] FIG. 9 is a diagram showing an example of the arrangement of color filters arranged in the tile region 201 of this embodiment. In this example, a plurality of pixels 10 are arranged in a matrix in the tile region 201. Four green color filters G, a red color filter R, a blue color filter B, and a transparent color filter C (transparent layer) are arranged on the plurality of pixels 10. In this example, a transparent color filter C is arranged on the four event detection pixels 10a shown in FIG. 8A, and four color filters G, R, and B are arranged on the 12 imaging pixels 10b. In this example, four of each of four types of filters are arranged in the tile region 201. However, the number of color filters and the combination of types of color filters included in the photoelectric conversion device 100 according to the present disclosure are not limited to this example, and can be changed arbitrarily.

[0046] According to the present disclosure, each of the plurality of pixel units 301 in the tile region 201 includes a pixel 10 including an APD 11 and a counter 403 that stores a count value of incident photons. In addition, the plurality of pixel units 301 share a memory unit 302, a calculation unit 303, a threshold control unit 304, a comparison unit 305, and an input / output unit 306. A part of the plurality of pixel units 301 operates as a detection pixel unit for detecting an event. The detection pixel unit outputs a count value, which is a digital value, to the calculation unit 303, and the comparison unit 305 detects the occurrence of an event based on the count value. That is, the photoelectric conversion device according to the present disclosure does not have a problem of noise caused by an analog signal such as a photocurrent generated from a photodiode when detecting an event. In addition, the event detection pixel unit included in the photoelectric conversion device of the present disclosure has the same configuration as the imaging pixel unit. In addition, the memory unit, the calculation unit, the threshold control unit, the comparison unit, and the input / output unit used when detecting an event are shared by the plurality of event detection pixels. Therefore, according to the present disclosure, the circuit scale of the pixel units arranged in a matrix in the tile region can be uniformed. Furthermore, photoelectric conversion device 100 according to the present disclosure can cause all pixel units 301 included in tile region 201 to function as either event detection pixel units or imaging pixel units. That is, photoelectric conversion device 100 can separately set the number and arrangement of event detection pixel units and imaging pixel units in each tile region 201. Furthermore, since the event detection pixel units and the imaging pixel units have the same configuration, photoelectric conversion device 100 according to the present disclosure can simplify the manufacturing process of the pixel units and reduce manufacturing costs.

[0047] [Second embodiment] A photoelectric conversion device according to a second embodiment of the present disclosure will be described with reference to FIGS. 10 to 13, focusing on differences from the first embodiment.

[0048] FIG. 10 is a block diagram showing a schematic configuration of a photoelectric conversion device according to this embodiment. The photoelectric conversion device 100 according to this embodiment differs from the first embodiment in that it does not have a vertical output line and a horizontal output line. According to this embodiment, a signal output from a tile region 201 is transferred to another adjacent tile region 201. For example, a signal output from a tile region T11 according to this embodiment is directly transferred to adjacent tile regions T12 and T21 in the row and column directions, respectively, without passing through a vertical output line and a horizontal output line. A signal output from a tile region 201 is transmitted to a first processing unit 203 or a second processing unit 204 via one or more other tile regions 201. The control unit 202 controls the direction of signal transfer according to the type of signal output from the tile region 201. For example, the control unit 202 determines whether to transfer the signal in the row direction (tile region T12) or the column direction (tile region T21) based on the type of signal output from the tile region T11. That is, the control unit 202 determines whether to transmit a signal to the first processing unit 203 or the second processing unit 204 based on the type of signal output from the tile region 201. For example, the control unit 202 can transfer a pixel signal from the imaging pixel unit to an adjacent tile region 201 in the column direction and have the first processing unit 203 process the signal. The control unit 202 can also transfer a pixel signal from an event detection pixel to an adjacent tile region 201 in the row direction and have the second processing unit 204 process the signal.

[0049] 11 is a block diagram showing a schematic configuration of a tile region 201 in this embodiment. The input / output unit 306 included in the tile region 201 in this embodiment directly receives a signal from the input / output unit 306 of another tile region 201 adjacent in the row direction and the column direction. That is, the input / output unit 306 receives a signal DataH transferred from the input / output unit 306 of the tile region 201 adjacent in the row direction. The input / output unit 306 transfers the signal DataH to the input / output unit 306 of the other tile region 201 adjacent in the row direction. Similarly, the input / output unit 306 receives a signal DataV transferred from the input / output unit 306 of the tile region 201 adjacent in the column direction. The input / output unit 306 transfers the signal DataV to the input / output unit 306 of the other tile region 201 adjacent in the column direction. For example, the input / output unit 306 of the tile region T22 transfers a signal received from the tile region T21 to the input / output unit 306 of the tile region T23. Furthermore, the input / output unit 306 of tile area T22 transfers the signal received from tile area T12 to the input / output unit 306 of tile area T32. The signal transferred to the input / output unit 306 of tile area T23 is sequentially transferred to adjacent tile areas 201 in the row direction, and is finally transmitted to the second processing unit 204. Furthermore, the signal transferred to the input / output unit 306 of tile area T32 is sequentially transferred to adjacent tile areas 201 in the column direction, and is finally transmitted to the first processing unit 203. The first processing unit 203 and second processing unit 204 each process the transmitted signals.

[0050] FIG. 12 is a diagram showing a modified example of the layout of the pixels 10 in the tile region 201 of the present embodiment. In each of the tile regions 201 according to this modified example, a different number of pixels 10 are arranged in a matrix in the row direction and the column direction. Specifically, the layout of the pixels 10 in the tile region 201 according to this modified example can be determined according to the frequency with which signals from the tile region 201 are transferred in the row direction and the column direction. For example, when signals output from the tile region 201 are transferred relatively more frequently in the column direction than in the row direction, the pixels 10 in the tile region 201 are arranged relatively more in the column direction as shown in FIG. 12. In the example of the pixel layout shown in FIG. 12, two pixels 10 are arranged in the row direction and four pixels 10 are arranged in the column direction in the tile region 201. On the other hand, when signals output from the tile region 201 are transferred relatively more frequently in the row direction than in the column direction, the pixels 10 in the tile region 201 are arranged relatively more in the row direction. The number of pixels 10 arranged in the row direction and column direction may be changed depending on the frequency of signal transmission in each direction. For example, the relatively higher the frequency of pixel signals being transmitted in the column direction, the greater the number of pixels 10 arranged in the column direction in the tile region 201 may be set to be relative to the number of pixels 10 arranged in the row direction. On the other hand, the relatively higher the frequency of pixel signals being transmitted in the row direction, the greater the number of pixels 10 arranged in the row direction in the tile region 201 may be set to be relative to the number of pixels 10 arranged in the column direction.

[0051] According to this embodiment, a signal output from a tile region 201 is sequentially transferred to the first processing unit 203 or the second processing unit 204 via the input / output unit 306 of an adjacent tile region 201. Furthermore, according to the modification shown in Fig. 12, a relatively larger number of pixels 10 are arranged in each tile region 201 in a direction in which signals are transferred more frequently. That is, this modification can reduce the number of times pixel signals are transferred to adjacent tile regions 201 in a direction in which transfer frequency is high. Therefore, the photoelectric conversion device 100 according to this modification can reduce power consumed when transferring signals.

[0052] FIG. 13 is a diagram showing a modified example of a pattern for connecting tile regions 201 in this embodiment. A signal output from a tile region 201 according to this embodiment does not necessarily need to be transmitted to adjacent tile regions 201 in the row and column directions. For example, as shown in FIG. 13, tile region T11 is directly connected to non-adjacent tile region T13. Similarly, tile region T11 is directly connected to non-adjacent tile region T31. That is, a signal output from tile region T11 is directly transferred to non-adjacent tile regions T13 and T31 without passing through adjacent tile regions T12 and T21. According to this modified example, the number of times that a signal output from tile region 201 is transferred is further reduced. Therefore, the photoelectric conversion device 100 according to this modified example can further reduce the power consumed when transferring a signal.

[0053] Note that the pattern of connections between tile regions 201 according to this embodiment is not limited to the modified example shown in Fig. 13. For example, a signal output from tile region T11 may be configured to be directly transferred to a tile region (e.g., tile regions T14, T41) that is more remote than tile regions T13, T31.

[0054] According to this embodiment, a signal output from a tile region 201 is sequentially transferred to the input / output unit 306 of another tile region 201 via the input / output unit 306, and is finally transmitted to the first processing unit 203 or the second processing unit 204. This configuration makes it possible to reduce signal attenuation in the vertical output lines and horizontal output lines, which are extended as the pixel array 1a becomes larger. In other words, this modification eliminates the need for a separate circuit to compensate for signal attenuation in the output lines, simplifying the circuit design of the photoelectric conversion device.

[0055] [Third embodiment] A photoelectric conversion device according to a third embodiment of the present disclosure will be described with reference to FIGS. 14 and 15, focusing on differences from the first and second embodiments.

[0056] 14 is a diagram showing a schematic configuration of a pixel unit 301 in this embodiment. The pixel unit 301 has a pixel 10 including an APD 11, a PMOS transistor 401, a counter 403, and an AND circuit 405. A pulse signal CLKB is applied from the control unit 202 to the gate of the PMOS transistor 401 and the input terminal of the AND circuit 405. A voltage VH is applied to one of the source and drain of the PMOS transistor 401, and the other of the source and drain is connected to the cathode of the APD 11 and the input terminal of the AND circuit 405. A voltage VL is applied to the anode of the APD 11. The output terminal of the AND circuit 405 is connected to the counter 403. A signal RES is input to the counter 403 from the control unit 202. The control unit 202 controls the counter 403 via the signal RES.

[0057] The AND circuit 405 receives as an input an inverted signal of the signal Vc generated in response to the operation of the APD 11. That is, the AND circuit 405 outputs a waveform-shaped high-level pulse signal Vpulse to the counter 403 when the signal Vc is at a low level and the input signal CLKB is at a high level.

[0058] Fig. 15 is a timing diagram showing the operation of the pixel unit 301 in this embodiment. Fig. 15 shows timing diagrams of the incidence of photons on the pixel 10, the signal CLKB transmitted to the gate of the PMOS transistor 401 and the input terminal of the AND circuit 405, the signal Vc generated based on the operation of the APD 11, the pulse signal Vpulse waveform-shaped by the AND circuit 405, and the count value of the counter 403.

[0059] In the period before time t1, the signal CLKB is at a high level, the signal Vc is at a low level, the signal Vpulse is at a high level, and the count value is n. The signal CLKB at a high level is applied to the gate of the PMOS transistor 401, and the source-drain of the PMOS transistor 401 is in an off state. The signal Vc is at a low level, and the APD 11 is not charged.

[0060] At time t1, the signal CLKB transitions from a high level to a low level, and the PMOS transistor 401 switches from an off state to an on state. Also, at time t1, the low level signal CLKB is input to the AND circuit 405, and the signal Vpulse transitions from a high level to a low level. The signal CLKB is maintained at a low level for a predetermined period from time t1. The APD11 is charged via the PMOS transistor 401, and the signal Vc transitions from a low level to a high level. The APD11 is capable of generating avalanche multiplication by the incidence of photons. In response to the signal Vc transitioning to a high level, the signal Vpulse is maintained at a low level. After a predetermined period has elapsed from time t1, the signal CLKB transitions from a low level to a high level, and the PMOS transistor 401 switches from an on state to an off state. Since the signal Vc is maintained at a high level, the signal Vpulse is maintained at a low level.

[0061] At time t2, the signal CLKB transitions from a high level to a low level, and the PMOS transistor 401 switches from an off state to an on state. Since the APD 11 is already charged, the signal Vc is maintained at a high level, and the signal Vpulse is maintained at a low level. After a predetermined period has elapsed from time t2, the signal CLKB transitions from a low level to a high level, and the PMOS transistor 401 switches from an on state to an off state. Since the signal Vc is maintained at a high level, the signal Vpulse is maintained at a low level.

[0062] At time t3, a photon is incident on the APD 11, which generates avalanche multiplication. The signal Vc transitions from high level to low level due to the occurrence of avalanche multiplication, exceeding the logical threshold of the AND circuit 405. Furthermore, the signal CLKB input to the AND circuit 405 at time t3 is at high level. Therefore, the signal Vpulse transitions from low level to high level at time t3. The transition of the signal Vpulse to high level causes the counter 403 to increase the stored count value by 1 LSB from n to n+1.

[0063] During the period from time t3 to t4, the signal CLKB is maintained at a high level, and the PMOS transistor 401 is maintained in an off state. Therefore, during the period from time t3 to t4, the APD 11 is not recharged, the signal Vc is maintained at a low level, and the signal Vpulse is maintained at a high level.

[0064] At time t4, the signal CLKB transitions from a high level to a low level, and the PMOS transistor 401 switches from an off state to an on state. Also, at time t4, the signal CLKB transitions to a low level, so that the low level signal CLKB is input to the AND circuit 405 at time t4, and the signal Vpulse transitions from a high level to a low level. The signal CLKB is maintained at a low level for a predetermined period from time t4. The APD11 is recharged via the PMOS transistor 401, and the signal Vc transitions from a low level to a high level. The APD11 is able to generate avalanche multiplication by the incidence of photons. In response to the signal Vc transitioning to a high level, the signal Vpulse is maintained at a low level. After a predetermined period has elapsed from time t4, the signal CLKB transitions from a low level to a high level, and the PMOS transistor 401 switches from an on state to an off state. Since the signal Vc is maintained at a high level, the signal Vpulse is maintained at a low level.

[0065] At time t5, a photon is incident on the APD 11, which generates avalanche multiplication. The signal Vc transitions from high level to low level due to the occurrence of avalanche multiplication, exceeding the logical threshold of the AND circuit 405. Furthermore, the signal CLKB input to the AND circuit 405 at time t5 is at high level. Therefore, the signal Vpulse transitions from low level to high level at time t5. Due to the transition of the signal Vpulse to high level, the counter 403 increases the stored count value by 1 LSB from n+1 to n+2.

[0066] During the period from time t5 to t6, the signal CLKB is maintained at a high level, and the PMOS transistor 401 is maintained in an off state. Therefore, during the period from time t5 to t6, the APD 11 is not recharged, the signal Vc is maintained at a low level, and the signal Vpulse is maintained at a high level.

[0067] At time t6, a photon is incident on the APD 11. However, since the APD 11 is not recharged during the period from time t5 to t6, the signal Vc does not change from the low level and the signal Vpulse does not change from the high level at time t6. Therefore, at time t6, the counter 403 does not increment the stored count value and the count value is maintained at n+2.

[0068] At time t7, the signal CLKB transitions from a high level to a low level, and the PMOS transistor 401 switches from an off state to an on state. Also, at time t7, the low level signal CLKB is input to the AND circuit 405, and the signal Vpulse transitions from a high level to a low level. The signal CLKB is maintained at a low level for a predetermined period from time t7. The APD11 is charged via the PMOS transistor 401, and the signal Vc transitions from a low level to a high level. The APD11 becomes capable of generating avalanche multiplication by the incidence of photons. In response to the signal Vc transitioning to a high level, the signal Vpulse is maintained at a low level. After a predetermined period has elapsed from time t7, the signal CLKB transitions from a low level to a high level, and the PMOS transistor 401 switches from an on state to an off state. Since the signal Vc is maintained at a high level, the signal Vpulse is maintained at a low level.

[0069] In this embodiment, the period for counting photons is controlled by the signal CLKB. In addition, when the event detection pixel 10a and the imaging pixel 10b are driven at different frame rates, the signal CLKB is changed according to the respective frame rates. With this configuration, the event detection pixel 10a and the imaging pixel 10b can be controlled independently. Specifically, the event detection pixel 10a and the imaging pixel 10b can be operated based on separate signals CLKB having different frequencies. For example, when the exposure period of the imaging pixel 10b is set to twice the exposure period of the event detection pixel 10a, the control unit 202 supplies the imaging pixel 10b with another pulse signal CLKB having a counting period twice that of the pulse signal CLKB supplied to the event detection pixel 10a in one period. The counter 403 is efficiently utilized by making the ratio between the counting period and the period of the signal CLKB constant according to the maximum value that can be counted by the counter 403.

[0070] According to this embodiment, the APD 11 is charged based on the signal CLKB. For example, when two or more photons are incident within a period of one pulse of the signal CLKB under high illuminance, the counter 403 counts only the first incident photon. In other words, in an environment of a predetermined illuminance or higher, the upper limit of the number of counts counted by the counter 403 within a predetermined period can be set to the number of pulses of the signal CLKB input to the pixel unit 301 within the same period.

[0071] [Fourth embodiment] Fig. 16 is a block diagram of an imaging system in this embodiment. The photoelectric conversion device in the above-mentioned embodiment is applicable to various imaging systems. Examples of imaging systems include digital still cameras, digital camcorders, camera heads, copiers, fax machines, mobile phones, vehicle-mounted cameras, observation satellites, and surveillance cameras. Fig. 16 shows a block diagram of a digital still camera as an example of an imaging system.

[0072] The imaging system 7 includes a barrier 706, a lens 702, an aperture 704, an imaging device 70, a signal processing unit 708, a timing generating unit 720, an overall control and calculation unit 718, a memory unit 710, a recording medium control I / F unit 716, a recording medium 714, and an external I / F unit 712. The barrier 706 protects the lens, and the lens 702 forms an optical image of a subject on the imaging device 70. The aperture 704 varies the amount of light that passes through the lens 702. The imaging device 70 is configured like the photoelectric conversion device of the above-mentioned embodiment, and converts the optical image formed by the lens 702 into image data. The signal processing unit 708 performs various corrections and compresses the data on the imaging data output from the imaging device 70.

[0073] The timing generating unit 720 outputs various timing signals to the imaging device 70 and the signal processing unit 708. The overall control / calculation unit 718 controls the entire digital still camera, and the memory unit 710 temporarily stores image data. The recording medium control I / F unit 716 is an interface for recording or reading image data to the recording medium 714, and the recording medium 714 is a removable recording medium such as a semiconductor memory for recording or reading imaging data. The external I / F unit 712 is an interface for communicating with an external computer or the like. Timing signals and the like may be input from outside the imaging system, and the imaging system only needs to have at least the imaging device 70 and the signal processing unit 708 for processing image signals output from the imaging device 70.

[0074] In this embodiment, the imaging device 70 and the signal processing unit 708 are provided on separate semiconductor substrates, but the imaging device 70 and the signal processing unit 708 may be formed on the same semiconductor substrate.

[0075] Each pixel includes a first photoelectric conversion unit and a second photoelectric conversion unit. The signal processing unit 708 processes a pixel signal based on the charge generated in the first photoelectric conversion unit and a pixel signal based on the charge generated in the second photoelectric conversion unit, and can obtain distance information from the imaging device 70 to the subject.

[0076] [Fifth embodiment] FIG. 17 is a diagram of a light detection system in this embodiment, and is a block diagram of a range image sensor using the photoelectric conversion device described in the above embodiment.

[0077] 17, the distance image sensor 410 includes a photoelectric conversion device 400, an image processing circuit 404, a memory 406, a monitor 407, and an optical system 409. The distance image sensor 410 receives light (modulated light, pulsed light) emitted from a light source device 411 toward a subject and reflected from the surface of the subject. The distance image sensor 410 can obtain a distance image according to the distance to the subject, based on the time from emission to reception of light.

[0078] The optical system 409 includes one or more lenses, guides image light (incident light) from a subject to the photoelectric conversion device 400, and forms an image on the light receiving surface (sensor unit) of the photoelectric conversion device 400.

[0079] The photoelectric conversion devices according to the above-mentioned embodiments can be applied as the photoelectric conversion device 400. The photoelectric conversion device 400 supplies the image processing circuit 404 with a distance signal indicating a distance determined from a received light signal.

[0080] The image processing circuit 404 performs image processing to construct a distance image based on the distance signal supplied from the photoelectric conversion device 400. The distance image (image data) obtained by the image processing can be displayed on a monitor 407 and stored (recorded) in a memory 406.

[0081] The range image sensor 410 configured in this way can obtain a more accurate range image as the pixel characteristics are improved by applying the above-mentioned photoelectric conversion device.

[0082] [Sixth embodiment] The technology according to the present disclosure may be applied to various products. For example, the technology according to the present disclosure may be applied to an endoscopic surgery system.

[0083] Fig. 18 is a schematic diagram of the endoscopic surgery system in this embodiment. Fig. 18 shows a state in which an operator (doctor) 1131 performs surgery on a patient 1132 on a patient bed 1133 using an endoscopic surgery system 1103. As shown in the figure, the endoscopic surgery system 1103 includes an endoscope 1100, a surgical tool 1110, and a cart 1134 on which various devices for endoscopic surgery are mounted.

[0084] The endoscope 1100 includes a lens barrel 1101, a region of a predetermined length from the tip of which is inserted into a body cavity of a patient 1132, a camera head 1102 connected to a base end of the lens barrel 1101, and an arm 1121. Although Fig. 18 shows the endoscope 1100 configured as a so-called rigid scope having a rigid lens barrel 1101, the endoscope 1100 may also be configured as a so-called flexible scope having a flexible lens barrel.

[0085] An opening into which an objective lens is fitted is provided at the tip of the lens barrel 1101. A light source device 1203 is connected to the endoscope 1100, and light generated by the light source device 1203 is guided to the tip of the lens barrel by a light guide extending inside the lens barrel 1101, and is irradiated via the objective lens toward an observation target in the body cavity of the patient 1132. The endoscope 1100 may be a direct-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0086] An optical system and a photoelectric conversion device are provided inside the camera head 1102, and reflected light (observation light) from an observation target is focused on the photoelectric conversion device by the optical system. The observation light is photoelectrically converted by the photoelectric conversion device to generate an electrical signal corresponding to the observation light, i.e., an image signal corresponding to an observation image. As the photoelectric conversion device, the photoelectric conversion device described in each of the above-mentioned embodiments may be used. The image signal is transmitted to a camera control unit (CCU) 1135 as RAW data.

[0087] The CCU 1135 is configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and performs overall control of the operations of the endoscope 1100 and the display device 1136. Furthermore, the CCU 1135 receives an image signal from the camera head 1102, and performs various types of image processing on the image signal, such as development processing (demosaic processing), for displaying an image based on the image signal.

[0088] Under the control of the CCU 1135 , the display device 1136 displays an image based on an image signal that has been subjected to image processing by the CCU 1135 .

[0089] The light source device 1203 includes a light source such as an LED (Light Emitting Diode), and supplies the endoscope 1100 with irradiation light when imaging an operation site or the like.

[0090] The input device 1137 is an input interface for the endoscopic surgery system 1103. A user can input various information and instructions to the endoscopic surgery system 1103 via the input device 1137.

[0091] The treatment tool control device 1138 controls the driving of the energy treatment tool 1112 for cauterizing tissue, incising, sealing blood vessels, or the like.

[0092] The light source device 1203 can supply irradiation light to the endoscope 1100 when imaging the surgical site, and can be, for example, a white light source made of an LED, a laser light source, or a combination of these. When a white light source is configured by a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision. Therefore, the light source device 1203 can adjust the white balance of the captured image. In this case, the laser light from each of the RGB laser light sources can be irradiated to the observation target in a time-division manner, and the drive of the image sensor of the camera head 1102 can be controlled in synchronization with the irradiation timing. This makes it possible to capture images corresponding to each of the RGB colors in a time-division manner. According to this method, a color image can be obtained without providing a color filter to the image sensor.

[0093] Furthermore, the driving of the light source device 1203 may be controlled so that the intensity of the light output from the light source device 1203 is changed at predetermined time intervals. By controlling the driving of the image sensor of the camera head 1102 in synchronization with the timing of the change in the light intensity to acquire images in a time-division manner and synthesizing the images, it is possible to generate an image with a high dynamic range that is free of so-called blocked-up shadows and blown-out highlights.

[0094] Furthermore, the light source device 1203 may be configured to be capable of supplying light of a predetermined wavelength band corresponding to the special light observation. In the special light observation, for example, the wavelength dependency of light absorption in body tissue can be utilized. Specifically, a predetermined tissue such as blood vessels on the mucous membrane surface can be photographed with high contrast by irradiating light of a narrower band than the irradiation light (i.e., white light) during normal observation. Alternatively, in the special light observation, a fluorescent observation may be performed in which an image is obtained by fluorescence generated by irradiating excitation light. In the fluorescent observation, it is possible to irradiate excitation light to the body tissue and observe the fluorescence from the body tissue, or to locally inject a reagent such as indocyanine green (ICG) into the body tissue and irradiate the body tissue with excitation light corresponding to the fluorescent wavelength of the reagent to obtain a fluorescent image. The light source device 1203 may be configured to be capable of supplying narrow band light and / or excitation light corresponding to such special light observation.

[0095] [Seventh embodiment] The light detection system and the moving body of this embodiment will be described with reference to Figures 19A, 19B, and 20. In this embodiment, an example of an on-vehicle camera will be shown as the light detection system.

[0096] FIG. 19A is a schematic diagram of a light detection system in this embodiment, showing an example of a vehicle system and a light detection system mounted on the vehicle system. The light detection system 1301 includes a photoelectric conversion device 1302, an image preprocessing unit 1315, an integrated circuit 1303, and an optical system 1314. The optical system 1314 forms an optical image of a subject on the photoelectric conversion device 1302. The photoelectric conversion device 1302 converts the optical image of the subject formed by the optical system 1314 into an electrical signal. The photoelectric conversion device 1302 is any of the photoelectric conversion devices in each of the above-mentioned embodiments. The image preprocessing unit 1315 performs a predetermined signal processing on the signal output from the photoelectric conversion device 1302. The function of the image preprocessing unit 1315 may be incorporated in the photoelectric conversion device 1302. The light detection system 1301 is provided with at least two sets of an optical system 1314 , a photoelectric conversion device 1302 , and an image pre-processing unit 1315 , and the output from each set of the image pre-processing unit 1315 is input to the integrated circuit 1303 .

[0097] The integrated circuit 1303 is an integrated circuit for use in an imaging system, and includes an image processing unit 1304 including a storage medium 1305, an optical distance measuring unit 1306, a parallax calculation unit 1307, an object recognition unit 1308, and an abnormality detection unit 1309. The image processing unit 1304 performs image processing such as development processing and defect correction on the output signal of the image pre-processing unit 1315. The storage medium 1305 stores the primary storage of the captured image and the defective positions of the captured pixels. The optical distance measuring unit 1306 performs focusing and distance measurement of the subject. The parallax calculation unit 1307 calculates distance measurement information from multiple image data acquired by multiple photoelectric conversion devices 1302. The object recognition unit 1308 recognizes subjects such as cars, roads, signs, and people. When the abnormality detection unit 1309 detects an abnormality in the photoelectric conversion device 1302, it notifies the main control unit 1313 of the abnormality.

[0098] The integrated circuit 1303 may be realized by dedicated hardware, a software module, or a combination of these. It may also be realized by a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like, or a combination of these.

[0099] The main control unit 1313 supervises and controls the operations of the light detection system 1301, the vehicle sensor 1310, the control unit 1320, etc. The main control unit 1313 may not be provided, and the light detection system 1301, the vehicle sensor 1310, and the control unit 1320 may each have a communication interface and transmit and receive control signals via a communication network, for example, according to the CAN standard.

[0100] The integrated circuit 1303 has a function of receiving a control signal from the main control unit 1313 or transmitting a control signal or a set value to the photoelectric conversion device 1302 by its own control unit.

[0101] The optical detection system 1301 is connected to a vehicle sensor 1310, and can detect the vehicle's driving state, such as vehicle speed, yaw rate, and steering angle, as well as the state of the environment outside the vehicle, other vehicles, and obstacles. The vehicle sensor 1310 is also a distance information acquisition unit that acquires distance information to an object. The optical detection system 1301 is also connected to a driving assistance control unit 1311 that performs various driving assistance functions, such as automatic steering, automatic cruising, and collision prevention functions. In particular, the collision determination function determines whether or not a collision with another vehicle or obstacle has occurred based on the detection results of the optical detection system 1301 and the vehicle sensor 1310. This allows for avoidance control when a collision is estimated, and activation of a safety device when a collision occurs.

[0102] The light detection system 1301 is also connected to an alarm device 1312 that issues an alarm to the driver based on the result of the determination by the collision determination unit. For example, if the collision determination unit determines that there is a high possibility of a collision, the main control unit 1313 performs vehicle control such as applying the brakes, releasing the accelerator, and suppressing engine output to avoid a collision and reduce damage. The alarm device 1312 issues an alarm to the user using means such as issuing an alarm such as a sound, displaying alarm information on the display screen of a car navigation system or a meter panel, or applying vibration to a seat belt or steering wheel.

[0103] The light detection system 1301 in this embodiment can capture an image of the surroundings of the vehicle, for example, the front or rear of the vehicle. Fig. 19B is a schematic diagram of a moving body in this embodiment, and shows a configuration in which the light detection system 1301 captures an image of the area in front of the vehicle.

[0104] The two photoelectric conversion devices 1302 are disposed in front of the vehicle 1300. Specifically, it is preferable to regard the center line of the vehicle 1300 relative to the forward / backward direction or the outer shape (for example, the vehicle width) as an axis of symmetry, and to dispose the two photoelectric conversion devices 1302 in line symmetry with respect to the axis of symmetry. This makes it possible to effectively obtain distance information between the vehicle 1300 and an object to be photographed and to determine the possibility of a collision. In addition, it is preferable that the photoelectric conversion device 1302 is disposed in a position that does not obstruct the driver's field of vision when the driver visually checks the situation outside the vehicle 1300 from the driver's seat. It is preferable that the alarm device 1312 is disposed in a position that is easily within the driver's field of vision.

[0105] Next, a fault detection operation of the photoelectric conversion device 1302 in the photodetection system 1301 will be described with reference to Fig. 20. Fig. 20 is a flowchart showing the operation of the photodetection system in this embodiment. The fault detection operation of the photoelectric conversion device 1302 can be executed in accordance with steps S1410 to S1480.

[0106] In step S1410, startup settings are made for the photoelectric conversion device 1302. That is, setting information for the operation of the photoelectric conversion device 1302 is transmitted from outside the photodetection system 1301 (e.g., the main control unit 1313) or from inside the photodetection system 1301, and the photoelectric conversion device 1302 starts imaging operation and fault detection operation.

[0107] Next, in step S1420, the photoelectric conversion device 1302 acquires pixel signals from the valid pixels. In addition, in step S1430, the photoelectric conversion device 1302 acquires output values ​​from failure detection pixels provided for failure detection. The failure detection pixels include photoelectric conversion elements, like the valid pixels. A predetermined voltage is written to the photoelectric conversion elements. The failure detection pixels output signals corresponding to the voltage written to the photoelectric conversion elements. Note that steps S1420 and S1430 may be executed in the reverse order.

[0108] Next, in step S1440, the photodetection system 1301 performs a correspondence determination between the output expected value of the fault detection pixel and the actual output value from the fault detection pixel. If the correspondence determination result in step S1440 indicates that the output expected value and the actual output value match, the photodetection system 1301 proceeds to processing in step S1450, determines that the imaging operation is performed normally, and proceeds to processing in step S1460. In step S1460, the photodetection system 1301 transmits the pixel signal of the scanning row to the storage medium 1305 and temporarily stores it. Thereafter, the photodetection system 1301 returns to the processing in step S1420 and continues the fault detection operation. On the other hand, if the correspondence determination result in step S1440 indicates that the output expected value and the actual output value do not match, the photodetection system 1301 proceeds to processing in step S1470. In step S1470, the light detection system 1301 determines that there is an abnormality in the imaging operation, and issues an alarm to the main control unit 1313 or the alarm device 1312. The alarm device 1312 displays on the display unit that an abnormality has been detected. After that, in step S1480, the light detection system 1301 stops the photoelectric conversion device 1302, and ends the operation of the light detection system 1301.

[0109] In this embodiment, the flowchart is looped for each line, but the flowchart may be looped for each set of lines, or the fault detection operation may be performed for each frame. The issuance of the alarm in step S1470 may be notified to the outside of the vehicle via a wireless network.

[0110] In addition, in this embodiment, the control to prevent collision with other vehicles has been described, but the control can also be applied to automatic driving control to follow other vehicles, automatic driving control not to go out of the lane, etc. Furthermore, the light detection system 1301 can be applied not only to vehicles such as the vehicle itself, but also to moving bodies (moving devices) such as ships, aircraft, and industrial robots. In addition, the light detection system can be applied not only to moving bodies, but also to devices that widely use object recognition, such as intelligent transport systems (ITS). The photoelectric conversion device of the present invention may further be configured to be capable of acquiring various information such as distance information.

[0111] [Eighth embodiment] FIG. 21(a) is a diagram showing a specific example of an electronic device in this embodiment, showing glasses 1600 (smart glasses). The glasses 1600 are provided with a photoelectric conversion device 1602 described in each of the above-mentioned embodiments. A display device including a light-emitting device such as an OLED or LED may be provided on the back side of the lens 1601. The photoelectric conversion device 1602 may be one or more. Also, multiple types of photoelectric conversion devices may be combined. The arrangement position of the photoelectric conversion device 1602 is not limited to that shown in FIG. 21(a).

[0112] The glasses 1600 further include a control device 1603. The control device 1603 functions as a power source that supplies power to the photoelectric conversion device 1602 and the above-mentioned display device. The control device 1603 also controls the operations of the photoelectric conversion device 1602 and the display device. The lens 1601 is formed with an optical system for collecting light on the photoelectric conversion device 1602.

[0113] FIG. 21(b) shows glasses 1610 (smart glasses) according to one application example. The glasses 1610 have a control device 1612, and the control device 1612 is equipped with a photoelectric conversion device corresponding to the photoelectric conversion device 1602 and a display device. The lens 1611 is formed with a photoelectric conversion device in the control device 1612 and an optical system for projecting light emitted from the display device, and an image is projected onto the lens 1611. The control device 1612 functions as a power source that supplies power to the photoelectric conversion device and the display device, and controls the operation of the photoelectric conversion device and the display device. The control device 1612 may have a line of sight detection unit that detects the line of sight of the wearer. Infrared light may be used to detect the line of sight. The infrared light emission unit emits infrared light toward the eyeball of a user gazing at a display image. An image of the eyeball is obtained by detecting the reflected light of the emitted infrared light from the eyeball with an imaging unit having a light receiving element. By providing a reduction unit that reduces the amount of light from the infrared light emitting unit to the display unit in a plan view, degradation of image quality is reduced.

[0114] The gaze of the user with respect to the displayed image is detected from an image of the eyeball obtained by capturing infrared light. Any known method can be applied to gaze detection using the image of the eyeball. As an example, a gaze detection method based on a Purkinje image formed by reflection of irradiated light on the cornea can be used.

[0115] More specifically, a gaze detection process based on the pupil-corneal reflex method is performed. Using the pupil-corneal reflex method, a gaze vector that indicates the direction (rotation angle) of the eyeball is calculated based on the pupil image and the Purkinje image included in the captured image of the eyeball, thereby detecting the user's gaze.

[0116] The display device of this embodiment may have a photoelectric conversion device having a light receiving element, and may control the display image of the display device based on information on the user's line of sight from the photoelectric conversion device.

[0117] Specifically, the display device determines a first field of view area to which the user gazes and a second field of view area other than the first field of view area based on the line-of-sight information. The first field of view area and the second field of view area may be determined by a control device of the display device or an external control device. In the display area of ​​the display device, the display resolution of the first field of view area may be controlled to be higher than the display resolution of the second field of view area. In other words, the resolution of the second field of view area may be lower than that of the first field of view area.

[0118] The display area may include a first display area and a second display area different from the first display area. A high priority area may be determined from the first display area and the second display area based on line-of-sight information. The first field of view area and the second field of view area may be determined by a control device of the display device or an external control device. The resolution of the high priority area may be controlled to be higher than the resolution of areas other than the high priority area. In other words, the resolution of an area with a relatively low priority may be lowered.

[0119] In addition, AI (Artificial Intelligence) may be used in determining the first field of view area and the area with high priority. The AI ​​may be a model configured to estimate the angle of the line of sight and the distance to an object at the end of the line of sight from the image of the eyeball, using the image of the eyeball and the direction in which the eyeball in the image was actually looking as teacher data. The AI ​​program may be provided in either the display device or the photoelectric conversion device, or may be provided in an external device. When the external device has the AI ​​program, it may be transmitted from a server or the like to the display device via communication.

[0120] In the case where display control is performed based on visual recognition detection, the present embodiment can be preferably applied to smart glasses further including a photoelectric conversion device that captures an image of the outside world. The smart glasses can display captured outside information in real time.

[0121] [Other embodiments] The present invention is not limited to the above-described embodiments and can be modified in various ways. For example, an example in which a part of the configuration of any of the embodiments is added to another embodiment, or an example in which a part of the configuration of another embodiment is replaced with another embodiment, is also an embodiment of the present invention.

[0122] The disclosure of this specification includes the following configurations. (Configuration 1) a pixel array including a plurality of pixel units arranged in a matrix, each pixel unit having an avalanche photodiode and a counter that generates a count value based on photons incident on the avalanche photodiode; The pixel array is divided into a plurality of tile regions in a matrix form; a storage unit provided for each of the tile regions, the storage unit storing a reference value used for detecting an event based on the count value; A photoelectric conversion device comprising: (Configuration 2) 2. The photoelectric conversion device according to configuration 1, wherein each of the plurality of tile regions includes a comparison unit that detects the event based on a difference between the reference value and the count value. (Configuration 3) 3. The photoelectric conversion device according to configuration 1 or 2, wherein at least one of the plurality of pixel units in the tile region is an event detection pixel unit for detecting the event. (Configuration 4) the tile region includes a plurality of the event detection pixel units, the comparison unit is shared by the plurality of event detection pixel units; 4. The photoelectric conversion device according to configuration 3. (Configuration 5) the tile region includes a plurality of the event detection pixel units, The storage unit is shared by the plurality of event detection pixel units. The photoelectric conversion device according to configuration 3 or 4. (Configuration 6) 6. The photoelectric conversion device according to any one of configurations 3 to 5, wherein the storage unit is provided for each of the event detection pixel units. (Configuration 7) 7. The photoelectric conversion device according to any one of configurations 3 to 6, wherein the reference value is the count value of the event detection pixel unit during a period in which the event is detected. (Configuration 8) 8. The photoelectric conversion device according to any one of configurations 3 to 7, wherein, when the event is detected, the reference value is updated to the count value of the event detection pixel unit. (Configuration 9) 9. The photoelectric conversion device according to any one of configurations 2 to 8, wherein the comparison section compares the difference with a predetermined threshold and determines that the event has been detected. (Configuration 10) the predetermined threshold value includes a first threshold value and a second threshold value different from the first threshold value; If the difference is a positive number, the difference is compared to the first threshold; If the difference is a negative number, the difference is compared to the second threshold. 10. The photoelectric conversion device according to configuration 9. (Configuration 11) the first threshold and the second threshold are changed based on a change in the reference value; a change amount of the first threshold value based on a change in the reference value is different from a change amount of the second threshold value based on a change in the reference value; 11. The photoelectric conversion device according to configuration 10. (Configuration 12) 12. The photoelectric conversion device according to any one of configurations 1 to 11, wherein each of the plurality of tile regions includes an input / output unit that transfers a signal output from a first tile region to a second tile region. (Configuration 13) In each of the tile regions, the number of the pixel units arranged in a row direction is different from the number of the pixel units arranged in a column direction, the number of pixel units arranged in the row direction is set to be relatively larger than the number of pixel units arranged in the column direction as the frequency of the signals being transferred in the row direction is higher than the frequency of the signals being transferred in the column direction, the number of pixel units arranged in the column direction is set to be relatively larger than the number of pixel units arranged in the row direction as the frequency of the signals being transferred in the column direction is higher with respect to the frequency of the signals being transferred in the row direction; 13. The photoelectric conversion device according to claim 12. (Configuration 14) The avalanche photodiode is charged in response to a pulse signal input to the pixel unit, the plurality of pixel units include the event detection pixel unit and an imaging pixel unit for imaging, a frequency of the pulse signal input to the event detection pixel unit is different from a frequency of the pulse signal input to the imaging pixel unit; 14. The photoelectric conversion device according to any one of configurations 3 to 13. (Configuration 15) The avalanche photodiode is charged in response to a pulse signal input to the pixel unit, the plurality of pixel units include the event detection pixel unit and an imaging pixel unit for imaging, A photoelectric conversion device described in any one of configurations 3 to 14, wherein a period during which the count value is generated in one cycle of the pulse signal input to the event detection pixel unit is different from a period during which the count value is generated in one cycle of the pulse signal input to the imaging pixel unit. (Configuration 16) 16. The photoelectric conversion device according to any one of configurations 3 to 15, further comprising a selection switch for changing the number of the event detection pixel units. (Configuration 17) 17. The photoelectric conversion device according to any one of configurations 3 to 16, further comprising a selection switch for selecting a pixel unit to be operated as the event detection pixel unit from the plurality of pixel units. (Configuration 18) 18. The photoelectric conversion device according to any one of configurations 2 to 17, wherein in each of the plurality of tile regions, the counter is disposed relatively outside with respect to the memory unit and the comparison unit. (Configuration 19) 19. The photoelectric conversion device according to any one of configurations 1 to 18, which is a back-illuminated photoelectric conversion device. (Configuration 20) An imaging device including the photoelectric conversion device according to any one of configurations 1 to 18; a signal processing unit that processes imaging data output from the imaging device; An imaging system comprising: (Configuration 21) A mobile object, The photoelectric conversion device according to any one of configurations 1 to 18, a distance information acquisition unit that acquires distance information to an object from a signal output from the photoelectric conversion device; A control unit that controls the moving object based on the distance information; A moving object comprising:

[0123] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0124] It should be noted that the above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0125] 10 pixels 10a Event detection pixel 10b Imaging pixel 11 Avalanche photodiode 100 Photoelectric conversion device 201 tile area 202 Control section 203 First Processing Section 204 Second Processing Section 301 Pixel section 302 Storage section 303 Arithmetic section 304 Threshold control section 305 Comparison Section 306 Input / output section 403 Counter

Claims

1. a pixel array including a plurality of pixel units arranged in a matrix, each pixel unit having an avalanche photodiode and a counter that generates a count value based on photons incident on the avalanche photodiode; The pixel array is divided into a plurality of tile regions in a matrix form; a storage unit provided for each of the tile regions, the storage unit storing a reference value used for detecting an event based on the count value; A photoelectric conversion device comprising:

2. The photoelectric conversion device according to claim 1 , wherein each of the plurality of tile regions includes a comparison unit that detects the event based on a difference between the reference value and the count value.

3. The photoelectric conversion device according to claim 2 , wherein at least one of the plurality of pixel units in the tile region is an event detection pixel unit for detecting the event.

4. the tile region includes a plurality of the event detection pixel units, the comparison unit is shared by the plurality of event detection pixel units; The photoelectric conversion device according to claim 3 .

5. the tile region includes a plurality of the event detection pixel units, The storage unit is shared by the plurality of event detection pixel units. The photoelectric conversion device according to claim 3 .

6. The photoelectric conversion device according to claim 3 , wherein the storage unit is provided for each of the event detection pixel units.

7. The photoelectric conversion device according to claim 3 , wherein the reference value is the count value of the event detection pixel unit during a period in which the event is detected.

8. The photoelectric conversion device according to claim 3 , wherein, when the event is detected, the reference value is updated to the count value of the event detection pixel unit.

9. The photoelectric conversion device according to claim 2 , wherein the comparison section compares the difference with a predetermined threshold value and determines that the event has been detected.

10. the predetermined threshold value includes a first threshold value and a second threshold value different from the first threshold value; If the difference is a positive number, the difference is compared to the first threshold; If the difference is a negative number, the difference is compared to the second threshold. The photoelectric conversion device according to claim 9 .

11. the first threshold and the second threshold are changed based on a change in the reference value; a change amount of the first threshold value based on a change in the reference value is different from a change amount of the second threshold value based on a change in the reference value; The photoelectric conversion device according to claim 10.

12. The photoelectric conversion device according to claim 1 , wherein each of the plurality of tile regions includes an input / output unit that transfers a signal output from a first tile region to a second tile region.

13. In each of the tile regions, the number of the pixel units arranged in a row direction is different from the number of the pixel units arranged in a column direction, the number of pixel units arranged in the row direction is set to be relatively larger than the number of pixel units arranged in the column direction as the frequency of the signals being transferred in the row direction is higher than the frequency of the signals being transferred in the column direction, the number of pixel units arranged in the column direction is set to be relatively larger than the number of pixel units arranged in the row direction as the frequency of the signals being transferred in the column direction is higher with respect to the frequency of the signals being transferred in the row direction; The photoelectric conversion device according to claim 12.

14. The avalanche photodiode is charged in response to a pulse signal input to the pixel unit, the plurality of pixel units include the event detection pixel unit and an imaging pixel unit for imaging, a frequency of the pulse signal input to the event detection pixel unit is different from a frequency of the pulse signal input to the imaging pixel unit; The photoelectric conversion device according to claim 3 .

15. The avalanche photodiode is charged in response to a pulse signal input to the pixel unit, the plurality of pixel units include the event detection pixel unit and an imaging pixel unit for imaging, The photoelectric conversion device according to claim 3 , wherein a period during which the count value is generated in one cycle of the pulse signal input to the event detection pixel unit is different from a period during which the count value is generated in one cycle of the pulse signal input to the imaging pixel unit.

16. The photoelectric conversion device according to claim 3 , further comprising a selection switch for changing the number of the event detection pixel units.

17. The photoelectric conversion device according to claim 3 , further comprising a selection switch for selecting a pixel unit to be operated as the event detection pixel unit from the plurality of pixel units.

18. The photoelectric conversion device according to claim 2 , wherein in each of the plurality of tile regions, the counter is disposed relatively outside the storage section and the comparison section.

19. The photoelectric conversion device according to claim 1 , which is a back-illuminated type photoelectric conversion device.

20. An imaging device including the photoelectric conversion device according to any one of claims 1 to 18; a signal processing unit that processes imaging data output from the imaging device; An imaging system comprising:

21. A mobile object, The photoelectric conversion device according to any one of claims 1 to 18, a distance information acquisition unit that acquires distance information to an object from a signal output from the photoelectric conversion device; A control unit that controls the moving object based on the distance information; A moving object comprising:

Images