Photoelectric converter, imaging system, light detection system, and mobile body

Abstract

To reduce current consumption while reducing an error in an image signal.SOLUTION: A photoelectric conversion device in a disclosure has: an avalanche photodiode; a pulse generation unit that converts an output from the avalanche photodiode into a pulse signal; a pulse count unit that counts the pulse signals and outputs a pulse count value; a time count unit that outputs a time count value representing the time from when the pulse generation unit starts the operation; an output unit that, when the pulse count value does not exceed a threshold, outputs the pulse count value, and when the pulse count value exceeds the threshold, ends the counting in the pulse count unit and outputs the time count value at the time point when the pulse count value exceeds the threshold; and a threshold arithmetic unit that calculates the threshold by using the time count value.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a photoelectric conversion device, an imaging system, an optical detection system, and a moving body.

Background Art

[0002] A photon counting type photoelectric conversion device that counts the number of photons incident on a photodiode during an exposure period and outputs the counted value as a signal value is known. As a method for realizing the photon counting method, a method using an avalanche photodiode and a counter has been proposed. When a reverse bias voltage greater than the breakdown voltage is applied to the avalanche photodiode, the generated carriers due to the incidence of a single photon cause avalanche multiplication, and a large current is generated. By counting the pulse signal due to this current with a counter circuit, a signal value corresponding to the number of photons incident on the avalanche photodiode can be obtained. On the other hand, a large current is generated every time photons are incident on a plurality of pixels. When a high-intensity light continues to irradiate a multi-pixel image sensor, current continues to be repeatedly generated in many pixels, leading to an increase in power consumption. In order to solve this problem, techniques for reducing the power consumption when irradiated with high-intensity light have been proposed.

[0003] The technique described in Patent Document 1 determines whether to perform pulse counting in subsequent sub-frames based on the magnitude of the photon count value of a sub-frame. When the count value of the sub-frame is high, in the next sub-frame, pulse counting is not performed and the immediately previous count value is output. The technique described in Patent Document 2 estimates the count value on the high-intensity side exceeding the threshold based on the inclination of the increase in the count value at the time of exceeding the threshold, and outputs the estimated count value.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

[0005] However, in the technology described in Patent Document 1, when the subject being photographed changes, the error between the actual number of photons and the pulse count value becomes large, making it impossible to obtain a correct image signal. Furthermore, in the technology described in Patent Document 2, it is not possible to reduce the current consumption further than the current value corresponding to a predetermined threshold.

[0006] This invention has been made in view of the above-mentioned problems, and aims to reduce current consumption while reducing errors in image signals. [Means for solving the problem]

[0007] According to one disclosure of this specification, an avalanche photodiode, a pulse generation unit that converts the output from the avalanche photodiode into a pulse signal, a pulse count unit that counts the pulse signal and outputs a pulse count value, a time count unit that outputs a time count value representing the time since the pulse generation unit started operating, and an output unit that outputs the pulse count value when the pulse count value does not exceed a threshold, and terminates the count in the pulse count unit and outputs the time count value at the time when the pulse count value exceeds the threshold, Based on a comparison of the magnitude of the brightness change between the first frame and the second frame which is later than the first frame, and the set value, A photoelectric conversion device is provided, characterized by having a threshold setting unit that changes the threshold. [Effects of the Invention]

[0008] According to the present invention, it is possible to reduce current consumption while reducing errors in the image signal. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of the photoelectric conversion device in the first embodiment. [Figure 2] This figure shows an example of the arrangement of the sensor substrate in the first embodiment. [Figure 3] This figure shows an example of the circuit board arrangement in the first embodiment. [Figure 4] This is a circuit diagram of the APD and pulse generation unit in the first embodiment. [Figure 5] This figure shows the relationship between the operation of the APD and the output signal in the first embodiment. [Figure 6] This is a block diagram of the photoelectric converter in the first embodiment. [Figure 7] This is a flowchart illustrating the operation of the photoelectric converter in the first embodiment. [Figure 8] This is an example of a subject used to illustrate the operation of the photoelectric converter in the first embodiment. [Figure 9] This is another example of a subject used to illustrate the operation of the photoelectric converter in the first embodiment. [Figure 10] This is a diagram illustrating the operation of the photoelectric converter in the first embodiment. [Figure 11] This is a diagram illustrating the operation of the photoelectric converter in the first embodiment. [Figure 12] This is a diagram illustrating the operation of the photoelectric converter in the first embodiment. [Figure 13] This is a diagram illustrating the operation of the photoelectric converter in the first embodiment. [Figure 14] This is a diagram illustrating the operation of the photoelectric converter in the second embodiment. [Figure 15] This is a diagram illustrating the operation of the photoelectric converter in the second embodiment. [Figure 16] This is a diagram illustrating the operation of the photoelectric converter in the second embodiment. [Figure 17] This is a diagram illustrating the operation of the photoelectric converter in the second embodiment. [Figure 18]This is a diagram for explaining the operation of the photoelectric conversion device in the second embodiment. [Figure 19] This is a block diagram of the photoelectric conversion device in the third embodiment. [Figure 20] This is a diagram showing a configuration example of a pixel block in the third embodiment. [Figure 21] This is a block diagram of the photoelectric conversion device in the fourth embodiment. [Figure 22] This is a block diagram of the photoelectric conversion device in the fifth embodiment. [Figure 23] This is a diagram showing an arrangement example of a circuit board in the sixth embodiment. [Figure 24] This is a block diagram of the imaging system in the eighth embodiment. [Figure 25] This is a block diagram of the light detection system in the ninth embodiment. [ [Figure 26] This is a schematic diagram of an endoscopic surgery system in the tenth embodiment. [Figure 27A] This is a schematic diagram of the light detection system in the eleventh embodiment. [Figure 27B] This is a schematic diagram of a moving body in the twelfth embodiment. [[ID=三十二]] [Figure 28] This is a flowchart showing the operation of the light detection system in the thirteenth embodiment. [Figure 29] This is a diagram showing a specific example of the electronic device in the first embodiment.

Embodiments for Carrying Out the Invention

[0010] Embodiments of the present invention will be described below with reference to the drawings. The embodiments shown below are for embodying the technical idea of the present invention and do not limit the present invention. The sizes and positional relationships of the members shown in each drawing may be exaggerated for clarity of explanation. In the following description, the same configurations may be denoted by the same numbers and the description may be omitted.

[0011] [[ID=四十八]][First Embodiment] The configuration of the photoelectric converter in this embodiment will be explained using Figures 1 to 4. The photoelectric converter has SPAD pixels including an avalanche photodiode (hereinafter referred to as "APD"). The conductivity type of the charge used as the signal charge among the charge pairs generated in the APD is called the first conductivity type. The first conductivity type refers to a conductivity type in which the majority carriers are charges of the same polarity as the signal charge. The conductivity type opposite to the first conductivity type is called the second conductivity type. Below, an example is described in which the signal charge is an electron, the first conductivity type is N-type, and the second conductivity type is P-type, but the signal charge may be a hole, the first conductivity type may be P-type, and the second conductivity type may be N-type.

[0012] In this specification, "plan view" refers to viewing from a direction perpendicular to the light incident surface of the semiconductor substrate, as described later. "Cross section" refers to the surface of the sensor substrate 1 in a direction perpendicular to the light incident surface. If the light incident surface of the semiconductor layer is rough when viewed microscopically, the plan view is defined based on the light incident surface of the semiconductor layer when viewed macroscopically. "Depth direction" is the direction from the light incident surface (first surface) of the sensor substrate 1 toward the surface (second surface) on which the circuit board 2 is located.

[0013] Figure 1 is a schematic diagram of the photoelectric converter in this embodiment, showing the configuration of a stacked type photoelectric converter 100. The photoelectric converter 100 includes a sensor substrate (first substrate) 1 and a circuit board (second substrate) 2 stacked on top of each other, and the sensor substrate 1 and the circuit board 2 are electrically connected to each other. The photoelectric converter in this embodiment is a back-illuminated type photoelectric converter in which light is incident from the first surface of the sensor substrate 1 and the circuit board 2 is arranged on the second surface of the sensor substrate 1. The sensor substrate 1 has a first semiconductor layer having a photoelectric conversion element described later and a first wiring structure. The circuit board 2 has a second semiconductor layer having a circuit such as a signal processing unit described later and a second wiring structure. The second semiconductor layer, the second wiring structure, the first wiring structure, and the first semiconductor layer are stacked in that order to constitute the photoelectric converter 100.

[0014] In the following description, the sensor substrate 1 and the circuit board 2 may be, but are not limited to, diced chips. For example, each substrate may be a wafer. Furthermore, each substrate may be stacked in a wafer state and then diced, or it may be made into chips and then stacked and bonded together. The sensor substrate 1 has a pixel region 1a, and the circuit board 2 has a circuit region 2a that processes the signals detected by the pixel region 1a.

[0015] Figure 2 shows an example of the arrangement of the sensor substrate 1. Multiple pixels 10 each contain an APD 11 and are arranged in a two-dimensional array in a planar view, forming a pixel region 1a.

[0016] Pixel 10 is typically a pixel used to form an image, but when used in TOF (Time of Flight), it is not necessarily required to form an image. In other words, pixel 10 may be a pixel used to measure the time and amount of light that arrives.

[0017] Figure 3 shows an example of the arrangement of the circuit board 2. The circuit board 2 has a signal processing unit 20, a vertical scanning circuit 251, a readout circuit 253, a horizontal scanning circuit 257, an output calculation unit 254, a control pulse generation circuit 255, scan lines 256, and signal lines 259. In a plan view, the circuit area 2a is arranged in the area that overlaps with the pixel area 1a in Figure 2. Furthermore, in a plan view, the vertical scanning circuit 251, readout circuit 253, horizontal scanning circuit 257, output calculation unit 254, and control pulse generation circuit 255 are arranged so as to overlap with the area between the edge of the sensor board 1 in Figure 2 and the edge of the pixel area 1a. In other words, the sensor board 1 has a pixel area 1a and a non-pixel area arranged around the pixel area 1a, and the vertical scanning circuit 251, readout circuit 253, horizontal scanning circuit 257, output calculation unit 254, and control pulse generation circuit 255 are arranged in the area that overlaps with the non-pixel area in a plan view.

[0018] The signal processing unit 20 is electrically connected to the pixels 10 via connecting wiring provided for each pixel 10, and is arranged in a two-dimensional array in a planar view, similar to the pixels 10. The signal processing unit 20 includes a binary counter that counts the photons incident on the pixels 10.

[0019] The vertical scanning circuit 251 receives control pulses supplied from the control pulse generation circuit 255 and supplies control pulses to the signal processing unit 20 corresponding to the pixels 10 of each row via the scan line 256. The vertical scanning circuit 251 may be composed of logic circuits such as a shift register and an address decoder.

[0020] The readout circuit 253 acquires the pulse count value of the digital signal from the signal processing unit 20 of each row via the signal line 259. Then, via the output calculation unit 254, it outputs the output signal to an external signal processing circuit (signal processing device) of the photoelectric converter 100. The readout circuit 253 may also have the function of a signal processing circuit that performs correction of the count value. The horizontal scanning circuit 257 receives a control pulse from the control pulse generation circuit 255 and sequentially outputs the count value of each column in the readout circuit 253 to the output calculation unit 254. As described later, if the pulse count value exceeds a threshold, the output calculation unit 254 estimates the actual image signal (pulse count value) based on the time count value and threshold included in the additional information and replaces the pulse count value with the estimated pulse count value (extrapolation). On the other hand, if the pulse count value is below the threshold, the pulse count value is output as is as an image signal.

[0021] The output calculation unit 254 performs predetermined processing on the pulse count value read by the readout circuit 253 and outputs the image signal to the outside. Furthermore, as will be described later, the output calculation unit 254 can perform processing such as calculation of the pulse count value when the pulse count value exceeds a threshold.

[0022] In Figure 2, the arrangement of photoelectric conversion elements in the pixel region may be arranged in a one-dimensional manner. Furthermore, the effects of the present invention can be achieved even in a configuration with only one pixel, and a configuration with one pixel can also be included in the present invention. In a photoelectric conversion device having multiple pixels, the effect of suppressing the circuit size according to this embodiment becomes even more pronounced. The signal processing unit 20 does not necessarily need to be provided for each pixel 10; for example, one signal processing unit 20 may be shared by multiple pixels 10, and signal processing may be performed sequentially.

[0023] Figure 4 is a block diagram of the APD and pulse generation unit in this embodiment. Figure 4 shows the pixel 10 of the sensor substrate 1 and the pulse generation unit 201 in the signal processing unit 20 of the circuit board 2. An APD 11 is placed in the pixel 10. The pulse generation unit 201 includes a quench element 2011, a waveform shaping unit 2012, a counter circuit 2013, and a selection circuit 2014.

[0024] The APD11 generates charge pairs corresponding to incident light through photoelectric conversion. A voltage VL (first voltage) is supplied to the anode of the APD11. In addition, a voltage VH (second voltage), which is higher than the voltage VL supplied to the anode, is supplied to the cathode of the APD11. A reverse bias voltage is applied to the anode and cathode, making the APD11 capable of avalanche multiplication. When a photon is incident on the APD11 under the reverse bias voltage, the charge generated by the photon undergoes avalanche multiplication, and an avalanche current is generated.

[0025] Depending on the reverse bias voltage, the APD11 can operate in Geiger mode or linear mode. Geiger mode is operation when the potential difference between the anode and cathode is greater than the breakdown voltage, while linear mode is operation when the potential difference between the anode and cathode is near or below the breakdown voltage. An APD operating in Geiger mode is specifically called a SPAD. For example, the voltage VL (first voltage) may be -30V and the voltage VH (second voltage) may be 1V. The APD11 may operate in linear mode or in Geiger mode. When the APD11 operates as a SPAD, the potential difference becomes larger compared to the linear mode APD11, and the voltage withstand effect becomes more pronounced, so it is preferable for the APD11 to operate as a SPAD.

[0026] Quench element 2011 is placed between the power line supplying voltage VH and the cathode of APD11. Quench element 2011 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, suppressing the voltage supplied to APD11 and thereby suppressing avalanche multiplication (quench operation). Quench element 2011 also works to restore the voltage supplied to APD11 to voltage VH by flowing current to compensate for the voltage drop caused by the quench operation (recharge operation).

[0027] The waveform shaping unit 2012 functions as a signal generation unit that generates a detection pulse based on the output generated by the incidence of photons. That is, the waveform shaping unit 2012 shapes the potential change of the cathode of the APD11 obtained during photon detection and outputs a rectangular wave pulse signal (detection pulse). For example, an inverter circuit can be used as the waveform shaping unit 2012. Figure 4 shows an example in which one inverter is used as the waveform shaping unit 2012, but a circuit in which multiple inverters are connected in series may be used. In addition, other circuits that have a waveform shaping effect may be used.

[0028] The counter circuit 2013 counts the pulse signal output from the waveform shaping unit 2012 and holds the count value. The counter circuit 2013 is also supplied with control pulses from the vertical scanning circuit 251 (Figure 3) via the drive line 2016. When the control pulses become active, the signal held by the counter circuit 2013 is reset.

[0029] The selection circuit 2014 includes a switch circuit, a buffer circuit for outputting signals, and the like. The selection circuit 2014 is supplied with control pulses from the vertical scanning circuit 251 in Figure 3 via the drive line 2017. In response to the control pulses, the selection circuit 2014 switches the electrical connection between the counter circuit 2013 and the signal line 219.

[0030] Furthermore, switches such as transistors may be provided between the quench element 2011 and the APD11, and between the APD11 and the signal processing unit 20. In addition, the supply of voltage VH or voltage VL may be electrically switched by switches such as transistors.

[0031] Figure 5 shows the relationship between the operation of the APD and the output signal in this embodiment. Figure 5(a) is an excerpt of the APD11, quench element 2011, and waveform shaping unit 2012 from Figure 4. When the input side of the waveform shaping unit 2012 is nodeA and the output side is nodeB, Figure 5(b) shows the waveform change of nodeA, and Figure 5(c) shows the waveform change of nodeB.

[0032] Between time t0 and time t1, a reverse bias voltage of VH-VL is applied to APD11. When a photon is incident on APD11 at time t1, avalanche multiplication occurs in APD11, an avalanche multiplication current flows through the quench element 2011, and the voltage at nodeA drops. As the voltage drop increases further and the potential difference applied to APD11 decreases, the avalanche multiplication of APD11 stops at time t3, and the voltage level at nodeA stops dropping below a certain value. Subsequently, between time t3 and time t5, a current flows from voltage VL to nodeA to compensate for the voltage drop, and at time t5, nodeA settles to its original voltage level. At this time, between time t2 and time t4, if the voltage level at nodeA falls below the threshold of the waveform shaping unit 2012, nodeB becomes high level. In other words, the voltage waveform of nodeA is shaped by the waveform shaping unit 2012, and a square wave pulse signal is output from nodeB.

[0033] Figure 6 is a block diagram of the photoelectric converter in this embodiment. In Figure 6, for the sake of simplicity, only one of the signal processing unit 20 and output calculation unit 254 of the photoelectric converter 100 in Figure 3 is shown, and the readout circuit 253 and other components between the signal processing unit 20 and the output calculation unit 254 are omitted.

[0034] The signal processing unit 20 is provided for each pixel 10 and includes a pulse generation unit 201, a pulse counting unit 202, a time counting unit 203, a threshold calculation unit 204, and an output unit 205. As explained in Figure 4, the pulse generation unit 201 converts photons incident through the lens into pulse signals.

[0035] The pulse counting unit 202 counts the pulses generated by the pulse generation unit 201 and outputs a pulse count value. The pulse counting unit 202 is composed of, for example, a flip-flop and may be a binary counter having a predetermined bit length.

[0036] The time counting unit 203 measures the time elapsed since the pulse counting unit 202 started counting pulses and outputs it as a time count value. The time count value is obtained by counting the clock signal. For example, suppose the period of the clock signal is 0.1 microseconds and the time count value has counted up from "0" to "10" (decimal notation). In this case, the time count value represents that 1 microsecond has elapsed since the pulse counting unit 202 started counting photons. It is preferable that the period of the clock signal be sufficiently shorter than the frequency of photon incidence in order to calculate the rate of change (slope) of the pulse count value per unit time.

[0037] The threshold calculation unit 204 calculates a threshold pulse count value that terminates the operation of the pulse generation unit 201 and the pulse count unit 202. The threshold pulse count value is expressed as the number of pulses. For example, the threshold calculation unit 204 calculates the rate of change (slope) of the pulse count value per unit time based on the pulse count value of the pulse count unit 202 and the time count value of the time count unit 203, and changes the threshold based on the rate of change. Specifically, if the difference between the current rate of change and the rate of change of the previous frame is less than or equal to a predetermined set value, the threshold is lowered, and if the difference exceeds the set value, the threshold is raised. The threshold calculation unit 204 includes a memory that holds the rate of change of the pulse count value in the previous frame in order to compare the rates of change between frames.

[0038] The output unit 205 outputs the pulse count value as is, and also outputs an eigenvalue as one of the additional pieces of information, if the pulse count value is below the threshold. The eigenvalue is a value that indicates that the pulse count value has not exceeded the threshold. The eigenvalue can be any value that is easily distinguishable from the pulse count value, for example, a value in which all bits of the pulse count value are "1" or "0". Alternatively, the eigenvalue may be assigned to the most significant bit of the pulse count value. If the pulse count value exceeds the threshold, the pulse count unit 202 terminates the pulse count operation, and the output unit 205 outputs time information (time count value) at the time the pulse count value exceeded the threshold. At the same time, the output unit 205 outputs the threshold as one of the additional pieces of information.

[0039] If the pulse count value exceeds the threshold, the output calculation unit 254 calculates an estimated value of the actual image signal (pulse count value) based on the time count value and threshold included in the additional information, and replaces (extrapolates) the pulse count value from the output unit 205 with the estimated value. On the other hand, if the pulse count value is below the threshold, the pulse count value from the output unit 205 is output as the image signal as is.

[0040] Figure 7 is a flowchart illustrating the operation of the photoelectric converter in this embodiment, showing the process from the pulse signal generation process of the pulse generation unit 201 to the output of the image signal of the output calculation unit 254 during video capture.

[0041] As soon as the photoelectric converter 100 starts imaging, the signal processing unit 20 resets the pulse count value of the pulse count unit 202, and the time count unit 203 starts counting the time count value. The threshold calculation unit 204 sets the threshold to a predetermined initial value.

[0042] In step S101, the photoelectric converter 100 determines whether or not an operation to terminate imaging has been performed. If an operation or instruction to terminate imaging has been performed (YES in step S101), the photoelectric converter 100 terminates the imaging operation. Unless an operation or instruction to terminate imaging has been performed (NO in step S101), the photoelectric converter 100 repeatedly executes the processes in steps S102 to S114.

[0043] In step S102, if no photons are incident and the pulse generation unit 201 does not generate a pulse (NO in step S102), the signal processing unit 20 proceeds to the processing in step S104. On the other hand, if photons are incident and the pulse generation unit 201 generates a pulse (YES in step S102), the pulse generation unit 201 detects the rising edge of the pulse and increments the pulse count value.

[0044] Next, the output unit 205 determines whether the pulse count value has exceeded a preset threshold. If the pulse count value has not exceeded the threshold (NO in step S103), or if no pulses are detected due to incident light (NO in step S102), the output unit 205 waits until the time count value is counted up (step S104). Here, it is assumed that the time count value is counted up by the clock signal, and that the period of the clock signal is sufficiently faster than the frequency of pulse count occurrence.

[0045] In step S105, the output unit 205 determines whether the time count value has reached the exposure time for one frame period. If the time count value has not reached the exposure time for one frame period (NO in step S105), the process returns to step S102. On the other hand, if the time count value has reached the exposure time for one frame period (YES in step S105), the output unit 205 outputs the pulse count value at the current time and additional information including an eigenvalue indicating that the pulse count value has not exceeded the threshold (step S106).

[0046] In step S107, the threshold calculation unit 204 calculates and stores the amount of change per unit time of the pulse count value, i.e., the rate of change (slope), based on the pulse count value and the time count value corresponding to the pulse count value.

[0047] On the other hand, in step S103, if the pulse count value exceeds a threshold (YES in step S103), the pulse generation unit 201 and the pulse count unit 202 stop operating, and the output unit 205 acquires the current time count value. The photoelectric converter 100 may also cut off the current supplied to the APD 11 and stop the operation of the APD 11.

[0048] In step S109, the output unit 205 waits until the time count value reaches the exposure time corresponding to the frame period. Then, in step S110, the output unit 205 outputs the current time count value and additional information including a threshold value.

[0049] In step S111, the threshold calculation unit 204 calculates and holds the rate of change of the pulse count value based on the threshold value and the time count value. The time count value at this time corresponds to the time from when the pulse count value starts counting until the pulse count value exceeds the threshold value.

[0050] In step S112, the threshold calculation unit 204 determines whether the difference in the rate of change of the pulse count value between past frames and the current frame is within a predetermined set value. If the difference in the rate of change of the pulse count value is within the set value (YES in step S112), the threshold calculation unit 204 determines that it is a scene with little change and lowers the threshold by a predetermined amount (step S113). The set value represents the allowable error value for the rate of change between frames. The difference in the rate of change and the set value can also be expressed as absolute values.

[0051] In step S112, if the difference in the rate of change between past frames and the current frame is greater than a set value (NO in step S112), the threshold calculation unit 204 increases the threshold (step S114). The amount of the threshold increase may be a preset value, or the amount of increase may be determined based on the current threshold. The threshold may also be reset to its initial value. Furthermore, the threshold may be changed or reset to its initial value at the time when a certain number of frames have elapsed. In addition, as will be described later, the threshold may be limited by an upper limit.

[0052] Note that the processing in steps S112, S113, and S114 requires the rate of change of the pulse count value in past frames. For this reason, the processing in steps S112, S113, and S114 does not need to be performed in the processing of the first frame at the start of shooting. In steps S105 and S109, the frame period is determined by the frame rate of the moving image to be captured. For example, if the frame rate is 30fps, the frame period will be 1 / 30 second. Also, if a bright scene is expected, a period shorter than the frame period may be used. In other words, instead of the frame period, different periods may be used depending on the brightness of the scene.

[0053] Here, we will use Figures 8 and 9 as examples to specifically explain the effects of this embodiment. Figure 8 shows one frame of a bright video, such as a daytime shooting scene. In Figure 8, vehicle 302 is moving from left to right, and vehicle 303 is moving toward the shooting position with its headlights on. Also, the LED traffic light 301 is located above vehicle 302. Because the shooting scene in Figure 8 is generally high brightness, the pulse count value becomes high for most pixels, resulting in high power consumption for conventional photon counting image sensors and photoelectric converters. Figure 9 is similar to Figure 8, except that it shows one frame of a video in a dark state, such as at night. That is, vehicle 302 is moving from left to right, and vehicle 303 is moving toward the shooting position with its headlights on. Also, the LED traffic light 301 is located above vehicle 302. In Figure 9, the moving vehicle 302 and the vehicle 303 with its headlights on are prone to sudden changes in brightness. Therefore, if video recording continues with the threshold set too low, the error in the pulse count value will increase, and the captured image may differ from the actual shooting scene. According to this embodiment, in both shooting scenes shown in Figures 8 and 9, it is possible to reduce the error in the pulse count value and reduce the difference between the image signal and the actual shooting scene.

[0054] The threshold reduction amount per frame (a predetermined amount) may be a pre-set fixed value, or the reduction amount may be determined based on the current threshold. For example, the higher the current threshold, the greater the threshold reduction amount may be. When a relatively large change in brightness occurs, such as the change from the scene in Figure 8 to the scene in Figure 9, or when the moving vehicle 302 is moving at high speed, the difference in the rate of change will be large. On the other hand, depending on the shooting scene, the difference in the rate of change may be less apparent. When the difference from the threshold of a previous frame is small, the threshold reduction amount per frame may be reduced to reduce distortion of the captured image. Also, as will be described later, a lower limit of the threshold may be set so that the threshold does not fall below the lower limit.

[0055] Furthermore, in step S112, the setting value used for comparison with the difference between frames may be a fixed value such as "10" (decimal), or it may be determined based on the current threshold. For example, the higher the current threshold, the higher the setting value may be set, or vice versa. The flowchart described above is merely an example, and the order of the processing steps may be changed as long as the processing result does not change.

[0056] According to this embodiment, when the difference in the rate of change of the pulse count value between frames is small, the threshold is controlled to be lower. A lower threshold increases the power consumption reduction effect. Conversely, when the difference in the rate of change is large, the threshold is controlled to be higher, making it possible to reduce the difference between the pulse count value and the actual scene.

[0057] Figures 10 and 11 are diagrams illustrating the operation of the photoelectric converter in this embodiment, showing the relationship between the pulse count value and the threshold value. The horizontal axis represents the exposure time, and the vertical axis represents the pulse count value.

[0058] Figures 10 and 11 show the transition of pulse count values ​​in scenes with relatively high brightness and little change between frames, such as the daytime scene in Figure 8. In scenes with high brightness and little change between frames, the pulse count value is high, and the difference in the rate of change of pulse count values ​​between frames is generally small.

[0059] Figures 10(A)-(D) and 11(E)-(G) show the transitions of pulse count values ​​and threshold Th from the first to the seventh frame. In the figures, "Th_H" and "Th_L" represent the upper and lower limits of the threshold Th, respectively. The upper limit Th_H and lower limit Th_L can be set to values ​​other than the most significant and least significant bits. The threshold Th is clipped at the upper limit Th_H and lower limit Th_L. Note that the threshold Th does not necessarily have to be limited to the range of the upper limit Th_H and lower limit Th_L, and clipping is not mandatory.

[0060] At times t111 to t112 in Figure 10(A) (first frame), the pulse count value exceeds the threshold Th, and the pulse generation unit 201 and pulse count unit 202 stop operating. Compared to the case where no threshold is set, the exposure time is shortened by the duration of the period t111 to t112. Since pulse counting is no longer performed after time t111, it is possible to reduce the power consumption of the photoelectric converter and imaging equipment. The output unit 205 outputs the time count value and threshold Th at time t111, when the pulse count value exceeds the threshold Th, as additional information. The threshold calculation unit 204 calculates and holds the rate of change (slope) of the pulse count value using the time count value and threshold Th value at time t111, when the threshold Th exceeds. The output calculation unit 254 estimates and outputs the pulse count value after the threshold Th is exceeded based on the rate of change. Note that in the first frame at the start of imaging, the rate of change of past frames is not held, so the threshold calculation unit 204 does not decide whether or not to change the threshold Th.

[0061] In Figure 10(B) (second frame), the output unit 205 outputs the time count value and threshold Th at time t121 when the pulse count value exceeds the threshold, as additional information, similar to the first frame. Furthermore, the threshold calculation unit 204 calculates the rate of change of the pulse count value and compares the rate of change in the first frame with the rate of change in the second frame. In scenes with little change, the rate of change between frames will be approximately the same, and the difference in the rate of change between frames will be smaller than the set value.

[0062] In Figure 10(C) (third frame), the threshold calculation unit 204 lowers the threshold Th based on the judgment result in the second frame. When the threshold Th is lowered, the exposure time is further shortened by the period t131 to t142. That is, pulse counting is not performed during the period t131 to t142, so it is possible to reduce the power consumption of the photoelectric converter and imaging device compared to when a fixed threshold Th is used.

[0063] In Figures 10(D) (fourth frame) and 11(E) (fifth frame), the difference in the rate of change between frames is small, so the threshold calculation unit 204 further lowers the threshold Th. In the fourth frame, the exposure time is shortened by the period t141 to t142. In the fifth frame, the threshold Th decreases to the lower limit Th_L, and the exposure time is shortened even further by the period t151 to t152. In this way, the power consumption reduction effect increases as the threshold Th decreases.

[0064] In Figure 11(F) (6th frame), the threshold Th remains at its lower limit Th_L. In the 6th frame as well, the exposure time is shortened by only the period t161~t162, resulting in reduced power consumption.

[0065] As shown in Figures 11(E) and (F), if the threshold Th remains at the lower limit Th_L for an extended period, the difference between the captured image and the actual scene may increase. Therefore, when a predetermined number of frames have reached the lower limit Th_L, for example in Figure 11(G) (the 7th frame), the threshold calculation unit 204 increases the threshold Th and lengthens the pulse count time. This process reduces the difference between the captured image and the actual scene. Even in this case, the pulse count stops after time t171, and the effect of reducing current consumption is achieved.

[0066] Figure 12 is a diagram illustrating the operation of the photoelectric converter in this embodiment, showing an example of operation when the subject is moving at high speed or when there is a sudden change in brightness.

[0067] Figures 12(A) to (O) show the relationship between pulse count values ​​and threshold Th for each of the 1st to 15th frames. The horizontal axis represents exposure time, and the vertical axis represents pulse count values. Note that the graph of pulse count values ​​after the threshold Th is exceeded shows the actual number of incident photons. After the pulse count value exceeds the threshold Th, the pulse count actually stops and is replaced with an estimated value.

[0068] Figures 12(A) to (E) (frames 1 to 5) show the pulse count value and threshold Th when the brightness is low. Figures 12(F) to (J) (frames 6 to 10) show the pulse count value and threshold Th when the brightness changes rapidly from a low state to a high state. Figures 16(K) to (O) (frames 11 to 15) show the pulse count value and threshold Th when the high brightness state continues.

[0069] In Figures 12(A) to (E) (frames 1 to 5), the difference in the rate of change of the pulse count value is small, so the threshold Th gradually decreases from the upper limit Th_H with each frame. For example, the difference between the rate of change of the pulse count value in Figure 12(A) (frame 1) and the rate of change in Figure 12(B) (frame 2) is approximately zero, and the threshold in Figure 12(C) (frame 3) decreases by a predetermined amount. In Figure 12(E) (frame 5), the threshold Th decreases to the lower limit Th_L. In Figures 12(A) to (E) (frames 1 to 5), the brightness is low, so the pulse count value does not exceed the threshold Th.

[0070] In Figure 12(F) (6th frame) and Figure 12(G) (7th frame), the difference in the rate of change of the pulse count value is small, but since the threshold Th has reached the lower limit Th_L, no process is performed to lower the threshold Th. Also, in Figure 12(H) (8th frame), the rate of change A08 in the 8th frame is larger than the rate of change A07 in the 7th frame, creating a difference between the rates of change A07 and A08. However, since the difference between the rates of change A07 and A08 is smaller than the set value, the threshold Th is maintained at the lower limit Th_L.

[0071] In Figure 12(I) (9th frame), the difference between the rate of change A08 in the 8th frame and the rate of change A09 in the 9th frame exceeds the set value. Therefore, in Figure 12(J) (10th frame), the threshold Th rises to the upper limit Th_H. The output calculation unit estimates and outputs the pulse count value after the threshold is exceeded based on the rate of change A09. In the figure, the pulse count value after the threshold Th is exceeded represents the actual number of incident photons, and the pulse count value after the threshold Th is exceeded is the graph of the rate of change A09. In Figure 12(J) (10th frame), the difference between the rate of change A09 in Figure 12(I) (9th frame) and the rate of change A10 in Figure 12(J) (10th frame) exceeds the set value. However, since the threshold Th is the upper limit Th_H, no process is performed to increase the threshold Th.

[0072] In Figure 12(K) (frame 11), an even brighter subject is captured, and the difference between the rate of change A10 in Figure 12(J) (frame 10) and the rate of change A11 in Figure 12(K) (frame 11) exceeds the set value. However, since the threshold Th is at the upper limit Th_H, the threshold Th is maintained at the upper limit Th_H.

[0073] In Figures 12(L) to (O) (frames 12 to 15), high-luminance subjects are captured, but the change in luminance is small. Therefore, the difference in the rate of change between frames becomes small, and the threshold Th gradually decreases towards the lower limit Th_L.

[0074] Figure 13 is a diagram illustrating the operation of the photoelectric converter in this embodiment, and shows other examples of operation in cases of high-speed movement of the subject or rapid changes in brightness.

[0075] Figures 13(A) to (O) show the relationship between pulse count values ​​and threshold Th for each of the 1st to 15th frames. The horizontal axis represents exposure time, and the vertical axis represents pulse count values. Note that after the pulse count value exceeds the threshold Th, the count actually stops and is replaced with an estimated value, but Figure 13 shows the actual number of incident photons for reference.

[0076] In Figures 13(A) to (E) (frames 1 to 5), a high-luminance state continues, and the threshold Th gradually decreases from the upper limit Th_H to the lower limit Th_L with each frame. In Figures 13(F) to (J) (frames 6 to 10), the luminance slowly decreases. In Figure 13(G) (frame 7), the difference between the rate of change A06 in frame 6 and the rate of change A07 in frame 7 exceeds the set value, and the threshold Th in Figure 13(H) (frame 8) rises to the upper limit Th_H. In Figures 13(I) to (O) (frames 9 to 15), a low-luminance state continues. As the difference with the rate of change of the previous frame becomes smaller, the threshold Th gradually decreases toward the lower limit Th_L. The threshold Th does not go below the lower limit Th_L, but in the low-luminance state, the pulse count value does not exceed the threshold Th.

[0077] As described above, in this embodiment, pulse counting stops when the pulse count value exceeds a threshold. This reduces the current consumption of the photoelectric converter. Furthermore, by using the time count value at the time the pulse count value exceeds the threshold and the threshold value, it becomes possible to more accurately estimate the pulse count value after the threshold is exceeded and output an image signal with less error. Therefore, according to this embodiment, it is possible to reduce current consumption while reducing errors in the image signal.

[0078] Furthermore, by changing the threshold based on the difference in the rate of change of pulse count values ​​between frames, it becomes possible to reduce the difference between the image signal and the actual scene.

[0079] [Second Embodiment] Next, the photoelectric converter in this embodiment will be described. The photoelectric converter in this embodiment differs from the photoelectric converter in the first embodiment in that it performs processing in the subframe. The following description will focus on the configuration and operation that differ from the first embodiment. Components common to the first embodiment will be denoted by the same reference numerals, and their descriptions may be omitted.

[0080] In this embodiment, a comparison is performed between the pulse count value and the threshold Th for each subframe. The operation and effects of this embodiment are described in detail below.

[0081] Figure 14 is a diagram illustrating the operation of the photoelectric converter in this embodiment, showing the pulse count value in a scene with little change in brightness. The graph of pulse count values ​​after exceeding the threshold Th shows the actual number of incident photons. After the pulse count value exceeds the threshold Th, the pulse count actually stops and is replaced with an estimated value.

[0082] Figure 14(A) shows the pulse count value for one frame, and Figure 14(B) shows the pulse count values ​​for multiple subframes obtained by dividing one frame. In Figures 14(A) and (B), the horizontal axis represents exposure time, and the vertical axis represents the pulse count value. One frame in Figure 14(A) is divided into multiple temporally adjacent subframes T1 to T5 as shown in Figure 14(B), and pulse counting is performed in each of the subframes T1 to T5. The pulse count values ​​of each subframe T1 to T5 are combined, and the combined data is output as an image signal for one frame. In addition, a comparison is performed between the pulse count value and a threshold Th in each of the subframes T1 to T5. Note that the number of subframes divided into one frame is not limited to the example above, and can be any number of two or more.

[0083] In Figure 14(A), the pulse count value begins to increase simultaneously with the start of the frame, and the pulse count value exceeds the threshold Th at time t201. The pulse count value for the period t201-t202, during which the threshold Th is exceeded, is estimated by the output calculation unit 254 (Figure 6). In Figure 14(A), an estimated pulse count value for a relatively long period t201-t202 is calculated, but because the change in brightness is small, the difference between the captured image and the actual scene is small.

[0084] In Figure 14(B), the pulse count value begins to increase simultaneously with the start of subframe T1, and the pulse count value exceeds the threshold Th at time t211. The pulse count value at times t211 to t212, when the threshold Th is exceeded, is estimated by the output calculation unit 254. Similarly, the pulse count value is counted in each of subframes T1 to T5, compared with the threshold Th, and the pulse count value is estimated. In subframes T1 to T5, the time interval t211 to t212 when the pulse count value exceeds the threshold Th is relatively short. Also, the sum of the time intervals t211 to t212 for each of subframes T1 to T5 is shorter than the time intervals t201 to t202 for one frame. Therefore, in this embodiment, the period for estimating the pulse count value is shortened, making it possible to reduce the difference between the image signal and the actual scene. However, in scenes with little change in brightness, the difference between the cases in Figures 14(A) and (B) is small.

[0085] Although not shown in Figures 14(A) and (B), an upper limit Th_H and a lower limit Th_L can also be set for the threshold Th in this embodiment. Since the threshold Th falls within the range of the upper limit Th_H and the lower limit Th_L, the effect of reducing the error in the estimated pulse count value can be obtained.

[0086] Figure 15 is a diagram illustrating the operation of the photoelectric converter in this embodiment, showing the pulse count value in a scene where the brightness changes gradually and then increases sharply. The graph of the pulse count value after exceeding the threshold Th shows the actual number of incident photons. After the pulse count value exceeds the threshold Th, the pulse count actually stops and is replaced with an estimated value.

[0087] In Figure 15(A), the pulse count value begins to increase simultaneously with the start of the frame, exceeds the threshold Th at time t221, and the pulse count stops after time t221. The pulse count value after time t221 is output as the estimated value B10. The actual brightness value increases sharply after time t221. Therefore, in the period t221-t222 after the threshold Th is exceeded, the estimated value B10 of the pulse count value differs significantly from the actual brightness value, resulting in a large discrepancy between the captured image and the actual scene.

[0088] In Figure 15(B), the pulse count value increases simultaneously with the start of subframe T1, and is reset upon the end of subframe T1. Subsequently, the pulse count value is counted up and reset repeatedly in each of the subframes T2 to T5. In subframe T4, the pulse count value exceeds the threshold Th, but the estimated value B11 of the pulse count value after exceeding the threshold Th approximates the actual brightness value. In subframe T5, the pulse count value also exceeds the threshold Th, but the estimated value B12 of the pulse count value after exceeding the threshold Th approximates the actual brightness value. The pulse count values ​​of each subframe T1 to T5 are combined and output as a single image signal. According to this embodiment, by dividing one frame into multiple subframes T1 to T5, the error in the estimated value after the pulse count value exceeds the threshold Th can be reduced. This makes it possible to reduce the difference between the image signal and the actual scene.

[0089] Figure 16 is a diagram illustrating the operation of the photoelectric converter in this embodiment, showing the pulse count value in a scene where the brightness rapidly increases and then gradually changes. The graph of the pulse count value after exceeding the threshold Th shows the actual number of incident photons. After the pulse count value exceeds the threshold Th, the pulse count actually stops and is replaced with an estimated value.

[0090] In Figure 16(A), the pulse count value begins to increase simultaneously with the start of the frame, and the pulse count value exceeds the threshold Th at time t231. After time t231, the estimated pulse count value B21 differs significantly from the actual brightness value, resulting in a large discrepancy between the image signal and the actual scene.

[0091] In Figure 16(B), the pulse count value increases simultaneously with the start of subframe T1 and exceeds the threshold Th at time t241. After exceeding the threshold Th, the estimated pulse count value B22 is calculated. The pulse count value is reset at the end of subframe T1. The pulse count value increases simultaneously with the start of subframe T2 and exceeds the threshold Th at time t242. After exceeding the threshold Th, the estimated pulse count value B23 is calculated. In subframes T3 to T5, the pulse count value does not exceed the threshold Th, so no estimate is calculated. The pulse count values ​​from subframes T1 to T5 are combined and output as a single image frame. In Figure 16(B) as well, the errors in the estimated values ​​B22 and B23 are smaller than the errors in the estimated value B21 in Figure 16(A), making it possible to reduce the difference between the image signal and the actual scene.

[0092] As described above, according to this embodiment, by dividing one frame into multiple subframes and comparing the pulse count value with a threshold value and calculating an estimated pulse count value in each subframe, it is possible to reduce errors in the image signal. Furthermore, in this embodiment, since one frame is divided into multiple subframes, the above effects can be achieved not only in video recording but also in still image recording.

[0093] Next, referring to Figures 17 and 18, the operation of changing the threshold Th in multiple subframes will be explained. In this embodiment as well, the power consumption reduction effect can be enhanced when there is high-speed movement of the subject or abrupt changes in brightness. Furthermore, by dividing one frame into multiple subframes, it is possible to reduce the difference between the captured image and the actual scene.

[0094] Figure 17 is a diagram illustrating the operation of the photoelectric converter in this embodiment, and illustrates the change in threshold Th across multiple subframes. In Figures 17(A), (B), and (C), the upper graph represents the pulse count value in one frame, and the lower graph represents the pulse count value, rate of change, and threshold Th in multiple subframes, respectively. Figures 17(A), (B), and (C) correspond to the first to third frames.

[0095] In Figure 17(A) (first frame), the brightness is low and the change in brightness is small. In subframe T1, the threshold Th is initially set to the upper limit Th_H. In subframe T2, the difference between the rate of change of the pulse count value in subframe T1 and the rate of change of the pulse count value in subframe T2 is less than or equal to the set value. Therefore, the threshold calculation unit 204 (Figure 6) lowers the threshold Th for the next subframe T3. Similarly, in subframes T3 to T5, the threshold Th gradually decreases, and in subframe T5, the threshold Th becomes the lower limit Th_L. Since the pulse count value does not exceed the threshold Th in subframes T1 to T5, the pulse count value estimation process is not performed.

[0096] In Figure 17(B) (second frame), the brightness increases rapidly. In subframes T1 and T2, the rate of change of the pulse count value is small, and the pulse count value is below the threshold Th. Therefore, the threshold Th maintains its lower limit Th_L. The difference between the rate of change A22 in subframe T2 and the rate of change A23 in subframe T3 is below the set value, so the threshold Th in subframe T4 maintains its lower limit Th_L. In subframe T4, the pulse count value increases rapidly, and the rate of change A24 also increases. The difference between the rate of change A23 in subframe T3 and the rate of change A24 in subframe T4 exceeds the set value, and the threshold Th in subframe T5 becomes the upper limit Th_H. Also, in subframe T4, an estimated value of the pulse count value after exceeding the threshold Th is calculated. In subframe T5, the brightness value continues to increase, and the difference between the rate of change A24 in subframe T4 and the rate of change A25 in subframe T5 exceeds the set value. At this point, the threshold Th is already at its upper limit Th_H, so the threshold Th does not change. In addition, in subframe T5, an estimated value of the pulse count after the threshold Th is exceeded is calculated.

[0097] In the third frame shown in Figure 17(C), the brightness is high. In subframe T1, the pulse count value increases at a constant rate of change A31. Therefore, the graph of the pulse count value (brightness value) and the graph of the rate of change A31 are shown superimposed. The difference between the rate of change A31 in subframe T1 of the third frame and the rate of change A25 in subframe T5 of the second frame exceeds the set value. However, since the threshold Th is at its upper limit Th_H, the threshold Th does not change. In subframes T2 to T5, since the rate of change of the pulse count value is constant, the threshold Th gradually decreases, and in subframe T5, the threshold Th becomes the lower limit Th_L. In addition, in subframes T1 to T5, an estimated value of the pulse count value after exceeding the threshold Th is calculated.

[0098] Figure 18 is a diagram illustrating the operation of the photoelectric converter in this embodiment, and shows an example of brightness change that is the opposite of the brightness change shown in Figure 17. Similar to Figure 17, in Figures 18(A), (B), and (C), the upper graph represents the pulse count value in one frame, and the lower graph represents the pulse count value, rate of change, and threshold Th in multiple subframes, respectively. Also, Figures 18(A), (B), and (C) correspond to the first to third frames.

[0099] In the first frame shown in Figure 18(A), the brightness is high. In subframes T1 to T5, the pulse count value increases at a constant rate of change. Therefore, the difference in the rate of change of the pulse count value becomes constant, and the threshold Th gradually decreases, reaching the lower limit Th_L in subframe T5. In addition, in subframes T1 to T5, an estimated value of the pulse count value after exceeding the threshold Th is calculated.

[0100] In the second frame shown in Figure 18(B), the brightness increases rapidly and then changes gradually. The rate of change A21 in subframe T1 matches the rate of change A15 in subframe T5 of the first frame, and the threshold Th remains at its lower limit Th_L. The difference between the rate of change A21 in subframe T1 and the rate of change A22 in subframe T2 exceeds the set value, and the threshold Th in subframe T3 becomes the upper limit Th_H. The difference between the rates of change A22 and A23 also exceeds the set value, and the threshold Th in subframe T4 remains at its upper limit Th_H. In subframe T4, the difference between the rates of change A23 and A24 becomes less than or equal to the set value, and the threshold Th in subframe T5 becomes lower.

[0101] In the third frame shown in Figure 18(C), the brightness is low and the change in brightness is small. In subframes T1 to T5, the difference in the rate of change of the pulse count value is approximately constant and below the set value. Therefore, the threshold Th decreases, and in subframes T2 to T5, the threshold Th becomes the lower limit Th_L. Since the pulse count value does not exceed the threshold Th in subframes T1 to T5, the pulse count value estimation process is not performed.

[0102] As described above, according to this embodiment, one frame is divided into subframes, and a comparison of the pulse count value and the threshold Th is performed for each subframe. Therefore, it is possible to reduce the error in the estimated pulse count value when the threshold is exceeded. In other words, when the threshold is exceeded, the difference between the actual scene and the captured image can be reduced.

[0103] Furthermore, in this embodiment as well, if the pulse count value exceeds the threshold, the pulse count stops, thereby reducing the current consumption of the photoelectric converter. In addition, based on the time count at the point when the pulse count value exceeds the threshold and the threshold itself, it becomes possible to more accurately estimate the pulse count value after the threshold is exceeded, and output an image signal with less error. As a result, it becomes possible to reduce current consumption while reducing errors in the image signal.

[0104] [Third Embodiment] Next, the photoelectric conversion device in this embodiment will be described. In this embodiment, some circuits are shared by multiple signal processing units 20. The following description will focus on the configuration and operation that differ from the first and second embodiments.

[0105] Figure 19 is a block diagram of the photoelectric converter in this embodiment. In Figure 19, for the sake of simplicity, only two signal processing units 20A and 20B and an output calculation unit 254 of the photoelectric converter 100 in Figure 3 are shown, and other components are omitted. The signal processing unit 20A includes a pulse generation unit 201A, a pulse counting unit 202A, a time counting unit 203A, a threshold calculation unit 204A, and an output unit 205A. The signal processing unit 20B includes a pulse generation unit 201B, a pulse counting unit 202B, and an output unit 205B, but does not include a time counting unit or a threshold calculation unit.

[0106] The time count value from the time count unit 203A and the threshold value Th from the threshold calculation unit 204A are input to the output unit 205A of the signal processing unit 20A and the output unit 205B of the signal processing unit 20B, respectively. In other words, the signal processing units 20A and 20B operate based on a common time count value and threshold value Th.

[0107] In the first embodiment shown in Figure 6, since a time counting unit 203 and a threshold calculation unit 204 are provided for each signal processing unit 20, the area of ​​each pixel circuit increases, and the integration density may decrease. According to this embodiment, it is possible to reduce the circuit size by sharing some circuits among multiple signal processing units 20.

[0108] Figure 20 shows an example of the configuration of a pixel block in this embodiment. The circuit region 2a is divided into (m × n) pixel blocks 21. For example, if 1600 pixels are arranged in the X direction and 900 pixels are arranged in the Y direction, (20 × 10) pixel blocks 21 may be provided. In this case, one pixel block 21 contains, for example, (100 × 100) signal processing units 20. A time counting unit 203 and a threshold calculation unit 204 are provided for each pixel block 21. That is, one pixel block 21 comprises one signal processing unit 20A including a time counting unit 203 and a threshold calculation unit 204, and multiple signal processing units 20B that do not include a time counting unit 203 and a threshold calculation unit 204. The multiple pixel blocks 21 are arranged, for example, in the region shown by the dotted lines in Figures 8 and 9, and threshold calculation and calculation of estimated pulse count values ​​are performed for each pixel block 21.

[0109] The number of pixel blocks 21 and the number of pixels in the X and Y directions are not limited to the examples described above. Furthermore, the shape and aspect ratio of the signal processing unit (pixel) 20 are not limited; for example, it may be rectangular instead of square. Additionally, the pixel block 21 may consist of only one signal processing unit 20.

[0110] As described above, according to this embodiment, it is possible to reduce the circuit size by sharing some circuits with multiple signal processing units.

[0111] [Fourth Embodiment] Next, the photoelectric conversion device in this embodiment will be described. The following description will focus on the configuration and operation that differ from the first and second embodiments.

[0112] Figure 21 is a block diagram of the photoelectric converter in this embodiment, showing the configuration of the signal processing unit 20. The signal processing unit 20 comprises a pulse generation unit 201, a pulse counting unit 202, a time counting unit 203, a threshold calculation unit 204, an output unit 205, and an output calculation unit 206. The pulse generation unit 201, pulse counting unit 202, time counting unit 203, and output unit 205 are configured substantially the same as in the above-described embodiment. In this embodiment, unlike the above-described embodiment, the pixel value from the output calculation unit 206 is input to the threshold calculation unit 204, and the threshold calculation unit 204 holds past pixel values ​​instead of the rate of change of past pulse count values. The most recent pixel value is also held in the threshold calculation unit 204. Here, the pixel value is the pulse count value when the pulse count value does not exceed the threshold, or an estimated value of the pulse count value after the pulse count value exceeds the threshold.

[0113] The operation of the photoelectric converter in this embodiment differs from the above-described embodiment in the following points in the flowchart of Figure 7. In steps S107 and S111, the pixel value is held instead of the rate of change. In step S112, the output calculation unit 254 determines whether the difference in pixel values ​​is within a set value. If the difference in pixel values ​​between frames is within a predetermined value (YES in step S112), the threshold becomes lower (step S113). On the other hand, if the difference in pixel values ​​between frames exceeds a predetermined value (NO in step S112), the threshold becomes higher (step S114).

[0114] According to this embodiment, the threshold calculation unit 204 does not need to calculate the rate of change of the pulse count value, and can determine the change of the threshold based on the difference or ratio between the current pixel value and the previous pixel value. Therefore, it is possible to reduce the circuit for calculating the rate of change and reduce the overall circuit size.

[0115] [Fifth Embodiment] Next, the photoelectric conversion device in this embodiment will be described. The following description will focus on the configuration and operation that differ from the fourth embodiment.

[0116] Figure 22 is a block diagram of the photoelectric converter in this embodiment. In this embodiment, the pulse generation unit 201, pulse counting unit 202, and output unit 205 are provided in the signal processing unit 20, while the time counting unit 203, threshold calculation unit 204, and output calculation unit 206 are provided in the photoelectric converter 100. Since the time counting unit 203, threshold calculation unit 204, and output calculation unit 206 are shared by multiple signal processing units 20, the circuit size can be reduced.

[0117] In the above-described embodiment, a photoelectric converter 100 in which a sensor substrate 1 and a circuit board 2 are stacked was described, but the photoelectric converter 100 may be configured using a single substrate. In this case, the APD 11, which is the light receiving unit, can be provided in the signal processing unit 20. In this embodiment, since the time counting unit 203, threshold calculation unit 204, and output calculation unit 206 are provided outside the signal processing unit 20, it is possible to increase the light receiving area of ​​the APD 11.

[0118] [Sixth Embodiment] Next, the photoelectric conversion device in this embodiment will be described. The flowchart shown in Figure 7 is merely an example and can be modified. For example, although the difference in the rate of change of the pulse count value between frames is described as an absolute value, the difference in the rate of change may be expressed as a positive or negative value. That is, if the rate of change of the pulse count value in the current frame is increasing from the rate of change of the pulse count value in past frames, the difference in the rate of change may be expressed as a positive value, and if the rate of change of the pulse count value in the current frame is decreasing from the rate of change of the pulse count value in past frames, the difference in the rate of change may be expressed as a negative value. In this case, in the comparison between the rate of change and the set value (step S112 in Figure 7), the set value may also be expressed as a positive or negative value.

[0119] Furthermore, in Figure 7, if the pulse count value does not exceed the threshold Th (NO in step S103), the threshold calculation unit 204 may return to the process in step S101 after calculating the rate of change (step S107) without performing the threshold change process (steps S112 to S114). In this way, various changes can be made in the flowchart of Figure 7, and in this embodiment as well, it is possible to reduce current consumption while reducing errors in the image signal.

[0120] [Seventh Embodiment] Figure 23 shows an example of the arrangement of the circuit board 2 in this embodiment, illustrating a modified version of the arrangement of the circuit board 2 in Figure 3. In Figure 3, the readout circuit 253 reads signals from the signal lines 259 connected to the signal processing units 20 of each column, but in Figure 23, the readout circuit 221 may read signals from the signal lines 227 connected to the pixels 10 of each row. Thus, the circuit board 2 can be configured in various arrangements. This embodiment can also achieve the same effects as the embodiment described above.

[0121] [Eighth Embodiment] Figure 24 is a block diagram of the imaging system in this embodiment. The photoelectric conversion device in the above-described embodiment is applicable to various imaging systems. Examples of imaging systems include digital still cameras, digital camcorders, camera heads, photocopiers, fax machines, mobile phones, in-vehicle cameras, observation satellites, and surveillance cameras. Figure 24 shows a block diagram of a digital still camera as an example of an imaging system.

[0122] The imaging system 7 includes a barrier 706, a lens 702, an aperture 704, an imaging device 70, a signal processing unit 708, a timing generation unit 720, an overall control / calculation unit 718, a memory unit 710, a recording medium control I / F unit 716, a recording medium 714, and an external I / F unit 712. The barrier 706 protects the lens, and the lens 702 forms an optical image of the subject on the imaging device 70. The aperture 704 varies the amount of light passing through the lens 702. The imaging device 70 is configured as a photoelectric converter in the above embodiment and converts the optical image formed by the lens 702 into image data. The signal processing unit 708 performs various corrections and data compression on the imaging data output from the imaging device 70.

[0123] The timing generation unit 720 outputs various timing signals to the imaging device 70 and the signal processing unit 708. The overall control and calculation unit 718 controls the entire digital still camera, and the memory unit 710 temporarily stores image data. The recording medium control I / F unit 716 is an interface for recording or reading image data to or from the recording medium 714, which is a removable recording medium such as a semiconductor memory for recording or reading imaging data. The external I / F unit 712 is an interface for communicating with an external computer or the like. Timing signals and the like may be input from outside the imaging system, and the imaging system only needs to have at least the imaging device 70 and the signal processing unit 708 that processes the image signals output from the imaging device 70.

[0124] In this embodiment, the imaging device 70 and the signal processing unit 708 are provided on separate semiconductor substrates, but the imaging device 70 and the signal processing unit 708 may be formed on the same semiconductor substrate.

[0125] Furthermore, each pixel includes a first photoelectric conversion unit and a second photoelectric conversion unit. The signal processing unit 708 processes the pixel signal based on the charge generated in the first photoelectric conversion unit and the pixel signal based on the charge generated in the second photoelectric conversion unit, and can acquire distance information from the imaging device 70 to the subject.

[0126] [Ninth Embodiment] Figure 25 is a diagram of the light detection system in this embodiment, and is a block diagram of a distance image sensor using the photoelectric conversion device described in the above embodiment.

[0127] As shown in Figure 25, the distance image sensor 401 comprises an optical system 402, a photoelectric converter 403, an image processing circuit 404, a monitor 405, and a memory 406. The distance image sensor 401 receives light (modulated light, pulsed light) that is emitted from the light source device 411 toward the subject and reflected from the surface of the subject. Based on the time from emission to reception, the distance image sensor 401 can acquire a distance image corresponding to the distance to the subject.

[0128] The optical system 402 includes one or more lenses and guides the image light (incident light) from the subject to the photoelectric converter 403, where it forms an image on the light-receiving surface (sensor part) of the photoelectric converter 403.

[0129] The photoelectric converter 403 can be any of the photoelectric converters described in the above-described embodiments. The photoelectric converter 403 supplies a distance signal indicating the distance obtained from the received light signal to the image processing circuit 404.

[0130] The image processing circuit 404 performs image processing to construct a distance image based on the distance signal supplied from the photoelectric converter 403. The distance image (image data) obtained through image processing can be displayed on the monitor 405 and stored (recorded) in the memory 406.

[0131] The distance image sensor 401 configured in this way can acquire more accurate distance images by applying the photoelectric conversion device described above, as the characteristics of the pixels are improved.

[0132] [Tenth Embodiment] The technology described herein can be applied to a variety of products. For example, the technology described herein may be applied to an endoscopic surgical system.

[0133] Figure 26 is a schematic diagram of the endoscopic surgical system in this embodiment. Figure 26 shows a surgeon (physician) 1131 performing surgery on a patient 1132 on a patient bed 1133 using the endoscopic surgical system 1103. As shown in the figure, the endoscopic surgical system 1103 includes an endoscope 1100, surgical instruments 1110, and a cart 1134 on which various devices for endoscopic surgery are mounted.

[0134] The endoscope 1100 comprises a barrel 1101, the tip of which is inserted into the body cavity of the patient 1132 for a predetermined length, a camera head 1102 connected to the base end of the barrel 1101, and an arm 1121. Figure 27 shows the endoscope 1100 configured as a so-called rigid endoscope having a rigid barrel 1101, but the endoscope 1100 may also be configured as a so-called flexible endoscope having a flexible barrel.

[0135] An opening into which an objective lens is fitted is provided at the tip of the endoscope tube 1101. A light source device 1203 is connected to the endoscope 1100, and the light generated by the light source device 1203 is guided to the tip of the endoscope tube by a light guide extending inside the endoscope tube 1101, and is irradiated through the objective lens towards the object to be observed inside the body cavity of the patient 1132. The endoscope 1100 may be a straight-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0136] The camera head 1102 contains an optical system and a photoelectric converter. Reflected light from the object being observed (observation light) is focused by the optical system into the photoelectric converter. The photoelectric converter converts the observation light into electrical signals, generating an electrical signal corresponding to the observation light, i.e., an image signal corresponding to the observed image. The photoelectric converter can be any of the photoelectric converters described in the embodiments described above. The image signal is transmitted as RAW data to the camera control unit (CCU) 1135.

[0137] The CCU1135 consists of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and comprehensively controls the operation of the endoscope 1100 and the display device 1136. Furthermore, the CCU1135 receives image signals from the camera head 1102 and performs various image processing operations on these image signals, such as development processing (demosaic processing), to display images based on the image signals.

[0138] The display device 1136 displays an image based on an image signal that has been processed by the CCU 1135, under control from the CCU 1135.

[0139] The light source device 1203 is equipped with a light source such as an LED (Light Emitting Diode) and supplies illumination light to the endoscope 1100 when photographing the surgical area, etc.

[0140] The input device 1137 is an input interface for the endoscopic surgical system 1103. The user can input various types of information and instructions to the endoscopic surgical system 1103 via the input device 1137.

[0141] The treatment instrument control device 1138 controls the driving of the energy treatment instrument 1112 for purposes such as tissue cauterization, incision, or blood vessel sealing.

[0142] The light source device 1203 is capable of supplying illumination light to the endoscope 1100 when photographing the surgical area, and may be, for example, an LED, a laser light source, or a combination thereof to form a white light source. When a white light source is formed by a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision. Therefore, the white balance of the captured image can be adjusted in the light source device 1203. In this case, the laser light from each of the RGB laser light sources may be irradiated onto the observation target in a time-division manner, and the drive of the image sensor of the camera head 1102 may be controlled in synchronization with the irradiation timing. This makes it possible to capture images corresponding to each of the RGB colors in a time-division manner. With this method, a color image can be obtained without providing a color filter on the image sensor.

[0143] Furthermore, the drive of the light source device 1203 may be controlled so that the intensity of the light output from the light source device 1203 is changed at predetermined time intervals. By controlling the drive of the image sensor of the camera head 1102 in synchronization with the timing of the change in light intensity to acquire images in time division and combining these images, it is possible to generate a high dynamic range image without so-called black crushing and white clipping.

[0144] Furthermore, the light source device 1203 may be configured to supply light in a predetermined wavelength band corresponding to special light observation. In special light observation, for example, the wavelength dependence of light absorption in body tissue can be utilized. Specifically, by irradiating with narrowband light compared to the irradiation light used during normal observation (i.e., white light), predetermined tissues such as blood vessels on the surface of mucosa can be imaged with high contrast. Alternatively, in special light observation, fluorescence observation may be performed to obtain an image from fluorescence generated by irradiation with excitation light. In fluorescence observation, excitation light can be irradiated onto body tissue and fluorescence from the body tissue can be observed, or a reagent such as indocyanine green (ICG) can be injected into body tissue and excitation light corresponding to the fluorescence wavelength of the reagent can be irradiated onto the body tissue to obtain a fluorescence image. The light source device 1203 may be configured to supply narrowband light and / or excitation light corresponding to such special light observation.

[0145] [Embodiment No. 11] The light detection system and mobile body of this embodiment will be described with reference to Figures 27A, 27B, and 28. In this embodiment, an example of an in-vehicle camera is shown as the light detection system.

[0146] Figure 27A is a schematic diagram of the photodetection system in this embodiment, showing an example of a vehicle system and a photodetection system mounted on the vehicle system. The photodetection system 1301 includes a photoelectric converter 1302, an image preprocessing unit 1315, an integrated circuit 1303, and an optical system 1314. The optical system 1314 forms an optical image of the subject on the photoelectric converter 1302. The photoelectric converter 1302 converts the optical image of the subject formed by the optical system 1314 into an electrical signal. The photoelectric converter 1302 is one of the photoelectric converters in each of the embodiments described above. The image preprocessing unit 1315 performs predetermined signal processing on the signal output from the photoelectric converter 1302. The functions of the image preprocessing unit 1315 may be incorporated into the photoelectric converter 1302. The photodetection system 1301 is provided with at least two sets of optical systems 1314, photoelectric converters 1302, and image preprocessing units 1315, and the output from each set of image preprocessing units 1315 is input to the integrated circuit 1303.

[0147] The integrated circuit 1303 is an integrated circuit for imaging system applications and includes an image processing unit 1304 with a storage medium 1305, an optical distance measuring unit 1306, a parallax calculation unit 1307, an object recognition unit 1308, and an anomaly detection unit 1309. The image processing unit 1304 performs image processing such as development and defect correction on the output signal of the image preprocessing unit 1315. The storage medium 1305 stores the primary storage of the captured image and the defect positions of the captured pixels. The optical distance measuring unit 1306 focuses on the subject and measures the distance. The parallax calculation unit 1307 calculates distance measurement information from multiple image data acquired by multiple photoelectric converters 1302. The object recognition unit 1308 recognizes subjects such as cars, roads, signs, and people. When the anomaly detection unit 1309 detects an anomaly in the photoelectric converter 1302, it alerts the main control unit 1313 to the anomaly.

[0148] The integrated circuit 1303 may be implemented by specially designed hardware, by a software module, or by a combination of these. It may also be implemented by an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a combination of these.

[0149] The main control unit 1313 coordinates and controls the operation of the light detection system 1301, vehicle sensor 1310, control unit 1320, etc. Alternatively, the main control unit 1313 may be omitted, and the light detection system 1301, vehicle sensor 1310, and control unit 1320 may each have their own communication interfaces, and each may send and receive control signals via a communication network, for example, using the CAN standard.

[0150] The integrated circuit 1303 has the function of receiving control signals from the main control unit 1313 or transmitting control signals and set values ​​to the photoelectric converter 1302 via its own control unit.

[0151] The light detection system 1301 is connected to the vehicle sensor 1310 and can detect the vehicle's driving conditions, such as vehicle speed, yaw rate, and steering angle, as well as the external environment and the state of other vehicles and obstacles. The vehicle sensor 1310 also functions as a distance information acquisition unit, acquiring distance information to objects. Furthermore, the light detection system 1301 is connected to the driver assistance control unit 1311, which performs various driving assistance functions such as automatic steering, automatic cruising, and collision avoidance. In particular, regarding the collision judgment function, the system determines whether a collision with another vehicle or obstacle has occurred and estimates the collision based on the detection results of the light detection system 1301 and the vehicle sensor 1310. This enables avoidance control when a collision is estimated and activation of safety devices in the event of a collision.

[0152] Furthermore, the light detection system 1301 is also connected to a warning device 1312 that issues a warning to the driver based on the judgment result of the collision judgment unit. For example, if the collision judgment unit determines that there is a high probability of collision, the main control unit 1313 performs vehicle control such as applying the brakes, releasing the accelerator, and suppressing engine output to avoid a collision or mitigate damage. The warning device 1312 issues a warning to the user by means of emitting a sound or other warning, displaying warning information on display screens such as the car navigation system and meter panel, and applying vibration to the seat belt and steering wheel.

[0153] The light detection system 1301 in this embodiment can capture images of the area around the vehicle, for example, in front of or behind it. Figure 27B is a schematic diagram of a moving object in this embodiment, showing a configuration in which the area in front of the vehicle is captured by the light detection system 1301.

[0154] The two photoelectric converters 1302 are positioned in front of the vehicle 1300. Specifically, it is preferable that the center line of the vehicle 1300 with respect to its direction of movement or external shape (e.g., vehicle width) is considered as the axis of symmetry, and the two photoelectric converters 1302 are positioned symmetrically with respect to the axis of symmetry. This makes it possible to effectively acquire distance information between the vehicle 1300 and the object being photographed and to determine the possibility of collision. Furthermore, it is preferable that the photoelectric converters 1302 are positioned so as not to obstruct the driver's field of view when the driver is viewing the situation outside the vehicle 1300 from the driver's seat. The warning device 1312 is preferably positioned so as to be easily visible to the driver.

[0155] Next, the fault detection operation of the photoelectric converter 1302 in the photodetection system 1301 will be explained using Figure 28. Figure 28 is a flowchart showing the operation of the photodetection system in this embodiment. The fault detection operation of the photoelectric converter 1302 may be performed according to steps S1410 to S1480.

[0156] In step S1410, the startup settings for the photoelectric converter 1302 are performed. Specifically, setting information for the operation of the photoelectric converter 1302 is transmitted from outside the photodetection system 1301 (e.g., the main control unit 1313) or from inside the photodetection system 1301, and the photoelectric converter 1302 starts the imaging operation and fault detection operation.

[0157] Next, in step S1420, the photoelectric converter 1302 acquires a pixel signal from the active pixels. Also, in step S1430, the photoelectric converter 1302 acquires an output value from a fault detection pixel provided for fault detection. This fault detection pixel is equipped with a photoelectric conversion element, just like the active pixels. A predetermined voltage is written to this photoelectric conversion element. The fault detection pixel outputs a signal corresponding to the voltage written to this photoelectric conversion element. Note that steps S1420 and S1430 may be executed in the reverse order.

[0158] Next, in step S1440, the light detection system 1301 determines whether the expected output value of the fault detection pixel matches the actual output value from the fault detection pixel. If the result of the matching determination in step S1440 shows that the expected output value and the actual output value match, the light detection system 1301 proceeds to step S1450, determines that the imaging operation is being performed normally, and proceeds to step S1460. In step S1460, the light detection system 1301 transmits the pixel signals of the scan row to the storage medium 1305 for temporary storage. After that, the light detection system 1301 returns to step S1420 and continues the fault detection operation. On the other hand, if the result of the matching determination in step S1440 shows that the expected output value and the actual output value do not match, the light detection system 1301 proceeds to step S1470. In step S1470, the light detection system 1301 determines that there is an abnormality in the imaging operation and issues an alarm to the main control unit 1313 or the alarm device 1312. The alarm device 1312 displays that an abnormality has been detected on its display unit. Subsequently, in step S1480, the light detection system 1301 stops the photoelectric converter 1302 and terminates the operation of the light detection system 1301.

[0159] In this embodiment, an example is shown where the flowchart is looped every row, but the flowchart may be looped every multiple rows, or the fault detection operation may be performed every frame. The alarm in step S1470 may be notified to an external party via a wireless network.

[0160] Furthermore, although this embodiment describes control to avoid collisions with other vehicles, it can also be applied to control that automatically follows other vehicles or control that automatically drives without deviating from the lane. In addition, the light detection system 1301 can be applied not only to vehicles such as the vehicle itself, but also to moving objects (mobile devices) such as ships, aircraft, or industrial robots. Moreover, it can be applied not only to moving objects, but also to a wide range of devices that utilize object recognition, such as intelligent transportation systems (ITS). The photoelectric conversion device of the present invention may further be configured to acquire various types of information, such as distance information.

[0161] [Twelfth Embodiment] Figure 29(a) shows a specific example of an electronic device in this embodiment, and shows eyeglasses 1600 (smart glasses). The eyeglasses 1600 are equipped with the photoelectric converter 1602 described in each of the embodiments described above. A display device including a light-emitting device such as an OLED or LED may be provided on the back side of the lens 1601. There may be one or more photoelectric converters 1602. In addition, multiple types of photoelectric converters may be combined. The arrangement position of the photoelectric converter 1602 is not limited to that shown in Figure 29(a).

[0162] The eyeglasses 1600 further include a control device 1603. The control device 1603 functions as a power source that supplies power to the photoelectric converter 1602 and the display device. The control device 1603 also controls the operation of the photoelectric converter 1602 and the display device. The lens 1601 has an optical system formed therein for focusing light onto the photoelectric converter 1602.

[0163] Figure 29(b) shows eyeglasses 1610 (smart glasses) relating to one application example. The eyeglasses 1610 have a control device 1612, which is equipped with a photoelectric converter corresponding to a photoelectric converter 1602 and a display device. The lens 1611 has an optical system formed therein for projecting the photoelectric converter in the control device 1612 and the light emitted from the display device, and an image is projected onto the lens 1611. The control device 1612 functions as a power supply that provides power to the photoelectric converter and the display device, and also controls the operation of the photoelectric converter and the display device. The control device 1612 may have a gaze detection unit that detects the wearer's gaze. Gaze detection may use infrared light. The infrared light emitter emits infrared light towards the eyeball of the user who is gazing at the displayed image. An imaging unit having a light-receiving element detects the reflected light from the eyeball of the emitted infrared light, thereby obtaining an image of the eyeball. By having a reduction mechanism that reduces the amount of light transmitted from the infrared light-emitting part to the display part in a planar view, the degradation of image quality is reduced.

[0164] The user's gaze towards a displayed image is detected from an image of the eyeball obtained by imaging with infrared light. Any known method can be applied to gaze detection using an image of the eyeball. As an example, a gaze detection method based on the Purkinje image obtained by the reflection of the irradiated light from the cornea can be used.

[0165] More specifically, gaze detection processing is performed based on the pupil-corneal reflection method. Using the pupil-corneal reflection method, a gaze vector representing the orientation (rotation angle) of the eyeball is calculated based on the pupil image and Purkinje image contained in the captured image of the eyeball, thereby detecting the user's gaze.

[0166] The display device of this embodiment includes a photoelectric converter having a light-receiving element, and may control the display image of the display device based on the user's gaze information from the photoelectric converter.

[0167] Specifically, the display device determines, based on gaze information, a first field of view area that the user is fixated on and a second field of view area other than the first field of view area. The first and second field of view areas may be determined by the control device of the display device or by an external control device. Within the display area of ​​the display device, the display resolution of the first field of view area may be controlled to be higher than that of the second field of view area. In other words, the resolution of the second field of view area may be lower than that of the first field of view area.

[0168] Furthermore, the display area may include a first display area and a second display area different from the first display area. Based on gaze information, a higher priority area may be determined from the first and second display areas. The first and second view areas may be determined by the control device of the display device or by an external control device. The resolution of the higher priority area may be controlled to be higher than the resolution of the areas other than the higher priority area. In other words, the resolution of areas with relatively lower priority may be lower.

[0169] Artificial Intelligence (AI) may be used in determining the first field of view area and the area with the highest priority. The AI ​​may be a model configured to estimate the angle of gaze and the distance to the target object at the end of the line of sight from the image of the eyeball, using the image of the eyeball and the direction the eyeball was actually looking in the image as training data. The AI ​​program may be installed in the display device, the photoelectric converter, or an external device. If the external device has the AI ​​program, it may be transmitted to the display device from a server or the like via communication.

[0170] When display control is performed based on visual detection, this embodiment can be preferably applied to smart glasses further comprising a photoelectric converter for capturing images of the surroundings. The smart glasses can display the captured external information in real time.

[0171] [Other embodiments] The present invention is not limited to the embodiments described above and can be modified in various ways. For example, an example in which a part of the configuration of one embodiment is added to another embodiment, or in which a part of the configuration of another embodiment is replaced, is also an embodiment of the present invention.

[0172] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0173] It should be noted that the embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various ways without departing from its technical concept or its main features. [Explanation of Symbols]

[0174] 1 Sensor board 1a Pixel region 2 Circuit boards 10 pixels 11 APD 20 Signal Processing Unit 201 Pulse generation unit 202 Pulse counting section 203 Time Count Section 204 Threshold Calculation Unit 254 Output Calculation Unit

Claims

1. Avalanche photodiode and A pulse generation unit that converts the output from the avalanche photodiode into a pulse signal, A pulse counting unit that counts the pulse signal and outputs a pulse count value, A time counting unit outputs a time count value representing the time since the pulse generation unit started operating, If the pulse count value does not exceed a threshold, the output unit outputs the pulse count value; if the pulse count value exceeds the threshold, the output unit terminates the count in the pulse counting unit and outputs the time count value at the point when the pulse count value exceeded the threshold. A photoelectric conversion device characterized by having a threshold setting unit that changes the threshold based on a comparison between the magnitude of the change in brightness of a first frame and a second frame that occurs after the first frame and a set value.

2. The photoelectric conversion device according to claim 1, characterized in that the output unit outputs the threshold value when the pulse count value exceeds the threshold value.

3. The photoelectric conversion device according to claim 1 or 2, characterized in that the output unit outputs an eigenvalue indicating whether or not the pulse count value exceeds the threshold.

4. The photoelectric converter according to any one of claims 1 to 3, further comprising an output calculation unit that calculates the pulse count value after the threshold is exceeded based on the ratio of the time count value at the time the pulse count value exceeds the threshold to the threshold, if the pulse count value exceeds the threshold.

5. The threshold setting unit is, If the pulse count value does not exceed the threshold, the rate of change of the pulse count value is calculated based on the pulse count value and the time count value corresponding to the pulse count value. If the pulse count value exceeds the threshold, the rate of change of the pulse count value is calculated based on the time count value at the time the pulse count value exceeds the threshold and the threshold, The difference between the rate of change in the first frame and the rate of change in the second frame, which is later than the first frame, is calculated. The photoelectric conversion device according to any one of claims 1 to 4, characterized in that the threshold is lowered when the difference is less than or equal to the set value, and the threshold is raised when the difference exceeds the set value.

6. The threshold setting unit The difference between the pulse count value output in the first frame and the pulse count value output in the second frame, which is later than the first frame, is compared. The photoelectric conversion device according to any one of claims 1 to 4, characterized in that the threshold is lowered when the difference is less than or equal to the set value, and the threshold is raised when the difference exceeds the set value.

7. The photoelectric conversion apparatus according to claim 5 or 6, characterized in that the threshold setting unit changes the threshold within the range of upper and lower limits other than the most significant bit and least significant bit of the pulse count value.

8. The first frame and the second frame are first and second subframes that are divided from one frame and are temporally adjacent. The photoelectric converter according to any one of claims 5 to 7, characterized in that it combines the pulse count values ​​in each of the first subframe and the second subframe and outputs them as the pulse count value for one frame.

9. The photoelectric conversion device according to claim 7, characterized in that the threshold setting unit raises the threshold to the upper limit when the difference exceeds the set value.

10. The photoelectric conversion apparatus according to any one of claims 5 to 9, characterized in that the threshold setting unit determines the set value according to the threshold.

11. The photoelectric conversion apparatus according to any one of claims 5 to 9, characterized in that the threshold setting unit lowers the threshold by a predetermined amount when the difference is less than or equal to the set value.

12. The photoelectric conversion apparatus according to claim 11, characterized in that the threshold setting unit determines the predetermined amount according to the difference.

13. The photoelectric conversion device according to claim 11 or 12, characterized in that the threshold setting unit reduces the predetermined amount as the threshold decreases.

14. The photoelectric conversion device according to any one of claims 5 to 13, characterized in that the threshold setting unit does not change the threshold during the first frame period after the start of shooting.

15. It comprises multiple pixel circuits provided corresponding to multiple avalanche photodiodes arranged in an array, The photoelectric conversion device according to any one of claims 1 to 14, characterized in that the time counting unit and the threshold setting unit are shared by the plurality of pixel circuits.

16. The system comprises multiple pixel blocks, each containing multiple pixel circuits, The photoelectric conversion device according to claim 15, characterized in that the time counting unit and the threshold setting unit are provided for each pixel block.

17. The photoelectric conversion device according to claim 15, characterized in that a first substrate on which a plurality of avalanche photodiodes are provided and a second substrate on which a plurality of pixel circuits are provided are stacked.

18. A photoelectric conversion device according to any one of claims 1 to 17, A photodetection system characterized by comprising a signal processing device that processes the signal output from the aforementioned photoelectric converter.

19. The light detection system according to claim 18, characterized in that the signal processing device generates a distance image representing distance information to an object based on the signal.

20. The photoelectric conversion apparatus according to any one of claims 1 to 18, characterized in that the threshold setting unit performs calculation processing.

21. It is a mobile object, A photoelectric conversion device according to any one of claims 1 to 17, A distance information acquisition unit acquires distance information to an object from the signal output from the aforementioned photoelectric converter, A control unit that controls the moving body based on the distance information. A mobile body characterized by having the following features.

Images