Imaging apparatus, imaging method, and computer program

Abstract

To provide an imaging apparatus that has an avalanche photodiode, and that can prevent the occurrence of a count operation less affecting on an image.SOLUTION: An imaging apparatus has: a photoelectric conversion element that performs photoelectric conversion using an avalanche photodiode; determination means that determines the output level of the photoelectric conversion element on the basis of a first threshold; and voltage control means that controls reverse bias voltage to be applied to the avalanche photodiode on the basis of a result of the determination made by the determination means.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to an imaging apparatus, an imaging method, a computer program, and the like. [Background technology]

[0002] In recent years, imaging devices have been developed that digitally count the number of photons arriving at a single-photon avalanche photodiode (SPAD) and output the counted value from a pixel as a photoelectrically converted digital signal.

[0003] Patent Document 1 describes a method for controlling the frequency of avalanche multiplication. That is, the document describes a configuration for controlling a standby state in which avalanche multiplication of a photodiode is possible and a recharge state in which the photodiode is returned to a state in which avalanche multiplication is possible again, and detecting the occurrence of avalanche multiplication during the standby state.

[0004] In this device, the number of periods in which avalanche multiplication occurs among multiple periods in the waiting state for avalanche multiplication is counted, thereby obtaining a count value (hereinafter referred to as sensor output) corresponding to the number of photons incident on the pixel. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2020-123847 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the device shown in Patent Document 1, the relationship between the number of incident photons and the number of counts for the number of periods in which avalanche multiplication occurs is not linear. Therefore, when the sensor output is converted to a linear format, the dynamic range depends on the output after linear conversion (hereinafter, the linear output). In addition, since the linear output saturates earlier than the sensor output, counting beyond the sensor output at which the linear output saturates does not affect the image, and therefore excessive avalanche multiplication occurs.

[0007] SUMMARY OF THE PRESENT EMBODIMENTS Accordingly, an object of the present invention is to provide an imaging device having an avalanche photodiode that is capable of suppressing the occurrence of counting operations that have a small effect on an image. [Means for solving the problem]

[0008] An imaging device according to one aspect of the present invention comprises: a photoelectric conversion element that performs photoelectric conversion using an avalanche photodiode; A determination means for determining an output level of the photoelectric conversion element based on a first threshold value; a voltage control means for controlling a reverse bias voltage applied to the avalanche photodiode based on a result of the determination by the determination means; The present invention is characterized by having the following. Effect of the Invention

[0009] According to the present invention, it is possible to provide an imaging device having an avalanche photodiode, which is capable of suppressing the occurrence of counting operations that have a small effect on an image. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a photoelectric conversion element according to a first embodiment of the present invention. [Diagram 2] 2 is a diagram showing a configuration example of a sensor substrate 11. FIG. [Diagram 3] 2 is a diagram showing an example of the configuration of a circuit board 21. FIG. [Figure 4]2 and 3, and shows an equivalent circuit of a pixel 101 and a signal processing circuit 103 corresponding to the pixel 101. [Diagram 5] 2 is a diagram illustrating a relationship between the operation of an APD 201 and an output signal. FIG. [Figure 6] 1 is a functional block diagram showing an example of the configuration of an imaging device 300 according to a first embodiment. [Figure 7] 5A to 5C are diagrams for explaining a SPAD sensor output and a linear output with respect to the amount of incident light according to the first embodiment. [Figure 8] 5A to 5C are diagrams for explaining threshold setting of the SPAD sensor output according to the first embodiment. [Figure 9] FIG. 4 is an explanatory diagram relating to an example of timing for changing a reverse bias voltage according to the first embodiment. [Figure 10] 6 is a flowchart showing an example of an imaging method performed by a control unit 304 according to the first embodiment. [Figure 11] FIG. 4 is a diagram illustrating a threshold setting for a linear output according to the first embodiment. [Figure 12] 13 is a flowchart showing an example of an imaging method performed by a control unit 304 according to the second embodiment. [Figure 13] 13 is a continuation of the flowchart of FIG. 12. [Figure 14] 13 is a diagram illustrating an example of a SPAD sensor output relative to the amount of incident light according to the third embodiment. FIG. [Figure 15] 13 is a flowchart showing an example of an operation of reducing the number of photon counts performed by a control unit 304 according to the third embodiment. [Figure 16] 15 is a continuation of the flowchart in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0012] <Embodiment 1> 1 is a diagram showing a configuration example of a photoelectric conversion element according to a first embodiment of the present invention. In the following, the photoelectric conversion element 100 will be described taking as an example an imaging device having a so-called stacked structure, in which two substrates, a sensor substrate 11 and a circuit substrate 21, are stacked and electrically connected.

[0013] However, a so-called non-laminated structure in which the components included in the sensor substrate and the components included in the circuit substrate are disposed on a common semiconductor layer may be used. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0014] 2 is a diagram showing a configuration example of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across a plurality of rows and columns. The pixels 101 include a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, APD). That is, the photoelectric conversion element 100 performs photoelectric conversion by the avalanche photodiode.

[0015] Here, the photoelectric conversion unit 102 functions as a sensor unit that emits a pulse at a frequency corresponding to the frequency of receiving photons. The number of rows and columns of the pixel array that constitutes the pixel region 12 is not particularly limited.

[0016] Fig. 3 is a diagram showing an example of the configuration of the circuit board 21. The circuit board 21 has a signal processing circuit 103 that processes charges photoelectrically converted by each photoelectric conversion unit 102 in Fig. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114.

[0017] The vertical scanning circuit 110 receives a control pulse supplied from a control pulse generating unit 115, and sequentially supplies the control pulse to a plurality of pixels arranged in the row direction. The vertical scanning circuit 110 uses logic circuits such as a shift register and an address decoder.

[0018] The signal output from the photoelectric conversion unit 102 of each pixel is processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter, a memory, etc., and a digital value is stored in the memory. The horizontal scanning circuit 111 inputs a control pulse that sequentially selects each column to the signal processing circuit 103 in order to read the signal from the memory of each pixel in which the digital signal is stored.

[0019] A signal is output to the vertical signal line 113 from the signal processing circuit 103 of the pixel of the row selected by the vertical scanning circuit 110. The signal output to the vertical signal line 113 is output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has a plurality of buffers built therein and connected to the vertical signal line 113.

[0020] 2 and 3, a plurality of signal processing circuits 103 are arranged in a region overlapping the pixel region 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged so as to overlap between an end of the sensor substrate 11 and an end of the pixel region 12 in a plan view.

[0021] In other words, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region overlapping the non-pixel region in a plan view.

[0022] The arrangement of the vertical signal lines 113, the readout circuits 112, and the output circuits 114 are not limited to the example shown in Fig. 3. For example, the vertical signal lines 113 may be arranged extending in the row direction, and the readout circuits 112 may be arranged at the ends of the vertical signal lines 113. Also, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit, and one signal processing circuit may be shared by a plurality of photoelectric conversion units to perform signal processing sequentially.

[0023] Fig. 4 is a diagram showing an equivalent circuit of the pixel 101 in Fig. 2 and Fig. 3 and the signal processing circuit 103 corresponding to the pixel 101. As shown in Fig. 4, each pixel 101 includes a photoelectric conversion unit 102 as a sensor unit, a counter circuit 211, a memory circuit 212, etc.

[0024] The APD 201 included in the photoelectric conversion unit 102 generates a pair of electric charges according to incident light by photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VL (first voltage) is supplied. The other of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VH (second voltage) higher than the driving voltage VL is supplied.

[0025] In Fig. 4, one node of the APD 201 is the anode, and the other node of the APD is the cathode. A reverse bias voltage (VR) is supplied to the anode and cathode of the APD 201 so that the APD 201 performs avalanche multiplication. By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, generating an avalanche current.

[0026] When a reverse bias voltage is supplied, there are two modes: Geiger mode, in which the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, in which the voltage difference between the anode and cathode is close to or less than the breakdown voltage. An APD that operates in Geiger mode is called a SPAD. In the case of a SPAD, for example, the drive voltage VL (first voltage) is -30V, and the drive voltage VH (second voltage) is 1V.

[0027] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quench element 202 is connected to a power supply line to which a driving voltage VH is supplied and one of the anode and cathode nodes of the APD 201.

[0028] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, and has a function of suppressing avalanche multiplication by suppressing the voltage supplied to the APD 201 (quench operation).The quench element 202 also has a function of returning the voltage supplied to the APD 201 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0029] 4 shows an example in which the signal processing circuit 103 has a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212 in addition to the quench element 202. However, it is sufficient for the signal processing circuit 103 to have at least one of the waveform shaping unit 210, the counter circuit 211, and the memory circuit 212.

[0030] The waveform shaping unit 210 shapes the voltage change of the cathode of the APD 201 obtained when a photon is detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. Although an example in which one inverter is used as the waveform shaping unit 210 is shown in Fig. 4, a circuit in which a plurality of inverters are connected in series may be used, or another circuit having a waveform shaping effect may be used.

[0031] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. When a control pulse RES is supplied via a drive line 213, the signal held in the counter circuit 211 is reset. Here, the counter circuit 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0032] 3 to the memory circuit 212 via a drive line 214 (not shown in FIG. 3) in FIG. 4, and switches between electrical connection and disconnection between the counter circuit 211 and the vertical signal line 113. The memory circuit 212 functions as a memory that temporarily stores the count value of the counter, and outputs an output signal from the counter circuit 211 of the pixel to the vertical signal line 113.

[0033] Note that electrical connections may be switched by disposing a switch such as a transistor between the quench element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the driving voltage VH or the driving voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0034] 5 is a diagram showing a schematic diagram of the relationship between the operation of the APD 201 and the output signal. The input side of the waveform shaping unit 210 is nodeA, and the output side is nodeB. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at nodeA drops.

[0035] When the amount of voltage drop further increases and the potential difference applied to the APD 201 decreases, the avalanche multiplication of the APD 201 stops as at time t2, and the voltage level of nodeA does not drop below a certain value.

[0036] After that, between time t2 and time t3, a current flows through nodeA to compensate for the voltage drop from the drive voltage VL, and at time t3, nodeA is stabilized to its original potential level. At this time, the portion of the output waveform at nodeA that exceeds a certain threshold value is shaped by the waveform shaping section 210 and is output as a pulse signal at nodeB.

[0037] In addition, since the breakdown voltage of the APD 201 has temperature dependency, it is desirable to set the reverse bias voltage (VR) in conjunction with the temperature. In this embodiment, the temperature of the photoelectric conversion element 100 is detected, and the reverse bias voltage (VR) is corrected to match the temperature dependency of the breakdown voltage.

[0038] Fig. 6 is a functional block diagram showing an example of the configuration of the imaging device 300 according to the embodiment 1. Note that some of the functional blocks shown in Fig. 6 are realized by causing a computer (not shown) included in the imaging device 600 and the moving object 700 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0039] However, some or all of these may be realized by hardware. For the hardware, a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP) may be used. Furthermore, each functional block shown in Fig. 6 does not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths.

[0040] The imaging device 300 includes a photoelectric conversion element 100, a coupling optical system 301, a linear conversion unit 302, an image processing unit 303, and a control unit 304. The imaging device 300 further includes a storage unit 305, an exposure parameter determination unit 306, a voltage parameter determination unit 307, an imaging power supply unit 308, an aperture control unit 309, an aperture mechanism 310, a communication unit 311, and the like.

[0041] The combined optical system 301 is composed of multiple lenses and a holding member for holding them. In addition, zoom and focus can be controlled by a motor or the like. Furthermore, it has a configuration in which an infrared cut filter that cuts infrared wavelengths can be inserted and removed. The aperture mechanism 310 is a mechanism for reducing the amount of light entering the photoelectric conversion element 100, and is controlled by the aperture control unit 309.

[0042] The signal output from the photoelectric conversion element 100 is processed in the linear conversion section 302 so that the output characteristic becomes linear. That is, the linear conversion section 302 functions as a linear conversion means that linearly converts the output of the photoelectric conversion element.

[0043] Figure 7 is a diagram for explaining the SPAD sensor output and linear output relative to the amount of incident light in embodiment 1, and shows a schematic diagram of the sensor output output from the photoelectric conversion element 100 and the linear output output from the linear conversion section 302.

[0044] The relationship between the number of photons incident on the SPAD sensor and the number of counts for the number of periods in which avalanche multiplication occurs is nonlinear as shown in Figure 7, and gradually saturates as the amount of incident light increases, but is converted to a linear value in the linear conversion section 302 as shown in Figure 7.

[0045] The signal output from the linear conversion unit 302 is subjected to image processing such as black level correction, gamma curve correction, noise reduction, data compression, etc. in the image processing unit 303 to generate final image data. If an on-chip RGB color filter is disposed on the light receiving surface of the photoelectric conversion element 100, processing such as white balance correction and color conversion is also performed.

[0046] In addition, the image processing unit 303 calculates an evaluation value of AE (Auto Exposure) based on the luminance information of the acquired pixel data. The imaging device 300 sets the aperture, exposure time (photoelectric conversion time) or digital gain based on the AE evaluation value so as to obtain image data with appropriate brightness. Note that in this embodiment, the terms exposure and exposure are used interchangeably.

[0047] Furthermore, the brightness of the image data can be optimized by amplifying or attenuating the gain of the luminance level in accordance with the AE evaluation value in the image processing unit 303. In this embodiment, the data input to the image processing unit 303 is referred to as pixel data, and the data output from the image processing unit 303 is referred to as image data.

[0048] The storage unit 305 can temporarily store and read out image data output by the image processing unit 303. Furthermore, the storage unit 305 is also used as a storage area for programs and various parameters executed by the CPU 312 of the control unit 304, and as a work area during program execution.

[0049] The communication unit 311 converts image data in accordance with a communication protocol and distributes the converted image data to an external device (client device) (not shown). The communication unit 311 also performs compression encoding processing such as H.264 and H.265.

[0050] Furthermore, the communication unit 311 receives setting commands for various parameters from the external device, outputs them to the control unit 304, and also transmits a response to the external device. The external device can instruct the imaging device 300 to zoom or focus, and can also enlarge and display a distributed image. In this embodiment, a system including the imaging device and the above external device is called an imaging system.

[0051] The exposure parameter determination unit 306 determines the exposure parameters of the imaging device 300. The exposure parameters include at least one of the exposure time (photoelectric conversion time), digital gain, and aperture transition amount or transition time (transition count, transition interval, and transition timing).

[0052] The voltage parameter determination unit 307 determines the reverse bias voltage (VR) of the photoelectric conversion element 100. The voltage parameters include at least one of the transition amount or transition time (number of transitions, transition interval, and transition timing) of the reverse bias voltage VR. The voltage parameter determination unit 307 reflects the determined voltage parameters in the imaging power supply unit 308.

[0053] The imaging power supply unit 308 includes a control IC or the like that can variably control an output voltage such as a drive voltage supplied to the photoelectric conversion element 100. That is, the output voltage can be adjusted by rewriting a register in the control IC. However, this is not limiting. For example, the imaging power supply unit 308 may include multiple power supplies and be configured to select one of the multiple power supplies to supply power to the photoelectric conversion element 100 based on a command from the control unit 304.

[0054] The aperture control unit 309 controls the aperture mechanism 310 so as to achieve the aperture amount determined by the image processing unit 303 and the exposure parameter determination unit 306 .

[0055] The control unit 304 includes a CPU 312 as a computer and a memory (not shown) for storing computer programs. The CPU 312 executes the computer programs stored in the memory to control each unit of the imaging device 300.

[0056] In this embodiment, the reverse bias voltage applied to the avalanche photodiode is appropriately changed during operation of the imaging device in order to correct the temperature dependency of the reverse bias voltage, adjust the sensitivity of the avalanche photodiode, etc. That is, by reducing the reverse bias voltage of the avalanche photodiode, the sensitivity is increased, and by increasing the reverse bias voltage, the sensitivity is decreased.

[0057] The operation of reducing the number of photon counts by changing the reverse bias voltage will be described with reference to Fig. 8. Fig. 8 is a diagram for explaining the setting of a threshold value for the SPAD sensor output according to the first embodiment, and shows an example of the relationship between the SPAD sensor output with respect to the amount of incident light, the linear output obtained by linearly converting the sensor output, and the threshold value setting.

[0058] Region 1 is the effective range of input luminance values ​​in the image processing unit 303 subsequent to the linear conversion unit 302. In Fig. 8, the sensor output of the SPAD sensor is compressed in a region where the amount of incident light is large.

[0059] In an area where the output is compressed to a certain degree or more, if linear conversion is performed, gradation steps become noticeable. Therefore, as shown in area 1, an effective range of input luminance values ​​(incident light amount values) in the image processing unit 303 is set, and the linear conversion unit 302 linearly converts the sensor output within the effective range of input luminance values ​​in the image processing unit 303.

[0060] Region 2 is a region outside the valid range of input luminance values ​​in image processing unit 303. When the sensor output is within the range of region 2, linear conversion unit 302 outputs a saturated output (point 1 on the linear output) as an output after linear conversion.

[0061] Therefore, even if the sensor output increases within the range of region 2, the signal for image processing is saturated and does not affect the image. In other words, above the sensor output at which the linear output saturates (point 3 on the sensor output), an extra photon count operation is occurring in the SPAD sensor.

[0062] Therefore, in this embodiment, a threshold value is set for the sensor output, and the sensitivity of the SPAD sensor is adjusted by controlling the reverse bias voltage applied to the SPAD sensor according to the sensor output. Note that when the sensor output is within the range of region 1, the reverse bias voltage is not controlled because controlling the reverse bias voltage would cause uneven sensitivity for each pixel.

[0063] On the other hand, in region 2 where pixel saturation occurs, the reverse bias voltage is controlled to suppress unnecessary photon counting in region 2.

[0064] A specific method for controlling the reverse bias voltage will be described below. Threshold value 1 is a threshold value for determining whether the sensor output is outside the effective range of the input luminance value in the image processing unit 303, and in this embodiment, it is set to point 2 on the sensor output.

[0065] Threshold 2 is a threshold (point 3 on the sensor output) for determining the sensor output after the reverse bias voltage is changed. In this embodiment, threshold 2 is set to a value lower than threshold 1, and is the sensor output at the point when the linear output is saturated.

[0066] In this embodiment, when the output obtained from the SPAD sensor is greater than threshold value 1, the reverse bias voltage applied to the SPAD sensor is set to low sensitivity mode. In the low sensitivity mode, the reverse bias voltage is set to a higher voltage than the default voltage, thereby reducing the sensitivity of the SPAD sensor.

[0067] If the output obtained from the SPAD sensor is less than or equal to threshold 2, the reverse bias voltage is set to normal mode, where a default constant reverse bias voltage is set to drive the SPAD sensor.

[0068] In addition, when the reverse bias voltage is in the low sensitivity mode and the sensor output falls below threshold value 1, the reverse bias voltage is set to normal mode. This makes it possible to suppress the occurrence of counting operations that do not affect the image when the output brightness of the SPAD sensor is above the saturation output. Also, by providing two threshold values, it is possible to prevent frequent switching between low sensitivity mode and normal mode.

[0069] 9 is an explanatory diagram of an example of timing for changing the reverse bias voltage according to the first embodiment. Voltage 1 is the reverse bias voltage in the normal mode, and voltage 2 is the reverse bias voltage in the low sensitivity mode. Fig. 9 shows an example in which the SPAD sensor is driven in the normal mode in the first frame and transitions to the low sensitivity mode in the second frame.

[0070] In the first frame, photoelectric conversion is performed in exposure period 1, and the sensor output is read out in the subsequent readout period 1. Then, based on the readout sensor output, the control unit 304 compares it with a threshold value and determines whether or not to change the reverse bias voltage. In the example of FIG. 9, it is assumed that the output obtained from the SPAD sensor in readout period 1 of the first frame is equal to or greater than threshold value 1. In that case, the reverse bias voltage is changed to low sensitivity mode in blank period 1.

[0071] When changing the reverse bias voltage (VR), it is desirable to change the voltage during the voltage change period from the end of exposure period 1 of the first frame to the start of exposure period 2 of the second frame. This allows the exposure period 2 of the second frame of the SPAD sensor to start with the reverse bias voltage changed to the low sensitivity mode.

[0072] Fig. 10 is a flowchart showing an example of an imaging method performed by the control unit 304 according to embodiment 1. Note that the CPU 312 as a computer in the control unit 304 executes a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowchart in Fig. 6.

[0073] First, in step S1000, the CPU 312 acquires the output of the SPAD sensor. Next, in step S1001, the CPU 312 determines the sensor output based on the sensor output acquired in step S1000.

[0074] Here, step S1001 functions as a determination step (determination means) for determining the output level of the photoelectric conversion element based on a first threshold value. If the sensor output is greater than threshold value 1, the process proceeds to step S1002. If the sensor output is equal to or less than threshold value 1, the process proceeds to step S1003.

[0075] In step S1002, since the sensor output is greater than threshold value 1, CPU 312 sets the reverse bias voltage to low sensitivity mode and reduces the sensitivity of the SPAD sensor. That is, in step S1002, when it is determined that the output level of the photoelectric conversion element is greater than the first threshold value, the reverse bias voltage is controlled to be increased from the first voltage to the second voltage. Then, the flow of FIG. 10 ends.

[0076] On the other hand, in step S1003, the CPU 312 determines whether the reverse bias voltage has been changed to the low sensitivity mode. That is, it determines whether the current reverse bias voltage is in the low sensitivity mode. If the reverse bias voltage has been changed to the low sensitivity mode, the process proceeds to step S1004. If the reverse bias voltage has not been changed, the process proceeds to step S1006.

[0077] In step S1004, CPU 312 determines the sensor output based on the sensor output acquired in step S1000. If the sensor output is greater than threshold value 2, the process proceeds to step S1005. If the sensor output is equal to or less than the threshold value, the process proceeds to step S1006.

[0078] In step S1005, since the reverse bias voltage is in the low sensitivity mode and the sensor output is greater than threshold value 2, CPU 312 does not change the reverse bias voltage from the low sensitivity mode.

[0079] If the sensor output is equal to or lower than the threshold value 1 and has not been changed to the reverse bias voltage, in step S1006, the CPU 312 changes the reverse bias voltage to the normal mode. That is, if it is determined that the output level of the photoelectric conversion element is equal to or lower than the first threshold value, the reverse bias voltage is controlled to the first voltage.

[0080] Moreover, when the reverse bias voltage is in the low sensitivity mode and the sensor output is equal to or lower than threshold value 2, the reverse bias voltage is changed from the low sensitivity mode to the normal mode. That is, when the reverse bias voltage is controlled to be equal to or higher than the second voltage, if the output level of the photoelectric conversion element is equal to or lower than the second threshold value, the reverse bias voltage is switched to the first voltage.

[0081] After the processes of steps S1005 and S1006, the flow of Fig. 10 ends. The flow of Fig. 10 is periodically repeated for each frame. Steps S1002, S1005, and S1006 function as a voltage control step (voltage control means) that controls the reverse bias voltage applied to the avalanche photodiode based on the determination result of the determination step (determination means).

[0082] According to this embodiment, by controlling as shown in Figure 10, in an imaging device having a SPAD, the occurrence of photon counting operations that do not affect the image can be suppressed by controlling the reverse bias voltage according to the output of the SPAD sensor.

[0083] 8 is desirably set to a sensor output value equal to or greater than the saturation output in order to reduce the effect of changes in the sensitivity of the SPAD sensor on the image. In this embodiment, threshold value 2 is set to a value lower than threshold value 1, but threshold value 2 and threshold value 1 may be set to the same value.

[0084] Also, a threshold value may be set for the linear output, and the reverse bias voltage may be controlled based on the result of the determination of the linear output. That is, the output level of the linear conversion means may be determined as the output level of the photoelectric conversion element.

[0085] Fig. 11 is a diagram for explaining threshold setting of the linear output according to the embodiment 1. As shown by threshold 3 and threshold 4 in Fig. 11, thresholds may be set for the linear output, and the reverse bias voltage may be controlled based on the determination result of the linear output.

[0086] In that case, in step S1001 of FIG. 10, it is judged whether the linear output>threshold value 3, and in step S1004 it is judged whether the linear output>threshold value 4.

[0087] In this embodiment, the reverse bias voltage is set to one value in the low sensitivity mode. However, in the low sensitivity mode, the reverse bias voltage may be further switched depending on the sensor output.

[0088] That is, for example, the relationship between the reverse bias voltage and the sensitivity may be tabulated in advance, and when the sensor output exceeds threshold value 1, the amount of change in the reverse bias voltage may be controlled so that the sensitivity is reduced more as the difference between the sensor output and threshold value 1 becomes greater.

[0089] In this case, it is desirable to set an upper limit to the amount of change in the reverse bias voltage, and to set the upper limit based on the range in which the avalanche photodiode can detect photons. In this way, by changing the reverse bias voltage within the range in which the SPAD sensor can detect photons after the sensor output exceeds threshold value 1, the sensor output can be maintained even after the reverse bias voltage is changed.

[0090] Incidentally, if it is determined whether or not to change the reverse bias voltage based on the result of comparing the sensor output and the threshold value in some pixels, an unintended decrease in sensor sensitivity may occur.

[0091] For example, when strong directional light, such as from a vehicle headlight, is incident on part of the sensor, the reverse bias voltage of the entire photoelectric conversion element may transition to low sensitivity mode based on the judgment results of some saturated pixels.

[0092] In order to avoid such a malfunction, a statistical value may be calculated by averaging the pixel outputs on the entire surface of the sensor or passing a median filter through the sensor, and the calculated statistical value may be compared with a threshold value. In other words, a statistical value of the output levels of multiple pixels of a photoelectric conversion element may be determined as the output level of the photoelectric conversion element.

[0093] Also, the determination as to whether or not the reverse bias voltage should be changed may be made based on the results of statistical processing of the comparison results between the pixels on the entire surface of the sensor and the threshold value. Alternatively, the determination as to whether or not the reverse bias voltage should be changed may be made based on the number of pixels that exceed the threshold value by comparing pixels arbitrarily selected from the sensor with the threshold value. In other words, the output level of each pixel of the photoelectric conversion element may be compared with the first threshold value, and it may be determined whether or not to control the reverse bias voltage based on the number of pixels that exceed the first threshold value.

[0094] <Embodiment 2> In the first embodiment, whether or not to change the reverse bias voltage of the avalanche photodiode is determined based on the comparison result between the sensor output and the threshold value. In the second embodiment, an example will be described in which the presence or absence of a change in the reverse bias voltage is determined according to the sensor output, the exposure setting, and the state of a predetermined filter control.

[0095] In the second embodiment, an ND filter is used as the predetermined filter. Threshold value 1 and Threshold value 2 are set in the same manner as in the first embodiment.

[0096] Fig. 12 is a flowchart showing an example of an imaging method performed by the control unit 304 according to the second embodiment, and Fig. 13 is a flowchart continuing from Fig. 12. Note that the CPU 312 as a computer in the control unit 304 executes a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowcharts of Figs. 12 and 13.

[0097] First, in step S2000, the CPU 312 obtains the exposure parameters of the SPAD sensor determined by the exposure parameter determination unit.

[0098] Next, in step S2001, CPU 312 determines whether the exposure state is minimum based on the exposure parameters acquired in step S2000. Minimum exposure state refers to a state in which the exposure time is the shortest within the settable range and the aperture is the smallest. If the exposure state is minimum, the process proceeds to step S2002. If the exposure state is not minimum, the process proceeds to step S2011.

[0099] In step S2002, the CPU 312 determines whether an ND filter is inserted or removed. If an ND filter is inserted, the process proceeds to step S2003. If an ND filter is not inserted, the process proceeds to step S2011.

[0100] In step S2003, the CPU 312 acquires the output of the SPAD sensor, and proceeds to step S2004. In step S2004, the CPU 312 determines the sensor output based on the sensor output acquired in step S2003. If the sensor output is greater than threshold value 1, the process proceeds to step S2005. If the sensor output is equal to or less than threshold value 1, the process proceeds to step S2007.

[0101] In step S2005, since the sensor output is greater than threshold value 1, CPU 312 sets the reverse bias voltage to low sensitivity mode to reduce the sensitivity of the SPAD sensor.

[0102] Then, in step S2006, the CPU 312 fixes the exposure to the minimum exposure state. Fixing the exposure while changing the sensitivity of the SPAD sensor suppresses AE malfunctions caused by the change in sensitivity of the SPAD sensor. After the process of step S2006, the flow of FIG. 12 ends.

[0103] In step S2007, the CPU 312 determines whether the reverse bias voltage has been changed to the low sensitivity mode. That is, it determines whether the current reverse bias voltage is in the low sensitivity mode. If the reverse bias voltage has been changed to the low sensitivity mode, the process proceeds to step S2008. If the reverse bias voltage has not been changed, the process proceeds to step S2011.

[0104] In step S2008, CPU 312 determines the sensor output based on the sensor output acquired in step S2003. If the sensor output is greater than threshold value 2, the process proceeds to step S2009. If the sensor output is equal to or less than threshold value 2, the process proceeds to step S2011.

[0105] In step S2009, the CPU 312 sets the reverse bias voltage to the low sensitivity mode because the reverse bias voltage has already been changed to the low sensitivity mode and the sensor output is greater than threshold value 2. That is, the low sensitivity mode is maintained.

[0106] In step S2010, since the reverse bias voltage is in the low sensitivity mode, the CPU 312 fixes the exposure at the minimum exposure. In step S2011, the CPU 312 sets the reverse bias voltage to the normal mode. In step S2012, the CPU 312 sets the exposure control to the normal exposure control. After the processing of steps S2010 and S1012, the flow of Fig. 12 and Fig. 13 ends. Note that the flow of Fig. 12 and Fig. 13 is periodically repeated for each frame.

[0107] According to the second embodiment, in an imaging device having a SPAD, the occurrence of photon counting operations that do not affect an image can be suppressed by controlling the reverse bias voltage according to the output of the SPAD sensor. Also, the change in the reverse bias voltage is controlled based on the aperture state of the optical system that forms an optical image on the photoelectric conversion element, or the insertion / removal state of a filter that can be inserted / removed from the optical system.

[0108] Therefore, it is possible to suppress unnecessary sensitivity changes in the SPAD sensor when the exposure is not at minimum or when a specified filter is not inserted, etc. Furthermore, when controlling the reverse bias voltage, it is possible to suppress malfunctions of the AE and the specified filter.

[0109] In the flow of Fig. 12, it is also possible to control so that voltage adjustment is possible only when the brightness value is equal to or greater than the threshold value and the exposure is at maximum. In that case, if the exposure is at minimum and the brightness upper limit is exceeded, the difference may be converted into sensitivity, and the applied voltage may be calculated from the sensitivity. Alternatively, it is also possible to adjust the AE and reverse bias voltage, and then remove the IRCF if it is still dark.

[0110] <Embodiment 3> In the first and second embodiments, an example has been described in which the occurrence of photon counting operations that do not affect images is suppressed by controlling the reverse bias voltage based on the comparison result between the sensor output and the threshold value, etc.

[0111] In the third embodiment, control of the reverse bias voltage when the output luminance of the SPAD sensor is in an output range that affects an image will be described. Also, in the third embodiment, a case will be described in which the change amount of the reverse bias voltage is not fixed but is changed according to the sensor output.

[0112] 14 is a diagram illustrating an example of the SPAD sensor output with respect to the amount of incident light according to the third embodiment. Threshold value 5 and threshold value 6 are threshold values ​​for determining the sensor output and are set within the output range of the sensor output. Threshold value 6 is a threshold value for determining the sensor output after the reverse bias voltage is changed and is set to a value lower than threshold value 5.

[0113] When the output obtained from the SPAD sensor is equal to or greater than threshold value 5, the reverse bias voltage is controlled so that the sensitivity of the SPAD sensor decreases. Specifically, the difference between the output of the SPAD sensor and threshold value 5 is calculated, and the reverse bias voltage is set high so that the output of the SPAD sensor decreases by that difference.

[0114] The relationship between the reverse bias voltage and the sensitivity of the SPAD sensor is stored in advance as a table. For signals acquired by a SPAD sensor to which a reverse bias voltage controlled to reduce the sensitivity is applied, it is desirable to correct the reduction in sensitivity of the SPAD sensor.

[0115] Based on the table, the reverse bias voltage is controlled in multiple steps or continuously according to the output of the SPAD sensor, and the change in sensitivity of the SPAD sensor due to the change in the reverse bias voltage is corrected. That is, the reverse bias voltage is controlled in multiple steps or continuously based on a comparison between the output level of the photoelectric conversion element and the first threshold value.

[0116] As a result, the number of photon count operations for the same amount of incident light can be reduced. After the reverse bias voltage is changed, that is, in a state in which the reverse bias voltage is changed, when it falls below threshold value 6, the normal reverse bias voltage state is restored.

[0117] Fig. 15 is a flowchart showing an example of an operation of reducing the number of photon counts performed by the control unit 304 according to the third embodiment, and Fig. 16 is a flowchart continuing from Fig. 15. Note that the CPU 312 as a computer in the control unit 304 executes a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowcharts of Figs. 15 and 16.

[0118] First, in step S3000, CPU 312 acquires the output of the SPAD sensor. Next, in step S3001, CPU 312 judges the sensor output based on the sensor output acquired in step S3000. If the sensor output is greater than threshold value 5, the process proceeds to step S3002. If the sensor output is equal to or less than threshold value 5, the process proceeds to step S3008.

[0119] In step S3002, CPU 312 calculates the difference between the sensor output and threshold value 5, and in step S3003, calculates the amount of change in the reverse bias voltage based on the difference calculated in step S3002. For example, the relationship between the change in the reverse bias voltage and the sensor sensitivity is stored in a table. Then, assume that in a given output range, increasing the reverse bias voltage by, for example, 100 mV reduces the sensitivity of the SPAD sensor by 10%.

[0120] At that time, if the difference calculated in step S3002 is a voltage difference equivalent to a 10% decrease in sensitivity, the voltage change amount is set to 100 mV. Also, it is desirable to set an upper limit for the change amount of the reverse bias voltage within a range in which the SPAD sensor can detect photons. This makes it possible to obtain the sensor output and perform judgment processing even after changing the reverse bias voltage applied to the SPAD sensor.

[0121] In step S3004, the CPU 312 calculates a change in sensitivity of the SPAD sensor based on the change amount of the reverse bias voltage calculated in step S3003. That is, when controlling the reverse bias voltage to be higher than the first voltage, the CPU 312 calculates a change in sensitivity due to a change in the voltage of the photoelectric conversion element from the change amount of the reverse bias voltage.

[0122] In step S3005, the CPU 312 sets the reverse bias voltage to be applied to the SPAD sensor based on the amount of change in the reverse bias voltage calculated in step S3003.

[0123] In step S3006, the CPU 312 acquires the output of the SPAD sensor in the next frame. In step S3007, the CPU 312 corrects the sensor output acquired in step S3006 based on the change in sensitivity of the SPAD sensor caused by the change in the reverse bias voltage. That is, the output level of the photoelectric conversion element is corrected based on the change in sensitivity caused by the voltage change. After the processing of step S3007, the flow of FIG. 15 ends.

[0124] In step S3008, the CPU 312 determines whether the reverse bias voltage has been changed to the low sensitivity mode. That is, it determines whether the current reverse bias voltage is in the low sensitivity mode. If the reverse bias voltage has been changed from the default voltage, the process proceeds to step S3009. If the reverse bias voltage has not been changed from the default voltage, the process proceeds to step S3012.

[0125] In step S3009, CPU 312 determines the sensor output based on the sensor output acquired in step S3000. If the sensor output is greater than threshold value 6, the process proceeds to step S3010. If the sensor output is equal to or less than threshold value 6, the process proceeds to step S3012.

[0126] In step S3010, the CPU 312 sets (maintains) the reverse bias voltage to the same value as the current one because the reverse bias voltage has been changed and the sensor output is greater than threshold value 6. In step S3011, the CPU 312 calculates the change in sensitivity of the SPAD sensor due to the change in the reverse bias voltage to the same value as the current one because the reverse bias voltage has been changed and the sensor output is greater than threshold value 6.

[0127] In step S3012, since the sensor output is equal to or lower than the threshold value, the CPU 312 sets the reverse bias voltage to a default value. In step S3013, the CPU 312 sets the change in sensitivity of the SPAD sensor due to the change in the reverse bias voltage to zero.

[0128] After the processes of steps S3011 and S3013, the flows in Figures 115 and 16 are terminated. Note that the flows in Figures 15 and 16 are periodically repeated for each frame.

[0129] According to the third embodiment, in an imaging device having a SPAD, the reverse bias voltage is controlled in multiple steps or continuously according to the output of the SPAD sensor, and the change in sensitivity of the SPAD sensor caused by the change in the reverse bias voltage is corrected. Therefore, the number of photon count operations for the same amount of incident light can be reduced.

[0130] In the above embodiment, if the brightness value (output level of the photoelectric conversion element) is equal to or higher than the threshold value, the reverse bias voltage of the SPAD sensor may be adjusted (sensitivity may be reduced). In this case, a brightness threshold value lower than the brightness saturation may be set, and the brightness at which the brightness value saturates may be shifted to prevent the brightness saturation as much as possible. Alternatively, the voltage may be controlled to change gradually.

[0131] Although the present invention has been described in detail based on the preferred embodiment, the present invention is not limited to the above embodiment, and various modifications are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. The present invention includes the following combinations.

[0132] (Configuration 1) An imaging device comprising: a photoelectric conversion element that performs photoelectric conversion using an avalanche photodiode; a determination means that determines the output level of the photoelectric conversion element based on a first threshold value; and a voltage control means that controls a reverse bias voltage applied to the avalanche photodiode based on the determination result of the determination means.

[0133] (Configuration 2) The imaging device described in configuration 1, characterized in that the voltage control means controls the reverse bias voltage to be increased from a first voltage to a second voltage when the determination means determines that the output level of the photoelectric conversion element is greater than the first threshold value.

[0134] (Configuration 3) The imaging device described in Configuration 2, characterized in that the voltage control means controls the reverse bias voltage to the first voltage when the determination means determines that the output level of the photoelectric conversion element is equal to or lower than the first threshold value.

[0135] (Configuration 4) The imaging device according to any one of configurations 1 to 3, wherein an upper limit of the reverse bias voltage is set based on a range in which the avalanche photodiode can detect photons.

[0136] (Configuration 5) When the reverse bias voltage is controlled to be equal to or higher than the second voltage, The imaging device according to configuration 2, characterized in that, in the determination means, when the output level of the photoelectric conversion element is equal to or lower than a second threshold value that is equal to or lower than the first threshold value, the reverse bias voltage is switched to the first voltage.

[0137] (Configuration 6) An imaging device described in any one of configurations 1 to 5, characterized in that the voltage control means controls the reverse bias voltage in multiple stages or continuously based on a comparison between the output level of the photoelectric conversion element and the first threshold value.

[0138] (Configuration 7) An imaging device as described in configuration 2 or 5, characterized in that when the voltage control means controls the reverse bias voltage to be higher than the first voltage, a change in sensitivity due to a change in voltage of the photoelectric conversion element is calculated from an amount of change in the reverse bias voltage.

[0139] (Configuration 8) The imaging device described in Configuration 7, wherein the voltage control means, when controlling the reverse bias voltage to be higher than the first voltage, corrects the output level of the photoelectric conversion element based on the change in sensitivity due to the voltage change.

[0140] (Configuration 9) The imaging device according to any one of configurations 1 to 8, further comprising linear conversion means for linearly converting the output of the photoelectric conversion element.

[0141] (Configuration 10) The imaging device according to configuration 9, wherein the determining means determines an output level of the linear conversion means as the output level of the photoelectric conversion element.

[0142] (Configuration 11) The imaging device according to any one of configurations 1 to 10, wherein the determining means determines a statistical value of output levels of a plurality of pixels of the photoelectric conversion element as the output level of the photoelectric conversion element.

[0143] (Configuration 12) An imaging device described in any one of configurations 1 to 11, characterized in that the change in the reverse bias voltage is controlled based on the aperture state of an optical system that forms an optical image on the photoelectric conversion element, or the insertion / removal state of a filter that can be inserted or removed from the optical system.

[0144] (Configuration 13) An imaging device described in any one of configurations 1 to 12, characterized in that the determination means compares the output level of each pixel of the photoelectric conversion element with the first threshold value, and determines whether or not to control the reverse bias voltage based on the number of pixels that exceed the first threshold value.

[0145] (Method) An imaging method for controlling a photoelectric conversion element that performs photoelectric conversion using an avalanche photodiode, comprising: a determination step for determining the output level of the photoelectric conversion element based on a first threshold value; and a voltage control step for controlling the reverse bias voltage applied to the avalanche photodiode based on the determination result of the determination step.

[0146] (Program) A computer program for controlling each means of the imaging device according to any one of configurations 1 to 12 by a computer.

[0147] A computer program for implementing all or part of the control in this embodiment may be provided to the imaging device via a network or various storage media. Then, a computer (or a CPU, MPU, etc.) in the imaging device may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0148] 100: Photoelectric conversion element 11: Sensor board 12: Pixel area 21: Circuit board 22: Circuit area 101: Pixel 102: Photoelectric conversion unit 103: Signal processing circuit 110: Vertical scanning circuit 111: Horizontal scanning circuit 112: Readout circuit 113: Signal line 114: Output circuit 115: Control pulse generation unit 201: Avalanche photodiode 202: Quench element 210: Waveform shaping section 211: Counter circuit 212: Selection circuit 213: Drive wire 300: Imaging device 301: Imaging optical system 302: Signal processing unit 303: Image generation unit 304: Correction processing unit 305: Storage section

Claims

1. a photoelectric conversion element that performs photoelectric conversion using an avalanche photodiode; a determination means for determining an output level of the photoelectric conversion element based on a first threshold value; a voltage control means for controlling a reverse bias voltage applied to the avalanche photodiode based on a result of the determination by the determination means; An imaging device comprising:

2. 2. The imaging device according to claim 1, wherein the voltage control means controls the reverse bias voltage to be increased from a first voltage to a second voltage when the determination means determines that the output level of the photoelectric conversion element is greater than the first threshold value.

3. 3. The imaging device according to claim 2, wherein the voltage control means controls the reverse bias voltage to the first voltage when the determination means determines that the output level of the photoelectric conversion element is equal to or lower than the first threshold value.

4. 2. The imaging device according to claim 1, wherein an upper limit of the reverse bias voltage is set based on a range in which the avalanche photodiode can detect photons.

5. When the reverse bias voltage is controlled to be equal to or higher than the second voltage, 3. The imaging device according to claim 2, wherein the determining means switches the reverse bias voltage to the first voltage when the output level of the photoelectric conversion element is equal to or lower than a second threshold value that is equal to or lower than the first threshold value.

6. 2. The image pickup apparatus according to claim 1, wherein the voltage control means controls the reverse bias voltage in multiple steps or continuously based on a comparison between the output level of the photoelectric conversion element and the first threshold value.

7. When the voltage control means controls the reverse bias voltage to be higher than the first voltage, 3. The imaging device according to claim 2, wherein a change in sensitivity of the photoelectric conversion element caused by a change in voltage is calculated from an amount of change in the reverse bias voltage.

8. When the voltage control means controls the reverse bias voltage to be higher than the first voltage, 8. The imaging device according to claim 7, wherein the output level of the photoelectric conversion element is corrected based on the change in sensitivity caused by the change in voltage.

9. 2. The image pickup apparatus according to claim 1, further comprising linear conversion means for linearly converting an output of the photoelectric conversion element.

10. 10. The image pickup apparatus according to claim 9, wherein the determining means determines an output level of the linear conversion means as the output level of the photoelectric conversion element.

11. 2. The image pickup apparatus according to claim 1, wherein the determining means determines a statistical value of output levels of a plurality of pixels of the photoelectric conversion element as the output level of the photoelectric conversion element.

12. 2. The imaging device according to claim 1, wherein the change in the reverse bias voltage is controlled based on a state of an aperture of an optical system that forms an optical image on the photoelectric conversion element, or a state of insertion or removal of a filter that can be inserted into or removed from the optical system.

13. 2. The imaging device according to claim 1, wherein the determination means compares an output level of each pixel of the photoelectric conversion element with the first threshold value, and determines whether or not to control the reverse bias voltage based on the number of pixels that exceed the first threshold value.

14. A control method for controlling a photoelectric conversion element that performs photoelectric conversion using an avalanche photodiode, comprising: a determining step of determining an output level of the photoelectric conversion element based on a first threshold value; a voltage control step of controlling a reverse bias voltage applied to the avalanche photodiode based on a result of the determination step; 13. An imaging method comprising:

15. A computer program for controlling each unit of the imaging apparatus according to any one of claims 1 to 13 by a computer.

Images