Photoelectric conversion device, photoelectric conversion method, computer program product, and storage medium

Abstract

This invention relates to a photoelectric conversion device, a photoelectric conversion method, a computer program product, and a storage medium. The photoelectric conversion device includes a photoelectric conversion element comprising an avalanche photodiode for outputting a signal based on incident photons, and a counter for counting and outputting the number of output signals from the avalanche photodiode. It determines whether the number of output signals is equal to or greater than a threshold. The exposure time of the avalanche photodiode and the recharge cycle of the avalanche photodiode during the exposure time are controlled. At least one of the threshold, the exposure time, and the recharge cycle is controlled such that a maximum number of recharges determined based on the exposure time and the recharge cycle is greater than the threshold.

Description

Technical Field

[0001] This disclosure relates to photoelectric conversion devices, photoelectric conversion methods, computer program products, and storage media. Background Technology

[0002] In recent years, photoelectric conversion elements have been proposed that digitally measure the number of photons arriving at an avalanche photodiode (APD) and output the measured value as a digital signal after photoelectric conversion from the pixel.

[0003] Japanese Patent Application Publication No. 2020-123847 discloses a photoelectric conversion device that enables the appropriate detection of the number of avalanche multiplication cycles by controlling the timing of APD recharging using a pulse signal.

[0004] In the photoelectric conversion device disclosed in Japanese Patent Application Publication No. 2020-123847, one photon can be counted for each cycle of a pulse generated by a pulse generation circuit. In this scheme, when multiple photons are incident during one pulse cycle, count omissions may occur.

[0005] However, when photographing general steady-state light, the effect of count omissions can be reduced through correction processing by statistically predicting the number of photons incident in one pulse cycle of the pulse.

[0006] On the other hand, in environments where illumination is provided by light sources that flicker within a frame, or particularly LED light sources using pulse modulation driving methods, or when using light sources that move within a frame, many count omissions occur compared to the case of using steady-state light. Therefore, the accuracy of the process for correcting these count omissions may deteriorate, and color variations compared to the actual environment may occur. Summary of the Invention

[0007] According to one aspect of this disclosure, a photoelectric conversion device includes: a photoelectric conversion element comprising an avalanche photodiode for outputting a signal based on incident photons, and a counter for counting the number of output signals from the avalanche photodiode and outputting the number of output signals. The device determines whether the number of output signals is equal to or greater than a threshold. It controls the exposure time of the avalanche photodiode and the recharge cycle of the avalanche photodiode during the exposure time. It controls at least one of the threshold, the exposure time, and the recharge cycle such that a maximum number of recharges determined based on the exposure time and the recharge cycle is greater than the threshold.

[0008] Further features of the invention will become apparent from the following description of embodiments with reference to the accompanying drawings. Attached Figure Description

[0009] Figure 1 This is a diagram illustrating an example configuration of a photoelectric conversion element according to a first embodiment of the present disclosure.

[0010] Figure 2 This is a diagram illustrating an example configuration of a sensor substrate according to the first embodiment.

[0011] Figure 3 This is a diagram illustrating an example of the configuration of a circuit board according to the first embodiment.

[0012] Figure 4 This is a diagram illustrating an example of an equivalent circuit of a pixel and a corresponding signal processing circuit according to the first embodiment.

[0013] Figure 5 This is a timing diagram of the photoelectric conversion element according to the first embodiment.

[0014] Figure 6A This is a diagram illustrating the counting operation when steady-state light is used as illumination.

[0015] Figure 6B This is a diagram illustrating a counting operation when a light source that flickers over time is used for illumination.

[0016] Figure 6C This is a diagram illustrating the illumination intensity of the region of interest when a light source is present and moving within a frame.

[0017] Figure 7A The following diagram illustrates an example: In existing photoelectric conversion elements, the maximum number of recharge cycles is increased by shortening the recharge cycle, and the saturation judgment count is set to be less than the maximum number of recharge cycles.

[0018] Figure 7B The following diagram illustrates an example: In a prior art photoelectric conversion element, the saturation judgment count is set to be less than the maximum number of recharges by extending the exposure time.

[0019] Figure 7C The following diagram illustrates an example: In existing photoelectric conversion elements, the number of saturation judgment counts is set to be less than the maximum number of recharge cycles by reducing the number of saturation judgment counts.

[0020] Figure 7D This is a diagram illustrating an example of the reduction in the number of photons incident on each pixel of the photoelectric conversion element per unit time.

[0021] Figures 8A to 8D This is a diagram illustrating an example of the response characteristics of a photoelectric conversion element according to the first embodiment.

[0022] Figure 9This is a diagram illustrating an example of the relationship between the saturation determination count, exposure time, and recharge cycle in a photoelectric conversion element according to the first embodiment.

[0023] Figure 10 This is a diagram illustrating an example of the relationship between the saturation determination count, exposure time, and recharge cycle in a photoelectric conversion element according to the second embodiment.

[0024] Figure 11 This is a diagram illustrating an example of the relationship between the saturation determination count, exposure time, and recharge cycle in a photoelectric conversion element according to the third embodiment.

[0025] Figure 12 This is a functional block diagram illustrating an example configuration of a photoelectric conversion device using a photoelectric conversion element according to the first to fourth embodiments. Detailed Implementation

[0026] In the following description, with reference to the accompanying drawings, advantageous modes of the present disclosure will be used to illustrate the embodiments. In the drawings, the same reference numerals are applied to the same components or elements, and repeated descriptions will be omitted or simplified.

[0027] First Embodiment Figure 1 This is a diagram illustrating an example configuration of a photoelectric conversion element 100 according to a first embodiment of the present disclosure. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 for processing signals detected in the pixel region 12.

[0028] In this embodiment, an example of a photoelectric conversion device having a so-called stacked structure will be described, in which the photoelectric conversion element 100 is configured such that two substrates serving as a sensor substrate 11 and a circuit substrate 21 are stacked and electrically connected. However, the photoelectric conversion device may have a so-called non-stacked structure, in which the configuration including the sensor substrate and the configuration including the circuit substrate are arranged in a common semiconductor layer.

[0029] Figure 2 This is a diagram illustrating an example configuration of the sensor substrate 11 according to the first embodiment. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged in a two-dimensional manner across multiple rows and columns. Each pixel 101 includes a photoelectric conversion unit 102 comprising an avalanche photodiode (hereinafter referred to as an APD). There are no particular limitations on the number of rows and columns of the pixel array forming the pixel region 12.

[0030] Figure 3 This is a diagram illustrating an example configuration of the circuit board 21 according to the first embodiment. The circuit board 21 includes components for... Figure 2The photoelectric conversion unit 102 processes the charge after photoelectric conversion, and includes a signal processing circuit 103, a reading circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a signal line 113, and a vertical scanning circuit 110.

[0031] The vertical scanning circuit 110 receives control pulses supplied from the control pulse generation unit 115 and supplies the control pulses to each pixel. The vertical scanning circuit 110 uses logic circuits such as shift registers or address decoders.

[0032] The signal output from the photoelectric conversion unit 102 of the pixel is processed by the signal processing circuit 103. The signal processing circuit 103 includes a counter or memory and stores the digital value in the memory. The horizontal scanning circuit 111 inputs control pulses for sequentially selecting each column to the signal processing circuit 103 to read signals from the memory of each pixel that stores the digital signal.

[0033] For the selected column, the signal processing circuit 103 of the pixel selected by the vertical scanning circuit 110 outputs a signal to the signal line 113. The signal output to the signal line 113 is output to the outside of the photoelectric conversion element 100 via the output circuit 114.

[0034] like Figure 2 and Figure 3 As illustrated, multiple signal processing circuits 103 are arranged in the region overlapping with the pixel region 12 in the plan view. In the plan view, the vertical scanning circuit 110, the horizontal scanning circuit 111, the readout circuit 112, the output circuit 114, and the control pulse generation unit 115 are arranged to overlap between the end of the sensor substrate 11 and the end of the pixel region 12.

[0035] In other words, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. The vertical scanning circuit 110, the horizontal scanning circuit 111, the readout circuit 112, the output circuit 114, and the control pulse generation unit 115 are arranged in the region that overlaps with the non-pixel region in the plan view.

[0036] The array of signal lines 113 and the arrays of read circuit 112 and output circuit 114 are not limited to Figure 3 An array. For example, signal line 113 may extend in the row direction, and read circuitry 112 may be arranged at the extension of signal line 113.

[0037] It is not necessary to include the function of a signal processing unit in every single photoelectric conversion unit. Instead, a signal processing unit can be shared by multiple photoelectric conversion units, and signal processing can be performed sequentially.

[0038] Figure 4 This is a diagram illustrating an example of a pixel 101 and an equivalent circuit of a signal processing circuit 103 corresponding to the pixel 101 according to the first embodiment.

[0039] The APD 201 generates electron-hole pairs corresponding to incident light through photoelectric conversion. Of the two nodes in the APD 201, one node is connected to a power supply line supplied with the driving voltage VL (first voltage).

[0040] Of the two nodes in APD 201, the other node can be connected via switch 202 to a power supply line supplied with a drive voltage VH (second voltage) that is higher than voltage VL. Figure 4 In this design, one node of APD 201 is the anode, and the other node of APD 201 is the cathode.

[0041] A reverse bias voltage is supplied to the anode and cathode of the APD 201 to cause the APD 201 to perform avalanche multiplication operation. When such a voltage is supplied, the charge generated by the incident light causes avalanche multiplication to generate an avalanche current.

[0042] There are Geiger mode and linear mode. In Geiger mode, operation occurs when the voltage difference between the anode and cathode is greater than the breakdown voltage when a reverse bias voltage is supplied. In linear mode, operation occurs when the voltage difference between the anode and cathode is close to, equal to, or less than the breakdown voltage. An APD operating in Geiger mode is called a SPAD. In the case of a SPAD, for example, the voltage VL (first voltage) is -30V and the voltage VH (second voltage) is 1V.

[0043] Switch 202 is connected to a power supply line supplied with drive voltage VH and one of the nodes of the anode and cathode of APD 201. Switch 202 switches the resistance value between APD 201 and the power supply line supplied with drive voltage VH.

[0044] In other words, switch 202 is connected to one of the nodes of the anode and cathode of the avalanche photodiode and the power line to which the driving voltage is applied, and switches the resistance value between that node and the power line.

[0045] During the switching of resistance values, the resistance value preferably changes by a factor of 10 or more, and more preferably by a factor of 100 or more. In the following text, the state of decreasing resistance value is referred to as the ON state of switch 202, and the state of increasing resistance value is referred to as the OFF state of switch 202.

[0046] Switch 202 functions as a quenching element. That is, switch 202 acts as a load circuit (quenching circuit) during signal amplification through avalanche multiplication, suppressing the voltage supplied to APD 201 and performing a quenching operation to suppress avalanche multiplication.

[0047] Switch 202 performs a recharge operation to return the voltage supplied to APD 201 to the drive voltage VH by allowing current to flow corresponding to the voltage drop via a quenching operation. Switch 202 may be configured with a MOS transistor. Figure 4 Example of a switch 202 as a PMOS transistor.

[0048] A control signal CLK for controlling the switch 202 is applied from the control pulse generation unit 115 to the gate electrode of the MOS transistor included in the switch 202. The control pulse generation unit 115 controls the recharge cycle of the avalanche photodiode by controlling the voltage applied to the gate electrode of the switch 202 and controlling the switch 202 to turn on and off.

[0049] The signal processing circuit 103 includes a waveform shaping unit 210, a counter circuit 211, and a selection circuit 212. The signal processing circuit 103 may include at least one of the waveform shaping unit 210, the counter circuit 211, and the selection circuit 212.

[0050] The waveform shaping unit 210 shapes the voltage change of the cathode of the APD 201 obtained during photon detection and outputs a pulse signal. For example, an inverter circuit can be used as the waveform shaping unit 210. As the inverter circuit, a circuit with multiple inverters connected in series can be used, or other circuits with waveform shaping effects can be used.

[0051] The counter circuit 211 counts the pulse signal output from the waveform shaping unit 210 and holds the count value. When a control pulse RES is supplied via the drive line 213, the signal held in the counter circuit 211 is reset.

[0052] A saturation judgment unit 215, which determines that the number of output signals measured by the counter circuit is saturated, is connected to the counter circuit 211. When the number of output signals measured by the counter circuit 211 reaches the saturation judgment count, the saturation judgment unit 215 suspends the counter circuit from counting the pulse signals upwards and keeps the saturation judgment count as the count value in the counter circuit 211.

[0053] In the photoelectric conversion element according to this embodiment, by controlling the control pulse generation unit 115 through the control unit 801 (described below), the saturation judgment count becomes less than the maximum number of recharges determined by the quotient obtained by dividing the exposure time of the APD by the recharge cycle. As a detail described below, by controlling the saturation judgment count through the saturation judgment unit 215 as described above, a photoelectric conversion device that can suppress color changes even in environments where a flickering light source is used as illumination light can be provided.

[0054] The saturation count can be a fixed value, as long as it is less than the maximum number of recharge cycles, but it is preferably configured to vary depending on the exposure time or recharge cycle. When the photoelectric conversion element has multiple driving modes, it can have a mode where the saturation count is equal to or greater than the maximum number of recharge cycles. The saturation count for each pixel can be controlled by a signal line (not shown).

[0055] Select circuit 212 from Figure 3 The vertical scanning circuit 110 in the middle is via Figure 4 Drive line 214 in Figure 3 (Not illustrated) A control pulse SEL is supplied to switch between an electrical connection and a non-connection between the counter circuit 211 and the signal line 113. The selection circuit 212 includes, for example, a buffer circuit for the output signal.

[0056] Figure 4 The illustrated output signal OUT is used as the output signal for pixel output and also outputs the count value of the counter circuit 211. That is, the counter circuit 211 functions as a counter that counts the number of output signals from the avalanche photodiode until the number of count signals reaches saturation, determines the count value, and outputs that value.

[0057] Switches such as transistors can be arranged between switch 202, which serves as a quenching element, and APD 201, or between photoelectric conversion unit 102 and signal processing circuit 103, to switch electrical connections. Similarly, switches such as transistors can be used to electrically switch the supply of voltage VH or voltage VL to photoelectric conversion unit 102.

[0058] Quenching and recharging operations can be performed, in which switch 202 is used in response to avalanche multiplication in APD 201. However, in some cases, the signal is not determined to be an output signal based on the photon detection timing.

[0059] For example, suppose that at a time when avalanche multiplication occurs in the APD, the input voltage to node A goes low and a recharge operation is performed. Typically, the decision threshold of waveform shaping unit 210 is set to a voltage higher than the voltage difference at which avalanche multiplication occurs in the APD.

[0060] When a photon is incident on the APD when the voltage of nodeA is lower than the threshold value due to a recharging operation and the APD is in a voltage state where avalanche multiplication may occur, avalanche multiplication occurs in the APD and the voltage of nodeA decreases.

[0061] In other words, because the voltage at node A drops below the threshold for detection, the output voltage from node B remains unchanged despite the detection of photons. Therefore, although avalanche multiplication occurs, it is not recognized as a signal.

[0062] In particular, under high illumination, photons enter continuously within a short period of time, making it difficult to identify them as a signal. Therefore, despite high illumination, the actual number of incident photons can easily deviate from the output signal.

[0063] However, in this embodiment, by applying the control signal CLK to the switch 202 and switching the switch 202 between on and off, even when photons continuously enter the APD in a short period of time, it can be determined as a signal.

[0064] Figure 5 This is a timing diagram used to describe the photoelectric conversion element of the first embodiment, and schematically illustrates the relationship between the control signal CLK of the switch, the voltage of node A, the voltage of node B, and the output signal. Figure 5 This example illustrates the case where the control signal CLK is a pulse signal with a constant repetition period.

[0065] In the photoelectric conversion device according to this embodiment, when the control signal CLK is at a high level, it is difficult to supply the drive voltage VH to the APD. When the control signal CLK is at a low level, the drive voltage VH is supplied to the APD. The high level of the control signal CLK is, for example, 1V, and the low level of the control signal CLK is, for example, 0V.

[0066] When the control signal CLK is high, switch 202 is off. When the control signal CLK is low, switch 202 is on. The resistance of switch 202 during the high level of the control signal CLK is higher than the resistance of switch 202 during the low level of the control signal CLK.

[0067] When the control signal CLK is high, although avalanche multiplication occurs in the APD, recharging is difficult to perform. Therefore, the voltage supplied to the APD becomes equal to or less than the APD's breakdown voltage. Thus, avalanche multiplication operation in the APD stops.

[0068] At time t1, the control signal CLK changes from high to low, switch 202 becomes on, and the APD's recharging operation begins. Therefore, the voltage at the APD's cathode becomes high.

[0069] Then, the voltage difference between the anode and cathode applied to the APD makes avalanche multiplication possible. The cathode voltage is the same as the voltage at node A. Therefore, when the cathode voltage transitions from low to high, the voltage at node A at time t2 becomes equal to or greater than the judgment threshold.

[0070] At this time, the pulse signal output from nodeB reverses from high level to low level. When recharging is complete, the voltage difference between drive voltages VH and VL is applied to APD 201. Afterwards, control signal CLK goes high and switch 202 turns off.

[0071] Subsequently, at time t3, when a photon is incident on APD 201, avalanche multiplication occurs in APD 201. The avalanche multiplication current flows to switch 202, and the voltage of the cathode decreases. That is, the voltage of node A decreases.

[0072] When the voltage of node A drops below a threshold during a voltage decrease period, the voltage of node B changes from low to high. That is, the portion of the output waveform in node A that exceeds the threshold is shaped by the waveform shaping unit 210 and output as a signal in node B. Then, the counter circuit counts, and the count value of the counter signal output from the counter circuit increases by one LSB.

[0073] Between time t3 and time t4, a photon is incident on the APD, but switch 202 is in the off state, and the voltage applied to APD 201 is not the voltage difference that makes avalanche multiplication possible. Therefore, the voltage level of node A does not exceed the judgment threshold.

[0074] At time t4, the control signal CLK changes from high to low, and switch 202 becomes on. Therefore, current flows to node A to compensate for the voltage drop from the drive voltage VL, and the voltage at node A returns to its original level. At this point, since the voltage at node A becomes at or above the judgment threshold at time t5, the pulse signal at node B reverses from high to low.

[0075] At time t6, nodeA is statistically determined to be at its original voltage level, and the control signal CLK changes from low to high. Thereafter, as described from time t1 to time t6, the voltage of each node or signal line, etc., changes in response to the control signal CLK or the incidence of photons. Thus, by applying the control signal CLK to switch 202 and switching switch 202 between on and off, the recharging frequency of the APD can be controlled.

[0076] When the control signal CLK is not used, the following problem occurs: the actual count value becomes less than the count value corresponding to the brightness of the incident light during the high brightness period. However, this problem can be solved by applying the control signal CLK to switch 202 and toggling the switch 202 between on and off states.

[0077] Here, when the recharge frequency of the APD is controlled by the control signal CLK and multiple photons are incident during one pulse period, count omissions may occur. Therefore, the relationship between the number of input signals and the number of output signals is not linear.

[0078] When shooting typical steady-state light, the relationship between the number of input signals and the number of output signals can theoretically be derived from the statistical distribution of the number of incident photons. Specifically, when the number of input signals is Nph, the number of output signals is Nct, the recharge period of the control signal CLK is T, and the exposure time is t, the following equation 1 holds. (Equation 1)

[0079] Therefore, in this embodiment, Equation 2 is used to statistically predict the number of photons incident during one pulse period based on the number of output signals Nct. Thus, the effect of counting omissions can be compensated for by the correction processing in the signal processing unit 804 described below. That is, the nonlinear correction of the number of output signals Nct of the counter circuit can be achieved by performing Equation 2 processing in the signal processing unit 804. (Equation 2)

[0080] On the other hand, in environments where illumination is provided by light sources that flicker within a single frame, or particularly LED light sources using pulse modulation driving methods, numerous counting omissions occur compared to the use of steady-state light. Therefore, the accuracy of the processing to correct for these counting omissions may deteriorate, and color variations may occur in real-world environments.

[0081] Therefore, in this embodiment, in the saturation determination unit 215, the saturation determination count when it is determined that the number of output signals measured by the counting unit has become saturated is set to be less than the maximum number of recharge cycles determined by the quotient obtained by dividing the exposure time by the recharge cycle. Therefore, even in environments where a flickering light source is used for illumination, a photoelectric conversion element capable of suppressing color changes can be provided. This will be described below.

[0082] Figure 6A This is a diagram illustrating the counting operation when steady-state light is used as illumination. Figure 6B This is a diagram illustrating the counting operation when a light source that flickers over time is used for illumination, and it is also a diagram illustrating the reason for the color change when using a prior art photoelectric conversion element in an environment where a flickering light source is used for illumination.

[0083] exist Figure 6A and Figure 6B In this design, photoelectric conversion elements, including on-chip color filters such as the so-called RGB Bayer array, are used as photoelectric conversion elements. That is, each pixel receives light via one of the R, G, and B color filters. An example is described where white light with a photon ratio of 1:2:1 incident on the R:G:B pixels per unit time is used for illumination.

[0084] When Figure 6A When steady-state light with constant illumination intensity is incident, the number of photons incident in each recharge cycle is small for each pixel of R, G, and B. Therefore, the count omission in each recharge cycle is small.

[0085] On the other hand, such as Figure 6B Therefore, when a flickering light source with discrete illumination intensity is used for illumination, photons are incident and concentrated in specific recharge cycles, leading to many count omissions. That is, when illumination is used with light that produces different numbers of photons incident on R:G:B pixels per unit time, the number of photons missed in pixels of a specific color (G:B) will vary. Figure 6B Many count omissions occur in G).

[0086] Therefore, the balance of R pixel:G pixel:B pixel counts may break down, and color changes may occur. For example, in Figure 6B In the example, as a result of many count omissions occurring in the G pixel, a color change toward magenta occurs.

[0087] exist Figure 6B The example illustrates the case where a flashing light source is used for illumination. A similar count omission occurs even when a moving light source is used for illumination. Figure 6C This is a diagram illustrating the illumination intensity of the region of interest when a light source is present and moving within a frame.

[0088] according to Figure 6C It is understandable that within the region of interest, there are moments of strong illumination intensity and moments of weak illumination intensity within a single frame. Therefore, as in the case of using a flickering light source for illumination, photons are incident and concentrated in specific recharge cycles, leading to many missed counts.

[0089] However, in the photoelectric conversion element according to this embodiment, the saturation determination count Nsat in the saturation determination unit 215 is set to be less than the maximum number of recharges determined by the quotient obtained by dividing the exposure time t by the recharge cycle T. That is, the saturation determination count Nsat, the exposure time t, and the recharge cycle T are set such that the following equation 3 holds. (Equation 3)

[0090] In this embodiment, by setting the saturation judgment count Nsat to satisfy Equation 3, color changes can be suppressed compared to existing photoelectric conversion elements. In the following text, reference will be made to... Figures 7A to 7C Two specific examples are described in this embodiment.

[0091] Figure 7A The following diagram illustrates an example: In a prior art photoelectric conversion element, the maximum number of recharge cycles t / T is increased by shortening the recharge cycle T, and the saturation judgment count is set to be less than the maximum number of recharge cycles t / T.

[0092] like Figure 7A As illustrated, the number of photons incident in each recharge cycle can be reduced by shortening the recharge cycle. Therefore, count omissions can be suppressed. Consequently, color changes can be suppressed.

[0093] Figure 7B The diagram illustrates the following example: In existing photoelectric conversion elements, the saturation judgment count is set to be less than the maximum number of recharge cycles t / T by extending the exposure time t. Because... Figure 7B The recharge cycle and Figure 6B The recharge cycles are the same, therefore, as Figure 6B That would result in count omissions during each recharge cycle.

[0094] However, due to the long exposure time, the number of counts tends to saturate in both G pixels with relatively large count omissions and R or B pixels with relatively small count omissions. Therefore, color variations can be suppressed.

[0095] Figure 7C The diagram illustrates the following example: In prior art photoelectric conversion elements, the saturation judgment count is set to be less than the maximum number of recharge cycles t / T by reducing the number of saturation judgment counts. Because... Figure 7C The recharge cycle and Figure 6B The recharge cycles are the same, therefore, as Figure 6B That would result in count omissions during each recharge cycle.

[0096] However, because the saturation detection count is small, saturation is easily achieved in G pixels with relatively large count omissions and R or B pixels with relatively small count omissions. Therefore, color changes can be suppressed.

[0097] For example, when in Figure 7C When the saturation judgment count is set to six, the color is not as... Figure 6B That way, the color changes towards magenta, and all R, G, and B pixels are saturated to six counts and represented as white. Here, when the saturation count is small, the color change is suppressed. However, as from... Figure 7C It is understood that overexposure of white in an image is prone to occur.

[0098] Therefore, in order to Figure 7D To further suppress color changes, it is preferable to reduce the number of photons incident on each pixel of the photoelectric conversion element per unit time by means of an aperture or ND filter.

[0099] Figure 7D This diagram illustrates an example of the decrease in the number of photons incident on each pixel of the photoelectric conversion element per unit time. Preferably, as the number of photons per unit time decreases, the signal level is amplified by means of digital gain or the like.

[0100] Figures 8A to 8D This is a graph illustrating an example of the response characteristics of the photoelectric conversion element according to the first embodiment, and it is a double logarithmic curve with the horizontal axis representing Nph and the vertical axis representing Nct. Figure 8A In the figure, the solid line is a graphical representation of Equation 1, which shows the response characteristics of the photoelectric conversion element using the control signal CLK, and indicates the response characteristics of the photoelectric conversion element in the prior art.

[0101] Figure 8B The dashed lines in the text indicate, for example, Figure 7A Examples of shortening recharge cycles in that way. Figure 8C The chain line indicator in the middle is as follows Figure 7B Examples of extending the exposure time, and Figure 8D dotted line indicators in the middle, such as Figure 7C That would result in response characteristics in examples where the number of saturation judgment counts is small. It can be understood that, relative to... Figure 8B The dotted lines in the middle Figure 8C The chain and Figure 8D The dotted lines in the diagram indicate all response characteristics. Figure 8A The nonlinearity of the response characteristics is mitigated.

[0102] The nonlinearity of the response characteristics occurs due to the influence of counting loss, therefore from Figure 8B , Figure 8C and Figure 8D It can also be understood that color changes are suppressed by setting the saturation judgment count Nsat to be less than the maximum number of recharges t / T.

[0103] According to Equation 1 and Figures 8A to 8D It can be understood that the smaller the ratio of the saturation judgment count Nsat to the maximum number of recharge cycles t / T, the more the nonlinearity of the response characteristics is mitigated. Therefore, it is preferable that the ratio of the saturation judgment count to the maximum number of recharge cycles, determined by dividing the exposure time by the recharge cycle, is small, as this can suppress color changes.

[0104] Specifically, preferably, the number of saturation judgment counts is equal to or less than 1 / 2 of the maximum number of recharges determined by the quotient obtained by dividing the exposure time by the recharge cycle.

[0105] exist Figure 7A In this system, the recharge cycle is changed by altering the frequency of the control signal CLK. In other words, the maximum number of recharges is controlled by changing the frequency of the signal used to control the recharge cycle.

[0106] However, the recharge cycle can be altered by masking a portion of the control signal CLK. In other words, the maximum number of recharges can be controlled by masking a portion of the signal used to control the recharge cycle. In this case, the period of the unmasked control signal CLK corresponds to the recharge cycle T.

[0107] Furthermore, the maximum number of recharges can be controlled by keeping the control signal CLK constantly on (that is, by fixing the logic and changing the recharge period T). In this case, the recharge period T is not determined by the period of the control signal CLK (= zero), but by the period determined by the analog characteristics of the peripheral circuitry of the APD.

[0108] Next, the photoelectric conversion element according to each embodiment will be described with particular focus on the control of the saturation judgment count, which is a characteristic of this embodiment, and the recharge cycle and exposure time used to determine the maximum number of recharges.

[0109] In the photoelectric conversion element described in the first embodiment, the maximum number of recharges (and the number of saturation judgment counts) is fixed regardless of the exposure time, by changing the recharge cycle according to the exposure time. Specifically, the recharge cycle changes proportionally to the exposure time.

[0110] Figure 9 This is a diagram illustrating an example of the relationship between the saturation judgment count Nsat, the exposure time t, and the recharge cycle T in the photoelectric conversion element according to the first embodiment. By keeping the saturation judgment count Nsat and the maximum number of recharges t / T (=Nrch) constant regardless of the exposure time, color changes can be suppressed similarly at any exposure time. Here, it is assumed that Nrch > Nsat.

[0111] As understood from Equation 2, since the correction process used to correct for the effects of count omissions can be standardized, it is preferable to keep the saturation judgment count Nsat and the maximum number of recharges t / T (=Nrch) fixed regardless of the exposure time.

[0112] As in this embodiment, it is preferable to make the saturation judgment count Nsat consistent with the maximum count that can be counted by the counter circuit 211 of the photoelectric conversion element 100, because the circuit size of the counter circuit 211 can be reduced. That is, it is preferable that the saturation judgment count is equal to the maximum count that can be counted by the avalanche photodiode.

[0113] In this embodiment, an example has been described where one of the saturation judgment count, exposure time, and recharge cycle is controlled such that the saturation count is greater than the maximum count. However, the relationship that the saturation count is greater than the maximum count can be achieved by combining the saturation judgment count, exposure time, and recharge cycle.

[0114] In other words, the control unit 801 can control the saturation judgment count, exposure time, and recharge cycle in combination, such that the saturation count is greater than the maximum count. Therefore, the control unit 801 can control at least one of the saturation judgment count, exposure time, and recharge cycle, such that the saturation count is greater than the maximum count.

[0115] As described in this embodiment, as a countermeasure against point light sources and moving light sources, at least one of the saturation judgment count (threshold), exposure time, and recharge cycle is controlled such that the saturation judgment count is greater than the maximum count.

[0116] As described above, since countermeasures can be taken against point light sources and moving light sources, it is preferable to perform the above control when the detection unit (not shown) detects a scene where a point light source or moving light source is present. When the above control is performed when there is no point light source or moving light source, the maximum count may unnecessarily increase. Therefore, there is a possibility that power consumption will increase unnecessarily.

[0117] Therefore, the detection unit (not shown) can detect whether there are light sources with different brightness between frames, or whether a light source with brightness equal to or greater than a predetermined threshold is moving, and when the above scene is detected, the above control can be performed.

[0118] Conversely, when the detection unit does not detect a scene with a point light source or a moving light source, the saturation judgment count is set to be equal to or less than the maximum count. This further reduces power consumption and allows for countermeasures against point light sources or moving light sources.

[0119] Furthermore, as described above, a first mode and a normal mode (second mode) can be prepared, and the mode can be switched in response to a user's command. In the first mode, control is performed to make the saturation judgment count greater than the maximum count, while in the normal mode (second mode), the above control is not performed. In other words, countermeasure modes for point light sources or moving light sources, as well as a normal mode, can be prepared in advance, allowing the user to switch modes arbitrarily.

[0120] Second Embodiment In the photoelectric conversion element according to the second embodiment, the recharge cycle is fixed, and the saturation determination count is changed according to the exposure time. That is, the recharge cycle is fixed regardless of the exposure time.

[0121] As can be understood from Equations 1 and 2, since the maximum number of recharge cycles t / T (=Nrch) is larger, the nonlinear response characteristics caused by count omissions are mitigated, and color changes can be suppressed. Therefore, to suppress color changes, it is preferable to increase the maximum number of recharge cycles t / T (=Nrch). That is, it is preferable to make the recharge period T smaller.

[0122] However, as mentioned above, the minimum value of the recharge period T is not the period of the control signal CLK, and is limited by the analog characteristics of the peripheral circuitry of the APD. Therefore, in order to suppress color changes most efficiently, it is preferable to implement the following embodiment: regardless of the exposure time, the recharge period is fixed to the minimum value T0 determined by the analog characteristics of the peripheral circuitry of the APD.

[0123] In this embodiment, since the maximum number of recharges increases proportionally to the exposure time t, the number of saturation judgment counts can be changed proportionally to the exposure time. That is, regardless of the exposure time, the ratio of the number of saturation judgment counts to the maximum number of recharges can remain equal.

[0124] In other words, the control signal CLK can remain normally ON, or the period of the control signal CLK can be set to a minimum value limited by the analog characteristics of the APD's peripheral circuitry. Here, instead of keeping the control signal CLK normally ON, it is preferable to set the period of the control signal CLK to a minimum value T0 limited by the analog characteristics of the APD's peripheral circuitry, because the number of avalanche multiplication periods during high brightness can be appropriately detected.

[0125] Figure 10 This is a diagram illustrating an example of the relationship between the saturation judgment count Nsat, exposure time t, and recharge cycle T in a photoelectric conversion element according to the second embodiment. (As shown from...) Figure 10 It is understood that in the photoelectric conversion element according to the second embodiment, the longer the exposure time, the greater the number of saturation judgment counts.

[0126] In other words, a longer exposure time results in a larger dynamic range. Due to the influence of photon shot noise, a longer exposure time can result in a higher image SN ratio.

[0127] Therefore, in the photoelectric conversion device using the photoelectric conversion element according to the second embodiment, it is preferable to perform exposure control so that the exposure time of the photoelectric conversion element can be made as long as possible by using an aperture or an ND filter to reduce light.

[0128] As understood from Equation 2, in the photoelectric conversion element according to the second embodiment, the saturation determination count Nsat and the maximum number of recharges t / T change with the exposure time. Therefore, it is preferable to change the correction processing used to correct for the effects of count omissions according to the exposure time.

[0129] Third Embodiment In the photoelectric conversion element according to the second embodiment, a long exposure time results in a high dynamic range or signal-to-noise ratio (SN ratio) of the image. However, a short exposure time can result in a low dynamic range or SN ratio. Therefore, in the third embodiment, a long exposure time allows color variation over the image's dynamic range or SN ratio.

[0130] Figure 11 This is a diagram illustrating an example of the relationship between the saturation judgment count Nsat, exposure time t, and recharge cycle T in a photoelectric conversion element according to the third embodiment. (As shown from...) Figure 11As understood, in the photoelectric conversion element according to the third embodiment, the saturation judgment count Nsat is fixed regardless of the exposure time t. Therefore, the decrease in dynamic range or SN ratio during short exposure periods, which occurs in the photoelectric conversion element according to the second embodiment, can be suppressed.

[0131] On the other hand, since the recharge cycle T is fixed regardless of the exposure time t, the longer the exposure time t, the greater the maximum number of recharges t / T(Nrch). The shorter the exposure time t, the smaller the maximum number of recharges t / T(Nrch).

[0132] Therefore, as the exposure time t increases, the ratio of the saturation judgment count Nsat to the maximum number of recharges t / T decreases, thereby suppressing color changes. Thus, in the third embodiment, control is implemented such that the longer the exposure time, the smaller the ratio of the saturation judgment count to the maximum number of recharges.

[0133] In a photoelectric conversion device using a photoelectric conversion element according to the third embodiment, it is preferable to perform exposure control so that the exposure time of the photoelectric conversion element is as long as possible by reducing light using an aperture or an ND filter.

[0134] Figure 11 This example illustrates the case where all exposure times, including the shortest exposure time t0, satisfy Equation 3. However, it is possible to implement a configuration where Equation 3 is not satisfied when the exposure time is short. That is, when the exposure time is less than a predetermined threshold, the saturation judgment count Nsat can be equal to or greater than the maximum number of recharges t / T.

[0135] As understood from Equation 2, in the photoelectric conversion element according to the third embodiment, the maximum number of recharge cycles t / T varies depending on the exposure time. Therefore, it is preferable to change the correction process used to correct for the effects of count omissions according to the exposure time.

[0136] Fourth embodiment The photoelectric conversion element according to the fourth embodiment has two driving modes, namely, a first driving mode for suppressing color changes and a second driving mode for suppressing the reduction of dynamic range or SN ratio.

[0137] As the first driving mode for suppressing color changes, the driving mode of the first embodiment is preferably used. By keeping the saturation judgment count Nsat1 and the maximum number of recharges Nrch1 fixed regardless of the exposure time, color changes can be suppressed similarly at any exposure time. That is, in the first driving mode, the saturation judgment count is set to be less than the maximum number of recharges regardless of the exposure time.

[0138] On the other hand, in the second drive mode used to suppress the decrease in dynamic range or SN ratio, it is preferable to set the saturation judgment count Nsat2 to be equal to or greater than the maximum number of recharges Nrch2, regardless of the exposure time.

[0139] It is preferable to set Nsat2=Nrch2. That is, when Nsat2=Nrch2 is set regardless of the exposure time, the decrease in the SN ratio can be suppressed to the greatest extent at any exposure time. Therefore, it is preferable to set Nsat2=Nrch2.

[0140] At this point, it is preferable to make Nsat1 and Nsat2 consistent with the maximum number of counts that can be counted by the counter circuit 211 of the photoelectric conversion element 100, because the circuit size can be reduced. That is, it is preferable to set the saturation judgment count Nsat1 in the first driving mode to be equal to the saturation judgment count Nsat2 in the second driving mode.

[0141] Preferably, the maximum number of recharges Nrch1 in the first drive mode is set to be greater than the maximum number of recharges Nrch2 in the second drive mode.

[0142] The dynamic range or SN ratio of the image differs in the first driving mode used to suppress color changes and the second driving mode used to suppress the reduction in dynamic range or SN ratio. Therefore, the parameters of signal processing can be changed.

[0143] For example, in the second drive mode, the dynamic range is wide. Therefore, the gamma curve may tend to be flatter. On the other hand, since the signal density (SN) is relatively low in the first drive mode, noise reduction processing can be enhanced.

[0144] Fifth Embodiment Figure 12 This is a functional block diagram illustrating an example configuration of a photoelectric conversion device using a photoelectric conversion element according to the first to fourth embodiments. Figure 12 Some of the illustrated functional blocks are implemented by causing the CPU or the like, which is used as a computer in a photoelectric conversion device, to execute computer programs stored in a memory that serves as a storage medium.

[0145] However, some or all of these functional blocks can be implemented in hardware. As hardware, this can be achieved using application-specific integrated circuits (ASICs) or processors (reconfigurable processors or DSPs).

[0146] Figure 12The illustrated functional blocks may not be embedded in the same housing, or may be composed of different devices connected to each other via signal lines. The photoelectric conversion device 800 includes a photoelectric conversion element 100, a control unit 801, a storage unit 802, a communication unit 803, a signal processing unit 804, and an imaging optical system 805.

[0147] The photoelectric conversion element 100 captures an optical image formed by the imaging optical system 301. The signal read from the photoelectric conversion element 100 is supplied to the signal processing unit 804. The signal processing unit 804 performs processes such as black level correction, gamma curve adjustment, noise reduction, data compression, white balance correction, and color conversion.

[0148] The signal processing unit 804 uses Equation 2 above to perform nonlinear correction on the quantity Nct of the output signal of the counter circuit 211. Therefore, the final image is generated. In this configuration, even in environments where flickering light sources are used for illumination, a photoelectric conversion element capable of suppressing color changes can be implemented.

[0149] The control unit 801 includes a CPU 806 that functions as a computer and acts as a control unit for controlling the operation of each unit in the entire photoelectric conversion device 800 based on a computer program stored in a memory 807 that serves as a storage medium.

[0150] The control unit 801 controls the length of the exposure cycle of each frame of the photoelectric conversion element 100 and the timing of the control signal CLK through the control pulse generation unit 115 of the photoelectric conversion element 100.

[0151] The control unit 801 controls the exposure time of the avalanche photodiode and the recharge cycle of the avalanche photodiode. Furthermore, as described above, the control unit 801 controls the process such that the number of saturation judgment counts is less than the maximum number of recharges determined by the quotient obtained by dividing the exposure time by the recharge cycle.

[0152] Storage unit 802 includes, for example, a recording medium such as a memory card or a hard disk. Communication unit 803 includes a wireless or wired interface and outputs the generated image to the outside of photoelectric conversion device 800, and also receives signals from the outside.

[0153] While this disclosure has been described with reference to embodiments, it should be understood that this disclosure is not limited to the disclosed embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

[0154] Furthermore, as part or all of the control according to the embodiments, a computer program that implements the functions of the above embodiments can be supplied to a photoelectric conversion device, etc., via a network or various storage media. Then, the computer (or CPU or MPU, etc.) of the photoelectric conversion device, etc., can be configured to read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention.

[0155] Additionally, this disclosure includes those implemented using at least one processor or circuitry configured to perform the functions of the above embodiments. For example, multiple processors may be used for distributed processing to perform the functions of the above embodiments.

[0156] This application claims the benefit of Japanese Patent Application 2024-224645, filed on December 20, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. A photoelectric conversion device, comprising: A photoelectric conversion element includes an avalanche photodiode for outputting a signal based on incident photons, and a counter for counting the number of output signals from the avalanche photodiode and outputting the number of output signals. A judgment unit is used to determine whether the number of the output signals is equal to or greater than a threshold. as well as A control unit is configured to control the exposure time of the avalanche photodiode and the recharge cycle of the avalanche photodiode during the exposure time. The control unit controls at least one of the threshold, the exposure time, and the recharge cycle, such that the maximum number of recharges determined based on the exposure time and the recharge cycle is greater than the threshold.

2. The photoelectric conversion device according to claim 1, wherein, The control unit controls at least one of the threshold, the exposure time, and the recharge cycle such that the maximum number of recharges, determined by the quotient obtained by dividing the exposure time by the recharge cycle, is greater than the threshold.

3. The photoelectric conversion device according to claim 1, wherein, If the detection unit detects a flickering light source in an image generated from the output signal of the avalanche photodiode, the control unit controls the system to make the maximum number of recharges greater than the threshold.

4. The photoelectric conversion device according to claim 1, further comprising: A switch, connected to one of the nodes of the anode and cathode of the avalanche photodiode and a power line to which a driving voltage is applied, is configured to control the recharge cycle by switching the resistance value between the node and the power line.

5. The photoelectric conversion device according to claim 4, wherein, The maximum number of recharge cycles is controlled by logic that fixes the switch.

6. The photoelectric conversion device according to claim 1, wherein, The threshold is equal to or less than 1 / 2 of the maximum number of recharges.

7. The photoelectric conversion device according to claim 1, wherein, The control unit controls the maximum number of recharge cycles by changing the frequency of the signal used to control the recharge cycle.

8. The photoelectric conversion device according to claim 1, wherein, The control unit controls the maximum number of recharge cycles by shielding a portion of the signal used to control the recharge cycle.

9. The photoelectric conversion device according to claim 1, wherein, The control unit changes the recharge cycle according to the exposure time, so that the maximum number of recharges remains constant regardless of the exposure time.

10. The photoelectric conversion device according to claim 9, wherein, The control unit keeps the maximum number of recharge cycles constant by changing the recharge cycle proportionally to the exposure time, regardless of the exposure time.

11. The photoelectric conversion device according to claim 1, wherein, The control unit keeps the threshold constant regardless of the exposure time.

12. The photoelectric conversion device according to claim 11, wherein, The control unit sets the threshold to be equal to the maximum number of counts that the avalanche photodiode can count.

13. The photoelectric conversion device according to claim 1, wherein, The control unit keeps the recharge cycle constant regardless of the exposure time.

14. The photoelectric conversion device according to claim 13, wherein, Regardless of the exposure time, the control unit sets the ratio of the threshold to the maximum number of recharges to be equal.

15. The photoelectric conversion device according to claim 14, wherein, As the exposure time becomes longer, the control unit sets the ratio of the threshold to the maximum number of recharges to decrease.

16. The photoelectric conversion device according to claim 15, wherein, If the exposure time is less than a certain threshold, the control unit sets the threshold to be equal to or greater than the maximum number of recharges.

17. The photoelectric conversion device according to claim 1, wherein, The control unit has a first driving mode and a second driving mode. In the first driving mode, the threshold is less than the maximum number of recharges regardless of the exposure time. In the second driving mode, the threshold is equal to or greater than the maximum number of recharges regardless of the exposure time.

18. The photoelectric conversion device according to claim 17, wherein, The control unit sets the threshold of the first driving mode to be equal to the threshold of the second driving mode.

19. The photoelectric conversion device according to claim 17, wherein, The control unit sets the maximum number of recharge cycles for the first drive mode to be greater than the maximum number of recharge cycles for the second drive mode.

20. The photoelectric conversion device according to claim 1, wherein, The avalanche photodiode includes an on-chip color filter.

21. The photoelectric conversion device according to claim 1, further comprising: The signal processing unit is used to perform nonlinear correction on the number Nct of the output signal of the counter using the following formula, where the length of the exposure time is t, the recharge cycle is T, and the number of input signals of the counter is Nph. 。 22. A photoelectric conversion method for controlling a photoelectric conversion element, the photoelectric conversion element comprising an avalanche photodiode for outputting a signal based on incident photons, and a counter for counting and outputting the number of output signals from the avalanche photodiode, the photoelectric conversion method comprising: Determine whether the number of output signals is equal to or greater than a threshold; Control the exposure time of the avalanche photodiode and the recharge cycle of the avalanche photodiode during the exposure time; as well as Control at least one of the threshold, the exposure time, and the recharge cycle such that the maximum number of recharges determined based on the exposure time and the recharge cycle is greater than the threshold.

23. A computer program product comprising a computer program for causing a computer to perform the photoelectric conversion method according to claim 22.

24. A computer-readable storage medium storing a computer program for causing a computer to perform the photoelectric conversion method according to claim 22.

Images