Photoelectric conversion device, photoelectric conversion method, and computer program

Abstract

This invention provides a photoelectric converter that can suppress changes in color even in environments where flashing light sources or light sources moving within a frame are used as illumination. [Solution] The photoelectric conversion device includes an avalanche photodiode, a counter that counts and outputs the number of output signals from the avalanche photodiode until it reaches a saturation determination count, and a control unit that controls the exposure time of the avalanche photodiode and the recharge period of the avalanche photodiode, and sets the ratio of the saturation determination count to the maximum recharge number, which is determined by the quotient obtained by dividing the exposure time by the recharge period, as the first ratio in the first frame, and controls the ratio of the saturation determination count to the maximum recharge number to be a second ratio which is smaller than the first ratio in the second frame which is different from the first frame.

Description

Technical Field

[0001] The present invention relates to a photoelectric conversion device, a photoelectric conversion method, a computer program, and the like.

Background Art

[0002] In recent years, a photoelectric conversion element has been proposed that digitally counts the number of photons arriving at an avalanche photodiode (APD) and outputs the count value from a pixel as a digitally converted photoelectric signal.

[0003] Patent Document 1 describes a photoelectric conversion device in which the timing for charging an APD is controlled by a pulse signal so that the number of periods during which avalanche multiplication occurs can be suitably detected.

[0004] In the photoelectric conversion device of Patent Document 1, photons can be counted one by one for each period of the pulse generated by the pulse generation circuit. When this method is used, count leakage occurs when a plurality of photons enter during one period of the pulse.

[0005] However, when shooting general steady light, by statistically predicting the number of photons incident in one period of the pulse, the influence of count leakage can be reduced by correction processing.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] On the other hand, when using light sources that blink within a single frame, particularly LED light sources using pulse modulation drive methods, or light sources that move within a frame, many count omissions occur compared to when using steady light. As a result, the accuracy of the count omission correction process decreases, and the color appears different from that of the actual environment.

[0008] Therefore, one of the objectives of the present invention is to provide a photoelectric conversion device that can suppress changes in color even in environments where flashing light sources or light sources moving within a frame are used as illumination. [Means for solving the problem]

[0009] One aspect of the present invention is a photoelectric conversion device, Avalanche photodiode and A counter that counts and outputs the number of output signals from the avalanche photodiode until it reaches a saturation count, The exposure time of the avalanche photodiode is controlled, and the recharge period of the avalanche photodiode is controlled. A control unit controls the first frame to have a first ratio, which is the ratio of the saturation determination count to the maximum recharge count determined by the quotient obtained by dividing the exposure time by the recharge cycle, and the second frame, which is different from the first frame, to have a second ratio, which is smaller than the first ratio. It is characterized by having the following features. [Effects of the Invention]

[0010] According to the present invention, a photoelectric conversion device can be realized that can suppress changes in color even in environments where flashing light sources or light sources moving within a frame are used as illumination light. [Brief explanation of the drawing]

[0011] [Figure 1]This figure shows an example of the configuration of a photoelectric conversion element 100 according to Embodiment 1 of the present invention. [Figure 2] This figure shows an example configuration of the sensor substrate 11 according to Embodiment 1. [Figure 3] This figure shows an example configuration of the circuit board 21 according to Embodiment 1. [Figure 4] This figure shows an example of an equivalent circuit of a pixel 101 and a signal processing circuit 103 corresponding to the pixel 101 according to Embodiment 1. [Figure 5] This is a timing chart of the photoelectric conversion element for illustrating Embodiment 1. [Figure 6] (A) is a diagram showing an example of the pixel arrangement of the photoelectric conversion element of Embodiment 1, and (B) is a diagram showing an example of the drive timing chart of the photoelectric conversion element of Embodiment 1. [Figure 7A] This diagram illustrates the counting operation when using continuous light as illumination. [Figure 7B] This diagram illustrates the counting operation when a time-blinking light source is used as illumination. [Figure 7C] This diagram illustrates the illumination intensity of a region of interest when there is a light source moving within a single frame. [Figure 8A] This figure shows an example where reducing the saturation detection count makes the second ratio (saturation detection count divided by the maximum recharge count) smaller than the first ratio. [Figure 8B] This figure shows an example where shortening the recharge cycle makes the second ratio (saturation detection count divided by the maximum recharge count) smaller than the first ratio. [Figure 8C] This figure shows an example where increasing the exposure time resulted in a second ratio (saturation detection count divided by the maximum recharge count) being smaller than the first ratio. [Figure 9] Figures (A) and (B) represent the pixel response characteristics of the first and second frames, respectively. [Figure 10] (A) is a figure showing an example of the response characteristics of pixels in the first frame, and (B) is a figure showing an example of the response characteristics of pixels in the second frame. [Figure 11] It is a functional block diagram showing a configuration example of a photoelectric conversion device using the photoelectric conversion element of Embodiments 1 to 4.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each figure, the same members or elements are given the same reference numerals, and duplicate explanations are omitted or simplified.

[0013] <Embodiment 1> FIG. 1 is a diagram showing a configuration example of a photoelectric conversion element 100 according to Embodiment 1 of the present invention. The sensor substrate 11 includes a pixel region 12, and the circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0014] In addition, in the present embodiment, an example of a so-called stacked structure photoelectric conversion device in which the photoelectric conversion element 100 is configured by stacking and electrically connecting two substrates, namely, the sensor substrate 11 and the circuit substrate 21, will be described. However, a so-called non-stacked structure in which the configuration included in the sensor substrate and the configuration included in the circuit substrate are arranged in a common semiconductor layer may also be used.

[0015] FIG. 2 is a diagram showing a configuration example of the sensor substrate 11 according to Embodiment 1. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally in a plurality of rows and columns. The pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, APD).

[0016] FIG. 3 is a diagram showing a configuration example of the circuit substrate 21 according to Embodiment 1. The circuit substrate 21 has a signal processing circuit 103 that processes charges photoelectrically converted by the photoelectric conversion unit 102 of FIG. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, signal lines 113, and a vertical scanning circuit 110.

[0017] The vertical scanning circuit 110 receives control pulses supplied from the control pulse generation unit 115 and supplies control pulses to each pixel. Logic circuits such as a shift register and an address decoder are used in the vertical scanning circuit 110.

[0018] The signal output from the photoelectric conversion unit 102 of each pixel is processed by the signal processing circuit 103. The signal processing circuit 103 is equipped with a counter and memory, and digital values ​​are stored in the memory. The horizontal scanning circuit 111 inputs control pulses to the signal processing circuit 103 to sequentially select each column in order to read the signal from the memory of each pixel in which the digital signal is stored.

[0019] Signals are output to signal line 113 from the signal processing circuit 103 of the pixels selected by the vertical scanning circuit 110 for the selected column. The signals output to signal line 113 are then output to the outside of the photoelectric conversion element 100 via the output circuit 114.

[0020] As shown in Figures 2 and 3, multiple signal processing circuits 103 are arranged in the region that overlaps with the pixel region 12 in a plan view. Then, a vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generation unit 115 are arranged so as to overlap between the edge of the sensor substrate 11 and the edge of the pixel region 12 in a plan view.

[0021] In other words, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generation unit 115 are arranged in the region that overlaps with the non-pixel region in a plan view.

[0022] Note that the arrangement of the signal line 113, the read circuit 112, and the output circuit 114 is not limited to Figure 3. For example, the signal line 113 may be arranged extending in the row direction, and the read circuit 112 may be placed at the end of the signal line 113.

[0023] Furthermore, the signal processing unit does not necessarily need to be provided in one unit for each photoelectric conversion unit; a single signal processing unit may be shared by multiple photoelectric conversion units, and the signal processing may be performed sequentially.

[0024] Figure 4 shows an example of an equivalent circuit of a pixel 101 and a signal processing circuit 103 corresponding to the pixel 101 according to Embodiment 1.

[0025] The APD201 generates charge pairs corresponding to incident light through photoelectric conversion. One of the two nodes of the APD201 is connected to a power line that supplies the drive voltage VL (first voltage).

[0026] Furthermore, the other of the two nodes of the APD201 can be connected via switch 202 to a power line supplied with a drive voltage VH (second voltage) higher than voltage VL. In Figure 4, one node of the APD201 is the anode, and the other node of the APD is the cathode.

[0027] A reverse bias voltage is supplied to the anode and cathode of the APD201, causing it to perform avalanche multiplication. This voltage supply causes the charge generated by the incident light to undergo avalanche multiplication, resulting in the generation of an avalanche current.

[0028] Furthermore, when a reverse bias voltage is supplied, there are two modes of operation: Geiger mode, where the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, where the voltage difference between the anode and cathode is near or below the breakdown voltage. An APD operating in Geiger mode is called a SPAD. In the case of a SPAD, for example, the voltage VL (first voltage) is -30V and the voltage VH (second voltage) is 1V.

[0029] Switch 202 is connected to the power line to which the drive voltage VH is supplied and to one of the nodes, either the anode or the cathode, of the APD201. Switch 202 then switches the resistance value between the APD201 and the power line to which the drive voltage VH is supplied.

[0030] Specifically, switch 202 is connected to one of the anodes or cathodes of the avalanche photodiode and to a power line to which the drive voltage is applied, and switches the resistance value between the node and the power line. Switch 202 is also used to control the recharge period.

[0031] Here, switching the resistance value means that it is desirable to change the resistance value by 10 times or more, and more desirable to change the resistance value by 100 times or more. In the following, a decrease in the resistance value will be referred to as turning switch 202 on, and a increase in the resistance value will be referred to as turning switch 202 off.

[0032] Switch 202 functions as a quench element. Specifically, when the signal is amplified by avalanche multiplication, switch 202 acts as a load circuit (quench circuit), suppressing the voltage supplied to APD201 and performing a quench operation to suppress avalanche multiplication.

[0033] Furthermore, switch 202 performs a recharge operation to return the voltage supplied to APD201 to the drive voltage VH by supplying current to compensate for the voltage drop caused by the quench operation. Switch 202 can be constructed using a MOS transistor, and Figure 4 shows the case where switch 202 is a PMOS transistor.

[0034] The control signal CLK for controlling switch 202 is applied from the control pulse generation unit 115, which acts as a signal generation unit, to the gate electrode of the MOS transistor that constitutes switch 202.

[0035] The control pulse generation unit 115 controls the recharge period of the avalanche photodiode by controlling the voltage applied to the gate electrode of the switch 202, thereby controlling the on and off states of the switch 202. Furthermore, the exposure time (count time) of the avalanche photodiode is controlled by the control unit 801, which will be described later, as an exposure time control unit.

[0036] The signal processing circuit 103 includes a waveform shaping unit 210, a counter circuit 211, and a selection circuit 212. However, the signal processing circuit 103 only needs to have at least one of the waveform shaping unit 210, the counter circuit 211, and the selection circuit 212.

[0037] The waveform shaping unit 210 shapes the cathode voltage change of the APD201 obtained during photon detection and outputs a pulse signal. For example, an inverter circuit can be used as the waveform shaping unit 210. The inverter circuit may be a circuit in which multiple inverters are connected in series, or other circuits that have a waveform shaping effect may be used.

[0038] The counter circuit 211 counts the pulse signal output from the waveform shaping unit 210 and holds the count value. When the control pulse RES is supplied via the drive line 213, the signal held by the counter circuit 211 is reset.

[0039] A saturation determination unit 215 is connected to the counter circuit 211, which determines when the number of output signals measured by the counter circuit has reached saturation. When the number of output signals measured by the counter circuit 211 reaches the saturation determination count, the saturation determination unit 215 stops the count-up of the pulse signal by the counter circuit and maintains the saturation determination count as the count value in the counter circuit 211. The saturation determination count can be set as a predetermined threshold.

[0040] In this embodiment, the quotient obtained by dividing the exposure time by the recharge cycle is called the maximum recharge count. Furthermore, in the first frame, the ratio of the saturation determination count to the maximum recharge count is defined as the first ratio, and in the second frame, which is different from the first frame, the ratio of the saturation determination count to the maximum recharge count is controlled to be a second ratio, which is smaller than the first ratio.

[0041] Specifically, the control unit 801 controls at least one of the recharge cycle, exposure time, and saturation determination count so that the second ratio is smaller than the first ratio. To control the saturation determination count for each frame, the control pulse generation unit 115 can change the count used for saturation determination using a signal line (not shown).

[0042] The selection circuit 212 receives a control pulse SEL from the vertical scanning circuit 110 in Figure 3 via the drive line 214 (not shown in Figure 3) in Figure 4, which switches the electrical connection between the counter circuit 211 and the signal line 113. The selection circuit 212 includes, for example, a buffer circuit for outputting a signal.

[0043] The output signal OUT shown in Figure 4 is the output signal as a pixel output, and the count value of the counter circuit 211 is output. In other words, the counter circuit 211 functions as a counter that counts and outputs the number of output signals from the avalanche photodiode until it reaches the saturation judgment count.

[0044] Furthermore, switches such as transistors may be placed between the switch 202 (which acts as a quench element) and the APD201, or between the photoelectric conversion unit 102 and the signal processing circuit 103, to switch the electrical connections. Similarly, the supply of voltage VH or voltage VL to the photoelectric conversion unit 102 may be electrically switched using switches such as transistors.

[0045] Furthermore, while it is possible to perform quench and recharge operations using switch 202 in response to avalanche multiplication in APD201, depending on the timing of photon detection, these may not be recognized as output signals.

[0046] For example, consider a scenario where avalanche multiplication occurs in the APD, causing the input voltage to nodeA to become low, and a recharge operation is performed. In this case, the judgment threshold of the waveform shaping unit 210 is set to a voltage higher than the voltage difference at which avalanche multiplication occurs in the APD.

[0047] When a photon is incident while the voltage of node A is below the threshold due to the recharge operation and the voltage is within the range where avalanche multiplication is possible at the APD, avalanche multiplication occurs at the APD and the voltage of node A decreases.

[0048] In other words, because the voltage at nodeA drops below the detection threshold, the output voltage from nodeB does not change despite the detection of photons. Consequently, even though avalanche multiplication is occurring, it is no longer recognized as a signal.

[0049] In particular, under high illumination conditions, photons enter the field in rapid succession over a short period, making it difficult to recognize them as a signal. As a result, even under high illumination conditions, there is a tendency for the actual number of incident photons to deviate from the output signal.

[0050] In contrast, in this embodiment, by applying the control signal CLK to the switch 202 and switching the switch 202 on and off, it becomes possible to determine that photons are entering the APD in succession in a short period of time as a signal.

[0051] Figure 5 is a timing chart of a photoelectric conversion element to illustrate Embodiment 1, schematically showing the relationship between the switch control signal CLK, the voltage at node A, the voltage at node B, and the output signal. Note that Figure 5 shows the case where the control signal CLK is a pulse signal with a constant repetition period.

[0052] In the photoelectric converter of this embodiment, when the control signal CLK is at a high level, the drive voltage VH is less likely to be supplied to the APD, and when the control signal CLK is at a low level, the drive voltage VH is supplied to the APD. A high level control signal CLK is, for example, 1V, and a low level control signal CLK is, for example, 0V.

[0053] Switch 202 is turned off when the control signal CLK is at a high level, and turned on when the control signal CLK is at a low level. The resistance of switch 202 when the control signal CLK is at a high level is higher than the resistance of switch 202 when the control signal CLK is at a low level.

[0054] When the control signal CLK is at a high level, even if avalanche multiplication occurs in the APD, recharging is less likely to occur, resulting in the voltage supplied to the APD being below the APD's breakdown voltage. Consequently, the avalanche multiplication operation in the APD stops.

[0055] At time t1, the control signal CLK changes from a high level to a low level, switching 202 on and initiating the APD recharge operation. This causes the APD cathode voltage to transition to a high level.

[0056] Then, the voltage difference between the voltages applied to the anode and cathode of the APD becomes a state where avalanche multiplication is possible. The cathode voltage is the same as that of node nodeA. Therefore, when the cathode voltage transitions from a low level to a high level, the voltage of node nodeA becomes greater than or equal to the threshold at time t2.

[0057] At this time, the pulse signal output from nodeB inverts, changing from a high level to a low level. Once recharging is complete, the APD201 is subjected to a voltage difference between the drive voltage VH and the drive voltage VL. Subsequently, the control signal CLK becomes high, and switch 202 turns off.

[0058] Next, at time t3, when a photon is incident on APD201, avalanche multiplication occurs in APD201, an avalanche multiplication current flows through switch 202, and the cathode voltage drops. In other words, the voltage at node A drops.

[0059] If the voltage at nodeA falls below the threshold during the voltage drop, the voltage at nodeB changes from a low level to a high level. In other words, the portion of the output waveform at nodeA that exceeds the threshold is shaped by the waveform shaping unit 210 and output as a signal at nodeB. This is then counted by the counter circuit, and the count value of the counter signal output from the counter circuit increases by 1 LSB.

[0060] Although photons are incident on the APD between time t3 and time t4, switch 202 is in the off state, and the voltage applied to APD201 is not a voltage difference that allows for avalanche multiplication, so the voltage level of node A does not exceed the judgment threshold.

[0061] At time t4, the control signal CLK changes from high level to low level, and switch 202 turns on. Consequently, a current flows through node A to compensate for the voltage drop from the drive voltage VL, and the voltage at node A returns to its original voltage level. At this time, at time t5, the voltage at node A exceeds the judgment threshold, so the pulse signal at node B inverts, changing from high level to low level.

[0062] At time t6, node A settles back to its original voltage level, and the control signal CLK changes from a low level to a high level. Subsequently, as explained from time t1 to time t6, the voltages of each node and signal lines change in response to the control signal CLK and the incidence of photons. In this way, the recharge frequency of the APD can be controlled by applying the control signal CLK to switch 202 and switching it on and off.

[0063] If the control signal CLK is not used, there is a problem in that the actual count value becomes smaller than the count value corresponding to the brightness of the incident light at high brightness. However, this problem can be solved by applying the control signal CLK to switch 202 and switching switch 202 on and off.

[0064] The recharge cycle corresponds to the time from t1, when the recharge operation begins, to t4, when the next recharge operation begins. Furthermore, the length of the period during which the recharge frequency is controlled by the control signal CLK (from t1 to tN in Figure 5) corresponds to the exposure time.

[0065] However, when controlling the APD recharge frequency with the control signal CLK, counting errors occur if multiple photons are incident in one pulse period. Therefore, the relationship between the number of output signals and the number of input signals is not linear.

[0066] When photographing general continuous light, the relationship between the number of input signals and the number of output signals can be theoretically derived from the statistical distribution of the number of incident photons. Specifically, when the number of input signals is Nph, the number of output signals is Nct, the recharge period of the control signal CLK is T, and the exposure time is t, the following equation 1 holds.

[0067]

number

[0068] Therefore, in this embodiment, the number of photons incident in one period of the pulse is statistically predicted from the number of output signals Nct using the following Equation 2. This allows the effects of count omissions to be recovered by the correction process in the signal processing unit 804 described later. That is, the nonlinearity of the number of output signals Nct of the counter circuit can be corrected by performing the process in Equation 2 in the signal processing unit 804.

[0069]

number

[0070] On the other hand, in environments where a light source that blinks within a single frame, particularly an LED light source using a pulse modulation driving method, is used as illumination, many count omissions occur compared to when using steady light. As a result, the accuracy of the count omission correction process decreases, and the color appears different from that of the actual environment.

[0071] Therefore, in this embodiment, if the ratio obtained by dividing the saturation detection count by the maximum recharge count in the first frame is the first ratio, then in the second frame, which is different from the first frame, the ratio obtained by dividing the saturation detection count by the maximum recharge count is controlled to be the second ratio. Moreover, the second ratio is controlled to be smaller than the first ratio.

[0072] This makes it possible to provide a photoelectric conversion element that can suppress both color changes and overexposure, even in environments where a flickering light source is used as illumination.

[0073] Figure 6(A) shows an example of the pixel arrangement of the photoelectric conversion element of Embodiment 1. As shown in Figure 6, the photoelectric conversion element of this embodiment is equipped with an on-chip color filter for each pixel having an avalanche photodiode, and the on-chip color filters are arranged in a so-called RGB Bayer array.

[0074] However, it is also acceptable to have pixels other than R, G, and B, such as Y (yellow) pixels, and to have pixels without an on-chip color filter (W pixels). Furthermore, the on-chip color filter does not have to be a Bayer array.

[0075] Figure 6(B) shows an example of a drive timing chart for the photoelectric conversion element of Embodiment 1. As shown in Figure 6(B), in this embodiment, the element is driven so that a first frame, in which the ratio obtained by dividing the saturation determination count by the maximum recharge count is the first ratio, and a second frame, in which the ratio obtained by dividing the saturation determination count by the maximum recharge count is the second ratio, alternate.

[0076] However, the frames do not necessarily have to alternate one frame at a time; either the first or second frame may appear consecutively for several frames, or the first or second frame may alternate every few frames. In other words, the first and second frames just need to alternate periodically.

[0077] Figure 7A illustrates the counting operation when steady light is used as illumination, and Figure 7B illustrates the counting operation when the averaged illumination intensity is the same as in Figure 7A, but the light source is blinking over time. In the following, we will explain using the example of using white light as illumination, where the number of photons incident on the R pixel:G pixel:B pixel per unit time is 1:2:1.

[0078] As shown in Figure 7A, when a steady light source is incident, the number of photons incident in each recharge period is small, so there are few missed counts in each recharge period. On the other hand, as shown in Figure 6(B), when a blinking light source is used as illumination, photons are concentrated and incident in specific recharge periods, resulting in many missed counts.

[0079] When using illumination light with different numbers of photons per unit time incident on the red, green, and blue pixels, a significant number of count misses are expected in pixels of a particular color. This disrupts the balance of the counts of the red, green, and blue pixels, causing a change in color. For example, in the example in Figure 7B, a significant number of count misses in the green pixels result in a shift towards magenta.

[0080] While Figure 7B shows the case where a blinking light source is used as illumination, counting errors also occur when a moving light source is used. Figure 7C illustrates the illumination intensity of the area of ​​interest when there is a moving light source within one frame.

[0081] Figure 7C shows that within a single frame, there are moments of high illumination intensity and moments of low illumination intensity in the region of interest. Therefore, similar to the case where a flickering light source is used as illumination, photons are concentrated and incident during specific recharge cycles, resulting in many missed counts.

[0082] On the other hand, in this embodiment, the photoelectric conversion element is controlled such that, if the ratio obtained by dividing the saturation determination count by the maximum recharge count in the first frame is the first ratio, then in the second frame, the ratio obtained by dividing the saturation determination count by the maximum recharge count becomes the second ratio. Note that the second frame is a different frame from the first frame, and the second ratio is made smaller than the first ratio. This makes it possible to suppress both color changes and blown-out highlights.

[0083] In the following explanation, we will use the example where the first ratio, obtained by dividing the saturation detection count in the first frame by the maximum recharge count, is equal to the ratio obtained by dividing the saturation detection count of a conventional photoelectric conversion element by the maximum recharge count.

[0084] In this case, the saturation detection count is set to, for example, 12 counts. In this case, the photons and count values ​​incident on the first frame when a blinking light source is used as illumination light are the same as in Figure 7B.

[0085] On the other hand, in this embodiment, the second ratio obtained by dividing the saturation determination count of the second frame by the maximum recharge count is made smaller than the ratio obtained by dividing the saturation determination count of a conventional photoelectric conversion element by the maximum recharge count. That is, at least one of the saturation determination count or the maximum recharge count of the second frame is different from that of the first frame. A specific example is shown below.

[0086] Figure 8A shows an example where the saturation detection count is reduced, making the second ratio (saturation detection count divided by the maximum recharge count) smaller than the first ratio. Since the recharge cycle in Figure 8A is the same as in Figure 7B, count omissions in each recharge cycle occur in the same way as in Figure 7B.

[0087] However, since the saturation detection count is suppressed to, for example, 6 in Figure 8A, both G pixels, which have a relatively large number of missed counts, and R and B pixels, which have a relatively small number of missed counts, are more likely to reach saturation, thus suppressing changes in color.

[0088] In other words, for example, if the saturation detection count is set to 6 in Figure 8A, the color does not change towards magenta as shown in Figure 7B, and all R, G, and B pixels are saturated, i.e., represented as white.

[0089] In this way, by reducing the saturation detection count, color changes can be suppressed. Specifically, the saturation detection count in the second frame is set to be smaller than that in the first frame, so the color changes are more suppressed in the count obtained in the second frame than in the first frame.

[0090] On the other hand, as can be seen by comparing Figure 7B and Figure 8A, if the saturation detection count is set to, for example, 6 counts as in Figure 8A, the count value obtained in the second frame is more prone to image clipping than the count value obtained in the first frame. In other words, image clipping is suppressed more effectively with the count value obtained in the first frame than with the count value obtained in the second frame.

[0091] Figures 8B and 8C show examples where increasing the maximum recharge count makes the second ratio (saturation detection count divided by the maximum recharge count) smaller than the first ratio. In other words, these are examples where the maximum recharge count of the second frame is greater than the maximum recharge count of the first frame. Note that in Figures 8B and 8C, the saturation detection count is set to 12, for example, as in Figures 7A and 7B.

[0092] Since the maximum number of recharges is calculated by dividing the exposure time by the recharge cycle, to increase the maximum number of recharges, you can either shorten the recharge cycle or lengthen the exposure time.

[0093] Figure 8B shows an example where the second ratio, obtained by dividing the saturation detection count by the maximum recharge count, is smaller than the first ratio by shortening the recharge period. In other words, in Figure 8B, the recharge period of the second frame is shorter than the recharge period of the first frame.

[0094] As shown in Figure 8B, shortening the recharge period reduces the number of photons incident in each recharge period, thereby suppressing counting errors. As a result, color changes can be suppressed.

[0095] In other words, as shown in Figure 8B, by shortening the recharge period, the count values ​​of the R and B pixels become 6, and the count value of the G image becomes 12, thus suppressing color changes. That is, the color changes are suppressed more in the count obtained in the second frame than in the first frame.

[0096] However, in the example in Figure 8B, the percentage of G pixel count misses is smaller than in Figure 7B, making it easier for the G pixel count value to increase. In other words, the count value obtained in the second frame is more prone to image clipping than the count value obtained in the first frame. Conversely, the count value obtained in the first frame suppresses image clipping more effectively than the count value obtained in the second frame.

[0097] In Figure 8B, the recharge period was changed by changing the frequency of the control signal CLK. That is, by changing the frequency of the pulse signal used to control the switching of the switch, the recharge period of the second frame was made shorter than the recharge period of the first frame.

[0098] However, the recharge period may be changed by masking a portion of the control signal CLK. That is, by masking a portion of the pulse signal used to control the switching of the switch, the recharge period of the second frame may be made shorter than the recharge period of the first frame.

[0099] Furthermore, the recharge period T can be changed by keeping the control signal CLK permanently ON, i.e., fixing its logic. That is, by fixing the logic of the switch in the second frame, the recharge period of the second frame can be made shorter than the recharge period of the first frame. In that case, the recharge period T is not determined by the period of the control signal CLK (= zero), but rather by the analog characteristics of the peripheral circuitry of the APD.

[0100] Figure 8C shows an example where the exposure time is increased so that the second ratio, obtained by dividing the saturation detection count by the maximum recharge count, is smaller than the first ratio. In other words, it shows an example where the exposure time of the second frame is longer than the exposure time of the first frame. If the recharge cycle in Figure 8C is the same as in Figure 7B, for example, count omissions in each recharge cycle will occur in the same way as in Figure 7B.

[0101] However, due to the long exposure time, both G pixels, which have a relatively large number of missed counts, and R and B pixels, which have a relatively small number of missed counts, tend to increase in count within the exposure time. In other words, all R, G, and B pixels are prone to saturation. Therefore, as with the case where the saturation judgment count in Figure 8A is reduced, the change in color is suppressed.

[0102] As described above, color changes can be suppressed by increasing the exposure time. In other words, the count obtained in the second frame shows less color change than the count obtained in the first frame.

[0103] On the other hand, as can be seen from Figure 8C, the count values ​​obtained in the second frame are more prone to image clipping than those obtained in the first frame. In other words, image clipping is suppressed more effectively with the count values ​​obtained in the first frame than with those obtained in the second frame.

[0104] Figures 9(A) and 9(B) show the response characteristics of pixels in the first and second frames, respectively, and are log-log graphs with Nph on the horizontal axis and Nct on the vertical axis. The solid line 91 in Figure 9(A) shows the response characteristics of the first frame, and the dashed line 92 in Figure 9(B) shows the response characteristics of pixels in the second frame when the saturation judgment count is reduced, as in Figure 8A.

[0105] Furthermore, in Figure 9(B), the dotted line 93 shows the response characteristics when the recharge period is shortened, as in Figure 8B, and the dashed line 94 shows the response characteristics of the pixels in the second frame when the exposure time is extended, as in Figure 8C.

[0106] As can be seen by comparing Figure 9(A) and Figure 9(B), the nonlinearity of the response characteristics is mitigated for all three lines: dashed line 92, dotted line 93, and chained line 94. Since the nonlinearity of the response characteristics is caused by the effect of count loss, Figures 9(A) and 9(B) also show that the color change is suppressed better in the second frame than in the first frame.

[0107] Furthermore, from the response characteristics in Figures 9(A) and 9(B), it can be seen that the dynamic range on the bright side is wider in the first frame than in the second frame. In other words, the first frame has relatively less blown-out highlights, while the second frame has relatively less change in color.

[0108] Note that 92d, 93d, and 94d in Figure 9(B) represent the dynamic ranges corresponding to 92, 93, and 94, respectively. As can be seen from Figure 9(B), changing the recharge period as shown by the dotted line 93 results in a wider dynamic range for the pixels in the second frame than lowering the saturation detection count as shown by the dashed line 92 or changing the exposure time as shown by the dotted line 94. Therefore, changing the recharge period is the most desirable approach.

[0109] However, shortening the recharge cycle increases the power consumption of the APD. Since the power consumption of the APD is proportional to Nct, the most desirable way to prioritize reducing power consumption is to change the saturation detection count.

[0110] On the other hand, when the exposure time is changed (dashed line 94), the slope of Nct relative to Nph changes. (However, since Figure 9 is a logarithmic graph, the dashed line 94 is shifted parallel to the dashed line 92 and the dotted line 93.) Therefore, it is desirable to combine the first and second frames to generate an image with a wide dynamic range.

[0111] Specifically, for dark subjects, the count value obtained in the second frame should be used, and for bright subjects, the count value obtained in the first frame should be used. Furthermore, multiple combinations of saturation detection counts, exposure time, and recharge cycles may be changed between the first and second frames.

[0112] As mentioned above, the smaller the ratio of the saturation detection count to the maximum recharge count, the more the color change is suppressed, and the larger the ratio, the more the overexposure is suppressed.

[0113] Therefore, the greater the difference between the first ratio (the saturation detection count of the first frame divided by the maximum recharge count) and the second ratio (the saturation detection count of the second frame divided by the maximum recharge count), the more desirable it is, as this allows for both color shift and suppression of overexposure. Specifically, the second ratio should ideally be half or less of the first ratio, and even more ideally 1 / 8 or less.

[0114] As described above, in the photoelectric conversion element of Embodiment 1, in an environment where a blinking light source is used as illumination light, it is possible to obtain a signal with suppressed overexposure using the first frame and a signal with suppressed color changes using the second frame.

[0115] <Embodiment 2> The photoelectric conversion element (photoelectric conversion device) of Embodiment 2 has a determination unit that determines the degree of color change by comparing the count value of a nearby first frame with the count value of a second frame, compared with the photoelectric conversion element of Embodiment 1.

[0116] Furthermore, the determination unit may be located inside the photoelectric conversion element, specifically on the circuit board 21, or it may be located outside the photoelectric conversion element and consist of a computer program, a dedicated circuit such as the signal processing unit 804 described later, or a processor. Also, the determination unit and the photoelectric conversion element do not have to be housed in the same enclosure, and may be composed of separate devices connected to each other via signal paths.

[0117] Figure 10(A) shows an example of the response characteristics of pixels in the first frame. The dashed line 121 shows the response characteristics when a steady light source is used as illumination, and the dotted line 122 shows the response characteristics when a blinking light source is used as illumination. Figure 10(B) shows an example of the response characteristics of pixels in the second frame. The solid line 123 in Figure 10(B) shows the cases where a steady light source is used as illumination, and where a blinking light source is used as illumination. Figures 10(A) and 10(B) show, for example, the case where the recharge period is changed between the first and second frames.

[0118] When a steady light source is used as illumination, the response characteristics of the pixels in the first and second frames correspond to Equation 1 described above. However, when a blinking light source is used as illumination, the output decreases when the number of incident photons is large due to the effect of counting errors.

[0119] As mentioned above, the impact of counting errors can be reduced in the second frame, so the output reduction when using a flashing light source as illumination is small, as shown in Figure 10(B). On the other hand, the impact of counting errors is large in the first frame, so the output reduction when using a flashing light source as illumination is large, as shown in Figure 10(A).

[0120] Therefore, if the third ratio NR, calculated by Equation 3 below, which is the ratio of the number of incident photons obtained from the output values ​​of the pixels in the first frame to the number of incident photons obtained from the output values ​​of the pixels in the second frame, is close to 1, it indicates that steady light is being used as illumination light. In other words, the change in color can be determined based on the ratio of the number of incident photons obtained from the output values ​​of the first frame to the number of incident photons obtained from the output values ​​of the second frame.

[0121] On the other hand, if the third ratio NR, which is calculated using Equation 3 and is the ratio of the output value of the pixels in the first frame to the output value of the pixels in the second frame, is less than 1, then it can be seen that a blinking light source is being used as illumination. In Equation 3, T1 is the recharge period of the first frame, and T2 is the recharge period of the second frame.

[0122]

number

[0123] The third ratio NR can be calculated for each smallest pixel unit in the pixel array (4 pixels in a 2x2 arrangement in the case of a Bayer array in Figure 6(A)), or a single third ratio NR can be calculated for multiple pixel units. When calculating a single third ratio NR for multiple pixel units, the calculation in Equation 3 can be performed for each of the first and second frames using the average or median values ​​of the multiple pixels.

[0124] Furthermore, as shown in Figure 6(A), if the photoelectric conversion element 100 has pixels equipped with on-chip color filters of multiple colors, it is preferable to determine the third ratio NR for each color. From the third ratio NR for each color, it is possible to determine how much count loss occurs in the output value of the pixels of each color.

[0125] Therefore, by determining the third ratio NR for each color and finding the difference in the third ratio for each color, the change in color can be determined with greater accuracy than simply using Equation 3. Specifically, the third ratio NR_R for the R pixel, the third ratio NR_G for the G pixel, and the third ratio NR_B for the B pixel should be determined, and then NR_R / NR_G and NR_B / NR_G should be calculated.

[0126] Thus, it is desirable to determine the change in color by calculating a third ratio for each color: the number of incident photons obtained from the output value of the first frame to the number of incident photons obtained from the output value of the second frame.

[0127] Furthermore, it is preferable that the first and second frames used to determine the change in color are temporally close to each other, and it is particularly desirable that the first and second frames be adjacent to each other. Therefore, it is desirable to drive the system so that the first and second frames alternate, as shown in Figure 6(B).

[0128] <Embodiment 3> The photoelectric conversion element (photoelectric conversion device) of Embodiment 3 has a synthesis processing unit that, according to the determination result determined by the determination unit, combines the output value of a nearby first frame and the output value of a pixel in a second frame to generate a composite image that suppresses overexposure and color changes.

[0129] The synthesis processing unit may be located inside the photoelectric conversion element, specifically on the circuit board 21, or it may consist of the photoelectric conversion element and a dedicated circuit such as a computer program with a determination unit or a signal processing unit 804 described later, or a processor. Furthermore, the determination unit and the photoelectric conversion element do not have to be housed in the same enclosure, and may be composed of separate devices connected to each other via signal paths.

[0130] To generate an image with suppressed overexposure and color shifts using judgment information, for example, the following can be done: When a steady light source is used as illumination and suppressing overexposure is important at the pixel level, the output value of the pixel in the first frame is used. Then, when a blinking light source is used as illumination and suppressing color shifts is important at the pixel level, the output value of the pixel in the second frame is used.

[0131] In other words, if the third ratio NR is above a certain threshold, the output value of the first frame is used to generate the composite image; if the third ratio NR is below a certain threshold, the output value of the second frame is used. The third ratio NR is, as described above, the ratio of the number of incident photons obtained from the output value of the first frame to the number of incident photons obtained from the output value of the second frame.

[0132] As mentioned above, if the third ratio NR is less than, for example, 1, it indicates that a flickering light source is being used as illumination, so it is desirable to use, for example, 1 as the specific threshold. However, because of the influence of errors when deriving the third ratio NR, it is more desirable to set the specific threshold to 1 / 2 or greater.

[0133] When performing image synthesis, it is desirable to have a larger number of pixels for which the third ratio NR is calculated. This is because it suppresses errors in the third ratio NR caused by factors such as light shot noise, and improves the accuracy of determining the extent of color changes.

[0134] On the other hand, especially when both continuous light and flashing light sources are used for illumination, the fewer pixels required to calculate the third ratio NR, the more precisely it is possible to select which pixel output from the first or second frame to use for each individual pixel. As a result, the effectiveness of suppressing blown-out highlights and color shifts is improved, which is desirable.

[0135] As mentioned above, the number of output signals Nct (count value) of a pixel and the number of photons Nph incident on the pixel are different, and the response characteristics are determined by the ratio of the saturation judgment count to the maximum recharge number. Therefore, it is preferable to perform the synthesis process using values ​​obtained by applying the correction process of Equation 2 to each of the output values ​​of the pixels in the first frame and the second frame.

[0136] Specifically, the output values ​​of the pixels in the first frame should be corrected using Equation 4, and the output values ​​of the pixels in the second frame should be corrected using Equation 5. In other words, it is desirable that the image synthesis processing unit performs different nonlinearity correction processes on the output values ​​of the pixels in the first frame and the output values ​​of the pixels in the second frame, and then generates a composite image.

[0137]

number

[0138]

number

[0139] <Embodiment 4> In Embodiment 4, by using the determination information obtained from the first and second frames, the driving conditions corresponding to the illumination light are selected in the frame following the acquisition of the determination information.

[0140] Specifically, in the frame following the acquisition of the judgment information, the system selects driving conditions that match the illumination light by changing at least one of the following: the saturation judgment count, exposure time, and recharge cycle.

[0141] In other words, the photoelectric conversion element (photoelectric conversion device) of Embodiment 4 differs from the photoelectric conversion element of Embodiment 2 in that it has a selection unit that selects the driving conditions for the next frame according to the determination result determined by the determination unit.

[0142] The selection unit may be located inside the photoelectric conversion element, specifically on the circuit board 21, or it may consist of the photoelectric conversion element and a dedicated circuit such as a computer program having a determination unit or a signal processing unit 804 described later, or a processor. Furthermore, the determination unit and the photoelectric conversion element do not have to be housed in the same enclosure, and may be composed of separate devices connected to each other via a signal path.

[0143] Furthermore, in pixel-level areas where the judgment information indicates that steady light is being used as illumination, it is important to suppress overexposure, and therefore it is desirable to use the driving conditions of the first frame.

[0144] On the other hand, in pixel units where the detection information indicates that a blinking light source is being used as illumination, it is desirable to use the driving conditions of the second frame, as suppressing color changes is important. That is, if the third ratio NR is above a certain threshold, it is desirable to use the driving conditions of the first frame in the next frame.

[0145] On the other hand, if the third ratio NR is less than a certain threshold (e.g., 1 / 2), it is desirable to use the driving conditions of the second frame in the next frame. As mentioned above, the third ratio NR is the ratio of the number of incident photons obtained from the output value of the first frame to the number of incident photons obtained from the output value of the second frame.

[0146] Furthermore, it is preferable to use, for example, 1 as the specific threshold. However, due to the influence of errors when deriving the third ratio NR, it is preferable that the specific threshold be between 1 / 2 and 1. In other words, due to the influence of errors when deriving the third ratio NR, it is preferable that the specific threshold be greater than or equal to 1 / 2.

[0147] Instead of changing the driving conditions for each pixel in the frame following the acquisition of judgment information, the same driving conditions may be used for all pixels. Specifically, for example, if it is determined from the judgment information of more than half of the pixels that continuous light is being used, it is desirable to use the driving conditions of the first frame for all pixels.

[0148] On the other hand, if, for example, it is determined from the determination information of more than half of the pixels that flashing illumination light is being used, it is desirable to use the driving conditions of the second frame for all pixels. That is, it is desirable to determine the driving conditions for the frames after the one in which the determination information was acquired, based on the determination information of more than half of the pixels.

[0149] If, at this point, the driving conditions of the first or second frame are used for all frames after the judgment information has been obtained, it will become impossible to obtain a judgment result for color change in subsequent frames.

[0150] Therefore, it is even more desirable to use different drive conditions for a small number of frames than for the majority of other frames. Specifically, it is desirable to adjust the frequency of these few frames so that the ratio of the frequency of frames using the first frame's drive conditions to the frequency of frames using the second frame's drive conditions is 1 / 4 or less, or 4 or more.

[0151] As can be seen from Figure 9, the larger the ratio of the saturation detection count to the maximum recharge count, the more white clipping is suppressed, and the smaller the ratio, the more color changes are suppressed. Therefore, it may be possible to select a different drive condition for the next frame, other than the drive conditions for the first frame and the drive conditions for the second frame.

[0152] For example, assuming that the overall illumination intensity of a flashing light source is constant, the shorter the on-time of the flashing light source, the stronger the illumination light becomes at the moment it is on, making it easier for counting errors to occur and resulting in greater color changes.

[0153] Therefore, if the judgment information indicates that the change in color is greater than a predetermined first value, it is desirable to reduce the ratio obtained by dividing the saturation judgment count by the maximum recharge count in the next frame by the predetermined first percentage compared to the second frame.

[0154] Furthermore, if it is determined that the change in color is smaller than a predetermined second value, in the next frame, the ratio obtained by dividing the saturation judgment count by the maximum recharge count is increased by a predetermined second percentage compared to the second frame. In this case, the first percentage and the second percentage may be different. It is desirable that the first value mentioned above be greater than the second value.

[0155] Furthermore, if the judgment information indicates that the change in color is small, the ratio obtained by dividing the saturation judgment count by the maximum recharge count in the next frame may be set to a value between the ratio in the first frame and the ratio in the second frame.

[0156] <Embodiment 5> Figure 11 is a functional block diagram showing an example configuration of a photoelectric conversion device using the photoelectric conversion elements of Embodiments 1 to 4. Note that some of the functional blocks shown in Figure 11 are realized by having the CPU, which acts as a computer within the photoelectric conversion device, execute a computer program stored in the memory, which acts as a storage medium.

[0157] However, some or all of these may be implemented in hardware. Hardware options include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, each functional block shown in Figure 11 does not necessarily have to be housed in the same enclosure; they may be composed of separate devices connected to each other via signal paths.

[0158] The photoelectric conversion device 800 includes a photoelectric conversion element 100, a control unit 801, a storage unit 802, a communication unit 803, a signal processing unit 804, and an imaging optical system 805. The photoelectric conversion element 100 captures an optical image formed by the imaging optical system 805. The signal read from the photoelectric conversion element 100 is supplied to the signal processing unit 804.

[0159] The signal processing unit 804 performs processing such as black level correction, gamma curve adjustment, noise reduction, data compression, white balance correction, and color conversion. Note that some parts of the signal processing unit 804 may be included in the signal processing circuit 103, and conversely, some parts of the signal processing circuit 103 may be included in the signal processing unit 804.

[0160] The signal processing unit 804 uses equation 2 described above to correct the nonlinearity of the output signal number Nct of the counter circuit 211. This generates the final image. With this configuration, a photoelectric conversion element that can suppress color changes even in environments where a flickering light source is used as illumination can be realized.

[0161] The control unit 801 has a built-in CPU 806, which functions as a control means to control the operation of each part of the entire photoelectric converter 800 based on a computer program stored in the memory 807, which serves as a storage medium.

[0162] Furthermore, the control unit 801 controls the exposure time of each frame of the photoelectric conversion element 100 and the timing of the control signal CLK via the control pulse generation unit 115 of the photoelectric conversion element 100. In other words, the control unit 801 controls the exposure time of the avalanche photodiode and also controls the recharge period of the avalanche photodiode.

[0163] The storage unit 802 includes, for example, a recording medium such as a memory card or a hard disk. The communication unit 803 has wireless and wired interfaces and outputs the generated image to the outside of the photoelectric converter 800 and receives signals from the outside.

[0164] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible in accordance with the spirit of the present invention, and these are not excluded from the scope of the present invention. Furthermore, some of the above embodiments may be combined as appropriate.

[0165] Furthermore, the present invention includes, for example, a system that implements the functions of the above embodiment using at least one processor such as a CPU, memory, and circuitry (e.g., an ASIC). Alternatively, multiple processors may be used for distributed processing.

[0166] Furthermore, in order to implement some or all of the control in the above embodiment, a computer program that implements the functions of the above embodiment may be supplied to the photoelectric converter or the like via a network or various storage media.

[0167] Furthermore, the computer (or CPU, MPU, etc.) in the photoelectric conversion, etc., may read and execute the program. In that case, the program and the storage medium in which the program is stored constitute the present invention. The present invention includes the following combinations.

[0168] (Configuration 1) A photoelectric conversion device comprising: an avalanche photodiode; a counter that counts and outputs the number of output signals from the avalanche photodiode until it reaches a saturation determination count; and a control unit that controls the exposure time of the avalanche photodiode and the recharge period of the avalanche photodiode, and sets the ratio of the saturation determination count to the maximum recharge number, which is determined by the quotient obtained by dividing the exposure time by the recharge period, as the first ratio in the first frame, and controls the ratio of the saturation determination count to the maximum recharge number to be a second ratio which is smaller than the first ratio in the second frame which is different from the first frame.

[0169] (Configuration 2) The photoelectric conversion device according to Configuration 1, characterized in that the saturation determination count of the second frame is smaller than the saturation determination count of the first frame.

[0170] (Configuration 3) The photoelectric conversion device according to Configuration 1 or 2, characterized in that the maximum number of recharges of the second frame is greater than the maximum number of recharges of the first frame.

[0171] (Configuration 4) The photoelectric conversion device according to Configuration 3, characterized in that the recharge period of the second frame is shorter than the recharge period of the first frame.

[0172] (Configuration 5) A photoelectric conversion device according to any one of Configurations 1 to 4, characterized in that it has a switch connected to a power line to which a driving voltage is applied, which is connected to one of the nodes of the avalanche photodiode, the anode or the cathode, and which controls the recharge period by switching the resistance value between the one node and the power line.

[0173] (Configuration 6) The photoelectric converter according to Configuration 5, characterized in that the recharge period of the second frame is made shorter than the recharge period of the first frame by changing the frequency of the pulse signal for controlling the switching of the switch.

[0174] (Configuration 7) The photoelectric converter according to Configuration 5 or 6, characterized in that the recharge period of the second frame is made shorter than the recharge period of the first frame by masking a portion of the pulse signal for controlling the switching of the switch.

[0175] (Configuration 8) A photoelectric converter according to any one of Configurations 5 to 7, characterized in that the logic of the switch in the second frame is fixed so that the recharge period of the second frame is shorter than the recharge period of the first frame.

[0176] (Configuration 9) A photoelectric conversion device according to any one of Configurations 1 to 8, characterized in that the exposure time of the second frame is longer than the exposure time of the first frame.

[0177] (Configuration 10) A photoelectric conversion device according to any one of Configurations 1 to 9, characterized in that the second ratio is half or less of the first ratio.

[0178] (Configuration 11) The photoelectric conversion device according to Configuration 10, characterized in that the second ratio is 1 / 8 times or less of the first ratio.

[0179] (Configuration 12) A photoelectric conversion device according to any one of Configurations 1 to 11, characterized in that the first frame and the second frame alternate periodically.

[0180] (Configuration 13) A photoelectric conversion device according to any one of Configurations 1 to 12, characterized in that it has a determination unit that determines the degree of color change by comparing the count value of the first frame with the count value of the second frame.

[0181] (Configuration 14) The photoelectric conversion device according to Configuration 13, characterized in that it determines the change in color based on the ratio of the number of incident photons obtained from the output value of the first frame to the number of incident photons obtained from the output value of the second frame.

[0182] (Configuration 15) The photoelectric conversion device according to Configuration 13 or 14, characterized in that it has a synthesis processing unit that synthesizes the output values ​​of the first frame and the second frame according to the determination result determined by the determination unit and generates a composite image with suppressed color changes.

[0183] (Configuration 16) The photoelectric conversion device according to Configuration 15, characterized in that a composite image is generated by using the output value of the first frame if the third ratio of the number of incident photons obtained from the output value of the first frame to the number of incident photons obtained from the output value of the second frame is greater than or equal to a certain threshold, and by using the output value of the second frame if the third ratio is less than the certain threshold.

[0184] (Configuration 17) The photoelectric conversion apparatus according to Configuration 15 or 16, characterized in that the synthesis processing unit performs different nonlinear correction processing on each of the output values ​​of the first frame and the output values ​​of the second frame, and then generates the synthesized image.

[0185] (Configuration 18) A photoelectric conversion device according to any one of Configurations 13 to 17, characterized in that it has a selection unit that selects the driving conditions for the next frame according to the determination result determined by the determination unit.

[0186] (Configuration 19) The photoelectric conversion device according to Configuration 18, characterized in that if a third ratio of the number of incident photons obtained from the output value of the first frame to the number of incident photons obtained from the output value of the second frame is greater than or equal to a specific threshold, the driving conditions of the first frame are used in the next frame, and if the third ratio is less than the specific threshold, the driving conditions of the second frame are used in the next frame.

[0187] (Configuration 20) A photoelectric conversion device according to Configuration 18 or 19, characterized in that it determines the driving conditions for the next frame and beyond based on the determination information of more than half of the pixels.

[0188] (Configuration 21) A photoelectric conversion device according to any one of Configurations 18 to 20, characterized in that the ratio of the frequency of frames using the driving conditions of the first frame to the frequency of frames using the driving conditions of the second frame is 1 / 4 or less, or 4 or more.

[0189] (Configuration 22) A photoelectric conversion device according to any one of Configurations 1 to 21, characterized in that each pixel having the avalanche photodiode is equipped with an on-chip color filter.

[0190] (Configuration 23) The photoelectric conversion device according to Configuration 22, characterized in that a change in color is determined by calculating the ratio of the number of incident photons obtained from the output value of the first frame to the number of incident photons obtained from the output value of the second frame for each color.

[0191] (Method) A photoelectric conversion method for controlling a photoelectric conversion element having an avalanche photodiode and a counter that counts and outputs the number of output signals from the avalanche photodiode until it reaches a saturation determination count, characterized in that the exposure time of the avalanche photodiode and the recharge period of the avalanche photodiode are controlled, and in a first frame, the ratio of the saturation determination count to the maximum recharge number, which is determined by the quotient obtained by dividing the exposure time by the recharge period, is set as the first ratio, and in a second frame different from the first frame, the ratio of the saturation determination count to the maximum recharge number is controlled to be a second ratio which is smaller than the first ratio.

[0192] (Program) A computer program for controlling each part of the photoelectric converter described in any one of configurations 1 to 23. [Explanation of Symbols]

[0193] 100: Photoelectric conversion element 11: Sensor board 12: Pixel area 13: Minimum pixel unit 21: Circuit board 22: Circuit area 101: Pixels 102: Photoelectric conversion unit 103: Signal Processing Circuit 110: Vertical scanning circuit 111: Horizontal scanning circuit 112: Readout circuit 113: Signal line 114: Output Circuit 115: Control pulse generation unit 201: Avalanche photodiode 202: Switch 210: Waveform shaping section 211: Counter circuit 212: Selection Circuit 215: Saturation judgment section 800: Photoelectric converter 801: Control Unit 802: Storage section 803: Communications Department 804: Signal Processing Unit 805: Imaging Optical System 806:CPU 807: Memory

Claims

1. Avalanche photodiode and A counter that counts and outputs the number of output signals from the avalanche photodiode until it reaches a saturation count, The exposure time of the avalanche photodiode is controlled, and the recharge period of the avalanche photodiode is controlled. A control unit controls the first frame to have a first ratio, which is the ratio of the saturation determination count to the maximum recharge count determined by the quotient obtained by dividing the exposure time by the recharge cycle, and the second frame, which is different from the first frame, to have a second ratio, which is smaller than the first ratio. A photoelectric conversion device characterized by having the following features.

2. The photoelectric conversion device according to claim 1, characterized in that the saturation determination count of the second frame is smaller than the saturation determination count of the first frame.

3. The photoelectric conversion device according to claim 1, characterized in that the maximum number of recharges of the second frame is greater than the maximum number of recharges of the first frame.

4. The photoelectric conversion device according to claim 3, characterized in that the recharge period of the second frame is shorter than the recharge period of the first frame.

5. The photoelectric conversion device according to claim 1, characterized in that it has a switch connected to one of the nodes of the avalanche photodiode, either the anode or the cathode, and to a power line to which a driving voltage is applied, and the switch controls the recharge period by switching the resistance value between the one node and the power line.

6. The photoelectric converter according to claim 5, characterized in that the recharge period of the second frame is made shorter than the recharge period of the first frame by changing the frequency of the pulse signal for controlling the switching of the switch.

7. The photoelectric converter according to claim 5, characterized in that the recharge period of the second frame is made shorter than the recharge period of the first frame by masking a portion of the pulse signal for controlling the switching of the switch.

8. The photoelectric converter according to claim 5, characterized in that the logic of the switch in the second frame is fixed so that the recharge period of the second frame is shorter than the recharge period of the first frame.

9. The photoelectric conversion apparatus according to claim 1, characterized in that the exposure time of the second frame is longer than the exposure time of the first frame.

10. The photoelectric conversion device according to claim 1, characterized in that the second ratio is less than or equal to half of the first ratio.

11. The photoelectric conversion device according to claim 10, characterized in that the second ratio is 1 / 8 times or less of the first ratio.

12. The photoelectric conversion device according to claim 1, characterized in that the first frame and the second frame alternate periodically.

13. The photoelectric conversion device according to claim 1, characterized in that it has a determination unit that determines the degree of color change by comparing the count value of the first frame with the count value of the second frame.

14. The photoelectric conversion device according to claim 13, characterized in that it determines the change in color based on the ratio of the number of incident photons obtained from the output value of the first frame to the number of incident photons obtained from the output value of the second frame.

15. The photoelectric conversion apparatus according to claim 13, further comprising a synthesis processing unit that synthesizes the output values ​​of the first frame and the second frame according to the determination result determined by the determination unit, and generates a composite image with suppressed color changes.

16. The photoelectric conversion apparatus according to claim 15, characterized in that a composite image is generated by using the output value of the first frame if a third ratio of the number of incident photons obtained from the output value of the first frame to the number of incident photons obtained from the output value of the second frame is greater than or equal to a specific threshold, and using the output value of the second frame if the third ratio is less than the specific threshold.

17. The photoelectric conversion apparatus according to claim 15, characterized in that the synthesis processing unit performs different nonlinear correction processing on the output value of the first frame and the output value of the second frame, and then generates the synthesized image.

18. The photoelectric conversion device according to claim 13, further comprising a selection unit that selects the driving conditions for the next frame according to the determination result determined by the determination unit.

19. The photoelectric converter according to claim 18, characterized in that if a third ratio of the number of incident photons obtained from the output value of the first frame to the number of incident photons obtained from the output value of the second frame is greater than or equal to a specific threshold, the driving conditions of the first frame are used in the next frame, and if the third ratio is less than the specific threshold, the driving conditions of the second frame are used in the next frame.

20. The photoelectric converter according to claim 18, characterized in that it determines the driving conditions for the next frame and beyond based on the determination information of more than half of the pixels.

21. The photoelectric conversion device according to claim 18, characterized in that the ratio of the frequency of frames using the driving conditions of the first frame to the frequency of frames using the driving conditions of the second frame is 1 / 4 or less, or 4 or more.

22. The photoelectric conversion device according to claim 1, characterized in that each pixel having the avalanche photodiode is equipped with an on-chip color filter.

23. The photoelectric conversion apparatus according to claim 22, characterized in that a change in color is determined by calculating the ratio of the number of incident photons obtained from the output value of the first frame to the number of incident photons obtained from the output value of the second frame for each color.

24. A photoelectric conversion method for controlling a photoelectric conversion element having an avalanche photodiode and a counter that counts and outputs the number of output signals from the avalanche photodiode until it reaches a saturation determination count, The exposure time of the avalanche photodiode is controlled, and the recharge period of the avalanche photodiode is controlled. A photoelectric conversion method characterized in that, in a first frame, the ratio of the saturation determination count to the maximum recharge count, which is determined by the quotient obtained by dividing the exposure time by the recharge period, is defined as the first ratio, and in a second frame different from the first frame, the ratio of the saturation determination count to the maximum recharge count is controlled to be a second ratio which is smaller than the first ratio.

25. A computer program for controlling each part of the photoelectric converter according to any one of claims 1 to 23 by computer.

Images