Photoelectric conversion device, photoelectric conversion method, and computer program

Abstract

The present invention provides a photoelectric conversion device that can suppress changes in color even in environments where flashing light sources or light sources moving within a frame are used as illumination. [Solution] The photoelectric conversion device is characterized by comprising: an avalanche photodiode; a counter that counts and outputs the number of output signals from the avalanche photodiode until it reaches a saturation determination count; and a control unit that controls the exposure time of the avalanche photodiode and the recharge period of the avalanche photodiode, thereby setting a first pixel whose ratio to the saturation determination count with respect to the maximum recharge number, which is determined by the quotient obtained by dividing the exposure time by the recharge period, is a first ratio; and a second pixel whose ratio to the saturation determination count with respect to the maximum recharge number is a second ratio smaller than the first ratio.

Description

Technical Field

[0001] The present invention relates to a photoelectric conversion device, a photoelectric conversion method, a computer program, and the like.

Background Art

[0002] In recent years, a photoelectric conversion element has been proposed that digitally counts the number of photons arriving at an avalanche photodiode (APD) and outputs the count value as a digitally converted photoelectric signal from a pixel.

[0003] Patent Document 1 describes a photoelectric conversion device in which the timing for charging an APD is controlled by a pulse signal so that the number of periods during which avalanche multiplication occurs can be suitably detected.

[0004] In the photoelectric conversion device of Patent Document 1, photons can be counted one by one for each period of the pulse generated by the pulse generation circuit. When this method is used, if a plurality of photons enter during one period of the pulse, count omission occurs.

[0005] However, when capturing general steady light, by statistically predicting the number of photons incident during one period of the pulse, the influence of count omission can be reduced by correction processing.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] On the other hand, when using light sources that blink within a single frame, particularly LED light sources using pulse modulation drive methods, or light sources that move within a frame, many count omissions occur compared to when using steady light. As a result, the accuracy of the count omission correction process decreases, and the color appears different from that of the actual environment.

[0008] Therefore, one of the objectives of the present invention is to provide a photoelectric conversion device that can suppress changes in color even in environments where flashing light sources or light sources moving within a frame are used as illumination. [Means for solving the problem]

[0009] One aspect of the present invention is a photoelectric conversion device, Avalanche photodiode and A counter that counts and outputs the number of output signals from the avalanche photodiode until it reaches a saturation count, By controlling the exposure time of the avalanche photodiode and the recharge period of the avalanche photodiode, A control unit sets a first pixel whose ratio of the saturation determination count to the maximum recharge count, which is determined by the quotient obtained by dividing the exposure time by the recharge cycle, is a first ratio, and a second pixel whose ratio of the saturation determination count to the maximum recharge count is a second ratio which is smaller than the first ratio. It is characterized by having the following features. [Effects of the Invention]

[0010] According to the present invention, a photoelectric conversion device can be realized that can suppress changes in color even in environments where flashing light sources or light sources moving within a frame are used as illumination. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an example of the configuration of a photoelectric conversion element 100 according to Embodiment 1 of the present invention. [Figure 2] This figure shows an example configuration of the sensor substrate 11 according to Embodiment 1. [Figure 3] This figure shows an example configuration of the circuit board 21 according to Embodiment 1. [Figure 4] This figure shows an example of an equivalent circuit of a pixel 101 and a signal processing circuit 103 corresponding to the pixel 101 according to Embodiment 1. [Figure 5] This is a timing chart of the photoelectric conversion element for illustrating Embodiment 1. [Figure 6] This figure shows an example of the pixel arrangement of the first pixel 101a and the second pixel 101b of the photoelectric conversion element 10 of Embodiment 1. [Figure 7A] This diagram illustrates the counting operation when using continuous light as illumination. [Figure 7B] This diagram illustrates the counting operation when a time-blinking light source is used as illumination. [Figure 7C] This diagram illustrates the illumination intensity of a region of interest when there is a light source moving within a single frame. [Figure 8A] This figure shows an example where reducing the saturation detection count makes the second ratio (saturation detection count divided by the maximum recharge count) smaller than the first ratio. [Figure 8B] This figure shows an example where shortening the recharge cycle makes the second ratio (saturation detection count divided by the maximum recharge count) smaller than the first ratio. [Figure 8C] This figure shows an example where increasing the exposure time resulted in a second ratio (saturation detection count divided by the maximum recharge count) being smaller than the first ratio. [Figure 9] Figures (A) and (B) represent the response characteristics of the first and second pixels, respectively. [Figure 10] (A) is a diagram showing an example of the response characteristics of the first pixel, and (B) is a diagram showing an example of the response characteristics of the second pixel. [Figure 11A] This diagram shows an example where the positions of the first and second pixels are swapped in each frame. [Figure 11B]It is a diagram showing an example of using driving conditions different from those of many other pixels in a small number of pixels. [Figure 12] It is a functional block diagram showing a configuration example of a photoelectric conversion device using the photoelectric conversion elements of Embodiments 1 to 4.

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each figure, the same members or elements are denoted by the same reference numerals, and duplicate explanations are omitted or simplified.

[0013] <Embodiment 1> FIG. 1 is a diagram showing a configuration example of a photoelectric conversion element 100 according to Embodiment 1 of the present invention. The sensor substrate 11 includes a pixel region 12, and the circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0014] In this embodiment, an example of a so-called stacked structure photoelectric conversion device in which the photoelectric conversion element 100 is composed of two substrates, a sensor substrate 11 and a circuit substrate 21, which are stacked and electrically connected, will be described. However, a so-called non-stacked structure in which the configurations included in the sensor substrate and the configurations included in the circuit substrate are arranged in a common semiconductor layer may also be used.

[0015] FIG. 2 is a diagram showing a configuration example of the sensor substrate 11 according to Embodiment 1. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 two-dimensionally arranged in a plurality of rows and columns. The pixel 101 includes a photoelectric conversion unit​​​​​

[0017] Figure 3 shows an example of the configuration of a circuit board 21 according to Embodiment 1. The circuit board 21 has a signal processing circuit 103 for processing the charge photoelectrically converted by the photoelectric conversion unit 102 in Figure 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a signal line 113, and a vertical scanning circuit 110.

[0018] The vertical scanning circuit 110 receives control pulses supplied from the control pulse generation unit 115 and supplies control pulses to each pixel. Logic circuits such as a shift register and an address decoder are used in the vertical scanning circuit 110.

[0019] The signal output from the photoelectric conversion unit 102 of each pixel is processed by the signal processing circuit 103. The signal processing circuit 103 is equipped with a counter and memory, and digital values ​​are stored in the memory. The horizontal scanning circuit 111 inputs control pulses to the signal processing circuit 103 to sequentially select each column in order to read the signal from the memory of each pixel in which the digital signal is stored.

[0020] Signals are output to signal line 113 from the signal processing circuit 103 of the pixels selected by the vertical scanning circuit 110 for the selected column. The signals output to signal line 113 are then output to the outside of the photoelectric conversion element 100 via the output circuit 114.

[0021] As shown in Figures 2 and 3, multiple signal processing circuits 103 are arranged in the region that overlaps with the pixel region 12 in a plan view. Then, a vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generation unit 115 are arranged so as to overlap between the edge of the sensor substrate 11 and the edge of the pixel region 12 in a plan view.

[0022] In other words, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generation unit 115 are arranged in the region that overlaps with the non-pixel region in a plan view.

[0023] Note that the arrangement of the signal line 113, the read circuit 112, and the output circuit 114 is not limited to Figure 3. For example, the signal line 113 may be arranged extending in the row direction, and the read circuit 112 may be placed at the end of the signal line 113.

[0024] Furthermore, the signal processing unit does not necessarily need to be provided in one unit for each photoelectric conversion unit; a single signal processing unit may be shared by multiple photoelectric conversion units, and the signal processing may be performed sequentially.

[0025] Figure 4 shows an example of an equivalent circuit of a pixel 101 and a signal processing circuit 103 corresponding to the pixel 101 according to Embodiment 1.

[0026] The APD201 generates charge pairs corresponding to incident light through photoelectric conversion. One of the two nodes of the APD201 is connected to a power line that supplies the drive voltage VL (first voltage).

[0027] Furthermore, the other of the two nodes of the APD201 can be connected via switch 202 to a power line supplied with a drive voltage VH (second voltage) higher than voltage VL. In Figure 4, one node of the APD201 is the anode, and the other node of the APD is the cathode.

[0028] A reverse bias voltage is supplied to the anode and cathode of the APD201, causing it to perform avalanche multiplication. This voltage supply causes the charge generated by the incident light to undergo avalanche multiplication, resulting in the generation of an avalanche current.

[0029] Furthermore, when a reverse bias voltage is supplied, there are two modes of operation: Geiger mode, where the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, where the voltage difference between the anode and cathode is near or below the breakdown voltage. An APD operating in Geiger mode is called a SPAD. In the case of a SPAD, for example, the voltage VL (first voltage) is -30V and the voltage VH (second voltage) is 1V.

[0030] Switch 202 is connected to the power line to which the drive voltage VH is supplied and to one of the nodes, either the anode or the cathode, of the APD201. Switch 202 then switches the resistance value between the APD201 and the power line to which the drive voltage VH is supplied.

[0031] In other words, switch 202 is connected to one of the anodes or cathodes of the avalanche photodiode and to a power line to which the drive voltage is applied, and switches the resistance value between the one node and the power line.

[0032] Here, switching the resistance value means that it is desirable to change the resistance value by 10 times or more, and more desirable to change the resistance value by 100 times or more. In the following, a decrease in the resistance value will be referred to as turning switch 202 on, and a increase in the resistance value will be referred to as turning switch 202 off.

[0033] Switch 202 functions as a quench element. Specifically, when the signal is amplified by avalanche multiplication, switch 202 acts as a load circuit (quench circuit), suppressing the voltage supplied to APD201 and performing a quench operation to suppress avalanche multiplication.

[0034] Furthermore, switch 202 performs a recharge operation to return the voltage supplied to APD201 to the drive voltage VH by supplying current to compensate for the voltage drop caused by the quench operation. Switch 202 can be constructed using a MOS transistor, and Figure 4 shows the case where switch 202 is a PMOS transistor.

[0035] The control signal CLK for controlling switch 202 is applied from the control pulse generation unit 115, which acts as a signal generation unit, to the gate electrode of the MOS transistor that constitutes switch 202.

[0036] The control pulse generation unit 115 controls the recharge period of the avalanche photodiode by controlling the voltage applied to the gate electrode of the switch 202, thereby controlling the on and off states of the switch 202. Furthermore, the exposure time (count time) of the avalanche photodiode is controlled by the control unit 801, which will be described later, as an exposure time control unit.

[0037] The signal processing circuit 103 includes a waveform shaping unit 210, a counter circuit 211, and a selection circuit 212. However, the signal processing circuit 103 only needs to have at least one of the waveform shaping unit 210, the counter circuit 211, and the selection circuit 212.

[0038] The waveform shaping unit 210 shapes the cathode voltage change of the APD201 obtained during photon detection and outputs a pulse signal. For example, an inverter circuit can be used as the waveform shaping unit 210. The inverter circuit may be a circuit in which multiple inverters are connected in series, or other circuits that have a waveform shaping effect may be used.

[0039] The counter circuit 211 counts the pulse signal output from the waveform shaping unit 210 and holds the count value. When the control pulse RES is supplied via the drive line 213, the signal held by the counter circuit 211 is reset.

[0040] A saturation determination unit 215 is connected to the counter circuit 211, which determines when the number of output signals measured by the counter circuit has reached saturation. When the number of output signals measured by the counter circuit 211 reaches the saturation determination count, the saturation determination unit 215 stops the count-up of the pulse signal by the counter circuit and maintains the saturation determination count as the count value in the counter circuit 211. The saturation determination count can be set as a predetermined threshold.

[0041] In this embodiment, the quotient obtained by dividing the exposure time by the recharge cycle is called the maximum recharge number. Furthermore, the plurality of pixels of the photoelectric conversion element 100 in this embodiment consist of a plurality of first pixels 101a and a plurality of second pixels 101b. Also, the number of first pixels 101a and the number of second pixels 101b are approximately the same, as described above.

[0042] Furthermore, the first pixel 101a is configured such that the ratio obtained by dividing the saturation detection count by the maximum recharge count is the first ratio, and the second pixel 101b is configured such that the ratio obtained by dividing the saturation detection count by the maximum recharge count is smaller than the first ratio, resulting in a second ratio.

[0043] To achieve this, in this embodiment, the counter circuit (counter) of the second pixel is made smaller than the counter circuit (counter) of the first pixel. This makes it possible to make the saturation determination count of the second pixel smaller than the saturation determination count of the first pixel.

[0044] Alternatively, the configuration of the counter circuits for the first and second pixels may remain the same, and control may be achieved by changing the number of counts used for saturation determination from the control pulse generation unit 115 via a signal line (not shown). With this configuration, the saturation determination counts for the first and second pixels may be changed according to the exposure time and recharge period of each pixel.

[0045] Furthermore, as mentioned above, since the maximum recharge count is the exposure time divided by the recharge cycle, it is desirable to be able to change the saturation detection count for the first and second pixels depending on the exposure time and the recharge cycle.

[0046] The selection circuit 212 receives a control pulse SEL from the vertical scanning circuit 110 in Figure 3 via the drive line 214 (not shown in Figure 3) in Figure 4, which switches the electrical connection between the counter circuit 211 and the signal line 113. The selection circuit 212 includes, for example, a buffer circuit for outputting a signal.

[0047] The output signal OUT shown in Figure 4 is the output signal as a pixel output, and the count value of the counter circuit 211 is output. In other words, the counter circuit 211 functions as a counter that counts and outputs the number of output signals from the avalanche photodiode until it reaches the saturation judgment count.

[0048] Furthermore, switches such as transistors may be placed between the switch 202 (which acts as a quench element) and the APD201, or between the photoelectric conversion unit 102 and the signal processing circuit 103, to switch the electrical connections. Similarly, the supply of voltage VH or voltage VL to the photoelectric conversion unit 102 may be electrically switched using switches such as transistors.

[0049] Furthermore, while it is possible to perform quench and recharge operations using switch 202 in response to avalanche multiplication in APD201, depending on the timing of photon detection, these may not be recognized as output signals.

[0050] For example, consider a scenario where avalanche multiplication occurs in the APD, causing the input voltage to nodeA to become low, and a recharge operation is performed. In this case, the judgment threshold of the waveform shaping unit 210 is set to a voltage higher than the voltage difference at which avalanche multiplication occurs in the APD.

[0051] When a photon is incident while the voltage of node A is below the threshold due to the recharge operation and the voltage is within the range where avalanche multiplication is possible at the APD, avalanche multiplication occurs at the APD and the voltage of node A decreases.

[0052] In other words, because the voltage at nodeA drops below the detection threshold, the output voltage from nodeB does not change despite the detection of photons. Consequently, even though avalanche multiplication is occurring, it is no longer recognized as a signal.

[0053] In particular, under high illumination conditions, photons enter the field in rapid succession over a short period, making it difficult to recognize them as a signal. As a result, even under high illumination conditions, there is a tendency for the actual number of incident photons to deviate from the output signal.

[0054] In contrast, in this embodiment, by applying the control signal CLK to the switch 202 and switching the switch 202 on and off, it becomes possible to determine that photons are entering the APD in succession in a short period of time as a signal.

[0055] Figure 5 is a timing chart of a photoelectric conversion element to illustrate Embodiment 1, schematically showing the relationship between the switch control signal CLK, the voltage at node A, the voltage at node B, and the output signal. Note that Figure 5 shows the case where the control signal CLK is a pulse signal with a constant repetition period.

[0056] In the photoelectric converter of this embodiment, when the control signal CLK is at a high level, the drive voltage VH is less likely to be supplied to the APD, and when the control signal CLK is at a low level, the drive voltage VH is supplied to the APD. A high level control signal CLK is, for example, 1V, and a low level control signal CLK is, for example, 0V.

[0057] Switch 202 is turned off when the control signal CLK is at a high level, and turned on when the control signal CLK is at a low level. The resistance of switch 202 when the control signal CLK is at a high level is higher than the resistance of switch 202 when the control signal CLK is at a low level.

[0058] When the control signal CLK is at a high level, even if avalanche multiplication occurs in the APD, recharging is less likely to occur, resulting in the voltage supplied to the APD being below the APD's breakdown voltage. Consequently, the avalanche multiplication operation in the APD stops.

[0059] At time t1, the control signal CLK changes from a high level to a low level, switching 202 on and initiating the APD recharge operation. This causes the APD cathode voltage to transition to a high level.

[0060] Then, the voltage difference between the voltages applied to the anode and cathode of the APD becomes a state where avalanche multiplication is possible. The cathode voltage is the same as that of node nodeA. Therefore, when the cathode voltage transitions from a low level to a high level, the voltage of node nodeA becomes greater than or equal to the threshold at time t2.

[0061] At this time, the pulse signal output from nodeB inverts, changing from a high level to a low level. Once recharging is complete, the APD201 is subjected to a voltage difference between the drive voltage VH and the drive voltage VL. Subsequently, the control signal CLK becomes high, and switch 202 turns off.

[0062] Next, at time t3, when a photon is incident on APD201, avalanche multiplication occurs in APD201, an avalanche multiplication current flows through switch 202, and the cathode voltage drops. In other words, the voltage at node A drops.

[0063] If the voltage at nodeA falls below the threshold during the voltage drop, the voltage at nodeB changes from a low level to a high level. In other words, the portion of the output waveform at nodeA that exceeds the threshold is shaped by the waveform shaping unit 210 and output as a signal at nodeB. This is then counted by the counter circuit, and the count value of the counter signal output from the counter circuit increases by 1 LSB.

[0064] Although photons are incident on the APD between time t3 and time t4, switch 202 is in the off state, and the voltage applied to APD201 is not a voltage difference that allows for avalanche multiplication, so the voltage level of node A does not exceed the judgment threshold.

[0065] At time t4, the control signal CLK changes from high level to low level, and switch 202 turns on. Consequently, a current flows through node A to compensate for the voltage drop from the drive voltage VL, and the voltage at node A returns to its original voltage level. At this time, at time t5, the voltage at node A exceeds the judgment threshold, so the pulse signal at node B inverts, changing from high level to low level.

[0066] At time t6, node A settles back to its original voltage level, and the control signal CLK changes from a low level to a high level. Subsequently, as explained from time t1 to time t6, the voltages of each node and signal lines change in response to the control signal CLK and the incidence of photons. In this way, the recharge frequency of the APD can be controlled by applying the control signal CLK to switch 202 and switching it on and off.

[0067] If the control signal CLK is not used, there is a problem in that the actual count value becomes smaller than the count value corresponding to the brightness of the incident light at high brightness. However, this problem can be solved by applying the control signal CLK to switch 202 and switching switch 202 on and off.

[0068] The recharge cycle corresponds to the time from t1, when the recharge operation begins, to t4, when the next recharge operation begins. Furthermore, the length of the period during which the recharge frequency is controlled by the control signal CLK (from t1 to tN in Figure 5) corresponds to the exposure time.

[0069] However, when controlling the APD recharge frequency with the control signal CLK, counting errors occur if multiple photons are incident in one pulse period. Therefore, the relationship between the number of output signals and the number of input signals is not linear.

[0070] When photographing general continuous light, the relationship between the number of input signals and the number of output signals can be theoretically derived from the statistical distribution of the number of incident photons. Specifically, when the number of input signals is Nph, the number of output signals is Nct, the recharge period of the control signal CLK is T, and the exposure time is t, the following equation 1 holds.

number

[0071] Therefore, in this embodiment, the number of photons incident in one period of the pulse is statistically predicted from the number of output signals Nct using the following Equation 2. This allows the effects of count omissions to be recovered by the correction process in the signal processing unit 804 described later. That is, the nonlinearity of the number of output signals Nct of the counter circuit can be corrected by performing the process in Equation 2 in the signal processing unit 804.

number

[0072] On the other hand, in environments where a light source that blinks within a single frame, particularly an LED light source using a pulse modulation driving method, is used as illumination, many count omissions occur compared to when using steady light. As a result, the accuracy of the count omission correction process decreases, and the color appears different from that of the actual environment.

[0073] Therefore, in this embodiment, a first pixel is set where the ratio obtained by dividing the saturation determination count by the maximum recharge count is a first ratio, and a second pixel is set where the ratio obtained by dividing the saturation determination count by the maximum recharge count is a second ratio which is smaller than the first ratio. This makes it possible to realize a photoelectric conversion element that can suppress both color changes and overexposure even in environments where a flickering light source is used as illumination.

[0074] Figure 6 shows an example of the pixel arrangement of the first pixel 101a and the second pixel 101b of the photoelectric conversion element 10 of Embodiment 1. Each pixel of the photoelectric conversion element 10 of this embodiment is equipped with an on-chip color filter of one of the so-called R (red), G (green), or B (blue) colors, and both the multiple first pixels 101a and the multiple second pixels 101b each contain pixels of all types of colors: R, G, and B.

[0075] Specifically, multiple types of on-chip color filters are placed on multiple first pixels, and multiple types of on-chip color filters are placed on multiple second pixels.

[0076] Furthermore, within the pixel region 12, the smallest unit in which pixels of all types of colors (R, G, and B) are included in each of the multiple first pixels and multiple second pixels is called the smallest pixel unit 13.

[0077] Figure 6 shows an example where the minimum pixel unit is 16 pixels in a 4x4 grid, but the minimum pixel unit may also be 64 pixels in an 8x8 grid, and the number of rows and columns in the minimum pixel unit may differ. Furthermore, the on-chip color filter may include Y (yellow) pixels other than R, G, and B, and may also have pixels without an on-chip color filter (W pixels).

[0078] Figure 7A illustrates the counting operation when a steady light source is used as illumination. Figure 7B illustrates the counting operation when a time-blinking light source is used as illumination.

[0079] Figure 7B is a diagram illustrating why, when using a conventional photoelectric conversion element, a change in color occurs when the averaged illumination intensity is the same as in Figure 7A, but the light source is blinking over time.

[0080] In Figures 7A and 7B, the photoelectric conversion element has an on-chip color filter in a so-called RGB Bayer array, and each pixel receives light through one of the R, G, or B color filters. Furthermore, white light is used as illumination, with a 1:2:1 ratio of photons incident on the R, G, and B pixels per unit time. Also, in Figures 7A and 7B, the saturation detection count is set to, for example, 12 counts.

[0081] As shown in Figure 7A, when constant illumination light is incident, the number of photons incident in each of the R, G, and B pixels during their respective recharge periods is small, resulting in minimal counting errors during each recharge period.

[0082] On the other hand, as shown in Figure 7B, when a flickering light source with discrete illumination intensity is used as the illumination light, photons are concentrated and incident during specific recharge periods, resulting in many missed counts. Therefore, when illumination lights with different numbers of photons per unit time incident on R, G, and B pixels are used, many missed counts are expected to occur in pixels of a particular color.

[0083] This disrupts the balance of R, G, and B pixel counts, causing a change in color. For example, in the example in Figure 7B, many G pixels are missed, and if the saturation detection count is set to, say, 12 counts, the color shifts towards magenta.

[0084] While Figure 7B shows the case where a blinking light source is used as illumination, counting errors also occur when a moving light source is used. Figure 7C illustrates the illumination intensity of the area of ​​interest when there is a moving light source within one frame.

[0085] Figure 7C shows that within a single frame, there are moments of high illumination intensity and moments of low illumination intensity in the region of interest. Therefore, similar to the case where a flickering light source is used as illumination, photons are concentrated and incident during specific recharge cycles, resulting in many missed counts.

[0086] On the other hand, the photoelectric conversion element of this embodiment includes a first pixel where the ratio of the saturation determination count divided by the maximum recharge count is a first ratio, and a second pixel where the ratio of the saturation determination count divided by the maximum recharge count is a second ratio which is smaller than the first ratio. This makes it possible to suppress both color changes and blown-out highlights.

[0087] In the following explanation, we will use the example where the first ratio, obtained by dividing the saturation detection count of the first pixel by the maximum recharge count, is equal to the ratio obtained by dividing the saturation detection count of a conventional photoelectric conversion element by the maximum recharge count. In this case, the saturation detection count is assumed to be 12 counts. In this case, the photons incident on the first pixel and the count value when a blinking light source is used as illumination light are the same as in Figure 7B.

[0088] On the other hand, the second ratio, obtained by dividing the saturation detection count of the second pixel by the maximum recharge count, is smaller than the ratio obtained by dividing the saturation detection count of a conventional photoelectric conversion element by the maximum recharge count. In other words, at least one of the saturation detection count or the maximum recharge count of the second pixel is different from that of the first pixel. A specific example is shown below.

[0089] Figure 8A shows an example where the saturation detection count is reduced, making the second ratio (saturation detection count divided by the maximum recharge count) smaller than the first ratio. Since the recharge cycle in Figure 8A is the same as in Figure 7B, count omissions in each recharge cycle occur in the same way as in Figure 7B.

[0090] However, since the saturation detection count is suppressed to, for example, 6 in Figure 8A, both G pixels, which have a relatively large number of missed counts, and R and B pixels, which have a relatively small number of missed counts, are more likely to reach saturation in their counts, thus suppressing changes in color.

[0091] In other words, for example, if the saturation detection count is set to 6 in Figure 8A, the color does not change towards magenta as in Figure 7B; instead, all R, G, and B pixels are saturated, i.e., represented as white.

[0092] In this way, by reducing the saturation detection count, color changes can be suppressed. That is, the color change is suppressed more effectively when using the count obtained from the second pixel than from the first pixel.

[0093] On the other hand, as can be seen by comparing Figure 7B and Figure 8A, if the saturation detection count is set to, for example, 6 counts as in Figure 8A, the count value obtained from the second pixel is more prone to image clipping than the count value obtained from the first pixel. In other words, image clipping is suppressed more effectively when using the count value obtained from the first pixel than when using the count value obtained from the second pixel.

[0094] Figures 8B and 8C show an example where increasing the maximum recharge count makes the second ratio (saturation detection count divided by the maximum recharge count) smaller than the first ratio. In other words, the maximum recharge count of the second pixel is made larger than the maximum recharge count of the first pixel.

[0095] Note that in Figures 8B and 8C, the saturation detection count is set to 12, for example, as in Figures 7A and 7B. Since the maximum recharge count is the exposure time divided by the recharge period, to increase the maximum recharge count, you can either shorten the recharge period or lengthen the exposure time.

[0096] Figure 8B shows an example where the second ratio, obtained by dividing the saturation detection count by the maximum recharge count, is made smaller than the first ratio by shortening the recharge period. In other words, the recharge period of the second pixel is made shorter than the recharge period of the first pixel.

[0097] As shown in Figure 8B, shortening the recharge period reduces the number of photons incident in each recharge period, thereby suppressing counting errors. As a result, color changes can be suppressed.

[0098] In other words, as shown in Figure 8B, the count values ​​for the R and B pixels are 6, and the count value for the G image is 12, so by shortening the recharge period, color changes can be suppressed. That is, the color change is suppressed more in the count obtained from the second pixel than in the first pixel.

[0099] On the other hand, in Figure 8B, the rate of G pixel count loss is smaller than in Figure 7B, making it easier for the G pixel count value to increase. In other words, the count value obtained from the second pixel is more likely to cause image clipping than the count value obtained from the first pixel. Conversely, the count value obtained from the first pixel suppresses image clipping more effectively than the count value obtained from the second pixel.

[0100] In Figure 8B, the recharge period was changed by changing the frequency of the control signal CLK. That is, by changing the frequency of the pulse signal used to control the switching of the switch, the recharge period of the second pixel was made shorter than the recharge period of the first pixel.

[0101] However, the recharge period may be changed by masking a portion of the control signal CLK. That is, by masking a portion of the pulse signal used to control the switching of switch 202, the recharge period of the second pixel may be made shorter than the recharge period of the first pixel.

[0102] Furthermore, the recharge period T may be changed by keeping the control signal CLK permanently ON, i.e., fixing its logic. That is, by fixing the logic of the switch 202 of the second pixel, the recharge period of the second pixel may be made shorter than that of the first pixel. In that case, the recharge period T is not determined by the period of the control signal CLK (= zero), but by the analog characteristics of the peripheral circuit of the APD.

[0103] Figure 8C shows an example where the exposure time is increased to make the second ratio (saturation detection count divided by the maximum recharge count) smaller than the first ratio. In other words, it shows an example where the exposure time of the second pixel is longer than the exposure time of the first pixel. If the recharge cycle in Figure 8C is the same as in Figure 7B, for example, count omissions in each recharge cycle will occur in the same way as in Figure 7B.

[0104] However, due to the long exposure time, both G pixels, which have a relatively large number of missed counts, and R and B pixels, which have a relatively small number of missed counts, tend to increase in count within the exposure time. In other words, all R, G, and B pixels are prone to saturation. Therefore, as with the case where the saturation judgment count in Figure 8A is reduced, the change in color is suppressed.

[0105] As described above, color changes can be suppressed by increasing the exposure time. In other words, the color change is suppressed more in the count obtained from the second pixel than in the first pixel.

[0106] On the other hand, as can be seen from Figure 8C, the count value obtained from the second pixel is more prone to image clipping than the count value obtained from the first pixel. In other words, the image clipping is suppressed more effectively when using the count value obtained from the first pixel than when using the count value obtained from the second pixel.

[0107] Figures 9(A) and 9(B) show the response characteristics of the first and second pixels, respectively, and are log-log graphs with Nph on the horizontal axis and Nct on the vertical axis. The solid line 91 in Figure 9(A) shows the response characteristics of the first pixel, and the dashed line 92 in Figure 9(B) shows the response characteristics of the second pixel when the saturation judgment count is reduced, as in Figure 8A.

[0108] Furthermore, in Figure 9(B), the dotted line 93 represents the response characteristics when the recharge period is shortened, as shown in Figure 8B, and the dashed line 94 represents the response characteristics of the second pixel when the exposure time is extended, as shown in Figure 8C.

[0109] As can be seen by comparing Figure 9(A) and Figure 9(B), the nonlinearity of the response characteristics is mitigated in all three cases: dashed line 92, dotted line 93, and chained line 94. In other words, since the nonlinearity of the response characteristics is caused by the effect of count loss, Figures 9(A) and 9(B) also show that the second pixel suppresses the change in color better than the first pixel.

[0110] Similarly, the response characteristics in Figures 9(A) and 9(B) show that the first pixel has a wider dynamic range on the brighter side than the second pixel. In other words, the first pixel has relatively less overexposure, while the second pixel has relatively less color change.

[0111] Furthermore, as can be seen from Figure 9(B), changing the recharge period as shown by the dotted line 93 results in a wider dynamic range for the second pixel than lowering the saturation detection count as shown by the dashed line 92 or changing the exposure time as shown by the dashed line 94. Note that 92d, 93d, and 94d represent the dynamic ranges corresponding to 92, 93, and 94, respectively. Therefore, changing the recharge period is the most desirable approach.

[0112] However, shortening the recharge cycle increases the power consumption of the APD. Since the power consumption of the APD is proportional to Nct, the most desirable way to prioritize reducing power consumption is to change the saturation detection count.

[0113] On the other hand, when the exposure time is changed (dashed line 94), the slope of Nct relative to Nph changes. (However, since Figure 9 is a logarithmic graph, the dashed line 94 is shifted parallel to the dashed line 92 and the dotted line 93.) Therefore, it is desirable to combine the count value of the first pixel and the count value of the second pixel to generate an image with a wide dynamic range.

[0114] Specifically, for dark subjects, the count value obtained from the second pixel should be used, and for bright subjects, the count value obtained from the first pixel should be used. Furthermore, multiple combinations of saturation detection counts, exposure time, and recharge cycles may be changed between the first and second pixels.

[0115] As mentioned above, the smaller the ratio of the saturation detection count to the maximum recharge count, the more the color change can be suppressed, and the larger the ratio, the more the overexposure can be suppressed.

[0116] Therefore, the greater the difference between the first ratio (the saturation detection count of the first pixel divided by the maximum recharge count) and the second ratio (the saturation detection count of the second pixel divided by the maximum recharge count), the more desirable it is, as this allows for both color shift and suppression of overexposure. Specifically, the second ratio should ideally be half or less of the first ratio, and even more ideally 1 / 8 or less.

[0117] As described above, according to the photoelectric conversion element of Embodiment 1, in an environment where a blinking light source is used as illumination light, a signal with suppressed overexposure can be obtained using the first pixel, and a signal with suppressed color changes can be obtained using the second pixel.

[0118] <Embodiment 2> In a configuration like that of the embodiment, the extent to which a change in color caused by a blinking light source is occurring can be determined by comparing the count values ​​of a first pixel and a second pixel that are located in close proximity to each other.

[0119] Therefore, the photoelectric conversion element (photoelectric conversion device) of Embodiment 2 uses the photoelectric conversion element of Embodiment 1 and is equipped with a determination unit that determines the extent of color change by comparing the count values ​​of a first pixel and a second pixel located in close proximity to each other.

[0120] The determination unit may be located inside the photoelectric conversion element, specifically on the circuit board 21, or it may be located outside the photoelectric conversion element and consist of a dedicated circuit such as a computer program or the signal processing unit 804 described later, or a processor. Furthermore, the determination unit and the photoelectric conversion element do not have to be housed in the same enclosure; they may be composed of separate devices connected to each other via signal paths.

[0121] Figure 10(A) shows an example of the response characteristics of the first pixel. The dashed line 121 shows the response characteristics when a steady light source is used as illumination, and the dotted line 122 shows the response characteristics when a blinking light source is used as illumination. Figure 10(B) shows an example of the response characteristics of the second pixel.

[0122] The solid line 123 in Figure 10(B) represents the case where a steady light source is used as illumination, and the case where a blinking light source is used as illumination; the two cases almost overlap. Figures 10(A) and 10(B) show, for example, the case where the recharge period is changed between the first and second pixels.

[0123] When a steady light source is used as illumination, the response characteristics of the first and second pixels correspond to Equation 1 described above. However, when a blinking light source is used as illumination, the output decreases when the number of incident photons is large due to the effect of counting errors.

[0124] As mentioned above, the second pixel is less affected by counting errors, so as shown in Figure 10(B), the output reduction when using a blinking light source as illumination is small. On the other hand, the first pixel is more affected by counting errors, so as shown in Figure 10(A), the output reduction when using a blinking light source as illumination is large.

[0125] Therefore, if the third ratio NR, calculated using Equation 3 below, which is the ratio of the number of incident photons obtained from the output value of the first pixel to the number of incident photons obtained from the output value of the second pixel, is close to 1, it indicates that steady light is being used as illumination light. Note that the third ratio NR can also be expressed as the ratio of the output value of the first pixel to the output value of the second pixel.

[0126] On the other hand, if the third ratio NR obtained by Equation 3 is less than 1, it indicates that a blinking light source is being used as illumination. In other words, it is possible to determine the change in color based on the ratio of the number of incident photons obtained from the output value of the first pixel to the number of incident photons obtained from the output value of the second pixel. In Equation 3 below, T1 is the recharge period of the first pixel, and T2 is the recharge period of the second pixel.

number

[0127] The third ratio NR may be calculated for each minimum pixel unit, or a single third ratio NR may be calculated for multiple pixel units. Furthermore, as shown in Figure 6, if there are multiple first and second pixels in the minimum pixel unit, or if a single third ratio NR is to be calculated for multiple pixel units, Equation 3 can be derived using the average or median values ​​of the multiple first and second pixels.

[0128] Furthermore, as shown in Figure 6, if the photoelectric conversion element 100 has pixels equipped with on-chip color filters of multiple colors, it is preferable to determine the third ratio NR for each color. From the third ratio NR for each color, it is possible to determine how much count loss occurs in the output value of the pixels of each color.

[0129] Therefore, by determining the third ratio NR for each color and finding the difference in the third ratio for each color, the change in color can be determined with greater accuracy than simply using Equation 3. Specifically, the third ratio NR_R for the R pixel, the third ratio NR_G for the G pixel, and the third ratio NR_B for the B pixel can be determined, and then NR_R / NR_G and NR_B / NR_G can be calculated.

[0130] Thus, it is desirable to accurately determine changes in color by calculating a third ratio for each color: the number of incident photons obtained from the output value of the first pixel to the number of incident photons obtained from the output value of the second pixel.

[0131] Furthermore, it is desirable that the number of first pixels and second pixels within the smallest pixel unit be approximately the same. Specifically, it is desirable that the ratio of the number of second pixels to the number of first pixels within the smallest pixel unit be between 1 / 2 and 2.

[0132] This configuration is desirable because it improves the accuracy of color change detection, as multiple second pixels are arranged near the first pixel, and multiple first pixels are arranged near the second pixels. In particular, it is desirable to arrange the first and second pixels in a staggered pattern, as shown in Figure 6.

[0133] Figure 11A shows an example where the positions of the first and second pixels are swapped in each frame. This configuration is desirable because it can further improve the accuracy of detecting changes in color.

[0134] Specifically, in Figure 11A, at least one of the saturation detection count, exposure time, and recharge cycle for each pixel is changed frame by frame. This allows the positions of the first and second pixels to be swapped frame by frame, improving the accuracy of color change detection while reducing the influence of the subject's pattern.

[0135] <Embodiment 3> The photoelectric conversion element (photoelectric conversion device) of Embodiment 3 has a synthesis processing unit that combines the output values ​​of the first pixel and the second pixel according to the determination result determined by the determination unit of Embodiment 2, and generates a composite image that suppresses overexposure and color changes.

[0136] The synthesis processing unit may be located inside the photoelectric conversion element, specifically on the circuit board 21, or it may consist of the photoelectric conversion element and a dedicated circuit such as a computer program with a determination unit or a signal processing unit 804 described later, or a processor. Furthermore, the determination unit and the photoelectric conversion element do not have to be housed in the same enclosure, and may be composed of separate devices connected to each other via signal paths.

[0137] To generate an image with suppressed overexposure and color changes using judgment information, for example, the following can be done: Specifically, when a steady light source is used as illumination and suppressing overexposure is important for a pixel unit, the output value of the first pixel is used; when a blinking light source is used as illumination and suppressing color changes is important for a pixel unit, the output value of the second pixel is used.

[0138] In other words, if the third ratio NR is above a certain threshold, the output value of the first pixel should be used; if the third ratio NR is below a certain threshold, the output value of the second pixel should be used. Specifically, if the third ratio of the number of incident photons obtained from the output value of the first pixel to the number of incident photons obtained from the output value of the second pixel is above a certain threshold, the output value of the first pixel should be used; if the third ratio is below a certain threshold, the output value of the second pixel should be used. A composite image is then generated.

[0139] As mentioned above, if the third ratio NR is less than, for example, 1, it can be determined that a blinking light source is being used as illumination, so a specific threshold of, for example, 1 can be used. However, because of the influence of errors when deriving the third ratio NR, it is desirable to set the specific threshold to 1 / 2 or greater.

[0140] When performing image synthesis, it is desirable to have a larger number of pixels for which the third ratio NR is calculated. This is because it suppresses errors in the third ratio NR caused by factors such as light shot noise, and improves the accuracy of determining the extent of color changes.

[0141] On the other hand, especially when both continuous light and flashing light sources are used for illumination, the fewer pixels required to determine the third ratio NR, the more precisely it is possible to select which output from the first or second pixel to use for each individual pixel. As a result, the effectiveness of suppressing blown-out highlights and color changes is improved, which is desirable.

[0142] As mentioned above, the number of output signals (count value) Nct of a pixel and the number of photons incident on the pixel Nph are different, and the response characteristics are determined by the ratio of the saturation judgment count to the maximum recharge number. Therefore, it is preferable to perform the synthesis process using values ​​obtained by applying the correction process described in Equation 2 above to the output value of the first pixel and the output value of the second pixel, respectively.

[0143] Specifically, the output value of the first pixel should be corrected using Equation 4, and the output value of the second pixel should be corrected using Equation 5. In other words, it is desirable that the image synthesis processing unit performs different nonlinear correction processes on the output value of the first pixel and the output value of the second pixel before generating the composite image.

number

number

[0144] <Embodiment 4> In Embodiment 4, for example, depending on the determination result determined by the determination unit of Embodiment 2, driving conditions suitable for illumination light are selected by changing at least one of the saturation determination count, exposure time, and recharge cycle of the next frame from which determination information has been acquired.

[0145] In other words, the photoelectric conversion element (photoelectric conversion device) of Embodiment 4 has a selection unit that selects the driving conditions for the next frame from which determination information has been acquired, in accordance with the determination result determined by the determination unit, compared to the photoelectric conversion element of Embodiment 2.

[0146] The selection unit may be located inside the photoelectric conversion element, specifically on the circuit board 21, or it may consist of the photoelectric conversion element and a dedicated circuit such as a computer program having a determination unit or a signal processing unit 804 described later, or a processor. Furthermore, the determination unit and the photoelectric conversion element do not have to be housed in the same enclosure, and may be composed of separate devices connected to each other via a signal path.

[0147] Furthermore, in pixel-level settings where the judgment information indicates that continuous light is being used as illumination, suppressing overexposure is important. Therefore, the driving conditions of the first pixel in the frame from which the judgment information was acquired are used for both the first and second pixels.

[0148] On the other hand, at the pixel level, where it is determined that a blinking light source is being used as illumination based on the judgment information, suppressing changes in color is important. Therefore, the driving conditions of the second pixel in the frame from which the judgment information was acquired are used for both the first and second pixels.

[0149] In other words, if the third ratio NR is greater than or equal to a specific threshold (e.g., 1 / 2), the driving conditions of the first pixel from the frame in which the determination information was acquired are applied to both the first and second pixels in the next frame. The third ratio is, as described above, the ratio of the number of incident photons obtained from the output value of the first pixel to the number of incident photons obtained from the output value of the second pixel.

[0150] On the other hand, if the third ratio NR is less than a specific threshold (e.g., 1 / 2), the driving conditions of the second pixel from the frame in which the judgment information was obtained are applied to both the first and second pixels in the next frame. It is desirable to use a specific threshold of 1 / 2 to 1. In other words, because there is an influence of errors when deriving the third ratio NR, it is desirable to set the specific threshold to 1 / 2 or greater.

[0151] Instead of changing the driving conditions for each pixel in the frame following the acquisition of judgment information, the same driving conditions may be used for all pixels. Specifically, for example, if it is determined from the judgment information of more than half of the pixels that continuous light is being used, it is desirable to use the driving conditions of the first pixel in the frame from which the judgment information was acquired for all pixels.

[0152] On the other hand, if, for example, it is determined from the judgment information of more than half of the pixels that flashing illumination is being used, it is desirable to use the driving condition of the second pixel in the frame from which judgment information has been obtained for all pixels.

[0153] In other words, it is desirable to determine the driving conditions for all pixels in the next frame from which judgment information has been acquired, based on the judgment results of more than half of the pixels, for example. However, the above judgment information is not limited to being acquired from more than half of the pixels. For example, it may be acquired from more than 3 / 4 of the pixels.

[0154] At this time, if the same driving conditions are set for all pixels, it will not be possible to determine the change in color in the relevant frame. Therefore, it is more desirable to use different driving conditions for a small number of pixels than for the many other pixels, as shown in Figure 11B.

[0155] Figure 11B shows an example where different driving conditions are used for a small number of pixels compared to a large number of other pixels. Specifically, as shown in Figure 11B, the ratio of the number of pixels using the first pixel driving condition to the number of pixels using the second pixel driving condition is set to, for example, 1 / 4 or less, or for example, 4 or more.

[0156] As can be seen from Figure 9, the larger the ratio obtained by dividing the saturation detection count by the maximum recharge count, the more white clipping is suppressed, and the smaller the ratio obtained by dividing the saturation detection count by the maximum recharge count, the more color changes are suppressed. Therefore, as the driving conditions for the next frame, you may select a different driving condition other than the driving conditions for the first pixel and the driving conditions for the second pixel of the frame from which the detection information was acquired.

[0157] For example, assuming that the overall illumination intensity of a flashing light source is constant, the shorter the on-time of the flashing light source, the stronger the illumination light becomes at the moment it is on, making it easier for counting errors to occur and resulting in greater color changes.

[0158] Therefore, if the judgment information indicates a significant change in color, it is desirable to make the ratio of the saturation judgment count divided by the maximum recharge count smaller in the next frame than the ratio of the second pixel in the frame from which the judgment information was obtained.

[0159] Furthermore, if the judgment information indicates that the color change is small, it is desirable to set the ratio obtained by dividing the saturation judgment count by the maximum recharge count in the next frame to, for example, an intermediate ratio between the ratios of the first and second pixels in the frame from which the judgment information was obtained.

[0160] <Embodiment 5> Figure 12 is a functional block diagram showing an example configuration of a photoelectric conversion device using the photoelectric conversion elements of Embodiments 1 to 4. Note that some of the functional blocks shown in Figure 12 are realized by having the CPU, which acts as a computer within the photoelectric conversion device, execute a computer program stored in the memory, which acts as a storage medium.

[0161] However, some or all of these may be implemented in hardware. Hardware options include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, each functional block shown in Figure 12 does not necessarily have to be housed in the same enclosure; they may be composed of separate devices connected to each other via signal paths.

[0162] The photoelectric conversion device 800 includes a photoelectric conversion element 100, a control unit 801, a storage unit 802, a communication unit 803, a signal processing unit 804, and an imaging optical system 805. The photoelectric conversion element 100 captures an optical image formed by the imaging optical system 805. The signal read from the photoelectric conversion element 100 is supplied to the signal processing unit 804.

[0163] The signal processing unit 804 performs processing such as black level correction, gamma curve adjustment, noise reduction, data compression, white balance correction, and color conversion. Note that some parts of the signal processing unit 804 may be included in the signal processing circuit 103, and conversely, some parts of the signal processing circuit 103 may be included in the signal processing unit 804.

[0164] The signal processing unit 804 uses equation 2 described above to correct the nonlinearity of the output signal number Nct of the counter circuit 211. This generates the final image. With this configuration, a photoelectric conversion element that can suppress color changes even in environments where a flickering light source is used as illumination can be realized.

[0165] The control unit 801 has a built-in CPU 806, which functions as a control means to control the operation of each part of the entire photoelectric converter 800 based on a computer program stored in the memory 807, which serves as a storage medium.

[0166] Furthermore, the control unit 801 controls the exposure time of each frame of the photoelectric conversion element 100 and the timing of the control signal CLK via the control pulse generation unit 115 of the photoelectric conversion element 100.

[0167] Furthermore, the control unit 801 controls the exposure time of the avalanche photodiode and the recharge period of the avalanche photodiode. In addition, as described above, the control unit 801 controls the setting of a first pixel where the ratio of the saturation determination count to the maximum recharge count is a first ratio, and a second pixel where the ratio of the saturation determination count to the maximum recharge count is a second ratio that is smaller than the first ratio.

[0168] Furthermore, the control unit 801 may determine the degree of color change caused by a blinking light source by comparing the count values ​​of a first pixel and a second pixel located in close proximity to each other. In this case, the control unit 801 functions as a determination unit that performs the above determination.

[0169] Furthermore, the control unit 801 and the signal processing unit 804 may function as a composite processing unit that combines the output values ​​of the first pixel and the second pixel according to the result of the above determination, and generates a composite image that achieves both suppression of overexposure and suppression of color changes.

[0170] Furthermore, the control unit 801 may also control the system to select driving conditions suitable for illumination light by changing at least one of the saturation determination count, exposure time, and recharge cycle for the next frame after acquiring determination information based on the above determination result.

[0171] The storage unit 802 includes, for example, a recording medium such as a memory card or a hard disk. The communication unit 803 has wireless and wired interfaces and outputs the generated image to the outside of the photoelectric converter 800 and receives signals from the outside.

[0172] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible in accordance with the spirit of the present invention, and these are not excluded from the scope of the present invention. Furthermore, some of the above embodiments may be combined as appropriate.

[0173] Furthermore, the present invention includes, for example, a system that realizes the functions of the above embodiment using at least one processor such as a CPU, memory, and circuitry (e.g., an ASIC). Alternatively, multiple processors may be used for distributed processing.

[0174] Furthermore, in order to implement some or all of the control in the above embodiment, a computer program that implements the functions of the above embodiment may be supplied to the photoelectric converter or the like via a network or various storage media.

[0175] Furthermore, the computer (or CPU, MPU, etc.) in the photoelectric conversion, etc., may read and execute the program. In that case, the program and the storage medium in which the program is stored constitute the present invention. The present invention includes the following combinations.

[0176] (Composition 1) A photoelectric conversion device comprising: an avalanche photodiode; a counter that counts and outputs the number of output signals from the avalanche photodiode until it reaches a saturation determination count; and a control unit that controls the exposure time of the avalanche photodiode and the recharge period of the avalanche photodiode, thereby setting a first pixel whose ratio to the saturation determination count with respect to the maximum recharge number, which is determined by the quotient obtained by dividing the exposure time by the recharge period, is a first ratio; and a second pixel whose ratio to the saturation determination count with respect to the maximum recharge number is a second ratio smaller than the first ratio.

[0177] (Configuration 2) The photoelectric conversion device according to Configuration 1, characterized in that the saturation determination count of the second pixel is smaller than the saturation determination count of the first pixel.

[0178] (Configuration 3) The photoelectric conversion device according to Configuration 1 or 2, characterized in that the counter of the second pixel is smaller than the counter of the first pixel.

[0179] (Configuration 4) A photoelectric conversion device according to any one of Configurations 1 to 3, characterized in that the maximum number of recharges of the second pixel is greater than the maximum number of recharges of the first pixel.

[0180] (Configuration 5) The photoelectric conversion device according to Configuration 4, characterized in that the recharge period of the second pixel is shorter than the recharge period of the first pixel.

[0181] (Configuration 6) A photoelectric conversion device according to any one of Configurations 1 to 5, characterized in that it has a switch connected to a power line to which a driving voltage is applied, which is connected to one of the nodes of the avalanche photodiode, the anode or the cathode, and which controls the recharge period by switching the resistance value between the one node and the power line.

[0182] (Configuration 7) The photoelectric converter according to Configuration 6, characterized in that the recharge period of the second pixel is made shorter than the recharge period of the first pixel by changing the frequency of the pulse signal for controlling the switching of the switch.

[0183] (Configuration 8) The photoelectric converter according to Configuration 6 or 7, characterized in that the recharge period of the second pixel is made shorter than the recharge period of the first pixel by masking a portion of the pulse signal for controlling the switching of the switch.

[0184] (Configuration 9) A photoelectric converter according to any one of Configurations 6 to 8, characterized in that the recharge period of the second pixel is made shorter than the recharge period of the first pixel by fixing the logic of the switch of the second pixel.

[0185] (Configuration 10) A photoelectric conversion device according to any one of Configurations 1 to 9, characterized in that the exposure time of the second pixel is longer than the exposure time of the first pixel.

[0186] (Composition 11) A photoelectric conversion device according to any one of configurations 1 to 10, characterized in that the second ratio is less than or equal to half of the first ratio.

[0187] (Configuration 12) The photoelectric conversion device according to Configuration 10, characterized in that the second ratio is 1 / 8 times or less of the first ratio.

[0188] (Configuration 13) A photoelectric conversion device according to any one of Configurations 1 to 12, characterized in that the arrangement of the first pixel and the second pixel is swapped for each frame.

[0189] (Configuration 14) A photoelectric conversion device according to any one of Configurations 1 to 13, characterized in that the first pixel and the second pixel are arranged in a staggered pattern.

[0190] (Configuration 15) A photoelectric conversion device according to any one of Configurations 1 to 14, characterized in that the ratio of the number of second pixels to the number of first pixels is 1 / 2 or more and 2 or less.

[0191] (Configuration 16) A photoelectric conversion device according to any one of Configurations 1 to 15, characterized in that it has a determination unit that determines the degree of color change by comparing the count values ​​of the first pixel and the second pixel.

[0192] (Configuration 17) The photoelectric conversion device according to Configuration 16, characterized in that it determines a change in color based on the ratio of the number of incident photons obtained from the output value of the first pixel to the number of incident photons obtained from the output value of the second pixel.

[0193] (Configuration 18) The photoelectric conversion device according to Configuration 16 or 17, characterized in that it has a synthesis processing unit that synthesizes the output values ​​of the first pixel and the second pixel according to the determination result determined by the determination unit and generates a composite image that suppresses overexposure and color changes.

[0194] (Configuration 19) The photoelectric conversion device according to Configuration 18, characterized in that a composite image is generated by using the output value of the first pixel if the third ratio of the number of incident photons obtained from the output value of the first pixel to the number of incident photons obtained from the output value of the second pixel is greater than or equal to a specific threshold, and by using the output value of the second pixel if the third ratio is less than the specific threshold.

[0195] (Configuration 20) The photoelectric conversion device according to Configuration 19, characterized in that the synthesis processing unit performs different nonlinear correction processing on the output value of the first pixel and the output value of the second pixel, and then generates the synthesized image.

[0196] (Configuration 21) A photoelectric conversion device according to any one of Configurations 16 to 20, characterized in that it has a selection unit that selects the driving conditions for the next frame according to the determination result determined by the determination unit.

[0197] (Configuration 22) The photoelectric conversion device according to Configuration 21, characterized in that if the third ratio of the number of incident photons obtained from the output value of the first pixel to the number of incident photons obtained from the output value of the second pixel is greater than or equal to a specific threshold, the driving conditions of the first pixel in the frame from which the determination result was obtained are applied to both the first pixel and the second pixel in the next frame, and if the third ratio is less than the specific threshold, the driving conditions of the second pixel in the frame from which the determination result was obtained are applied to both the first pixel and the second pixel in the next frame.

[0198] (Configuration 23) A photoelectric conversion device according to Configuration 21 or 22, characterized in that the driving conditions for all pixels of the next frame are determined based on the determination results of more than half of the pixels.

[0199] (Configuration 24) A photoelectric conversion device according to any one of Configurations 21 to 23, characterized in that the ratio of the number of pixels to which the driving conditions for the first pixels are applied to the number of pixels to which the driving conditions for the second pixels are applied is 1 / 4 or less, or 4 or more.

[0200] (Configuration 25) A photoelectric converter according to any one of Configurations 1 to 24, characterized in that each pixel is provided with an on-chip color filter.

[0201] (Configuration 26) The photoelectric conversion device according to Configuration 25, characterized in that multiple types of on-chip color filters are arranged in each of the multiple first pixels, and multiple types of on-chip color filters are arranged in each of the multiple second pixels.

[0202] (Configuration 27) A photoelectric conversion device according to Configuration 25 or 26, characterized in that a change in color is determined by calculating the ratio of the number of incident photons obtained from the output value of the first pixel to the number of incident photons obtained from the output value of the second pixel for each color.

[0203] (Method) A photoelectric conversion method for controlling a photoelectric conversion element having an avalanche photodiode and a counter that counts and outputs the number of output signals from the avalanche photodiode until it reaches a saturation determination count, By controlling the exposure time of the avalanche photodiode and the recharge period of the avalanche photodiode, A photoelectric conversion method characterized by setting a first pixel whose ratio of the saturation determination count to the maximum recharge number, which is determined by the quotient obtained by dividing the exposure time by the recharge period, is a first ratio, and a second pixel whose ratio of the saturation determination count to the maximum recharge number is a second ratio smaller than the first ratio.

[0204] (Program) A computer program for controlling each part of the photoelectric converter described in any one of configurations 1 to 27. [Explanation of symbols]

[0205] 100: Photoelectric conversion element 11: Sensor board 12: Pixel area 13: Minimum pixel unit 21: Circuit board 22: Circuit area 101: Pixels 101a: First pixel 101b: Second pixel 102: Photoelectric conversion unit 103: Signal Processing Circuit 110: Vertical scanning circuit 111: Horizontal scanning circuit 112: Readout circuit 113: Signal line 114: Output Circuit 115: Control pulse generation unit 201: Avalanche photodiode 202: Switch 210: Waveform shaping section 211: Counter circuit 212: Selection Circuit 213: Drive line 214: Drive line 215: Saturation judgment section 800: Photoelectric converter 801: Control Unit 802: Storage section 803: Communications Department 804: Signal Processing Unit 805: Imaging Optical System 806:CPU 807: Memory

Claims

1. Avalanche photodiode and A counter that counts and outputs the number of output signals from the avalanche photodiode until it reaches a saturation count, By controlling the exposure time of the avalanche photodiode and the recharge period of the avalanche photodiode, A control unit sets a first pixel whose ratio of the saturation determination count to the maximum recharge count, which is determined by the quotient obtained by dividing the exposure time by the recharge cycle, is a first ratio, and a second pixel whose ratio of the saturation determination count to the maximum recharge count is a second ratio which is smaller than the first ratio. A photoelectric conversion device characterized by having the following features.

2. The photoelectric conversion device according to claim 1, characterized in that the saturation determination count of the second pixel is smaller than the saturation determination count of the first pixel.

3. The photoelectric conversion device according to claim 1, characterized in that the counter of the second pixel is smaller than the counter of the first pixel.

4. The photoelectric conversion device according to claim 1, characterized in that the maximum recharge number of the second pixel is greater than the maximum recharge number of the first pixel.

5. The photoelectric conversion device according to claim 4, characterized in that the recharge period of the second pixel is shorter than the recharge period of the first pixel.

6. The photoelectric conversion device according to claim 1, characterized in that it has a switch connected to one of the nodes of the avalanche photodiode, either the anode or the cathode, and to a power line to which a driving voltage is applied, and the switch controls the recharge period by switching the resistance value between the one node and the power line.

7. The photoelectric converter according to claim 6, characterized in that the recharge period of the second pixel is made shorter than the recharge period of the first pixel by changing the frequency of the pulse signal for controlling the switching of the switch.

8. The photoelectric converter according to claim 6, characterized in that the recharge period of the second pixel is made shorter than the recharge period of the first pixel by masking a portion of the pulse signal for controlling the switching of the switch.

9. The photoelectric converter according to claim 6, characterized in that the recharge period of the second pixel is made shorter than the recharge period of the first pixel by fixing the logic of the switch of the second pixel.

10. The photoelectric conversion device according to claim 1, characterized in that the exposure time of the second pixel is longer than the exposure time of the first pixel.

11. The photoelectric conversion device according to claim 1, characterized in that the second ratio is less than or equal to half of the first ratio.

12. The photoelectric conversion device according to claim 10, characterized in that the second ratio is 1 / 8 times or less of the first ratio.

13. The photoelectric conversion device according to claim 1, characterized in that the arrangement of the first pixel and the second pixel is swapped for each frame.

14. The photoelectric conversion device according to claim 1, characterized in that the first pixel and the second pixel are arranged in a staggered pattern.

15. The photoelectric conversion device according to claim 1, characterized in that the ratio of the number of second pixels to the number of first pixels is 1 / 2 or more and 2 or less.

16. The photoelectric conversion device according to claim 1, further comprising a determination unit that determines the degree of color change by comparing the count values ​​of the first pixel and the second pixel.

17. The photoelectric conversion device according to claim 16, characterized in that it determines the change in color based on the ratio of the number of incident photons obtained from the output value of the first pixel to the number of incident photons obtained from the output value of the second pixel.

18. The photoelectric conversion apparatus according to claim 16, further comprising a synthesis processing unit that synthesizes the output values ​​of the first pixel and the second pixel according to the determination result determined by the determination unit, and generates a composite image that suppresses overexposure and color changes.

19. The photoelectric conversion apparatus according to claim 18, characterized in that a composite image is generated by using the output value of the first pixel if the third ratio of the number of incident photons obtained from the output value of the first pixel to the number of incident photons obtained from the output value of the second pixel is greater than or equal to a specific threshold, and by using the output value of the second pixel if the third ratio is less than the specific threshold.

20. The photoelectric conversion apparatus according to claim 19, characterized in that the synthesis processing unit performs different nonlinear correction processing on the output value of the first pixel and the output value of the second pixel, and then generates the synthesized image.

21. The photoelectric conversion device according to claim 16, further comprising a selection unit that selects the driving conditions for the next frame according to the determination result determined by the determination unit.

22. The photoelectric converter according to claim 21, characterized in that, if a third ratio of the number of incident photons obtained from the output value of the first pixel to the number of incident photons obtained from the output value of the second pixel is greater than or equal to a specific threshold, the driving conditions for the first pixel of the frame from which the determination result was obtained are applied to both the first pixel and the second pixel in the next frame, and if the third ratio is less than the specific threshold, the driving conditions for the second pixel of the frame from which the determination result was obtained are applied to both the first pixel and the second pixel in the next frame.

23. The photoelectric conversion device according to claim 21, characterized in that the driving conditions for all pixels of the next frame are determined based on the determination results of more than half of the pixels.

24. The photoelectric conversion device according to claim 21, characterized in that the ratio of the number of pixels to which the first pixel driving condition is applied to the number of pixels to which the second pixel driving condition is applied is 1 / 4 or less, or 4 or more.

25. The photoelectric converter according to claim 1, characterized in that each pixel is provided with an on-chip color filter.

26. The photoelectric conversion device according to claim 25, characterized in that multiple types of on-chip color filters are arranged in each of the multiple first pixels, and multiple types of on-chip color filters are arranged in each of the multiple second pixels.

27. The photoelectric conversion device according to claim 25, characterized in that it determines the change in color by determining the ratio of the number of incident photons obtained from the output value of the first pixel to the number of incident photons obtained from the output value of the second pixel for each color.

28. A photoelectric conversion method for controlling a photoelectric conversion element having an avalanche photodiode and a counter that counts and outputs the number of output signals from the avalanche photodiode until it reaches a saturation determination count, By controlling the exposure time of the avalanche photodiode and the recharge period of the avalanche photodiode, A photoelectric conversion method characterized by setting a first pixel whose ratio of the saturation determination count to the maximum recharge number, which is determined by the quotient obtained by dividing the exposure time by the recharge period, is a first ratio, and a second pixel whose ratio of the saturation determination count to the maximum recharge number is a second ratio smaller than the first ratio.

29. A computer program for controlling each part of the photoelectric converter according to any one of claims 1 to 27 by computer.

Images