Imaging device, imaging method, and computer program

Abstract

The present invention provides an imaging device that can suppress image quality degradation during the recharge period of an avalanche photodiode. [Solution] Each pixel comprises a photoelectric conversion element having a sensor unit that emits pulses in response to photons incident on an avalanche photodiode, a counter that counts the number of pulses, a memory that stores the count value of the counter, and a switch that switches the avalanche photodiode between a standby state capable of avalanche multiplication and a recharge state; a signal generation unit that supplies a clock signal to the switch; a light emission unit that emits pulsed light to illuminate a subject in synchronization with the clock signal; and a control unit that performs multiple exposure operations by the counter according to the timing of pulse emission and the predetermined imaging distance range in order to image a subject that is within a predetermined imaging distance range, and shifts the relative timing of the clock signal and pulse emission by a predetermined phase for each predetermined exposure operation.

Description

Technical Field

[0001] The present invention relates to an imaging device, an imaging method, a computer program, and the like.

Background Art

[0002] A camera called a range-gated camera is known. That is, it is a technology that emits pulsed light in front of the camera at a predetermined cycle, and the image sensor inside the camera is exposed at a predetermined timing according to the target distance range, so that only the subject in the target distance range can be clearly imaged.

[0003] Hereinafter, such a technology is called range-gated control. By using this range-gated control, for example, even in bad weather, a subject (object) at a predetermined distance can be clearly imaged.

[0004] In addition, mounting an SPAD (Single Photon Avalanche Diode) sensor with excellent low-light performance on the above range-gated camera has been considered. However, in the SPAD sensor, when using the conventional passive charging method, when light enters the photodiode under high brightness, for example, there may be a case where photons enter the photodiode at the timing of recharging the photodiode or immediately after that.

[0005] In this case, the potential of the photon counting part remains in the state where photons are detected and does not change. As a result, this period will not be counted as the period when photons are obtained. Therefore, in a high-brightness environment, the actual coefficient value becomes smaller than the coefficient value with respect to the brightness of the incident light, and the brightness of the image becomes lower than the original brightness.

[0006] In contrast, Patent Document 1 proposes a driving method called clock decharging for using a SPAD sensor under high brightness conditions. Patent Document 1 discloses a pixel having an APD, an APD, a quench circuit connected to the APD, a signal control circuit that receives the signal output from the APD, and a pulse generation circuit connected to the quench circuit and the signal control circuit.

[0007] The pulse generation circuit controls the on / off state of the quench circuit. Furthermore, it is disclosed that the potential of the APD's output node is reset to output a pulse signal corresponding to the input photon even under high brightness conditions. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] Japanese Patent Publication No. 2020-123847 [Overview of the project] [Problems that the invention aims to solve]

[0009] However, in the configuration described in Patent Document 1, photons entering during the recharge period will not be detected. Since the recharge period is usually very short, typically a few nanoseconds to tens of nanoseconds, this is not a problem in normal shooting modes.

[0010] However, in range gate cameras, light with an emission time of several nanoseconds to tens of nanoseconds is counted within an exposure period (photon counting period) of several nanoseconds to tens of nanoseconds. As a result, reflected light from the portion that passed through during the recharging period cannot be exposed. In other words, there is a problem in that reflected light from a specific distance range cannot be imaged, leading to a deterioration in image quality.

[0011] Therefore, one of the objectives of the present invention is to provide an imaging device that can suppress image quality degradation during the recharge period of an avalanche photodiode. [Means for solving the problem]

[0012] In an imaging device, A photoelectric conversion element having multiple pixels, wherein the pixels are A sensor unit that emits pulses in response to photons incident on an avalanche photodiode, A counter for counting the number of pulses, A memory for storing the count value of the aforementioned counter, The avalanche photodiode is equipped with a switch that allows switching between a standby state capable of avalanche multiplication and a recharge state, and the photoelectric conversion element is equipped with a switch that allows switching between these states. A signal generation unit that supplies a clock signal to the switch, A light-emitting unit that emits pulsed light in synchronization with the aforementioned clock signal to illuminate the subject, In order to image a subject located within a predetermined imaging distance range, the counter performs multiple exposure operations according to the timing of the pulse emission and the predetermined imaging distance range, A control unit that shifts the relative timing of the clock signal and the pulse emission by a predetermined phase for each predetermined exposure operation, It is characterized by having the following features. [Effects of the Invention]

[0013] According to the present invention, it is possible to provide an imaging device that can suppress image quality degradation during the recharge period of an avalanche photodiode. [Brief explanation of the drawing]

[0014] [Figure 1] This figure shows an example configuration of the photoelectric conversion element 100 in Embodiment 1 of the present invention. [Figure 2] This figure shows an example configuration of the sensor board 11. [Figure 3] This figure shows an example of the configuration of the circuit board 21. [Figure 4] Figure 2 shows the equivalent circuits of the photoelectric conversion unit 102 for each pixel and the signal processing circuit 103 corresponding to the photoelectric conversion unit 102. [Figure 5]It is a diagram schematically showing the relationship between the operation of the clock charging method and the output signal. [Figure 6] It is a functional block diagram showing a configuration example of the light emitter 500, the camera 600, and the mobile body 700 according to Embodiment 1. [Figure 7] It is a diagram showing an example of the relationship between the emitted light from the light emitter 500, the progress of the reflected light thereof, and the exposure timing of the camera 600 according to Embodiment 1. [Figure 8] It is a timing chart explaining the control operation for obtaining a range gate image in one frame period according to Embodiment 1. [Figure 9] (A) and (B) are diagrams for explaining the relationship between the control signal CLK and the exposure timing according to Embodiment 1. [Figure 10] It is a diagram showing an example of the relationship between the light emission and the control signal CLK at each exposure timing according to Embodiment 1. [Figure 11] It is a flowchart showing details of an operation example of the imaging method according to Embodiment 1. [Figure 12] (A) and (B) are diagrams showing a configuration example of an image sensor having two photoelectric conversion parts in one pixel according to Embodiment 2. [Figure 13] It is a diagram showing an equivalent circuit of a signal processing circuit corresponding to one pixel in the pixel group 1000 in FIG. 12. [Figure 14] (A) to (C) are diagrams for explaining an example of an operation mode when using an image sensor having two photoelectric conversion parts in one pixel. [Figure 15] It is a diagram showing a problem when counting both the first photoelectric conversion signal and the second photoelectric conversion signal. [Figure 16] It is a diagram showing an example of a method for solving the problem explained in FIG. 15.

Mode for Carrying Out the Invention

[0015] Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to the following embodiments. In each drawing, the same reference numeral is used for the same member or element, and redundant explanations are omitted or simplified.

[0016] <Embodiment 1> Figure 1 shows an example of the configuration of a photoelectric conversion element 100 according to Embodiment 1 of the present invention. In the following description, a photoelectric conversion device having a so-called stacked structure in which a sensor substrate 11 and a circuit substrate 21 are stacked and electrically connected will be used as an example.

[0017] However, the configurations included in the sensor substrate 11 and the circuit board 21 may be arranged on a common semiconductor layer, resulting in a so-called non-stacked structure. The sensor substrate 11 includes a pixel region 12, and the circuit board 21 includes a circuit region 22 for processing signals detected in the pixel region 12.

[0018] Figure 2 shows an example of the configuration of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across multiple rows and columns. That is, the photoelectric conversion element 100 has a plurality of pixels. Each pixel 101 includes a photoelectric conversion unit 102 which includes an avalanche photodiode (hereinafter referred to as APD).

[0019] Here, the photoelectric conversion unit 102 functions as a sensor unit that emits pulses in response to photons incident on the avalanche photodiode. The number of rows and columns of the pixel array constituting the pixel region 12 are not particularly limited.

[0020] Figure 3 shows an example of the configuration of the circuit board 21. The circuit board 21 has a signal processing circuit 103 for processing the charge photoelectrically converted by each photoelectric conversion unit 102 in Figure 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114.

[0021] The vertical scanning circuit 110 receives control pulses supplied from the control pulse generation unit 115 and sequentially supplies control pulses to multiple pixels arranged in the row direction, row by row. Logic circuits such as a shift register and an address decoder are used in the vertical scanning circuit 110.

[0022] The photoelectric conversion signals output from the photoelectric conversion unit 102 of each pixel are processed by each signal processing circuit 103. The signal processing circuit 103 is equipped with a counter and memory, and digital values ​​are stored in the memory. The horizontal scanning circuit 111 inputs control pulses to the signal processing circuit 103 to sequentially select each column in order to read the signal from the memory of each pixel in which the digital signal is stored.

[0023] Signals are output to the vertical signal line 113 from multiple signal processing circuits 103 for multiple pixels in a row selected by the vertical scanning circuit 110. The signals output to the vertical signal line 113 are output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has multiple buffers built in that are connected to the vertical signal line 113.

[0024] As shown in Figures 2 and 3, multiple signal processing circuits 103 are arranged in the region that overlaps with the pixel region 12 in a plan view. Then, a vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generation unit 115 are arranged so as to overlap with the peripheral area of ​​the pixel region 12 in a plan view.

[0025] Specifically, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generation unit 115 are arranged in the region that overlaps with the non-pixel region in a plan view.

[0026] Furthermore, the arrangement of the vertical signal line 113, the readout circuit 112, and the output circuit 114 is not limited to the example shown in Figure 3. For example, the vertical signal line 113 may extend in the row direction, and the readout circuit 112 may be placed at the end of the vertical signal line 113. Also, the signal processing circuit 103 does not necessarily need to be provided for each photoelectric conversion unit; a single signal processing unit may be shared by multiple photoelectric conversion units, and sequential signal processing may be performed.

[0027] Figure 4 shows the equivalent circuits of the photoelectric conversion unit 102 for each pixel in Figure 2 and the signal processing circuit 103 corresponding to the photoelectric conversion unit 102.

[0028] The APD201 included in the photoelectric conversion unit 102 generates charge pairs corresponding to incident light through photoelectric conversion. One of the two nodes of the APD201 is connected to a power line to which a drive voltage VL (first voltage) is supplied. The other of the two nodes of the APD201 is connected via switch 202 to a power line to which a drive voltage VH (second voltage), which is higher than the drive voltage VL, is supplied.

[0029] In Figure 4, one node of the APD201 is the anode, and the other node of the APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD201 so that the APD201 performs avalanche multiplication. By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, and an avalanche current is generated.

[0030] Furthermore, when a reverse bias voltage is supplied, there are two modes: Geiger mode, in which the device operates when the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, in which the device operates when the voltage difference between the anode and cathode is near or below the breakdown voltage.

[0031] An APD that operates in Geiger mode is called a SPAD. In the case of a SPAD, for example, the drive voltage VL (first voltage) is -30V and the drive voltage VH (second voltage) is 1V. Note that SPADs are a type of APD.

[0032] The signal processing circuit 103 includes a switch 202, a waveform shaping unit 210, a counter circuit 211, a memory circuit 212, and a signal generation unit 215. The switch 202 is connected to a power line to which a drive voltage VH is supplied and to one of the nodes, either the anode or the cathode, of the APD 201.

[0033] The switch 202 then switches the resistance value between the APD201 and the power line to which the drive voltage VH is supplied. When switching the resistance value, it is preferable to change the resistance value by a factor of 10 or more, and more preferable to change the resistance value by a factor of 100 or more.

[0034] In the following, a decrease in the resistance value of switch 202 will be referred to as "switch 202 being on," and a increase in that resistance value will be referred to as "switch 202 being off." Switch 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, suppressing the voltage supplied to APD201 and performing a quench operation to suppress avalanche multiplication.

[0035] Furthermore, switch 202 can perform a recharge operation, which restores the voltage supplied to APD201 to the drive voltage VH by supplying current to compensate for the voltage drop caused by the quench operation. In other words, switch 202 has the function of switching the avalanche photodiode between a standby state in which avalanche multiplication is possible and a recharge state.

[0036] The switch 202 can be constructed using, for example, a MOS transistor, and Figure 4 shows the case where the switch 202 is a PMOS transistor. The control signal CLK (clock signal) for the switch 202 supplied from the signal generation unit 215 is applied to the gate electrode of the MOS transistor constituting the switch 202. In this embodiment, the on and off states of the switch 202 are controlled by controlling the voltage applied to the gate electrode of the switch 202.

[0037] Figure 4 shows an example where the signal processing circuit 103 includes a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212 in addition to the switch 202. The counter circuit 211 counts pulses from the waveform shaping unit 210, and the memory circuit 212 can store the count value of the counter circuit 211. The counter circuit 211 functions as a counter that counts the number of pulses.

[0038] The waveform shaping unit 210 shapes the cathode voltage change of the APD201 obtained during photon detection and outputs a pulse signal. The input node of the waveform shaping unit 210 is nodeA, and the output node is nodeB. The waveform shaping unit 210 changes the output voltage from nodeB depending on whether the input voltage to nodeA is above or below a predetermined value.

[0039] When the input voltage to node A becomes higher than the threshold voltage, the output voltage from node B becomes low. Conversely, when the input voltage to node A becomes lower than the threshold voltage, the output voltage from node B becomes high. For example, an inverter circuit can be used as the waveform shaping unit 210.

[0040] Figure 4 shows an example where one inverter is used as the waveform shaping unit 210, but a circuit with multiple inverters connected in series may also be used, or other circuits that have a waveform shaping effect may be used.

[0041] First, let's explain the passive recharging method. In response to the avalanche multiplication in APD201, it is possible to perform quench and recharge operations using switch 202, but depending on the timing of photon detection, it may not be recognized as an output signal.

[0042] For example, consider a scenario where avalanche multiplication occurs in the APD, causing the input voltage to nodeA to become low, and a recharge operation is performed. Note that the judgment threshold of the waveform shaping unit 210 is set to a voltage higher than the voltage difference at which avalanche multiplication occurs in the APD.

[0043] When a photon is incident while the voltage of node A is below the threshold due to the recharge operation and the voltage is within the range where avalanche multiplication is possible at the APD, avalanche multiplication occurs at the APD and the voltage of node A decreases.

[0044] In other words, because the voltage at nodeA drops below the threshold voltage, the output voltage from nodeB does not change despite the detection of photons. Consequently, even though avalanche multiplication is occurring, the count value of counter circuit 211 does not increase.

[0045] In particular, under high illumination conditions, photons enter continuously in short intervals, making it difficult for the count value to increase. As a result, even under high illumination conditions, there is a tendency for the actual number of incident photons and the count value to diverge.

[0046] In contrast, in the clock recharging method according to this embodiment, as shown in Figure 5 described later, the control signal CLK (clock signal) shown in Figure 5 is applied to the switch 202 to periodically switch the switch 202 on and off. This makes it possible to determine the signal even when photons enter the APD continuously in a short period of time.

[0047] As described above, the counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and stores the count value. Furthermore, when the control pulse RES is supplied via the RES signal line 213, the count value of the counter circuit 211 is stored in the memory circuit 212 and the counter value of the counter circuit 211 is reset.

[0048] Here, the counter circuit 211 counts up from the start to the end of the exposure period (accumulation period). In this embodiment, there are multiple exposure periods in one frame period, and the count value for each exposure period is accumulated and added within one frame period.

[0049] The memory circuit 212 receives a control pulse SEL from the vertical scanning circuit 110 in Figure 3 via the drive line 214 (not shown in Figure 3) in Figure 4, which switches the electrical connection between the memory circuit 212 and the vertical signal line 113.

[0050] The memory circuit 212 functions as a memory that temporarily stores the counter's count value. After temporarily storing the output signal from the pixel's counter circuit 211, it outputs it to the vertical signal line 113 when the control pulse SEL is supplied.

[0051] Furthermore, switches such as transistors may be placed between switch 202 and APD201, or between photoelectric conversion unit 102 and signal processing circuit 103, to switch the electrical connections. Similarly, the supply of the drive voltage VH or drive voltage VL supplied to the photoelectric conversion unit 102 may be electrically switched using switches such as transistors.

[0052] Furthermore, as shown in Figure 4, it is preferable to configure the switch 202 with a single transistor and perform both the quench operation and the recharge operation with that single transistor. This makes it possible to reduce the number of circuits compared to when the quench operation and the recharge operation are performed by different circuit elements.

[0053] In particular, when each pixel has a counter circuit and reads out the SPAD signal for each pixel, it is preferable to reduce the circuit area for switch 202 in order to arrange the counter circuit, and the effect of configuring switch 202 with a single transistor becomes significant.

[0054] Figure 5 schematically illustrates the relationship between the operation of the clock recharging method and the output signal. In Figure 5, an example is shown where the control signal CLK, which is the clock signal, is a pulse signal with a repeating period. That is, in Figure 5, the on / off state of switch 202 is switched at a predetermined clock frequency.

[0055] Furthermore, Figure 5 schematically shows the relationship between the control signal CLK of switch 202, the voltage at node A, the voltage at node B, and the output signal. In this embodiment, when the control signal CLK is at a high level, the drive voltage VH is less likely to be supplied to the APD, and when the control signal CLK is at a low level, the drive voltage VH is supplied to the APD.

[0056] In other words, a high level control signal CLK is, for example, 1V, and a low level control signal CLK is, for example, 0V. Switch 202 is turned off when the control signal CLK is high, and turned on when the control signal CLK is low.

[0057] When the control signal CLK is at a high level, the resistance of switch 202 is higher than when the control signal CLK is at a low level. Therefore, when the control signal CLK is at a high level, even if avalanche multiplication occurs in the APD, no recharge operation is performed, the voltage supplied to the APD becomes below the breakdown voltage of the APD, and the avalanche multiplication operation in the APD stops.

[0058] At time tA in Figure 5, the control signal CLK changes from a high level to a low level, switching 202 on and initiating the APD recharge operation. This causes the APD cathode voltage to transition to a high level.

[0059] Then, the voltage difference between the anode and cathode of the APD becomes greater than or equal to the breakdown voltage, and the device enters a state where avalanche multiplication is possible. Since the cathode voltage is the same as that of node A, when the cathode voltage transitions from a low level to a high level, the voltage of node A becomes greater than or equal to the threshold at time tB.

[0060] At this point, the pulse signal output from nodeB inverts, changing from a high level to a low level. Subsequently, the voltage difference between the drive voltage VH and the drive voltage VL is applied to APD201. After that, the control signal CLK becomes high, and switch 202 turns off.

[0061] Next, at time tC, as shown by the black circle and downward arrow in Figure 5, when a photon is incident on APD201, avalanche multiplication occurs in APD201, an avalanche multiplication current flows through switch 202, and the cathode voltage drops. In other words, the voltage at node A drops.

[0062] As the voltage drop increases further and the voltage difference applied to APD201 decreases, the avalanche multiplication of APD201 stops, as at time tC, and the voltage level of node A will no longer drop below a certain value. If the voltage of node A falls below the threshold during the voltage drop, the voltage of node B will move from a low level to a high level.

[0063] In other words, if the output waveform at nodeA falls below the judgment threshold, the waveform is shaped by the waveform shaping unit 210 and output as a high-level signal at nodeB. The rising edge of nodeB is then counted by the counter circuit, and the count value of the counter signal output from the counter circuit increases by 1 LSB to n+1.

[0064] In this way, each time the rising edge of the nodeB waveform occurs in accordance with the control signal CLK, the count value of the counter circuit 211 increases. Finally, at the end of the predetermined exposure period, the counter circuit 211 generates a final count value.

[0065] In Figure 5, photons are incident on the APD between time tC and time tD, but since switch 202 is in the off state, the voltage applied to APD201 is not a voltage difference that allows for avalanche multiplication, and therefore the voltage level of node A does not exceed the judgment threshold.

[0066] At time tD, the control signal CLK changes from a high level to a low level, and switch 202 turns on. Consequently, a current flows through node A to compensate for the voltage drop from the drive voltage VL, and the voltage at node A returns to its original voltage level.

[0067] At this time, the voltage of node A exceeds the threshold at time tE, causing the pulse signal of node B to invert and change from a high level to a low level.

[0068] At time tF, node A settles back to its original voltage level, and then the control signal CLK changes from a low level to a high level. Consequently, switch 202 is turned off. Subsequently, as explained from time tA to time tF, the voltages of each node and signal line change in response to the control signal CLK and the incidence of photons.

[0069] Next, the imaging device of this embodiment, consisting of a light emitter 500, a camera 600, and a mobile unit 700, will be described.

[0070] Figure 6 is a functional block diagram showing an example configuration of the light emitter 500, camera 600, and mobile body 700 according to Embodiment 1. Note that some of the functional blocks shown in Figure 6 are realized by having the computers (not shown) included in the light emitter 500, camera 600, and mobile body 700, respectively, execute computer programs stored in memory (not shown) which is a storage medium.

[0071] However, some or all of these may be implemented in hardware. Hardware options include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, each functional block shown in Figure 6 does not necessarily have to be housed in the same enclosure; they may be composed of separate devices connected to each other via signal paths.

[0072] The camera 600 includes a photoelectric conversion element 100, an imaging optical system 601, an image processing unit 603, a recognition unit 604, a camera control unit 605, a memory unit 606, a communication unit 607, etc. The photoelectric conversion element 100 is composed of an avalanche photodiode, as described in Figures 1 to 5, for photoelectric conversion of an optical image.

[0073] The imaging device (camera 600 and light emitter 500) of this embodiment is mounted on a mobile body 700, and the imaging unit 602, which consists of an imaging optical system 601 and a photoelectric conversion element 100, is configured to capture images in at least one direction, such as the front, rear, or side of the mobile body. Multiple imaging units 602 may be provided on the mobile body 700, or multiple imaging devices may be provided on the mobile body 700.

[0074] The image processing unit 603 performs image processing on the image signal acquired by the photoelectric conversion element 100, such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaicing, and data compression, to generate the final image signal. If the photoelectric conversion element 100 has an on-chip color filter such as RGB, the image processing unit 603 may also perform processing such as white balance correction and color conversion.

[0075] Furthermore, the output of the image processing unit 603 is supplied to the recognition unit 604 and the camera control unit 605, as well as to the ECU (Electric Control Unit) 701 of the mobile unit 700. The recognition unit 604 performs image recognition based on the image signal to recognize surrounding objects such as people and vehicles. Deep learning is used for this recognition process.

[0076] For example, it is preferable to use YOLO (You Only Look Once) as a deep learning model, as it is easy to train and has fast detection speed. Alternatively, other deep learning models such as SSD (Single Shot Multi Box Detector), FasterR-CNN (Regional Convolution Neural Network), FastR-CNN, and R-CNN may also be used.

[0077] In this embodiment, the recognition unit 604 calculates the distance to the recognized object. That is, the recognition unit 604 calculates, for example, a first distance range and a second distance range by recognizing the subject. As a distance measurement method, distance estimation may be performed using deep learning, for example. That is, for example, the distance value may be calculated by analyzing information such as the blur of the image of the detected object using deep learning.

[0078] Alternatively, the imaging device may be a stereo camera, and the distance may be measured using the principle of triangulation. Or, the photoelectric conversion element may be a phase difference detection type image sensor, and the distance may be measured using the phase difference signal from the photoelectric conversion element. Recognition processing, including distance estimation, is performed on the color image and IR (infrared) image input from the image processing unit 603, and the recognition results are output to the subsequent ECU 701.

[0079] In this embodiment, the mobile body 700 is described using the example of an automobile, but the mobile body can be anything that is mobile, such as an aircraft, train, ship, drone, AGV, or robot.

[0080] The camera control unit 605 has a built-in CPU and memory that stores computer programs, and the CPU controls various parts of the camera 600 by executing the computer programs stored in the memory.

[0081] Furthermore, the camera control unit 605 functions as a control means, for example, via the control pulse generation unit of the photoelectric conversion element 100, it controls the length of the exposure period (accumulation period, count period in the counter) for each frame of the photoelectric conversion element 100, as well as the timing of the control signal. In addition, the camera control unit 605 also transmits the same signal as the aforementioned reference signal to the light emitter 500 via the communication unit 607.

[0082] The same reference signal transmitted to the photoelectric conversion element 100 is also transmitted to the light emitter 500, and the light emitter 500 performs light emission control based on the reference signal, thereby synchronizing the exposure timing inside the photoelectric conversion element 100 with the light emission timing by the light emitter 500.

[0083] The storage unit 606 includes, for example, a recording medium such as a memory card or hard disk, and can store and read image signals. The communication unit 607 has wireless and wired interfaces and outputs the generated image signals to the outside of the camera 600 and receives various signals from the outside.

[0084] Furthermore, in this embodiment, the communication unit 607 is connected to the communication unit 503 of the light emitter 500 and is also responsible for transmitting the aforementioned reference signal and control commands from the camera control unit 605 to the light emitter 500.

[0085] The light emitter 500 includes a light-emitting unit 501, a light-emitting control unit 502, and a communication unit 503. The light-emitting unit 501 includes, for example, a near-infrared LED for illuminating a subject in front of the moving body 700, and is combined with a lens to emit a beam of light. The light-emitting unit also outputs near-infrared pulsed light for a predetermined emission time in response to a pulse signal output from the light-emitting control unit 502.

[0086] The light emission control unit 502 receives a reference signal transmitted by the camera control unit 605 of the camera 600 via the communication unit 503, generates a pulse signal at a predetermined timing using the reference signal as a reference, and outputs it to the light emission unit 501.

[0087] Here, the light emission control unit 502 can set the period from the reference signal to the output of a pulse, the pulse output width, the pulse non-output width, and the repetition period and number of repetitions from one pulse output to the next.

[0088] The camera control unit 605 sets a predetermined value to the light emission control unit 502 via the communication unit 607 and the communication unit 503, so that a pulse signal is output to the light emission unit 501 at a predetermined timing based on the reference signal, and the light emission period of the light emitter 500 is controlled.

[0089] In this way, the light emission control unit 502 controls the light emission using the same signal as the reference signal input to the photoelectric conversion element 100 as a reference. That is, the light emission unit 501 emits pulsed light in synchronization with the control signal CLK, which is a clock signal, to illuminate the subject.

[0090] The communication unit 503 communicates with the communication unit 607 of the camera 600, receives setting information and reference signals from the camera control unit 605 to the light emission control unit 502, and transmits them to the light emission unit 501.

[0091] The ECU701 has a built-in CPU and memory that stores computer programs, and the CPU executes the computer programs stored in the memory to control various parts of the mobile unit 700.

[0092] The output of the ECU 701 is supplied to the vehicle control unit 702 and the display unit 703. The vehicle control unit 702 functions as a motion control means that controls the driving, stopping, and direction of the vehicle as a moving body based on the output of the ECU 701. The display unit 703 functions as a display means and includes a display element such as a liquid crystal device or an organic EL, and is mounted on the moving body 700.

[0093] In this embodiment, the ECU 701 receives recognition result information from the recognition unit 604 and can execute vehicle stopping control (such as automatic braking) according to the content of the recognition result. The ECU 701 also receives a color image and an IR image from the image processing unit 603 and transmits them to the display unit 703 along with the recognition result.

[0094] Based on the output of the ECU 701, the display unit 703 displays various information to the driver of the mobile unit 700, such as images acquired by the photoelectric conversion element 100, recognition results from the recognition unit 604, and the vehicle's driving status, for example, using a GUI.

[0095] Furthermore, the image processing unit 603, recognition unit 604, etc. in Figure 6 do not necessarily have to be mounted on the mobile body 700. For example, they may be installed on an external terminal provided separately from the mobile body 700 for remotely controlling the mobile body 700 or for monitoring the movement of the mobile body.

[0096] Figure 7 is a diagram illustrating an example of the relationship between the propagation of the light emitted from the light emitter 500 and its reflected light, and the exposure timing of the camera 600, according to Embodiment 1. In Figure 7, the horizontal axis represents distance and the vertical axis represents time.

[0097] As shown in Figure 7, in this embodiment, an image (range gate image) capturing the target distance range is acquired by performing control (range gate control) that synchronizes the light emission timing and exposure timing according to the target distance range.

[0098] In this embodiment, a camera that acquires a target distance image by range gate control in this manner is called a range gate camera.

[0099] First, let's explain the horizontal axis. In the example shown in Figure 7, fog 810 is located between distances x1 and x2, and vehicle 820 is located at distance x3. Also, in Figure 7, range gate control starts from the position at distance D to the target vehicle 820, and range gate images are acquired within the target distance range R from there.

[0100] In this case, the target distance range R is the target distance range to be imaged. At this time, the vehicle 820 is located within the target distance range R.

[0101] Next, let's explain the vertical axis. Time 0 is defined as the start time of light emission at light emitter 500, and time tf is defined as the end time of light emission. In this case, the light emission period is time tf. Also, when acquiring a range gate image within the target distance range R, with distance D as the starting point, the exposure start time is defined as time t1 and the exposure end time as time t2.

[0102] Time t1 is the moment when the synchrotron radiation emitted from the light emitter 500 at time 0 returns to camera 600 as reflected light from an object at distance D. Time t2 is the moment when the synchrotron radiation emitted from the light emitter 500 at time tf returns to camera 600 as reflected light from a point that has traveled a distance R from the object at distance D.

[0103] Furthermore, time t3 is defined as the timing when the first reflected light from fog 810 returns to camera 600, and time t4 is defined as the timing when the last reflected light from fog 810 returns to camera 600.

[0104] In range gate control, no exposure is performed during the period t3 to t4 when the reflected light from the fog 810 reaches the camera 600. Then, by performing exposure only during the period t1 to t2 when the reflected light from distance D reaches the camera within the target distance range R, the fog 810 can be removed while a clear image of the vehicle 820 can be acquired.

[0105] Here, we will explain the time it takes for reflected light from an object at distance x to return to the camera 600. Let time tr be the time it takes for the emitted light from the light emitter 500 to strike an object at distance x and return to the imaging unit as reflected light. At this time, the relationship between the time tr for the reflected light to return and the distance x to the object to be imaged is given by the following equation (1).

[0106] Time tr=2x / speed of light c (approximately 3×10^8m / s)...Equation (1)

[0107] As shown in Figure 7, when the imaging range is defined as the target distance range R from distance D, the exposure timing time t1 at the start of the time range corresponding to the target distance range R can be obtained by substituting distance D into distance x in equation (1) above, resulting in the following equation (2).

[0108] Time t1=2D / speed of light c...Equation (2)

[0109] Furthermore, the exposure timing time t2 at the end of the range can be obtained by substituting the distance D + target distance range R into the distance x in equation (1) above and adding the time tf, resulting in the following equation (3).

[0110] Time t2=time tf+2(D+R) / speed of light c...Equation (3)

[0111] In this way, the time tf from the start to the end of light emission, the time t1 from the start of light emission to the start of exposure, and the time t2 until the end of exposure are controlled according to the distance x (target distance range R) to be imaged. This enables range gate control that allows clear imaging of subjects within the target distance range even if there is fog or other obstructions between the camera and the target distance range.

[0112] Figure 8 is a timing chart illustrating the control operation for obtaining a range gate image during a one-frame period according to Embodiment 1. In this embodiment, the range gate image is generated as described above by exposure synchronized with the emission from the light emitter 500.

[0113] In Figure 8, the "Vertical Sync Signal" indicates the frame period of the image, with one frame period being the time between one low pulse and the next low pulse. "Emission Control" indicates the emission timing of the light emitter 500, with light emission occurring from the light emitter 500 while the level is high. "Exposure Control" indicates the counting period of the counter circuit, with the counter circuit counting the number of photons while the level is high.

[0114] The "counter value" indicates the increase or decrease in the photon count of the counter circuit. The "RES signal" indicates a control pulse supplied to the counter circuit via the RES signal line 213, and the pulse resets the count value held in the counter circuit.

[0115] Next, range gate control for obtaining a range gate image will be described. In this embodiment, the light emission period is controlled in a pulsed manner by the light emission control unit 502, and the number of photons is counted only for reflected light from a predetermined target distance range.

[0116] Thus, the emission period from the start to the end of emission is defined as time tf, the time from the start of emission to the start of exposure (photon counting) is defined as time t1, and the time from the start of emission to the end of exposure (photon counting) is defined as time t2.

[0117] In this case, time t1 represents the period from the start of light emission until the light reaches the target distance range and the reflected light returns to camera 600. The period from time t1 to time t2 is the period during which the number of photons of reflected light within the target distance range is counted, and this period is from the start to the end of exposure. The counter value increases according to the number of photons during the exposure period.

[0118] For range gate control to function correctly, it is necessary to synchronize the timing of light emission start and exposure start according to a predetermined target distance range. In this embodiment, the camera control unit 605 synchronizes the operation timing of the counter circuit 211 and the light emission control unit 502 by transmitting the same reference signal to both.

[0119] Furthermore, as shown in the timing chart of Figure 7, the period from the start of one light emission to the start of the next constitutes one range gate operation cycle. The counter value counted in one range gate operation cycle is retained, and the counter value is accumulated in the next range gate operation cycle. The period from one light emission to the next is set based on the time it takes for the reflected light to sufficiently attenuate and no longer return to the camera 600.

[0120] As shown in Figure 8, a predetermined number of range gate operation cycles (e.g., several hundred to tens of thousands of times) are performed within one frame period, and the count value of the counter circuit is accumulated each time. Then, the RES signal sends the information of the last accumulated counter value within one frame period from the counter circuit 211 to the memory circuit 212, and the counter value is then reset by the RES signal.

[0121] In this range gate control, the exposure period is synchronized with the light emitted by the light emitter 500, making it possible to obtain a clear image of the target range even in adverse weather conditions such as fog.

[0122] Next, Figures 9(A) and 9(B) are diagrams illustrating the relationship between the control signal CLK and the exposure timing according to Embodiment 1. Figure 9 shows the relationship between the propagation of the light emitted from the light emitter 500 and its reflected light, and the range imaged by the exposure of the camera 600, when a range gate camera and a clock decharging type SPAD are combined.

[0123] Figure 9(A) illustrates the inability to expose reflected light from a specific distance range during the recharge period using the clock decharging method when acquiring range gate images.

[0124] In Figure 9, similar to Figure 7, the horizontal axis represents distance and the vertical axis represents time. Time 0 is defined as the start time of light emission from light emitter 500, and time tf is defined as the end time of light emission. In this case, the light emission period is time tf. When the target distance range is distance x4 to distance x5, the exposure start time is time t5 and the exposure end time is time t6. It is also assumed that vehicle 820 is located within this target distance range.

[0125] During the exposure period from time t5 to time t6, a control signal CLK is applied to switch 202 using a clock recharging method, as explained in Figure 4, and a recharge operation is performed.

[0126] Figure 9(A) explains the case where there are 6 clock cycles between time t5 and time t6. The respective clocks are CLK1, CLK2, CLK3, CLK4, CLK5, and CLK6, and their timings are shown on the vertical axis of Figure 9(A).

[0127] Figure 9(B) shows the distance timing of CLK1 to CLK6. This Figure 9(B) displays only the portion corresponding to the target distance range during the exposure period from time t5 to time t6 on the vertical axis of Figure 9(A).

[0128] As explained in the description of the clock recharging method in Figure 5, photons incident during the period from when the control signal CLK goes low until the threshold voltage is exceeded by the recharge operation are not counted.

[0129] In other words, within the six clock cycles from CLK1 to CLK6, photons are not counted during the period before the threshold voltage is exceeded due to the recharge operation. Therefore, exposure (counting operation by counter circuit 211) does not occur during that period.

[0130] As shown in Figure 9(B), the clock duty cycle is 50:50, and the period until the threshold voltage is exceeded by the recharge operation is equivalent to the clock's low period. In this case, even if the control signal CLK is at its current maximum value of 30MHz, the period of CLK will be 33.3ns.

[0131] If we consider this in terms of the speed of light, we can see that light travels approximately 10m in 33.3ns. At this time, since the clock duty cycle is 50:50, if we express the low period in terms of distance, we can see that each clock cycle corresponds to approximately 5m.

[0132] The inability to image a distance of approximately 5 meters means that, as shown in Figure 9(B), it is possible that an entire car may not be exposed.

[0133] Next, Figure 10 is a diagram showing an example of the relationship between light emission and the control signal CLK at each exposure timing according to Embodiment 1, and, similar to Figure 9(B), it shows the relationship between the target distance range and the control signal CLK in range gate operation.

[0134] As explained in Figure 9(B), when exposure is performed within the target distance range, if there are six control signal CLK signals, then photons cannot be counted during the period when the control signal CLK goes low and the recharge operation exceeds the threshold voltage.

[0135] Therefore, that range cannot be imaged. However, in a range gate camera, as explained in Figure 8, in order to acquire a single range gate image, the exposure operation is repeated hundreds to tens of thousands of times using range gate operation cycles, and the image is acquired by accumulating and adding the count values ​​from those exposure operations.

[0136] In other words, in this embodiment, in order to image a subject that is within a predetermined imaging distance range, the counter circuit 211 performs multiple exposure operations (counting operations) according to the timing of pulse emission and the predetermined imaging distance range.

[0137] Furthermore, as shown in Figure 10, the clock duty cycle may be 50:50 and the period until the threshold voltage is exceeded by the recharge operation may be equivalent to the low period of CLK. In that case, by changing the phase of the control signal CLK by 180° between the 2n (even) exposure period and the 2n+1 (odd) exposure period, it becomes possible to acquire an image across the entire region.

[0138] In other words, in this embodiment, by shifting the control signal CLK by half a phase between even-numbered exposure periods and odd-numbered exposure periods, the timing of light emission and exposure relative to the control signal CLK changes.

[0139] Thus, in this embodiment, a control step is performed to shift the relative timing of the control signal CLK, which serves as a clock signal, and the pulse emission by a predetermined phase for each predetermined exposure operation. This prevents photon counting errors in a predetermined imaging range during range gate control.

[0140] Furthermore, while Figure 10 explains that the phase is shifted by 180° between even-numbered and odd-numbered cycles, the period and amount of phase shift can be arbitrarily set according to the time until the threshold voltage is exceeded by the recharge operation and the clock frequency. In other words, the predetermined phase can be set based on the ratio of the period of the control signal CLK as the clock signal and the period of the avalanche photodiode's recharge state.

[0141] For example, if the relationship between the time it takes for the threshold voltage to be exceeded by the recharge operation and the period of one clock cycle is 1:4, then by shifting the phase by 1 / 4 each time at the 4n, 4n+1, 4n+2, and 4n+3 cycles, images can be acquired uniformly across the entire region.

[0142] Figure 11 is a flowchart detailing an example of the operation of the imaging method according to Embodiment 1. In this flowchart, each step from step S101 to step S116 is executed sequentially by the CPU, which acts as a computer within the camera control unit 605, executing a computer program stored in memory.

[0143] In step S101 of Figure 11, the camera control unit 605 acquires weather information by determining the weather conditions (sunny, rainy, foggy, etc.) in front of the vehicle using the recognition unit 604 of the camera 600, and then proceeds to step S102.

[0144] In step S102, the camera control unit 605 determines whether the weather information acquired in step S101 indicates bad weather. If it is determined to be bad weather, the process proceeds to step S103. If it is determined not to be bad weather, the process proceeds to step S115.

[0145] In step S103, the camera control unit 605 starts range gate control mode. That is, the camera control unit 605 puts the camera 600 into range gate control mode and sends a control signal to the light emitter 500 via the communication unit 607, putting the light emitter 500 into range gate control mode. After that, the process proceeds to step S104.

[0146] In step S104, the camera control unit 605 sets i=1 and j=1, and then proceeds to step S105.

[0147] In step S105, the camera control unit 605 starts imaging the i-th target distance range out of the N target distance ranges in range gate control mode. Then, the process proceeds to step S106.

[0148] In step S106, the camera control unit 605 determines whether j is odd or not. Here, j refers to which of the predetermined number of exposures (hundreds to tens of thousands) within a one-frame period is set in range gate control.

[0149] If j is odd, the process proceeds to step S107; if j is even, it proceeds to step S108. In this embodiment, depending on the determination result in step S106, the process changes whether to shift the light emission timing relative to the control signal CLK applied to the switch 202 by half a phase. That is, the control signal CLK is alternately shifted by half a phase for each exposure period.

[0150] In step S107, light emission is executed at a predetermined timing with respect to the control signal CLK. That is, the light emitter 500 is caused to emit light at the normal timing without shifting the light emission timing with respect to the control signal CLK applied by the camera control unit 605 to the switch 202 by half a phase. After the light emission of the light emitter 500 is completed, the process proceeds to step S109.

[0151] In step S108, light emission is executed at a timing shifted by half a phase from the predetermined timing with respect to the control signal CLK. That is, the light emitter 500 is caused to emit light at a timing in which the light emission timing with respect to the control signal CLK applied by the camera control unit 605 to the switch 202 is shifted by half a phase. After the light emission of the light emitter 500 is completed, the process proceeds to step S109.

[0152] In step S109, exposure is performed within the target distance range. That is, the camera control unit 605 performs exposure (counting operation by the counter circuit 211) in the imaging unit 602 at a timing according to the target distance range. After the exposure is completed, the process proceeds to step S110.

[0153] In step S110, the camera control unit 605 sets j = j + 1, increments j by 1, and proceeds to step S111.

[0154] In step S111, it is determined whether j < M. That is, the camera control unit 605 determines whether the number of exposures M, which is a predetermined number of exposures within one target distance range, has been reached in the range gate control. If it is determined No in step S111, that is, if M has been reached, the process proceeds to step S112. If it is determined Yes in step S111, that is, if M has not been reached, the process returns to step S106.

[0155] In step S112, an image is acquired. That is, the imaging unit 602 generates image data based on the cumulative count value obtained by M exposure operations, and sends the image data to the image processing unit 603. After completion of the image processing in the image processing unit 603, the data is transferred from the image processing unit 603 to the subsequent recognition unit 604 and ECU 701. Then, the process proceeds to step S113.

[0156] In step S113, the camera control unit 605 sets i = i + 1, increments i by 1, and proceeds to step S114.

[0157] In step S114, it is determined whether i < N. That is, it is determined whether all of the N target distance ranges set by the camera control unit 605 have been completed. If the determination in step S114 is Yes, that is, if i is less than or equal to N, the process returns to step S105. If the determination in step S114 is No, that is, if i is greater than N, the processing flow of FIG. 11 is terminated.

[0158] In step S115, the normal shooting mode is started. That is, the camera control unit 605 transmits a control signal so as to operate in the normal shooting mode. After starting shooting in the normal shooting mode, the process proceeds to step S116.

[0159] Here, the normal shooting mode refers to a mode in which shooting is performed to generate an image by performing a counting operation (exposure operation) by the counter circuit 211 for a predetermined exposure period within one frame period without light emission by the light emitting unit.

[0160] In step S116, an image is acquired. That is, the image data captured by the imaging unit 602 is sent to the image processing unit 603. After completion of the image processing in the image processing unit 603, the data is transferred from the image processing unit 603 to the subsequent recognition unit 604 and ECU 701. Then, the processing flow of FIG. 11 is terminated.

[0161] Although the flowchart in Figure 11 shows an example of shifting the phase by half, as mentioned in Figure 10, the amount of phase shift and the period can be arbitrarily set according to the ratio of the period of the control signal CLK to the recharge period.

[0162] Furthermore, in this embodiment, the phase was changed after each exposure, but the frequency of the change does not necessarily have to be after each exposure. The phase may be changed every few exposures, or the phase may be changed between the first and second halves of the total number of exposures.

[0163] Furthermore, in this embodiment, the timing of light emission is changed by half a phase shift with respect to the control signal CLK, but it is also possible to keep the timing of light emission constant and change the control signal CLK for each light emission.

[0164] Thus, according to this embodiment, in a range gate camera using a clock recharging SPAD, even when range gate control is performed, the distance range that cannot be imaged due to the recharge period is less likely to occur, and the overall image quality of the screen can be improved.

[0165] <Embodiment 2> Embodiment 2 of the present invention will be described below. Embodiment 1 described a method using a clock decharging SPAD sensor. Embodiment 2 describes a case in which a clock decharging SPAD sensor is further equipped with two photoelectric conversion units within one pixel, thereby enabling the acquisition of two images with parallax and stereo distance measurement.

[0166] Furthermore, the functional block diagram in Embodiment 2 is substantially the same as that shown in Figure 6, except that the imaging unit 602 has been changed to a SPAD sensor with a structure in which two photoelectric conversion units are located within one pixel.

[0167] Figures 12(A) and (B) show an example configuration of an image sensor having two photoelectric conversion units within one pixel according to Embodiment 2. Figure 12(A) is a top view of the sensor substrate 11 as seen from the direction of light incidence. The sensor substrate 11 is constructed by arranging multiple pixel groups 1000, each consisting of 4 pixels arranged in a 2x2 grid, in a matrix.

[0168] The pixel group 1000 includes green pixels G1 and G2 for detecting green light, red pixels R for detecting red light, and blue pixels B for detecting blue light. In the pixel group 1000, green pixels G1 and G2 are arranged diagonally.

[0169] Furthermore, each pixel has a first photoelectric conversion unit 102A and a second photoelectric conversion unit 102B that receive light from different pupils. The first photoelectric conversion unit 102A and the second photoelectric conversion unit 102B can be controlled independently.

[0170] Figure 12(B) is a cross-sectional view of the pixel group 1000 in the I-I' section of Figure 12(A). Each pixel consists of a microlens 1003, a light guide layer 1004, and a light receiving layer 1005.

[0171] The light guide layer 1004 includes a microlens 1003 for efficiently guiding the light beam incident on the pixel to the light receiving layer 1005, a color filter that allows light in a wavelength band corresponding to the color of light detected by each pixel to pass through, and a light guide member having wiring for image readout and pixel driving.

[0172] The light-receiving layer 1005 is a photoelectric conversion unit that converts light incident through the light guide layer 1004 into electrical signals and outputs them as electrical signals, and has a first photoelectric conversion unit 102A and a second photoelectric conversion unit 102B.

[0173] Furthermore, in the above explanation, the pixel group 1000 was arranged as green pixel G1, green pixel G2, red pixel R, and blue pixel B as shown in Figure 12(A). However, this arrangement is not limited to this, and infrared pixels such as IR that receive infrared light may also be included, and the order of arrangement is not limited.

[0174] In the following explanation, the output of the first photoelectric conversion unit 102A will be referred to as the first photoelectric conversion signal, and the output of the second photoelectric conversion unit 102B will be referred to as the second photoelectric conversion signal. Furthermore, the image signal generated by the first photoelectric conversion signals of multiple pixels will be referred to as the first image signal, and the image signal generated from the second photoelectric conversion signals of multiple pixels will be referred to as the second image signal.

[0175] Furthermore, since the first photoelectric conversion unit 102A and the second photoelectric conversion unit 102B each receive light from different exit pupils of the photographic lens via microlenses and perform photoelectric conversion, the first photoelectric conversion signal and the second photoelectric conversion signal have parallax. Therefore, the first image signal and the second image signal have a phase difference corresponding to the amount of parallax (positional displacement), and the distance to the subject can be calculated based on this phase difference. Furthermore, when acquiring an image for display, the first image signal and the second image signal are added together.

[0176] Figure 13 shows the equivalent circuit of a signal processing circuit corresponding to one pixel in the pixel group 1000 shown in Figure 12. It is basically the same as the one shown in Figure 4, but since there is a first photoelectric conversion unit 102A and a second photoelectric conversion unit 102B in one pixel, two sets of sets consisting of APD, switch, waveform shaping unit and counter circuit are provided in the pixel.

[0177] Specifically, 201A is an APD included in the first photoelectric conversion unit 102A, and 201B is an APD included in the second photoelectric conversion unit 102B. Furthermore, 202A and 202B are switches connected to the cathodes of APD201A and 201B, respectively, and 210A and 210B are waveform shaping units connected to the cathodes of APD201A and 201B, respectively.

[0178] 222A, 222B, 223A, and 223B are pixel switches. Pixel switches 222A, 222B, 223A, and 223B switch whether each counter is supplied with a first photoelectric conversion signal, a second photoelectric conversion signal, or both the first and second photoelectric conversion signals.

[0179] Furthermore, 211A and 211B are counter circuits connected to the outputs of OR circuits 224A and 224B, respectively. Note that counter circuits 211A and 211B function as a first counter and a second counter, respectively, capable of counting the outputs of the first and second photoelectric conversion sections. The outputs of OR circuits 224A and 224B are connected to counter circuits 211A and 211B, respectively.

[0180] Furthermore, as described above, the pixel switches 222A, 222B, 223A, and 223B selectively connect the output of the first photoelectric conversion unit and the output of the second photoelectric conversion unit to the first counter and the second counter, respectively.

[0181] For example, when the counter circuit 211A counts the first photoelectric conversion signal, the pixel switch 222A is turned ON and the pixel switch 222B is turned OFF. Also, when the counter circuit 211A counts the second photoelectric conversion signal, the pixel switch 222B is turned ON and the pixel switch 222A is turned OFF.

[0182] Thus, in this embodiment, the pixel switch has a first switching state in which one of the outputs of the first photoelectric conversion unit and the second photoelectric conversion unit is connected to the first counter, and the other is connected to the second counter.

[0183] When counting both the first and second photoelectric conversion signals in the counter circuit 211A, pixel switches 222A and 222B are turned ON. The same applies to the counter circuit 211B.

[0184] Thus, in this embodiment, the pixel switch has a second switching state in which both the output of the first photoelectric conversion unit and the output of the second photoelectric conversion unit are connected to, for example, the first counter.

[0185] Figures 14(A) to 14(C) illustrate examples of operating modes when using an image sensor with two photoelectric conversion units within a single pixel. Figure 14(A) shows a mode in which distance measurement is performed while capturing the entire image (imaging range R1) in normal shooting mode.

[0186] In this normal shooting mode, only pixel switches 222A and 223B, as explained in Figure 13, are turned ON. As a result, counter circuits 211A and 211B acquire the first and second photoelectric conversion signals, respectively, and further acquire subject distance data from multiple pixels based on the phase difference between the first and second image signals.

[0187] In other words, the subject distance can be calculated based on the output of the first counter and the output of the second counter in the first switching state described above.

[0188] On the other hand, when acquiring an image for display, an image can be obtained by adding the count values ​​of counter circuit 211A and counter circuit 211B, thereby adding the first image signal and the second image signal. That is, an image for display can be generated based on, for example, the output of the first counter in the second switching state described above.

[0189] Figure 14(B) illustrates an example of performing range gate control to acquire images only within the imaging range R2. By acquiring the first image signal and the second image signal within the imaging range R2, subject distance data can be obtained.

[0190] In this mode as well, similar to Figure 14(A), only the pixel switches 222A and 223B described in Figure 13 are turned ON, and the first photoelectric conversion signal and the second photoelectric conversion signal are acquired by the counter circuits 211A and 211B, respectively.

[0191] Then, based on the outputs of the counter circuits 211A and 211B for multiple pixels, a first image signal and a second image signal are generated, respectively, and subject distance data is calculated based on the phase difference between the first image signal and the second image signal.

[0192] Furthermore, in Embodiment 2, when using the clock decharging method, the phase of the timing of light emission with respect to the control signal CLK is changed to prevent any range from being affected by imaging, similar to Embodiment 1.

[0193] Figure 14(C) shows an example of acquiring the target distance range of two locations simultaneously using range gate control. By acquiring the target distance range of two locations at the same time, the number of flashes can be reduced. In addition, there is the advantage of being able to narrow the target distance range for each location.

[0194] In this embodiment, when imaging is performed for two target distance ranges, counter circuits 211A and 211B count both the first and second photoelectric conversion signals for each target distance range. Based on the count values ​​of counter circuits 211A and 211B, images corresponding to each target distance range are acquired.

[0195] For example, suppose the counter circuit 211A acquires the imaging range R3 and the counter circuit 211B acquires the imaging range R4. In this case, when the imaging range R3 is exposed, both pixel switches 222A and 222B are turned ON, and when the imaging range R4 is exposed, both pixel switches 223A and 223B are turned ON.

[0196] At this time, as explained in Embodiment 1, the phase of the timing of light emission relative to the control signal CLK is changed so that no range of imaging is not possible. However, as shown in Figure 14(C), when the first photoelectric conversion signal and the second photoelectric conversion signal are counted with a single counter, counting errors may occur in the clock decharging method.

[0197] In other words, if photons enter both the first photoelectric conversion unit 102A and the second photoelectric conversion unit 102B simultaneously within the same clock cycle, a counting error occurs. This will be illustrated and explained in detail in Figure 15.

[0198] Figure 15 illustrates the challenges when counting both the first and second photoelectric conversion signals, showing the relationship between the target distance range and the control signal CLK. The incident light to the first photoelectric conversion unit 102A is denoted as incident light A, and the incident light to the second photoelectric conversion unit 102B is denoted as incident light B.

[0199] The control signals CLK to each photoelectric conversion unit are CLKA and CLKB. However, in the clock decharging method, as explained in Embodiment 1, there is a problem that photons incident during the low period of the clock cannot be counted.

[0200] Furthermore, when counting both the first and second photoelectric conversion signals, as shown in the range of CLK3 in Figure 15, there are cases where photons of incident light A and incident light B are simultaneously incident on the first photoelectric conversion unit 102A and the second photoelectric conversion unit 102B, respectively, within the same clock cycle. In such cases, only one count is possible within the same clock cycle, resulting in count omissions.

[0201] In particular, if this counting error occurs frequently in areas where objects are present, the actual count will not match the original count, resulting in an image with a poor signal-to-noise ratio.

[0202] In contrast, Figure 16 is a diagram showing an example of a method for solving the problem described in Figure 15, and shows an example of the relationship between the control signal CLK and the light emission timing relative to the control signal CLK in each photoelectric conversion unit according to Embodiment 2.

[0203] First, to prevent leakage of the imaging area during the Low period, similar to Embodiment 1, the timing of light emission from the light emitter 500 can be shifted by half a phase during the 2n-th exposure period and the 2n+1-th exposure period to prevent leakage of the imaging area.

[0204] Furthermore, in order to prevent counting errors, the light emission timing is shifted by half a phase between the 2n exposure period and the 2n+1 exposure period, and the control signal CLK applied to the first photoelectric conversion unit 102A and the control signal CLK applied to the second photoelectric conversion unit 102B are also shifted by half a phase.

[0205] That is, as shown in Figure 16, during the 2n exposure period, the control signal CLKA applied to the first photoelectric conversion unit 102A and the control signal CLKB applied to the second photoelectric conversion unit 102B are shifted by half a phase. However, it is not limited to half a phase. That is, the phases of the clock signals supplied to the avalanche photodiode of the first photoelectric conversion unit and the avalanche photodiode of the second photoelectric conversion unit can be shifted by a predetermined phase.

[0206] Furthermore, the control signal CLKA during the 2n-th exposure period and the control signal CLKA during the 2n+1-th exposure period are shifted by half a phase, and the control signal CLKB during the 2n-th exposure period and the control signal CLKB during the 2n+1-th exposure period are also shifted by half a phase. This reduces counting errors.

[0207] Furthermore, similar to Embodiment 1, Figures 15 and 16 illustrate an example where the duty cycle of the control signal CLK is set to 50:50, and the time until the threshold is exceeded by the recharge operation is the same as the low period of CLK.

[0208] Therefore, although I explained that the phase is shifted by half a phase between even-numbered and odd-numbered calls, the period and the amount of phase shift can be arbitrarily changed according to the ratio of the period of the control signal CLK to the recharge period.

[0209] For example, if the ratio of the period of the control signal CLK to the recharge period is 25:75, then by shifting the phase by 1 / 4 at the 4n, 4n+1, 4n+2, and 4n+3 times, images can be acquired uniformly across the entire region.

[0210] Also, while the phase was changed after each exposure in this example, the frequency of the change does not necessarily have to be after every exposure. It would also be acceptable to change the phase every few exposures, or to change the phase between the first and second halves of the total number of exposures.

[0211] Furthermore, in this embodiment, an example was described in which the timing of light emission is changed by half a phase shift with respect to the control signal CLK. However, the timing of light emission may be kept constant, and the control signal CLK may be changed for each light emission.

[0212] Furthermore, the relative timing of pulse emission with respect to the fixed clock signal may be shifted by a predetermined phase for each exposure operation. Alternatively, the relative timing of the clock signal and pulse emission may be shifted by a predetermined phase for each exposure operation, every predetermined number of times, or between the exposure operation in the first half of the frame period and the exposure operation in the second half.

[0213] Alternatively, based on the ratio of the clock signal period to the recharge period, a predetermined number of pulse emission timings may be performed at predetermined phases, and the subsequent multiple pulse emission timings may be periodically shifted by multiple phases corresponding to the duty cycle of the clock signal.

[0214] Alternatively, the timing of the pulse emission may be kept constant, and the duty cycle of the clock signal may be changed according to the timing of the pulse emission.

[0215] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate.

[0216] Furthermore, the present invention includes, for example, a system that implements the functions of the above embodiment using at least one processor such as a CPU, memory, and circuitry (e.g., an ASIC). Alternatively, multiple processors may be used for distributed processing.

[0217] Furthermore, in order to implement some or all of the control in the above embodiment, a computer program that implements the functions of the above embodiment may be supplied to the imaging device, etc., via a network or various storage media.

[0218] The computer (or CPU or MPU, etc.) in the imaging device may read and execute the program. In that case, the program and the storage medium storing the program constitute the present invention. The present invention includes the following combinations.

[0219] (Configuration 1) An imaging device characterized by comprising: a photoelectric conversion element having a plurality of pixels, each pixel comprising: a sensor unit that emits pulses in response to photons incident on an avalanche photodiode; a counter that counts the number of pulses; a memory that stores the count value of the counter; and a switch that switches the avalanche photodiode between a standby state capable of avalanche multiplication and a recharge state; a signal generation unit that supplies a clock signal to the switch; a light emission unit that emits pulsed light in synchronization with the clock signal to illuminate a subject; and a control unit that performs multiple exposure operations using the counter according to the timing of the pulsed light emission and the predetermined imaging distance range in order to image a subject located within a predetermined imaging distance range, and shifts the relative timing of the clock signal and the pulsed light emission by a predetermined phase for each predetermined exposure operation.

[0220] (Configuration 2) The imaging apparatus according to Configuration 1, characterized in that the predetermined phase is set based on the ratio of the period of the clock signal to the period of the recharge state of the avalanche photodiode.

[0221] (Configuration 3) The imaging apparatus according to Configuration 1 or 2, characterized in that the pixel has a first photoelectric conversion unit and a second photoelectric conversion unit that receive light from different pupils, a first counter and a second counter that can count the output of the first photoelectric conversion unit and the output of the second photoelectric conversion unit, respectively, and a pixel switch for selectively connecting the output of the first photoelectric conversion unit and the output of the second photoelectric conversion unit to the first counter and the second counter, respectively.

[0222] (Configuration 4) The imaging apparatus according to Configuration 3, characterized in that the control unit shifts the phase of the clock signal supplied to the avalanche photodiode of the first photoelectric conversion unit and the avalanche photodiode of the second photoelectric conversion unit.

[0223] (Configuration 5) The imaging apparatus according to Configuration 3 or 4, characterized in that the pixel switch has a first switching state in which one of the outputs of the first photoelectric conversion unit and the second photoelectric conversion unit is connected to the first counter and the other is connected to the second counter.

[0224] (Configuration 6) The imaging apparatus according to Configuration 5, characterized in that the control unit calculates the subject distance based on the output of the first counter and the output of the second counter in the first switching state.

[0225] (Configuration 7) An imaging apparatus according to any one of Configurations 3 to 6, characterized in that the pixel switch has a second switching state in which both the output of the first photoelectric conversion unit and the output of the second photoelectric conversion unit are connected to the first counter.

[0226] (Configuration 8) The imaging apparatus according to Configuration 7, characterized in that the control unit generates a display image based on the output of the first counter in the second switching state.

[0227] (Configuration 9) The imaging apparatus according to any one of Configurations 1 to 8, characterized in that the control unit shifts the relative timing of the pulse emission with respect to the fixed clock signal by a predetermined phase for each exposure operation.

[0228] (Configuration 10) The imaging apparatus according to any one of Configurations 1 to 9, wherein the control unit shifts the relative timing of the clock signal and the pulse emission by a predetermined phase for each exposure operation, every predetermined number of times, or between the first half of the exposure operation and the second half of the exposure operation.

[0229] (Configuration 11) The imaging apparatus according to any one of Configurations 1 to 10, characterized in that the control unit performs a predetermined number of pulse emission timings at predetermined phases based on the ratio of the period of the clock signal to the period of the recharge state, and periodically shifts the timing of subsequent multiple pulse emission timings by a plurality of phases corresponding to the duty cycle of the clock signal.

[0230] (Configuration 12) The imaging apparatus according to any one of Configurations 1 to 11, characterized in that the control unit keeps the timing of the pulse emission constant and changes the duty cycle of the clock signal according to the timing of the pulse emission.

[0231] (Method) An imaging method using an imaging device comprising: a sensor unit having a plurality of pixels, each pixel emitting pulses in response to photons incident on an avalanche photodiode; a counter for counting the number of pulses; a memory for storing the count value of the counter; and a switch for switching the avalanche photodiode between a standby state capable of avalanche multiplication and a recharge state, wherein the imaging device comprises a signal generation unit supplying a clock signal to the switch and a light emission unit that emits pulsed light in synchronization with the clock signal to illuminate a subject, and the imaging method is characterized by having a control step of performing multiple exposure operations by the counter according to the timing of the pulsed light emission and the predetermined imaging distance range in order to image a subject located within a predetermined imaging distance range, and shifting the relative timing of the clock signal and the pulsed light emission by a predetermined phase for each predetermined exposure operation.

[0232] (Program) A computer program for controlling each part of the imaging device described in any one of configurations 1 to 12. [Explanation of symbols]

[0233] 11: Sensor board 12: Pixel area 21: Circuit board 22: Circuit area 30: Color Filters 31: RGB Filter 32: IR filter 100: Photoelectric conversion element 101: Pixels 102: Photoelectric conversion unit 103: Signal Processing Circuit 110: Vertical scanning circuit 111: Horizontal scanning circuit 112: Readout circuit 113: Vertical signal line 114: Output Circuit 115: Control pulse generation unit 201:APD 202: Switch 210: Waveform shaping section 211: Counter circuit 212: Memory Circuit 213: Drive line 214: Drive line 222: Switch 224: OR Circuit 500: Light emitter 501: Light-emitting part 502: Light emission control unit 503: Communications Department 600: Camera 601: Imaging Optical System 602: Imaging Department 603: Image Processing Unit 604: Recognition part 605: Camera Control Unit 606: Storage section 607: Communications Department 700: Mobile 701: ECU 702: Vehicle Control Unit 703: Display section 810: Fog 820: Vehicle 1000: Pixel group 102A: First photoelectric conversion unit 102B: Second photoelectric conversion unit 1003: Microlens 1004: Light guide layer 1005: Light receiving layer

Claims

1. It has multiple pixels, and the pixels are A sensor unit that emits pulses in response to photons incident on an avalanche photodiode, A counter for counting the number of pulses, A memory for storing the count value of the aforementioned counter, The avalanche photodiode is equipped with a switch that allows switching between a standby state capable of avalanche multiplication and a recharge state, and the photoelectric conversion element is equipped with a switch that allows switching between these states. A signal generation unit that supplies a clock signal to the switch, A light-emitting unit that emits pulsed light in synchronization with the aforementioned clock signal to illuminate the subject, In order to image a subject located within a predetermined imaging distance range, the counter performs multiple exposure operations according to the timing of the pulse emission and the predetermined imaging distance range, A control unit that shifts the relative timing of the clock signal and the pulse emission by a predetermined phase for each predetermined exposure operation, An imaging device characterized by having the following features.

2. The imaging apparatus according to claim 1, characterized in that the predetermined phase is set based on the ratio of the period of the clock signal to the period of the recharge state of the avalanche photodiode.

3. The aforementioned pixel comprises a first photoelectric conversion unit and a second photoelectric conversion unit that receive light from different pupils, A first counter and a second counter capable of counting the output of the first photoelectric conversion unit and the output of the second photoelectric conversion unit, respectively. A pixel switch for selectively connecting the output of the first photoelectric conversion unit and the output of the second photoelectric conversion unit to the first counter and the second counter, respectively, The imaging device according to claim 1, characterized by having the following features.

4. The imaging apparatus according to claim 3, characterized in that the control unit shifts the phase of the clock signal supplied to the avalanche photodiode of the first photoelectric conversion unit and the avalanche photodiode of the second photoelectric conversion unit.

5. The imaging apparatus according to claim 3, characterized in that the pixel switch has a first switching state in which one of the outputs of the first photoelectric conversion unit and the second photoelectric conversion unit is connected to the first counter and the other is connected to the second counter.

6. The imaging apparatus according to claim 5, characterized in that the control unit calculates the subject distance based on the output of the first counter and the output of the second counter in the first switching state.

7. The imaging apparatus according to claim 3, characterized in that the pixel switch has a second switching state in which both the output of the first photoelectric conversion unit and the output of the second photoelectric conversion unit are connected to the first counter.

8. The imaging apparatus according to claim 7, characterized in that the control unit generates a display image based on the output of the first counter in the second switching state.

9. The imaging apparatus according to claim 1, characterized in that the control unit shifts the relative timing of the pulse emission with respect to the fixed clock signal by a predetermined phase for each exposure operation.

10. The imaging apparatus according to claim 1, characterized in that the control unit shifts the relative timing of the clock signal and the pulse emission by a predetermined phase for each exposure operation, every predetermined number of times, or between the first half of the exposure operation and the second half of the exposure operation.

11. The imaging apparatus according to claim 1, characterized in that the control unit performs a predetermined number of pulse emission timings at predetermined phases based on the ratio of the period of the clock signal to the period of the recharge state, and then periodically shifts the timing of subsequent pulse emission timings by a plurality of phases corresponding to the duty cycle of the clock signal.

12. The imaging apparatus according to claim 1, characterized in that the control unit keeps the timing of the pulse emission constant and changes the duty cycle of the clock signal according to the timing of the pulse emission.

13. It has multiple pixels, and the pixels are A sensor unit that emits pulses in response to photons incident on an avalanche photodiode, A counter for counting the number of pulses, A memory for storing the count value of the aforementioned counter, An imaging method using an imaging device equipped with a switch that switches the avalanche photodiode between a standby state capable of avalanche multiplication and a recharge state, wherein The imaging device is A signal generation unit that supplies a clock signal to the switch, The system includes a light-emitting unit that emits pulsed light in synchronization with the aforementioned clock signal to illuminate an object, The imaging method is: In order to image a subject located within a predetermined imaging distance range, the counter performs multiple exposure operations according to the timing of the pulse emission and the predetermined imaging distance range, A control step of shifting the relative timing of the clock signal and the pulse emission by a predetermined phase for each predetermined exposure operation, An imaging method characterized by having the following features.

14. A computer program for controlling each part of the imaging apparatus described in any one of claims 1 to 12 by computer.

Images