Image pickup apparatus, image pickup method, computer program product, and storage medium

Abstract

The invention relates to an imaging apparatus, an imaging method, a computer program product, and a storage medium. The image pickup apparatus includes: a photoelectric conversion element having a plurality of pixels including a sensor unit, the sensor unit including: an avalanche photodiode generating a pulse in response to photons incident thereon; the counter is used for counting the number of the pulses; a memory for storing the count value of the counter; and a switch that switches the avalanche photodiode between a standby state in which avalanche multiplication may occur and a recharged state; a signal generation unit that supplies a clock signal to the switch; a light emitting unit that performs pulsed light emission for illuminating the subject in synchronization with the clock signal; and a control unit that uses the counter to perform a plurality of exposure operations according to the timing of the pulsed light emission and a predetermined imaging distance range to capture an image of a subject present within the predetermined imaging distance range, the relative timing of the clock signal and the pulsed light emission being shifted by a predetermined phase for each predetermined exposure operation.

Description

[0001] TECHNICAL FIELD The present disclosure relates to an imaging apparatus, an imaging method, and a computer program product, among others. BACKGROUND

[0002] A camera called a range-gated camera is known. That is, the range-gated camera employs a technique of emitting pulsed light in a forward direction from the camera at a predetermined cycle, and an image sensor inside the camera is exposed at a predetermined timing corresponding to a target distance range so that only a subject within the target distance range can be clearly photographed.

[0003] Hereinafter, this technique is referred to as range-gated control. By using the range-gated control, for example, a subject (object) at a predetermined distance can be clearly photographed even under bad weather.

[0004] Further, a SPAD (Single Photon Avalanche Diode) sensor that excels in low light performance is being considered for installation in the above-described range-gated camera. However, in the SPAD sensor, when a conventional passive recharge method is used, there is a case where a photon immediately hits a photodiode at or after a timing when the photodiode is recharged, such as a case where light hits the photodiode under high brightness.

[0005] In this case, the potential of the photon counting unit is maintained in a state where the photon is detected and does not change. As a result, this period is not counted as a period to obtain a photon. Therefore, under a high brightness environment, the actual coefficient value becomes smaller than the coefficient value corresponding to the incident light brightness, and the brightness of the image becomes lower than the original brightness.

[0006] In contrast, Japanese Patent Application Publication No. 2020-123847 proposes a driving method called a clocked recharge method in order to use a SPAD sensor under high brightness. Japanese Patent Application Publication No. 2020-123847 discloses a pixel having an APD including an APD, a quenching circuit connected to the APD, a signal control circuit into which a signal output from the APD is input, and a pulse generation circuit connected to the quenching circuit and the signal control circuit.

[0007] The pulse generation circuit controls the on / off of the quenching circuit. Further, it is disclosed that the potential of an output node of the APD is reset, and a pulse signal corresponding to an input photon is output even under high brightness.

[0008] Patent Literature 1: Japanese Patent Application Publication No. 2020-123847

[0009] However, in the structure described in Japanese Patent Laid-Open No. 2020-123847, photons that entered during a recharging period are not detected. The recharging period is typically several ns to several tens of ns, and because it is a very short time, the recharging period does not become a problem in a normal imaging mode.

[0010] However, in a range-gated camera, because light having an emission duration of several ns to several tens of ns is counted in an exposure period (photon counting period) of several ns to several tens of ns, reflected light that passes through a portion during a recharging period cannot be exposed. That is, there is a problem that reflected light from a specific distance range cannot be captured, resulting in image quality degradation. SUMMARY

[0011] An imaging apparatus includes: a photoelectric conversion element having a plurality of pixels, wherein the pixels each include a sensor unit including: an avalanche photodiode configured to generate a pulse in response to a photon incident on the avalanche photodiode; a counter configured to count a number of the pulses; a memory configured to store a count value of the counter; and a switch configured to switch the avalanche photodiode between a standby state in which avalanche multiplication can occur and a recharging state; a signal generation unit configured to supply a clock signal to the switch; a light emission unit configured to perform pulsed light emission for illuminating a subject in synchronization with the clock signal; and a control unit configured to perform a plurality of exposure operations for capturing an image of a subject present within a predetermined imaging distance range using the counter according to a timing of the pulsed light emission and the predetermined imaging distance range, and configured to shift a relative timing of the clock signal and the pulsed light emission by a predetermined phase for each predetermined exposure operation.

[0012] Further features of the present disclosure will become apparent from the following description of embodiments with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a diagram illustrating a structure example of a photoelectric conversion element 100 according to a first embodiment of the present disclosure.

[0014] Figure 2 is a diagram illustrating a structure example of a sensor substrate 11.

[0015] Figure 3 is a diagram illustrating a structure example of a circuit substrate 21.

[0016] Figure 4 is Figure 2The photoelectric conversion unit 102 of each pixel and the equivalent circuit of the signal processing circuit 103 corresponding to the photoelectric conversion unit 102.

[0017] Figure 5 This is a diagram that schematically illustrates the relationship between the operation and output signal of the clock-controlled recharge method.

[0018] Figure 6 This is a functional block diagram illustrating a structural example of the light emitter 500, camera 600, and moving body 700 according to the first embodiment.

[0019] Figure 7 This is a diagram illustrating an example of the relationship between the emission from the light emitter 500, the travel of reflected light, and the exposure timing of the camera 600 according to the first embodiment.

[0020] Figure 8 This is a timing diagram explaining the control operations for obtaining a range-gated image within a frame time period according to the first embodiment.

[0021] Figure 9A and Figure 9B This is a diagram used to explain the relationship between the control signal CLK and the exposure timing according to the first embodiment.

[0022] Figure 10 This is a diagram illustrating an example of the relationship between the emission and control signal CLK at each exposure timing according to the first embodiment.

[0023] Figure 11 This is a flowchart illustrating details of an operational example of the imaging method according to the first embodiment.

[0024] Figure 12A and Figure 12B This is a diagram illustrating a structural example of a camera element having two photoelectric conversion units in one pixel according to the second embodiment.

[0025] Figure 13 It is shown that... Figure 12A The diagram shows the equivalent circuit of the signal processing circuit corresponding to one pixel in pixel group 1000.

[0026] Figures 14A to 14C This is a diagram illustrating an example of the operating mode when using an imaging element with two photoelectric conversion units in one pixel.

[0027] Figure 15 This is a diagram illustrating the problems that arise when both the first and second photoelectric conversion signals are counted.

[0028] Figure 16 This shows the solution used to solve Figure 15A diagram illustrating an example of the methods used to explain the problem. Detailed Implementation

[0029] In the following description, with reference to the accompanying drawings, advantageous modes of the present disclosure will be used to illustrate the embodiments. In the drawings, the same reference numerals are applied to the same components or elements, and repeated descriptions will be omitted or simplified.

[0030] <First Embodiment>

[0031] Figure 1 This is a diagram illustrating a structural example of a photoelectric conversion element 100 according to a first embodiment of the present disclosure. The following explanation uses a photoelectric conversion device as an example, in which the photoelectric conversion element 100 has a so-called stacked structure, wherein two substrates (sensor substrate 11 and circuit substrate 21) are stacked and electrically connected.

[0032] However, a so-called non-stacked structure can be employed, in which components in the sensor substrate 11 and components included in the circuit substrate 21 are arranged in a common semiconductor layer. The sensor substrate 11 includes pixel regions 12, while the circuit substrate 21 includes circuit regions 22 for processing signals detected in the pixel regions 12.

[0033] Figure 2 This is a diagram illustrating an example of the structure of a sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged in a two-dimensional configuration in multiple rows and columns. That is, the photoelectric conversion element 100 includes a plurality of pixels. Each pixel 101 includes a photoelectric conversion unit 102, which includes an avalanche photodiode (hereinafter referred to as an APD).

[0034] In this context, the photoelectric conversion unit 102 serves as a sensor unit configured to emit pulses in response to photons incident on the avalanche photodiode. It should be noted that there are no particular limitations on the number of rows and columns of the pixel array configuring the pixel region 12.

[0035] Figure 3 This is a diagram illustrating an example of the structure of circuit board 21. Circuit board 21 includes components configured for processing in... Figure 2 The photoelectric conversion unit 102 includes a signal processing circuit 103, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114 for the charge converted by photoelectric conversion.

[0036] The vertical scanning circuit 110 receives control pulses supplied from the control pulse generation unit 115 and sequentially supplies the control pulses line by line to a plurality of pixels arranged in the row direction. Logic circuits such as offset registers or address decoders are used in the vertical scanning circuit 110.

[0037] The photoelectric conversion signals output from the photoelectric conversion unit 102 of each pixel are processed by the corresponding signal processing circuit 103. The signal processing circuit 103 is equipped with a counter, a memory, etc., and the digital values ​​are stored in the memory. The horizontal scanning circuit 111 inputs control pulses to the signal processing circuit 103 to sequentially select each column so as to read signals from the memory of each pixel in which the digital signals are stored.

[0038] Signals are output from multiple signal processing circuits 103 corresponding to multiple pixels in a row selected by the vertical scanning circuit 110 to the vertical signal line 113. The signals output to the vertical signal line 113 are output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. Multiple buffers connected to the vertical signal line 113 are built into the readout circuit 112.

[0039] like Figure 2 and Figure 3 As shown, multiple signal processing circuits 103 are arranged in the area overlapping with pixel region 12 in the planar view. Then, vertical scanning circuit 110, horizontal scanning circuit 111, readout circuit 112, output circuit 114 and control pulse generation unit 115 are arranged to overlap with the outer portion of pixel region 12 in the planar view.

[0040] That is, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. Then, the vertical scanning circuit 110, the horizontal scanning circuit 111, the readout circuit 112, the output circuit 114, and the control pulse generation unit 115 are arranged in the region that overlaps with the non-pixel region in the plan view.

[0041] It should be noted that the configuration of the vertical signal line 113, the readout circuit 112, and the output circuit 114 is not limited to... Figure 3 The example shown. For example, the vertical signal line 113 can be arranged to extend in the row direction, and the readout circuit 112 can be arranged at the end of the range of the vertical signal line 113. Furthermore, it is not necessarily necessary to provide a signal processing circuit 103 for each photoelectric conversion unit, and the signal processing circuit 103 can have a structure in which one of the signal processing units is shared by multiple photoelectric conversion units and performs signal processing sequentially.

[0042] Figure 4 It is shown Figure 2 The diagram shows the photoelectric conversion unit 102 of each pixel and the equivalent circuit of the signal processing circuit 103 corresponding to the photoelectric conversion unit 102.

[0043] The APD 201 included in the photoelectric conversion unit 102 generates charge pairs corresponding to the incident light through photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line supplied with a driving voltage VL (first voltage). In addition, the other node of the APD 201 is connected via a switch 202 to a power supply line supplied with a driving voltage VH (second voltage) higher than the driving voltage VL.

[0044] exist Figure 4 In this configuration, one node of the APD 201 is the anode, and the other node is the cathode. A reverse bias voltage is supplied to both the anode and cathode of the APD 201, causing the APD 201 to perform avalanche multiplication operation. By establishing this state of supplying this voltage, the charge generated by the incident light causes avalanche multiplication, and an avalanche current is generated.

[0045] It should be noted that when a reverse bias voltage is supplied, there are Geiger mode and linear mode. In Geiger mode, the voltage difference between the anode and cathode operates at a voltage difference greater than the breakdown voltage. In linear mode, the voltage difference between the anode and cathode operates at a voltage difference close to the breakdown voltage or at a voltage difference equal to or less than the breakdown voltage.

[0046] An APD that operates in Geiger mode is called a SPAD. In the case of a SPAD, for example, the drive voltage VL (first voltage) is -30V and the drive voltage VH (second voltage) is 1V. It should be noted that SPAD is included in the APD category.

[0047] The signal processing circuit 103 includes a switch 202, a waveform shaping unit 210, a counter circuit 211, a memory circuit 212, and a signal generation unit 215. The switch 202 is connected to the power supply line supplied with the drive voltage VH, and is also connected to one of the anode and cathode nodes of the APD 201.

[0048] Then, switch 202 switches the resistance value between APD 201 and the power supply line to which the drive voltage VH is supplied. Here, when switching the resistance value, the resistance value preferably changes by a factor of 10 or more, and more preferably by a factor of 100 or more.

[0049] In the following text, a low resistance value of switch 202 is referred to as switch 202 being turned on, and a high resistance value is referred to as switch 202 being turned off. Switch 202 can be used as a load circuit (quenching circuit) when signal multiplication is performed by avalanche multiplication, and can suppress the voltage supplied to APD 201 and thereby suppress the quenching operation of avalanche multiplication.

[0050] Furthermore, switch 202 can perform a recharging operation, which has the function of returning the voltage supplied to APD 201 to the drive voltage VH by allowing current to flow corresponding to the voltage drop caused by the quenching operation. In other words, switch 202 has the function of switching the avalanche photodiode between a standby state and a recharging state, in which avalanche multiplication is possible.

[0051] Switch 202 can be configured, for example, by a MOS transistor, and Figure 4 The example shown illustrates the case where switch 202 is a PMOS transistor. A control signal CLK (clock signal) for switch 202, supplied from signal generation unit 215, is applied to the gate electrode of the MOS transistor configuring switch 202. In this embodiment, the switching on and off of switch 202 is controlled by controlling the voltage applied to the gate electrode of switch 202.

[0052] Figure 4 An example is shown where, in addition to switch 202, signal processing circuit 103 also includes waveform shaping unit 210, counter circuit 211, and memory circuit 212. Counter circuit 211 counts pulses from waveform shaping unit 210, and memory circuit 212 can hold the count value of counter circuit 211. It should be noted that counter circuit 211 is used as a counter to count the number of pulses.

[0053] The waveform shaping unit 210 shapes the voltage change of the cathode of the APD 201 obtained during photon detection and outputs a pulse signal. The input-side node of the waveform shaping unit 210 is referred to as node A, and the output-side node is referred to as node B. The waveform shaping unit 210 changes the output voltage from node B according to whether the input voltage to node A is equal to or greater than a predetermined value or lower than a predetermined value.

[0054] When the input voltage to node A becomes a high voltage equal to or greater than a predetermined threshold, the output voltage from node B becomes low. Then, when the input voltage to node A becomes a voltage lower than the predetermined threshold, the output voltage from node B becomes high. For example, the inverter circuit is used as waveform shaping unit 210.

[0055] although Figure 4 An example of one inverter being used as waveform shaping unit 210 is shown, but a circuit in which multiple inverters are connected in series can be used, or other circuits with waveform shaping effects can be used.

[0056] First, an explanation of the passive recharging method will be provided. Although the quenching and recharging operations using switch 202 can be performed based on avalanche multiplication in APD 201, there is a possibility that photon detection events may not be recognized as output signals due to reliance on photon detection timing.

[0057] For example, assuming an avalanche multiplication occurs in the APD, the input voltage to node A becomes low, and a recharging operation is in progress. It should be noted that the determination threshold of the waveform shaping unit 210 is set to a voltage higher than the voltage difference at which avalanche multiplication occurs in the APD.

[0058] When a photon is incident at a voltage below a certain threshold at node A due to a recharge operation and at a voltage at which avalanche multiplication may occur in the APD, the voltage at node A decreases due to avalanche multiplication occurring in the APD.

[0059] In other words, because the voltage at node A decreases below a predetermined threshold, the output voltage from node B does not change despite the detection of photons. Therefore, despite avalanche multiplication, the count value of counter circuit 211 does not increase.

[0060] In particular, under high illumination, because photons enter continuously within a short period of time, it becomes difficult to increase the count value. As a result, despite high illumination, the actual number of incident photons and the count value can easily deviate from each other.

[0061] In contrast, according to the following description Figure 5 In the clock-controlled recharging method of this embodiment shown, by... Figure 5 A control signal CLK (clock signal) is applied to switch 202 to periodically switch switch 202 on and off. Therefore, signal detection is possible even when photons continuously enter the APD within a short period of time.

[0062] As described above, the counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. Furthermore, when a control pulse RES is supplied via the RES signal line 213, the count value of the counter circuit 211 is stored in the memory circuit 212, and at the same time, the count value of the counter circuit 211 is reset.

[0063] Here, the counter circuit 211 increments the count from the beginning to the end of the exposure period (cumulative period). It should be noted that in this embodiment, there are multiple exposure periods within a frame period, and the count value in each exposure period is accumulated within a frame period.

[0064] Control pulse SEL via Figure 4 drive line 214 ( Figure 3 (not shown in the image) From Figure 3 The vertical scanning circuit 110 supplies power to the memory circuit 212 and switches the electrical connection and disconnection between the memory circuit 212 and the vertical signal line 113.

[0065] The memory circuit 212 serves as a temporary storage for the count value of the counter, and after temporarily storing the output signal from the counter circuit 211 of each pixel, when the control pulse SEL is supplied, the memory circuit 212 outputs the signal to the vertical signal line 113.

[0066] It should be noted that switches such as transistors can be arranged between switch 202 and APD 201 or between photoelectric conversion unit 102 and signal processing circuit 103 to switch electrical connections. Similarly, the supply of drive voltage VH or drive voltage VL to photoelectric conversion unit 102 can be electrically switched by using switches such as transistors.

[0067] It should be noted that, such as Figure 4 As shown, switch 202 is preferably configured with a single transistor, and both quenching and recharging operations are preferably performed by a single transistor. Therefore, the number of circuits can be reduced compared to the case where quenching and recharging operations are performed by separate circuit elements.

[0068] In particular, when each pixel has a counter circuit and the SPAD signal is read out for each pixel, the circuit area of ​​the switch 202 is preferably minimized due to the presence of the counter circuit, and the effect of configuring the switch 202 with a single transistor becomes significant.

[0069] Figure 5 It is a diagram schematically illustrating the relationship between the operation and output signal of the clock-controlled recharge method, and Figure 5 The example explained is that the control signal CLK, as a clock signal, is a pulse signal with a repetitive period. That is, in... Figure 5 In this circuit, the switch 202 is switched on and off at a predetermined clock frequency.

[0070] also, Figure 5 The diagram schematically illustrates the relationship between the control signal CLK of switch 202, the voltage of node A, the voltage of node B, and the output signal. In this embodiment, when the control signal CLK is high, it becomes difficult to establish a state that supplies the drive voltage VH to the APD, and when the control signal CLK is low, it establishes a state that supplies the drive voltage VH to the APD.

[0071] In other words, the high level of the control signal CLK is, for example, 1V, and the low level of the control signal CLK is, for example, 0V. When the control signal CLK is at a high level, switch 202 is off, and when the control signal CLK is at a low level, switch 202 is on.

[0072] When the control signal CLK is high, the resistance of switch 202 becomes higher than that when the control signal CLK is low. Therefore, when the control signal CLK is high, since no recharging operation occurs even when avalanche multiplication occurs in the APD, the voltage supplied to the APD becomes equal to or less than the breakdown voltage of the APD, and the avalanche multiplication operation in the APD stops.

[0073] exist Figure 5 At time tA, the control signal CLK changes from high to low, and switch 202 turns on, initiating the APD recharging operation. As a result, the voltage at the APD's cathode turns high.

[0074] Then, the voltage difference between the anode and cathode applied to the APD becomes equal to or greater than the breakdown voltage, establishing a state capable of avalanche multiplication. Because the cathode voltage is the same as the voltage at node A, when the cathode voltage transitions from low to high, the voltage at node A becomes equal to or greater than a predetermined threshold at time tB.

[0075] At this point, the pulse signal output from node B is inverted, and the pulse signal changes from high level to low level. Subsequently, a state is established where the voltage difference between the drive voltage VH and the drive voltage VL is applied to APD 201. Furthermore, the control signal CLK then goes high, and switch 202 is turned off.

[0076] Next, at time tC, as Figure 5 As indicated by the black circle and downward arrow, when a photon is incident on APD 201, avalanche multiplication occurs in APD 201. The avalanche multiplication current flows in switch 202, and the voltage at the cathode decreases. That is, the voltage at node A decreases.

[0077] As the voltage drop becomes larger and the voltage difference applied to APD 201 becomes smaller, the avalanche multiplication of APD 201 stops as it did at time tC, and the voltage level of node A does not drop to a value equal to or less than a certain constant. When the voltage of node A falls below a predetermined threshold while the voltage of node A is decreasing, the voltage of node B changes from low to high.

[0078] In other words, when the output waveform drops below a certain threshold at node A, waveform shaping unit 210 performs waveform shaping, and the output waveform is output as a high-level signal at node B. Then, the rising edge of node B is counted by the counter circuit, and the count value of the counter signal output from the counter circuit increases by 1 LSB and becomes n+1.

[0079] In this way, the counter circuit 211 increments its count value whenever the rising edge of the waveform at node B occurs according to the control signal CLK. Then, at the end of the predetermined exposure period, a final count value is generated in the counter circuit 211.

[0080] exist Figure 5 In the process, although photons are incident on the APD between time tC and time tD, the voltage level of node A does not exceed a predetermined threshold because switch 202 is in the off state and the voltage applied to APD 201 does not establish a voltage difference that can avalanche multiplication.

[0081] At time tD, the control signal CLK changes from high to low, and switch 202 turns on. With this change, current compensating for the voltage drop flows to node A, and the voltage at node A changes from the drive voltage VL back to its original level.

[0082] At this point, because the voltage of node A becomes equal to or greater than a certain threshold at time tE, the pulse signal of node B reverses and changes from high level to low level.

[0083] At time tF, node A stabilizes at its original voltage level, and thereafter, the control signal CLK changes from low to high. Therefore, switch 202 turns off. Even thereafter, as explained from time tA to time tF, the voltages of the nodes and signal lines, etc., change according to the control signal CLK and the incidence of photons.

[0084] Next, an explanation will be provided for the light emitter 500, camera 600, and moving body 700, which are the imaging devices in this embodiment.

[0085] Figure 6 This is a functional block diagram illustrating a structural example of the emitter 500, camera 600, and moving body 700 according to the first embodiment. It should be noted that... Figure 6 Some of the functional blocks shown are implemented by causing the computer (not shown) included in the illuminator 500, the computer (not shown) included in the camera 600, and the computer (not shown) included in the moving body 700 to execute computer programs stored in a memory that serves as a storage medium (not shown).

[0086] However, some or all of the functional blocks can be implemented in hardware. As for the hardware, dedicated application-specific integrated circuits (ASICs) and processors (reconfigurable processors, DSPs, etc.) can be used. Furthermore, Figure 6 The functional blocks shown do not need to be built into the same housing, and the functional blocks can be configured by separate devices connected to each other via signal paths.

[0087] The camera 600 includes a photoelectric conversion element 100, an imaging optical system 601, an image processing unit 603, a recognition unit 604, a camera control unit 605, a storage unit 606, and a communication unit 607. The photoelectric conversion element 100 comprises components for photoelectric conversion of optical images. Figures 1 to 5 The avalanche photodiode configuration described in [the document].

[0088] The imaging device (camera 600 and emitter 500) of this embodiment is mounted on the moving body 700, and includes an imaging unit 602 consisting of an imaging optical system 601 and a photoelectric conversion element 100, configured to capture images in at least one direction, such as the forward, backward, and lateral directions of the moving body. It should be noted that multiple imaging units 602, or multiple imaging devices, can be provided on the moving body 700.

[0089] The image processing unit 603 performs image processing on the image signal acquired by the photoelectric conversion element 100, such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaic processing, and data compression, and generates a final image signal. It should be noted that when the photoelectric conversion element 100 has an on-chip color filter such as RGB, the image processing unit 603 can perform processing such as white balance correction and color conversion.

[0090] Furthermore, the output of the image processing unit 603 is supplied to the recognition unit 604, the camera control unit 605, and the ECU (electronic control unit) 701 of the moving body 700. The recognition unit 604 performs image recognition based on image signals to identify objects such as people or vehicles in the vicinity. Deep learning is used for this recognition process.

[0091] For example, YOLO (YouOnly Look Once) is preferred as a deep learning method, as it enables easy learning and fast detection. Other types of deep learning methods include SSD (Single-Shot Multi-Box Detector), faster R-CNN (Region Convolutional Neural Network), Fast R-CNN, and R-CNN.

[0092] Furthermore, in this embodiment, the identification unit 604 calculates the distance to the identified object. That is, the identification unit 604 calculates, for example, a first distance range and a second distance range by identifying the subject. It should be noted that, as a distance measurement method, distance estimation can be performed using, for example, deep learning. That is, for example, the distance value can be calculated by using deep learning to analyze information such as the blurriness of the image of the detected object.

[0093] As an alternative method, a distance measurement can be performed using a camera device that functions as a stereo camera and employing the principle of triangulation. Alternatively, by making the photoelectric conversion element a phase difference detection type camera element, the distance can be measured using the phase difference signal from the photoelectric conversion element. Recognition processing, including distance estimation, is performed on each of the color image and IR (infrared) image input from the image processing unit 603, and the recognition results are output to the ECU 701 in a subsequent stage.

[0094] It should be noted that although this embodiment will be explained by using a car as an example of a mobile body 700, the mobile body can be any type of mobile body, as long as the mobile body can move, such as aircraft, trains, ships, drones, AGVs and robots, etc.

[0095] The camera control unit 605 includes a CPU that functions as a computer and a memory that stores computer programs, and controls various parts of the camera 600 by causing the CPU to execute the computer programs stored in the memory.

[0096] It should be noted that the camera control unit 605 serves as a control component, and controls, for example, the length of the exposure period (cumulative period, counting period in the counter) of each frame of the photoelectric conversion element 100 and the timing of the control signal via the control pulse generation unit of the photoelectric conversion element 100. Furthermore, the camera control unit 605 transmits a signal identical to the aforementioned reference signal to the light emitter 500 via the communication unit 607.

[0097] In this way, the same reference signal transmitted to the photoelectric conversion element 100 is also transmitted to the light emitter 500, and the light emitter 500 performs light emission control based on the reference signal, thereby performing synchronous control of the exposure timing inside the photoelectric conversion element 100 and the light emission timing of the light emitter 500.

[0098] Storage unit 606 includes a recording medium such as a memory card or hard disk, and can store and read image signals. Communication unit 607 includes wireless and wired interfaces, outputs generated image signals to the outside of camera 600, and receives various signals from the outside.

[0099] Furthermore, in this embodiment, the communication unit 607 is connected to the communication unit 503 of the emitter 500, and is also used to transmit the aforementioned reference signals and control commands from the camera control unit 605 to the emitter 500.

[0100] The emitter 500 includes an emitter unit 501, an emitter control unit 502, and a communication unit 503. The emitter unit 501 includes, for example, a near-infrared LED for illuminating a subject in front of the moving body 700, and the emitter unit 501 can be combined with a lens to illuminate a light beam. Furthermore, the emitter unit outputs near-infrared pulsed light at a predetermined emission time according to a pulse signal output from the emitter control unit 502.

[0101] The light-emitting control unit 502 receives a reference signal transmitted by the camera control unit 605 of the camera 600 via the communication unit 503, generates a pulse signal based on the reference signal at a predetermined timing, and outputs the pulse signal to the light-emitting unit 501.

[0102] In this context, the light emission control unit 502 can set the time period from the reference signal to the pulse output, the pulse output width, the pulse non-output width, and the repetition period and repetition count from the pulse output to the next pulse output.

[0103] The camera control unit 605 sets a predetermined value for the light-emitting control unit 502 via the communication unit 607 and the communication unit 503, outputs a pulse signal to the light-emitting unit 501 at a predetermined timing based on the reference signal, and controls the light-emitting period of the light-emitting unit 500.

[0104] In this way, the light-emitting control unit 502 performs light-emitting control based on the same signal as the reference signal input to the photoelectric conversion element 100. That is, the light-emitting unit 501 performs pulsed light emission for illuminating the subject in sync with the control signal CLK, which is a clock signal.

[0105] The communication unit 503 communicates with the communication unit 607 of the camera 600 and receives setting information and reference signals sent by the camera control unit 605 to the light-emitting control unit 502.

[0106] ECU 701 includes a CPU that functions as a computer and a memory that stores computer programs, and controls the various parts of the moving body 700 by having the CPU execute the computer programs stored in the memory.

[0107] The output of ECU 701 is supplied to vehicle control unit 702 and display unit 703. Vehicle control unit 702 serves as a movement control component that performs driving, stopping, and directional control of the vehicle as a moving body based on the output of ECU 701. Furthermore, display unit 703 serves as a display component, including display elements such as liquid crystal devices and organic EL, and is mounted on the moving body 700.

[0108] In this embodiment, the ECU 701 receives the recognition result information from the recognition unit 604, and the ECU 701 can perform vehicle stop control (automatic braking, etc.) according to the content of the recognition result. In addition, the ECU 701 receives color images and IR images from the image processing unit 603, and transmits the color images and IR images together with the recognition result to the display unit 703.

[0109] Based on the output of ECU 701, display unit 703 displays various information to the driver of vehicle 700, such as images acquired by photoelectric conversion element 100, recognition results of recognition unit 604, and vehicle driving status, using a GUI, for example.

[0110] It should be noted that Figure 6 The image processing unit 603 and the recognition unit 604 do not need to be installed on the mobile body 700. The image processing unit 603 and the recognition unit 604 can be set in an external terminal for remotely controlling the mobile body 700 or for monitoring the movement of the mobile body, and can be set separately from the mobile body 700.

[0111] Figure 7 This is a diagram illustrating an example of the relationship between the emission from the light emitter 500, the travel of reflected light, and the exposure timing of the camera 600 according to the first embodiment. Figure 7 The distance on the horizontal axis and the time on the vertical axis are shown.

[0112] like Figure 7 As shown, in this embodiment, an image (range-gated image) of the target distance range is obtained by controlling the synchronous emission timing and exposure timing according to the target distance range (distance gating control).

[0113] It should be noted that in this embodiment, the camera that acquires the target distance image through distance gating control in this manner is called a distance-gated camera.

[0114] First, an explanation will be provided for the horizontal axis. Figure 7 In the example shown, fog 810 exists between distance x1 and distance x2, and vehicle 820 exists at distance x3. Furthermore, in Figure 7 In this process, the distance gating control uses the position of the distance D to the vehicle 820 used as the target as the starting point, and acquires a distance gating image of the target distance range R from that position.

[0115] In this case, the target distance range R becomes the target distance range to be photographed. At this time, vehicle 820 exists within the target distance range R.

[0116] Next, an explanation will be provided for the vertical axis. Time 0 is set as the start timing for the emission of the emitter 500, and time tf is set as the end timing for the emission. The emission period is then defined as time tf. Furthermore, when acquiring a distance-gated image within the target distance range R using the location at distance D as the starting point, the exposure start time is set to time t1, and the exposure end time is set to time t2.

[0117] Time t1 is the timing at which the radiant light emitted from the emitter 500 returns to the camera 600 as reflected light from an object at a distance D at time 0. Furthermore, time t2 is the timing at which the radiant light emitted from the emitter 500 returns to the camera 600 as reflected light from an object at a distance (D+R) at time tf.

[0118] Furthermore, the timing for the initial reflected light from fog 810 to return to camera 600 is set to time t3, and the timing for the final reflected light from fog 810 to return to camera 600 is set to time t4.

[0119] In distance gating control, no exposure is performed during the time period from time t3 to time t4 when reflected light from fog 810 arrives at camera 600. Then, by performing exposure only during the time period from time t1 to time t2 when reflected light from a range extending from distance D through the target distance range R arrives, a clear image of vehicle 820 can be acquired while removing fog 810.

[0120] Here, an explanation will be provided regarding the time until the reflected light from the target object located at a distance x returns to the camera 600. The time until the radiated light emitted from the emitter 500 strikes the target object located at a distance x and returns to the camera unit as reflected light is set as time tr. At this time, the relationship between the time tr until the reflected light returns and the distance x to the target object to be photographed is the following equation (1).

[0121] Time tr = 2x / speed of light c (approximately 3 × 10⁻⁶) 8 (m / s) Equation (1)

[0122] like Figure 7 As shown, when the target distance range R extending from distance D is set as the camera range, the exposure timing t1 of the starting point of the time range corresponding to the target distance range R can be obtained by substituting distance D for distance x in the above equation (1) in the following equation (2).

[0123] Time t1 = 2D / speed of light c Equation (2)

[0124] In addition, the exposure timing t2 of the end point of the range can be obtained by replacing the distance x in the above equation (1) with the distance D + target distance range R and adding the time tf to the result in the following equation (3).

[0125] Time t2 = Time tf + 2(D + R) / Speed ​​of light c Equation (3)

[0126] In this way, the time tf from the start of illumination to the end of illumination, the time t1 from the start of illumination to the start of exposure, and the time t2 until the end of exposure are controlled according to the distance x (target distance range R) to be photographed. Thus, even when there is fog or other conditions between the camera and the target distance range, distance gating control is achieved to clearly photograph subjects within the target distance range.

[0127] Figure 8 This is a timing diagram explaining the control operations for obtaining a range-gated image within a one-frame time period according to the first embodiment. In this embodiment, as described above, the range-gated image is generated by exposure synchronized with the emission of the emitter 500.

[0128] exist Figure 8 In this context, "Vertical Sync Signal" indicates the frame period for image capture, and the time interval from a low pulse to the next low pulse is one frame period. "Emitting Control" indicates the emitting timing of the emitter 500, and the emitter 500 emits light during the high level period. "Exposure Control" indicates the counting period of the counter circuit, and the counter circuit counts photons during the high level period.

[0129] The “counter value” indicates the increase or decrease in the number of photons counted by the counter circuit. The “RES signal” indicates the control pulse supplied to the counter circuit via the RES signal line 213, and the count value held in the counter circuit is reset by this pulse.

[0130] Next, an explanation of the range gating control used to obtain the range-gated image will be provided. In this embodiment, the emission period of light is controlled in a pulse manner by the light emission control unit 502, and photon counting is performed only on reflected light from a predetermined target distance range.

[0131] In this way, the period of light emission from the start to the end of light emission is set as time tf, the time from the start of light emission to the start of exposure (photon counting) is set as time t1, and the time from the start of light emission to the end of exposure (photon counting) is set as time t2.

[0132] At this point, time t1 indicates the period from the start of illumination until the reflected light from the target distance range returns to the camera 600. Furthermore, the period from time t1 to time t2 is the period during which the number of photons of the reflected light from the target distance range is counted, and this period extends from the start to the end of the exposure. The counter value increases according to the number of photons during the exposure period.

[0133] To perform distance gating control correctly, the timing of the start of illumination and the start of exposure needs to be synchronized according to a predetermined target distance range. In this embodiment, the camera control unit 605 synchronizes the operating timing of both the counter circuit 211 and the illumination control unit 502 by transmitting the same reference signal to both.

[0134] In addition, such as Figure 8 As shown in the timing diagram under "Emitting Light Control," the time interval from the start of emitting light to the start of the next emitting light constitutes a distance gating operation cycle. Then, while the counter value counted in one distance gating operation cycle is maintained, the counter value is incremented in the next distance gating operation cycle. It should be noted that the time interval from the start of emitting light to the next emitting light is set based on the time until the reflected light sufficiently decays and stops returning to the camera 600.

[0135] like Figure 8 As shown, a predetermined number of distance gating operations (e.g., hundreds to tens of thousands) are performed within a frame period, and the count value of the counter circuit is accumulated each time. Then, the last accumulated counter value information within a frame period is sent from the counter circuit 211 to the memory circuit 212 via the RES signal, and thereafter the counter value is reset by the RES signal.

[0136] In this way, in distance gating control, because the exposure period is synchronized with the emission of the illuminator 500, a clear image of the target range can be obtained even in adverse weather conditions such as fog.

[0137] Next, Figure 9A and Figure 9B This is a diagram used to explain the relationship between the control signal CLK and the exposure timing according to the first embodiment. Figure 9A and Figure 9B This illustrates the relationship between the radiated light from the emitter 500, the progression of its reflected light, and the range captured by the camera 600 through exposure when a SPAD combining a distance-gated camera and a clock-controlled recharge method is used.

[0138] It should be noted that Figure 9A This diagram is used to explain why reflected light from a specific distance range cannot be exposed during the recharge period of the clocked recharge method when acquiring a range-gated image.

[0139] exist Figure 9A and 9B In, similar to Figure 7 The distance is shown on the horizontal axis and the time is shown on the vertical axis. Furthermore, time 0 is set as the start timing for the emission of the emitter 500, and time tf is set as the end timing for the emission. The emission period is then defined as time tf. When the target distance range is set from distance x4 to distance x5, the exposure start time is set to time t5, and the exposure end time is set to time t6. It is also assumed that vehicle 820 is within the target distance range.

[0140] During the exposure period from time t5 to time t6, such as Figure 4 As explained in the text, the clock-controlled recharging method is used to apply the control signal CLK to the switch 202 and perform a recharging operation.

[0141] exist Figure 9A The document provides an explanation for the case where there are six clocks between time t5 and time t6. Each clock is designated as CLK1, CLK2, CLK3, CLK4, CLK5, and CLK6, and the timing of each clock is as follows: Figure 9A As shown on the vertical axis.

[0142] Figure 9B This is a diagram showing the distance timing from CLK1 to CLK6. Figure 9B Only the portion corresponding to the target distance range between time t5 and time t6 is displayed, where time t5 to time t6 is Figure 9A The exposure time on the vertical axis.

[0143] As in Figure 5 As described in the explanation of the clock-controlled recharge method, incident photons are not counted during the period from when the control signal CLK transitions to low until the threshold voltage is exceeded through the recharge operation.

[0144] In other words, for each of the six clocks from CLK1 to CLK6, since the incident photons are not counted during the period before the threshold voltage is exceeded by the recharge operation, no exposure will occur during those recharge periods when the photons are not counted (using the counting operation of counter circuit 211).

[0145] like Figure 9B As shown, the clock duty cycle is 50:50, and the period until the threshold voltage is exceeded through a recharge operation is assumed to be equivalent to the low period of the clock. In this case, even when the control signal CLK is 30MHz (which is its current maximum value), the period of CLK becomes 33.3ns.

[0146] When considering the speed of light, light travels approximately 10 meters in 33.3 ns. At this point, since the clock duty cycle is 50:50, the low-period corresponds to a light travel distance of approximately 5 meters per clock cycle.

[0147] like Figure 9B As shown, when a distance of approximately 5 meters cannot be captured, a complete vehicle may not be fully exposed.

[0148] Next, Figure 10 This is a diagram illustrating an example of the relationship between emission and control signal CLK at each exposure timing according to the first embodiment, and also showing... Figure 9B The relationship between the target distance range and the control signal CLK in a similar distance gating operation.

[0149] like Figure 9B As explained in the text, when exposure is performed within the target distance range, it is assumed that there are six control signal CLK cycles. During the period from when the control signal CLK transitions to low until the threshold voltage is exceeded through a recharge operation, photons cannot be counted.

[0150] Therefore, image capture is not possible within this range. However, in distance-gated cameras, such as... Figure 8 As explained in the text, in order to acquire a range-gated image, the exposure operation is repeated hundreds to tens of thousands of range-gated operation cycles, and the image is acquired by accumulating the count values ​​obtained from these exposure operations.

[0151] In other words, in this embodiment, in order to capture an image of a subject existing within a predetermined camera distance range, the counter circuit 211 performs multiple exposure operations (counting operations) based on the timing of pulse emission and the predetermined camera distance range.

[0152] It should be noted that, such as Figure 10 As shown, the clock duty cycle can be 50:50, and the period before the threshold voltage is exceeded by the recharge operation can be equal to the low period of CLK. In this case, by changing the phase of the control signal CLK by 180° between the 2n (even) exposure period and the 2n+1 (odd) exposure period, images of the entire target distance range can be acquired.

[0153] In other words, in this embodiment, by shifting the control signal CLK by half a phase between even-numbered and odd-numbered exposure periods, the timing of light emission and exposure is changed based on the control signal CLK.

[0154] In this way, in this embodiment, a control step is performed to offset a predetermined phase between the control signal CLK, which serves as a clock signal for each predetermined exposure operation, and the pulse emission. Therefore, in distance gating control, photon counting omissions within a predetermined imaging range can be prevented.

[0155] Furthermore, despite Figure 10 The explanation provided is that the phase is shifted by 180° between even-numbered and odd-numbered exposure periods, but the amount of phase shift and the period in which it is shifted can be arbitrarily set based on the time period until the threshold voltage is exceeded through recharge operation and the clock frequency. That is, the predetermined phase can be set based on the ratio between the period of the control signal CLK, which serves as the clock signal, and the time period of the recharge state of the avalanche photodiode.

[0156] For example, if the relationship between the period until the threshold voltage is exceeded by a recharge operation and the period of a clock is 1:4, images can be uniformly acquired across the entire target distance range by shifting the phase by 1 / 4 in each of the 4n, 4n+1, 4n+2, and 4n+3 exposure periods.

[0157] Figure 11 This is a flowchart illustrating details of an operational example of the imaging method according to the first embodiment. In this flowchart, each step from S101 to S116 is executed sequentially by a computer program stored in memory, executed by a CPU or the like, which is a computer in the camera control unit 605.

[0158] exist Figure 11 In step S101, the camera control unit 605 obtains weather information by having the recognition unit 604 of the camera 600 determine the weather conditions (sunny, rainy, foggy, etc.) in front of the vehicle, and the process proceeds to step S102.

[0159] In step S102, the camera control unit 605 determines whether the weather information acquired in step S101 indicates severe weather conditions. If it is determined that the weather conditions are severe, the process proceeds to step S103. If it is determined that the weather conditions are not severe, the process proceeds to step S115.

[0160] In step S103, the camera control unit 605 initiates the distance strobe control mode. That is, the camera control unit 605 sets the camera 600 to the distance strobe control mode and sends a control signal to the emitter 500 via the communication unit 607 to set the emitter 500 to the distance strobe control mode. After this, the process proceeds to step S104.

[0161] In step S104, the camera control unit 605 sets i=1 and j=1, and the process proceeds to step S105.

[0162] In step S105, the camera control unit 605 begins recording the i-th target distance range out of N target distance ranges during recording in distance gating control mode. Afterward, the process proceeds to step S106.

[0163] In step S106, the camera control unit 605 determines whether j is odd. In this context, j indicates the j-th exposure operation among a predetermined exposure operation (hundreds to tens of thousands) within a frame period set in the distance gating control.

[0164] If j is odd, the process proceeds to step S107; if j is even, the process proceeds to step S108. In this embodiment, based on the determination result in step S106, the timing of the light emission is adjusted to offset the control signal CLK applied to switch 202 by half a phase. That is, for each exposure period, the control signal CLK is alternately offset by half a phase.

[0165] In step S107, light emission is performed at a predetermined timing for the control signal CLK. That is, the camera control unit 605 causes the emitter 500 to emit light at a normal timing without shifting the light emission timing by half a phase relative to the control signal CLK applied to the switch 202. After the emitter 500 completes its light emission, the process proceeds to step S109.

[0166] In step S108, illumination is performed at a timing offset by half a phase from the predetermined timing relative to the control signal CLK. That is, the camera control unit 605 causes the emitter 500 to emit light at a timing offset by half a phase from the illumination timing of the control signal CLK applied to the switch 202. After the emitter 500 completes illumination, the process proceeds to step S109.

[0167] In step S109, exposure is performed within the target distance range. That is, the camera control unit 605 performs exposure in the imaging unit 602 at a timing point corresponding to the target distance range (using the counting operation of the counter circuit 211). After the exposure is completed, the process proceeds to step S110.

[0168] In step S110, the camera control unit 605 sets j=j+1 to increment j by 1, and the process proceeds to step S111.

[0169] In step S111, it is judged whether j < M. That is, the camera control unit 605 judges whether the M - time exposure has been reached. The M - time exposure is a predetermined exposure operation within a target distance range in the distance - gated control. When it is judged as "No" in step S111, that is, when the M - time exposure has been reached, the process proceeds to step S112. When it is judged as "Yes" in step S111, that is, when the M - time exposure has not been reached, the process returns to step S106.

[0170] In step S112, an image is acquired. That is, the imaging unit 602 generates image data based on the cumulative count value of the M - time exposure operation and sends the image data to the image processing unit 603. After the image processing in the image processing unit 603 is completed, the data is transferred from the image processing unit 603 to the subsequent recognition unit 604 and ECU 701. Thereafter, the process proceeds to step S113.

[0171] In step S113, the camera control unit 605 sets i = i + 1 to increment i by 1, and the process proceeds to step S114.

[0172] In step S114, it is judged whether i < N. That is, it is judged whether all N target distance ranges set by the camera control unit 605 have been completed. When it is judged as "Yes" in step S114, that is, when i is less than N, the process returns to step S105. When it is judged as "No" in step S114, that is, when i is greater than or equal to N, Figure 11 the processing flow ends.

[0173] In step S115, the normal imaging mode is started. That is, the camera control unit 605 transmits a control signal to operate in the normal imaging mode. After the imaging starts in the normal imaging mode, the process proceeds to step S116.

[0174] It should be noted that the normal imaging mode in this article refers to a mode in which, without the light - emitting unit emitting light, imaging is performed by counting operations (exposure operations) by the counter circuit 211 only for a predetermined exposure period within one frame period to generate an image.

[0175] In step S116, an image is acquired. That is, the imaging unit 602 sends the captured image data to the image processing unit 603. After the image processing in the image processing unit 603 is completed, the data is transferred from the image processing unit 603 to the subsequent recognition unit 604 and ECU 701. Thereafter, Figure 11 the processing flow ends.

[0176] It should be noted that althoughFigure 11 The flowchart describes an example of offsetting half a phase, but as... Figure 10 As described, the phase offset and the period at which the offset occurs can be arbitrarily set based on the ratio between the period of the control signal CLK and the recharge period.

[0177] Furthermore, although the phase is changed for each exposure operation in this embodiment, the frequency of change does not necessarily need to be specific to each exposure operation. The phase can be changed at intervals between multiple exposure operations, or the phase can be changed between the first and second halves of the total number of exposure operations.

[0178] Furthermore, although in this embodiment, the half-phase shift is achieved by changing the timing of the emission with the control signal CLK as a reference, the timing of the emission can remain constant, and the control signal CLK can be changed for each emission.

[0179] In this way, according to this embodiment, in a distance-gated camera using a SPAD employing a clock-controlled recharge method, even when distance gating is performed, the range of distances that cannot be captured due to the recharge period is unlikely to occur, and the overall image quality can be improved.

[0180] <Second Embodiment>

[0181] The following explanation will focus on a second embodiment of the present disclosure. In the first embodiment, an explanation of a method for a SPAD sensor using a clocked recharge method was provided. In the second embodiment, an explanation will be provided for a sensor that can acquire two images with parallax and perform stereo distance measurement by further having two photoelectric conversion units in a pixel of the clocked recharge method SPAD sensor.

[0182] It should be noted that the functional block diagram in the second embodiment is different from... Figure 6 The functional block diagram described in the previous section is basically the same, and the camera unit 602 is changed to a SPAD sensor with a structure having two photoelectric conversion units in one pixel.

[0183] Figure 12A and Figure 12B This is a diagram illustrating a structural example of a camera element having two photoelectric conversion units in one pixel according to the second embodiment. Figure 12A This is a top view of the sensor substrate 11 as seen from the direction of light incidence. The sensor substrate 11 is configured by arranging multiple pixel groups 1000, each consisting of four pixels arranged in a 2x2 matrix.

[0184] Pixel group 1000 has a green pixel G1 for detecting green light, a green pixel G2 for detecting green light, a red pixel R for detecting red light, and a blue pixel B for detecting blue light. In pixel group 1000, green pixel G1 and green pixel G2 are arranged diagonally.

[0185] Furthermore, each pixel has a first photoelectric conversion unit 102A and a second photoelectric conversion unit 102B that receive light from different pupils. The control of the first photoelectric conversion unit 102A and the second photoelectric conversion unit 102B can be controlled independently.

[0186] Figure 12B yes Figure 12A A cross-sectional view of pixel group 1000 along the I-I' section. Each pixel is configured with a microlens 1003, a light guide layer 1004, and a light-receiving layer 1005.

[0187] The light guide layer 1004 includes a microlens 1003 for effectively guiding the light flux incident on the pixel to the light receiving layer 1005, a color filter that allows light of a wavelength band corresponding to the color of the light detected by each pixel to pass through, and a light guide member having wiring for image readout and pixel driving.

[0188] The light-receiving layer 1005 is a photoelectric conversion unit that outputs an electrical signal by photoelectric conversion of light incident through the light guide layer 1004, and the light-receiving layer 1005 has a first photoelectric conversion unit 102A and a second photoelectric conversion unit 102B.

[0189] Furthermore, in the above explanation, such as Figure 12A As shown, although the pixel group 1000 is arranged as green pixel G1, green pixel G2, red pixel R and blue pixel B, the arrangement of the pixel group 1000 is not limited to this, and infrared pixels IR that receive infrared light can also be arranged, and the arrangement order is not restricted.

[0190] It should be noted that, in the following explanation, the output of the first photoelectric conversion unit 102A is referred to as the first photoelectric conversion signal, and the output of the second photoelectric conversion unit 102B is referred to as the second photoelectric conversion signal. Furthermore, the image signal generated from the first photoelectric conversion signals of multiple pixels is referred to as the first image signal, and the image signal generated from the second photoelectric conversion signals of multiple pixels is referred to as the second image signal.

[0191] It should be noted that because the first photoelectric conversion unit 102A and the second photoelectric conversion unit 102B each perform photoelectric conversion by receiving light from different exit pupils of the imaging lens via microlenses, the first photoelectric conversion signal and the second photoelectric conversion signal have parallax. Therefore, the first image signal and the second image signal have a phase difference corresponding to the parallax amount (positional deviation amount), and the distance to the subject can be calculated based on the phase difference. Furthermore, when acquiring an image for display, the first image signal and the second image signal are added together.

[0192] Figure 13 It is shown that... Figure 12A The diagram shows the equivalent circuit of the signal processing circuit corresponding to one pixel in pixel group 1000. Although the equivalent circuit is basically similar to... Figure 4 The equivalent circuit shown is provided, but because a pixel has a first photoelectric conversion unit 102A and a second photoelectric conversion unit 102B, two sets of circuits, each including an APD, a switch, a waveform shaping unit and a counter circuit, are set in the pixel.

[0193] In other words, 201A is an APD included in the first photoelectric conversion unit 102A, and 201B is an APD included in the second photoelectric conversion unit 102B. Furthermore, switch 202A is connected to the cathode of APD 201A, switch 202B is connected to the cathode of APD 201B, waveform shaping unit 210A is connected to the cathode of APD 201A, and waveform shaping unit 210B is connected to the cathode of APD 201B.

[0194] Pixel switches 222A, 222B, 223A, and 223B are pixel switches. Pixel switches 222A, 222B, 223A, and 223B switch whether to supply a first photoelectric conversion signal to each counter, whether to supply a second photoelectric conversion signal to each counter, or whether to supply both the first and second photoelectric conversion signals to each counter.

[0195] Furthermore, counter circuit 211A is connected to the output of OR circuit 224A, and counter circuit 211B is connected to the output of OR circuit 224B. It should be noted that counter circuit 211A serves as a first counter capable of counting the output of the first photoelectric conversion unit, and counter circuit 211B serves as a second counter capable of counting the output of the second photoelectric conversion unit. The output of OR circuit 224A is connected to counter circuit 211A, and the output of OR circuit 224B is connected to counter circuit 211B.

[0196] It should be noted that, as described above, pixel switches 222A, 222B, 223A, and 223B selectively connect the output of the first photoelectric conversion unit to the first counter and connect the output of the second photoelectric conversion unit to the second counter.

[0197] For example, when the first photoelectric conversion signal is counted by the counter circuit 211A, pixel switch 222A is turned on and pixel switch 222B is turned off. Furthermore, when the second photoelectric conversion signal is counted by the counter circuit 211A, pixel switch 222B is turned on and pixel switch 222A is turned off.

[0198] In this manner, this embodiment has a first switching state in which the pixel switch connects one output from the first photoelectric conversion unit to the first counter and another output from the second photoelectric conversion unit to the second counter.

[0199] When both the first and second photoelectric conversion signals are counted by counter circuit 211A, pixel switches 222A and 222B are turned on. The same applies to counter circuit 211B.

[0200] In this manner, this embodiment has a second switching state in which the outputs of the first photoelectric conversion unit and the second photoelectric conversion unit are connected to, for example, a first counter via a pixel switch.

[0201] Figures 14A to 14C This is a diagram illustrating an example of the operating mode when using an imaging element with two photoelectric conversion units in a single pixel. Figure 14A This diagram illustrates the mode for distance measurement when shooting the entire area (shooting range R1) in normal camera mode.

[0202] In normal camera mode, only Figure 13 Pixel switches 222A and 223B, as explained in the diagram, are turned on. Therefore, counter circuit 211A acquires a first photoelectric conversion signal, and counter circuit 211B acquires a second photoelectric conversion signal. Furthermore, subject distance data is acquired based on the phase difference between the first and second image signals from multiple pixels.

[0203] In other words, the distance to the subject can be calculated based on the outputs of the first counter and the second counter in the first switching state described above.

[0204] In contrast, when acquiring an image for display, the image obtained by adding the first image signal and the second image signal can be acquired by adding the count values ​​of counter circuit 211A and counter circuit 211B. That is, the display image can be generated based on, for example, the output of the first counter in the second switching state described above.

[0205] Figure 14B This diagram illustrates an example of performing distance gating control and acquiring an image within the range of the camera's field of view R2. Subject distance data can be obtained by acquiring both a first image signal and a second image signal within the range of the camera's field of view R2.

[0206] Similarly, in this mode, it is similar to Figure 14A ,only Figure 13 The pixel switches 222A and 223B are turned on, and the first photoelectric conversion signal is obtained by the counter circuit 211A, and the second photoelectric conversion signal is obtained by the counter circuit 211B.

[0207] Then, a first image signal is generated based on the output of the counter circuit 211A with multiple pixels, and a second image signal is generated based on the output of the counter circuit 211B with multiple pixels. The subject distance data is calculated based on the phase difference between the first image signal and the second image signal.

[0208] Furthermore, in the second embodiment, when using the clock-controlled recharge method, similar to the first embodiment, the operation is performed by changing the phase of the light emission timing relative to the control signal CLK, so that the area that cannot be photographed will not occur.

[0209] Figure 14C This diagram illustrates an example of simultaneously acquiring the target distance range at two locations using distance gating control. The number of illumination cycles can be reduced by simultaneously acquiring the target distance range at two locations. Furthermore, this provides the advantage that the target distance range at each location can be further narrowed.

[0210] In this embodiment, when images are captured for target distance ranges at two locations in this manner, counter circuits 211A and 211B count both the first and second photoelectric conversion signals for each target distance range. Then, images corresponding to each target distance range are acquired based on the count values ​​of counter circuits 211A and 211B.

[0211] For example, suppose the camera range R3 is obtained by counter circuit 211A, and suppose the camera range R4 is obtained by counter circuit 211B. At this time, at the timing of exposing the camera range R3, both pixel switches 222A and 222B are turned on, and at the timing of exposing the camera range R4, both pixel switches 223A and 223B are turned on.

[0212] At this point, as explained in the first embodiment, operation is performed by changing the phase of the emission timing relative to the control signal CLK, so that the area that cannot be photographed does not occur. However, as Figure 14C As shown, when the first photoelectric conversion signal and the second photoelectric conversion signal are counted by a counter, there is a possibility of count omission in the clock-controlled recharge method.

[0213] In other words, when photons simultaneously enter the first photoelectric conversion unit 102A and the second photoelectric conversion unit 102B within the same clock cycle, a counting omission occurs. Details of the counting omission are... Figure 15 Examples and explanations are provided.

[0214] Figure 15 This diagram illustrates the problems that arise when both the first and second photoelectric conversion signals are counted, and shows the relationship between the target distance range and the control signal CLK. The incident light to the first photoelectric conversion unit 102A is set as incident light A, and the incident light to the second photoelectric conversion unit 102B is set as incident light B.

[0215] The control signals CLK for each photoelectric conversion unit are designated as CLK A and CLK B. At this time, in the clock-controlled recharge method, as explained in the first embodiment, there is a problem that photons incident during the low periods of the clock cannot be counted.

[0216] Furthermore, when counting the first photoelectric conversion signal and the second photoelectric conversion signal, such as Figure 15 As shown in the range of CLK3, there exists a situation where, within the same clock cycle, photons of incident light A simultaneously strike the first photoelectric conversion unit 102A and photons of incident light B simultaneously strike the second photoelectric conversion unit 102B. In this case, because only one count can be performed within the same clock cycle, a count omission occurs.

[0217] Especially at locations where objects exist, when count omissions occur frequently, the original counts do not match, resulting in images with a poor S / N ratio.

[0218] In comparison, Figure 16 This shows the solution used to solve Figure 15The diagram illustrates an example of the method for explaining the problem, and shows an example of the relationship between the emission timing and the control signal CLK in each photoelectric conversion unit according to the second embodiment, in the context of the control signal CLK and each exposure timing.

[0219] First, to prevent the camera range from being missed during low-period exposures, similar to the first embodiment, the camera range can be prevented by shifting the light emission timing of the emitter 500 by half a phase during the 2nth exposure period and the 2n+1th exposure period.

[0220] In addition, to prevent counting omissions, the emission timing is shifted by half a phase during the 2nth exposure period and the 2n+1th exposure period, and the control signal CLK applied to the first photoelectric conversion unit 102A and the control signal CLK applied to the second photoelectric conversion unit 102B are also shifted by half a phase.

[0221] In other words, such as Figure 16 As shown, during the 2nth exposure period, the control signal CLK A applied to the first photoelectric conversion unit 102A and the control signal CLK B applied to the second photoelectric conversion unit 102B are offset by half a phase. It should be noted that the offset is not limited to half a phase. That is, the clock signals supplied to the avalanche photodiode of the first photoelectric conversion unit and the avalanche photodiode of the second photoelectric conversion unit can be controlled to offset their phases by a predetermined phase.

[0222] Furthermore, the control signals CLK A in the 2nth exposure period and CLK B in the 2n+1th exposure period are shifted by half a phase, and the control signals CLK B in the 2nth exposure period and CLK B in the 2n+1th exposure period are also shifted by half a phase. Therefore, count omissions can be reduced.

[0223] It should be noted that, similar to the first embodiment, Figure 15 and Figure 16 The example explained is that the duty cycle of the control signal CLK is set to 50:50, and the time until the threshold is exceeded by the recharge operation is the same as the low period of CLK.

[0224] Therefore, although an explanation is provided for shifting the phase by half a phase between even-numbered and odd-numbered exposure periods, the amount of phase shift and the period in which the shift occurs can be arbitrarily changed depending on the ratio between the period of the control signal CLK and the recharge period.

[0225] For example, when the ratio between the period of the control signal CLK and the recharge period is 25:75, the image of the entire region can be uniformly acquired by shifting the phase by 1 / 4 at the 4nth, 4n+1th, 4n+2th and 4n+3rd times.

[0226] Furthermore, although the phase is changed for each exposure operation under the current conditions, the frequency of change is not necessarily specific to each exposure operation. The phase can be changed at intervals of multiple exposure operations, or the phase can be changed between the first and second halves of the total number of exposure operations.

[0227] Furthermore, although this embodiment explains an example of shifting half a phase by changing the timing of light emission based on the control signal CLK, the timing of light emission can remain constant, and the control signal CLK can be changed for each light emission.

[0228] Furthermore, the relative timing of pulsed emission relative to a fixed clock signal can be offset by a predetermined phase for each exposure operation. Additionally, the relative timing of the clock signal and pulsed emission can be offset by a predetermined phase for each exposure operation, or at intervals of a predetermined number of exposure operations, or between exposure operations in the first half and the second half of a frame period.

[0229] Furthermore, based on the ratio between the period of the clock signal and the time period of the recharge state, multiple pulse emission timings can be performed with a predetermined phase, and subsequent multiple pulse emission timings can be periodically offset with multiple phases corresponding to the duty cycle of the clock signal.

[0230] Furthermore, the timing of the pulsed light emission can be kept constant, and the duty cycle of the clock signal can be controlled to change according to the timing of the pulsed light emission.

[0231] While this disclosure has been described with reference to embodiments, it should be understood that this disclosure is not limited to the disclosed embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

[0232] Furthermore, as part or all of the control according to the embodiments, a computer program implementing the functions of the above embodiments can be supplied to a camera device, etc., via a network or various storage media. Then, the computer (or CPU, MPU, etc.) of the camera device, etc., can be configured to read and execute the program. In this case, the program and the storage medium storing the program are configured according to this disclosure.

[0233] Furthermore, this disclosure includes those implemented using at least one processor or circuitry configured to perform the functions of the embodiments described above. For example, multiple processors may be used to distribute processing to perform the functions of the embodiments explained above.

[0234] This application claims the benefit of Japanese Patent Application 2024-153665, filed on September 6, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. An image pickup apparatus comprising: a photoelectric conversion element having a plurality of pixels, wherein the pixels each include a sensor unit including an avalanche photodiode configured to generate a pulse in response to a photon incident on the avalanche photodiode, a counter configured to count a number of the pulses, a memory configured to store a count value of the counter, and a switch configured to switch the avalanche photodiode between a standby state in which avalanche multiplication is likely to occur and a recharging state; a signal generation unit configured to supply a clock signal to the switch; a light emission unit configured to perform pulsed light emission for illuminating a subject in synchronization with the clock signal; and a control unit configured to perform a plurality of exposure operations with the counter for capturing an image of a subject present within a predetermined image pickup distance range according to a timing of the pulsed light emission and the predetermined image pickup distance range, and configured to shift a relative timing of the clock signal and the pulsed light emission by a predetermined phase for each predetermined exposure operation.

2. The image pickup apparatus according to claim 1, wherein The predetermined phase is set based on a ratio between a period of the clock signal and a period of the recharging state of the avalanche photodiode.

3. The apparatus according to claim 1, wherein The pixels include: first and second photoelectric conversion units configured to receive light from mutually different pupils, respectively; first and second counters capable of counting outputs of the first and second photoelectric conversion units, respectively; and a pixel switch configured to selectively connect the outputs of the first and second photoelectric conversion units to the first and second counters, respectively.

4. The apparatus according to claim 3, wherein The control unit is configured to shift phases of the clock signal supplied to the avalanche photodiodes of the first and second photoelectric conversion units.

5. The image pickup apparatus according to claim 3, comprising a first switching state in which the pixel switch connects one of the outputs of the first and second photoelectric conversion units to the first counter and the other output to the second counter.

6. The apparatus according to claim 5, wherein The control unit is configured to calculate a subject distance based on the outputs of the first and second counters in the first switching state.

7. The image pickup apparatus according to claim 3, comprising a second switching state in which the outputs of the first and second photoelectric conversion units are both connected to the first counter by the pixel switch.

8. The apparatus according to claim 7, wherein The control unit is configured to generate a display image based on the output of the first counter in the second switching state.

9. The apparatus according to claim 1, wherein The control unit is configured to shift the relative timing of the clock signal and the pulsed light by a predetermined phase for each of the exposure operations.

10. The apparatus according to claim 1, wherein The control unit is configured to shift the relative timing of the clock signal and the pulsed light by a predetermined phase for each of the exposure operations, or at intervals of a predetermined number of exposure operations, or between the first-half exposure operations and the second-half exposure operations.

11. The apparatus according to claim 1, wherein The control unit is configured to perform predetermined multiple pulsed light timings at predetermined phases based on a ratio between a period of the clock signal and a period of the recharging state, and is configured to periodically shift subsequent multiple pulsed light timings by a number of phases corresponding to a duty ratio of the clock signal.

12. The apparatus according to claim 1, wherein The control unit is configured to keep the timing of the pulsed light constant, and to change the duty ratio of the clock signal in accordance with the timing of the pulsed light.

13. An image capturing method using an image capturing apparatus including a plurality of pixels, wherein The pixels each include a sensor unit including an avalanche photodiode configured to generate a pulse in response to a photon incident on the avalanche photodiode, a counter configured to count a number of the pulses, a memory configured to store a count value of the counter, and a switch configured to switch the avalanche photodiode between a standby state in which avalanche multiplication is likely to occur and a recharging state, wherein the imaging device includes a signal generation unit configured to supply a clock signal to the switch, and a light emission unit configured to perform pulsed light for illuminating an object in synchronization with the clock signal, and wherein the imaging method includes performing multiple exposure operations with the counter for photographing an image of an object present within a predetermined imaging distance range in accordance with a timing of the pulsed light and the predetermined imaging distance range, and shifting the relative timing of the clock signal and the pulsed light by a predetermined phase for each of the exposure operations.

14. A computer program product including a program for causing a computer to execute the method according to claim 13.

15. A computer-readable storage medium storing a program for causing a computer to execute the method according to claim 13.

Images