Distance image creation device and apparatus

Abstract

To provide a technique advantageous for achieving both a frame rate and distance measurement accuracy.SOLUTION: A distance image creation device comprises a plurality of pixels, and an image creation unit for creating a distance image in each of a plurality of distance measurement frame periods. In each distance measurement frame period, a plurality of sub-frames are acquired by a plurality of detection operations different from each other in the time from light emission timing in a light source until an exposure period in which each pixel detects light. The distance image creation device further comprises an event detection unit that, for each of the plurality of pixels, detects a change in a signal value between at least two distance measurement frame periods. In each distance measurement frame period, for the pixel for which the event detection unit detects the change, the image creation unit acquires the pixel value of the distance image on the basis of the plurality of sub-frames. For the pixel for which the event detection unit does not detect the change, the image creation unit does not acquire the pixel value of the distance image, or when a set condition is satisfied, acquires the pixel value of the distance image on the basis of the plurality of sub-frames.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to a distance image generating device and apparatus. [Background technology]

[0002] A Time of Flight (ToF) distance measuring device is known, which measures distance by emitting light from a light source and detecting the light reflected by an object. Patent Document 1 describes distance measurement using a time gate method using a Single Photon Avalanche Diode (SPAD) sensor. In the time gate method, a laser beam is repeatedly emitted toward an object at a predetermined frequency, a specific exposure period associated with the timing of the laser beam emission is set in the SPAD sensor, and photons are detected during this exposure period. By shifting the waiting time between the emission of the laser beam and the exposure period, the distance to the object is obtained from the waiting time until the exposure period when photons are detected. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2017 / 0052065 Summary of the Invention [Problem to be solved by the invention]

[0004] In the time gate method, in order to improve the distance measurement accuracy, it is conceivable to detect photons a predetermined number of times within the same waiting time, but increasing the number of detections within the same waiting time slows down the frame rate.

[0005] An object of the present invention is to provide a technique that is advantageous in achieving both a high frame rate and high distance measurement accuracy. [Means for solving the problem]

[0006] In view of the above problems, a distance image generating device according to an embodiment of the present invention is a distance image generating device comprising a plurality of pixels each having a photoelectric conversion element, and an image generation unit for generating a distance image based on output from the plurality of pixels in each of a plurality of ranging frame periods, wherein in each ranging frame period, a plurality of subframes are acquired by a plurality of detection operations having different times from the light emission timing of a light source to the exposure period in which light is detected in the photoelectric conversion element, and further comprising an event detection unit for detecting a change in signal value for each of the plurality of pixels between at least two of the plurality of ranging frame periods, wherein in each ranging frame period, the image generation unit acquires pixel values ​​of the distance image based on the plurality of subframes for pixels in which the event detection unit detects a change, and in each ranging frame period, the image generation unit does not acquire pixel values ​​of the distance image for pixels in which the event detection unit does not detect a change, or acquires pixel values ​​of the distance image based on the plurality of subframes when a set condition is satisfied. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a technique that is advantageous in achieving both a high frame rate and high distance measurement accuracy. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a hardware block diagram showing an example of the configuration of a distance image generating apparatus according to an embodiment of the present invention. [Figure 2] 2 is a diagram showing an example of the configuration of a photoelectric conversion device of the distance image generating device of FIG. 1; [Figure 3] 3 is a diagram showing an example of the configuration of a sensor substrate of the photoelectric conversion device of FIG. 2; [Figure 4] 3 is a diagram showing an example of the configuration of a circuit board of the photoelectric conversion device of FIG. 2. [Figure 5] 3 is a block diagram showing an example of the configuration of one pixel of the photoelectric conversion device of FIG. 2. [Figure 6] 3 is a diagram for explaining an example of the operation of the avalanche photodiode of the photoelectric conversion device of FIG. 2. [Figure 7] FIG. 2 is a functional block diagram showing an example of the configuration of the distance image generating device of FIG. 1. [Figure 8] FIG. 2 is a diagram showing the drive timing of the distance image generating device of FIG. 1. [Figure 9] 2A and 2B are diagrams for explaining distance measurement frames, subframes, and microframes of the distance image generating device of FIG. 1. [Figure 10] 3 is a flowchart showing the operation of the distance image generating device of FIG. 1 in generating a distance image. [Figure 11] 3 is a flowchart showing the operation of the distance image generating device of FIG. 1 in generating a distance image. [Figure 12] 3 is a diagram showing an example of the configuration of a circuit board of the photoelectric conversion device of FIG. 2. [Figure 13] 3 is a block diagram showing an example of the configuration of one pixel of the photoelectric conversion device of FIG. 2. [Figure 14] FIG. 2 is a functional block diagram showing an example of the configuration of the distance image generating device of FIG. 1. [Figure 15] 2A and 2B are diagrams for explaining distance measurement frames, subframes, and microframes of the distance image generating device of FIG. 1. [Figure 16] 2A and 2B are diagrams for explaining distance measurement frames, subframes, and microframes of the distance image generating device of FIG. 1. [Figure 17] FIG. 1 is a diagram showing an example of the configuration of a device incorporating a distance image generating device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] A distance image generating device according to an embodiment of the present disclosure will be described with reference to Figures 1 to 16. Figure 1 is a hardware block diagram that schematically shows an example configuration of a distance image generating device 30 according to this embodiment. The distance image generating device 30 includes a light emitting device 31, a light receiving device 32, and a signal processing circuit 33. The configuration of the distance image generating device 30 shown in this embodiment is an example, and is not limited to the configuration shown in the figures.

[0011] The distance image generating device 30 is a device that measures the distance from the distance image generating device 30 to the object X being measured using technology such as Light Detection and Ranging (LiDAR). The distance image generating device 30 measures the distance from the distance image generating device 30 to the object X based on the time difference between when a light emitting device 31 emits light and when the emitted light is reflected by the object X and received by a light receiving device 32. The distance image generating device 30 can also measure the distance to multiple points in a two-dimensional manner by emitting laser light within a predetermined distance range that includes the object X and receiving the reflected light using a pixel array. This allows the distance image generating device 30 to generate and output a distance image.

[0012] The light received by the light receiving device 32 includes ambient light such as sunlight in addition to the reflected light from the object X. Therefore, the distance image generating device 30 measures the incident light in each of a plurality of periods (bin periods) and determines that the reflected light is incident in the bin period when the amount of light is at its peak, thereby performing distance measurement that suppresses the influence of ambient light.

[0013] The light emitting device 31 functions as a light source that emits light such as laser light to the outside of the distance image generating device 30. The signal processing circuit 33 may include a processor that performs arithmetic processing of digital signals, a memory that stores digital signals, etc. The memory may be, for example, a semiconductor memory such as an SRAM or a DRAM.

[0014] The light receiving device 32 generates a pulse signal including a pulse based on the incident light. The light receiving device 32 is, for example, a photoelectric conversion device including an avalanche photodiode (APD) as a photoelectric conversion element. In this case, when one photon is incident on the APD and an electric charge is generated, avalanche multiplication occurs, and one pulse is generated according to the avalanche current. However, the light receiving device 32 may also be a device using a photoelectric conversion element that uses another photodiode, such as a PN diode or a PIN diode.

[0015] In this embodiment, the light receiving device 32 includes a pixel array in which a plurality of pixels, each including a photoelectric conversion element, are arranged in an array of a plurality of rows and a plurality of columns. Here, a photoelectric conversion device, which is a specific configuration example of the light receiving device 32, will be described with reference to FIGS. 2 to 6(a) and 6(b). The configuration example of the photoelectric conversion device described below is just one example. Photoelectric conversion devices applicable to the light receiving device 32 are not limited to the devices described below, and may be any device that can realize the functions described below with reference to FIGS. 7 to 16.

[0016] FIG. 2 is a schematic diagram showing the overall configuration of a photoelectric conversion device 100 incorporated into the light-receiving device 32 of this embodiment. The photoelectric conversion device 100 includes a sensor substrate 11 and a circuit substrate 21 stacked on top of each other. The sensor substrate 11 and the circuit substrate 21 are electrically connected to each other. The sensor substrate 11 has a pixel region 12 including a plurality of pixels 101 arranged in a plurality of rows and a plurality of columns. The circuit substrate 21 has a circuit region 22 including a plurality of pixel signal processing units 103 arranged in a plurality of rows and a plurality of columns, and a circuit region 23 arranged on the periphery of the circuit region 22. The circuit region 23 may include circuits for controlling the plurality of pixel signal processing units 103. The sensor substrate 11 has a light incident surface that receives incident light and a connection surface opposite the light incident surface. The sensor substrate 11 is connected to the circuit substrate 21 on the connection surface side. In other words, the photoelectric conversion device 100 is a so-called back-illuminated photoelectric conversion device.

[0017] In the following description, the sensor substrate 11 and the circuit board 21 are described as being diced chips, but the sensor substrate 11 and the circuit board 21 are not limited to being chips. For example, the sensor substrate 11 and the circuit board 21 may be in the form of a wafer. Furthermore, when the sensor substrate 11 and the circuit board 21 are diced chips, the photoelectric conversion device 100 may be manufactured by stacking them in the wafer state and then dicing them, or by stacking them after dicing.

[0018] 3 is a schematic block diagram showing an example of the arrangement of the sensor substrate 11. The pixel region 12 includes a plurality of pixels 101 arranged to form a plurality of rows and a plurality of columns. Each of the plurality of pixels 101 includes a photoelectric conversion unit 102 in the sensor substrate 11, the photoelectric conversion element of which includes an APD 201.

[0019] The conductivity type of the charge pairs generated in the APD and used as signal charges is referred to as the first conductivity type. The first conductivity type refers to a conductivity type in which charges of the same polarity as the signal charges serve as majority carriers. The conductivity type opposite to the first conductivity type, i.e., a conductivity type in which charges of a different polarity than the signal charges serve as majority carriers, is referred to as the second conductivity type. In the following description, the anode of the APD is at a fixed potential, and a signal is output from the cathode of the APD. Therefore, the first conductivity type semiconductor region is an N-type semiconductor region, and the second conductivity type semiconductor region is a P-type semiconductor region. However, this is not a limitation. The cathode of the APD may also be at a fixed potential, and a signal may also be output from the anode of the APD. In this case, the first conductivity type semiconductor region is a P-type semiconductor region, and the second conductivity type semiconductor region is an N-type semiconductor region. In the following description, a case in which one node of the APD is at a fixed potential is described, but the potentials of both nodes may fluctuate.

[0020] 4 is a schematic block diagram showing an example configuration of the circuit board 21. The circuit board 21 has a circuit area 22 that includes a plurality of pixel signal processing units 103 arranged to form a plurality of rows and a plurality of columns.

[0021] The circuit board 21 also has a vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, pixel output signal lines 113, an output circuit 114, and a control signal generation unit 115 arranged thereon. Additionally, the circuit board 21 also has a vertical scanning circuit 116, a horizontal scanning circuit 117, a readout circuit 118, pixel output signal lines 119, an output circuit 120, and a control signal generation unit 121 arranged thereon. The multiple photoelectric conversion units 102 shown in FIG. 3 and the multiple pixel signal processing units 103 shown in FIG. 4 are electrically connected to each other via connection wiring provided for each pixel 101. In this embodiment, the pixel signal processing unit 103 is configured to be able to output two types of signals with different exposure periods from the pixels 101.

[0022] The control signal generation unit 115 is a control circuit that generates control signals for driving the vertical scanning circuit 110, the horizontal scanning circuit 111, and the readout circuit 112, respectively, and supplies the control signals to each of these components. In this way, the control signal generation unit 115 controls the drive timing and the like of each of these components. In addition, the control signal generation unit 121 is a control circuit that generates control signals for driving the vertical scanning circuit 116, the horizontal scanning circuit 117, and the readout circuit 118, and supplies the control signals to each of these components. In this way, the control signal generation unit 121 controls the drive timing and the like of each of these components.

[0023] The vertical scanning circuit 110 supplies a control signal to each of the plurality of pixel signal processing units 103 based on a control signal supplied from the control signal generation unit 115. The vertical scanning circuit 110 supplies a control signal to each pixel signal processing unit 103 for each row via a drive line provided for each row of the circuit area 22. As will be described later, a plurality of drive lines may be provided for each row. The vertical scanning circuit 110 may include logic circuits such as a shift register and an address decoder. In this way, the vertical scanning circuit 110 selects a row from which the pixel signal processing unit 103 will output a signal.

[0024] The vertical scanning circuit 116 supplies a control signal to each of the plurality of pixel signal processing units 103 based on the control signal supplied from the control signal generation unit 121. The vertical scanning circuit 116 supplies a control signal to each pixel signal processing unit 103 for each row via a drive line provided for each row of the circuit area 22. As will be described later, a plurality of drive lines may be provided for each row. The vertical scanning circuit 116 may include logic circuits such as a shift register and an address decoder. In this way, the vertical scanning circuit 116 selects a row from which the pixel signal processing unit 103 will output a signal.

[0025] The signal output from the photoelectric conversion unit 102 of the pixel 101 is processed by the pixel signal processing unit 103. The pixel signal processing unit 103 counts the number of pulses output from the APD included in the photoelectric conversion unit 102 to acquire and hold a digital signal having multiple bits.

[0026] The horizontal scanning circuit 111 supplies a control signal to the readout circuit 112 based on a control signal supplied from the control signal generation unit 115. The pixel signal processing units 103 are connected to the readout circuit 112 via pixel output signal lines 113 provided for each column of the circuit area 22. The pixel output signal line 113 for one column is shared by multiple pixel signal processing units 103 for the corresponding column. The pixel output signal line 113 includes multiple wirings. The pixel output signal line 113 has at least the function of outputting a digital signal from each pixel signal processing unit 103 to the readout circuit 112 and the function of supplying the pixel signal processing unit 103 with a control signal for selecting a column from which to output a signal. The readout circuit 112 outputs a signal to a storage unit or signal processing unit external to the photoelectric conversion device 100 via the output circuit 114 based on the control signal supplied from the control signal generation unit 115.

[0027] The horizontal scanning circuit 117 supplies a control signal to the readout circuit 118 based on a control signal supplied from the control signal generation unit 121. The pixel signal processing units 103 are connected to the readout circuit 118 via pixel output signal lines 119 provided for each column of the circuit area 22. The pixel output signal line 119 for one column is shared by multiple pixel signal processing units 103 for the corresponding column. The pixel output signal line 119 includes multiple wirings. The pixel output signal line 119 has at least the function of outputting a digital signal from each pixel signal processing unit 103 to the readout circuit 118 and the function of supplying the pixel signal processing unit 103 with a control signal for selecting a column from which to output a signal. The readout circuit 118 outputs a signal to a storage unit or signal processing unit external to the photoelectric conversion device 100 via the output circuit 120 based on the control signal supplied from the control signal generation unit 121.

[0028] 2 to 4, the photoelectric conversion units 102 are arranged in a two-dimensional array in the pixel region 12, but this is not limiting. The photoelectric conversion units 102 may be arranged one-dimensionally in the pixel region 12. Furthermore, it is not necessary to provide one pixel signal processing unit 103 for each pixel 101. For example, one pixel signal processing unit 103 may be shared by multiple pixels 101. In this case, the pixel signal processing unit 103 provides a signal processing function to each pixel 101 by sequentially processing signals output from each photoelectric conversion unit 102.

[0029] 3 and 4 , a circuit region 22 in which a plurality of pixel signal processing units 103 are arranged is arranged in a region overlapping the pixel region 12 in a planar view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, a control signal generating unit 115, and the like are arranged so as to overlap between an edge of the sensor substrate 11 and an edge of the pixel region 12 in a planar view. Similarly, a vertical scanning circuit 116, a horizontal scanning circuit 117, a readout circuit 118, an output circuit 120, and a control signal generating unit 121 are also arranged so as to overlap between an edge of the sensor substrate 11 and an edge of the pixel region 12. In other words, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. In the circuit board 21, a circuit area 23 is arranged in an area that overlaps the non-pixel area in a planar view, in which a vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, a control signal generating unit 115, a vertical scanning circuit 116, a horizontal scanning circuit 117, a readout circuit 118, an output circuit 120, and a control signal generating unit 121 are arranged.

[0030] The arrangement of the pixel output signal lines 113, the arrangement of the readout circuits 112, and the arrangement of the output circuits 114 are not limited to those shown in Fig. 4. For example, the pixel output signal lines 113 may be arranged to extend in the row direction and be shared by multiple pixel signal processing units 103 in the corresponding row. The readout circuits 112 may then be arranged so that the pixel output signal lines 113 in each row are connected to them.

[0031] Similarly, the arrangement of the pixel output signal lines 119, the arrangement of the readout circuits 118, and the arrangement of the output circuits 120 are not limited to those shown in Fig. 4. For example, the pixel output signal lines 119 may be arranged to extend in the row direction and be shared by multiple pixel signal processing units 103 in the corresponding row. The readout circuits 118 may then be arranged so that the pixel output signal lines 119 in each row are connected.

[0032] 5 is a schematic block diagram showing an example of the configuration of one pixel of the photoelectric conversion unit 102 and pixel signal processing unit 103 according to this embodiment. FIG. 5 schematically shows a more specific example of the configuration than that described above, including the connection relationship between the photoelectric conversion unit 102 arranged on the sensor substrate 11 and the pixel signal processing unit 103 arranged on the circuit board 21. In FIG. 5, the drive lines between the vertical scanning circuit 110 and the pixel signal processing unit 103 in FIG. 4 are shown as drive lines 213, 214, and 215. Similarly, the drive lines between the vertical scanning circuit 116 and the pixel signal processing unit 103 are shown as drive lines 219, 220, and 221.

[0033] The photoelectric conversion unit 102 includes an APD 201. The pixel signal processing unit 103 includes a quenching element 202, a waveform shaping unit 210, a counter circuit 211, a selection circuit 212, and a gating circuit 216. The pixel signal processing unit 103 further includes a counter circuit 217, a selection circuit 218, and a gating circuit 222.

[0034] The APD 201 generates charges according to incident light through photoelectric conversion. A potential VL is supplied to the anode of the APD 201. The cathode of the APD 201 is connected to one main terminal of the quench element 202 and an input terminal of the waveform shaping unit 210. A potential VH higher than the potential VL supplied to the anode is supplied to the cathode of the APD 201 via the quench element 202. As a result, a reverse bias voltage is supplied between the anode and cathode of the APD 201 such that the APD 201 performs avalanche multiplication. When charges are generated by incident light in the APD 201 to which the reverse bias voltage is supplied, these charges undergo avalanche multiplication, generating an avalanche current.

[0035] There are two operating modes when a reverse bias voltage is supplied to the APD 201: Geiger mode and linear mode. The Geiger mode is a mode in which the APD 201 operates at a potential difference (voltage) between the anode and cathode that is greater than the breakdown voltage of the APD 201. The linear mode is a mode in which the APD 201 operates at a potential difference between the anode and cathode that is close to or less than the breakdown voltage of the APD 201.

[0036] The APD 201 operated in Geiger mode is called a Single Photon Avalanche Diode (SPAD). In this case, for example, the potential VL may be about −30 V, and the potential VH may be about 1 V. The APD 201 may be operated in either linear mode or Geiger mode. In the case of a SPAD, the potential difference is larger than in a linear mode APD, and the effect of avalanche multiplication becomes more pronounced. Therefore, the APD 201 may be operated as a SPAD.

[0037] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication. The quench element 202 suppresses the voltage supplied to the APD 201 and suppresses avalanche multiplication (quench operation). The quench element 202 also returns the voltage supplied to the APD 201 to the potential VH by passing a current corresponding to the voltage drop caused by the quench operation (recharge operation). The quench element 202 may be, for example, a resistive element.

[0038] The waveform shaping unit 210 shapes the potential change of the cathode of the APD 201 obtained when photons are detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. FIG. 5 shows an example in which one inverter is used as the waveform shaping unit 210. However, this is not limiting. The waveform shaping unit 210 may be a circuit in which multiple inverters are connected in series, or may be any other circuit that has a waveform shaping effect.

[0039] The gating circuit 216 and the gating circuit 222 are circuits that perform gating, allowing the pulse signal output from the waveform shaping unit 210 to pass for a predetermined period. The input terminals of the gating circuit 216 and the gating circuit 222 are connected in parallel to the output terminal of the APD 201. During the period during which the pulse signal can pass through the gating circuit 216, photons incident on the APD 201 are counted by the counter circuit 211 in the subsequent stage. Similarly, during the period during which the pulse signal can pass through the gating circuit 222, photons incident on the APD 201 are counted by the counter circuit 217 in the subsequent stage. Therefore, the gating circuit 216 and the gating circuit 222 control the period during which a signal is generated in the pixel 101 based on incident light. The period during which the pulse signal can pass through the gating circuit 216 is controlled by a control signal supplied from the vertical scanning circuit 110 via the drive line 215. Similarly, the period during which the pulse signal is allowed to pass through the gating circuit 222 is controlled by a control signal supplied from the vertical scanning circuit 116 via a drive line 221 .

[0040] 5 shows an example in which one AND circuit is used as each of the gating circuit 216 and the gating circuit 222. The AND circuit constituting the gating circuit 216 has two input terminals, one of which receives a pulse signal from the waveform shaping unit 210 and the other of which receives a control signal supplied from the vertical scanning circuit 110 via a drive line 215. The AND circuit serving as the gating circuit 216 outputs a logical product of these signals to the counter circuit 211. The AND circuit constituting the gating circuit 222 has two input terminals, one of which receives a pulse signal from the waveform shaping unit 210 and the other of which receives a control signal supplied from the vertical scanning circuit 116 via a drive line 221. The AND circuit serving as the gating circuit 222 outputs a logical product of these signals to the counter circuit 217. Here, the gating circuits 216 and 222 only need to have the function of achieving the gating described above, and circuit configurations other than AND circuits may be used.

[0041] The counter circuit 211 counts the pulse signals output from the waveform shaping unit 210 via the gating circuit 216 and holds a digital signal indicating the count value. When a control signal is supplied from the vertical scanning circuit 110 via the drive line 213, the counter circuit 211 resets the signal it holds.

[0042] The counter circuit 217 counts the pulse signals output from the waveform shaping unit 210 via the gating circuit 222 and holds a digital signal indicating the count value. When a control signal is supplied from the vertical scanning circuit 116 via a drive line 219, the counter circuit 217 resets the signal it holds.

[0043] A control signal is supplied to the selection circuit 212 from the vertical scanning circuit 110 shown in Fig. 4 via a drive line 214 shown in Fig. 5. In response to this control signal, the selection circuit 212 switches between electrical connection and disconnection between the counter circuit 211 and the pixel output signal line 113. The selection circuit 212 includes, for example, a buffer circuit for outputting a signal corresponding to the value held in the counter circuit 211.

[0044] A control signal is supplied to the selection circuit 218 from the vertical scanning circuit 116 shown in Fig. 4 via a drive line 220 shown in Fig. 5. In response to this control signal, the selection circuit 218 switches between electrical connection and disconnection between the counter circuit 217 and the pixel output signal line 119. The selection circuit 218 includes, for example, a buffer circuit for outputting a signal corresponding to the value held in the counter circuit 217.

[0045] In the configuration shown in FIG. 5 , the selection circuit 212 switches between electrical connection and disconnection between the counter circuit 211 and the pixel output signal line 113. Similarly, the selection circuit 218 switches between electrical connection and disconnection between the counter circuit 217 and the pixel output signal line 119. However, the method for controlling the signal output to the pixel output signal line 113 and the pixel output signal line 119 is not limited thereto. For example, a switch such as a transistor may be disposed at a node between the quenching element 202 and the APD 201, or between the photoelectric conversion unit 102 and the pixel signal processing unit 103. The signal output to the pixel output signal line 113 and the pixel output signal line 119 may be controlled by switching between electrical connection and disconnection of this switch. Alternatively, the signal output to the pixel output signal line 113 and the pixel output signal line 119 may be controlled by changing the value of the potential VH or potential VL supplied to the photoelectric conversion unit 102 using a switch such as a transistor.

[0046] Figures 6(a) and 6(b) are diagrams illustrating the operation of the APD 201 of this embodiment. Figure 6(a) shows the APD 201, quench element 202, and waveform shaping unit 210 from the configuration shown in Figure 5. As shown in Figure 6(a), the connection node between the APD 201, quench element 202, and input terminals of the waveform shaping unit 210 is referred to as node A. Also, as shown in Figure 6(a), the output side of the waveform shaping unit 210 is referred to as node B.

[0047] FIG. 6(b) shows the time changes in the potentials of node A and node B in FIG. 6(a). From time t0 to time t1, a voltage of potential VH-potential VL is applied to the APD 201 in FIG. 6(a). When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201. This causes an avalanche current to flow through the quench element 202, causing the potential of node A to drop. The potential drop then increases, and the voltage applied to the APD 201 gradually decreases. Finally, at time t2, avalanche multiplication in the APD 201 stops. As a result, the voltage level of node A no longer drops below a certain value. Then, from time t2 to time t3, a current flows through node A from the node at potential VH to compensate for the voltage drop, and at time t3, node A stabilizes to its original potential.

[0048] In the above process, the potential of node B becomes high during the period when the potential of node A is lower than a certain threshold. In this way, the waveform of the drop in the potential of node A caused by the incidence of a photon is shaped by waveform shaping section 210 and output as a pulse to node B.

[0049] Next, the overall configuration and operation of distance image generation device 30 will be described in more detail. Figure 7 is a functional block diagram showing an example of the schematic configuration of distance image generation device 30 of this embodiment. Figure 7 shows a more detailed configuration of light-emitting device 31, light-receiving device 32, and signal processing circuit 33 described in Figure 1.

[0050] The light emitting device 31 includes a pulsed light source 311 and a light source control unit 312. The pulsed light source 311 is a light source such as a semiconductor laser device that emits pulsed light throughout the entire distance measurement range. The pulsed light source 311 may also be a surface light source such as a surface emitting laser. The light source control unit 312 is a control circuit that controls the timing of light emission from the pulsed light source 311.

[0051] The light receiving device 32 includes an imaging section 321, a gate pulse generating section 322, a microframe readout section 323, a microframe adder 324, an addition count control section 325, and a subframe output section 326. The light receiving device 32 further includes a microframe readout section 327, a microframe adder 328, and an event detecting section 329. The imaging section 321 may use a photoelectric conversion device including a pixel region 12 in which pixel circuits including the above-mentioned APD 201 are arranged two-dimensionally. This allows the distance image generating device 30 to acquire a two-dimensional distance image.

[0052] The gate pulse generation unit 322 is a control circuit that outputs a control signal that controls the drive timing of the imaging unit 321. The gate pulse generation unit 322 synchronizes the pulsed light source 311 and the imaging unit 321 by transmitting and receiving control signals to and from the light source control unit 312. This enables imaging while controlling the time difference between when light is emitted from the pulsed light source 311 and when it is received by the imaging unit 321. In this embodiment, the gate pulse generation unit 322 drives the imaging unit 321 using global gate driving. Global gate driving is a driving method in which incident light is simultaneously detected in all pixels 101 arranged in the imaging unit 321 during the same exposure period, based on the emission time of pulsed light from the pulsed light source 311. In this embodiment, global gate driving sequentially shifts the timing from light emission to collective exposure, and repeatedly detects incident light. As a result, each pixel 101 arranged in the imaging unit 321 simultaneously generates a 1-bit signal indicating the presence or absence of incident photons during each of multiple exposure periods. The generated 1-bit signal may be held in counter circuit 211 and counter circuit 217. In this case, counter circuit 211 and counter circuit 217 are configured as 1-bit counters (memory circuits). Therefore, counter circuit 211 and counter circuit 217 may each be referred to as a memory circuit.

[0053] This global gate driving is achieved by inputting a high-level signal to the input terminals of the gating circuits 216 and 222 of each pixel 101 during the gating period based on a control signal supplied from the gate pulse generation unit 322. Hereinafter, this embodiment will be described assuming that the counter circuit 211 holds a signal for acquiring data for generating a distance image, and the counter circuit 217 holds a signal for acquiring data for event detection. Therefore, the gate pulse generation unit 322 of this embodiment generates gate pulses indicating different exposure periods for the gating circuits 216 and 222. A specific example will be described with reference to FIG. 8. FIG. 8 is a drive timing diagram showing the timing of gate pulses in this embodiment.

[0054] "Light emission" in Fig. 8 indicates the light emission timing of the pulsed light source 311. As shown in Fig. 8, the pulsed light source 311 emits light at a constant cycle under the control of the light source control unit 312. "P_G1" and "P_G2" indicate the input timing of multiple types of gate pulses input from the gate pulse generation unit 322 to the imaging unit 321. "P_G1" corresponds to the control signal input to the gating circuit 216 via the drive line 215 in Fig. 5. "P_G2" corresponds to the control signal input to the gating circuit 222 via the drive line 221 in Fig. 5.

[0055] A gate pulse G01 is input to the gating circuit 216 connected to the counter circuit 211 at a timing synchronized with the light emission timing L01. A gate pulse G02 is input to the gating circuit 222 connected to the counter circuit 217 for a predetermined exposure period during which imaging is performed. By varying the timing of the gate pulse G01 and the gate pulse G02 in this way, distance information is obtained in the counter circuit 211 and luminance information for event detection is obtained in the counter circuit 217 for each pixel 101 arranged in the imaging unit 321.

[0056] The microframe readout unit 323, microframe adder 324, addition count control unit 325, microframe readout unit 327, and microframe adder 328 function as a readout unit that reads out 1-bit signals that make up microframes from the imaging unit 321. Furthermore, these circuits are signal processing circuits that perform predetermined signal processing on the readout 1-bit signals. Details of the operation of each of these components will be described later with reference to the flowchart in FIG. 10.

[0057] The subframe output unit 326 is an interface that outputs a signal in accordance with a predetermined standard from the light receiving device 32 to the signal processing circuit 33. The subframe output unit 326 transmits a signal from the memory in the light receiving device 32 to the memory in the signal processing circuit 33.

[0058] The event detection unit 329 detects a change in signal value between at least two ranging frame periods among a plurality of ranging frame periods for each of the plurality of pixels 101. More specifically, the event detection unit 329 detects an event for each pixel 101 from signal values ​​obtained in at least two ranging frames acquired from the microframe adder 328, and outputs event information to the signal processing circuit 33. For example, for each pixel 101, if the difference in signal value between consecutive nth and (n+1)th frames exceeds a predetermined threshold, the event detection unit 329 outputs an event detection signal for each pixel 101. The event detection signal is, for example, a binary signal. Here, event detection does not necessarily have to be performed for two consecutive frames. For example, the event detection signal may be generated based on whether the difference in signal value between the nth and (n+2)th frames exceeds a predetermined threshold.

[0059] Signal processing circuit 33 includes a subframe adder 331 and an image generator 332. Signal processing circuit 33 can be a computer including a processor that operates as image generator 332 and a memory that operates as subframe adder 331. Image generator 332 is a distance image generator that generates a distance image based on outputs from multiple pixels 101 during each of multiple ranging frame periods. The operation of each of these components will also be described later with reference to the flowchart in FIG. 10.

[0060] Next, before describing the flow of generating a distance image according to this embodiment, the configuration of distance measurement frames, subframes, and microframes will be described with reference to Fig. 9. Fig. 9 schematically shows the acquisition periods of distance measurement frames corresponding to distance image data, subframes used to generate the distance measurement frames, and microframes used to generate the subframes, arranged in blocks horizontally. The horizontal direction in Fig. 9 indicates the passage of time, and each block represents the acquisition period of one distance measurement frame, subframe, or microframe.

[0061] The ranging frame F1 corresponds to data for one distance image. That is, the ranging frame F1 contains information corresponding to the distance to the object X for each of the plurality of pixels 101, obtained from the time difference between when light is emitted and when it is received. The information corresponding to the distance to the object X may be calculated by performing various arithmetic operations on the information corresponding to the distance to the object X for each of the plurality of pixels 101. In this embodiment, it is assumed that the distance image is acquired as a video, and one ranging frame F1 is repeatedly acquired every time one ranging frame period T1 elapses. As a result, a distance image is generated based on the output from the plurality of pixels 101 in each of the plurality of ranging frame periods T1.

[0062] One ranging frame F1 is generated from multiple sub-frames F2. One ranging frame period T1 includes multiple sub-frame periods T2. One sub-frame F2 is repeatedly acquired every time one sub-frame period T2 elapses. The sub-frame F2 is composed of a multi-bit signal corresponding to the amount of light incident during the sub-frame period T2.

[0063] One subframe F2 is generated from multiple microframes F3. One subframe period T2 includes multiple microframe periods T3. One microframe F3 is repeatedly acquired every time one microframe period T3 elapses. The microframe F3 is composed of a 1-bit signal indicating the presence or absence of incident light on the photoelectric conversion element (APD 201) during the microframe period T3. One subframe F2 with a multi-bit signal is generated by adding and combining multiple microframes F3 with a 1-bit signal. As a result, one subframe F2 can include a multi-bit signal corresponding to the number of microframes F3 in which incident light was detected during the subframe period T2.

[0064] Each subframe F2 may be generated as a multi-bit signal by adding and combining multiple frames across multiple ranging frames F1. Hereinafter, the operation of generating one subframe F2 may be referred to as a detection operation. In each ranging frame period T1, multiple subframes F2 are acquired by multiple detection operations with different times from the light emission timing of the pulsed light source 311, which is the light source, to the exposure period in which light is detected by the photoelectric conversion element (APD 201). Details will be described later with reference to the flowchart in FIG. 10.

[0065] In this way, multiple subframes F2 are acquired, each with a different period during which incident light is detected from the light emission timing of the pulsed light source 311. The signal acquisition time during the period during which incident light is detected from the light emission timing of the pulsed light source 311 can be associated with the distance from the distance image generating device 30 to the object to be measured. The signal acquisition time at which the signal value is maximized can be determined from the distribution of the signal acquisition times and the signal values ​​of the multiple subframes F2. Since it is estimated that reflected light is incident on the imaging unit 321 at the time when the signal value is maximized, the distance can be obtained by converting the signal acquisition time at which the signal value is maximized into the distance to the object X. Furthermore, a distance image can be generated by acquiring the distance for each pixel 101 and obtaining a two-dimensional distribution of the distances.

[0066] 9, the length of the ranging frame period T1 required to acquire one ranging frame F1 depends on the length of the subframe period T2 required to acquire one subframe F2, and the length of the subframe period T2 depends on the number of microframes F3. Since the noise reduction effect increases with the number of microframes F3, there is a trade-off between ranging accuracy and frame rate with respect to the number of microframes.

[0067] 10 is a flowchart showing the operation of distance image generation by the distance image generating device 30. The operation of this embodiment will be described with reference to the flowchart of FIG.

[0068] In the flowchart shown in Fig. 10, one cycle of processing in the loop from S11 to S13 represents processing performed in a microframe period T3 for acquiring one microframe F3 in Fig. 9. One cycle of processing in the loop from S11 to S16 represents processing performed in a subframe period T2 for acquiring one subframe F2 in Fig. 9. One cycle of processing in the loop from S11 to S19 and the loop from S11 to S21 represents processing performed in a ranging frame period T1 for acquiring one ranging frame F1 in Fig. 8.

[0069] In S11, the light source control unit 312 controls the pulsed light source 311 to emit pulsed light within a predetermined distance measurement range. Synchronously with this, the gate pulse generation unit 322 controls the imaging unit 321 to start detecting incident light by global gate driving. Here, the gate pulse generation unit 322 generates gate pulses indicating different detection periods for the gating circuit 216 and the gating circuit 222, as described with reference to FIG. 8 . The gate pulse generation unit 322 supplies gate pulses to the gating circuit 216 so that multiple detection operations are performed with different times (signal acquisition times) from the light emission timing of the pulsed light source 311 to the exposure period during which light is detected in the photoelectric conversion element (APD 201). The gate pulse generation unit 322 also supplies gate pulses to the gating circuit 222 so that an event detection operation is performed in which a signal is acquired from the photoelectric conversion element (APD 201) during an exposure period whose length is different from the exposure period of the detection operation. 8, the exposure period of the event detection operation (gate pulse G02) may be longer than the exposure period of the detection operation (gate pulse G01). The event detection unit 329 detects a change in the signal value of each pixel 101 based on the signal obtained by the event detection operation.

[0070] Next, in S12, the microframe readout unit 323 and the microframe readout unit 327 each read a microframe F3 from the imaging unit 321 every time a microframe period T3 elapses. The microframe readout unit 323 reads a 1-bit signal constituting the microframe F3 stored in the counter circuit 211. The microframe readout unit 327 reads a 1-bit signal constituting the microframe F3 stored in the counter circuit 217. The read microframes are stored in the memories of the microframe adder 324 and the microframe adder 328. This memory has a storage capacity sufficient to store multiple bits of data for each pixel 101. Each time a microframe F3 is read out, the microframe adder 324 and the microframe adder 328 sequentially add the value of the microframe F3 to the value stored in the memory. In this way, the microframe adder 324 and the microframe adder 328 add multiple microframes F3 within the subframe period T2 to obtain a subframe F2. In this way, the microframe readout unit 323 and the microframe adder 324 function as readout units that read out signals of the detection operation for generating a distance image. Similarly, the microframe readout unit 327 and the microframe adder 328 function as readout units that read out signals of the event detection operation for detecting an event. In this way, the detection operation and the event detection operation may be performed in parallel.

[0071] The number of additions performed by the microframe adder 324 and the microframe adder 328 is controlled by the addition count control unit 325. For this control, the addition count control unit 325 holds information about a preset number of additions. The number of additions performed by the microframe adder 324 and the microframe adder 328 may be the same or different. In this way, the microframe readout unit 323 and the microframe readout unit 327 function as acquisition units that acquire microframes F3 composed of 1-bit signals based on light incident on the photoelectric conversion element (APD 201). The microframe adder 324 and the microframe adder 328 also function as synthesis units that synthesize multiple microframes F2 acquired during different periods. For example, if the number of additions is 16, a subframe F2 signal having 4-bit gradation can be generated by adding and synthesizing 16 microframes F3.

[0072] In S13, the microframe adder 324 and the microframe adder 328 determine whether or not a preset number of microframes F3 have been added. If the preset number of microframes F3 have not been added (NO in S13), the process proceeds to S11, where the next microframe F3 is read. If the preset number of microframes F3 have been added (YES in S13), the process proceeds to S14.

[0073] Next, in S14, the subframe output unit 326 reads the subframe F2 for which the addition has been completed from the memory of the microframe adder 324, and outputs it to the subframe adder 331. The subframe adder 331 stores the subframe F2 output from the subframe output unit 326 in its memory. The subframe adder 331 is configured to be able to store the multiple subframes F2 used to generate one ranging frame F1 individually for each pixel 101 and for each subframe period T2.

[0074] In S15, which may be performed in parallel with S14, the event detection unit 329 reads the subframe F2 for which addition has been completed from the memory of the microframe adder 328 and stores the subframe F2 in the memory of the event detection unit 329. As described above, the event detection unit 329 outputs event information to the signal processing circuit 33 by comparing the signal values ​​of the pixel 101 during two or more ranging frame periods T1.

[0075] In S16, the signal processing circuit 33 determines whether the subframe adder 331 has completed acquisition of a predetermined number of subframes F2 (i.e., the number of ranging points). If acquisition of subframes F2 for the number of ranging points has not been completed (NO in S16), the process proceeds to S11, where multiple microframes F3 are acquired and added again to read out the next subframe F2. At this time, the time from the light emission timing of the pulsed light source 311 to the start time of global gate driving for exposure is shifted (gate shift) with respect to the previous subframe period T2, and processing is performed to acquire each microframe F3. If acquisition of subframes F2 for the number of ranging points has been completed (YES in S16), the process proceeds to S17. Subframes F2 for the number of ranging points are acquired by looping from S11 to S16.

[0076] In S17, the signal processing circuit 33 determines whether or not an event has been detected for each pixel 101 based on the event information from the event detection unit 329. If an event has not been detected for that pixel (NO in S17), the process proceeds to S18. If an event has been detected for that pixel (YES in S17), the process proceeds to S20.

[0077] In S18, the subframe adder 331 sequentially adds the multiple subframes F2 acquired from the subframe output unit 326 to the value stored for each subframe period T2. In other words, the subframe adder 331 stores multiple subframes F2 for pixels 101 for which the event detection unit 329 does not detect any change during each ranging frame period T1. When the event detection unit 329 does not detect any change during consecutive ranging frame periods T1, the subframe adder 331 adds the multiple subframes F2 for each pixel 101 for consecutive ranging frame periods T1 stored for each pixel 101 for each subframe period T2. The maximum number of additions is preset. For example, if the number of additions is four, four subframes with 4-bit gradation (additive synthesis of 16 microframes F3) generated by the microframe adder 324 are added together. This allows for the generation of a frame signal with a maximum of 6-bit gradation (additive synthesis of 64 microframes). The addition of the subframes F2 may be performed by the light receiving device 32. In this case, the subframe adder 331 functions as a storage unit that stores the added data of the subframe F2 output from the subframe output unit 326 in a memory.

[0078] In S19, the signal processing circuit 33 determines whether a preset number of subframe-by-subframe additions have been completed. If the set number of subframe-by-subframe additions have not been completed (NO in S19), the process proceeds to S11, where multiple microframes F3 are acquired and added again to read the next ranging frame F1. In this case, the start time of global gate drive relative to the light emission timing is reset to the first subframe period T3, and the same process is performed. If the set number of subframe-by-subframe additions have been completed (YES in S19), the process proceeds to S20.

[0079] In S20, the image generation unit 332 acquires multiple subframes F2 in one ranging frame period T1 from the subframe adder 331. The image generation unit 332 generates a distance image showing a two-dimensional distribution of distances by acquiring the distance corresponding to the subframe in which the signal value is maximum for each pixel 101. At this time, the image generation unit 332 acquires pixel values ​​of the distance image corresponding to the distance based on the multiple subframes F2 for pixels 101 in which the event detection unit 329 detected an event, in other words, for pixels 101 in which a change in signal value was detected during the ranging frame period T1. In this embodiment, the image generation unit 332 acquires pixel values ​​of the distance image based on the multiple subframes F2 of the ranging frame period T1 in which the event detection unit 329 detected an event.

[0080] Meanwhile, in S20, the image generation unit 332 does not acquire pixel values ​​of the distance image for pixels for which the event detection unit 329 does not detect an event, or acquires pixel values ​​of the distance image based on the multiple subframes if a set condition is met. More specifically, for pixels 101 for which no event is detected (NO in S17) and addition for each subframe F2 is not complete (NO in S19), pixel values ​​of the distance image are not acquired, and the pixel values ​​of the previous ranging frame period T1 are used as is for the distance image. Also, if the event detection unit 329 does not detect an event in a predetermined number of consecutive ranging frame periods T1 (NO in S17 and YES in S19), the image generation unit 332 acquires pixel values ​​of the distance image based on the data added by the subframe adder 331. In other words, pixel values ​​of the distance image are acquired for pixels 101 for which no event is detected (NO in S17) and addition for each subframe F2 is complete (YES in S19). Therefore, for example, when capturing a distance image as a video, the number of times pixel values ​​are acquired for each pixel 101 may be less than the number of times the distance measurement frame periods T1 are repeated. This is because if the event detection unit 329 does not detect an event and the set condition is not met (addition for each subframe F2 is not completed), the image generation unit 332 does not acquire pixel values ​​of the distance image. The image generation unit 332 then outputs the distance image data to a device external to the signal processing circuit 33. This distance image can be used, for example, to detect the environment surrounding the vehicle. The image generation unit 332 may also store the pixel values ​​of the distance image in a memory internal to the distance image generation device 30. The pixel values ​​of the distance image stored in the memory can be used, for example, as pixel values ​​of pixels for which distance image pixel values ​​were not acquired.

[0081] After generating the distance image, in S21, the image generation unit 332 resets the data held in the subframe adder 331 corresponding to the pixel 101 from which the pixel value of the distance image was acquired, as well as the number of additions. Then, the process proceeds to S11, where multiple microframes F3 are acquired and added again to read the next ranging frame F1. In this case, the start time of global gate driving relative to the light emission timing of the pulsed light source 311 is reset to the period of the first subframe F2, and similar processing is performed.

[0082] As described above, in this embodiment, for a pixel 101 in which an event is detected, pixel values ​​of a distance image indicating a distance corresponding to the time from the irradiation of light by the pulsed light source 311 to the detection of the light are acquired, and the pixel values ​​of the distance image are updated. On the other hand, for a pixel 101 in which no event is detected, pixel values ​​of a distance image are not acquired until a preset condition is met, more specifically, until a predetermined number of subframes F2 have been added, and subframes F2 are added. As a result, for an area in which a moving object, in which the amount of light incident on the photoelectric conversion element (APD 201) changes, is captured, pixel values ​​are updated every ranging frame period T1, thereby suppressing blur without reducing the frame rate. Furthermore, for an area in which a stationary object or background, in which the amount of light incident on the photoelectric conversion element (APD 201) does not change, pixel values ​​are generated using subframes F2 acquired over multiple ranging frame periods T1, thereby reducing noise and improving ranging accuracy. In this way, the distance image generating device 30 of this embodiment is able to achieve both a high frame rate and high ranging accuracy.

[0083] Here, the voltage (potential VH-potential VL) applied to the APD 201 may be configured to be controllable for each pixel 101 or for each pixel region formed by a plurality of pixels 101. In this case, power saving can be achieved by controlling the voltage for the pixel 101 or pixel region in which no event is detected.

[0084] Next, an example of the processing flow for generating a distance image different from that shown in Fig. 10 will be described with reference to Fig. 11. Descriptions of configurations and flows that may be similar to those of the above-described embodiment will be omitted or simplified as appropriate.

[0085] Fig. 11 is a flowchart showing the operation of distance image generation device 30. The operation of this embodiment will be described with reference to the flowchart in Fig. 11. In the flowchart shown in Fig. 11, the processing from S11 to S16 in the loop until sub-frames F2 for the number of distance measurement points are acquired is the same as the flow shown in Fig. 10. When acquisition of sub-frames F2 for the number of distance measurement points is completed in S16, the process proceeds to S111.

[0086] In S111, the subframe adder 331 adds up the most recent N subframes F2 acquired from the subframe output unit 326 for each pixel 101 and for each subframe period T2, and stores the sum. In other words, the subframe adder 331 adds up the multiple subframes F2 for each of the predetermined number (N) of ranging frame periods T1 stored for each pixel 101 for each subframe period T2 in each ranging frame period T1. N, which indicates the number of times subframes are added, is preset. For example, if N is added four times as described above, a frame signal with 6-bit gradation (additive synthesis of 64 microframes) can be generated. In this embodiment, the subframe adder 331 is configured to individually store N subframes F2 used to generate one ranging frame F1 for each subframe period T2.

[0087] In S17, similar to the flow shown in FIG. 10, the signal processing circuit 33 determines for each pixel 101 whether an event has been detected based on the event information from the event detection unit 329. For a pixel 101 for which no event has been detected (NO in S17), the process proceeds to S11, where multiple microframes F3 are acquired and added again to read out the next ranging frame. In this case, the start time of global gate driving relative to the light emission timing is reset to the first subframe period T3, and the same process is performed. For a pixel 101 for which an event has been detected (YES in S17), the process proceeds to S20.

[0088] In S20, the image generation unit 332 acquires pixel values ​​of a distance image for pixels in which the event detection unit 329 detected a change, based on the data added by the subframe adder 331. The image generation unit 332 acquires the distance corresponding to the subframe in which the signal value is maximized for each pixel, thereby generating pixel values ​​of a distance image showing a two-dimensional distribution of distances. For pixels 101 in which no event was detected (NO in S17), pixel values ​​of the distance image are not acquired, and the result of the previous ranging frame is used as is. The process then proceeds to S11, where multiple microframes F3 are acquired and added again to read the next ranging frame F1. In this case, the start time of global gate drive relative to the light emission timing is reset to the first subframe period T2, and similar processing is performed.

[0089] As described above, in this embodiment, for all pixels 101, multiple subframes F2 of the latest predetermined number of ranging frame periods T1 are stored in the subframe adder 331. The image generator 332 acquires pixel values ​​of the distance image for a pixel 101 in which the event detector 329 detected an event based on the data added and stored by the subframe adder 331. As a result, the pixel value of the distance image pixel corresponding to the pixel 101 in which the event was detected is updated from the pixel value of the distance image acquired in the previous ranging frame period T1. As a result, the added data of the pixel 101 stored in the subframe adder 331 is ensured to be added a maximum number of times as set forth previously, thereby reducing the influence of noise in acquiring the distance of the pixel 101 in which an event was detected. On the other hand, the image generator 332 does not acquire pixel values ​​of the distance image for a pixel 101 in which the event detector 329 did not detect an event. Because the distance of the pixel 101 is not acquired until an event is detected, the processing load on the image generator 332 can be reduced.

[0090] 11, in the first ranging frame period T1, regardless of whether an event is detected, pixel values ​​of the distance image may be acquired for all pixels 101. Furthermore, in the up to (N-1) ranging frame periods T1, the subframe adder 331 may acquire the pixel value of the pixel 101 in which an event is detected using data obtained by adding together multiple subframes F2 acquired in the (N-1)th ranging frame period T1.

[0091] Next, modifications of the above-described distance image generating device 30 will be described with reference to Figures 12 to 16. Descriptions of configurations that may be similar to those described above will be omitted or simplified as appropriate.

[0092] Fig. 12 is a schematic block diagram showing an example of the configuration of a circuit board 21 of this embodiment. Unlike the circuit board 21 shown in Fig. 4, the circuit board 21 shown in Fig. 12 does not include a vertical scanning circuit 116, a horizontal scanning circuit 117, a readout circuit 118, a pixel output signal line 119, an output circuit 120, or a control signal generation unit 121. Therefore, in this embodiment, the pixel signal processing unit 103 is configured to be able to output only one type of signal from the pixel 101. The other configuration may be the same as that of the circuit board 21 shown in Fig. 4.

[0093] Fig. 13 is a schematic block diagram showing an example of the configuration of one pixel of the photoelectric conversion unit 102 and pixel signal processing unit 103 according to this embodiment. Unlike the pixel signal processing unit 103 shown in Fig. 5, the pixel signal processing unit 103 shown in Fig. 13 does not include the vertical scanning circuit 116, drive lines 219, 220, 221, counter circuit 217, selection circuit 218, or gating circuit 222. The rest of the configuration may be similar to that of the pixel signal processing unit 103 shown in Fig. 5.

[0094] Fig. 14 is a functional block diagram showing an example of the schematic configuration of a distance image generating device 30 according to this embodiment. The configuration of the light receiving device 34 differs from the configuration shown in Fig. 7. However, the configurations of the light emitting device 31 and the signal processing circuit 33 may be the same.

[0095] 7, the configuration of the imaging unit 321 described with reference to Fig. 12 and Fig. 13 is different from that of the imaging unit 321 shown in Fig. 7, and is configured to output only one type of signal from one pixel 101. Furthermore, the light receiving device 34 of this embodiment can only simultaneously acquire one type of microframe F3 per pixel, and does not include a microframe readout unit 327 or a microframe adder 328. In other words, the detection operation and the event detection operation described above cannot be performed in parallel.

[0096] The gate pulse generating unit 322, microframe reading unit 323, microframe adding unit 324, addition count control unit 325, subframe output unit 326, and event detecting unit 329 have the same configuration as that shown in Fig. 7. However, the processing they perform is different. More specifically, the event detecting unit 329 obtains an event detection operation signal for event detection from the microframe adding unit 324.

[0097] 15 is a timing diagram illustrating the relationship between a ranging frame, subframes, and an event detection frame obtained by the event detection operation of this embodiment. A subframe 151 is one subframe F2 obtained in one subframe period T2 shown in FIG. 9, and the number of subframes F2 in one ranging frame F1 corresponds to the number of ranging points. In this embodiment, the event detection frame 152 is obtained from the microframe adder 324, so the gate pulse control method by the gate pulse generator 322 is different.

[0098] 8 is input to the gating circuit 216. After acquiring the subframes 151 corresponding to the number of ranging points, the gate pulse G02 of "P_G2" is input to the gating circuit 216, just like the gate pulse G01. In this way, the gate pulses G01 and G02 are input in a time-division manner to acquire subframes 151 for generating a range image and frames 152 for event detection. Each ranging frame period T1 can be said to include a period during which a detection operation is performed to acquire multiple subframes 151, and a period during which an event detection operation is performed (a period during which event detection frames 152 are acquired).

[0099] In this embodiment, an example has been shown in which one event detection frame 152 is acquired at the end of acquisition of multiple sub-frames 151 in one ranging frame period T1, but the present invention is not limited to this. It is sufficient that the event detection unit 329 can detect an event using the event detection frame 152. For example, multiple event detection frames may be acquired in one ranging frame period T1, and the event detection frames may be acquired at any position (timing) relative to the sub-frame 151 for generating a distance image.

[0100] As described above, in this embodiment, the same effects as those of the above-described embodiments can be obtained while reducing the circuit configuration of the light receiving device 34. In other words, in the distance image generating device 30, both a high frame rate and high distance measurement accuracy can be achieved.

[0101] Next, another example of acquiring an event detection frame will be described using FIG. 16. FIG. 16 is a timing diagram illustrating the relationship between a ranging frame, subframes, and an event detection frame obtained by the event detection operation of this embodiment. A subframe 161 is one subframe F2 acquired in one subframe period T2 shown in FIG. 9, and the number of subframes F2 in one ranging frame F1 corresponds to the number of ranging points. In the configuration shown in FIG. 16, an event detection frame 162 is generated from the subframes 161 used to generate a distance image. The event detection unit 329 reads the subframe 161 after the addition of the microframe F3 has been completed from the memory of the microframe adder 324 and stores it in the memory of the event detection unit 329. Each time a subframe 161 is read, the event detection unit 329 sequentially adds the value of the subframe 161 to the value stored in the memory. One event detection frame 162 is generated by adding and combining subframes 161 equal to the number of ranging points. The event detection unit 329 detects an event, for example, by comparing an event detection frame 162 corresponding to a certain ranging frame period T1 with the event detection frame 162 corresponding to the immediately preceding ranging frame period T1. In other words, the event detection unit 329 detects a change (event) in the signal value of each pixel 101 based on data obtained by combining multiple subframes 161 in each ranging frame period T1. In this embodiment, the event detection unit 329 is configured to hold two or more event detection frames 162 obtained by adding and combining as many subframes 161 as there are ranging points.

[0102] As described above, in this embodiment, event detection frames 162 are generated from sub-frames 161 for generating a distance image. In a low-light environment such as a darkroom where it is difficult to capture an image using normal photography, the sub-frames 161 for generating a distance image are generated by detecting reflected light of laser emission. Because the event detection frames 162 are generated from multiple sub-frames 161, noise reduction is achieved during event detection.

[0103] An application example of the distance image generating device 30 according to the above embodiment will be described below. FIG. 17 is a schematic diagram of an apparatus EQP equipped with the distance image generating device 30. FIG. 17 shows the photoelectric conversion device 100 (image capturing unit 321) of the distance image generating device 30. As described above, the photoelectric conversion device 100 may be a semiconductor chip with a stacked structure provided with a pixel region 12 in which pixels 101 are arranged. As shown in FIG. 17, the photoelectric conversion device 100 is housed in a semiconductor package PKG. The package PKG may include a base to which the photoelectric conversion device 100 is fixed, a cover such as glass facing the photoelectric conversion device 100, and conductive connecting members such as bonding wires or bumps that connect terminals provided on the base to terminals provided on the photoelectric conversion device 100. The apparatus EQP may further include at least one of a control device CTRL, a processing device PRCS, a display device DSPL, and a memory device MMRY. The package PKG containing the photoelectric conversion device 100 may contain not only the photoelectric conversion device 100 but also other components of the light receiving device 32 including the photoelectric conversion device 100 (imaging unit 321) and a signal processing circuit 33. The package PKG may also contain a light emitting device 31. However, the present invention is not limited to this, and for example, the light emitting device 31 may be disposed separately from the package PKG.

[0104] The optical system OPT forms an image on the pixel region 12 and may be, for example, a lens, a shutter, or a mirror. The control device CTRL controls the operation of the distance image generating device 30 and may be, for example, a semiconductor device such as an ASIC. The processing device PRCS processes signals output from the distance image generating device 30 and may be, for example, a semiconductor device such as a CPU or an ASIC. The display device DSPL may be an EL display device or a liquid crystal display device that displays data obtained by the distance image generating device 30. The memory device MMRY is a magnetic device or a semiconductor device that stores data obtained by the distance image generating device 30. The memory device MMRY may be a volatile memory such as an SRAM or a DRAM, or a non-volatile memory such as a flash memory or a hard disk drive. The mechanical device MCHN may have a moving or propulsive part such as a motor or an engine. The mechanical device MCHN also drives components of the optical system OPT for, for example, zooming, focusing, and shutter operation. The device EQP displays the data output from the distance image generating device 30 on the display device DSPL, and transmits the data to the outside via a communication device (not shown) provided in the device EQP. For this purpose, the device EQP may be provided with a memory device MMRY and a processing device PRCS.

[0105] The device EQP incorporating the distance image generating device 30 can be applied to surveillance cameras, on-board cameras mounted on transportation equipment such as automobiles, railroad cars, ships, aircraft, and industrial robots. In addition, the device EQP incorporating the distance image generating device 30 can be applied not only to transportation equipment but also to a wide range of equipment that uses object recognition, such as intelligent transport systems (ITS).

[0106] The disclosure of this specification includes the following distance image generating devices and apparatus.

[0107] (Item 1) a plurality of pixels each having a photoelectric conversion element; and an image generation unit for generating a distance image based on outputs from the plurality of pixels during each of a plurality of ranging frame periods, In each ranging frame period, a plurality of subframes are acquired by a plurality of detection operations in which the time from the light emission timing of the light source to the exposure period in which the light is detected by the photoelectric conversion element is different from one another, an event detection unit that detects a change in a signal value between at least two consecutive ranging frame periods among the plurality of ranging frame periods for each of the plurality of pixels; In each ranging frame period, the image generation unit acquires pixel values ​​of a distance image based on the plurality of subframes for pixels in which a change has been detected by the event detection unit; A distance image generating device characterized in that during each ranging frame period, the image generating unit does not acquire pixel values ​​of a distance image for pixels for which the event detection unit does not detect a change, or acquires pixel values ​​of a distance image based on the multiple subframes when set conditions are met.

[0108] (Item 2) 2. The distance image generating device according to item 1, wherein in each ranging frame period, the image generating unit acquires pixel values ​​of a distance image for pixels in which a change has been detected by the event detection unit based on the plurality of subframes of the ranging frame period in which a change has been detected by the event detection unit.

[0109] (Item 3) a subframe adder configured to hold the plurality of subframes of a pixel for which the event detection unit does not detect a change in each ranging frame period, and to add up the plurality of subframes of the consecutive ranging frame periods held for each pixel when the event detection unit does not detect a change in consecutive ranging frame periods; 3. The distance image generating device according to item 1 or 2, characterized in that in each ranging frame period, for a pixel for which the event detection unit does not detect a change, if the event detection unit does not detect a change over a predetermined number of consecutive ranging frame periods, the image generating unit acquires the pixel value of the distance image based on the data added by the subframe adder.

[0110] (Item 4) The distance image generating device described in item 3 is characterized in that, when the image generating unit obtains pixel values ​​of a distance image based on data added by the subframe adder, the subframe adder resets the held data of the pixel whose pixel value has been obtained.

[0111] (Item 5) a subframe adder configured to hold the plurality of subframes of a predetermined number of latest ranging frame periods among the plurality of ranging frame periods, and to add up the plurality of subframes of the predetermined number of ranging frame periods held for each pixel, In each ranging frame period, the image generation unit acquires a pixel value of a distance image for a pixel in which a change has been detected by the event detection unit based on the data added by the subframe adder, 2. The distance image generating device according to item 1, wherein in each ranging frame period, the image generating unit does not acquire pixel values ​​of the distance image for pixels for which the event detecting unit does not detect a change.

[0112] (Item 6) an event detection operation is performed in each ranging frame period to acquire a signal from the photoelectric conversion element during an exposure period having a length different from that of each of the exposure periods of the plurality of detection operations; 6. The distance image generating device according to any one of items 1 to 5, wherein the event detection unit detects a change in the signal value of each pixel based on the signal obtained by the event detection operation.

[0113] (Item 7) further comprising a first readout unit that reads out signals of the plurality of detection operations and a second readout unit that reads out a signal of the event detection operation, 7. The distance image generating device according to item 6, wherein the plurality of detection operations and the event detection operation are performed in parallel.

[0114] (Item 8) 7. The distance image generating device according to item 6, wherein each ranging frame period includes a period in which the plurality of detection operations are performed to acquire the plurality of sub-frames and a period in which the event detection operation is performed.

[0115] (Item 9) 9. The distance image generating device according to any one of items 6 to 8, wherein the exposure period of the event detection operation is longer than the exposure periods of each of the plurality of detection operations.

[0116] (Item 10) The distance image generating device described in any one of items 1 to 6, characterized in that the event detection unit detects changes in the signal value of each pixel based on data obtained by combining the multiple subframes during each ranging frame period.

[0117] (Item 11) further comprising a memory for storing pixel values ​​of the distance image; A distance image generating device described in any one of items 1 to 10, characterized in that in each ranging frame period, for pixels for which pixel values ​​of the distance image are not acquired, the image generating unit uses the pixel value in the previous ranging frame period for the distance image.

[0118] (Item 12) 12. The distance image generating device according to any one of items 1 to 11, wherein the photoelectric conversion element includes an avalanche photodiode.

[0119] (Item 13) A distance image generating device according to any one of items 1 to 12, a processing device that processes a signal output from the distance image generating device; An apparatus characterized by comprising:

[0120] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0121] 30: distance image generating device, 101: pixel, 329: event detection unit, 332: image generating unit, F2: subframe, T1: distance measurement frame period

Claims

1. a plurality of pixels each having a photoelectric conversion element; and an image generation unit for generating a distance image based on outputs from the plurality of pixels during each of a plurality of ranging frame periods, In each ranging frame period, a plurality of subframes are acquired by a plurality of detection operations in which the time from the light emission timing of the light source to the exposure period in which the light is detected by the photoelectric conversion element is different from one another, an event detection unit that detects a change in a signal value between at least two ranging frame periods among the plurality of ranging frame periods for each of the plurality of pixels; In each ranging frame period, the image generation unit acquires pixel values ​​of a distance image based on the plurality of subframes for pixels in which a change has been detected by the event detection unit; A distance image generating device characterized in that during each ranging frame period, the image generating unit does not acquire pixel values ​​of a distance image for pixels for which the event detection unit does not detect a change, or acquires pixel values ​​of a distance image based on the multiple subframes when set conditions are met.

2. 2. The distance image generating device according to claim 1, wherein, in each ranging frame period, the image generating unit acquires pixel values ​​of a distance image for pixels in which a change has been detected by the event detection unit based on the plurality of subframes of the ranging frame period in which a change has been detected by the event detection unit.

3. a subframe adder configured to hold the plurality of subframes of a pixel for which the event detection unit does not detect a change in each ranging frame period, and to add up the plurality of subframes of the consecutive ranging frame periods held for each pixel when the event detection unit does not detect a change in consecutive ranging frame periods; The distance image generating device according to claim 1, characterized in that in each ranging frame period, for a pixel in which the event detection unit does not detect a change, if the event detection unit does not detect a change over a predetermined number of consecutive ranging frame periods, the image generating unit acquires the pixel value of the distance image based on the data added by the subframe adder.

4. The distance image generating device according to claim 3, characterized in that the subframe adder resets the held data of the pixel whose pixel value has been acquired when the image generating unit acquires the pixel value of the distance image based on the data added by the subframe adder.

5. a subframe adder configured to hold the plurality of subframes of a predetermined number of latest ranging frame periods among the plurality of ranging frame periods, and to add up the plurality of subframes of the predetermined number of ranging frame periods held for each pixel, In each ranging frame period, the image generation unit acquires a pixel value of a distance image for a pixel in which a change has been detected by the event detection unit based on the data added by the subframe adder, 2. The distance image generating device according to claim 1, wherein in each distance measurement frame period, the image generating section does not acquire pixel values ​​of the distance image for pixels for which the event detecting section does not detect a change.

6. an event detection operation is performed in each ranging frame period to acquire a signal from the photoelectric conversion element during an exposure period having a length different from that of each of the exposure periods of the plurality of detection operations; 2. The distance image generating device according to claim 1, wherein the event detection unit detects a change in the signal value of each pixel based on the signal obtained by the event detection operation.

7. further comprising a first readout unit that reads out signals of the plurality of detection operations and a second readout unit that reads out a signal of the event detection operation, 7. The distance image generating device according to claim 6, wherein the plurality of detection operations and the event detection operation are performed in parallel.

8. 7. The distance image generating device according to claim 6, wherein each ranging frame period includes a period in which the plurality of detection operations are performed to acquire the plurality of sub-frames and a period in which the event detection operation is performed.

9. 7. The distance image generating device according to claim 6, wherein an exposure period of the event detection operation is longer than an exposure period of each of the plurality of detection operations.

10. 2. The distance image generating device according to claim 1, wherein the event detection unit detects a change in the signal value of each pixel based on data obtained by combining the plurality of subframes in each distance measurement frame period.

11. further comprising a memory for storing pixel values ​​of the distance image; 2. The distance image generating device according to claim 1, wherein in each ranging frame period, for pixels for which pixel values ​​of the distance image are not acquired, the image generating unit uses the pixel value in the previous ranging frame period for the distance image.

12. 2. The distance image generating device according to claim 1, wherein the photoelectric conversion element includes an avalanche photodiode.

13. A distance image generating device according to any one of claims 1 to 12; a processing device that processes a signal output from the distance image generating device; An apparatus characterized by comprising:

Images