Imaging device, control device, and spiking neural network

Abstract

The present invention suppresses a decrease in sensitivity and controls output saturation of an event. The imaging apparatus includes: a light receiving unit disposed in each pixel arranged in a matrix shape in a row direction and a column direction, and outputting a pulse based on an incident photon; a counter provided in each pixel and counting the pulses output from the light receiving unit; and a comparator provided in each pixel and outputting an event based on a comparison result of a count value from the counter and a counter threshold value. The comparator may output an event when the count value is greater than a counter threshold. The comparator may reset the counter when the count value is greater than the counter threshold.

Description

Technical Field

[0001] The present technology relates to an imaging device, a control device, and a learning model. Specifically, the present technology relates to an imaging device, a control device, and a spiking neural network that can control the generation rate of events. Background Art

[0002] In an imaging device, there is a technique of generating events based on the amount of light incident on each pixel. At this time, in the case of high light intensity, event output may saturate because emissions occur frequently. To suppress the emission rate, for example, a technique of introducing a driving method for shortening the exposure time has been proposed (see, for example, Non-Patent Document 1).

[0003] Citation List

[0004] Non-Patent Document

[0005] Non-Patent Document 1: Ecole polytechnique fédéralede Lausanne (EPFL), Canon Inc., "Megapixel time-gated SPAD image sensor for 2D and 3D imaging applications", 2020 Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] However, in the above-described conventional technology, if the exposure time is shortened in order to suppress the saturation of event output, there is a possibility of a decrease in sensitivity and an increase in noise.

[0008] In view of this situation, the present technology has been made, and its object is to suppress the saturation of event output while suppressing a decrease in sensitivity.

[0009] Solutions to the Problems

[0010] The present technology has been made to solve the above problems, and a first aspect thereof is an imaging device including: a light receiving unit provided in pixels arranged in a matrix along a row direction and a column direction and outputting a pulse based on the incidence of photons; a counter provided in the pixels and counting the pulses output from the light receiving unit; and a comparator provided in the pixels and outputting an event based on a comparison result between a count value of the counter and a counter threshold. This has the effect of outputting an event at an output rate of a compressed pulse.

[0011] In addition, in the first aspect, when the count value exceeds the counter threshold, the comparator can output an event. This brings the following effect: compressing the output rate of the pulse based on the counter threshold.

[0012] In addition, in the first aspect, when the count value exceeds the counter threshold, the comparator can reset the counter. This brings the effect of starting to count from the beginning whenever an event is output.

[0013] In addition, in the first aspect, the counter and the comparator can be arranged below the light receiving portion. This brings the following effect: forming a counter and a comparator for each light receiving portion while suppressing an increase in the planar size of the imaging device.

[0014] In addition, in the first aspect, the light receiving portion can include a single photon avalanche diode (SPAD). This brings the effect of counting photons one by one.

[0015] In addition, in the first aspect, a control unit for controlling the counter threshold based on the event output rate can be included. This brings the effect of dynamically changing the counter threshold according to the amount of light.

[0016] In addition, in the first aspect, the control unit can include a spiking neural network that controls the counter threshold based on the event input. This brings the effect that the counter threshold can be controlled asynchronously.

[0017] In addition, in the first aspect, the spiking neural network can include: a plurality of first spiking neurons, each first spiking neuron receiving the event input and emitting based on the event input rate; and a second spiking neuron, the emissions of the plurality of first spiking neurons being respectively input to the second spiking neuron, and the second spiking neuron emitting based on the input rate of the emissions of the first spiking neurons. This brings the effect that the input rate in the spatial direction of the event and the input rate in the temporal direction of the event can be detected.

[0018] In addition, in the first aspect, the spiking neural network can be capable of detecting the input rate in the spatial direction of the event and the input rate in the temporal direction of the event. This brings the effect that the event rate can be controlled asynchronously while stabilizing the counter threshold with respect to temporal and spatial changes in the amount of light.

[0019] In addition, in the first aspect, each of the first spiking neurons in the first spiking neuron group may include: a high-rate detection first spiking neuron that fires based on an increase in the input rate of the event; and a low-rate detection first spiking neuron that fires based on a decrease in the input rate of the event. And the second spiking neuron may include: a high-rate detection second spiking neuron that is connected such that the neuron membrane potential rises based on an input from the firing of the high-rate detection first spiking neuron and the neuron membrane potential falls based on an input from the firing of the low-rate detection first spiking neuron; and a second low-rate detection spiking neuron that is connected such that the neuron membrane potential falls based on an input from the firing of the high-rate detection first spiking neuron and the neuron membrane potential rises based on an input from the firing of the low-rate detection first spiking neuron. This brings the following effects: while enabling asynchronous detection of the event rate, the counter threshold is updated upward / downward according to low and high light levels.

[0020] In addition, in the first aspect, the spiking neural network may be able to control the counter threshold at multiple levels. This brings the effect of fine-tuning the counter threshold according to the light amount.

[0021] In addition, in the first aspect, the spiking neural network may be able to control the counter threshold at a constant rate. This brings the effect of fine-tuning the counter threshold according to the light amount.

[0022] In addition, in the first aspect, the control unit may control the negative power supply voltage of the light receiving unit based on the output rate of the event. Thereby, the effect of adjusting the sensitivity of the light receiving unit according to the light reception amount of the light receiving unit is produced.

[0023] In addition, in the first aspect, a vertical arbiter may also be included, which arbitrates the output of events in a row based on the detection results of events in each row. This brings the effect of outputting events only from the row where the event occurs.

[0024] In addition, in the first aspect, a horizontal arbiter may also be included, which arbitrates the output of events in a column based on the detection results of events in each column. This brings the effect of outputting events only from the column where the event occurs.

[0025] In addition, the second aspect is a control device, including a control unit that receives the comparison result between the count value of the pulses based on the incident output of photons and the counter threshold as the input of the event, and controls the counter threshold based on the output rate of the event. This brings the following effect: while changing the compression ratio according to the light amount, outputting an event of the output rate of the compressed pulse.

[0026] In addition, in a second aspect, the control unit may include a spiking neural network that controls the counter threshold based on event inputs. This has the effect of being able to control the counter threshold asynchronously.

[0027] In addition, a third aspect is a spiking neural network, including: a plurality of first spiking neurons, each first spiking neuron receiving an input of pulses generated based on the incidence of photons at different spatial positions from each other and emitting based on the input rate of the pulses; and a second spiking neuron, the emissions of the plurality of first spiking neurons being respectively input to the second spiking neuron, and the second spiking neuron emitting based on the input rate of the emissions of the first spiking neurons. This has the effect of being able to asynchronously detect the input rate in the spatial direction of an event and the input rate in the temporal direction of the event.

[0028] In addition, in the third aspect, the input rate in the spatial direction of the detectable pulses and the input rate in the temporal direction of the pulses can be detected. This has the effect of being able to asynchronously detect the event rate while stabilizing the counter threshold with respect to temporal and spatial changes in the light amount.

[0029] In addition, in the third aspect, each of the first spiking neurons may include: a high-rate detection first spiking neuron that emits based on an increase in the input rate of the pulses; and a low-rate detection first spiking neuron that emits based on a decrease in the input rate of the pulses, and the second spiking neuron may include: a high-rate detection second spiking neuron that is connected such that the neuron membrane potential rises based on the input of the emissions from the high-rate detection first spiking neuron and the neuron membrane potential falls based on the input of the emissions from the low-rate detection first spiking neuron; and a second low-rate detection spiking neuron that is connected such that the neuron membrane potential falls based on the input of the emissions from the high-rate detection first spiking neuron and the neuron membrane potential rises based on the input of the emissions from the low-rate detection first spiking neuron. This has the following effects: while enabling the asynchronous detection of the event rate, the counter threshold is updated up / down according to low and high light amounts. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a block diagram showing a configuration example of an imaging device according to a first embodiment.

[0031] Figure 2 is a block diagram showing a configuration example of a solid-state imaging device according to a first embodiment.

[0032] Figure 3 is a block diagram showing a configuration example of a solid-state imaging device according to a first embodiment layer by layer.

[0033] Figure 4It is a circuit diagram showing a configuration example of pixels according to the first embodiment.

[0034] Figure 5 It is a diagram showing a configuration example of a control spiking neural network (SNN) that can detect the input rate of events in the time direction according to the first embodiment.

[0035] Figure 6 It is a sketch showing the firing probabilities of a low-rate detection spiking neuron and a high-rate detection spiking neuron according to the first embodiment.

[0036] Figure 7 It is a timing diagram showing the operation of a control SNN that can detect the input rate in the time direction of events according to the first embodiment.

[0037] Figure 8 It is a diagram showing a configuration example of a control SNN that can detect the input rate in the time and space directions of events according to the first embodiment.

[0038] Figure 9 It is a timing diagram showing the operation of a control SNN that can detect the input rate in the time and space directions of events according to the first embodiment.

[0039] Figure 10 It is a block diagram showing a configuration example of a solid-state imaging device according to the second embodiment layer by layer.

[0040] Figure 11 It is a block diagram showing a configuration example of a solid-state imaging device according to the third embodiment layer by layer.

[0041] Figure 12 It is a block diagram showing a configuration example of a solid-state imaging device according to the fourth embodiment layer by layer.

[0042] Figure 13 It is a block diagram showing a configuration example of a solid-state imaging device according to the fifth embodiment layer by layer.

[0043] Figure 14 It is a block diagram showing a configuration example of a control SNN according to the sixth embodiment.

[0044] Figure 15 It is a diagram showing a configuration example of a control SNN that can detect the input rate in the time and space directions of events according to the sixth embodiment.

[0045] Figure 16 It is a block diagram showing a configuration example of a control SNN according to the seventh embodiment.

[0046] Figure 17It is a block diagram showing a configuration example of a control SNN according to the eighth embodiment.

[0047] Figure 18 It is a block diagram showing a configuration example of a distance measuring device according to the ninth embodiment.

[0048] Figure 19 It is a block diagram showing a schematic configuration example of a vehicle control system.

[0049] Figure 20 It is an explanatory diagram showing an example of the installation position of an imaging unit. Detailed Embodiments

[0050] Hereinafter, modes for implementing the present technology (hereinafter referred to as embodiments) will be described. The description will be given in the following order.

[0051] 1. First Embodiment (Example of controlling the output rate of events based on the comparison result between the count value generated by a counter and a counter threshold)

[0052] 2. Second Embodiment (Example of arbitrating the output of events in a row based on the detection result of events in each row)

[0053] 3. Third Embodiment (Example of arbitrating the output of events in each row and each column based on the detection result of events in each row and each column)

[0054] 4. Fourth Embodiment (Example of controlling the negative power supply voltage of a SPAD based on the comparison result between the count value generated by a counter and a counter threshold)

[0055] 5. Fifth Embodiment (Example in which the counter threshold can be controlled at one level or more)

[0056] 6. Sixth Embodiment (Example in which the counter threshold can be controlled at multiple levels)

[0057] 7. Seventh Embodiment (Example in which the counter threshold can be controlled at a constant rate)

[0058] 8. Eighth Embodiment (Example in which the counter threshold can be controlled based on a level and a ratio)

[0059] 9. Ninth Embodiment (Example of applying the control of the output rate of events to a distance measuring device)

[0060] 10. Application Example of a Moving Body

[0061] <1. First Embodiment>

[0062] Figure 1 It is a block diagram showing a configuration example of an imaging device according to the first embodiment.

[0063] In the accompanying drawings, the imaging device 100 includes an optical system 101, a solid-state imaging device 102, an imaging control unit 103, an image processing unit 104, a storage unit 105, a display unit 106, and an operation unit 107. The imaging control unit 103, the image processing unit 104, the storage unit 105, the display unit 106, and the operation unit 107 are connected to each other via a bus 108. It should be noted that the imaging device 100 can be used alone, integrated into a portable terminal (e.g., a smart phone), or integrated into an authentication device or a monitoring device.

[0064] The optical system 101 guides light from an object into the solid-state imaging device 102 and forms an object image on the light-receiving surface of the solid-state imaging device 102. The optical system 101 may include, for example, a focusing lens, a zoom lens, an aperture, etc. The optical system 101 may include a plurality of lenses, such as a wide-angle lens, a standard lens, and a telephoto lens.

[0065] The solid-state imaging device 102 converts the light from the object into an electrical signal for each pixel, digitizes the electrical signal, and outputs it. The solid-state imaging device 102 may be, for example, an event-based vision sensor. The light received by the solid-state imaging device 102 may be visible light, near-infrared light (NIR), short-wavelength infrared light (SWIR), ultraviolet light, X-rays, etc.

[0066] The imaging control unit 103 controls the imaging of the solid-state imaging device 102 based on a command from the operation unit 107. At this time, the imaging control unit 103 may control the exposure conditions, imaging timing, etc. of the solid-state imaging device 102.

[0067] The image processing unit 104 performs image processing based on the output from the solid-state imaging device 102. The image processing unit 104 may include an application processor that performs processing based on software.

[0068] The storage unit 105 stores the captured images captured by the solid-state imaging device 102, and stores imaging parameters of the solid-state imaging device 102, etc. In addition, the storage unit 105 may store a program for operating the imaging device 100 based on software. The storage unit 105 may include a read-only memory (ROM), a random access memory (RAM), and a memory card.

[0069] The display unit 106 displays the captured image and displays various types of information that support the imaging operation. The display unit 106 may be a liquid crystal display or an organic electroluminescence (EL) display.

[0070] The operation unit 107 provides a user interface for operating the imaging device 100. The operation unit 107 may include, for example, buttons, dials, and switches provided in the imaging device 100. The operation unit 107 may include a touch panel configured together with the display unit 106.

[0071] Figure 2 It is a block diagram showing a configuration example of a solid-state imaging device according to the first embodiment.

[0072] In the drawings, the solid-state imaging device 102 includes a control unit 112, a pixel array unit 111, and a signal processing unit 113. These circuits may be provided on a single semiconductor substrate or may be provided on a stacked substrate.

[0073] In the pixel array unit 111, the pixels 110 are arranged in an array shape in the row direction and the column direction. Each pixel 110 is connected to the signal lines 141 of each column and the control lines 142 of each row. The pixel 110 outputs an event generated based on the comparison result between the count value of the pulse generated according to the incidence of photons and the count threshold as pixel data. It should be noted that the event is a signal indicating the brightness change of the incident light in the same direction. At this time, the pixel 110 may include a light-receiving unit and a circuit unit. The circuit unit may be arranged below the light-receiving unit. The light-receiving unit may include a single-photon avalanche diode (SPAD). The light-receiving unit may be a photodiode. In each pixel 110, the circuit unit may output an event to the signal line 141 based on the comparison result between the count value of the pulse output from the light-receiving unit and the counter threshold.

[0074] The control unit 112 sequentially selects rows in synchronization with the vertical synchronization signal. At this time, the control unit 112 can select the pixel 110 via the control line 142. The control unit 112 may include a vertical arbiter that arbitrates the selection of the row including the pixel 110 in which an event is detected. The signal processing unit 113 performs various types of signal processing on the image data in which the pixel data is arranged. The signal processing unit 113 may include a row scanner that scans columns. The signal processing unit 113 may include a horizontal arbiter that arbitrates the selection of the column including the pixel 110 in which an event is detected.

[0075] Figure 3 It is a block diagram showing a specific example of a solid-state imaging device according to the first embodiment.

[0076] In the drawings, the pixel array unit 111 includes a light-receiving array unit 120 and a circuit array unit 130. The light-receiving array unit 120 may be stacked on the circuit array unit 130. The light-receiving array unit 120 includes light-receiving units 121. The light-receiving units 121 are arranged in an array shape in the row direction and the column direction. The circuit array unit 130 includes circuit units 131. The circuit units 131 are arranged in an array shape in the row direction and the column direction. At this time, each pixel 110 may include each light-receiving unit 121 and each circuit unit 131. The circuit unit 131 may be provided directly below the light-receiving unit 121.

[0077] The circuit unit 131 can generate an event by compressing the rate of the pulses output from the optical receiving unit 121 and output the event to the signal line 141. At this time, the circuit unit 131 can output the event to the signal line 141 based on the comparison result between the count value of the pulses output from the optical receiving unit 121 and the counter threshold CTH. In the generation of the event, an exposure can be left. At this time, a non-exposure period may not be provided.

[0078] The signal processing unit 113 includes a line scanner 151, a main processor 152, a control SNN 153, and a threshold register 154.

[0079] The line scanner 151 scans the signal lines 141 of each column and reads events from the signal lines 141 of each column.

[0080] The main processor 152 processes the events read by the line scanner 151. For example, the main processor 152 can configure an image for viewing or can configure an image for sensing based on the events. In addition, the main processor 152 can perform image processing on these images.

[0081] The control SNN 153 updates the counter threshold CTH based on the input rate of the events output from the circuit unit 131. At this time, the control SNN 153 can detect the input rate in the spatial direction of the events and the input rate in the temporal direction of the events. The control SNN 153 can update the counter threshold CTH step by step, or at a constant rate, or in a mixed manner (both step by step and at a constant rate). At this time, the control SNN 153 can generate an up signal SU for increasing the counter threshold CTH and a down signal SD for decreasing the counter threshold CTH.

[0082] The threshold register 154 stores the counter threshold CTH for the count value of the pulses output from the optical receiving unit 121 and outputs the counter threshold CTH to the circuit unit 131. The threshold register 154 can update the counter threshold CTH based on the up signal SU and the down signal SD from the control SNN 153.

[0083] Figure 4 It is a circuit diagram showing a configuration example of a pixel according to the first embodiment.

[0084] In the drawing, the pixel 110 includes a SPAD 122, a quenching resistor 132, a P-channel transistor 133, an N-channel transistor 134, an inverter 135, a counter 136, a comparator 137, and a latch circuit 138.

[0085] The SPAD 122 detects photons one by one. At this time, the SPAD 122 can amplify the current based on avalanche amplification. However, in the SPAD 122, a negative voltage VN is set such that a voltage higher than the breakdown voltage is applied. At this time, the amplification factor of avalanche amplification is theoretically infinite. Therefore, the SPAD 122 can generate a saturated output current regardless of the amount of photons incident per unit time and can detect photons one by one.

[0086] The quenching resistor 132 forcibly stops the avalanche amplification of the SPAD 122. The quenching resistor 132 can use the resistance component of a MOS transistor. The power supply voltage VE can be applied to the MOS transistor. At this time, the resistance value of the quenching resistor 132 can be set based on a control signal CNT applied to the gate of the MOS transistor. A reverse voltage higher than the breakdown voltage is set to the SPAD 122 via the quenching resistor 132. Therefore, when a current flows through the SPAD 122 based on avalanche amplification, the voltage applied to the SPAD 122 decreases based on the voltage drop across the quenching resistor 132, and the avalanche amplification stops.

[0087] The P-channel transistor 133 and the N-channel transistor 134 are connected in series. The gate of the P-channel transistor 133 and the gate of the N-channel transistor 134 are connected to the cathode of the SPAD 122. The power supply voltage VE can be applied to the P-channel transistor 133.

[0088] The inverter 135 generates a pulse PL based on the output from the connection point between the P-channel transistor 133 and the N-channel transistor 134 and outputs the pulse PL to the counter 136. The power supply voltage VDDL can be applied to the inverter 135. The power supply voltage VDDL can be lower than the power supply voltage VE.

[0089] The counter 136 counts the pulses PL output from the inverter 135 and outputs the count value CNT to the comparator 137. The power supply voltage VDDL can be applied to the counter 136.

[0090] The comparator 137 outputs an event IVE based on the comparison result between the count value of the counter 136 and the counter threshold CTH. For example, when the count value exceeds the counter threshold CTH, the comparator 137 can output the event IVE. In addition, when the count value exceeds the counter threshold CTH, the comparator 137 can reset the counter 136. At this time, the comparator 137 can adjust the output rate of the event IVE based on the counter threshold CTH.

[0091] The latch circuit 138 latches the event IVE output from the comparator 137. Then, the latch circuit 138 outputs the event IVE to the signal line 141 based on a specified timing.

[0092] The pixel 110 can be formed in a stacked chip. At this time, the SPAD 122 can be formed on the upper chip 129. In the lower chip 139, a quenching resistor 132, a P-channel transistor 133, an N-channel transistor 134, an inverter 135, a counter 136, a comparator 137, and a latch circuit 138 can be formed.

[0093] The lower chip 139 and the upper chip 129 can be directly bonded. At this time, pad electrodes 229 and 239 can be formed on the lower chip 139 and the upper chip 129, respectively. The pad electrode 239 is connected to the quenching resistor 132, the gate of the P-channel transistor 133, and the gate of the N-channel transistor 134. The pad electrode 229 is connected to the SPAD 122. The pad electrodes 229 and 239 can be arranged to face each other. When directly bonding the lower chip 139 and the upper chip 129, hybrid bonding can be used. At this time, the pad electrodes 229 and 239 can be Cu-Cu connected. The materials of the semiconductor substrates for the lower chip 139 and the upper chip 129 can be Si, InGaAs, or InP.

[0094] Figure 5 It is a diagram showing a configuration example of a control SNN capable of detecting the input rate of an event in the time direction according to the first embodiment.

[0095] In the drawing, the control SNN 153 is provided with a high-rate detection spiking neuron 201 and a low-rate detection spiking neuron 202. The event IVE generated by the circuit unit 131 is input to the high-rate detection spiking neuron 201 and the low-rate detection spiking neuron 202. The high-rate detection spiking neuron 201 outputs a high-rate detection spike SPH based on the event IVE generated by the circuit unit 131. The low-rate detection spiking neuron 202 outputs a low-rate detection spike SPL based on the event IVE generated by the circuit unit 131.

[0096] Figure 6 It is a graph showing the firing probabilities of the low-rate detection spiking neuron and the high-rate detection spiking neuron according to the first embodiment.

[0097] In the drawing, in the high-rate detection spiking neuron 201, as the output rate FRQ of the event IVE generated in the circuit unit 131 increases, the firing probability PRB increases. In the low-rate detection spiking neuron 202, as the output rate FRQ of the event IVE generated in the circuit unit 131 decreases, the firing probability PRB increases.

[0098] Figure 7 It is a timing diagram showing the operation of a control SNN capable of detecting the input rate in the time direction of an event according to the first embodiment.

[0099] In the drawings, the neuron threshold NTH is set for the high-rate detection spiking neuron 201, and the neuron threshold NTL is set for the low-rate detection spiking neuron 202. Further, in the high-rate detection spiking neuron 201, when a pulse is input, the neuron membrane potential NVH rises, and when no pulse is input, the neuron membrane potential NVH gradually decreases. In the low-rate detection spiking neuron 202, the neuron membrane potential NVL gradually increases when no pulse is input, and the neuron membrane potential NVH drops when a pulse is input.

[0100] Then, during the high light intensity period KH, based on the photon incidence to the SPAD 122, the pulse PL is input to the counter 136. Whenever the pulse PL is input to the counter 136, the count value CNT is accumulated and input to the comparator 137. Then, in the comparator 137, when the count value CNT reaches the counter threshold CTH, the event IVE is output from the comparator 137 to the signal line 141, and the counter 136 is reset (from time T11 to t13).

[0101] The event IVE output to the signal line 141 is input to the high-rate detection spiking neuron 201 and the low-rate detection spiking neuron 202. Then, in the high-rate detection spiking neuron 201, when the output rate FRQ of the event IVE from the circuit unit 131 increases and the input of the next event IVE is repeated before the neuron membrane potential NVH completely drops, the neuron membrane potential NVH reaches the neuron threshold NTH. Then, when the neuron membrane potential NVH reaches the neuron threshold NTH, the high-rate detection spiking neuron 201 generates a high-rate detection spike SPH. At this time, the high-rate detection spiking neuron 201 can output the high-rate detection spike SPH to the threshold register 154 as an upward signal SU.

[0102] When the upward signal SU is input, the threshold register 154 increases the counter threshold CTH and outputs it to the circuit unit 131. In the circuit unit 131, when the counter threshold CTH increases, the output rate of the event IVE decreases, and saturation of the event IVE during the high light intensity period KH can be suppressed.

[0103] On the other hand, during the low light amount period KL, the pulse PL is input to the counter 136 based on the incidence of photons to the SPAD 122. Whenever the pulse PL is input to the counter 136, the count value CNT is accumulated and input to the comparator 137. Then, in the comparator 137, when the state where the count value CNT does not reach the counter threshold CTH continues, the state where the event IVE is not input to the low rate detection spiking neuron 202 continues. Then, when the state where the event IVE is not input to the low rate detection spiking neuron 202 continues, the neuron membrane potential NVL gradually increases, and the neuron membrane potential NVL reaches the neuron threshold NTL. Then, when the neuron membrane potential NVL reaches the neuron threshold NTL, the low rate detection spiking neuron 202 generates a low rate detection spike SPL(t14). At this time, the low rate detection spiking neuron 202 can output the low rate detection spike SPL as a down signal SD to the threshold register 154.

[0104] When the down signal SD is input, the threshold register 154 decreases the counter threshold CTH and outputs it to the circuit unit 131. In the circuit unit 131, when the counter threshold CTH decreases, the output rate of the event IVE increases, and the loss of the event IVE during the low light amount period KL can be suppressed.

[0105] Figure 8 FIG. is a diagram showing a configuration example of a control SNN capable of detecting the input rate in the time direction and space direction of an event according to the first embodiment. It should be noted that in the drawing, for the purpose of detecting the input rate in the space direction, the configuration of the control SNN 153 for three pixels having different spatial positions is regarded as an example, but the control SNN 153 can correspond to a larger number of pixels.

[0106] In the drawing, a first layer and a second layer are provided in the control SNN 153. In the first layer of the control SNN 153, a high rate detection spiking neuron 211 and a low rate detection spiking neuron 212 are provided for each circuit unit 131. A high rate detection spiking neuron 221 and a low rate detection spiking neuron 222 are provided in the second layer of the control SNN 153.

[0107] A positive connection 231 is provided between the high rate detection spiking neuron 211 and the high rate detection spiking neuron 221. A negative connection 232 is provided between the high rate detection spiking neuron 211 and the low rate detection spiking neuron 222. A negative connection 232 is provided between the low rate detection spiking neuron 212 and the high rate detection spiking neuron 221. A positive connection 231 is provided between the low rate detection spiking neuron 212 and the low rate detection spiking neuron 222.

[0108] The positive connection 231 can increase the neuron membrane potential NVH of the high-rate detection spiking neuron 221 based on the pulses input to the high-rate detection spiking neuron 221. In addition, the positive connection 231 can increase the neuron membrane potential NVL of the low-rate detection spiking neuron 222 based on the pulses input to the low-rate detection spiking neuron 222. On the other hand, the negative connection 232 can decrease the neuron membrane potential NVH of the high-rate detection spiking neuron 221 based on the pulses input to the high-rate detection spiking neuron 221. In addition, the negative connection 232 can decrease the neuron membrane potential NVL of the low-rate detection spiking neuron 222 based on the pulses input to the low-rate detection spiking neuron 222.

[0109] The event IVE generated in the circuit unit 131 is input to the high-rate detection spiking neuron 211 and the low-rate detection spiking neuron 212 of each circuit unit 131. The high-rate detection spiking neuron 221 outputs a high-rate detection spike SPH based on the positive connection 231 with the high-rate detection spiking neuron 211 and the negative connection 232 with the low-rate detection spiking neuron 212. The low-rate detection spiking neuron 222 outputs a low-rate detection spike SPL based on the negative connection 232 with the high-rate detection spiking neuron 211 and the positive connection 231 with the low-rate detection spiking neuron 212.

[0110] Figure 9 is a timing diagram showing the operation of the control SNN capable of detecting the input rate in the time direction and the spatial direction of an event according to the first embodiment. Note that in the drawings, the operation of the control SNN 153 of three pixels 110-1 to 110-3 having different spatial positions is taken as an example.

[0111] In the drawings, the neuron threshold NTH is set for the high-rate detection spiking neuron 221, and the neuron threshold NTL is set for the low-rate detection spiking neuron 222. In addition, in the high-rate detection spiking neuron 221, when a pulse is input via the positive connection 231, the neuron membrane potential NVH rises, and when a pulse is input via the negative connection 232, the neuron membrane potential NVH drops. In addition, in the high-rate detection spiking neuron 221, when no pulse is input, the neuron membrane potential NVH gradually decreases. In the low-rate detection spiking neuron 222, when a pulse is input via the positive connection 231, the neuron membrane potential NVL drops, and when a pulse is input via the negative connection 232, the neuron membrane potential NVL rises. In addition, in the low-rate detection spiking neuron 222, the neuron membrane potential NVL gradually decreases when no pulse is input.

[0112] Then, in the high light amount period KH, in each of pixels 110-1 to 110-3, based on the photon incidence to SPAD 122, pulse PL is input to counter 136. Whenever pulse PL is input to counter 136, count value CNT is accumulated and input to comparator 137. Then, in comparator 137, when count value CNT reaches counter threshold CTH, event IVE is output from comparator 137 to signal line 141, and counter 136 is reset.

[0113] The event IVE output of signal line 141 is input to high rate detection spiking neuron 211 and low rate detection spiking neuron 212 for each of pixels 110-1 to 110-3.

[0114] Here, in the high light amount period KH, it is assumed that high light amount is incident on pixels 110-1 and 110-2, and low light amount is incident on pixel 110-3. At this time, in high rate detection spiking neuron 221, when high rate detection spikes SPH1 and SPH2 are output from pixels 110-1 and 110-2, neuron membrane potential NVH rises (times t20 and t21). In addition, in high rate detection spiking neuron 221, when low rate detection spike SPL3 is output from pixel 110-3, neuron membrane potential NVH drops (time t22). Then, in high rate detection spiking neuron 221, when high rate detection spikes SPH1 and SPH2 are output from pixels 110-1 and 110-2 at a high rate, neuron membrane potential NVH reaches neuron threshold NTH (times t23 and t24). Then, when neuron membrane potential NVH reaches neuron threshold NTH, high rate detection spiking neuron 221 generates high rate detection spike SPH. At this time, high rate detection spiking neuron 221 can output high rate detection spike SPH as upward signal SU to threshold register 154.

[0115] When upward signal SU is input, threshold register 154 increases counter threshold CTH and outputs it to each of pixels 110-1 to 110-3. In each of pixels 110-1 to 110-3, when counter threshold CTH increases, the output rate of event IVE decreases, and saturation of event IVE in the high light amount period KH can be suppressed.

[0116] On the other hand, during the low light amount period KL, in each of the pixels 110-1 to 110-3, a pulse PL is input to the counter 136 based on the incidence of photons on the SPAD 122. Whenever the pulse PL is input to the counter 136, the count value CNT is accumulated and input to the comparator 137. Then, in the comparator 137, when the count value CNT reaches the counter threshold CTH, an event IVE is output from the comparator 137 to the signal line 141, and the counter 136 is reset.

[0117] The event IVE output of the signal line 141 is input to the high-rate detection spiking neuron 211 and the low-rate detection spiking neuron 212 for each of the pixels 110-1 to 110-3.

[0118] Here, during the low light amount period KL, low light amounts are incident on the pixels 110-1 and 110-2, and a high light amount is incident on the pixel 110-3. At this time, in the low-rate detection spiking neuron 222, when the low-rate detection spikes SPL1 and SPL2 are output from the pixels 110-1 and 110-2, the neuron membrane potential NVL rises (times t25 and t26). In addition, in the low-rate detection spiking neuron 222, when the high-rate detection spike SPH3 is output from the pixel 110-3, the neuron membrane potential NVL drops (time t27). Then, in the low-rate detection spiking neuron 222, when the low-rate detection spikes SPL1 and SPL2 are output from the pixels 110-1 and 110-2 at a high rate, the neuron membrane potential NVL reaches the neuron threshold NTL (times t28 and t29). Then, when the neuron membrane potential NVL reaches the neuron threshold NTL, the low-rate detection spiking neuron 222 generates a low-rate detection spike SPL. At this time, the low-rate detection spiking neuron 222 can output the low-rate detection spike SPL as a downward signal SD to the threshold register 154.

[0119] When the downward signal SD is input, the threshold register 154 decreases the counter threshold CTH and outputs it to each of the pixels 110-1 to 110-3. In each of the pixels 110-1 to 110-3, when the counter threshold CTH decreases, the output rate of the event IVE increases, and the loss of the event IVE during the low light amount period KL can be suppressed.

[0120] As described above, in the above first embodiment, each pixel 110 controls the output rate of the event IVE based on the comparison result between the count value CNT generated by the counter 136 and the counter threshold CTH. Therefore, the solid-state imaging device 102 can suppress the saturation of the event output and the lack of the event output while suppressing the reduction in sensitivity.

[0121] In addition, in order to control the output rate of events based on the counter threshold CTH, the control SNN 153 is used to update the counter threshold CTH. Therefore, the input rate in the spatial direction of the event IVE and the input rate in the temporal direction of the event IVE can be controlled, while realizing the asynchronous input of the event IVE. To this end, the counter threshold CTH can be stabilized with respect to the temporal change and spatial change of the light quantity, and the power consumption and latency can be reduced compared with the method of synchronizing the input of the event IVE. For example, even in an illumination environment where there is flickering under fluorescent lamps and the surrounding environment during travel at the entrance and exit of a tunnel, the counter threshold CTH can be stabilized while achieving low power consumption and low latency.

[0122] <2. Second Embodiment>

[0123] In the above first embodiment, the output rate of the event IVE is controlled based on the comparison result between the count value CNT generated by the counter and the counter threshold CTH. In the second embodiment, the output of the event IVE in each row is arbitrated based on the detection result of the event IVE in each row.

[0124] Figure 10 is a block diagram showing an example of the configuration of a solid-state imaging device according to the second embodiment layer by layer.

[0125] In the drawings, the solid-state imaging device 200 includes a vertical arbiter 252 in the control unit 112 of the above first embodiment. Other configurations of the solid-state imaging device 200 of the second embodiment are similar to those of the solid-state imaging device 102 of the above first embodiment.

[0126] The vertical arbiter 252 arbitrates the output of the event IVE in each row based on the detection result of the event IVE in each row. The vertical arbiter 252 is connected to the circuit unit 131 of each row via a control line 241. At this time, the vertical arbiter 252 can be connected to the output of the latch circuit 138 of the circuit unit 131.

[0127] When the output of the event IVE in any row is detected, the vertical arbiter 252 outputs the row number NOR of the row to the line scanner 151. When the row number NOR is output from the vertical arbiter 252, the line scanner 151 can scan the row specified by the row number NOR and read the event IVE from that row.

[0128] As described above, in the above second embodiment, the vertical arbiter 252 discriminates the output of the event IVE in each row based on the detection result of the event IVE in each row. Therefore, the line scanner 151 can scan only the rows where the event IVE occurs. Therefore, the line scanner 151 does not need to scan the rows where the event IVE does not occur, and the effect of improving the output rate of the compressed event IVE can be achieved.

[0129] <3. Third Embodiment>

[0130] In the above-described second embodiment, the output of the event IVE in each row is arbitrated based on the detection result of the event IVE in each row. In the third embodiment, the output of the event IVE in each row and each column is arbitrated based on the detection result of the event IVE in each row and each column.

[0131] Figure 11 is a block diagram showing an example of the configuration of a solid-state imaging device according to the third embodiment layer by layer.

[0132] In the drawings, the solid-state imaging device 300 includes a horizontal arbiter 351 instead of the line scanner 151 of the above-described second embodiment. Other configurations of the solid-state imaging device 300 of the third embodiment are similar to those of the solid-state imaging device 200 of the above-described second embodiment.

[0133] The horizontal arbiter 351 discriminates the output of the event IVE in the column based on the detection result of the event IVE in each column. The horizontal arbiter 351 is connected to the circuit unit 131 of each column through the signal line 141. At this time, the horizontal arbiter 351 can be connected to the output of the latch circuit 138 of the circuit unit 131.

[0134] When the output of the event IVE in any row is detected, the vertical arbiter 252 outputs the row number NOR of the row to the horizontal arbiter 351. When the row number NOR is output from the vertical arbiter 252, the horizontal arbiter 351 can read the event IVE from the column in which the event IVE has occurred in the row specified by the row number NOR.

[0135] As described above, in the third embodiment, the vertical arbiter 252 arbitrates the output of the event IVE in the row based on the detection result of the event IVE for each row, and the horizontal arbiter 351 arbitrates the output of the event IVE in the column based on the detection result of the event IVE for each column. Therefore, the horizontal arbiter 351 can read the event IVE only from the pixel 110 in which the event IVE has occurred. For this reason, the horizontal arbiter 351 does not need to scan the pixel 110 in which the event IVE has not occurred, and the effect of increasing the output rate of the compressed event IVE can be achieved.

[0136] <4. Fourth Embodiment>

[0137] In the above-described first embodiment, the output rate of the event IVE is controlled based on the comparison result between the count value CNT generated by the counter and the counter threshold CTH. In the fourth embodiment, the negative power supply voltage VN of the SPAD 122 is controlled based on the comparison result between the count value CNT generated by the counter 136 and the counter threshold CTH.

[0138] Figure 12It is a block diagram showing an example of the configuration of a solid-state imaging device according to the fourth embodiment layer by layer.

[0139] In the drawings, in the solid-state imaging device 400, a voltage setting register 454 and a negative power supply 455 are added to the solid-state imaging device 102 of the first embodiment described above. In addition, the solid-state imaging device 400 includes a control SNN 453 instead of the control SNN 153 of the first embodiment described above. The other configurations of the solid-state imaging device 400 of the fourth embodiment are similar to the configurations of the solid-state imaging device 102 of the first embodiment described above.

[0140] The control SNN 453 updates the counter threshold CTH and the negative power supply voltage VN based on the input rate of the event IVE output from the circuit unit 131. At this time, the control SNN 453 can detect the input rate in the spatial direction of the event IVE and the input rate in the temporal direction of the event IVE. The control SNN 453 can update the counter threshold CTH and the negative power supply voltage VN step by step, at a constant rate, or in a mixed manner (both step by step and at a constant rate). At this time, the control SNN 453 can generate an upward signal SU for increasing the counter threshold CTH and a downward signal SD for decreasing the counter threshold CTH. In addition, the control SNN 453 can generate an upward signal EU for increasing the negative power supply voltage VN and a downward signal ED for decreasing the negative power supply voltage VN.

[0141] The control SNN 453 can be configured to be similar to the control SNN 153 of the first embodiment described above. At this time, in addition to the high-rate detection spiking neurons and low-rate detection spiking neurons for updating the counter threshold CTH, high-rate detection spiking neurons and low-rate detection spiking neurons for updating the negative power supply voltage VN can be provided in the second layer of the control SNN 453.

[0142] The voltage setting register 454 stores the set value of the negative power supply voltage VN and outputs the set value to the negative power supply 455. The voltage setting register 454 can update the set value of the negative power supply voltage VN based on the upward signal EU and the downward signal ED from the control SNN 453.

[0143] The negative power supply 455 raises and lowers the negative power supply voltage VN based on the upward signal EU and the downward signal ED output from the voltage setting register 454, and supplies the negative power supply voltage VN to the SPAD 122.

[0144] As described above, in the fourth embodiment described above, the control SNN 453 controls the negative power supply voltage VN of the SPAD 122 based on the input rate of the event IVE. Therefore, the solid-state imaging device 400 can adjust the sensitivity of the light receiving unit 121 according to the amount of light received by the light receiving unit 121.

[0145] In addition, in order to control the negative power supply voltage VN of the SPAD 122 based on the output rate of the event IVE, the control SNN 453 is used to update the negative power supply voltage VN. Therefore, the input rate in the spatial direction of the event IVE and the input rate in the temporal direction of the event IVE can be controlled, and asynchronous input of the event IVE can be achieved simultaneously. Therefore, the negative power supply voltage VN can be stabilized with respect to the temporal change and spatial change of the light quantity, and low power consumption and low latency can be achieved compared with the method of synchronizing the input of the event IVE.

[0146] <5. Fifth Embodiment>

[0147] In the above-described first embodiment, the control SNN 153 generates an upward signal SU for increasing the counter threshold CTH and a downward signal SD for decreasing the counter threshold CTH. In the fifth embodiment, the control SNN generates an upward signal SU1 and a downward signal SD1. The upward signal SU1 increases the counter threshold CTH by one step, and the downward signal SD1 decreases the counter threshold CTH by one level.

[0148] Figure 13 It is a block diagram showing a configuration example of the control SNN according to the fifth embodiment.

[0149] In the drawing, the control SNN 553 generates an upward signal SU1 for increasing the counter threshold CTH by one level and a downward signal SD1 for decreasing the counter threshold CTH by one level.

[0150] The threshold register 554 stores the counter threshold CTH for the count value CNT of the pulse output from the light receiving unit 121 and outputs the counter threshold CTH to the circuit unit 131. The threshold register 554 can update the counter threshold CTH based on the upward signal SU1 and the downward signal SD1 from the control SNN 553.

[0151] The control SNN 553 and the threshold register 554 can be applied to any of the solid-state imaging devices of the above-described first to fourth embodiments.

[0152] As described above, in the above-described fifth embodiment, the control SNN 553 increases or decreases the counter threshold CTH by one level. Therefore, the input rate in the spatial direction of the event IVE and the input rate in the temporal direction of the event IVE can be controlled while suppressing the expansion of the configuration of the control SNN 553, and the event IVE can be asynchronously input.

[0153] <6. Sixth Embodiment>

[0154] In the above fifth embodiment, the control SNN 553 increases or decreases the counter threshold CTH by one level. In the sixth embodiment, the control SNN increases or decreases the counter threshold CTH in two stages.

[0155] Figure 14 FIG. is a block diagram showing a configuration example of a control SNN according to the sixth embodiment.

[0156] In the drawing, the control SNN 653 generates an upward signal SU1 and SU2 for increasing the counter threshold CTH by two levels and a downward signal SD1 and SD2 for decreasing the counter threshold CTH by two levels.

[0157] The threshold register 654 stores the counter threshold CTH for the count value CNT of the pulses output from the optical reception unit 121 and outputs the counter threshold CTH to the circuit unit 131. The threshold register 654 can update the counter threshold CTH based on the upward signals SU1 and SU2 and the downward signals SD1 and SD2 from the control SNN 653.

[0158] The control SNN 653 and the threshold register 654 can be applied to any solid-state imaging device of the first to fourth embodiments described above.

[0159] Figure 15 FIG. is a diagram showing a configuration example of a control SNN capable of detecting the input rate in the time direction and the spatial direction of an event according to the sixth embodiment.

[0160] In the drawing, in the control SNN 653, a high-rate detection spiking neuron 621 and a low-rate detection spiking neuron 622 are added to the second layer of the control SNN 153 of the first embodiment described above. The other structure of the control SNN 653 of the sixth embodiment is the same as the structure of the control SNN 153 of the first embodiment described above.

[0161] The neuron threshold NTH2 is set for the high-rate detection spiking neuron 621, and the neuron threshold NTL2 is set for the low-rate detection spiking neuron 622. The neuron thresholds NTH and NTH2 may be different from each other. The neuron thresholds NTL and NTL2 may be different from each other. Therefore, it is possible to make the reaction conditions of the high-rate detection spiking neurons 221, 621 and the low-rate detection spiking neurons 222, 622 different, so that the counter threshold CTH can be moved up and down by more than two levels.

[0162] A positive connection 231 is provided between the high-rate detection spiking neuron 211 and the high-rate detection spiking neuron 621. A negative connection 232 is provided between the high-rate detection spiking neuron 211 and the low-rate detection spiking neuron 622. A negative connection 232 is provided between the low-rate detection spiking neuron 212 and the high-rate detection spiking neuron 621. A positive connection 231 is provided between the low-rate detection spiking neuron 212 and the low-rate detection spiking neuron 622.

[0163] Based on the positive connection 231 with the high-rate detection spiking neuron 211 and the negative connection 232 with the low-rate detection spiking neuron 212, the high-rate detection spiking neuron 621 outputs a high-rate detection spike SPHB. Based on the negative connection 232 with the high-rate detection spiking neuron 211 and the positive connection 231 with the low-rate detection spiking neuron 212, the low-rate detection spiking neuron 622 outputs a low-rate detection spike SPLB. The high-rate detection spike SPH can be used as an upward signal SU1. The high-rate detection spike SPHB can be used as an upward signal SU2. The low-rate detection spike SPL can be used as a downward signal SD1. The low-rate detection spike SPLB can be used as a downward signal SD2.

[0164] As described above, in the above sixth embodiment, the control SNN 653 increases or decreases the counter threshold CTH by two levels. Therefore, the counter threshold CTH can be updated more finely while suppressing the expansion of the configuration of the control SNN 653, and the event rate of the event IVE can be controlled asynchronously.

[0165] <7. Seventh Embodiment>

[0166] In the above fifth embodiment, the control SNN 553 increases or decreases the counter threshold CTH by one level. In the seventh embodiment, the counter threshold CTH is increased or decreased based on the rate.

[0167] Figure 16 It is a block diagram showing a configuration example of the control SNN according to the seventh embodiment.

[0168] In the drawing, the control SNN 753 generates an upward signal PU1 that increases the counter threshold CTH at a constant rate and a downward signal PD1 that decreases the counter threshold CTH at a constant rate. The rate at which the counter threshold CTH increases or decreases can be, for example, 10% or 20%.

[0169] The threshold register 754 stores the counter threshold CTH for the count value CNT of the pulses output from the optical receiving unit 121 and outputs the counter threshold CTH to the circuit unit 131. The threshold register 754 can update the counter threshold CTH based on the upward signal PU1 and the downward signal PD1 from the control SNN 753.

[0170] The control SNN 753 and the threshold register 754 can be applied to any solid-state imaging device of the first to fourth embodiments described above.

[0171] As described above, in the seventh embodiment, the control SNN 753 increases or decreases the counter threshold CTH at a constant rate. As a result, it is possible to detect the input rate in the spatial direction of the event IVE and the input rate in the temporal direction of the event IVE while suppressing the expansion of the configuration of the control SNN 753, and it is possible to asynchronously control the event rate of the event IVE.

[0172] <8. Eighth Embodiment>

[0173] In the sixth embodiment described above, the control SNN 653 increases or decreases the counter threshold CTH by two levels. In the eighth embodiment, the control SNN 653 increases or decreases the counter threshold CTH based on the phase and rate.

[0174] Figure 17 FIG. is a block diagram showing a configuration example of the control SNN according to the eighth embodiment.

[0175] In the drawing, the control SNN 853 generates an up signal SU1 and an up signal PU1. The up signal SU1 increases the counter threshold CTH by one level, and the up signal PU1 increases the counter threshold CTH at a constant rate. In addition, the control SNN 853 generates a down signal SD1 that decreases the counter threshold CTH by one level and a down signal PD1 that decreases the counter threshold CTH at a constant rate.

[0176] The threshold register 854 stores the counter threshold CTH for the count value CNT of the pulse output from the light receiving unit 121 and outputs the counter threshold CTH to the circuit unit 131. The threshold register 854 can update the counter threshold CTH based on the up signals SU1 and PU1 and the down signals SD1 and PD1 from the control SNN 853.

[0177] The control SNN 853 and the threshold register 854 can be applied to any solid-state imaging device of the first to fourth embodiments described above.

[0178] As described above, in the eighth embodiment, the control SNN 853 increases or decreases the counter threshold CTH based on the phase and rate. Therefore, it is possible to more finely update the counter threshold CTH while suppressing the expansion of the configuration of the control SNN 853, and it is possible to asynchronously control the event rate of the event IVE.

[0179] <9. Ninth Embodiment>

[0180] Figure 18 This is a block diagram showing a configuration example of a distance measuring device according to the ninth embodiment.

[0181] In the drawing, the distance measurement device 1000 captures a distance image based on, for example, time of flight (ToF). A distance image can be generated from the distance pixel signals based on the distance to each pixel in the depth direction from the distance measurement device 1000 to the object 1001.

[0182] The distance measurement device 1000 includes a light emitting device 1100 and an imaging device 1200. The light emitting device 1100 includes a light emission control unit 1101 and a light emission unit 1102.

[0183] The light emission control unit 1101 controls the light irradiation pattern of the light emission unit 1102 according to the control of the control unit 1202. The light emission unit 1102 emits light within a predetermined wavelength range under the control of the light emission control unit 1101. The predetermined wavelength range can be the infrared range. The light emission unit 1102 can be a laser diode or a light emitting diode.

[0184] The imaging device 1200 receives the reflected light obtained by reflecting the light emitted from the light emitting device 1100 by the object 1001 for each pixel, and generates a distance image. The imaging device 1200 includes an imaging unit 1201, a control unit 1202, a storage unit 1203, and a display unit 1204. The imaging unit 1201 includes an optical system 1211, a light receiving unit 1221, and a signal processing unit 1231.

[0185] The optical system 1211 forms an image of the incident light on the light receiving surface of the light receiving unit 1221. Note that the optical system 1211 can include a lens, a filter, an aperture, etc.

[0186] The light receiving unit 1221 receives the reflected light reflected by the object 1001. The light receiving unit 1221 can be a SPAD or a photodiode. Under the control of the control unit 1202, the light receiving unit 1221 receives the reflected light from the object 1001, and provides the thus obtained pixel signal to the signal processing unit 1231. This pixel signal represents a digital count value obtained by counting the time from when the irradiation light is emitted from the light emitting device 1100 to when the irradiation light is received by the light receiving unit 1221. A light emission timing signal indicating the timing of the light emission of the light emission unit 1102 is also provided from the control unit 1202 to the light receiving unit 1221. The imaging unit 1201 can include any one of the solid-state imaging devices of the first to fourth embodiments described above.

[0187] The signal processing unit 1231 processes the pixel signals provided by the light receiving unit 1221 under the control of the control unit 1202. For example, the signal processing unit 1231 detects the distance from each pixel to the object based on the pixel signals provided by the light receiving unit 1221, and generates a distance image indicating the distance from each pixel to the object. For example, the signal processing unit 1231 obtains the time when each pixel of the light receiving unit 1221 receives light multiple times for each pixel from the time when the light emitting unit 1102 emits light. The signal processing unit 1231 creates a histogram corresponding to the obtained time. Then, by detecting the peak of the histogram, the signal processing unit 1231 determines the time until the light emitted from the light emitting unit 1102 is reflected by the object 1001 and returns. In addition, the signal processing unit 1231 performs calculations based on the determined time and the speed of light to obtain the distance to the object. The signal processing unit 1231 provides the generated distance image to the control unit 1202.

[0188] The control unit 1202 controls the light emission control unit 1101 and the light receiving unit 1221. For example, the control unit 1202 provides an irradiation signal to the light emission control unit 1101 and provides a light emission timing signal to the light receiving unit 1221. The light emitting unit 1102 emits irradiation light according to the irradiation signal. The light emission timing signal may be the irradiation signal provided to the light emission control unit 1101. In addition, the control unit 1202 provides the distance image obtained from the imaging unit 1201 to the display unit 1204 and causes the display unit 1204 to display the distance image. In addition, the control unit 1202 stores the distance image obtained from the imaging unit 1201 in the storage unit 1203. The control unit 1202 may include a processor such as a central processing unit (CPU) or a graphics processing unit (GPU). In addition, the control unit 1202 may include a hardware circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0189] The display unit 1204 displays the distance image, the user interface screen, etc. The display unit 1204 may be a liquid crystal display device or an organic EL display device. The storage unit 1203 stores the distance image, the setting information for distance measurement, etc. The storage unit 1203 may include a semiconductor memory such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), or may include a storage device such as a hard disk device or a solid state drive (SSD).

[0190] As described above, in the above ninth embodiment, the imaging unit 1201 generates pixel signals based on events in which the output rate of the pulses output from the light receiving unit 1221 is compressed. Therefore, it is possible to suppress saturation of event output and insufficiency of event output while suppressing a decrease in sensitivity, and it is possible to suppress a decrease in ranging accuracy caused by changes in the surrounding environment.

[0191] <10. Application Example of a Moving Body>

[0192] The technology according to the present disclosure (this technology) can be applied to various products. For example, the technology according to an embodiment of the present disclosure can also be implemented as a device installed on any type of mobile body (such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, and a robot).

[0193] Figure 19 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile body control system to which the technology according to the present disclosure can be applied.

[0194] The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. In Figure 19 the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside vehicle information detection unit 12030, an inside vehicle information detection unit 12040, and an integrated control unit 12050. In addition, as a functional structure of the integrated control unit 12050, a microcomputer 12051, a sound / image output unit 12052, and a vehicle-mounted network interface (I / F) 12053 are illustrated.

[0195] The drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 serves as a control device for a driving force generation device (such as an internal combustion engine, a drive motor, etc.) for generating the driving force of the vehicle, a driving force transmission mechanism for transmitting the driving force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, a braking device for generating the braking force of the vehicle, etc.

[0196] The body system control unit 12020 controls the operation of various devices provided on the vehicle body according to various programs. For example, the body system control unit 12020 serves as a control device for a keyless entry system, a smart key system, an electric window device, or various lights such as a headlight, a rear backup light, a brake light, a turn signal, a fog light, etc. In this case, radio waves or signals of various switches sent from a mobile device as an alternative to a button can be input to the body system control unit 12020. The body system control unit 12020 receives these input radio waves or signals and controls the door lock device, the electric window device, the lights, etc. of the vehicle.

[0197] The vehicle exterior information detection unit 12030 detects information outside the vehicle including the vehicle control system 12000. For example, an imaging unit 12031 is connected to the vehicle exterior information detection unit 12030. The vehicle exterior information detection unit 12030 causes the imaging unit 12031 to capture an image of the outside of the vehicle and receives the captured image. In addition, the vehicle exterior information detection unit 12030 can also perform processing for detecting objects such as people, vehicles, obstacles, signs, and characters on the road surface, or processing for detecting their distances, based on the received image.

[0198] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of the received light. The imaging unit 12031 can output the electrical signal as an image, or can output the electrical signal as information regarding the measured distance. In addition, the light received by the imaging unit 12031 can be visible light, or can be invisible light such as infrared light.

[0199] The vehicle interior information detection unit 12040 detects information regarding the interior of the vehicle. The vehicle interior information detection unit 12040 is connected, for example, to a driver state detection unit 12041 that detects the state of the driver. The driver state detection unit 12041 includes, for example, a camera that captures the driver. Based on the detection information input from the driver state detection unit 12041, the vehicle interior information detection unit 12040 can calculate the driver's fatigue level or the driver's concentration, or can determine whether the driver is dozing off.

[0200] The microcomputer 12051 can calculate control target values for a driving force generation device, a steering mechanism, or a braking device based on information regarding the interior or exterior of the vehicle obtained by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can execute cooperative control for implementing functions of an advanced driver assistance system (ADAS), which includes anti-collision or shock absorption for the vehicle, following driving based on a following distance, maintaining the vehicle speed of driving, warning of a vehicle collision, warning of deviation of the vehicle from a lane, and the like.

[0201] In addition, the microcomputer 12051 can execute cooperative control for autonomous driving by controlling a driving force generation device, a steering mechanism, a braking device, etc. based on information regarding the exterior or interior of the vehicle obtained by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, which enables the vehicle to automatically drive without relying on the driver's operations, etc.

[0202] In addition, the microcomputer 12051 can output a control command to the vehicle body system control unit 12020 based on the information about the outside of the vehicle obtained by the outside vehicle information detection unit 12030. For example, the microcomputer 12051 can perform cooperative control aimed at preventing glare by controlling the headlamp to change from high beam to low beam according to the position of the preceding vehicle or oncoming vehicle detected by the outside vehicle information detection unit 12030.

[0203] The sound / image output unit 12052 sends an output signal of at least one of sound and image to an output device that can visually or auditorily notify information to the vehicle occupants or outside the vehicle. In Figure 19 the example, the audio speaker 12061, the display unit 12062, and the instrument panel 12063 are shown as output devices. For example, the display unit 12062 may include at least one of an on-board display and a head-up display.

[0204] Figure 20 is a diagram illustrating an example of the installation position of the imaging unit 12031.

[0205] In Figure 20 this, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0206] The imaging units 12101, 12102, 12103, 12104, 12105 are provided, for example, at positions such as the front nose, side mirrors, rear bumper, rear door, and upper part of the windshield inside the vehicle 12100. The imaging unit 12101 provided at the front nose inside the vehicle and the imaging unit 12105 provided at the upper part of the windshield mainly obtain images in front of the vehicle 12100. The imaging units 12102 and 12103 provided to the side mirrors mainly obtain images on the sides of the vehicle 12100. The imaging unit 12104 provided to the rear bumper or rear door mainly obtains images at the rear of the vehicle 12100. The imaging unit 12105 provided at the upper part of the windshield inside the vehicle is mainly used to detect preceding vehicles, pedestrians, obstacles, signals, traffic signs, lanes, etc.

[0207] Note that Figure 20An example of the imaging ranges of imaging units 12101 to 12104 is shown. Imaging range 12111 represents the imaging range of imaging unit 12101 provided at the front nose. Imaging ranges 12112 and 12113 respectively represent the imaging ranges of imaging units 12102 and 12103 provided at the side mirrors. Imaging range 12114 represents the imaging range of imaging unit 12104 provided at the rear bumper or the rear door. For example, a bird's-eye view image of vehicle 12100 viewed from above is obtained by superimposing the image data imaged by imaging units 12101 to 12104.

[0208] At least one of imaging units 12101 to 12104 may have a function of obtaining distance information. For example, at least one of imaging units 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0209] For example, the microcomputer 12051 can determine the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the time change of the distance (relative speed with respect to vehicle 12100) based on the distance information obtained from imaging units 12101 to 12104. Thus, it extracts objects present on the driving path of vehicle 12100 moving in substantially the same direction as vehicle 12100 at a prescribed speed (e.g., equal to or greater than 0 km / h). In addition, the microcomputer 12051 can preset a following distance to maintain in front of a preceding vehicle and perform automatic braking control (including following stop control), automatic acceleration control (including following start control), etc. Thereby, it is possible to perform coordinated control for autonomous driving that enables the vehicle to automatically drive without depending on the driver's operation, etc.

[0210] For example, the microcomputer 12051 can classify three-dimensional object data related to three-dimensional objects into three-dimensional object data of two-wheeled vehicles, standard vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on the distance information obtained from imaging units 12101 to 12104, extract the classified three-dimensional object data, and use the extracted three-dimensional object data for automatically avoiding obstacles. For example, the microcomputer 12051 identifies obstacles around vehicle 12100 as obstacles that can be visually recognized by the driver of vehicle 12100 and obstacles that are difficult for the driver of vehicle 12100 to visually recognize. Then, the microcomputer 12051 determines a collision risk indicating the risk of collision with each obstacle. When the collision risk is equal to or higher than a set value and thus there is a possibility of collision, the microcomputer 12051 outputs a warning to the driver via the audio speaker 12061 or the display unit 12062, and performs forced deceleration or avoidance steering via the driving system control unit 12010. The microcomputer 12051 can thereby assist driving to avoid collisions.

[0211] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. The microcomputer 12051 can, for example, identify a pedestrian by determining whether a pedestrian exists in the captured images of the imaging units 12101 to 12104. Such identification of a pedestrian is performed, for example, through a process of extracting feature points in the captured images of the imaging units 12101 to 12104 that are infrared cameras and through a process of performing pattern matching processing on a series of feature points representing the outline of an object to determine whether it is a pedestrian. When the microcomputer 12051 determines that a pedestrian exists in the captured images of the imaging units 12101 to 12104 and thus identifies the pedestrian, the sound / image output unit 12052 controls the display unit 12062 such that a square contour line for emphasis is displayed superimposed on the identified pedestrian. The sound / image output unit 12052 may also control the display unit 12062 such that an icon or the like representing the pedestrian is displayed at a desired position.

[0212] Examples of a vehicle control system to which the technology according to the present disclosure can be applied have been described above. The technology of the present disclosure can be applied to the imaging unit 12031 among the above configurations. Specifically, for example, the above solid-state imaging device can be applied to the imaging unit 12031. By applying the technology according to the present disclosure to the vehicle control system 12000, saturation of the imaging output and lack of the imaging output can be suppressed, while a decrease in the sensitivity of the imaging unit 12031 is suppressed.

[0213] It should be noted that the above embodiments show examples for embodying the present technology, and there is a corresponding relationship between the matters in the embodiments and the matters specifying the present invention in the claims. Similarly, there is a corresponding relationship between the matters specifying the present invention in the claims and the matters having the same name in the embodiments of the present technology. However, the present technology is not limited to the embodiments, and various modifications can be applied to the embodiments without departing from the gist of the present technology. In addition, the effects described in this specification are merely examples and are not limited, and other effects can be provided.

[0214] It should be noted that the present technology may also have the following configuration.

[0215] (1) An imaging device, comprising:

[0216] A light receiving unit, provided in pixels arranged in a matrix along a row direction and a column direction, and outputting a pulse based on the incidence of photons;

[0217] A counter, provided in the pixel and counting the pulses output from the light receiving unit; and

[0218] A comparator, provided in the pixel and outputting an event based on a comparison result between the count value of the counter and a counter threshold value.

[0219] (2) The imaging device according to (1) above, wherein when the count value exceeds the counter threshold, the comparator outputs an event.

[0220] (3) The imaging device according to (1) or (2) above, wherein

[0221] When the count value exceeds the counter threshold, the comparator resets the counter.

[0222] (4) The imaging device according to any one of (1) to (3) above, wherein

[0223] The counter and the comparator are arranged below the light receiving unit.

[0224] (5) The imaging device according to any one of (1) to (4) above, wherein

[0225] The light receiving unit includes a single photon avalanche diode (SPAD).

[0226] (6) The imaging device according to any one of (1) to (5) above, further comprising:

[0227] A control unit that controls the counter threshold based on the output rate of the event.

[0228] (7) The imaging device according to (6) above, wherein

[0229] The control unit includes a spiking neural network that controls the counter threshold based on the input of the event.

[0230] (8) The imaging device according to (7) above, wherein

[0231] The spiking neural network includes:

[0232] A plurality of first spiking neurons, each of which receives the input of the event and emits based on the input rate of the event; and

[0233] A second spiking neuron, the emissions of the plurality of first spiking neurons are respectively input to the second spiking neuron, and the second spiking neuron emits based on the input rate of the emissions of the first spiking neurons.

[0234] (9) The imaging device according to (7) or (8) above, wherein

[0235] The spiking neural network

[0236] Is capable of detecting the input rate in the spatial direction of the event and the input rate in the temporal direction of the event.

[0237] (10) The imaging device according to (8) above, wherein

[0238] Each of the plurality of first spiking neurons includes:

[0239] A high-rate detection first spiking neuron that fires based on an increase in the input rate of the event; and

[0240] A low-rate detection first spiking neuron that fires based on a decrease in the input rate of the event,

[0241] The second spiking neuron includes:

[0242] A high-rate detection second spiking neuron that is connected such that the neuron membrane potential rises based on the firing input from the high-rate detection first spiking neuron and the neuron membrane potential falls based on the firing input from the low-rate detection first spiking neuron; and

[0243] A low-rate detection second spiking neuron that is connected such that the neuron membrane potential falls based on the firing input from the high-rate detection first spiking neuron and the neuron membrane potential rises based on the firing input from the low-rate detection first spiking neuron, and

[0244] The second spiking neuron includes:

[0245] A first rate detection spiking neuron connected to the first spiking neuron such that the neuron membrane potential rises based on the input of the event; and

[0246] A second rate detection spiking neuron connected to the first spiking neuron such that the neuron membrane potential falls based on the input of the event.

[0247] (11) The imaging device according to any one of (7) to (10) above, wherein

[0248] The spiking neural network can control the counter threshold at multiple levels.

[0249] (12) The imaging device according to any one of (7) to (11) above, wherein

[0250] The spiking neural network can control the counter threshold at a constant rate.

[0251] (13) The imaging device according to any one of (6) to (12) above, wherein

[0252] The control unit controls the negative power supply voltage of the light receiving unit based on the output rate of the event.

[0253] (14) The imaging device according to any one of (1) to (13) above further includes:

[0254] A vertical arbiter that arbitrates the output of the events in the row based on the detection results of the events in each row.

[0255] (15) The imaging device according to any one of (1) to (14) above further includes:

[0256] A horizontal arbiter that arbitrates the output of the events in the column based on the detection results of the events in each column.

[0257] (16) A control device includes:

[0258] A control unit that receives, as an input of an event, a comparison result between a count value of pulses based on incident outputs of photons and a counter threshold, and controls the counter threshold based on the output rate of the event.

[0259] (17) In the control device according to (16) above,

[0260] the control unit includes a spiking neural network that controls the counter threshold based on the input of the event.

[0261] (18) A spiking neural network includes:

[0262] A plurality of first spiking neurons, each of which receives an input of pulses generated based on the incidence of photons at different spatial positions from each other, and emits based on the input rate of the pulses; and

[0263] A second spiking neuron, the emissions of the plurality of first spiking neurons are respectively input to the second spiking neuron, and the second spiking neuron emits based on the input rate of the emissions of the first spiking neurons.

[0264] (19) In the spiking neural network according to (18) above,

[0265] It is capable of detecting the input rate in the spatial direction of the pulses and the input rate in the temporal direction of the pulses.

[0266] (20) In the spiking neural network according to (18) or (19) above,

[0267] Each of the plurality of first spiking neurons includes:

[0268] A high-rate detection first spiking neuron that emits based on an increase in the input rate of the pulses; and

[0269] A low-rate detection first spiking neuron that fires based on a decrease in the input rate of the pulses, and

[0270] The second spiking neuron includes:

[0271] A high-rate detection second spiking neuron that is connected such that the neuron membrane potential rises based on a firing input from the high-rate detection first spiking neuron and the neuron membrane potential falls based on a firing input from the low-rate detection first spiking neuron; and

[0272] A low-rate detection second spiking neuron that is connected such that the neuron membrane potential falls based on a firing input from the high-rate detection first spiking neuron and the neuron membrane potential rises based on a firing input from the low-rate detection first spiking neuron.

[0273] List of reference symbols

[0274] 100 Imaging device

[0275] 101 Optical system

[0276] 102 Solid-state imaging device

[0277] 103 Imaging control unit

[0278] 104 Image processing unit

[0279] 105 Storage unit

[0280] 106 Display unit

[0281] 107 Operation unit

[0282] 108 Bus

[0283] 110 Pixel

[0284] 111 Pixel array unit

[0285] 112 Control unit

[0286] 113 Signal processing unit

[0287] 120 Light receiving array unit

[0288] 121 Light receiving unit

[0289] 130 Circuit array unit

[0290] 131 Circuit unit

[0291] 141 Signal line

[0292] 142 Control line

[0293] 151 Line Scanner

[0294] 152 Main Processor

[0295] 153 Control SNN

[0296] 154 Threshold Register

[0297] 122 SPAD

[0298] 132 Quenching Resistor

[0299] 133 P-channel Transistor

[0300] 134 N-channel Transistor

[0301] 135 Inverter

[0302] 136 Counter

[0303] 137 Comparator

[0304] 138 Latch Circuit

[0305] 129 Upper Chip

[0306] 139 Lower Chip

[0307] 229, 239 Pad Electrodes

Claims

1. An imaging device, comprising: A light receiving portion, disposed in pixels arranged in a matrix along a row direction and a column direction, and outputting pulses based on the incidence of photons; A counter, disposed in the pixels and counting the pulses output from the light receiving portion; And A comparator, disposed in the pixels and outputting an event based on a comparison result between a count value of the counter and a counter threshold.

2. The imaging device according to claim 1, wherein, When the count value exceeds the counter threshold, the comparator outputs the event.

3. The imaging device according to claim 1, wherein When the count value exceeds the counter threshold, the comparator resets the counter.

4. The imaging device according to claim 1, wherein The counter and the comparator are disposed below the light receiving portion.

5. The imaging device according to claim 1, wherein The light receiving portion includes a single photon avalanche diode (SPAD).

6. The imaging device according to claim 1, further comprising A control portion, controlling the counter threshold based on an output rate of the event.

7. The imaging device according to claim 6, wherein The control portion includes a spiking neural network that controls the counter threshold based on an input of the event.

8. The imaging device according to claim 7, wherein The spiking neural network includes: A plurality of first spiking neurons, each first spiking neuron receiving an input of the event and emitting based on an input rate of the event; and A second spiking neuron, inputs of emissions of the plurality of first spiking neurons are respectively input to the second spiking neuron, and the second spiking neuron emits based on an input rate of the emissions of the first spiking neurons.

9. The imaging device according to claim 7, wherein The spiking neural network Is capable of detecting an input rate in a spatial direction of the event and an input rate in a temporal direction of the event.

10. The imaging device according to claim 8, wherein Each of the plurality of first spiking neurons includes: A high-rate detection first spiking neuron that emits based on an increase in the input rate of the event; and A low-rate detection first spiking neuron that emits based on a decrease in the input rate of the event, and The second spiking neuron includes: A high-rate detection second spiking neuron that is connected such that a neuron membrane potential rises based on an emission input from the high-rate detection first spiking neuron and the neuron membrane potential drops based on an emission input from the low-rate detection first spiking neuron; and A low-rate detection second spiking neuron that is connected such that a neuron membrane potential drops based on an emission input from the high-rate detection first spiking neuron and the neuron membrane potential rises based on an emission input from the low-rate detection first spiking neuron.

11. The imaging device according to claim 7, wherein The spiking neural network is capable of controlling the counter threshold at multiple levels.

12. The imaging device according to claim 7, wherein The spiking neural network can control the counter threshold at a constant rate.

13. The imaging device according to claim 6, wherein the control unit controls the negative power supply voltage of the light receiving unit based on the output rate of the event.

14. The imaging device according to claim 1, further comprising a vertical arbiter that arbitrates the output of the events in the row based on the detection results of the events in each row.

15. The imaging device according to claim 1, further comprising a horizontal arbiter that arbitrates the output of the events in the column based on the detection results of the events in each column.

16. A control device, comprising: a control unit that receives, as an input of an event, a comparison result between a count value of pulses based on incident outputs of photons and a counter threshold, and controls the counter threshold based on the output rate of the event.

17. The control device according to claim 16, wherein the control unit includes a spiking neural network that controls the counter threshold based on the input of the event.

18. A spiking neural network, comprising: a plurality of first spiking neurons, each first spiking neuron receiving an input of pulses generated based on the incidence of photons at different spatial positions, and emitting based on the input rate of the pulses; and a second spiking neuron, the emissions of the plurality of first spiking neurons being respectively input to the second spiking neuron, and the second spiking neuron emitting based on the input rate of the emissions of the first spiking neurons.

19. The spiking neural network according to claim 18, wherein it is possible to detect the input rate in the spatial direction of the pulse and the input rate in the temporal direction of the pulse.

20. The spiking neural network according to claim 18, wherein each of the plurality of first spiking neurons includes: a high-rate detection first spiking neuron that emits based on an increase in the input rate of the pulse; and a low-rate detection first spiking neuron that emits based on a decrease in the input rate of the pulse, and the second spiking neuron includes: a high-rate detection second spiking neuron that is connected such that the neuron membrane potential rises based on the emission input from the high-rate detection first spiking neuron and the neuron membrane potential falls based on the emission input from the low-rate detection first spiking neuron; and a low-rate detection second spiking neuron that is connected such that the neuron membrane potential falls based on the emission input from the high-rate detection first spiking neuron and the neuron membrane potential rises based on the emission input from the low-rate detection first spiking neuron.

Images