Device and method

Abstract

A device, wherein the device includes: a single-photon avalanche diode sensor including a plurality of single-photon avalanche diode pixels, each being configured to generate and output light detection events to a neuromorphic circuit; and the neuromorphic circuit implementing a plurality of spiking neurons, wherein the neuromorphic circuit is configured to process the light detection events using the plurality of spiking neurons.

Description

[0001] DEVICE AND METHOD

[0002] TECHNICAL FIELD

[0003] The present disclosure generally pertains to a device and a method.

[0004] TECHNICAL BACKGROUND

[0005] Generally, single-photon avalanche diode (“SPAD”) sensors are known, which are typically used in active light sensing devices such as LiDAR (“Light Detection And Ranging”) devices.

[0006] Recently, such SPAD sensors have been used in passive imaging devices, where passive SPAD sensors may provide useful characteristics to the imaging device.

[0007] Unlike conventional Complementary Metal-Oxide-Semiconductor (“CMOS”) image sensors, which integrate incident light over time, SPAD sensors operate by generating light detection events by photoelectric conversion on incident light to generate an avalanche current and outputting the light detection events for counting the number of generated light detection events.

[0008] Due to their high sensitivity, the SPAD sensor may be able to capture images of scenes at low- light conditions as it is typically sensitive to single-photon detections.

[0009] Furthermore, the circuity of the SPAD sensor typically operates in the digital domain, thereby avoiding noise which commonly arises due to analog-to-digital conversion (“ADC”) operations.

[0010] However, their extreme sensitivity results in a large data rate which may be, in some cases, difficult to handle, for example, in some mobile applications. Moreover, in dark scenes the SPAD sensor may sometimes not be able to output a sufficient amount of data for image reconstruction.

[0011] Although there exist techniques for passive SPAD sensor-based imaging, it is generally desirable to improve the existing techniques.

[0012] SUMMARY

[0013] According to a first aspect, the disclosure provides a device, comprising: a single-photon avalanche diode sensor including a plurality of single-photon avalanche diode pixels, each being configured to generate and output light detection events to a neuromorphic circuit; and the neuromorphic circuit implementing a plurality of spiking neurons, wherein the neuromorphic circuit is configured to process the light detection events using the plurality of spiking neurons. According to a second aspect, the disclosure provides a method, comprising: generating and outputting light detection events to a neuromorphic circuit; and processing, by the neuromorphic circuit implementing a plurality of spiking neurons, the light detection events using the plurality of spiking neurons.

[0014] Further aspects are set forth in the dependent claims, the drawings and the following description.

[0015] BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Embodiments are explained by way of example with respect to the accompanying drawings, in which:

[0017] Fig. 1 schematically illustrates in a block diagram an embodiment of a device including a SPAD sensor;

[0018] Fig. 2 schematically illustrates an embodiment of a SPAD pixel;

[0019] Fig. 3 schematically illustrates an embodiment of a spiking neuron;

[0020] Fig. 4 schematically illustrates in a block diagram an embodiment of a device including a SPAD sensor and a neuromorphic circuit;

[0021] Fig. 5 schematically illustrates in a block diagram an embodiment of a device including a SPAD sensor and a neuromorphic circuit;

[0022] Fig. 6 schematically illustrates in a block diagram an embodiment of a processing of light detection events by a plurality of spiking neurons as a pre-processing method for image reconstruction methods;

[0023] Fig. 7 schematically illustrates in Fig. 7A an embodiment of generating an output rate of spikes that is lower than an input rate of light detection events, and in Fig. 7B an embodiment of generating an output rate of spikes that is higher than an input rate of light detection events;

[0024] Fig. 8 schematically illustrates an example of a relation between a compression ratio and a peak signal-to-noise ratio of a reconstructed image;

[0025] Fig. 9 schematically illustrates in a flow diagram an embodiment of a method;

[0026] Fig. 10 schematically illustrates in a flow diagram an embodiment of a method; and

[0027] Fig. 11 schematically illustrates in a flow diagram an embodiment of a method. DETAILED DESCRIPTION OF EMBODIMENTS

[0028] Before a detailed description of the embodiments under reference of Fig. 4 is given, general explanations are made.

[0029] As mentioned in the outset, recently, single-photon avalanche diode (“SPAD”) sensors have been used in passive imaging devices, where the passive SPAD sensors may provide useful characteristics to the imaging device.

[0030] The data that is generated by the SPAD sensor is different than data coming from other imaging sensors.

[0031] Compared to standard CMOS image sensors, the SPAD sensor encodes information in time. To reconstruct the scene, the 1 -bit data coming from the SPAD sensor is temporally accumulated.

[0032] Compared to an Event-Based Sensor (EVS), the data from the SPAD sensor is dense in space. While an EVS responds to changes in the incident light, the SPAD sensors may collect all incoming photons.

[0033] As further mentioned in the outset, due to their high sensitivity, the SPAD sensor may be able to capture images of scenes at low-light conditions as it is typically sensitive to single-photon detections.

[0034] Furthermore, the circuity of the SPAD sensor typically operates in the digital domain, thereby avoiding noise which commonly arises due to analog-to-digital conversion (“ADC”) operations.

[0035] However, their extreme sensitivity results in a large data rate which may be, in some cases, difficult to handle, for example, in some mobile applications.

[0036] Hence, the large amount of data that the SPAD sensor may generate may be challenging to handle in some use cases. For example, some SPAD sensors may be read out in the form of binary frames at about 100 kfps (“kilo frames per second”) such that a large amount of binary data is generated. In some cases, it may be power inefficient to get all this data from the SPAD sensor and may generate a communications bottleneck.

[0037] For enhancing the general understanding of the present disclosure, an embodiment of a device 1 which includes a SPAD sensor 2 is discussed in the following under reference of Fig. 1, which schematically illustrates the embodiment in a block diagram.

[0038] The device 1 may be, for instance, a mobile electronic device such as a smartphone or the like.

[0039] The device 1 includes the SPAD sensor 2, an input / output interface 4, a data bus 5 and an application processor 6. The SPAD sensor 2 includes a plurality of SPAD pixels 3 which are arranged in this embodiment as an array without limiting the disclosure in this regard.

[0040] Each SPAD pixel 3 generates and outputs light detection events when light is incident on the respective SPAD pixel 3 and the respective SPAD pixel 3 is activated.

[0041] For further enhancing the general understanding of the present disclosure, an embodiment of a SPAD pixel 3 is discussed in the following under reference of Fig. 2, which schematically illustrates the embodiment, and which may also apply to other embodiments of the present disclosure.

[0042] The SPAD pixel 3 includes a SPAD 10, a resistor 11, a switch 12 and a NOT gate 13.

[0043] The SPAD 10 is coupled with its anode to GND potential and with its cathode to the resistor 11 and the NOT gate 13.

[0044] The resistor 11 is coupled to the switch 12 and the switch 12 is further coupled to a bias voltage Vbias.

[0045] When the SPAD 10 is activated by closing the switch 12, the SPAD 10 becomes reverse-biased via the resistor 11 with the bias voltage Vbias which is above the breakdown voltage of the SPAD 10 to allow an avalanche current to be generated in the SPAD 10 when light is incident on the SPAD 10.

[0046] Once the avalanche current is generated in response to an incident photon, the cathodic voltage Vc of the SPAD 10 drops until it is below the breakdown voltage. The cathodic voltage Vc is quickly restored to the bias voltage Vbias such that the cathodic voltage Vc includes a short voltage pulse which is received by the NOT gate 13. The NOT gate 13 then outputs a light detection event LDE as a digital pulse representing 1 -bit of binary data.

[0047] Generally, the present disclosure is not limited to such embodiments of SPAD sensors and SPAD pixels and other embodiments may be apparent to the skilled person.

[0048] Returning to Fig. 1, each SPAD pixel 3 outputs its generated light detection events to an input / output interface 4 which is coupled to the data bus 5 (for the sake of illustration only some SPAD pixels 3 are shown without an explicit connection to the input / output interface 4).

[0049] The input / output interface 4 may, for example, include a 1 -bit memory for the output of each SPAD pixel 3. The 1 -bit memories may then be read out at a certain rate (frame rate).

[0050] The input / output interface 4 then outputs binary frames of light detection events, for instance, at a high frame rate of about 100 kfps via the data bus 5 to the application processor 6. The application processor 6 may perform various high-level functions such as image reconstruction, object tracking and / or image segmentation based on the binary frames of light detection events.

[0051] Returning to the general explanations, as mentioned above, typically, binary data is generated with a high data rate by a SPAD sensor which may lead to a large amount of binary data. In some cases, it may be power inefficient to get all this binary data from the SPAD sensor and may generate a communications bottleneck at a data bus.

[0052] However, simply reducing the frame rate of the binary frames may result in a lower quality of the reconstructed image, since some generated light detection events would simply be dropped (as the 1 -bit memory may have already stored a “1” indicating a light detection event) and, thus, are not accounted for in the reconstructed image and, thus, a loss of information may appear.

[0053] The underlying operation mode of a SPAD sensor leads to a generation of image data in a fundamentally different way than conventional image sensors (e.g., CMOS sensors). It has thus been recognized that the processing of the binary data from a SPAD sensor may be performed in a different way to efficiently extract the information from the noisy and dense stream of binary data.

[0054] In view of the large amount of binary data a SPAD sensor may generate, and the possible power inefficiency and communications bottleneck associated with it, it has further been recognized that the data rate of the binary data from the SPAD sensor may be adapted to a more efficient representation using a neuromorphic processing algorithm. In some embodiments, efficient means that the amount of information is optimized, while the amount of binary data is modulated. In some embodiments, the neuromorphic processing algorithm may provide a stream of compressed binary data.

[0055] As mentioned above, every light detection event may be accounted for, in some embodiments, to avoid information loss to optimize the amount of information, and it has been recognized that a neuromorphic processing algorithm may be used to accomplish this.

[0056] The parameters of the neuromorphic processing algorithm may be trained, in some embodiments, in a way to optimize them for a specific downstream task such as image reconstruction, object detection and / or image segmentation. The parameters may be fixed or may be adapted online (in real-time).

[0057] Hence, some embodiments pertain to device, wherein the device includes: a single-photon avalanche diode sensor including a plurality of single-photon avalanche diode pixels, each being configured to generate and output light detection events to a neuromorphic circuit; and the neuromorphic circuit implementing a plurality of spiking neurons, wherein the neuromorphic circuit is configured to process the light detection events using the plurality of spiking neurons.

[0058] The device may be an electronic circuit, an electronic circuit board, an electronic module, an imaging module, a(n) (mobile) electronic device (e.g., a camera, a computer display, a smartphone, a tablet, a laptop, or the like), a robot, a vehicle, a medical device, etc.

[0059] Generally, neuromorphic circuits are known which realize brain-inspired computing principles on-chip.

[0060] In some embodiments, the neuromorphic circuit has at least the following properties: the memory is co-located with each processing element with no off-chip memory; the processing is event-based and asynchronous (without a clock), which refers to the fact that, in some embodiments, processing only happens if signals are present, thereby achieving low-power processing; the neuromorphic circuit implements a plurality of spiking neurons as their computational primitives, which may be implemented as a Spiking Neural Network (“SNN”) or without interconnection between different spiking neurons; and communications within and / or among different neuromorphic circuit cores is asynchronous via Network-on-Chip (NoC) technologies.

[0061] Examples of well-established neuromorphic circuits are Dynap™-CNN from SynSense™, Loihi and Loihi 2 from Intel™, wherein, in some embodiments, at least one of them is used.

[0062] In some embodiments, the single-photon avalanche diode sensor and the neuromorphic circuit are implemented on the same chip.

[0063] In some of such embodiments, the neuromorphic circuit is implemented within the single-photon avalanche diode sensor. In some of such embodiments, each spiking neuron is implemented within a different single-photon avalanche diode pixel (which may also be referred to as in-pixel implementation).

[0064] As generally known, SNNs are artificial neural networks which are inspired from the behavior and mechanisms of biological neural networks.

[0065] Spiking neurons, which are used in a SNN, have internal states which evolve in time and, thus, enabling dynamic processing of information such that the SNN may extract information from temporal information, in some embodiments, in a way that conventional stateless neural networks may not be able to do in some cases.

[0066] Thus, typically, SNNs process information in a spatio-temporally sparse way. Messages between different spiking neurons get typically exchanged in forms of spikes which can be binary or graded (i.e., valued).

[0067] It has thus been recognized that SPAD-based sensing and neuromorphic processing may suit together for improving passive SPAD sensor-based imaging, since, for example, SPAD sensors output their light detection events as binary events where a “1” corresponds to a photon detection and a “0” corresponds to no photon detection. A neuromorphic circuit typically operates also in a binary way where a “1” corresponds to a spike and a “0” corresponds to no spike, as will also be discussed further below in more detail.

[0068] Additionally, applying conventional algorithms to SPAD data may be inefficient because processing a “0” may be computationally as expensive as processing a “1”. A neuromorphic circuit typically starts processing if spikes are present and therefore a “0” may not cause further processing.

[0069] Moreover, some image reconstruction techniques utilizing SPAD data may typically require an accumulation of the temporal binary data stream. However, in some embodiments, spiking neurons implemented in a neuromorphic circuit may intrinsically integrate input binary data and may thus be suited to work with temporal SPAD data.

[0070] Generally, various spiking neuron models are known, which may be used in some embodiments.

[0071] In some embodiments, a state of each spiking neuron is modeled using a membrane potential. In some embodiments, the state is modeled further using a membrane current.

[0072] For further enhancing the general understanding of the present disclosure, an embodiment of a spiking neuron 20 is schematically illustrated in Fig. 3, which is discussed in the following, and which also applies to other embodiments of the present disclosure.

[0073] The spiking neuron 20 is connected to three different SPAD pixels 3 for the sake of illustration only. However, the spiking neuron 20 may as well be connected only to one SPAD pixel 3.

[0074] The spiking neuron 20 receives a first temporal sequence of light detection events 21-1, a second temporal sequence of light detection events 21-2 and a third temporal sequence of light detection events 21-3. Based on the input temporal sequences of light detection events 21-1 to 21-3, the spiking neuron 20 generates an output temporal sequence of spikes 22.

[0075] The output temporal sequence of spikes 22 depends on the implemented spiking neural model and the values of the parameters of the implemented spiking neural model.

[0076] Each spiking neuron 20 may have a membrane threshold and a synaptic weight. The spiking neuron 20 generates a spike- a binary message - when the membrane potential exceeds the membrane threshold and outputs the spike to other connected spiking neurons 20 or provides the spike at an output of the neuromorphic circuit.

[0077] A connection between two neurons is denoted as a synapse. A SPAD pixel 3 is considered as a neuron, in some embodiments, as it generates and outputs light detection events which are considered as spikes. Hence, in some embodiments, a synapse is present between a SPAD pixel 3 and a spiking neuron 20.

[0078] A synapse has its own weight which can be fixed (i.e., static) or plastic (i.e., adaptive). When propagating binary spikes between two neurons, the membrane potential of the post-synaptic neuron (the neuron which receives the spike) increases by the synaptic weight value.

[0079] A widely used spiking neuron model, which is used in some embodiments, is the CUrrent-BAsed Leaky Integrate-and-Fire (“CUBA-LIF”) model. In this model, the state of the spiking neuron 20 is modeled both by the membrane potential and the membrane current.

[0080] An input spike (here a light detection event) is scaled by the synaptic weight and is accumulated in the membrane current. This current is then accumulated in the membrane potential of the spiking neuron 20. The CUBA-LIF model has two additional parameters: the membrane current decay constant and the membrane potential decay constant, which influence how the state of the spiking neuron evolves in time. The state of the spiking neuron 20 implementing the CUBA-LIF model may change over time also in the absence of input spikes.

[0081] The CUBA-LIF model may be formulated as follows:

[0082] 0(t+l) = Reset(0(t)).

[0083] Here, the variable I (as a function of time) denotes the membrane current, the membrane current decay constant, T6the membrane potential decay constant, 0(t) the membrane potential (as a function of time), Sinput(t) the input spike (here a light detection event), w the synaptic weight connecting the pre- to the post synaptic neuron (here the SPAD pixel 3 to the spiking neuron 20), ©threshold is the membrane threshold and Soutput(t) the output spike.

[0084] A simplified spiking neuron model is the Integrate-and-Fire (“IF”) model. In this spiking neuron model the state of the spiking neuron 20 is modeled only by the membrane potential. An input spike is multiplied by the synaptic weight and is then accumulated in the membrane potential. If the membrane potential exceeds the membrane threshold, a spike is generated and passed downstream.

[0085] The IF model may be formulated as follows: e(t) = e(t-i) + sinput(t)-w,

[0086] 0(t+l) = Reset(©(t)).

[0087] Different reset mechanisms may be implemented in each spiking neuron model. A reset mechanism describes what happens to the membrane potential of the spiking neuron 20 after generating and outputting a spike. Typical reset mechanisms include either hard reset by resetting the membrane potential back to zero, or soft reset by subtracting the membrane threshold from the membrane potential. More complex reset mechanisms may be implemented depending on the application.

[0088] Returning to the general explanations, generally, each spiking neuron takes all of its input light detection events into account in the generation of output spikes such that all information from the SPAD sensor is used. The output spikes may thus be considered as integrated light detection events in some embodiments or enhanced light detection events in some embodiments.

[0089] Thus, it has been recognized that a neuromorphic circuit using a plurality of spiking neurons to process light detection events of a plurality of SPAD pixels preserves the data format of the SPAD sensor and, thus, the output of the neuromorphic circuit is compatible with known image reconstruction techniques that use SPAD data to reconstruct an image of a scene. For example, the output of the neuromorphic circuit may be read in a temporal sequence of binary frames as well.

[0090] In some embodiments, each spiking neuron has a membrane threshold and a connection to a different single-photon avalanche diode pixel or a different group of single-photon avalanche diode pixels to process the respective light detection events, wherein the connection is assigned a weight, as also discussed above under reference of Fig. 3.

[0091] The connection may at least be a logical connection in the sense that each spiking only processes the light detection events generated and output by a particular SPAD pixel or by a particular group of SPAD pixels. The connection may include a direct or indirect physical connection.

[0092] The group of SPAD pixels may include a plurality of adjacent SPAD pixels of the plurality of SPAD pixels.

[0093] By connecting the SPAD pixels to a spiking neuron in a 1-to-l fashion or in a patch-based fashion, the neuromorphic circuit may modify the stream of raw photon detections (light detection events) while preserving image quality.

[0094] As mentioned above, the amount of binary data from the SPAD sensor may be large and may thus cause communications bottlenecks due to a limited bandwidth of a data bus, however, simply reducing the frame rate of binary frames may result in a reduced quality of the reconstructed image, since some image information would not be used.

[0095] Hence, in some embodiments, a ratio of the membrane threshold and the weight is set larger than one such that an input rate of light detection events is higher than an output rate of spikes.

[0096] Each spiking neuron utilizes all of its input light detection events to generate and output spikes such that all information from the SPAD sensor is used and the output rate of spikes is reduced such that a reduced frame rate may be used without a sharp loss of image quality. In this way, an efficient compression of the light detection events may be performed. Moreover, the data format stays the same as the neuromorphic circuit outputs binary data similar as the SPAD sensor does.

[0097] It has further been recognized that the output rate of each spiking neuron may be adapted in accordance with the amount of light incident on the respective SPAD pixel or group of SPAD pixels, for example, to balance between highly illuminated and less illuminated parts of the scene. This may improve a (grey or color) tone mapping for the image reconstruction.

[0098] Hence, in some embodiments, the neuromorphic circuit is further configured to dynamically adapt, for each spiking neuron, at least one of the weight and the membrane threshold depending on an amount of light incident on the single-photon avalanche diode pixel or the group of singlephoton avalanche diode pixels connected to the respective spiking neuron.

[0099] In other words, some spiking neurons may increase the output rate of spikes compared to the input rate of light detection events when the connected SPAD pixels are less illuminated, and some spiking neurons may decrease the output rate of spikes compared to the input rate of light detection events when the connected SPAD pixels are highly illuminated.

[0100] In this way, on the one hand, an overall frame rate may be reduced for efficient compression, for example, in accordance with the highest ratio of the membrane threshold and the weight. On the other hand, highly and less illuminated parts of the SPAD sensor are more balanced to improve a (grey or color) tone mapping for the image reconstruction.

[0101] In some embodiments, the amount of light is represented by an input rate of light detection events to the respective spiking neuron.

[0102] In some embodiments, at least one of the weight and the membrane threshold are adapted based on local plasticity rules of the respective spiking neuron.

[0103] Local plasticity rules are generally known. An overview is given, for example, in: “Spike-based local synaptic plasticity: a survey of computational models and neurom orphic circuits”, Lyes Khacef et al., 2023, Neuromorph. Comput. Eng. 3, 042001, DOI 10.1088 / 2634-4386 / ad05da.

[0104] Local plasticity rules are also considered in: Albers C, Westkott M, Pawelzik K (2016), “Learning of Precise Spike Times with Homeostatic Membrane Potential Dependent Synaptic Plasticity.”, PLoS ONE 11(2): e0148948.doi:10.1371 / joumal.pone.0148948.

[0105] Such local plasticity rules may be used in some embodiments.

[0106] In some embodiments, the weight is adapted based on a deviation between a membrane potential and a first threshold and a second threshold.

[0107] In some embodiments, the weight is adapted based on a deviation of an output rate of spikes and a target output rate.

[0108] In some embodiments, the membrane threshold is increased by a predetermined value when a light detection event is processed, and wherein the increase decays with a predetermined time constant.

[0109] Some embodiments pertain to a method, wherein the method includes: generating and outputting light detection events to a neuromorphic circuit; and processing, by the neuromorphic circuit implementing a plurality of spiking neurons, the light detection events using the plurality of spiking neurons.

[0110] The method may be performed by the device as described herein. Returning to Fig. 4, there is schematically illustrated in a block diagram an embodiment of a device 30 which includes a SPAD sensor 2 and a neuromorphic circuit 31, which is discussed in the following under reference of Fig. 4, Fig. 6, Fig. 7 and Fig. 8.

[0111] The embodiment is based on the embodiments discussed under reference of Figs. 1, 2 and 3.

[0112] The device 30 may be, for example, a mobile electronic device such as a smartphone.

[0113] Compared to the known embodiment of the device 1 of Fig. 1, the device 30 includes the neuromorphic circuit 31 which implements a plurality of spiking neurons 20, wherein each SPAD pixel 3 is connected with a different spiking neuron 20 (for the sake of illustration only some spiking neurons 20 and their connections to SPAD pixels 3 are shown and others are omitted in the drawing).

[0114] Hence, each spiking neuron 20 processes light detection events from a different SPAD pixel 3 and generates, based thereon, spikes.

[0115] The neuromorphic circuit 31 outputs the generated spikes to the input / output interface 4 - and thus with a similar data format as the SPAD pixel 3 - for transmission via the data bus 5 to the application processor 6.

[0116] In such embodiments, the SPAD sensor 2 and the neuromorphic circuit 31 are each implemented as a separate chip.

[0117] However, in some embodiments, the SPAD sensor 2 and the neuromorphic circuit 31 are implemented on the same chip.

[0118] Fig. 5 schematically illustrates in a block diagram an embodiment of a device 30-1 including the SPAD sensor 2 or a SPAD sensor 2-1 and the neuromorphic circuit 31, which is discussed in the following under reference of Fig. 5.

[0119] In this embodiment, the SPAD sensor 2 and the neuromorphic circuit 31 are implemented on the same chip, as illustrated by the missing box around the neuromorphic circuit 31.

[0120] In some of such embodiments, the neuromorphic circuit 31 is implemented within the SPAD sensor 2-1, as illustrated by the dotted box with the reference number 2-1.

[0121] Moreover, in some of such embodiments of the SPAD sensor 2-1, each spiking neuron 20 is implemented within a different SPAD pixel 3.

[0122] Generally, the following explanations made under reference of Figs. 6 to 8 equally apply to the embodiments of the device 30 of Fig. 4 and the device 30-1 of Fig. 5. Fig. 6 schematically illustrates in a block diagram an embodiment of a processing of light detection events by a plurality of spiking neurons 20 as a pre-processing method for image reconstruction methods, which is discussed in the following.

[0123] The SPAD pixels 3 of Fig. 4 or Fig. 5 generate and output light detection events in a spatiotemporal way, which may be represented as a spatio-temporal cube.

[0124] These light detection events are input in a 1-to-l fashion to the spiking neurons 20 which output spikes in a similar spatio-temporal way.

[0125] However, the amount of binary data may be reduced. If, for example, the amount of binary data from the SPAD pixels 3 is X MB, then the amount of binary may be reduced by a factor N, since, as mentioned above, the spiking neurons 20 may be configured to have an output rate of spikes that is lower than an input rate of light detection events.

[0126] Hence, the frame rate of the binary frames (e.g., of the input / output interface 4 of Fig. 4 or Fig. 5) may be set reduced in accordance with a ratio of the output rate and the input rate.

[0127] In this way, the binary data from the SPAD sensor 2 may be compressed in an efficient way, since all light detection events are taken into account in the compressed output stream.

[0128] The compressed output stream is then input to an image reconstruction method 40 which, for example, uses Quanta Burst Imaging (“QBI”).

[0129] Fig. 7 schematically illustrates in Fig. 7A an embodiment of generating an output rate of spikes that is lower than an input rate of light detection events, and in Fig. 7B an embodiment of generating an output rate of spikes that is higher than an input rate of light detection events, which are discussed in the following.

[0130] The ratio 9threshoid / w, which relates the spiking neurons membrane threshold 9threshoid with the weight w connecting the SPAD pixels 3 with the spiking neurons 20, allows to adapt the output rate of the spikes with respect to the input rate of the light detection events: a ratio, 9threshoid / w > 1, decreases the output rate with respect to the input rate, while a ratio, 9threshoid / w < 1, increases the output rate with respect to the input rate.

[0131] Referring now to Fig. 7A, the ratio is set to 9threshoid / w = 4 and a hard reset is implemented.

[0132] A temporal sequence of light detection events 50-1 is input to a spiking neuron 20. With each light detection event, the membrane potential 50-2 is increased by w. Once the membrane potential 50-2 reaches or exceeds the membrane threshold (^threshold, the spiking neuron 20 generates and outputs a spike 50-3, which is configured here as to happen after four input light detection events.

[0133] As a result, the output rate of spikes is lower than the input rate of light detection events.

[0134] Referring now to Fig. 7B, the ratio is set to 9threshoid / w =0-25 and a soft reset is implemented by a subtracting the membrane threshold ^threshold from the membrane potential when a spike is generated.

[0135] A temporal sequence of light detection events 51-1 is input to a spiking neuron 20. With each light detection event, the membrane potential 51-2 is increased by w.

[0136] Once the membrane potential 51-2 reaches or exceeds the membrane threshold (^threshold, the spiking neuron 20 generates and outputs a spike 51-3. Here, after a light detection event, the membrane potential is still above the membrane threshold (^threshold such that three further spikes 51-3 are generated until the membrane potential 51-2 is below the membrane threshold (^threshold again.

[0137] As a result, the output rate of spikes is larger than the input rate of light detection events.

[0138] The parameters ^threshold and w may be configured manually, during training with backpropagation or dynamically based on local plasticity rules. The configuration may depend on the downstream task.

[0139] If the data bandwidth is to be reduced, then the ratio may be fixed such that it is larger than one such that the output rate of spikes is lower than the input rate of light detection events.

[0140] In some embodiments, a spiking neuron 20 functions as an integrator which accumulates an input and returns an output only if the accumulated input exceeds a certain value. This mapping between input and output is determined by two parameters, as mentioned above, the weight between the SPAD pixel 3 and the respective spiking neuron 20 and the membrane threshold ^threshold of the respective spiking neuron 20.

[0141] Basically, the ratio of the weight w and the membrane threshold ^threshold determines how many input photon detections (light detection events) are required to generate one spiking output. In this way, data compression may be achieved.

[0142] In order to quantify the effect of this filtering technique on the resulting reconstructed image, a comparison is made between the image reconstruction applied to the raw SPAD data stream and the neuromorphic filtered SPAD data stream, as depicted in Fig. 8, which schematically illustrates an example of a relation between compression ratio and peak signal to noise ratio of a reconstructed image, which is discussed in the following.

[0143] The effect of increasing the compression ratio (“CR”) on the quality of the reconstructed image in terms of Peak Signal -to-Noise Ratio (“PSNR”) is illustrated in the graph on the left side.

[0144] The uppermost left image on the right side depicts the ground truth image of the scene which the SPAD sensor 2 is capturing.

[0145] The uppermost right image on the right side depicts a result of QBI image reconstruction using the raw SPAD data.

[0146] The rest of the images show the reconstructed image when different compression ratios are used. These results show that the data rate may be reduced by a factor of eight, while maintaining basically the same image quality.

[0147] Returning to Fig. 7, apart from setting the same (fixed) ratio for each spiking neuron 20, the parameters of the implemented spiking neuron model may be adapted dynamically for each spiking neuron 20 individually by the neuromorphic circuit 31.

[0148] As SPAD sensors output their data as a temporal sequence of light detection events and as this data rate is proportional to the overall scene brightness incident on the respective SPAD pixel 3, it has been recognized that the ratio 9threshoid / wmay be dynamically adapted based on scene illumination or scene brightness.

[0149] It is envisaged, in some embodiments, that the neuromorphic circuit 31 increases the data rate in low-light conditions and decreases the data rate in high-light conditions, as will be discussed in the following.

[0150] In order to improve (grey or color) tone mapping for image reconstruction, the adaption strategy for the SPAD sensor 2 may be optimized online by adapting the parameters of the spiking neurons 20 dynamically. At least one of the membrane thresholds and the weights may be adapted with various mechanisms, which are based on local plasticity rules.

[0151] First, the membrane thresholds may be adapted dynamically by the neuromorphic circuit 31.

[0152] When a spiking neuron 20 generates a spike, the membrane threshold increases by a predetermined value which then decays with a predetermined time constant (e.g., exponentially) to its resting state. In this way, a short-term adaptation is introduced in the membrane threshold where it increases temporarily to regulate (i.e., reduce) the output rate of spikes of the respective spiking neuron 20, and then comes back to its resting value so that the spiking neuron 20 may generate a spike again.

[0153] Second, the weights may be adapted dynamically by the neuromorphic circuit 31.

[0154] Biological neurons learning ability is expressed mainly as the change in strength (i.e., weight) of the synapses that connect neurons, to adapt the structure and function of the underlying network. Neuromorphic processing takes inspiration from biology and uses brain-inspired local synaptic plasticity rules for online learning and adaptation on the fly.

[0155] In some embodiments, synapses with adaptive weights are used between the SPAD pixel 3 (pre- synaptic neuron) and the spiking neuron 20 (post-synaptic neuron), based on plasticity mechanisms that rely on at least one of the following local variables for on-chip (which may be energy efficient) and online (i.e., real-time) learning: pre-synaptic spike trace (low-pass filter of the pre-synaptic spikes); post-synaptic membrane potential; and post-synaptic spike trace (low- pass filter of the post-synaptic spikes).

[0156] For example, the Homeostatic Membrane Potential Dependent Plasticity (“MPDP”) may be used in some embodiments, which is able to keep the membrane potential between two thresholds. An embodiment of this local plasticity rule may be formulated as: w = T|-(-y[e(t)-eu]+- [eD-0(t)]+)-Sinput(t), wherein r is the learning rate, y is a (learned) coefficient, 0( (is the upper threshold and, e.g., between the membrane threshold and zero, 0Dis the lower threshold and, e.g., zero, and [...]+ are rectifying brackets.

[0157] In some embodiments, the Calcium MPDP may be used which targets a specific output rate. An embodiment of this local plasticity rule may be formulated as: wherein Steacher(t) is the target output rate, p(9(t)) is the current output rate of spikes and f(p) is a function which scales the current output rate of spikes.

[0158] Both synaptic local plasticity rules may regulate the activity of the spiking neuron 20 and may be used to adapt the SPAD pixels 3 input rate on the fly.

[0159] Fig. 9 schematically illustrates in a flow diagram an embodiment of a method 120, which is discussed in the following.

[0160] The method may be performed by the device as described herein.

[0161] At 121, light detection events are generated, as discussed herein. At 122, the light detection events are output to a neuromorphic circuit, as discussed herein.

[0162] At 123, by the neuromorphic circuit implementing a plurality of spiking neurons, the light detection events are processed using the plurality of spiking neurons, as discussed herein.

[0163] Fig. 10 schematically illustrates in a flow diagram an embodiment of a method 130, which is discussed in the following.

[0164] The method may be performed by the device as described herein.

[0165] At 131, light detection events are generated, as discussed herein.

[0166] At 132, the light detection events are output to a neuromorphic circuit implementing a plurality of spiking neurons, as discussed herein.

[0167] At 133, a ratio of a membrane threshold and a weight is set larger than one such that an input rate of light detection events is higher than an output rate of spikes, wherein each spiking neuron has the membrane threshold and a connection to a different single-photon avalanche diode pixel or a different group of single-photon avalanche diode pixels to process the respective light detection events, wherein the connection is assigned the weight, as discussed herein.

[0168] At 134, by the neuromorphic circuit, the light detection events are processed using the plurality of spiking neurons, as discussed herein.

[0169] Fig. 11 schematically illustrates in a flow diagram an embodiment of a method 140, which is discussed in the following.

[0170] The method may be performed by the device as described herein.

[0171] At 141, light detection events are generated, as discussed herein.

[0172] At 142, the light detection events are output to a neuromorphic circuit implementing a plurality of spiking neurons, as discussed herein.

[0173] At 143, by the neuromorphic circuit for each spiking neuron, at least one of the weight and the membrane threshold is dynamically adapted depending on an amount of light incident on the single-photon avalanche diode pixel or the group of single-photon avalanche diode pixels connected to the respective spiking neuron, wherein each spiking neuron has the membrane threshold and a connection to a different single-photon avalanche diode pixel or a different group of single-photon avalanche diode pixels to process the respective light detection events, wherein the connection is assigned the weight, as discussed herein.

[0174] At 144, by the neuromorphic circuit, the light detection events are processed using the plurality of spiking neurons, as discussed herein. It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding.

[0175] Note that the present technology can also be configured as described below.

[0176] (1) A device, wherein the device includes: a single-photon avalanche diode sensor including a plurality of single-photon avalanche diode pixels, each being configured to generate and output light detection events to a neuromorphic circuit; and the neuromorphic circuit implementing a plurality of spiking neurons, wherein the neuromorphic circuit is configured to process the light detection events using the plurality of spiking neurons.

[0177] (2) The device of (1), wherein each spiking neuron has a membrane threshold and a connection to a different single-photon avalanche diode pixel or a different group of singlephoton avalanche diode pixels to process the respective light detection events, wherein the connection is assigned a weight.

[0178] (3) The device of (2), wherein a ratio of the membrane threshold and the weight is set larger than one such that an input rate of light detection events is higher than an output rate of spikes.

[0179] (4) The device of (2), wherein the neuromorphic circuit is further configured to dynamically adapt, for each spiking neuron, at least one of the weight and the membrane threshold depending on an amount of light incident on the single-photon avalanche diode pixel or the group of singlephoton avalanche diode pixels connected to the respective spiking neuron.

[0180] (5) The device of (4), wherein the at least one of the weight and the membrane threshold are adapted based on local plasticity rules of the respective spiking neuron.

[0181] (6) The device of (5), wherein the weight is adapted based on a deviation between a membrane potential and a first threshold and a second threshold.

[0182] (7) The device of (5) or (6), wherein the weight is adapted based on a deviation of an output rate of spikes and a target output rate.

[0183] (8) The device of anyone of (5) to (7), wherein the membrane threshold is increased by a predetermined value when a light detection event is processed, and wherein the increase decays with a predetermined time constant. (9) The device of anyone of (4) to (8), wherein the amount of light is represented by an input rate of light detection events to the respective spiking neuron.

[0184] (10) The device of anyone of (1) to (9), wherein the single-photon avalanche diode sensor and the neuromorphic circuit are implemented on the same chip.

[0185] (11) A method, wherein the method includes: generating and outputting light detection events to a neuromorphic circuit; and processing, by the neuromorphic circuit implementing a plurality of spiking neurons, the light detection events using the plurality of spiking neurons.

[0186] (12) The method of (11), wherein each spiking neuron has a membrane threshold and a connection to a different single-photon avalanche diode pixel or a different group of singlephoton avalanche diode pixels to process the respective light detection events, wherein the connection is assigned a weight.

[0187] (13) The method of (12), further including setting a ratio of the membrane threshold and the weight larger than one such that an input rate of light detection events is higher than an output rate of spikes.

[0188] (14) The method of (12), further including dynamically adapting, by the neuromorphic circuit for each spiking neuron, at least one of the weight and the membrane threshold depending on an amount of light incident on the single-photon avalanche diode pixel or the group of singlephoton avalanche diode pixels connected to the respective spiking neuron.

[0189] (15) The method of (14), wherein the at least one of the weight and the membrane threshold are adapted based on local plasticity rules of the respective spiking neuron.

[0190] (16) The method of (15), wherein the weight is adapted based on a deviation between a membrane potential and a first threshold and a second threshold.

[0191] (17) The method of (15) or (16), wherein the weight is adapted based on a deviation of an output rate of spikes and a target output rate.

[0192] (18) The method of anyone of (15) to (17), wherein the membrane threshold is increased by a predetermined value when a light detection event is processed, and wherein the increase decays with a predetermined time constant.

[0193] (19) The method of anyone of (14) to (18), wherein the amount of light is represented by an input rate of light detection events to the respective spiking neuron. (20) The method of anyone of (11) to (19), wherein a state of each spiking neuron is modeled using a membrane potential, in particular, wherein the state is modeled further using a membrane current.

Claims

CLAIMS1. A device, comprising: a single-photon avalanche diode sensor including a plurality of single-photon avalanche diode pixels, each being configured to generate and output light detection events to a neuromorphic circuit; and the neuromorphic circuit implementing a plurality of spiking neurons, wherein the neuromorphic circuit is configured to process the light detection events using the plurality of spiking neurons.

2. The device of claim 1, wherein each spiking neuron has a membrane threshold and a connection to a different single-photon avalanche diode pixel or a different group of singlephoton avalanche diode pixels to process the respective light detection events, wherein the connection is assigned a weight.

3. The device of claim 2, wherein a ratio of the membrane threshold and the weight is set larger than one such that an input rate of light detection events is higher than an output rate of spikes.

4. The device of claim 2, wherein the neuromorphic circuit is further configured to dynamically adapt, for each spiking neuron, at least one of the weight and the membrane threshold depending on an amount of light incident on the single-photon avalanche diode pixel or the group of single-photon avalanche diode pixels connected to the respective spiking neuron.

5. The device of claim 4, wherein the at least one of the weight and the membrane threshold are adapted based on local plasticity rules of the respective spiking neuron.

6. The device of claim 5, wherein the weight is adapted based on a deviation between a membrane potential and a first threshold and a second threshold.

7. The device of claim 5, wherein the weight is adapted based on a deviation of an output rate of spikes and a target output rate.

8. The device of claim 5, wherein the membrane threshold is increased by a predetermined value when a light detection event is processed, and wherein the increase decays with a predetermined time constant.

9. The device of claim 4, wherein the amount of light is represented by an input rate of light detection events to the respective spiking neuron.

10. The device of claim 1, wherein the single-photon avalanche diode sensor and the neuromorphic circuit are implemented on the same chip.

11. A method, comprising: generating and outputting light detection events to a neuromorphic circuit; and processing, by the neuromorphic circuit implementing a plurality of spiking neurons, the light detection events using the plurality of spiking neurons.

12. The method of claim 11, wherein each spiking neuron has a membrane threshold and a connection to a different single-photon avalanche diode pixel or a different group of singlephoton avalanche diode pixels to process the respective light detection events, wherein the connection is assigned a weight.

13. The method of claim 12, further comprising setting a ratio of the membrane threshold and the weight larger than one such that an input rate of light detection events is higher than an output rate of spikes.

14. The method of claim 12, further comprising dynamically adapting, by the neuromorphic circuit for each spiking neuron, at least one of the weight and the membrane threshold depending on an amount of light incident on the single-photon avalanche diode pixel or the group of singlephoton avalanche diode pixels connected to the respective spiking neuron.

15. The method of claim 14, wherein the at least one of the weight and the membrane threshold are adapted based on local plasticity rules of the respective spiking neuron.

16. The method of claim 15, wherein the weight is adapted based on a deviation between a membrane potential and a first threshold and a second threshold.

17. The method of claim 15, wherein the weight is adapted based on a deviation of an output rate of spikes and a target output rate.

18. The method of claim 15, wherein the membrane threshold is increased by a predetermined value when a light detection event is processed, and wherein the increase decays with a predetermined time constant.

19. The method of claim 14, wherein the amount of light is represented by an input rate of light detection events to the respective spiking neuron.

20. The method of claim 11, wherein a state of each spiking neuron is modeled using a membrane potential, in particular, wherein the state is modeled further using a membrane current.

Images