Apparatus and method

Abstract

An apparatus, wherein the apparatus comprises: a single photon avalanche diode sensor comprising a plurality of single photon avalanche diode pixels, each single photon avalanche diode pixel configured to generate and output a light detection event to a neuromorphic circuit; and a neuromorphic circuit implementing a plurality of spiking neurons, wherein the neuromorphic circuit is configured to process the light detection events using the plurality of spiking neurons.

Description

Technical Field

[0001] This disclosure generally relates to apparatus and methods. Background Technology

[0002] Typically, single-photon avalanche diode ("SPAD") sensors are known to be used in active light sensing devices such as LiDAR ("light detection and ranging") devices.

[0003] Recently, such SPAD sensors have been used in passive imaging devices, where passive SPAD sensors can provide useful characteristics to the imaging device.

[0004] Unlike conventional complementary metal-oxide-semiconductor (“CMOS”) image sensors that integrate incident light over time, SPAD sensors generate photodetection events by photoelectric conversion of incident light to generate avalanche currents, and output photodetection events to operate by counting the number of photodetection events generated.

[0005] Due to the high sensitivity of SPAD sensors, they may be able to capture images of scenes in low-light conditions, as they are typically sensitive to single-photon detection.

[0006] In addition, the circuitry of SPAD sensors typically operates in the digital domain, thereby avoiding noise that is usually caused by analog-to-digital conversion (“ADC”) operations.

[0007] However, its extremely high sensitivity results in a large data rate, which can be difficult to handle in certain situations (e.g., in some mobile applications). Furthermore, in low-light scenes, SPAD sensors may sometimes be unable to output enough data to reconstruct an image.

[0008] Although imaging techniques based on passive SPAD sensors exist, improvements to existing techniques are generally expected. Summary of the Invention

[0009] According to a first aspect, this disclosure provides an apparatus comprising: A single-photon avalanche diode sensor includes multiple single-photon avalanche diode pixels, each single-photon avalanche diode pixel being configured to generate a light detection event and output the light detection event to a neuromorphic circuit; and A neuromorphic circuit is implemented to realize multiple spiking neurons, wherein the neuromorphic circuit is configured to use multiple spiking neurons to process light detection events.

[0010] According to the second aspect, this disclosure provides a method comprising: Generate light detection events and output them to a neuromorphic circuit; and Multiple spiking neurons are implemented using neuromorphic circuits, and these multiple spiking neurons are used to process light detection events.

[0011] Other aspects are set forth in the dependent claims, the drawings and the following description. Attached Figure Description

[0012] The embodiments are explained by way of example relative to the accompanying drawings, in which: Figure 1 An embodiment of a device including a SPAD sensor is illustrated schematically in block diagram form; Figure 2 An embodiment of the SPAD pixel is schematically illustrated; Figure 3 The implementation of a spiking neuron is illustrated schematically; Figure 4 A block diagram schematically illustrates an implementation of a device including a SPAD sensor and neuromorphic circuitry. Figure 5 A block diagram schematically illustrates an implementation of a device including a SPAD sensor and neuromorphic circuitry. Figure 6 A block diagram schematically illustrates an implementation method for processing light detection events using multiple spiking neurons as a preprocessing method for image reconstruction. Figure 7 exist Figure 7 A schematically illustrates an implementation of generating a pulse output rate that is lower than the input rate of the optical detection event, and... Figure 7 B schematically illustrates an implementation of generating a pulse output rate that is higher than the input rate of the optically detected event; Figure 8 An example illustrating the relationship between the compression ratio and peak signal-to-noise ratio of a reconstructed image is shown. Figure 9 The implementation of the method is illustrated schematically using a flowchart; Figure 10 The implementation of the method is schematically illustrated using a flowchart; and Figure 11 The implementation of the method is illustrated schematically using a flowchart. Detailed Implementation

[0013] Provide a reference Figure 4 Before a detailed description of the implementation methods, a general explanation will be given.

[0014] As mentioned at the beginning, single-photon avalanche diode (“SPAD”) sensors have recently been used in passive imaging devices, where passive SPAD sensors can provide useful characteristics to the imaging device.

[0015] The data generated by the SPAD sensor differs from data from other imaging sensors.

[0016] Compared to standard CMOS image sensors, SPAD sensors encode information over time. To reconstruct the scene, one bit of data from the SPAD sensor is accumulated over time.

[0017] Compared to event-based sensors (EVS), data from SPAD sensors is spatially dense. While EVS responds to changes in incident light, SPAD sensors can collect all incident photons.

[0018] As further mentioned at the beginning, due to the high sensitivity of SPAD sensors, SPAD sensors may be able to capture images of scenes in low-light conditions, as they are typically sensitive to single-photon detection.

[0019] In addition, the circuitry of SPAD sensors typically operates in the digital domain, thereby avoiding noise that is usually caused by analog-to-digital conversion (“ADC”) operations.

[0020] However, its extremely high sensitivity leads to large data rates, which can be difficult to handle in some situations (e.g., in some mobile applications).

[0021] Therefore, handling the large amounts of data that SPAD sensors may generate can be challenging in certain use cases. For example, some SPAD sensors can be read out in binary frames at around 100 kfps (“kiloframes per second”), resulting in a large amount of binary data. In some cases, acquiring all this data from a SPAD sensor can be power inefficient and may create communication bottlenecks.

[0022] To enhance the overall understanding of this disclosure, embodiments are schematically illustrated below with reference to block diagrams. Figure 1 The discussion includes implementation methods of device 1, including SPAD sensor 2.

[0023] Device 1 can be, for example, a mobile electronic device, such as a smartphone.

[0024] Device 1 includes a SPAD sensor 2, an input / output interface 4, a data bus 5, and an application processor 6.

[0025] SPAD sensor 2 includes a plurality of SPAD pixels 3, which are arranged in an array in this embodiment, without limiting the present disclosure in this respect.

[0026] When light is incident on the corresponding SPAD pixel 3 and the corresponding SPAD pixel 3 is activated, each SPAD pixel 3 generates and outputs a light detection event.

[0027] To further enhance the overall understanding of this disclosure, embodiments are illustrated below with reference to other embodiments that may also be applied to this disclosure. Figure 2 The implementation methods of SPAD pixel 3 are discussed.

[0028] SPAD pixel 3 includes SPAD 10, resistor 11, switch 12 and NOT gate 13.

[0029] The anode of SPAD 10 is coupled to the GND potential, and the cathode of the SPAD is coupled to resistor 11 and NOT gate 13.

[0030] Resistor 11 is coupled to switch 12, and switch 12 is further coupled to bias voltage Vbias.

[0031] When SPAD 10 is activated by closing switch 12, SPAD 10 is reverse biased via resistor 11 by a bias voltage Vbias that is higher than the breakdown voltage of SPAD 10, so as to allow avalanche current to be generated in SPAD 10 when light is incident on SPAD 10.

[0032] Once an avalanche current is generated in response to an incident photon, the cathode voltage Vc of SPAD 10 decreases until it falls below the breakdown voltage. The cathode voltage Vc quickly recovers to the bias voltage Vbias, such that it includes a short voltage pulse received by NOT gate 13. NOT gate 13 then outputs a photodetector event LDE as a digital pulse representing 1 bit of binary data.

[0033] Generally, this disclosure is not limited to such embodiments of SPAD sensors and SPAD pixels, and other embodiments may be apparent to those skilled in the art.

[0034] Back Figure 1 Each SPAD pixel 3 outputs its generated light detection event to an input / output interface 4 coupled to the data bus 5 (for illustrative purposes, only some SPAD pixels 3 are shown, and the explicit connections to the input / output interface 4 are not shown).

[0035] The input / output interface 4 may include, for example, a 1-bit memory for the output of each SPAD pixel 3. The 1-bit memory can then be read out at a certain rate (frame rate).

[0036] Then, input / output interface 4 outputs binary frames of light detection events to application processor 6 via data bus 5, for example at a high frame rate of about 100kfps.

[0037] Application processor 6 can perform a variety of advanced functions, such as image reconstruction, object tracking, and / or image segmentation, based on binary frames of light detection events.

[0038] Returning to the overall explanation, as mentioned above, SPAD sensors typically generate binary data at high data rates, which can result in a large amount of binary data. In some cases, acquiring all this binary data from a SPAD sensor can be power inefficient and may create a communication bottleneck at the data bus.

[0039] However, simply reducing the frame rate of binary frames may result in lower quality reconstructed images because some generated light detection events will be simply discarded (since 1-bit memory may have already stored "1" indicating a light detection event) and therefore not considered in the reconstructed image, potentially leading to information loss.

[0040] The underlying operating mode of SPAD sensors results in the generation of image data in a fundamentally different manner compared to conventional image sensors (e.g., CMOS sensors). Therefore, it has been recognized that binary data from SPAD sensors can be processed in different ways to efficiently extract information from noisy and dense binary data streams.

[0041] Given the large amounts of binary data that SPAD sensors may generate, and the associated potential power inefficiencies and communication bottlenecks, it is further recognized that the data rate of binary data from SPAD sensors can be adapted for more efficient representation using neuromorphic processing algorithms. In some implementations, efficiency means that the amount of information is optimized while the amount of binary data is modulated. In some implementations, neuromorphic processing algorithms can provide a stream of compressed binary data.

[0042] As mentioned above, in some implementations, each light detection event can be taken into account to avoid information loss and thus optimize the amount of information, and it has been recognized that this can be achieved using neuromorphic processing algorithms.

[0043] In some implementations, the parameters of the neuromorphic processing algorithm can be trained to optimize the parameters for specific downstream tasks, such as image reconstruction, object detection, and / or image segmentation. The parameters can be fixed or can be adjusted online (in real time).

[0044] Therefore, some embodiments relate to an apparatus comprising: A single-photon avalanche diode sensor includes multiple single-photon avalanche diode pixels, each single-photon avalanche diode pixel being configured to generate a light detection event and output the light detection event to a neuromorphic circuit; and A neuromorphic circuit is implemented to realize multiple spiking neurons, wherein the neuromorphic circuit is configured to use multiple spiking neurons to process light detection events.

[0045] The device can be an electronic circuit, electronic circuit board, electronic module, imaging module, (mobile) electronic device (e.g., camera, computer monitor, smartphone, tablet, laptop, etc.), robot, vehicle, medical device, etc.

[0046] Typically, neuromorphic circuits that implement the principles of on-chip neuromorphic computing are known.

[0047] In some implementations, the neuromorphic circuit has at least the following properties: memory is co-located with each processing element, with no off-chip memory; processing is event-based and asynchronous (without a clock), meaning that in some implementations, processing only occurs when a signal is present, thereby achieving low-power processing; the neuromorphic circuit implements multiple spiking neurons as its computational primitives, which may be implemented as spiking neural networks (“SNNs”) or without interconnection between different spiking neurons; and communication within and / or between different neuromorphic circuit cores is asynchronous via on-chip network (NoC) technology.

[0048] Examples of established neuromorphic circuits are Dynap™-CNN from SynSense™, Loihi and Loihi 2 from Intel™, wherein at least one of these circuits is used in some implementations.

[0049] In some implementations, the single-photon avalanche diode sensor and the neuromorphic circuitry are implemented on the same chip.

[0050] In some of these implementations, the neuromorphic circuitry is implemented within a single-photon avalanche diode sensor. In some of these implementations, each spiking neuron is implemented within a different single-photon avalanche diode pixel (this can also be referred to as in-pixel implementation).

[0051] As is generally known, SNNs are artificial neural networks inspired by the behavior and mechanisms of biological neural networks.

[0052] The spiking neurons used in SNNs have internal states that evolve over time, thus enabling dynamic processing of information. This allows SNNs to extract information from temporal information, and in some implementations, they employ methods that conventional stateless neural networks may not be able to do in certain situations.

[0053] Therefore, SNNs typically process information in a spatiotemporally sparse manner. Messages between different spiking neurons are usually exchanged in the form of pulses, which can be binary or hierarchical (i.e., have values).

[0054] Therefore, it has been recognized that SPAD-based sensing and neuromorphic processing can be combined to improve imaging based on passive SPAD sensors, since, for example, SPAD sensors output their light detection events as binary events, where "1" corresponds to photon detection and "0" corresponds to no photon detection. Neuromorphic circuits also typically operate in a binary manner, where "1" corresponds to a pulse and "0" corresponds to no pulse, as will be discussed in more detail below.

[0055] Furthermore, applying conventional algorithms to SPAD data can be inefficient because processing "0" can be computationally just as costly as processing "1". Neuromorphic circuits typically begin processing upon the presence of a pulse, so "0" may not trigger further processing.

[0056] Furthermore, some image reconstruction techniques utilizing SPAD data often require the accumulation of temporal binary data streams. However, in some implementations, spiking neurons implemented in neuromorphic circuits can inherently integrate the input binary data and may therefore be suitable for processing temporal SPAD data.

[0057] Typically, various spiking neuron models are known to be usable in some implementations.

[0058] In some implementations, the state of each spiking neuron is modeled using membrane potential. In some implementations, the state is further modeled using membrane current.

[0059] To further enhance the overall understanding of this disclosure, other embodiments discussed below, which also apply to this disclosure, will be discussed. Figure 3 The embodiment of the spiking neuron 20 is schematically illustrated in the figure.

[0060] For illustrative purposes only, the spiking neuron 20 is connected to three different SPAD pixels 3. However, the spiking neuron 20 may also be connected to only one SPAD pixel 3.

[0061] The spiking neuron 20 receives the first light detection event time sequence 21-1, the second light detection event time sequence 21-2, and the third light detection event time sequence 21-3.

[0062] Based on the input light detection event time series 21-1 to 21-3, the spiking neuron 20 generates a pulse output time series 22.

[0063] The pulse output time series 22 depends on the implemented spiking neural model and the values ​​of the parameters of the implemented spiking neural model.

[0064] Each spiking neuron 20 may have a membrane threshold and synaptic weights. When the membrane potential exceeds the membrane threshold, the spiking neuron 20 generates a pulse (binary message) and outputs the pulse to other connected spiking neurons 20, or provides a pulse at the output of a neuromorphic circuit.

[0065] The connection between two neurons is represented as a synapse. In some embodiments, SPAD pixel 3 is treated as a neuron because it generates and outputs light detection events as pulses. Therefore, in some embodiments, a synapse exists between SPAD pixel 3 and spiking neuron 20.

[0066] Synapses have their own weights, which can be fixed (i.e., static) or plastic (i.e., adaptive). When a binary pulse propagates between two neurons, the membrane potential of the postsynaptic neuron (the neuron receiving the pulse) increases the synaptic weight value.

[0067] The widely used spiking neuron model for some implementations is the current leakage integral and firing (“CUBA-LIF”) model. In this model, the state of the spiking neuron 20 is modeled by both membrane potential and membrane current.

[0068] The input pulse (in this case, a light-detected event) is scaled by synaptic weights and accumulated in the membrane current. This current then accumulates in the membrane potential of the spiking neuron 20. The CUBA-LIF model has two additional parameters: a membrane current decay constant and a membrane potential decay constant, which affect how the state of the spiking neuron evolves over time. The state of the spiking neuron 20 implementing the CUBA-LIF model may also change over time in the absence of an input pulse.

[0069] The CUBA-LIF model can be expressed as follows: , , , .

[0070] Here, the variable I (as a function of time) represents the membrane current, τ i τ represents the membrane current decay constant. θ Let S represent the membrane potential decay constant, θ(t) represent the membrane potential (as a function of time), and S... input(t) represents the input pulse (here, the light detection event), w represents the synaptic weight connecting the presynaptic neuron to the postsynaptic neuron (here, the synaptic weight connecting SPAD pixel 3 to spiking neuron 20), θ threshold It is the membrane threshold and S output (t) is the output pulse.

[0071] A simplified model of the spiking neuron is the Integral-and-Fire (“IF”) model. In this model, the state of the spiking neuron 20 is modeled solely by membrane potential. Input impulses are multiplied by synaptic weights and then accumulated in the membrane potential. If the membrane potential exceeds a membrane threshold, an impulse is generated and propagated downstream.

[0072] The IF model can be expressed as follows: , , .

[0073] Different reset mechanisms can be implemented in each spiking neuron model. The reset mechanism describes how the membrane potential of the spiking neuron 20 changes after it generates and outputs a pulse. Typical reset mechanisms include a hard reset by resetting the membrane potential to zero, or a soft reset by subtracting a membrane threshold from the membrane potential. More complex reset mechanisms can be implemented depending on the application.

[0074] Returning to a general interpretation, typically, each spiking neuron takes into account all its input photodetector events when generating an output pulse, thus utilizing all information from the SPAD sensor. Therefore, the output pulse can be considered as an integrated photodetector event in some embodiments, or as an enhanced photodetector event in others.

[0075] Therefore, it has been recognized that neuromorphic circuits that use multiple spiking neurons to process light detection events of multiple SPAD pixels preserve the data format of the SPAD sensor, thus making the output of the neuromorphic circuit compatible with known image reconstruction techniques that reconstruct scene images using SPAD data. For example, the output of the neuromorphic circuit can also be read in the form of a binary frame time series.

[0076] In some implementations, each spiking neuron has a membrane threshold and connections to different single-photon avalanche diode pixels or different groups of single-photon avalanche diode pixels to process corresponding light detection events, wherein these connections are weighted, as further referenced above. Figure 3 The subject of discussion.

[0077] The connection can be at least a logical connection, meaning that each pulse only processes and outputs light detection events generated by a specific SPAD pixel or a specific group of SPAD pixels. The connection can include direct or indirect physical connections.

[0078] A SPAD pixel group can include multiple adjacent SPAD pixels from multiple SPAD pixels.

[0079] By connecting SPAD pixels to spiking neurons in a one-to-one or block-based manner, neuromorphic circuits can modify the flow of the original photon detection (photodetection events) while preserving image quality.

[0080] As mentioned above, the amount of binary data from SPAD sensors can be large, and therefore may cause communication bottlenecks due to the limited bandwidth of the data bus. However, simply reducing the frame rate of the binary frames may result in a decrease in the quality of the reconstructed image, as some image information will not be used.

[0081] Therefore, in some implementations, the ratio of the membrane threshold to the weight is set to be greater than one, such that the input rate of the photodetector event is higher than the output rate of the pulse.

[0082] Each spiking neuron uses all its input light detection events to generate and output a pulse, ensuring that all information from the SPAD sensor is utilized. Furthermore, the reduced pulse output rate allows for a lower frame rate without a significant loss of image quality. This enables efficient compression of light detection events. Additionally, the data format remains identical to the binary data output by the neuromorphic circuit (similar to what the SPAD sensor does).

[0083] It has been further recognized that the output rate of each spiking neuron can be adjusted based on the amount of light incident on the corresponding SPAD pixel or group of SPAD pixels, for example, to balance the bright and dark parts of a scene. This can improve the (grayscale or color) tone mapping used for image reconstruction.

[0084] Therefore, in some embodiments, the neuromorphic circuit is also configured to dynamically adjust at least one of the weights and membrane thresholds for each spiking neuron based on the amount of light incident on the single-photon avalanche diode pixel or group of single-photon avalanche diode pixels connected to the respective spiking neuron.

[0085] In other words, when the illumination of the connected SPAD pixels is low, some spiking neurons can increase the output rate of the pulses compared to the input rate of the light detection event, and when the illumination of the connected SPAD pixels is high, some spiking neurons can decrease the output rate of the pulses compared to the input rate of the light detection event.

[0086] In this way, on the one hand, the overall frame rate can be reduced, for example, based on the highest ratio of the membrane threshold to the weight, to achieve efficient compression. On the other hand, the high-light and low-light portions of the SPAD sensor are more balanced, improving the (grayscale or color) tone mapping used for image reconstruction.

[0087] In some implementations, the amount of light is represented by the input rate of light detection events to the corresponding spiking neuron.

[0088] In some implementations, at least one of the weights and membrane thresholds is adjusted based on the local plasticity rules of the corresponding spiking neurons.

[0089] The rules of local plasticity are well known. For example, an overview can be found in: “Spike-based local synapticplasticity: a survey of computational models and neuromorphic circuits”, Lyes Khacef et al., 2023, Neuromorph. Comput. Eng. 3, 042001, DOI 10.1088 / 2634-4386 / ad05da.

[0090] The rule of local plasticity is also found in: Albers C, Westkott M, Pawelzik K (2016), “Learning of Precise Spike Times with Homeostatic Membrane Potential Dependent Synaptic Plasticity.”, PLoS ONE 11(2):e0148948.doi:10.1371 / journal.pone.0148948.

[0091] In some implementations, such local plasticity rules can be used.

[0092] In some implementations, the weights are adjusted based on the deviation between the membrane potential and a first threshold and a second threshold.

[0093] In some implementations, the weights are adjusted based on the deviation between the pulse output rate and the target output rate.

[0094] In some implementations, when processing a photodetection event, the membrane threshold increases by a predetermined value, wherein the increase decays with a predetermined time constant.

[0095] Some implementations relate to a method, wherein the method includes: Generate light detection events and output them to a neuromorphic circuit; and Multiple spiking neurons are implemented using neuromorphic circuits, and these multiple spiking neurons are used to process light detection events.

[0096] This method can be performed by an apparatus as described herein.

[0097] Back Figure 4 The block diagram schematically illustrates an embodiment of device 30, which includes a SPAD sensor 2 and a neuromorphic circuit 31, as described below. Figure 4 , Figure 6 , Figure 7 and Figure 8 Discuss this neuromorphic circuit.

[0098] Implementation method based on reference Figure 1 , Figure 2 and Figure 3 The implementation methods discussed.

[0099] Device 30 can be, for example, a mobile electronic device, such as a smartphone.

[0100] and Figure 1 Compared to known embodiments of device 1, device 30 includes a neuromorphic circuit 31 that implements a plurality of spiking neurons 20, wherein each SPAD pixel 3 is connected to a different spiking neuron 20 (for illustrative purposes, only some spiking neurons 20 and their connections to SPAD pixels 3 are shown in the figure, while others are omitted).

[0101] Therefore, each spiking neuron 20 processes light detection events from different SPAD pixels 3 and generates spiking events based on these events.

[0102] The neuromorphic circuit 31 outputs the generated pulses to the input / output interface 4, and therefore has a data format similar to that of the SPAD pixel 3, for transmission to the application processor 6 via the data bus 5.

[0103] In this type of implementation, the SPAD sensor 2 and the neuromorphic circuit 31 are each implemented as separate chips.

[0104] However, in some implementations, the SPAD sensor 2 and the neuromorphic circuit 31 are implemented on the same chip.

[0105] Figure 5 An embodiment of device 30-1 is schematically illustrated in a block diagram. This device includes a SPAD sensor 2 or SPAD sensor 2-1 and a neuromorphic circuit 31, as described below. Figure 5 This needs to be discussed.

[0106] In this embodiment, the SPAD sensor 2 and the neuromorphic circuit 31 are implemented on the same chip, as shown by the lack of a frame around the neuromorphic circuit 31.

[0107] In some of these embodiments, the neuromorphic circuitry 31 is implemented within the SPAD sensor 2-1, as shown by the dashed box with reference numeral 2-1.

[0108] Furthermore, in some of these embodiments of the SPAD sensor 2-1, each spiking neuron 20 is implemented within a different SPAD pixel 3.

[0109] Generally, refer to the following Figures 6 to 8 The interpretations made are equally applicable Figure 4 Embodiments of device 30 and Figure 5 Implementation of device 30-1.

[0110] Figure 6 A block diagram schematically illustrates an implementation of a preprocessing method for image reconstruction, in which multiple spiking neurons 20 process light detection events. This processing will be discussed below.

[0111] Figure 4 or Figure 5 SPAD pixel 3 generates and outputs light detection events in a spatiotemporal manner, which can be represented as a spatiotemporal cube.

[0112] These light detection events are input to the spiking neuron 20 in a one-to-one manner, which outputs pulses in a similar spatiotemporal manner.

[0113] However, the amount of binary data can be reduced. If, for example, the amount of binary data from SPAD pixel 3 is X MB, then the amount of binary data can be reduced by a factor of N because, as mentioned above, the spiking neuron 20 can be configured to have a pulse output rate that is lower than the input rate of the light detection event.

[0114] Therefore, the frame rate of binary frames (e.g., Figure 4 or Figure 5 The frame rate of input / output interface 4 can be set to decrease based on the ratio of output rate to input rate.

[0115] In this way, binary data from SPAD sensor 2 can be compressed efficiently because all light detection events are taken into account in the compressed output stream.

[0116] The compressed output stream is then input to image reconstruction method 40, which, for example, uses quantum burst imaging (“QBI”).

[0117] Figure 7 exist Figure 7 A schematically illustrates an implementation of generating a pulse output rate that is lower than the input rate of the optical detection event, and... Figure 7 B schematically illustrates an implementation of generating a pulse output rate higher than the input rate of the optical detection event, which will be discussed below.

[0118] The membrane threshold θ of the spiking neuron threshold The ratio θ associated with the weight w connecting SPAD pixel 3 to spiking neuron 20 threshold / w allows adjustment of the pulse output rate relative to the input rate of the optically detected event: ratio θ threshold / w>1 reduces the output rate relative to the input rate, while the ratio θ threshold / w<1 increases the output rate relative to the input rate.

[0119] Now refer to Figure 7 A, the ratio is set to θ threshold / w=4, and a hard reset was implemented.

[0120] The time series of light detection events 50-1 is input into the spiking neuron 20. With each light detection event, the membrane potential 50-2 increases by w.

[0121] Once the membrane potential 50-2 reaches or exceeds the membrane threshold θ threshold The spiking neuron 20 generates and outputs pulse 50-3, which is configured here to occur after four input light detection events.

[0122] As a result, the output rate of the pulse is lower than the input rate of the optically detected event.

[0123] Now refer to Figure 7 B, the ratio is set to θ threshold / w=0.25, and by subtracting the membrane threshold θ from the membrane potential when a pulse is generated. threshold To achieve a soft reset.

[0124] The time series of light detection events 51-1 is input into the spiking neuron 20. With each light detection event, the membrane potential 51-2 increases by w.

[0125] Once the membrane potential 51-2 reaches or exceeds the membrane threshold θ threshold Spiking neurons 20 generate and output pulses 51-3. Here, after the light detection event, the membrane potential remains above the membrane threshold θ. threshold This causes three additional pulses 51-3 to be generated until the membrane potential 51-2 falls below the membrane threshold θ again. threshold .

[0126] As a result, the output rate of the pulse is greater than the input rate of the optically detected event.

[0127] Parameter θ threshold `w` and `w` can be manually configured during training using backpropagation, or dynamically manually configured based on local plasticity rules. The configuration can depend on the downstream task.

[0128] If the data bandwidth is reduced, the ratio can be fixed so that it is greater than one, making the output rate of the pulse lower than the input rate of the optical detection event.

[0129] In some implementations, the spiking neuron 20 acts as an integrator that accumulates the input and returns an output only when the accumulated input exceeds a certain value. This mapping between input and output is determined by two parameters (as mentioned above, the weights between the SPAD pixel 3 and the corresponding spiking neuron 20, and the membrane threshold θ of the corresponding spiking neuron 20). threshold To determine.

[0130] Basically, the weight w is related to the membrane threshold θ threshold The ratio determines how many input photons need to be detected (photodetection events) to generate one pulse output. In this way, data compression can be achieved.

[0131] To quantify the impact of this filtering technique on the resulting reconstructed image, a comparison was made between the image reconstruction applied to the original SPAD data stream and the SPAD data stream after neuromorphic filtering, as illustrated in the example showing an example of the relationship between the compression ratio and peak signal-to-noise ratio of the reconstructed image. Figure 8 The following discussion will focus on what has been described.

[0132] The graph on the left illustrates the effect of increasing the compression ratio (“CR”) on the quality of the reconstructed image (in terms of peak signal-to-noise ratio (“PSNR”)).

[0133] The top left image on the right depicts a true image of the scene being captured by SPAD sensor 2.

[0134] The rightmost image on the right depicts the result of QBI image reconstruction using the original SPAD data.

[0135] The remainder of the image shows the reconstructed images when using different compression ratios. These results demonstrate that the data rate can be reduced by up to eight times while maintaining essentially the same image quality.

[0136] Back Figure 7 In addition to setting the same (fixed) ratio for each spiking neuron 20, the neuromorphic circuit 31 can dynamically adjust the parameters of the implemented spiking neuron model individually for each spiking neuron 20.

[0137] Since the SPAD sensor outputs its data as a time series of light detection events, and since this data rate is proportional to the overall scene brightness incident on the corresponding SPAD pixel 3, it has been recognized that the ratio θ can be dynamically adjusted based on scene illumination or scene brightness. threshold / w.

[0138] In some implementations, it is envisioned that the neuromorphic circuit 31 increases the data rate under low light conditions and decreases the data rate under high light conditions, as will be discussed below.

[0139] To improve the (grayscale or color) tone mapping used for image reconstruction, the adjustment strategy of the SPAD sensor 2 can be optimized online by dynamically adjusting the parameters of the spiking neuron 20. At least one of the membrane threshold and weights can be adjusted using various mechanisms based on local plasticity rules.

[0140] First, the membrane threshold can be dynamically adjusted by the neuromorphic circuit 31.

[0141] When the spiking neuron 20 generates a pulse, the membrane threshold increases by a predetermined value and then decays to its resting state with a predetermined time constant (e.g., exponentially). In this way, a short-term adjustment is introduced into the membrane threshold, in which the membrane threshold temporarily increases to regulate (i.e., reduce) the output rate of the corresponding spiking neuron 20, and then returns to its resting value, so that the spiking neuron 20 can generate a pulse again.

[0142] Second, the weights can be dynamically adjusted by the neuromorphic circuit 31.

[0143] Biological neurons primarily demonstrate their learning ability through changes in the strength (i.e., weights) of the synapses connecting neurons, adjusting the structure and function of the underlying network. Neuromorphic processing draws inspiration from biology and utilizes brain-like rules of local synaptic plasticity for online learning and real-time adjustment.

[0144] In some implementations, a synapse with adaptive weights is used between SPAD pixel 3 (presynaptic neuron) and spiking neuron 20 (postsynaptic neuron), based on a plasticity mechanism that depends on at least one of the following local variables for on-chip (which may be energy-efficient) and online (i.e., real-time) learning: presynaptic pulse trace (low-pass filter for presynaptic pulses); postsynaptic membrane potential; and postsynaptic pulse trace (low-pass filter for postsynaptic pulses).

[0145] For example, in some implementations, steady-state membrane potential-dependent plasticity (“MPDP”) capable of maintaining the membrane potential between two threshold values ​​can be used. An implementation of this local plasticity rule can be stated as: , Where r is the learning rate, γ is the (learning) coefficient, and θ U It is the upper threshold and, for example, between the membrane threshold and zero, θ D It is the lower threshold and is, for example, zero, and [...] + It's a rectifier bracket.

[0146] In some implementations, calcium MPDP can be used, with the goal of achieving a specific output rate. This implementation of the local plasticity rule can be expressed as: , Among them, S teacher θ(t) is the target output rate, ρ(θ(t)) is the current output rate of the pulse, and f(ρ) is a function that scales the current output rate of the pulse.

[0147] Both synaptic local plasticity rules can modulate the activity of spiking neurons 20 and can be used to adjust the input rate of SPAD pixels 3 in real time.

[0148] Figure 9 The implementation of method 120 is illustrated in a flowchart, and the method will be discussed below.

[0149] This method can be performed by an apparatus as described herein.

[0150] At position 121, a light detection event is generated, as discussed in this paper.

[0151] At position 122, the light detection event is output to the neuromorphic circuit, as discussed in this paper.

[0152] At position 123, multiple spiking neurons are implemented using neuromorphic circuits, and these multiple spiking neurons are used to process light detection events, as discussed in this paper.

[0153] Figure 10 The implementation of method 130 is illustrated schematically in flowchart form, and will be discussed below.

[0154] This method can be performed by an apparatus as described herein.

[0155] At position 131, a light detection event is generated, as discussed in this paper.

[0156] At position 132, the light detection event is output to a neuromorphic circuit that implements multiple spiking neurons, as discussed in this paper.

[0157] At position 133, the ratio of membrane threshold to weight is set to be greater than one, such that the input rate of the light detection event is higher than the output rate of the pulse. Each spiking neuron has a membrane threshold and connections to different single-photon avalanche diode pixels or different groups of single-photon avalanche diode pixels to process the corresponding light detection event. The connections are assigned weights as discussed in this paper.

[0158] At position 134, multiple spiking neurons are used via neuromorphic circuitry to process light-detected events, as discussed in this paper.

[0159] Figure 11 The implementation of method 140 is illustrated schematically in flowchart form, and will be discussed below.

[0160] This method can be performed by an apparatus as described herein.

[0161] At position 141, a light detection event is generated, as discussed in this paper.

[0162] At position 142, the light detection event is output to a neuromorphic circuit that implements multiple spiking neurons, as discussed in this paper.

[0163] At 143, based on the amount of light incident on the single-photon avalanche diode pixel or single-photon avalanche diode pixel group connected to the corresponding spiking neuron, the neuromorphic circuit dynamically adjusts at least one of the weights and membrane thresholds for each spiking neuron, wherein each spiking neuron has a membrane threshold and connections to different single-photon avalanche diode pixels or different single-photon avalanche diode pixel groups to process the corresponding light detection event, wherein the connections are assigned weights as discussed herein.

[0164] At position 144, multiple spiking neurons are used via neuromorphic circuitry to process light detection events, as discussed in this paper.

[0165] It should be recognized that the implementation methods describe the method using an exemplary order of method steps. However, the specific order of method steps given is for illustrative purposes only and should not be construed as a constraint.

[0166] It should be noted that this technology can also be configured as described below.

[0167] (1) An apparatus, wherein the apparatus comprises: A single-photon avalanche diode sensor includes a plurality of single-photon avalanche diode pixels, each single-photon avalanche diode pixel being configured to generate a light detection event and output the light detection event to a neuromorphic circuit; and The neuromorphic circuit implements multiple spiking neurons, wherein the neuromorphic circuit is configured to use the multiple spiking neurons to process the light detection event.

[0168] (2) According to the apparatus of (1), each spiking neuron has a membrane threshold and connections to different single-photon avalanche diode pixels or different groups of single-photon avalanche diode pixels to process corresponding light detection events, wherein the connections are assigned weights.

[0169] (3) According to the apparatus of (2), wherein the ratio of the membrane threshold to the weight is set to be greater than one, such that the input rate of the optical detection event is higher than the output rate of the pulse.

[0170] (4) According to the apparatus of (2), wherein the neuromorphic circuit is further configured to dynamically adjust at least one of the weights and membrane thresholds for each spiking neuron based on the amount of light incident on the single-photon avalanche diode pixel or single-photon avalanche diode pixel group connected to the corresponding spiking neuron.

[0171] (5) The apparatus according to (4), wherein at least one of the weights and membrane thresholds is adjusted based on the local plasticity rules of the corresponding spiking neurons.

[0172] (6) According to the apparatus of (5), the weights are adjusted based on the deviation between the membrane potential and the first threshold and the second threshold.

[0173] (7) According to the device of (5) or (6), the weights are adjusted based on the deviation between the pulse output rate and the target output rate.

[0174] (8) The apparatus according to any one of (5) to (7), wherein when processing a photodetection event, the membrane threshold increases by a predetermined value, and wherein the increase decays with a predetermined time constant.

[0175] (9) The apparatus according to any one of (4) to (8), wherein the amount of light is represented by the input rate of the light detection event to the corresponding spiking neuron.

[0176] (10) An apparatus according to any one of (1) to (9), wherein the single-photon avalanche diode sensor and the neuromorphic circuit are implemented on the same chip.

[0177] (11) A method, wherein the method comprises: Generate light detection events and output the light detection events to a neuromorphic circuit; and Multiple spiking neurons are implemented using neuromorphic circuits, and these multiple spiking neurons are used to process light detection events.

[0178] (12) According to the method of (11), each spiking neuron has a membrane threshold and connections to different single-photon avalanche diode pixels or different single-photon avalanche diode pixel groups to process the corresponding light detection event, wherein the connections are assigned weights.

[0179] (13) According to the method of (12), it further includes setting the ratio of the membrane threshold to the weight to be greater than one, such that the input rate of the photodetector event is higher than the output rate of the pulse.

[0180] (14) The method according to (12) further includes: dynamically adjusting at least one of the weights and membrane thresholds for each spiking neuron by means of a neuromorphic circuit, based on the amount of light incident on the single-photon avalanche diode pixel or single-photon avalanche diode pixel group connected to the corresponding spiking neuron.

[0181] (15) According to the method of (14), wherein at least one of the weights and membrane thresholds is adjusted based on the local plasticity rules of the corresponding spiking neurons.

[0182] (16) According to the method of (15), the weights are adjusted based on the deviation between the membrane potential and the first threshold and the second threshold.

[0183] (17) According to the method of (15) or (16), the weights are adjusted based on the deviation between the pulse output rate and the target output rate.

[0184] (18) The method according to any one of (15) to (17), wherein when processing a photodetection event, the membrane threshold increases by a predetermined value, and wherein the increase decays with a predetermined time constant.

[0185] (19) The method according to any one of (14) to (18), wherein the amount of light is represented by the input rate of the light detection event to the corresponding spiking neuron.

[0186] (20) The method of any one of (11) to (19), wherein the state of each spiking neuron is modeled using membrane potential, and specifically, wherein the state is further modeled using membrane current.

Claims

1. An apparatus comprising: A single-photon avalanche diode sensor includes a plurality of single-photon avalanche diode pixels, each single-photon avalanche diode pixel being configured to generate a light detection event and output the light detection event to a neuromorphic circuit; as well as The neuromorphic circuit implements multiple spiking neurons, wherein the neuromorphic circuit is configured to use the multiple spiking neurons to process the light detection event.

2. The apparatus according to claim 1, wherein, Each spiking neuron has a membrane threshold and connections to different single-photon avalanche diode pixels or different groups of single-photon avalanche diode pixels to process corresponding light detection events, wherein the connections are assigned weights.

3. The apparatus according to claim 2, wherein, The ratio of the membrane threshold to the weight is set to be greater than one, such that the input rate of the photodetector event is higher than the output rate of the pulse.

4. The apparatus according to claim 2, wherein, The neuromorphic circuit is also configured to dynamically adjust at least one of the weights and the membrane threshold for each spiking neuron, based on the amount of light incident on the single-photon avalanche diode pixel or single-photon avalanche diode pixel group connected to the corresponding spiking neuron.

5. The apparatus according to claim 4, wherein, At least one of the weights and the membrane threshold is adjusted based on the local plasticity rules of the corresponding spiking neurons.

6. The apparatus according to claim 5, wherein, The weights are adjusted based on the deviation between the membrane potential and the first and second thresholds.

7. The apparatus according to claim 5, wherein, The weights are adjusted based on the deviation between the pulse output rate and the target output rate.

8. The apparatus according to claim 5, wherein, When processing a light detection event, the membrane threshold is increased by a predetermined value, wherein the increase decays with a predetermined time constant.

9. The apparatus according to claim 4, wherein, The amount of light is represented by the input rate of light detection events to the corresponding spiking neuron.

10. The apparatus according to claim 1, wherein, The single-photon avalanche diode sensor and the neuromorphic circuit are implemented on the same chip.

11. A method comprising: Generate light detection events and output the light detection events to the neuromorphic circuit; as well as Multiple spiking neurons are implemented using neuromorphic circuits, and the optical detection events are processed using these multiple spiking neurons.

12. The method according to claim 11, wherein, Each spiking neuron has a membrane threshold and connections to different single-photon avalanche diode pixels or different groups of single-photon avalanche diode pixels to process corresponding light detection events, wherein the connections are assigned weights.

13. The method of claim 12, further comprising: The ratio of the membrane threshold to the weight is set to be greater than one, such that the input rate of the photodetector event is higher than the output rate of the pulse.

14. The method of claim 12, further comprising: Based on the amount of light incident on the single-photon avalanche diode pixel or the group of single-photon avalanche diode pixels connected to the corresponding spiking neuron, the neuromorphic circuit dynamically adjusts at least one of the weights and the membrane threshold for each spiking neuron.

15. The method according to claim 14, wherein, At least one of the weights and the membrane threshold is adjusted based on the local plasticity rules of the corresponding spiking neurons.

16. The method according to claim 15, wherein, The weights are adjusted based on the deviation between the membrane potential and the first and second thresholds.

17. The method according to claim 15, wherein, The weights are adjusted based on the deviation between the output rate of the pulse and the target output rate.

18. The method according to claim 15, wherein, When processing a light detection event, the membrane threshold is increased by a predetermined value, wherein the increase decays with a predetermined time constant.

19. The method of claim 14, wherein, The amount of light is represented by the input rate of the light detection events to the corresponding spiking neuron.

20. The method according to claim 11, wherein, The state of each spiking neuron is modeled using membrane potential, specifically, the state is further modeled using membrane current.

Images