Optical biasing system for event-based sensors (EBS)

Abstract

An optical biasing system that injects a synthetic light field into the path provided to an event-based sensor, shifting irradiance levels from low-light regions (where noise events are common) to a higher brightness region with more stable noise rates. In some embodiments, the optical biasing system determines the ambient brightness level of the captured scene and outputs a synthetic (e.g., a temporally varying) light field having an irradiance level selected to optimize the signal-to-noise ratio at the determined ambient brightness level. In some embodiments, the optical biasing system determines the ambient brightness level at each pixel and outputs a light field having spatially varying irradiance levels to optimize the signal-to-noise ratio of each pixel. The disclosed system provides a number of benefits over existing EBS noise correction methods, most notably because the disclosed method does not require modifying sensor hardware or using active algorithms to control sensor parameters.

Description

OPTICAL BIASING SYSTEM FOR EVENT-BASED SENSORS (EBS)CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Prov. Pat. Appl. No. 63 / 498,492, filed April 26, 2023, which is hereby incorporated by reference in its entirety.FEDERAL FUNDING

[0002] This invention was made with government support under Award Number FA9451- 20-1-0005, awarded by the U.S. Air Force Research Laboratory. The government has certain rights in the invention.BACKGROUND

[0003] Event-Based Sensor (EBS) imagers - also known as a neuromorphic cameras, silicon retina, or dynamic vision sensors - are imaging sensors that responds to local changes in brightness. Unlike traditional cameras, which capture images as complete frames using a shutter, each pixel inside an EBS imager operates independently and asynchronously, reporting changes in brightness as they occur and staying silent otherwise. Because EBS imagers can detect logarithmic changes in light intensity over a wide illumination range, EBS imagers have higher dynamic range and better low light sensitivity than traditional frame-based imagers. Accordingly, EBS imagers are used to autonomously capture image data in many applications, such as night-time surveillance, autonomous driving, wildlife observation, microscopy, astronomical imaging, etc.

[0004] Existing EBS imagers, however, are inherently limited by sensor transistor circuitry noise and shot noise from photons, particularly at excessively low illumination levels where signals are dominated by parasitic dark current that produce shot noise events that shroud the true signal. Accordingly, there is a desire to increase the signal-to-noise ratio of EBS imagers, particular in low illumination environments.SUMMARY

[0005] Disclosed is an optical biasing system that injects a synthetic light field into the path provided to an event-based sensor, shifting the irradiance levels perceived by the event-based sensor from a low-light region (where noise events are common) to a higher brightness region with more stable noise rates. In some embodiments, the optical biasing system determines the ambient brightness level of the captured scene and outputs a synthetic (e.g., a temporallyvarying) light field having an irradiance level selected to optimize the signal-to-noise ratio at the determined ambient brightness level. In some embodiments, the optical biasing system determines the ambient brightness level at each pixel and outputs a light field having spatially varying irradiance levels to optimize the signal-to-noise ratio of each pixel.|0006] The disclosed optical biasing system significantly reduces sensor noise, which is particularly useful for task applications involving excessively low illumination levels. The disclosed system provides a number of benefits over existing EBS noise correction methods, most notably because the disclosed method does not require modifying sensor hardware or using active algorithms to control sensor parameters.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Aspects of exemplary embodiments may be better understood with reference to the accompanying drawings. The components in the drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of exemplary embodiments.

[0008] FIG. 1 is a diagram abstracting the asynchronous operation of event-based sensor (EBS) pixels to detect changes in scene brightness.

[0009] FIG. 2 illustrates the functionality to tune a number of parameters of the EBS pixel of FIG. 1.

[0010] FIG. 3 is a graph illustrating the per pixel event rate of an EBS camera as a function of scene spectral irradiance levels.

[0011] FIG. 4 is a diagram of an optical biasing system according to exemplary embodiments.

[0012] FIG. 5 is a diagram of an optical biasing system according to an exemplary embodiment.10013] FIG. 6 is a diagram of a spatially varying optical biasing system according to an exemplary embodiment.

[0014] FIG. 7 is a diagram abstracting embodiments of the disclosed optical biasing system as an equivalent electrical circuit.DETAILED DESCRIPTION

[0015] Reference to the drawings illustrating various views of exemplary embodiments is now made. In the drawings and the description of the drawings herein, certain terminology isused for convenience only and is not to be taken as limiting the embodiments of the present invention. Furthermore, in the drawings and the description below, like numerals indicate like elements throughout.(0016] In event-based sensors (EBS), each pixel includes analog circuitry used to detect changes in scene brightness. FIG. l is a diagram abstracting the asynchronous operation of an existing EBS pixel 100 to detect changes in scene brightness. In the example of FIG. 1, the EBS pixel 100 includes a photoreceptor section 110, a source follower (SF) buffer 130, a switched capacitor change amplifier 150, voltage comparators 170, and a refractory period section 190.|0017] As described below, EBS pixels 100 register sequences of log intensity fluctuations above some predefined magnitudes. That function is modelled after the way fluctuations are perceived by the human visual system. Pixels 100 store log intensity values after an event is triggered and await a log intensity change surpassing those stored values. Events are encoded as e(p, x, y, t) using Address Event Representation (AER) protocol, which details the polarity of intensity change p, pixel coordinates x, y on the sensor array, and timestamp t of the occurred event. The AER event stream produced by the EBS is readily processable for fine information extraction in a host of task-specific applications (e.g., night-time surveillance, autonomous driving, wildlife observation, etc.).

[0018] As shown in FIG. 1, the photoreceptor section 110 includes a photodiode 112 that produces a photocurrent I in response to photon radiation (e.g., visible light or infrared or ultraviolet radiation). The photocurrent I is composed of signal photocurrent Ipas well as sensor dark photocurrent Idark- A field effect transistor 114 in the photoreceptor section 110 translates the photocurrent I into a voltage Vpthat logarithmically scales with input intensity (irradiance) that is directly proportional to the photocurrent I.

[0019] The source follower buffer 130 attenuates rapid spikes in the converted voltage Vpand provides the attenuated signal to the switched capacitor change amplifier 150, which compares the voltage Vpcurrently being sampled to the previously sampled voltage (stored in capacitor C2) and outputs the difference as voltage Vd. The comparators 170 compare the voltage Vdto thresholds 0ONand 0OFFand output binary signals (ON and OFF) in response to positive polarity events (i.e., the logarithmically scaled input intensity increasing by more than the threshold 0OW) and positive polarity events (i.e., decreasing by more than the threshold 0OFF). In the refractory period section 190, a rest pulse is sent out to hold Vdfor a configuredrefractory periodrefr, during which the pixel 100 does not respond to stimuli change.Accordingly, the pixel 100 detects an event when the following is true:where Ip[n] is the photocurrent Ipcurrently being sampled and Ip[n — 1] is the previously sampled photocurrent.

[0020] As shown in FIGS. 1 and 2, the EBS pixel 100 provides functionality to tune a number of parameters. The event thresholds 0OWand 0OFF, which are based on the ratio between the bias current I OFF or I0Nand the current Idcaused by the voltage Vd, can be tuned by adjusting the bias currents I0FFand / or I0N. The length of the refractory period ^refrcan be controlled by adjusting the current Irefr. And the sensor’s ability to respond fast (temporal bandwidth) is dictated by the readout circuity biases Iprand ISf relative to input illumination levels.

[0021] As briefly mentioned above, conventional EBS pixels 100 are inherently limited by sensor transistor circuitry noise (e.g., reset and readout transistors) and shot noise from unrelated arrival photons and electrons (e.g., originating at the photodiode 112) that generate random spikes in Vp. As described below, noise can be particularly excessive at low illumination levels where signals are dominated by parasitic dark current Idark, which produces shot noise events that shroud the true signal Ip. In many applications, noise greatly limits temporal contrast sensitivity and overall signal-to-noise ratio (SNR). Additionally, high EBS event rates may cause readout bus saturation and / or issues with motion blur bandwidth limitations, which introduce timing jitter and reduce the number of true brightness change events.

[0022] FIG. 3 is a graph illustrating the per pixel event rate of a DAVIS346 camera measured as a function of scene spectral irradiance levels at 633 nm using an optical power meter. As shown in FIG. 3, noise background event rates rise sharply after traversing into the low-light region 310, which is arbitrarily modelled to the left of a black vertical bar 350. Within the low-light region 310, sensor noise event rates are extremely high and peak at around 0.12 pW / cm2at 633 nm with a pixel event rate of 15.5 Hz. Accordingly, the usability of conventional EBS pixels 100 in low-light conditions is greatly limited by noise. By contrast, event rates quiet down and stabilize to approximately 0.2 Hz in a high-brightness region 390, indicating that higher background illumination levels improve noise minimization and background event rates.Optical Biasing System[00231 Accordingly, an optical biasing system is disclosed that reduces noise by shifting the perceived irradiance levels away from the low-light region 310 and into the region 390 with more stable noise rates.

[0024] FIG. 4 is a diagram of an optical biasing system 400 according to exemplary embodiments.

[0025] As shown in FIG. 4, the optical biasing system 400 improves the functioning of existing EBS cameras 410 (e.g., having an array of pixels 100 described above) capturing a scene 401 via an imaging path 411. To do so, the optical biasing system 400 includes a light source 420 that emits a light field via an injection path 422 and a beamsplitter 450 that combines light captured from the scene 401 and the light field emitted by the light source 420 and outputs the combined light field along a combined path 455 to the EBS camera 410. In some embodiments, the optical biasing system 400 may include neutral density filters 430 that filter the light captured from the scene 401 and the light field emitted by the light source 420.

[0026] By injecting light from the light source 420 into the imaging path, the optical biasing system 400 shifts the irradiance levels perceived by the EBS camera 410 away from the low- light region 310 (where noise events are more common) and into the region 390 with more stable noise rates. Accordingly, the optical biasing system 400 reduces the noise that can prevent existing EBS pixels 100 from performing in low-irradiance environments.]0027] While injecting a light field from an external light source 420 reduces noise, doing so may also unintentionally reduce signal event rates by reducing the overall object contrast. The amount of noise reduction and the amount of signal reduction both vary depending on the irradiance of the scene 401 and the irradiance of the injection 422 from the external light source 420. Therefore, in some embodiments, the optical biasing system 400 may select the optimal irradiance level for improving signal-to-noise ratio based on the irradiance of the scene 401.

[0028] FIG. 5 is a diagram of a dynamic optical biasing system 500 according to an exemplary embodiment.

[0029] As described above with reference to FIG. 4, an EBS camera 410 capturing a scene 401 (e.g., via an imaging lens 501) via a beamsplitter 450 while a light source 420 (e.g., a light emitting diode and a collimating lens 521) injects a light field into the imaging path via the beamsplitter 450. Meanwhile, in the embodiment of FIG. 5, the optical biasing system 500 also includes an ambient light sensor 580 (e.g., a photodiode and a condensing lens 581) thatmeasures the ambient brightness of the scene 401 and a controller 590 (e.g., an electronic processor) that selects the optimal irradiance level for improving the signal-to-noise ratio at the ambient brightness measured by the ambient light sensor 580. For instance, the optimal irradiance level for various ambient brightness levels may be determined during laboratory tests and stored (e.g., in a lookup table) and selected by the controller 590, which controls the light source 420 to inject light into the sensor path at the optimal level. In other embodiments, laboratory data may be used to determine a formula, executed by the controller 590, calculate the optimal irradiance level based on the ambient brightness level of the scene.

[0030] Accordingly, the dynamic optical biasing system 500 can inject a temporally varying the light field having an irradiance level selected to optimally reduce signal-to-noise ratio.

[0031] Because the ambient brightness levels may vary across the scene 401, the optimal irradiance level may vary for different pixels 100 of the EBS camera 410. Accordingly, in some embodiments, the optical biasing system 400 may output a spatially varying light field having irradiance levels selected to optimally increase the signal-to-noise ratio of each pixel 100.

[0032] FIG. 6 is a diagram of a spatially varying optical biasing system 600 according to an exemplary embodiment.

[0033] In the embodiment of FIG. 6, the ambient light sensor 580 (e.g., an array of photodiodes) measures the ambient brightness of the scene 401 captured by each pixel 100, the controller 590 selects the optimal irradiance level to increase the signal-to-noise ratio at the ambient brightness level of that pixel 100, and the light source 420 (e.g., an LED display) outputs a spatially varying light field wherein light is injected into each pixel 100 at the irradiance level selected for that pixel 100 by the controller 590.

[0034] FIG. 7 is a diagram abstracting embodiments of the disclosed optical biasing system as an equivalent electrical circuit. As described above, EBS pixels 100 provide functionality to tune a number of parameters (the event threshold 0, the refractory period Areyr, and / or the bandwidth Bpr) by adjusting various bias currents (J0FF and I0N, Irefr, and / or Iprand / sy). As shown in FIG. 7, the optical injection 710 provided to each pixel by the disclosed system 400 can be abstracted as providing an external bias current IBthat, much like the bias currents of prior art EBS pixels 100, can be tuned (e.g., by the controller 590) to provide an optimal signal to noise ratio for the ambient brightness of the scene (e.g., determined by the ambient brightness sensor

[0035] By shifting up the perceived irradiance levels away from the low-light region and into a region with stable noise rates (as shown in FIG. 3), the disclosed optical biasing system significantly reduces sensor noise, which is particularly useful for task applications involving excessively low illumination levels.Benefits Over Existing Noise Correction Methods

[0036] The disclosed EBS noise correction method provides a number of benefits over existing EBS noise correction methods, most notably because the disclosed EBS noise correction method does not require modifying sensor electronics or filters or the use of active algorithms for controlling key sensor parameters (contrast thresholds, refractory period, photoreceptor bias currents, etc.).

[0037] The most common existing noise reduction approaches utilize post-process background activity filters (BAF) to remove events with a lack of temporal correlation to other pixels in their spatial vicinity. The memory requirements of post-process background activity filters, however, make those methods difficult to implement, particular at very high event rates and / or in the low-light region 310. Furthermore, a faster noise event stream increases the probability of signal events being filtered away when a spatiotemporal correlation occurs with a noise event.

[0038] Less common denoising approaches are through hardware modifications, for example hardware neural networks and algorithms implemented with Field-Programmable Gate Arrays (FPGA) and other integrated circuits. Those methods delay processing speeds and have the drawback of dropping events when objects pass between adjacent binned areas for filtering. The various hardware denoisers are also subject to tradeoffs between memory minimization and denoising accuracies which may hinder real time performance or falsely remove desired events. Additionally, attempts have also been made to limit noise by expanding pixel size and increasing total photon collection area such that more light is accumulated. However, those expanded pixels sizes comes at the expense of sensor spatial resolution, which is already limited by complex circuitry.

[0039] While preferred embodiments have been described above, those skilled in the art who have reviewed the present disclosure will readily appreciate that other embodiments can be realized within the scope of the invention. Accordingly, the present invention should be construed as limited only by any appended claims.

Claims

CLAIMSWhat is claimed is:

1. A system for optically biasing an event-based sensor that captures light from a scene, the system comprising: a light source that outputs an external light field; and a beamsplitter that combines the light from the scene and the external light from the light source to form a combined light signal and provides the combined light signal to the event-based sensor.

2. The system of claim 1, further comprising: an ambient light sensor that determines an ambient brightness level of the light from the scene; and a controller that: selects an irradiance level based on the ambient brightness level of the light from the scene; and controls the light source to output the external light field at the selected irradiance level.

3. The system of claim 2, wherein the ambient light sensor comprises a photodiode.

4. The system of claim 2, wherein the controller stores a plurality of irradiance levels, each associated with a range of ambient brightness levels, and selects the irradiance level associated with the ambient brightness level determined by the ambient light sensor.

5. The system of claim 4, wherein the irradiance levels stored by the controller are pre-selected to optimize the signal-to-noise ratio within the associated range of ambient brightness levels.

6. The system of claim 2, wherein the controller uses a formula to calculate the irradiance level based on the ambient brightness level of the scene.

7. The system of claim 2, wherein: the event-based sensor includes a plurality of sensor pixels;the ambient light sensor determines an ambient brightness level of the light captured by each of the plurality of sensor pixels; the controller selects an irradiance level for each of the plurality of sensor pixels based on the ambient brightness level of the light captured by each sensor pixel; and the light source comprises a plurality of emitter pixels, each emitter pixel outputting an external light field to one of the plurality of sensor pixels having the irradiance level selected by the controller based on the ambient brightness level of the light captured by the sensor pixel.

8. The system of claim 7, wherein the ambient light sensor comprises an array of photodiodes.

9. The system of claim 7, wherein the light source comprises an array of light emitting diodes.

10. The system of claim 1, wherein the system adds a bias to a photocurrent produced by a photodiode of the event-based sensor.

11. A method for optically biasing an event-based sensor capturing light from a scene, the method comprising: outputting external light field from a light source; combining the light from the scene and the external light field from the light source to form a combined light signal; and providing the combined light signal to the event-based sensor.

12. The method of claim 11, further comprising: determining an ambient brightness level of the light from the scene; and selecting an irradiance level based on the ambient brightness level of the light from the scene; and controlling the light source to output the external light field at the selected irradiance level.

13. The method of claim 12, wherein the ambient brightness level is determined using a photodiode.

14. The method of claim 12, wherein selecting the irradiance level based on the ambient brightness level of the light from the scene comprises: storing a plurality of irradiance levels, each associated with a range of ambient brightness levels; and selecting the irradiance level associated with the ambient brightness level of the scene.

15. The method of claim 14, wherein the stored irradiance levels are pre-selected to optimize the signal-to-noise ratio within the associated range of ambient brightness levels.

16. The method of claim 12, wherein selecting the irradiance level based on the ambient brightness level of the light from the scene comprises using a formula to calculate the irradiance level based on the ambient brightness level of the scene.

17. The method of claim 12, wherein: the event-based sensor includes a plurality of sensor pixels; determining the ambient brightness level comprises determining an ambient brightness level captured by each of the plurality of sensor pixels; selecting the irradiance level comprises selecting an irradiance level for each of the plurality of sensor pixels based on the ambient brightness level of the light captured by each sensor pixel; and outputting the external light field comprises outputting an external light field to each of the plurality of sensor pixels having the irradiance level selected based on the ambient brightness level of the light captured by the sensor pixel.

18. The method of claim 17, wherein the ambient brightness level of the light captured by each sensor pixel is determined using an array of photodiodes.

19. The method of claim 17, wherein the external light field is output to each of the sensor pixels using an array of light emitting diodes.

20. The method of claim 11, wherein the external light field adds a bias to a photocurrent produced by a photodiode of the event-based sensor.

Images