Systems and methods for enhancing performance of event cameras

Abstract

Embodiments pertain to an event camera system including: an event camera; an optical encoder positioned between a scene and the event camera configured to optically encode an image of the scene into one or more encoded optical channels that are optically combined for capture of the event camera; and a processor configured to decode the one or more encoded optical channels in the capture of the event camera. The system may further comprise an optical multiplexor configured to perform the optical combining of the one or more optical channels. The system may further comprise an optical device positioned between the optical encoder and the scene.

Description

SYSTEMS AND METHODS FOR ENHANCING PERFORMANCE OF EVENT CAMERASCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This patent application claims priority and / or benefit from US provisional patent applications 63 / 531,323, filed August 8, 2023, and 63 / 545,372, filed October 24, 2023, both titled "Method and System for Removing Noise from Images Captured with Events Cameras by Multiplexing a Duplicated Image onto the Camera's Sensor", which applications are incorporated herein by reference in their entirety.FIELD

[0002] Embodiments disclosed herein relate generally to systems and methods for image capture using an event camera.BACKGROUND

[0003] An event camera is a type of vision sensor that differs from traditional frame-based cameras. Instead of capturing images at fixed intervals, event cameras may detect changes in the scene asynchronously and at a high temporal resolution. Event cameras may be particularly useful in scenarios where traditional cameras struggle, such as in environments with high-speed motion or varying lighting conditions.

[0004] The description above is presented as a general overview of related art in this field and should not be construed as an admission that any of the information it contains constitutes prior art against the present patent application.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Non-limiting examples of embodiments disclosed herein are described below with reference to figures attached hereto that are listed following this paragraph. The drawings and descriptions are meant to illuminate and clarify embodiments disclosed herein and should not be considered limiting in any way. Like elements in different drawings may be indicated by like numerals. Elements in the drawings are not necessarily drawn to scale.

[0006] FIG. 1 shows a graph of event generation by an event camera.

[0007] FIG. 1 and FIG. 3 illustrate the noise difference between event and RGB cameras in various light conditions.

[0008] FIGS. 4A-4C are block diagrams of an event camera system according to some disclosedembodiments.

[0009] FIG. 4D is an illustrative scene and corresponding combined encoded optical channels according to some disclosed embodiments.

[0010] FIG. 5 is a block diagram of an event camera system according to some disclosed embodiments.

[0011] FIG. 6A is a schematic drawing of an event camera system having an expanded FOV according to some disclosed embodiments.

[0012] FIG. 6B is a block diagram showing an MZ configuration for use as an encoder in an event camera system according to some disclosed embodiments.

[0013] FIGS. 6C and 6D are illustrative images showing events captured by the event camera of FIG. 6A.

[0014] FIGS. 7A and 7B are schematic drawings of event camera systems for capturing color event images according to some disclosed embodiments.

[0015] FIG. 8 shows point clouds representing events from two optical channels before and after encoding and multiplexing.

[0016] FIG. 9 is a block diagram of an exemplary computing device consistent with some disclosed embodiments.

[0017] FIG. 10 shows ROC curves generated from a denoising evaluation.

[0018] FIGS. 11A and 11B show an experimental apparatus and spatiotemporal three-dimensional (3D) representation of denoising of events in a denoising evaluation.DETAILED DESCRIPTION

[0019] Event cameras, aka Dynamic Vision System (DVS), or neuromorphic cameras, may represent a dramatic shift from conventional frame-based imaging technology. While traditional cameras operate by sampling a scene at a fixed temporal rate, typically around 30 frames per second, event cameras may detect dynamic changes in illumination. This difference may lead to several advantages such as, for example, a higher dynamic range, a higher sampling rate, and lower power consumption.

[0020] The potential applications for these sensors in domains where high dynamic range and speed are critical are diverse, ranging from robotics and autonomous vehicles, where they may enhance obstacle detection and navigation, to augmented and virtual reality, where they may offer low-latency tracking.Their utility may extend to particle tracking velocimetry in fluid dynamics, providing detailed, high-speed observations of fluid flow. In the field of surveillance, event cameras may aid in detecting unusual activities or tracking fast-moving objects due to their high dynamic range and ability to capture fast motion. Sports analytics may also benefit from the high-speed capture capabilities of event cameras for analyzing fast sports movements and improving performance understanding. High-speed manufacturing and quality control processes may leverage event cameras for rapid and efficient monitoring and inspection of products on assembly lines. The field of biomedical imaging may utilize these cameras for capturing high-speed biological events, like the motion of cells, with high precision. Astronomy may also benefit from the high dynamic range of event cameras for better observations of celestial events involving rapid changes in brightness. High-dynamic range photography and videography may handle scenes with a high contrast ratio, from deep shadows to bright highlights, which is challenging for traditional cameras.

[0021] The pixel circuit in a typical event camera may use a fast logarithmic photoreceptor, a differential amplification circuit that accurately amplifies the amount of change, and a low-cost two-transistor comparator. In some examples, each pixel may generate an event asynchronously and independently whenever it detects a brightness change that exceeds a predefined threshold rj. This may be mathematically expressed as:

[0022] is the brightness level (log intensity) at pixel (x, y) at time tirtlastis the time of occurrence of the previous event and etE {+1, —1} is the event polarity corresponding to the sign of the brightness change. As shown in FIG. 1, the output of the camera is a variable sequence of digital "events" or "spikes," each event representing a change of brightness at a pixel at a particular time. With reference to FIG. 1, when the voltage increases and the change is greater than a threshold, an ON event will be generated and when the voltage decreases and a change is greater than a threshold, an OFF event will be generated.

[0023] Despite the advantages, event cameras may produce uninformative background activity events, even under constant illumination, limiting their performance. The generation of background activity may be influenced by various factors such as, for example, charge injection, the leakage of the transistor section of the reset switch, shot and thermal noise. The occurrence of background activity may be random and less frequent than genuine events. Moreover, similar to the "hot pixel" phenomenon in traditional image sensors, event cameras may continuously output events if a pixel cannot be adequately reset, which resultsin further noise. In some examples, the terms "background activity" and "noise" may herein be used interchangeably.

[0024] While event cameras may generate noise in any lighting conditions, FIGS. 2 and 3 illustrate how an event camera may generate additional noise in poor lighting conditions and illustrate the difference to frame-based cameras. As shown in FIG. 2, the light intensity (upper images) is inversely correlated to the noise level of captured events (lower images), i.e., with the light intensity gradually decreasing, the noise level increases in the event frame. The main reason for this low-light noise is that the continual sampling (or differential sampling) in the event camera cannot smooth noisy signals.

[0025] In traditional cameras, such noise input may be largely suppressed by the smoothness of the integration function, thus potentially maintaining good imaging quality. However, in event cameras, the noise is much more pronounced due to the continuous differential sampler, and logarithmic operations may further amplify the noise, leading to the generation of BA. Therefore, an objective technical problem to be solved may be defined as the event camera based denoising.

[0026] Background activity differs from signal events in that there is a higher spatiotemporal correlation between signal events. In spatiotemporal images, signal events are denser, while background activity is sparser. This distinction may be leveraged by embodiments of the invention disclosed herein to, e.g., denoise the image, for example, by evaluating the spatiotemporal density of events. However, it should be appreciated that it may be challenging to differentiate between genuine events and noise based solely on a single event. It is noted that embodiments disclosed herein may be combined with known methods and / or systems.

[0027] Multiple noise filtering methods for event-based data have been proposed, such as a Nearest Neighbor (NNb) filter, variations of NNb filters, and other filters that consider differing polarity, refractory period, and inter-spike interval. However, many of these filters may be too simple, and, for a complex scene with a large number of noise events, may not effectively remove noise.

[0028] Other methods may employ Gaussian convolution in the temporal dimension to extract correlation or extend this approach by applying convolution across both spatial and temporal dimensions. These methods require each event to participate in multiple calculations, including critical comparisons.

[0029] Other methods may slice the 3D event stream in time and obtain multiple frame images, and then use traditional image / video denoising methods. However, such processing will lose the most importantcharacteristic of DVS, i.e., low latency during imaging and may cause a large irreversible loss of events as well as an irreversible loss of time attributes of an event.

[0030] When the trajectory of a target is known, it may be possible to calculate the temporal correlation of events generated by a single pixel through repeated or multiple recordings. However, applying this approach in real-world scenarios presents significant challenges as it is often impractical, if not impossible, to have advance knowledge of every target's motion information. Additionally, considering the cost associated with event-based cameras, deploying them in a way that would allow for such comprehensive tracking and analysis may be prohibitive.

[0031] A further disadvantage of current event cameras is that they can only capture changes in overall brightness without any spectral sensitivity, i.e.: "grayscale" event images. Unfortunately, there are fundamental limitations to adopting traditional color and spectral imaging techniques found in conventional cameras for DVS because of their specific sensing technology. For instance, attempts have been made to create DVS color-sensitive cameras using the Bayer filter technique commonly found in regular cameras. The popular Bayer filter generally has one red (R) and one blue (B) channel, along with two green (G) channels for a 2x2 binning configuration. However, this technique significantly compromises spatial resolution, which is already limited in DVS cameras. The resolution of DVS cameras is lower than that of conventional cameras due to the additional circuitry added to each pixel sensor, limiting pixel size minimization. As a result, there is a fundamental limitation to designing color event imagers at a similar resolution as achieved with conventional cameras. Another proposed method uses RGB flickering illumination to capture color images of static objects with a single event camera. The method does not sacrifice the native spatial resolution of the sensor, however, it significantly compromises the DVS's temporal resolution, and it is not designed to capture color events that represent instant changes in the scene.

[0032] Capturing multiple spectral bands beyond the RGB colors may be even more complicated since most of the spectral imaging techniques involve some kind of temporal scanning (e.g., push broom or spectral multiplexing), which contradicts the temporal-sensitive mechanism of DVS. Temporal scanning may conflict with the essence of DVS as an object-events sensor, as it generates events due to scanning, which hinders the detection of object events. While there are non-scanning spectral sensing techniques used with regular cameras, they are less suitable for DVS cameras because they sacrifice the spatial resolution for spectral sensing, which, as previously mentioned, is limited with DVS.

[0033] It would therefore be desirable to overcome the deficiencies of DVS such as, for example, by reducing background activity and to also enhance the practicality of event cameras and expand their potential use such as, for example, by enabling color and / or spectral imaging.

[0034] Disclosed herein are systems and methods that may enhance the performance of event cameras by reducing background activity and enabling, for example, enlarged field of view (FOV), and / or color, stereoscopic, polarized and / or spectral imaging. The proposed systems and methods overlap optical data from multiple optical channels onto a single event camera image sensor (or "pixel array") taking advantage of the fact that captured event camera data is sparse. It should be appreciated that this approach may be uniquely suited to event cameras and may not be applied to traditional conventional imaging systems as the captured data in such systems is not sparse.

[0035] An optical channel may be spatially encoded before capture by the event camera, and the captured event data may be decoded after acquisition, e.g., into event representations of the scene. In some embodiments, the number of event representations may correspond to the number of encoded optical channels. In some embodiments, the multiple optical channels may be optically multiplexed into a single image / channel before being provided to the event camera for capture. The term "event data" relates to the imaged scene.

[0036] The term "capture" as it relates to an event camera includes capture of event data in relation to a scene.

[0037] This approach may be used to extend the performance of DVS in many ways. This system and method may be used with a single event camera image sensor to detect and remove background activity, increase the resolution and / or field of view (FOV) of two-dimensional event imaging (without compromising its spatial resolution), to realize, for example, a panoramic (e.g., up to 360°) event camera, a multi-view event camera, to capture color event data, and / or to capture multi-dimensional event images, such as, for example, spectral, polarimetric, and / or light-field events. It is noted that the term "color" is not necessarily limited to the visible spectrum, but may also encompass wavelengths of the non-visible spectrum.

[0038] Although embodiments disclosed herein may pertain to systems that include an event camera, this should by no means be construed in a limiting manner. Embodiments of systems may not necessarily be considered to include the event camera. Embodiments of systems disclosed herein may be employed in conjunction with event cameras that may not necessarily be considered part of the system.

[0039] FIGS. 4A-4C are block diagrams of an event camera system 100 according to some disclosed embodiments. As shown, system 100 may include, for example, optical devices 110, optical encoders 120, an optical multiplexor 130, an event camera 140, and a processor 150. The output of the one or more optical encoders 120 are encoded optical channels 122. System 100 captures events from a scene 102. The term "optical" as used herein to describe components (optical device, optical encoder, etc.) implies that the image is transmitted through the device at least partially, or fully, in an optical form. Scene 102 may be any object or scene for capture by event camera 140.

[0040] As shown in FIG. 4A, one or more (e.g., N) optical devices (110-1.. ,110-N) may be provided. Optical devices 110 may include an optical element, such as, for example, a lens, lens assembly, color filter, spectral filter, mirror, and / or prism. In some embodiments, one or more of optical devices 110 may capture a different color, such as by use of a color filter placed in optical device 110. In some embodiments, each one of the optical devices 110 may capture a different color, such as, for example, by use of a color filter placed in optical device 110.

[0041] Although embodiments may pertain to use color-based encoding, this should not be construed as limiting. Accordingly, additional or alternative scene characteristics may be used for the purpose of encoding (e.g., spatial, spectral, phase, etc.). One or more optical devices may be operably coupled with one or more encoders.

[0042] In some embodiments, at least two or all of the optical devices 110 are substantially identical. In some embodiments, at least two or each of optical devices 110 captures a different spectral band. In some embodiments, at least two or each of optical devices 110 captures a different polarization. In some embodiments, at least two of the optical devices 110 capture a stereoscopic view. In some embodiments, at least two or each of optical devices 110 captures a substantially different FOV and / or overlapping FOVs. In some embodiments, at least two or each of optical devices 110 forms part of a plenoptic array. In some embodiments, such as shown in FIG. 4B, no optical device 110 is provided and scene 102 is captured directly by one or more encoders 120. In some embodiments, such as shown in FIG. 4C, a single optical device 110 may be provided between scene 102 and encoder 120.

[0043] It should be appreciated that any combination of the above arrangements may be provided including, for example, one or more optical devices 110 may be placed before only a portion of encoders 120. In some examples, an optical device may not be configured to capture a certain color.

[0044] In some embodiments, such as shown in FIGS. 4A and 4B, N optical encoders 120-1...120-N may be provided (e.g., each) producing a corresponding encoded optical channel 122-1.. ,122-N. In some embodiments, optical encoder 120 may spatially encode the output received from optical device 110 (FIGS. 4A, 4C) or directly encode scene 102 (FIG. 4B) to produce an optical channel 122. In some embodiments, the encoding may include optically shifting an image received from (e.g., each) optical device 110 or from scene 102. In some embodiments, spatial encoding may be according to an (e.g., predefined) pattern. In some embodiments, spatial encoding may include, for example, duplication and / or blurring, and / or other optical effects (e.g., purposely introducing codes in the form of "artifacts"). In some examples, an encoder may be configured or include an optical device. In some examples, an encoder may receive light from the scene without employing an optical device ("directly encodes scene").

[0045] Optical encoder 120 may be realized by various optical methods, such as, for example, employing a shear plate, a birefringent crystal prism, a diffractive optical element such as a phase mask or a blazed grating based on a point spread function, or a Mach-Zehnder (MZ) configuration (see FIG. 6B). In some embodiments, such as shown in FIG. 4C, a single optical encoder 120 may create N optical channels 122, for example, either from an optical device 110 as shown, or directly from scene 102 (see also the embodiment of FIG. 7B).

[0046] FIG. 4D shows scene 102 and corresponding combined encoded optical channels that show spatial encoding including position shifting and duplication. It should be appreciated that the provided figures show particular embodiments and further combinations and / or options are possible, such as encoding some of scene 102 but transmitting other parts or aspects of scene 102 without encoding (see FIG. 6B). In some embodiments, a single encoded optical channel 122 may be sufficient, for example, for detecting noise and / or unwanted background activity introduced by event camera 140.

[0047] Optical multiplexor 130 may optically combine the encoded optical channels 122 of the N or single optical encoders 120 into a single combined (or overlapped) optical channel provided to event camera 140 for capture of events. As shown in FIG. 4C, in some embodiments, encoder 120 produces encoded optical channels 122 as a combined overlapped output and no optical multiplexor 130 is required. As used herein the term "combine" refers to either of multiplexing together or overlapping output. In some embodiments, optical encoder 120 is a part of optical multiplexor 130 and where only the figure reference 130 is used in the figures, it may be understood to include both of optical encoder 120 and optical multiplexor 130.

[0048] Processor 150 may be a computing device such as computing device 500 described herein. Processor 150 may be in data communication with the components of system 100. Processor 150 mayoperate on a single computing device or may be separate computing devices that may include or may be in communication with a transitory and / or non-transitory computer readable medium (such as memory 512) containing instructions that when executed by at least one processor (such as processor 510) are configured to perform one or more of the functions and / or operations necessary to provide the functionality described herein. Where system 100 may be said herein to provide specific functionality or perform actions, it should be understood that the functionality or actions may be performed by a relevant processor such as processor 510 that is part of processor 150, that may call on other components of system 100 and memory such as memory 512 that may include instructions which, when executed by processor 510 may cause the execution of a method or process described herein.

[0049] In use, 1 to N encoded optical channels 122 may be multiplexed together to be superimposed on an image sensor of event camera 140. In some embodiments, one image of scene 102 may not be encoded (for example, pass through directly from optical device to multiplexor 130) but all other optical channels 122 may be encoded. Following capture by event camera 140, (e.g., each) optical channel 122 may be decoded by processor 150 according to its encoding.

[0050] In some examples, "decoding" as described herein implies the process whereby processor 150 may extract / identify the event data of images of scene 102 (and image properties, such as, for example, colors / spectral data according to the optical devices 110 used) from the encoded and multiplexed / overlapped captured event data.

[0051] It should be appreciated that the total image events captured may be N times larger than that of a conventional event camera. Since event data is typically highly sparse in the spatial-temporal domain, the probability of exact overlapping of two events originating from different optical channels 122 is negligible. As above, each optical channel 122 is spatially encoded, thus allowing extraction of its data from the captured multiplexed event data of event camera 140 by algorithmic or optical decoding.

[0052] FIG. 5 is a schematic drawing of event camera system 100 according to some disclosed embodiments. With reference to FIG. 5: the events generated by the camera may be given by:

[0053] where Tj(-) denotes a spatial encoding operator of the / 'th optical channel 122, Events{-}is the event-generating operator, and n(x, y, t) is a noise component. lt x,y, t) denotes the / 'th optical channel 122 carrying specific image information (such as, for example, the / 'th FOV, viewing direction, spectral component, polarimetric component, etc.).

[0054] The spatial encoding, Tj(-) , may be obtained, for example, by geometrical transformation of the image (e.g., rotation, shifting, flipping, etc.) from scene 102, or by blurring or distorting the image by encoders 120-1.. ,120-N. In some embodiments, a spatial encoding ?;(■) involving superimposing two replicas at a predefined distance of the image may be used for the purpose of denoising the event data.

[0055] The decoding of events (EI...EN) may be performed by applying an inverse, ortype of pseudo inverse of ?;(■)• Decoding may be performed by processor 150 or may be performed optically.

[0056] FIG. 6A is a schematic drawing of event camera system 100 having an expanded FOV according to some disclosed embodiments. In the example shown in FIG. 6A, Li and L2 are schematically illustrated as lenses used in optical devices 110-1 and 110-2 (N=2). In some examples, optical devices 110-1 and 110-2 may have slightly overlapping FOVs (FOV-1 and FOV-2) such that together the combined FOV-3 may be almost double what would be supported by event camera 140 alone without compromising the spatial resolution of event camera 140. Ti and T2 correspond to optical encoders 120-1 and 120-2. Multiplexor 130 includes mirrors Mi and M2 as well as an optical combiner C. In some embodiments, the FOVs may be nonoverlapping. In some embodiments, the FOVs may be adjacent.

[0057] In the embodiment of FIG. 6A, the multiplexed captured events may be given by:

[0058] In some embodiments, T^l^x, y, t)) = / (%, y, t), and T2= l2(x, y, t) + lShifted x,y, t) = / 2(%, y, t) + / (% + A, y + A, t). The shift inmay be realized by various optical methods, such as, for example, employing a shear plate, a birefringent crystal prism, a diffractive optical element such as, for example, a phase mask or a blazed grating based on a point spread function, or a Mach-Zehnder (MZ) configuration (FIG. 6B).

[0059] FIG. 6B is a block diagram showing an MZ configuration for use as an encoder / multiplexor 130 in an event camera system 100 according to some disclosed embodiments. As shown in FIG. 6B beam splitters 610 and planar mirrors 612 are arranged on the optical path. In the embodiment of FIG. 6B, a first image 104-1 of scene 102 is optically directed to event camera 140 without any encoding to form unencoded optical channel 122-1 (performingwhile a second image 104-2 of scene102 is optically encoded by MZ encoder 120 to form encoded optical channel 122-2 before being combined with image 104-1 in multiplexor 130.

[0060] By adjusting the position and orientation of these mirrors optical channel 122-2 may be a shifted duplicate of image 104-2 (performingv.y. tJ = / 2,.y, t) + 1 + A, y + A;t).jn(3). In some embodiments, such as for noise detection, only encoded optical channel 122-2 may be transmitted (without unencoded channel 122-1) such that after decoding the spatially encoded channel 122-2, pixels that were not unencoded may be identified as noise.

[0061] FIG. 6C shows events captured by the event camera 140 of FIG. 6A from the overlapped FOV-1 and FOV-2 channels (1±and 12), and FIG. 6D shows the same event capture after decoding and having a substantially doubled FOV-3.

[0062] FIGS. 7A and 7B are schematic drawings of event camera systems 100 for capturing color event images according to some disclosed embodiments. In FIG. 7A, Li- L3 are optical devices 110 (lenses) 110-1 to 110-3 (N=3), where (e.g., each) optical device 110 includes a respective color filter F1-F3. Optical devices 110-1 to 110-3 therefore each capture a different color of scene 102. Encoder / multiplexor 130 includes mirrors Mi and M2as well as an optical combiner C. The combined encoded optical channels 122-1...122-3 may be captured by event camera 140, and the captured events may be decoded (such as by processor 150) into the source optical channels based on the applied optical encoding to derive colored event information.

[0063] In FIG. 7B, scene 102 is imaged by a single optical device 110. Encoder 120 may include a diffractive optical element such as a phase mask or a blazed grating based on a point spread function that produces combined encoded optical channels 122-1....122-3 at the single output encoder 120. The combined encoded optical channels 122-1 to 122-3 may be captured by event camera 140, and the captured events may be decoded (such as by processor 150) into the source optical channels 122-1...122-3 based on the applied optical encoding to derive colored event information.

[0064] In some embodiments, system 100 may also be used for reduction of background activity without the need to map the event stream into frame images in advance, and / or the need for multiple recordings or cameras to thereby avoid the drawbacks that may be associated with the denoising methods described above. By treating events as point clouds in 3D space e^x^yi, tt) E M3it may be possible to leverage their intrinsic asynchronous and sparse nature to achieve full sensor resolution. As above, due to the sparse nature of the event data it is assumed that multiple encoded channels 122 of a scene 102 may be projected onto the full sensor array of event camera 140 without events coinciding with each other. By exploiting the co-occurrence of the events from optical channels 122 alongside the fact that background activity noise isindependent of the events from the optical channels 122, the actual object events may be separated from the noise events. In some embodiments, a single encoded channel may be sufficient for detection of noise.

[0065] Image, / (%, y, t) may be encoded {e.g., using encoder 120) T ■ l(x,y, t) for example, by adding a shifted replica T onto the full sensor array of event camera 140 such that the captured signal may be given by:E(x,y, t) = T1- l(x,y, t) + N x,y, t) = l x,y, t) + T ■ l x, y, t) + N(x,y, t), (4)

[0066] where N x,y, t) is the background activity noise component, and

[0067] FIG. 8 shows point clouds representing events from two optical channels 122 before and after encoding and decoding. FIG. 8 (a) shows the original optical channels I x,y, t), FIG. 8 (b) shows the shifted duplicate IShifted(.x>y> 0 optical channels, and FIG. 8 (c) shows the two encoded and multiplexed channels that may be processed, for example, by processor 150 to acquire E x,y, t). The noise term in the captured E(x,y, t) can thus be removed by exploiting the fact that / (%, y, t) and Shifted (,x> y> 0arecorrelated and mutually independent of N(x, y, t).

[0068] FIG. 9 is a block diagram of an exemplary computing device 500 consistent with some disclosed embodiments. The computing device 500 may include processor 510, such as, for example, a central processing unit (CPU). In some embodiments, the processor 510 may include, or may be a component of, a larger processing unit implemented with one or more processors. The one or more processors may be implemented with any combination of general-purpose microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate array (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic, discrete hardware components, dedicated hardware finite state machines, or any other suitable entities that can perform calculations or other manipulations of information. The processing circuitry such as processor 510 may be coupled to a memory 512.

[0069] Memory 512 may contain instructions that when executed by processor 510, may perform the methods described in more detail herein. Memory 512 may be further used as a working scratch pad for processor 510, a temporary storage, and others, as the case may be. Memory 512 may be a volatile memory such as, but not limited to, random access memory (RAM), or non-volatile memory (NVM), such as, but not limited to, flash memory.

[0070] Processor 510 may be further connected to a communications device 520, such as a network interface card, for providing wired or wireless connectivity between computing device 500 and a network (not shown). Processor 510 may be further coupled with a storage device 514. Storage device 514 may be used for the purpose of storing data for the purposes as described herein. While illustrated in FIG. 9 as a single device, it is to be understood that storage device 514 may include multiple devices either collocated or distributed.

[0071] Processor 510 and / or memory 512 may also include machine-readable media for storing software. "Software" as used herein refers broadly to any type of instructions, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. Instructions may include code (e.g., in source code format, binary code format, executable code format, or any other suitable format of code). The instructions, when executed by the one or more processors, may cause the processing system to perform the various functions described in further detail herein.

[0072] In some embodiments, device 500 may include one or more input interfaces 516. Input interface 516 may be configured to ingest and format data for use by device 500. In some embodiments, device 500 may include a configuration management module 518 which may be configured to configure device 500.

[0073] In some embodiments, device 500 may include a communication module 520 for enabling the transmission and / or reception of data. Communication module 520 may be used for communicating a notification or output. Communication module 520 may include human interface components (not shown) such as a display device for displaying information to a user and input devices such as a touch screen and / or a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to device 500.

[0074] Example - Denoising Evaluation

[0075] To test denoising accuracy, a clean event stream was created, either by transforming a frame-based video to its event format using v2e simulator

[0020] , which is based on an accurate DVS pixel model, or by aggressively denoising a recording. Next, prerecorded background activity noise events are inserted into the event stream. Then the system described herein may be used to denoise and collect the output, while tracking signal and noise events. Finally, the desired metrics may be calculated based on the collected output and input.

[0076] The denoising process may be regarded as a binary classification of signal or noise for each event. A positive classification means that the filter classifies an event as a signal event. To the best of theknowledge of the inventors, there is no unified performance evaluation (similar to SNR or PSNR) to judge the effectiveness of events denoising as of today. It was chosen to measure the denoising accuracy by the Receiver Operating Characteristic (ROC) and Area Under the Curve (AUC). The ROC method plots the True Positive Rate (TPR) and False Positive Rate (FPR) over all thresholds, providing a clear picture of the effect on signal and noise discrimination. Ideal denoising achieves zero FPR (noise misclassified as signal) and perfect TPR=1 (signal correctly classified as signal) and AUC=1. Higher AUC means a better classifier. Increasing TPR always increases FPR. The optimum TPR and FPR depends on the application: An always-on surveillance system might favor low FPR noise suppression while a mobile robot might favor high TPR signal retention.

[0077] Denoising metrics like Signal to Noise Ratio (SNR) depend on discrimination threshold; an infinite SNR is easily produced by aggressive denoising, but it would also remove many informative signal events. The AUC is likewise an arbitrary measure of accuracy, but unlike metrics that combine TPR and FPR at a selected discrimination threshold, the AUC removes bias by a particular choice of threshold. Using the methodology, the ROC curves were measured for comparing against two other off-the-shelf widely used background activity filters provided by iniVition python library: BANF & FDNF, both NNb filters. To generate each ROC curve, the filters' Ltparameter is swept through.

[0078] FIG. 10 shows all ROC curves and Table 1 summarizes denoising accuracy using AUC and two selected TPR / FPR operating points. The system described herein achieves a higher AUC than any of the other methods. FIG. 10 (and Table 1) illustrates the denoising performance of the disclosed system and method, showing a true positive rate of nearly 1 (i.e.: detection of all signal events) with a zero false positive rate (no noise events are detected as events). FIG. 10 (and Table 1) also indicates that the disclosed system and method displays superior performance when compared to other conventional methods (BANF FSNF).Table 1: Denoising accuracy comparisonFilter AUC FPR@TPR TPR@FPR 80% 5%System 0.994 0.80 100100BANF 0.962 5.60 75.7FDNF 0.861 20.0 23.7

[0079] The spatiotemporal three-dimensional (3D) representation of denoising the events captured by rotating a LED on a disk using the disclosed system and method versus the BANF can be seen in FIGS. 11A and 11B. FIG. 11A (a) shows the captured scene of a rotating disc with an LED, FIG. 11A (b) shows the captured 3D representation of captured events, FIG. 11B (c) shows denoised events using iniVition's BANF NNb filter, and FIG. 11B (d) shows denoised event stream using the disclosed system. It can be seen from FIG. 11A (b) that the background activity wraps the real event in it. In this controlled experiment the parameters used were: Lx= Ly= 2, N±= N2= 2, and Lt= 2ms for both BANF and the disclosed system.

[0080] It can be seen that the result with the BANF method still has some noise in it. This is because the space-time distance between background activity events is too small, so they were passed. If the filter is strict, the real events will be filtered out incorrectly, while the traces of the LED in the filtered noise will become clearer.

[0081] Additional examples:

[0082] Embodiments pertain to event camera systems, comprising:

[0083] at least one optical encoder positioned or positionable between a scene and an event camera;

[0084] wherein the at least one optical encoder is configured to optically encode an image of the scene into one or more encoded optical channels that are optically combined for capture of event data by the event camera; and

[0085] wherein the processor is configured to decode the captured event data. In some examples, the captured event data may be decoded, e.g., into event representations of the scene.

[0086] In some embodiments, the system includes the event camera.

[0087] In some embodiments, the system comprises a memory configured to store executable instructions and / or data processable by the processor for decoding the one or more encoded optical channels in the capture of the event camera.

[0088] In some embodiments, the system comprises an optical multiplexor configured to perform the optical combining of the one or more optical channels.

[0089] In some embodiments, the system comprises at least one optical device positioned between the optical encoder and the scene.

[0090] In some embodiments, the system comprises a plurality of optical devices positioned between the at least one optical encoder and the scene.

[0091] In some embodiments, the system comprises a plurality of optical encoders optically coupled with the at least one optical device.

[0092] In some embodiments, the system comprises at least one optical encoder which corresponds to a respective at least one optical channel.

[0093] In some embodiments, the optical encoding includes spatial encoding.

[0094] In some embodiments, the spatial encoding includes duplication and / or shifting and / or blurring.

[0095] In some embodiments, the optical encoder includes a Mach Zehnder interferometer.

[0096] In some embodiments, the optical encoder is configured to realize an (e.g., predefined) point spread function, e.g., by a blazed grating and / or phase mask.

[0097] In some embodiments, the processor is further configured to demultiplex the one or more optical channels from the capture by the event camera.

[0098] In some embodiments, the one or more optical channels corresponds to various characteristics captured from the scene such as, for example, wavelengths or wavelength ranges.

[0099] In some embodiments, each of the one or more optical channels corresponds to a different spectral band captured from the scene.

[0100] In some embodiments, each of the one or more optical channels corresponds to a different polarization captured from the scene.

[0101] In some embodiments, each of the one or more optical channels corresponds to a substantially different view of the scene.

[0102] In some embodiments, the processor is further configured to identify noise introduced by the event camera by analyzing at least one of the one or more encoded optical channels.

[0103] In some embodiments, the event camera includes a single event camera image sensor.

[0104] In some embodiments, the optical encoder and the optical multiplexor are combined.

[0105] In some embodiments, the at least two optical devices are substantially identical.

[0106] Embodiments pertain to a method for capturing an event relating to a scene, the methodcomprising:

[0107] optically encoding an image of the scene into one or more encoded optical channels;

[0108] optically combining the one or more encoded optical channels;

[0109] capturing, by an event camera, event data of the one or more optically combined encoded channels; and

[0110] decoding the captured event data.

[0111] In some embodiments, the method comprises positioning at least one optical encoder between the scene and an event camera.

[0112] In some embodiments, the optically encoding is performed by an optical encoder.

[0113] In some embodiments, the decoding is performed by a processor.

[0114] Embodiments pertain to a method for capturing an event, the method comprising:

[0115] positioning an optical encoder between a scene and an event camera configured to optically encode an image of the scene into one or more encoded optical channels that are optically combined for capture of the event camera;

[0116] capturing of the one or more combined encoded optical channels by the event camera; and

[0117] by a processor, decoding the one or more encoded optical channels in the capture by the event camera.

[0118] In embodiments, the method comprises providing a memory configured to store executable instructions and / or data processable by the processor for decoding the one or more encoded optical channels in the capture of the event camera.

[0119] In embodiments, the method comprises providing an optical multiplexor and optically combining the one or more optical channels with the optical multiplexor.

[0120] In embodiments, the method comprises providing an optical device positioned between the optical encoder and the scene.

[0121] In embodiments, the method comprises providing one or more optical devices positioned between the optical encoder and the scene each corresponding to the one or more encoded optical channels.

[0122] In embodiments of the method, the optical encoder comprises one or more optical encoders eachcorresponding to the one or more optical channels.

[0123] In embodiments of the method, the optical encoding includes spatial encoding.

[0124] In embodiments of the method, the spatial encoding includes duplication and / or shifting and / or blurring.

[0125] In embodiments of the method, the optical encoder includes a Mach Zehnder interferometer.

[0126] In embodiments, the method includes realizing a (e.g., predefined) point spread function, e.g., by a blazed grating or phase mask.

[0127] In embodiments of the method, at least two or each of the one or more optical channels corresponds to a different color captured from the scene.

[0128] In embodiments of the method, at least two or each of the one or more optical channels corresponds to a different spectral band captured from the scene.

[0129] In embodiments of the method, at least two or each of the one or more optical channels corresponds to a different polarization captured from the scene.

[0130] In embodiments of the method, at least two or each of the one or more optical channels corresponds to a substantially different view of the scene.

[0131] In embodiments, the method includes identifying noise introduced by the event camera by analyzing at least one of the one or more encoded optical channels.

[0132] In embodiments of the method, the event camera includes a single event camera image sensor.

[0133] In embodiments of the method, the optical encoder and optical multiplexor are combined.

[0134] In embodiments of the method, at least two optical devices are substantially identical.

[0135] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The materials, methods, and examples provided herein are illustrative only and not intended to be limiting.

[0136] Implementation of the method and system of the present disclosure may involve performing or completing certain selected tasks or steps manually, automatically, or a combination thereof. Moreover, according to actual instrumentation and equipment of preferred embodiments of the method and system of the present disclosure, several selected steps may be implemented by hardware (HW) or by software (SW) on any operating system of any firmware, or by a combination thereof. For example, as hardware,selected steps of the disclosure could be implemented as a processor chip or a circuit. As software or algorithm, selected steps of the disclosure could be implemented as a plurality of software instructions being executed by a computer / processor using any suitable operating system. In any case, selected steps of the method and system of the disclosure could be described as being performed by a data processor, such as a computing device for executing a plurality of instructions.

[0137] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0138] Any digital computer system, unit, device, module and / or engine exemplified herein can be configured or otherwise programmed to implement a method disclosed herein, and to the extent that the system, module and / or engine is configured to implement such a method, it is within the scope and spirit of the disclosure. Once the system, module and / or engine are programmed to perform particularfunctions pursuant to computer readable and executable instructions from program software that implements a method disclosed herein, it in effect becomes a special purpose computer particular to embodiments of the method disclosed herein. The methods and / or processes disclosed herein may be implemented as a computer program product that may be tangibly embodied in an information carrier including, for example, in a non-transitory tangible computer-readable and / or non-transitory tangible machine-readable storage device. The computer program product may be directly loadable into an internal memory of a digital computer, comprising software code portions for performing the methods and / or processes as disclosed herein.

[0139] The methods and / or processes disclosed herein may be implemented as a computer program that may be intangibly embodied by a computer readable signal medium. A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a non-transitory computer or machine-readable storage device and that can communicate, propagate, or transport aprogram for use by or in connection with apparatuses, systems, platforms, methods, operations and / or processes discussed herein.

[0140] It should be understood that where the claims or specification refer to "a" or "an" element and / or feature, such reference is not to be construed as there being only one of that element. Hence, reference to "an element" or "at least one element" for instance may also encompass "one or more elements".

[0141] Unless otherwise specified, the terms 'about', 'substantially', and / or ’close’ with respect to a magnitude or a numerical value may imply to be within an inclusive range of -10% to +10% of the respective magnitude or value.

[0142] It is important to note that the method may include is not limited to those diagrams or to the corresponding descriptions. For example, the method may include additional or even fewer processes or operations in comparison to what is described herein. In addition, embodiments of the method are not necessarily limited to the chronological order as illustrated and described herein.

[0143] Unless otherwise stated or applicable, the use of the expression "and / or" between the last two members of a list of options for selection indicates that a selection of one or more of the listed options is appropriate and may be made, and may be used interchangeably with the expressions "at least one of the following", "any one of the following" or "one or more of the following", followed by a listing of the various options.

[0144] As used herein, the phrase "A,B,C, or any combination of the aforesaid" should be interpreted as meaning all of the following: (i) A or B or C or any combination of A, B, and C, (ii) at least one of A, B, and C; and (iii) A, and / or B and / or C. This concept is illustrated for three elements (i.e., A,B,C), but extends to fewer and greater numbers of elements (e.g., A, B, C, D, etc.).

[0145] It is appreciated that certain features of the invention, which are, for clarity, described in the context of separate embodiments or example, may also be provided in combination in a single embodiment. Conversely, various features of the invention, which are, for brevity, described in the context of a single embodiment, example and / or option, may also be provided separately or in any suitable subcombination or as suitable in any other described embodiment, example, or option of the invention. Certain features described in the context of various embodiments, examples and / or options are not to be considered essential features of those embodiments, unless the embodiment, example and / or option is inoperative without those elements.

[0146] Throughout this application, various embodiments of this invention may be presented in a rangeformat. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the invention. Accordingly, the description of a range should be considered to have specifically disclosed all the possible subranges as well as individual numerical values within that range. For example, description of a range such as from 1 to 6 should be considered to have specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numbers within that range, for example, 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.

[0147] Whenever a numerical range is indicated herein, it is meant to include any cited numeral (fractional or integral) within the indicated range. The phrases "ranging / ranges between" a first indicate number and a second indicate number and "ranging / ranges from" a first indicate number "to" a second indicate number are used herein interchangeably and are meant to include the first and second indicated numbers and all the fractional and integral numerals therebetween.

[0148] While the invention has been described with respect to a limited number of embodiments, these should not be construed as limitations on the scope of the invention, but rather as exemplifications of some of the embodiments.

[0149] While certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes, and equivalents will now occur to those skilled in the art. It should be understood that they have been presented by way of example only, not limitation, and various changes in form and details may be made. Any portion of the apparatus and / or methods described herein may be combined in any combination, except mutually exclusive combinations. The implementations described herein can include various combinations and / or sub-combinations of the functions, components and / or features of the different implementations and embodiments described.

Claims

CLAIMSWHAT IS CLAIMED IS:

1. An event camera system comprising: at least one optical encoder positioned or positionable between a scene and an event camera; wherein the at least one optical encoder is configured to optically encode an image of the scene into one or more encoded optical channels that are optically combined for capture of event data by the event camera; and a processor configured to decode the captured event data.2 The event camera system of claim 1, further comprising the event camera.3 The system of claim 1 and / or claim 2, further comprising a memory configured to store executable instructions and / or data processable by the processor for decoding the one or more encoded optical channels in the capture of the event camera.4 The system of any one or more of the preceding claims, further comprising an optical multiplexor configured to perform the optical combining of the one or more optical channels.5 The system of any one or more of the preceding claims, further comprising at least one optical device positioned between the optical encoder and the scene.6 The system of any one or more of the preceding claims, comprising a plurality of optical devices positioned between the at least one optical encoder and the scene.7 The system of any one or more of claim 5 and / or claim 6, comprising a plurality of optical encoders optically coupled with the at least one optical device.8 The system of claim 7, wherein the at least one optical encoder corresponds to respective opticalchannel.

9. The system of any one or more of the preceding claims, wherein the optical encoding includes spatial encoding.

10. The system of claim 9, wherein the spatial encoding includes duplication and / or shifting and / or blurring.

11. The system of any one or more of the preceding claims, wherein the optical encoder includes a Mach Zehnder interferometer.

12. The system of any one or more of the preceding claims, wherein the optical encoder includes a blazed grating and / or phase mask realizing a predefined point spread function.

13. The system of any one or more of the preceding claims, wherein the processor is further configured to demultiplex the one or more optical channels from the capture by the event camera.

14. The system of any one or more of the preceding claims, wherein the one or more optical channels corresponds to a different color captured from the scene.

15. The system of any one or more of the preceding claims, wherein each of the one or more optical channels corresponds to a different spectral band captured from the scene.

16. The system of any one or more of the preceding claims, wherein each of the one or more optical channels corresponds to a different polarization captured from the scene.

17. The system of any one or more of the preceding claims, wherein each of the one or more optical channels corresponds to a substantially different view of the scene.

18. The system of any one or more of the preceding claims, wherein the processor is further configured to identify noise introduced by the event camera by analyzing at least one of the one or more encoded optical channels.

19. The system of any one or more of the preceding claims, wherein the event camera includes a single event camera image sensor.

20. The system of any one or more of the claims 4 to 19, wherein the optical encoder and the optical multiplexor are combined.

21. The system of any one or more of the claims 5 to 20, wherein the at least two optical devices are substantially identical.

22. A method for capturing an event relating to a scene, the method comprising: optically encoding an image of the scene into one or more encoded optical channels; optically combining the one or more encoded optical channels; and capturing, by an event camera, event data of the one or more optically combined encoded channels; decoding the captured event data.

23. The method of claim 22, comprising: positioning at least one optical encoder between the scene and an event camera; and wherein optically encoding is performed by an optical encoder; and wherein the decoding is performed by a processor.

24. The method of claim 22 and / or claim 23, further comprising:providing a memory configured to store executable instructions and / or data processable by the processor for decoding the one or more encoded optical channels in the capture of the event camera.

25. The method of claim 22 and / or claim 24, further comprising providing an optical multiplexor and optically combining the one or more optical channels with the optical multiplexor.

26. The method of any one or more of the claims 22 to 25, further comprising providing an optical device positioned between the optical encoder and the scene.

27. The method of any one or more of the claims 22 to 26, further comprising providing one or more optical devices positioned between the optical encoder and the scene each corresponding to the one or more encoded optical channels.

28. The method of claim 27, wherein the optical encoder comprises one or more optical encoders each corresponding to the one or more optical channels.

29. The method of any one or more of the claims 22 to 28, wherein the optical encoding includes spatial encoding.

30. The method of claim 29, wherein the spatial encoding includes duplication and / or shifting and / or blurring.

31. The method of any one or more of the claims 22 to 30, wherein the optical encoder includes a Mach Zehnder interferometer.

32. The method of any one or more of the claims 22 to 31, wherein the optical encoder includes a blazed grating or phase mask realizing a predefined point spread function.

33. The method of any one or more of the claims 22 to 32, wherein at least two or each of the one or more optical channels corresponds to a different color captured from the scene.

34. The method of any one or more of the claims 22 to 33, wherein at least two or each of the one or more optical channels corresponds to a different spectral band captured from the scene.

35. The method of any one or more of the claims 22 to 34, wherein at least two or each of the one or more optical channels corresponds to a different polarization captured from the scene.

36. The method of any one or more of the claims 22 to 35, wherein at least two or each of the one or more optical channels corresponds to a substantially different view of the scene.

37. The method of any one or more of the claims 22 to 36, wherein the processor is further configured to identify noise introduced by the event camera by analyzing at least one of the one or more encoded optical channels.

38. The method of any one or more of the claims 22 to 37, wherein the event camera includes a single event camera image sensor.

39. The method of any one or more of the claims 24 to 38, wherein the optical encoder and optical multiplexor are combined.

40. The method of any one or more of the claims 22 to 39, comprising at least two optical devices that are substantially identical.

41. The method of any one or more of the claims 22 to 40, comprising providing an event camera.