Event-based imaging system and method and spiking neural network therefor, training system for creating such spiking neural network, and vehicle, drone and robotic system

Abstract

Embodiments of the present disclosure relate to event-based imaging systems and methods and spiking neural networks therefor, training systems for creating such spiking neural networks, and vehicle, drone, and robotic systems. The method includes obtaining event data of an object in an environment of the event-based imaging system using an event-based visual sensor (EVS), and obtaining motion data of the object using a reference sensor. In addition, the method provides for training an artificial neural network (ANN) using the event data and the motion data to characterize and / or predict motion of the subject. The method further includes creating a spiking neural network (SNN) based on the trained ANN, the SNN configured to characterize and / or predict a motion of the subject.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to event-based imaging systems and methods and spiking neural networks for such imaging systems, training systems for creating such spiking neural networks, and vehicles, drones, and robotic systems. In particular, but not exclusively, embodiments of the present disclosure relate to the concept of high-speed prediction using event-based imaging systems. Background Art

[0002] Event-based imaging plays an increasingly important role in a variety of scientific and technological fields.

[0003] Detecting moving objects and estimating their speed is used in a variety of applications including autonomous driving and drone control. High-speed estimation is particularly important in rapidly moving or changing systems. Drones may need to avoid dynamic obstacles during flight, and autonomous vehicles may need to be able to predict the moving traffic participants in their environment.

[0004] In practice, cameras can be used to detect moving objects and estimate their speed. In doing so, the speed of the object can be obtained by detecting and tracking the object in multiple subsequent image frames. Such techniques can rely on accurate object position detection for measurement updates. This is done, for example, by convolutional neural networks (CNNs).

[0005] In practice, the update rate of the predictable velocity depends on the processing time of the object detection algorithm and the sensor measurement rate (ie, the frame rate if the sensor is a standard camera). Furthermore, by filtering the position data, additional delay is introduced.

[0006] CNNs can achieve sufficiently accurate velocity estimates, but the complex architecture significantly limits the inference time used to predict the object's velocity. In some applications, the inference time is on the order of a few milliseconds to tens of milliseconds. Traditional frame-based cameras, due to the high data rates they produce, can typically only achieve frame rates on the order of a few hundred Hz. In addition, the inference time of CNNs (the time used to estimate the camera's image data to obtain the object's velocity) may be insufficient for many applications.

[0007] LiDAR-based systems can achieve object detection at around 200fps. Recurrent neural networks learn motion information directly on sequential data and therefore do not rely on filtering methods for velocity prediction. However, due to their complex architecture, the prediction frequency is on the order of a few hundred Hz. Therefore, LiDAR-based systems also do not provide sufficient temporal resolution for many applications.

[0008] In some applications, a higher temporal resolution is required to determine the velocity of an object.

[0009] Therefore, an improved speed measurement / estimation concept is needed.

[0010] This need is met by the subject matter of the accompanying independent and dependent claims. Summary of the invention

[0011] Embodiments of the present disclosure are based on the discovery that parameters of an artificial neural network trained to determine the motion of an object can be transferred to a spiking neural network to combine the accuracy of the artificial neural network in determining the motion of the object with the temporal resolution of the spiking neural network. Another discovery is that real-world observations of reference sensors can be used to train the artificial neural network with higher accuracy than using manually generated or simulated training data.

[0012] Embodiments of the present disclosure provide a method for an event-based imaging system. The method includes: using an event-based visual sensor (EVS) to obtain event data of an object in an environment of the event-based imaging system, and using a reference sensor to obtain motion data of the object. Further, the method provides for training an artificial neural network (ANN) using the event data and the motion data to characterize and / or predict the motion of the object. The method also includes: creating a spiking neural network (SNN) based on the trained ANN, the SNN being configured to characterize and / or predict the motion of the object.

[0013] Using reference sensors to generate motion data from real-world observations eliminates the need for laborious manual generation or simulation of motion data and can produce higher accuracy for ANNs and SNNs than manual or simulated input motion data used to train ANNs.

[0014] Those skilled in the art who benefit from the present disclosure will appreciate that SNNs can provide faster processing rates than other ANNs, while on the other hand, other ANNs can be trained more efficiently and accurately than SNNs. The ANN can be, for example, any ANN that allows GPU (graphics processing unit) training to have higher convergence and higher accuracy than SNNs in determining the speed or other motion parameters of an object. By creating an SNN based on a trained ANN, the capabilities of the trained ANN can be transferred to the SNN. Thus, the SNN created based on the trained ANN combines the advantages of the higher accuracy of the ANN with the higher processing rate of the SNN. Therefore, unlike other ANNs, SNNs can not only provide sufficient accuracy for determining the motion of an object, but also provide sufficient time resolution (a few microseconds, μs) for applications in rapidly changing systems.

[0015] The reference sensor may include or correspond to any imaging sensor. In some embodiments, the reference sensor includes an RGB camera, a LiDAR sensor, a RADAR sensor, an infrared sensor, and / or an ultrasonic sensor.

[0016] In order to create an SNN, the activation function of the ANN can be replaced by an activation function that integrates the input over time with a leakage term. In some embodiments, creating the SNN includes replacing the activation function of the ANN with a leaky integrate-and-release (LIF) neuron or a LIF activation function. The activation function is, for example, a rectified linear unit (ReLU) or any other activation function (different from the LIF activation function). In practice, for example, the activation function in the input layer of the ANN is replaced by a LIF neuron, so that the input indicating event data is fed to the hidden layer of the SNN instantly with the time resolution or inherent delay of the LIF neuron. In practice, the time resolution is about 1 μs or a few microseconds. In this way, (pixel) events are integrated by LIF neurons, and the SNN continuously updates its results with additional event data within microseconds. Thereby, the judgment of the movement of the object can be accelerated.

[0017] In practice, creating an SNN may include implementing the SNN in a field programmable gate array (FPGA) or neuromorphic hardware. As will be appreciated by those skilled in the art, hardware-implemented neural networks allow for shorter processing times than, for example, purely software-implemented neural networks due to parallelization. Thus, implementing the SNN in an FPGA or neuromorphic hardware allows for faster motion prediction / determination.

[0018] Optionally, obtaining the motion data includes obtaining measurement data of the object from a reference sensor, and obtaining the motion data using a Bayesian filter and using the measurement data as an input to the Bayesian filter. Filtering the motion data by the Bayesian filter allows for more accurate determination of the motion data and thus enhances the training of the ANN, resulting in greater accuracy of the SNN in determining the motion of the object.

[0019] Optionally, the results of the Bayesian filter are filtered to further enhance the training of the ANN and the accuracy of the SNN. To this end, for example, obtaining motion data may also include: obtaining motion samples by applying a Bayesian filter to the measurement data; determining the uncertainty of the motion samples; and filtering the motion samples based on the uncertainty to obtain motion data. By filtering the motion samples based on the uncertainty, outliers that may deteriorate the training can be filtered out, thereby improving the training of the ANN. Additionally or alternatively, for better training data, a motion model can be used to determine trajectory samples of (multiple) objects, and motion samples, particularly velocity samples, can be derived from the trajectory samples.

[0020] In some embodiments, the uncertainty of the motion sample is represented by a Mahalanobis distance of the motion sample, and wherein filtering the motion samples includes filtering out motion samples having a Mahalanobis distance greater than a predefined threshold.

[0021] To further improve the SNN, the SNN may be further trained when the event-based imaging system is operating. To this end, the method may further include obtaining additional event data using the EVS, obtaining additional motion data using the reference sensor, providing the event data and the motion data to the SNN, and training the SNN based on the additional event data and the additional motion data provided to the SNN. In this way, the SNN is also adapted for even more accurate determination / prediction of the motion of the object.

[0022] In doing so, training the SNN includes, for example, using spike timing dependent backpropagation (STDB) to train the SNN.

[0023] Further embodiments of the present disclosure provide SNNs obtained by the methods proposed herein.

[0024] As explained in conjunction with the proposed approach, SNNs can be implemented in FPGA or neuromorphic hardware to provide higher temporal resolution than software-based / CPU-based or GPU-based implementations.

[0025] Likewise, an embodiment of the present disclosure provides an event-based imaging system for characterizing and / or predicting the motion of an object in an environment of the event-based imaging system. The event-based imaging system includes an EVS and a proposed SNN, the EVS being configured to provide event data of an object in the environment of the event-based imaging system, and the SNN being used to characterize and / or predict the motion of the object based on the event data.

[0026] As described above, using reference sensors to generate motion data based on real-world observations, on the one hand, eliminates the need for laborious manual generation or simulation of motion data, and on the other hand, can produce higher accuracy in determining the motion of objects for ANNs and SNNs than training ANNs with manual or simulated input motion data.

[0027] In practice, the SNN may include multiple layers including an input layer configured to receive event data and provide an output indicating the event data to another layer upon receiving the event data so as to process the event data faster than other ANNs. In an embodiment, the input layer is, for example, configured to provide an output indicating the event data to another layer immediately upon receiving the event data. To this end, the input layer includes, for example, LIF neurons.

[0028] In order to process the event data faster, in practice, the EVS can process the event data according to the Address Event Representation (AER) protocol to provide the event data to the SNN, and the SNN is configured to process the event data according to the AER protocol for characterizing and / or predicting the movement of the object. In this way, the amount of data to be transmitted for characterizing and / or predicting the movement of the object can be reduced, thereby reducing resource consumption and shortening processing time.

[0029] As described above, the SNN can also be trained. To this end, the event-based imaging system also includes a reference sensor for training the SNN. Accordingly, the event-based imaging system can be equipped with at least two types of sensors, an EVS and any kind of reference sensor. Therefore, this setup of the event-based imaging system can be called a "dual setup".

[0030] Another embodiment of the present disclosure provides a training system for creating an SNN, the SNN being configured to characterize and / or predict the motion of an object in an environment of an event-based imaging system. According to the description in conjunction with the proposed method, the training system includes an EVS and a reference sensor, the EVS being configured to provide event data of an object in an environment of an event-based imaging system, and the reference sensor being configured to provide motion data of the object. In addition, the training system includes an ANN, the ANN being configured to train the ANN to characterize and / or predict the motion of the object based on the event data and the motion data, and to create an SNN for characterizing and / or predicting the motion of the object based on the ANN.

[0031] In an embodiment, the reference sensor can be any kind of sensor. In some embodiments, the reference sensor includes an RGB camera, a LiDAR sensor, a RADAR sensor, an infrared sensor, and / or an ultrasonic sensor.

[0032] Other embodiments provide a vehicle, drone or robotic system including the event-based imaging system presented herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Some examples of apparatus and / or methods will be described below, by way of example only, and with reference to the accompanying drawings, in which:

[0034] Figure 1 A flow chart schematically illustrating an embodiment of a method for an event-based imaging system is shown;

[0035] Figure 2 A block diagram schematically illustrating an embodiment of a training system for creating a spiking neural network (SNN) configured to characterize and / or predict the motion of an object in the context of an event-based imaging system is shown;

[0036] Figure 3shows a block diagram schematically illustrating another embodiment of a training system;

[0037] Figure 4 Schematically illustrates the functional principles of ANN and SNN according to the proposed concept;

[0038] Figure 5 shows a block diagram schematically illustrating an embodiment of an event-based imaging system proposed herein;

[0039] Figure 6 shows a block diagram schematically illustrating another embodiment of an event-based imaging system; and

[0040] Figure 7 A block diagram schematically illustrating an exemplary application of the proposed event-based imaging system is shown. DETAILED DESCRIPTION

[0041] Some examples are now described in more detail with reference to the accompanying drawings. However, other possible examples are not limited to the features of these embodiments described in detail. Other examples may include modifications of features and equivalents and substitutions of features. In addition, the terms used to describe certain examples in this article should not limit other possible examples.

[0042] In the description of the accompanying drawings, the same or similar reference numerals represent the same or similar elements and / or features, which may be the same or implemented in a modified form while providing the same or similar functions. For clarity, the thickness of the lines, layers and / or regions in the drawings may also be exaggerated.

[0043] When two elements A and B are combined using "or", this should be understood to disclose all possible combinations, i.e., only A, only B and A and B, unless otherwise explicitly defined in individual cases. As an alternative expression of the same combination, "at least one of A and B" or "A and / or B" can be used. The same applies to combinations of more than two elements.

[0044] If singular forms such as "a", "an", and "the" are used, and the use of only a single element is not explicitly or implicitly defined as mandatory, additional examples may also use several elements to implement the same functionality. If a function is described below as being implemented using multiple elements, additional examples may use a single element or a single processing entity to implement the same functionality. It should also be understood that the terms "include", "including", "comprise" and / or "comprising" when used describe the presence of specified features, wholes, steps, operations, processes, elements, parts and / or groups thereof, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, processes, elements, parts and / or groups thereof.

[0045] As mentioned above, motion measurement systems have so far provided insufficient temporal resolution for applications in rapidly changing systems (e.g., drone or automotive applications).

[0046] In practice, systems using RGB cameras need to handle high data rates and therefore cannot provide sufficient temporal resolution for motion measurements (especially high-speed measurements). The concept of the present disclosure is to use an EVS with a temporal resolution of about 1 μs or a few microseconds (up to 10 μs) better than that of an RGB camera. Such an EVS asynchronously detects brightness changes at the pixel level and generates event data indicating such brightness changes. The event data may include time-stamped pixel events marked with polarity (i.e., "ON" and "OFF") according to the direction of the brightness change, where "ON" indicates an increase in brightness and "OFF" indicates a decrease in brightness. Therefore, the event data is sparser than an RGB frame, which allows faster processing than in a system using an RGB camera.

[0047] One challenge in the application of EVS is to provide a suitable means for determining motion from event data. Conventional algorithms for processing grayscale or RGB frames show insufficient performance / accuracy for event data due to the different data representations.

[0048] It is therefore an object of the present disclosure to provide improved devices, particularly processing devices, for determining the motion of an object from event data of an EVS. In particular, it is an object of the present invention to provide a hardware system that enables one-shot, high-frequency velocity prediction of moving targets using an event camera and an FPGA.

[0049] Figure 1 A flow chart schematically illustrating an embodiment of a method 100 for an event-based imaging system is shown, and Figure 2A block diagram schematically illustrating an embodiment of a training system 200 configured to apply the method 100 is shown. Various aspects and features are now described in more detail with reference to the proposed method 100 and training system 200.

[0050] As can be seen from the flow chart, method 100 includes obtaining 110 event data of an object in an environment of an event-based imaging system using an EVS. Accordingly, training system 200 includes an EVS 210 configured to provide event data of an object in an environment of an event-based imaging system. EVS 210, also known as a "neuromorphic camera, event camera, silicon retina, or dynamic vision sensor," includes an imaging sensor configured to respond to local changes in brightness. Thus, the event data indicates brightness changes measured by pixels of EVS 210.

[0051] The pixels of the EVS210 can respond independently when there is a change in brightness. Each pixel can store a reference brightness level and constantly compare it to the current brightness level. If the brightness difference exceeds a threshold, the pixel resets its reference level and generates an event: a discrete packet containing the address and timestamp of the pixel. An event can also contain the polarity of the brightness change (increase or decrease) or an instantaneous measurement of the lighting level. Thus, event data can include an asynchronous event stream triggered by changes in scene lighting and can indicate polarity, address, and timestamp.

[0052] Any (reflective) object may cause such a change in brightness. Thus, the object may be any (arbitrary) object. In order to train the event-based imaging system to determine the motion of the object, the object may be a moving object. In an embodiment, the object may be a target that is a feature of the application of the event-based imaging system. For automotive applications, the object may be, for example, any traffic participant, an infrastructure object or a suitable training surrogate for a traffic participant or an infrastructure object. For applications in drones, the object may be an aircraft or a suitable training surrogate for an aircraft.

[0053] In addition, the method 100 provides for acquiring 120 motion data of the object using a reference sensor. Therefore, the training system 200 also includes such a reference sensor 220 configured to provide motion data of the object. The reference sensor 220 is, for example, or includes an RGB camera, a LiDAR sensor, a RADAR sensor, an infrared sensor, an ultrasonic sensor, etc. In some embodiments, the reference sensor may also include a combination of these sensors or another sensor. In an embodiment, the motion data may include any kind of information about the motion of the object. Examples of motion data include information about the speed / rate of the object, about the direction of motion, about rotation, and / or about acceleration.

[0054] In order to obtain motion data, the measurement data from the reference sensor 220 may be appropriately processed. In practice, the measurement data may include, for example, images or frames captured by an RGB camera and / or point clouds from a LiDAR sensor. Thus, object detection and object tracking may be applied to determine motion data based on changes in the position of the object in subsequent images / frames and / or point clouds. To this end, for example, appropriate computer vision (CV) algorithms may be applied to the images / frames and / or point clouds.

[0055] Spiking neural networks (SNNs) process binary sparse events asynchronously as they arrive. The inference time is limited by the processor, but the hardware can be optimized to exploit the sparse and binary nature of the SNN activation signal. Custom hardware implements the SNN in its underlying circuits, and massively parallel neural computing speeds up the processing time. Therefore, the idea of ​​the present disclosure is to apply SNNs to determine / predict the motion of an object based on event data from EVS. However, training SNNs introduces another dimension, namely, time. Therefore, training SNNs requires more memory. It also requires a "surrogate function" to overcome the "non-differentiability" of the "pulse". Therefore, "direct" training of SNNs is avoided. However, the present disclosure provides an easier way to provide a suitable SNN for determining / predicting the motion of an object based on event data from EVS.

[0056] To this end, the method 100 includes applying 130 event data and motion data to train the ANN to characterize and / or predict the motion of the object. Therefore, the example of the training system 200 also includes an ANN 230, which is configured to train the ANN to characterize and / or predict the motion of the object based on the event data and the motion data, and to create an SNN for characterizing and / or predicting the motion of the object based on the ANN. In order to train the ANN 230, training data including paired samples of event data and motion data matched in time order can be generated, wherein the event data is used as real data for training the ANN 230 to characterize and / or predict the motion of the object and / or any other object based on the event data from the EVS 210. In an embodiment, the event data can be accumulated over a time period equal to the duration of the determination of the motion data. Accumulating event data, for example, includes summing pixel events. In doing so, two channels can be used, wherein the first channel only accumulates positive brightness changes ("ON") and the second channel only accumulates negative brightness changes ("OFF"). Therefore, a first image is generated based on positive brightness changes and a second image is generated based on negative brightness changes. Optionally, pixel events may be weighted to determine a weighted sum of pixel events. In doing so, for example, the most recent event is given the highest weight, while the oldest event is given the lowest weight (during the accumulation period).

[0057] In practice, the RGB camera may be operated at 100 Hz, such that event data is accumulated over 0.01 s (in other applications, other frame rates / times are possible) and then associated with corresponding frames of the RGB camera. For example, the corresponding frame is a frame recorded while or immediately after accumulating the event data. The ANN 230 may be implemented, for example, by software and / or hardware of the training system 200. To train the ANN 230, for example, supervised learning may be applied.

[0058] In some applications, it is particularly desirable for an event-based imaging system to be able to detect (high) speeds of objects. Thus, the motion data may be indicative of the speed of the object, and the ANN 230 may be specifically trained to determine the speed of the object based on the event data from the EVS 210.

[0059] In practice, the proposed method can be applied to multiple objects and different kinds of objects to achieve higher robustness of the trained ANN 230.

[0060] In order to establish an SNN according to the above ideas, the method 100 provides for creating 140 an SNN based on the trained ANN 230, which is configured to characterize and / or predict the movement of an object to transfer the ability of the ANN to determine the movement of the object to the SNN. To this end, the structure / architecture and weights of the ANN can be transferred to the SNN. Therefore, in addition to the specific structural characteristics of the SNN, the SNN can be created so that it exhibits the same weights and structure. The SNN, for example, includes the same number of layers as the ANN, the same number of neurons, and neuronal connections that exhibit the same weights.

[0061] The idea of ​​SNNs is that neurons in SNNs do not transmit information at every propagation cycle (like other multilayer perceptron networks (e.g., ANNs)), but rather transmit information asynchronously when the membrane potential (an intrinsic quality of a neuron related to its membrane charge) reaches a certain value (called the "threshold"). When the membrane potential of a neuron reaches the threshold, the neuron fires and generates a signal that travels to other neurons, which in turn increase or decrease their potential in response to this signal. The neuron model that is excited at the moment of threshold crossing is also called a spiking neuron model. Thus, this spiking neuron model is implemented in SNNs along with the capabilities of path-trained ANNs. Therefore, SNNs are configured to asynchronously process asynchronous event streams in event data in "real time", which processes events faster than other multilayer perceptron networks because other multilayer perceptron networks process data in predetermined propagation cycles of a few milliseconds.

[0062] The proposed method thus allows to determine and / or predict the motion, in particular the velocity, of a (moving) object at event rates and rates exceeding tens of kHz in real time and without the need for a prior motion model, such as used by a filter. Moreover, the use of motion data observed from the real world using reference data can provide a higher accuracy than in the case of manually generated or simulated real data.

[0063] An exemplary spiking neuron model is a leaky integrate-and-fire (LIF) model. Thus, creating 140 the SNN may include replacing the activation function of the ANN with the LIF neuron / activation function. In this process, in practice, the activation function of the neurons of the input layer may be replaced by an activation function according to the LIF model.

[0064] In the leaky integrate-and-fire model / activation function, the instantaneous activation level (modeled as a differential equation) is considered to be the state of the neuron, where incoming impulses push this value higher or lower until the state eventually decays, where some amount of current leaks to the next layer, or if the firing threshold is reached, the neuron fires, resetting the state variable to a lower value after firing.

[0065] In an embodiment, the SNN may be implemented in hardware, emulating the structure / architecture and characteristics (neuronal connections, weights) of the SNN to achieve a higher processing rate than a software-implemented neural network. Examples of suitable hardware include neuromorphic hardware and FPGAs. Thus, creating 140 the SNN may include implementing the SNN in a field programmable gate array FPGA or neuromorphic hardware. In this process, the SNN may be established through programmable logic blocks, application specific integrated circuits (ASICs), etc. Those skilled in the art will recognize that the hardware may be digital, analog, and / or a combination thereof.

[0066] Reference is now made to an exemplary embodiment 300 of the training system 200 which is schematically shown. Figure 3 , other aspects and features of the proposed method are described in more detail.

[0067] As from Figure 3 As can be seen in the block diagram in , training system 300 includes EVS 310 and RGB camera 320, which is configured to generate event data 350 indicative of object 370 (particularly its movement).

[0068] In addition, the training system 300 includes an ANN 340, which should be trained using input from the EVS 310 or any equivalent event camera to detect and predict the speed of moving objects. In order to train the ANN 340, the motion data / real data 330 of the training data is obtained using the RGB camera 320. To this end, CV techniques can be applied to frames of the RGB camera 320 running at a frequency of 100 Hz (within the range). The CV technique (for example) detects the color, shape and / or appearance of the object 370 to estimate the position of the object 370, and then tracks the position of the object 370 using a filter to determine the speed or multiple speeds of the object 370. In this process, for example, the position is filtered by the motion model and the speed of the object can be obtained as the real data 330 of the training data. In order to filter the position by the motion model, for example, a Kalman filter or any other Bayesian filter can be applied.

[0069] In order to avoid or eliminate objects where the real data 330 indicates false detections or false assumptions about the motion model and its current parameters, only velocities (motion samples) with acceptable uncertainty may be selected for the real data 330. To this end, for example, the Mahalanobis velocity distance is determined and (only) velocities with a Mahalanobis distance below a predetermined threshold are selected for the ground truth 330.

[0070] In this way, the (frame) rate of the RGB camera 320 can be utilized to generate the real data 330. The (frame) rate is, for example, 100 Hz, which is lower than the rate at which events are detected by the EVS 310. Therefore, the RGB camera 320 together with the Kalman filter can be referred to as a "low frequency reference system".

[0071] According to the proposed method, a low-frequency reference system is used to train / create a "high-frequency inference system" that is configured to more quickly determine / predict the velocity or other motion parameters of an object. In this process, samples of velocity and event data of real data that match in time order are grouped in pairs and used as training data to train the ANN 340 to characterize and / or predict the velocity of an object.

[0072] Then, the ANN 340 is converted into an SNN 360 by transferring the training capability of determining the motion of the object from the ANN 340 to the SNN 360. To this end, the ReLU activation / function of the neurons in the input layer is exchanged with the LIF neurons. The SNN 360 can be implemented in an FPGA and an efficient pulse-based communication can be established.

[0073] SNN 360 may then be applied to predict the velocity of a moving object (eg, object 370).

[0074] As outlined in more detail above, SNN 360 allows for the determination of motion (e.g., the velocity of a moving object) at a higher rate (i.e., with a higher temporal resolution) than ANN 340, which can operate at the frame rate of an RGB camera (100 Hz). Thus, EVS 310 together with SNN 360 may be referred to as a "high frequency inference system."

[0075] High-frequency inference systems can achieve a temporal resolution of 1 μs or a few microseconds and thus meet the temporal resolution needs of applications in rapidly changing systems (e.g., drone or automotive applications).

[0076] A person skilled in the art having the benefit of the present disclosure will recognize that even though the illustrated example primarily deals with prediction of object velocity, the proposed method may also be applied to determination of (current) velocity and determination and / or prediction of other motion parameters (e.g., acceleration, rotation, etc.). A person skilled in the art will also appreciate that the SNN 340 may be built in various ways, for example, using different programming and manufacturing means and techniques to convey the capabilities of the ANN 340 to the SNN 360. Likewise, a person skilled in the art will recognize that the proposed method is also applicable to different types or combinations of sensors used as reference sensors and is not limited to RGB cameras.

[0077] Refer to the following Figure 4 Explaining in more detail other details about the creation of SNN 360, Figure 4 The functional principles of ANN 340 and SNN 360 are schematically shown.

[0078] Figure 4 A diagram 410 schematically illustrating the functional principle of the ANN 340 and another diagram 420 schematically illustrating the functional principle of the SNN 360 are shown.

[0079] On the left hand side of Figures 410 and 420, simplified structures of ANN 340 (Figure 410) and SNN 360 (Figure 420) are schematically shown. As can be seen from Figures 410 and 420, it is proposed that ANN 340 and SNN 360 exhibit the same architecture / structure of interconnected neurons forming an input layer 414 / 424 and another layer 416 / 426 (e.g., a hidden layer of the ANN / SNN). In other words, the layers and their connections (here, for example, layers 424 and 426) and the connections between the neurons of layers 424 and 426 correspond to the structure of ANN 340 (i.e., the connections between layers 414 and 416 and their neurons). Similarly, the weights of the neurons of ANN 340 and SNN 360 can be corresponding.

[0080] In addition, the ReLU activation of the ANN 340 in the input layer 414 is replaced with a LIF activation in the input layer 424, which integrates the input over time and introduces a leakage term (like an RC circuit).

[0081] On the right hand side of the diagrams 410 and 420, exemplary outputs of input layers 414 and 424 are shown. As can be seen from the comparison of the output 412 of the input layer 414 and the output 422 of the input layer 424 plotted over time (see abscissas 418 and 428), pixel events from the EVS are no longer accumulated, but are fed to other layers (layer 426) of the SNN 360 as they occur (with a temporal resolution of 1 μs), while the output 412 is provided at predetermined periods at times t1 and t2. In practice, these periods take several milliseconds, resulting in a temporal resolution that is insufficient for many applications in rapidly changing systems.

[0082] Those skilled in the art will recognize that the LIF neurons of the input layer integrate the pixel events in the event data and that the SNN 360 continuously updates its representation and / or prediction of velocity (or other motion parameter) for each detected event in near real time within microseconds.

[0083] In hardware implementations of SNNs, such efficiency and low latency / high temporal resolution are achieved through two key advantages:

[0084] 1) Compared to other ANNs, neurons in SNNs may not provide the same accuracy as neurons in ANNs, but carry only one bit of information, which may enable higher processing rates than in other ANNs.

[0085] 2) According to the Address Event Representation (AER) communication protocol, zero values ​​are not transmitted. Therefore, in practice, only non-zero values ​​are transmitted, for example, through dense vectors carrying the index of neurons with non-zero values ​​(e.g., the index of the neuron of the input layer that received the detected event). In this way, the amount of data to be processed is reduced, thereby improving efficiency and time resolution / processing rate (reducing latency).

[0086] The EVS 310 may use the same AER protocol so that, as in 2), the sparseness of activity events results in a smaller amount of data to be transferred and processed.

[0087] It is further proposed that, optionally, the SNN 360 is further trained using the reference sensor (RGB camera 320) and the EVS (EVS 310). To this end, for example, additional event data may be obtained using the EVS 310, and additional motion data may be obtained using the RGB camera 320. Then, the SNN 360 may be further trained based on the additional event data and the additional motion data. In doing so, the weights of the SNN 360 may be further adjusted. To this end, for example, an STDB may be applied. In some embodiments, this may be done during the application of the SNN 360 to adapt the SNN 360 to the specific use case or environment of the application. Alternatively, this may also be done before use / application.

[0088] As will be appreciated by those skilled in the art, SNN 360 may be compatible not only with EVS 310, but also with other similar or equivalent EVSs. Therefore, it should be understood that SNN 360 may be used with EVS 310 or with another EVS.

[0089] Therefore, embodiments of the present disclosure may provide SNNs obtained by the methods and / or training systems proposed herein. As outlined in conjunction with the proposed methods / training systems, such SNNs are implemented, for example, in FPGAs or in neuromorphic hardware.

[0090] In fact, SNN can be deployed in event-based imaging systems. Figure 5 Describing exemplary applications of SNNs in more detail, Figure 5 A block diagram schematically illustrating an embodiment of an event-based imaging system 500 according to the present disclosure is shown.

[0091] For example, an event-based imaging system is configured to characterize and / or predict the motion of an object in an environment of the event-based imaging system, and includes an EVS 510 and an SNN 520 according to the present disclosure, wherein the EVS is configured to provide event data of an object in the environment of the event-based imaging system, and the SNN is used to characterize and / or predict the motion of the object based on the event data.

[0092] As outlined in conjunction with the methods and training systems for creating such SNNs, SNN 520 provides higher temporal resolution than other ANNs and therefore allows for the application of event-based imaging systems in rapidly changing systems such as drones, robotic systems, or automotive applications.

[0093] To this end, an input layer of the SNN 520 may be configured to receive event data and, when the input layer receives the event data, immediately provide output(s) indicative of the event data to another layer, except for the inherent latency of neurons in the SNN 520. In an embodiment, the inherent latency may be one microsecond or several microseconds. Thus, the input layer may be configured to provide an output in less than 1.5 microseconds or several microseconds after receiving the event data.

[0094] Due to the sparsity of the data to be processed for characterizing and / or predicting the motion of an object, the EVS 520 can be configured to process the event data according to the AER protocol to provide the event data to the SNN, which in turn can be configured to process the event data also according to the AER protocol for characterizing and / or predicting the motion of an object. In this process, the event data only includes the address of the pixel where the event was detected and the timestamp of the event. Similarly, the output of the neuron, for example, only includes the value and the index of the corresponding neuron. The sparsity thus obtained allows a faster processing rate than other less sparse data.

[0095] Figure 6 A block diagram schematically illustrating another embodiment 600 of an event-based imaging system is shown.

[0096] As can be seen from the block diagram, in addition to the EVS 610 and the SNN 620 as proposed herein, the event-based imaging system may further include a reference sensor, e.g., an RGB camera 630, for (further) training the SNN 620. The reference sensor may be any sensor configured to provide motion data of an object in the environment of the event-based imaging system to train the SNN 520 using the event data from the EVS 510 and the motion data of the reference sensor as training data, wherein the motion data may be real data, and the STDB may be applied to train the SNN 520.

[0097] In a use case, when the training system (e.g., the training system 200 or 300) is also used as an event-based imaging system, such a setup (also referred to as a "dual setup") is achieved, for example, by not removing the RGB camera (e.g., the RGB camera 320) after the initial training, or by using a camera that can capture frames and events simultaneously. This setup allows, in particular, the SNN 620 to be continuously trained on further observations of the EVS 610 and the RGB camera 630. To this end, for example, the object 660 is recorded by the EVS 610 and the RGB camera 630 to generate additional motion data 650 and additional event data 640, and the additional motion data 650 and the event data 640 are used as training data for training the SNN 620. To this end, for example, as described in conjunction with the training of the ANN, motion data used as real data can be obtained. In order to train the SNN 620, the weights of the SNN 620 can be updated using the STDB, wherein the weight update can be proportional to the spike timing difference between the current time and the most recent timestamp of the spike of the corresponding neuron. In this way, the SNN 620 is able to learn to predict the velocity of new objects in real time and optimize its predictions as the environment changes.

[0098] Event-based imaging systems can also apply and benefit from the inherent sparsity of the AER communication protocol and each processing scheme. Therefore, lower power consumption and lower latency can be achieved compared to other tracking systems. The LIF activation function in the input layer of the SNN integrates the pixel events from the EVS 610. The pixel events can reach a temporal resolution of 1μs, and the SNN 610 can use the new pixel events to update its predictions accordingly.

[0099] The proposed event-based imaging system can be deployed in various applications, including applications in drones, robotic systems, and vehicles.

[0100] Therefore, embodiments of the present disclosure may provide a vehicle, drone, or robot system including the event-based imaging system (e.g., event-based imaging system 500 or 600) proposed herein, as described below with reference to Figure 7 Explained in more detail.

[0101] Figure 7 A block diagram schematically illustrating an exemplary application of the proposed event-based imaging system is shown.

[0102] As can be seen from the block diagram, an event-based imaging system 710 according to the present disclosure can be installed in a device 700. Those skilled in the art who have benefited from the present disclosure will recognize that the device 700 can be any type of device. In an embodiment, the device is a vehicle, a drone, or a robotic system.

[0103] Additional embodiments relate to:

[0104] (1) A method for an event-based imaging system, the method comprising:

[0105] obtaining event data of an object in an environment of the event-based imaging system using an event-based vision sensor EVS;

[0106] obtaining motion data of the object using a reference sensor;

[0107] Using the event data and the motion data to train an artificial neural network (ANN) to characterize and / or predict the motion of the object; and

[0108] A spiking neural network (SNN) is created based on the trained ANN, and the spiking neural network (SNN) is configured to characterize and / or predict the motion of the object.

[0109] (2) The method according to (1), wherein the reference sensor includes an RGB camera, a LiDAR sensor, a RADAR sensor, an infrared sensor, and / or an ultrasonic sensor.

[0110] (3) The method of (1) or (2), wherein creating the SNN includes replacing the activation function of the ANN with a leaky integrate-and-fire (LIF) neuron.

[0111] (4) A method according to any one of (1) to (3), wherein creating the SNN includes implementing the SNN in a field programmable gate array (FPGA) or neuromorphic hardware.

[0112] (5) The method according to any one of (1) to (4), wherein obtaining the motion data comprises:

[0113] obtaining measurement data of the object from a reference sensor; and

[0114] The motion data is obtained using a Bayesian filter and using the measurement data as input to the Bayesian filter.

[0115] (6) The method according to (5), wherein obtaining the motion data further comprises:

[0116] obtaining motion samples by applying a Bayesian filter to the measurement data;

[0117] determining an uncertainty of the motion sample; and

[0118] The motion samples are filtered based on the uncertainty to obtain motion data.

[0119] (7) The method of (6), wherein the uncertainty of the motion sample is represented by the Mahalanobis distance of the motion sample, and wherein filtering the motion sample includes filtering out motion samples having a Mahalanobis distance greater than a predefined threshold.

[0120] (8) The method according to any one of (1) to (7), wherein the method further comprises:

[0121] Use EVS to obtain additional event data;

[0122] obtaining additional motion data using a reference sensor;

[0123] providing the event data and the motion data to an SNN; and

[0124] The SNN is trained based on the additional event data and the additional motion data provided to the SNN.

[0125] (9) The method according to (8), wherein training the SNN includes using spike timing dependent back propagation (STDB) to train the SNN.

[0126] (10) A spiking neural network (SNN), which is obtained by the method of any one of the preceding claims.

[0127] (11) The SNN of (10), wherein the SNN is implemented in a field programmable array (FPGA) or neuromorphic hardware.

[0128] (12) An event-based imaging system for characterizing and / or predicting motion of an object in an environment of the event-based imaging system, the event-based imaging system comprising:

[0129] an event-based vision sensor EVS configured to provide event data of objects in an environment of the event-based imaging system; and

[0130] The spiking neural network (SNN) according to (10) or (11) is used to characterize and / or predict the movement of the object based on the event data.

[0131] (13) The event-based imaging system of (12), wherein the SNN comprises a plurality of layers, the plurality of layers comprising an input layer, the input layer being configured as:

[0132] receiving the event data; and

[0133] When the input layer receives the event data, it immediately provides an output indicating the event data to another layer.

[0134] (14) An event-based imaging system according to (12) or (13), wherein the EVS is configured to process event data according to an Address Event Representation (AER) protocol to provide the event data to the SNN, and wherein the SNN is configured to process the event data according to the AER protocol for characterizing and / or predicting the motion of the object.

[0135] (15) The event-based imaging system according to any one of (12) to (14), wherein the event-based imaging system further includes a reference sensor for training the SNN.

[0136] (16) A training system for creating a spiking neural network (SNN) configured to characterize and / or predict motion of an object in an environment of an event-based imaging system, the training system comprising:

[0137] an event-based vision sensor EVS configured to provide event data of an object in an environment of the event-based imaging system;

[0138] a reference sensor configured to provide motion data of the object; and

[0139] An artificial neural network ANN is configured to train the ANN based on the event data and the motion data to characterize and / or predict the motion of the object, and to create an SNN for characterizing and / or predicting the motion of the object based on the ANN.

[0140] (17) The training system of (16), wherein the reference sensor comprises an RGB camera, a LiDAR sensor, a RADAR sensor, an infrared sensor, and / or an ultrasonic sensor.

[0141] (18) A vehicle, drone or robotic system comprising the event-based imaging system described in any one of (12) to (15).

[0142] Aspects and features described with respect to a particular example in the previous examples may also be combined with one or more of the other examples to replace the same or similar features of that other example or to otherwise introduce the features into that other example.

[0143] Examples may also be or relate to (computer) programs, which include program codes that execute one or more of the above methods when the program is executed on a computer, a processor, or other programmable hardware components. Therefore, the steps, operations, or processes of the different methods in the above methods may also be performed by a programmed computer, a processor, or other programmable hardware components. Examples may also cover program storage devices, such as digital data storage media, which are machine-readable, processor-readable, or computer-readable and encode and / or contain machine-executable, processor-executable, or computer-executable programs and instructions. Program storage devices may include or be digital storage devices, magnetic storage media (e.g., disks and tapes), hard drives, or optically readable digital data storage media. Other examples may also include computers, processors, control units, (field) programmable logic arrays ((F)PLA), (field) programmable gate arrays ((F)PGA), graphics processor units (GPUs), application-specific integrated circuits (ASICs), integrated circuits (ICs), or system-on-chip (SoC) systems programmed to perform the steps of the methods described above.

[0144] It should be further understood that the contents of several steps, processes, operations or functions disclosed in the specification or claims should not be interpreted as implying that these operations necessarily depend on the order described, unless explicitly stated in individual cases or necessary for technical reasons. Therefore, the foregoing description does not limit the execution of several steps or functions to a specific order. In addition, in other examples, a single step, function, process or operation may include and / or be decomposed into several sub-steps, sub-functions, sub-processes or sub-operations.

[0145] If some aspects have been described with respect to a device or system, these aspects should also be understood as descriptions of corresponding methods. For example, a block, device or functional aspect of a device or system may correspond to a feature of a corresponding method, such as a method step. Thus, aspects described with respect to a method should also be understood as descriptions of corresponding blocks or corresponding blocks, corresponding elements, characteristics or functional features of a corresponding system.

[0146] The following claims are hereby incorporated into the detailed description, wherein each claim may stand alone as a separate example. It should also be noted that although in the claims, a dependent claim refers to a specific combination with one or more other claims, other examples may also include a combination of a dependent claim with the subject matter of any other dependent claim or independent claim. Such combinations are hereby expressly proposed, unless it is clearly stated in an individual instance that a specific combination is not intended for use. In addition, features of a claim should also be included in any other independent claim, even if the claim is not directly qualified as dependent on other independent claims.

Claims

1. A method for an event-based imaging system, the method comprising: obtaining event data of an object in an environment of the event-based imaging system using an event-based vision sensor EVS; obtaining motion data of the object using a reference sensor; Using the event data and the motion data to train an artificial neural network (ANN) to characterize and / or predict the motion of the object; as well as A spiking neural network (SNN) is created based on the trained ANN, wherein the spiking neural network (SNN) is configured to characterize and / or predict the motion of the object.

2. The method according to claim 1, wherein: The reference sensor includes an RGB camera, a LiDAR sensor, a RADAR sensor, an infrared sensor and / or an ultrasonic sensor.

3. The method according to claim 1, wherein: Creating the SNN includes replacing the activation function of the ANN with leaky integrate-and-fire (LIF) neurons.

4. The method according to claim 1, wherein: Creating the SNN includes implementing the SNN in a field programmable gate array (FPGA) or neuromorphic hardware.

5. The method according to claim 1, wherein: Obtaining the motion data includes: obtaining measurement data of the object from the reference sensor; and The motion data is obtained using a Bayesian filter and using the measurement data as an input to the Bayesian filter.

6. The method according to claim 5, wherein: Obtaining the motion data further includes: obtaining motion samples by applying the Bayesian filter to the measurement data; determining an uncertainty of the motion sample; and The motion samples are filtered based on the uncertainty to obtain the motion data.

7. The method according to claim 6, wherein: The uncertainty of the motion samples is represented by a Mahalanobis distance of the motion samples, and wherein filtering the motion samples comprises filtering out motion samples having a Mahalanobis distance greater than a predefined threshold.

8. The method according to claim 1, wherein: The method further comprises: obtaining additional event data using the EVS; obtaining additional motion data using the reference sensor; providing the event data and the motion data to the SNN; and The SNN is trained based on the further event data and the further motion data provided to the SNN.

9. The method according to claim 8, wherein: Training the SNN includes training the SNN using spike timing dependent back propagation (STDB).

10. A spiking neural network (SNN), the spiking neural network (SNN) being obtained by the method according to any one of the preceding claims.

11. The SNN according to claim 10, wherein: The SNN is implemented in a field programmable array FPGA or neuromorphic hardware.

12. An event-based imaging system for characterizing and / or predicting motion of an object in an environment of the event-based imaging system, the event-based imaging system comprising: an event-based vision sensor EVS configured to provide event data of objects in an environment of the event-based imaging system; as well as A spiking neural network (SNN) according to claim 10 or 11, used to characterize and / or predict the movement of the object based on the event data.

13. The event-based imaging system of claim 12, wherein: The SNN comprises a plurality of layers, the plurality of layers comprising an input layer, the input layer being configured as: receiving the event data; as well as When the input layer receives the event data, an output indicating the event data is immediately provided to another layer.

14. The event-based imaging system of claim 12, wherein: The EVS is configured to process the event data according to an Address Event Representation (AER) protocol to provide the event data to the SNN, and wherein the SNN is configured to process the event data according to the AER protocol for characterizing and / or predicting the movement of the object.

15. The event-based imaging system of claim 12, wherein: The event-based imaging system further includes a reference sensor for training the SNN.

16. A training system for creating a spiking neural network (SNN) configured to characterize and / or predict motion of an object in an environment of an event-based imaging system, the training system comprising: an event-based vision sensor EVS configured to provide event data of said objects of said environment of said event-based imaging system; a reference sensor configured to provide motion data of the object; as well as An artificial neural network (ANN) is configured to train the ANN based on the event data and the motion data to characterize and / or predict the motion of the object, and to create an SNN based on the ANN to characterize and / or predict the motion of the object.

17. The training system according to claim 16, wherein: The reference sensor includes an RGB camera, a LiDAR sensor, a RADAR sensor, an infrared sensor, and / or an ultrasonic sensor.

18. A vehicle, drone or robotic system comprising the event-based imaging system according to claim 12.

Images