Low-latency event processing method and device
The method constructs event subgraphs with unidirectional edges and calculates node embeddings to process event data in real-time, addressing latency issues in existing technologies and enhancing applications like optical flow prediction.
Patent Information
- Application Number
- FR2023005360
- Authority / Receiving Office
- FR · FR
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-30
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-05-30
AI Technical Summary
Existing methods for processing event data using convolutional neural networks (CNNs) and vision transformers fail to leverage the sparse nature and low latency of event data, leading to performance degradation, particularly in applications like optical flow prediction, and graphical neural networks suffer from high latency in generating graphs.
A method involving a data processing device that constructs an event subgraph with unidirectional edges, calculates node embeddings, and applies a function to generate predictions, enabling real-time or near real-time processing of event data using a graphical neural network.
This approach achieves significantly reduced latency, typically by four orders of magnitude, and lower computational load, while maintaining efficient processing of event data, allowing for applications such as optical flow prediction.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Title of the invention: Method and device for low latency event processing Technical field
[0001] The present disclosure relates generally to the field of data processors and data processing methods and, in particular, to a data processor for processing event-based data generated by an event-based detector. Prior art
[0002] Event-based detectors are used in many applications and are particularly suitable for environments where input data is scarce. Indeed, an event-based detector provides significant power savings in such environments compared to a synchronous detector that periodically samples input data, even when there is no significant input signal. In addition, event-based detectors allow for a high data compression rate.
[0003] For example, event cameras differ from normal frame-based cameras in that instead of each pixel integrating a charge and recording a light intensity, event pixels generate binary events asynchronously as the light intensity changes. This allows information about motion in a scene to be significantly compressed. The same principle can also be applied to other detection modalities - for example, event-based audio.
[0004] Although convolutional neural networks (CNNs) and, more recently, vision transformers provide excellent means for solving frame-based computer vision tasks, they are not naturally compatible with event data. In order to process event data using such models, events are often embedded into so-called compact frames by, for example, counting the number of events generated per pixel in a given time window. However, this fails to take advantage of the sparse nature and low latency of event data. Furthermore, although these approaches allow the reuse of existing CNN architectures, and their optimized hardware implementations, the fine spatiotemporal details captured by event cameras are in practice ignored.This can lead to performance degradation, especially for applications such as optical flow prediction. Furthermore, compact frame CNNs do not easily take advantage of inherent sparsity. event data to reduce computational constraints.
[0005] It has been proposed to use graph neural networks to process asynchronous event data. One such example is the publication by Schaefer, Simon, Daniel Gehrig, and Davide Scaramuzza titled "AEGNN: Asynchronous Event-based Graph Neural Networks." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022. However, a technical difficulty with this solution is that graph generation is relatively slow, leading to relatively high latency.
[0006] There is a need in the state of the art for an improved device and method that can process event data in real time or near real time to enable processing closely coupled to the event detector. Summary of the invention
[0007] According to one embodiment, there is provided a method of generating, by a data processing device, a prediction based on event data captured by an event-based detector, the method comprising: receiving, by the data processing device, a new event from the event-based detector, the new event comprising a timestamp and an event data value having one or more dimensions; determining, by the data processing device in a list of past events, which of the past events fall within a time distance from the timestamp and within a data value distance from the event data value;constructing, by the data processing device, an event subgraph comprising a first node representing the new event, additional nodes representing each of the determined past events, and unidirectional edges connecting each additional node to the first node; calculating, by the data processing device, node embeddings for the first node based on edges consisting only of said unidirectional edges connecting each additional node to the first node; and generating the prediction by applying a function to the node embeddings.
[0008] According to one embodiment, the method further comprises controlling one or more actuators based on the prediction.
[0009] According to one embodiment, determining past events comprises performing a search for the K nearest neighboring events in a past search volume having an event data value radius in the one or more event data value dimensions, and a time radius in the time domain.
[0010] According to one embodiment, the event-based detector is a camera. event, and the prediction is an optical flow prediction.
[0011] According to one embodiment, the data processing device is configured to implement real-time processing of the new event, the data processing device being for example configured to generate the prediction during a processing delay of less than 100 microseconds, and, preferably, of less than 5 microseconds, after a time of the new event.
[0012] According to one embodiment, generating the prediction comprises passing the event subgraph into a graph neural network.
[0013] According to one embodiment, the graph neural network comprises a plurality of layers, and passing the event subgraph into the graph neural network comprises: calculating an embedding vector for the first node and for additional nodes of the subgraph using the embeddings of all events connected by said unidirectional edges, the embedding vector comprising a subvector for each of said layers; and applying a transformation to the embedding vector of each node, or to the input event data in the case of a first layer of the graph neural network, to generate a resulting embedding vector.
[0014] According to a further aspect, a data processing device is provided for generating a prediction based on event data captured by an event-based detector, the data processing device being configured to: receive a new event from the event-based detector, the new event comprising a timestamp and an event data value having one or more dimensions; determine, in a list of past events, those of the past events that fall within a time distance from the timestamp and within a data value distance from the event data value; construct an event subgraph comprising a first node representing the new event, additional nodes representing each of the determined past events, and unidirectional edges connecting each additional node to the first node;calculating node embeddings for the first node based on edges consisting only of said one-way edges connecting each additional node to the first node; and generating the prediction by applying a function to the node embeddings.;
[0015] According to one embodiment, the data processing device is further configured to control one or more actuators based on the prediction.
[0016] According to one embodiment, the data processing device is configured to determine said past events by performing a search for the K nearest neighboring events in a past search volume having a radius of event data value in the one or more dimensions of the event data value, and a time ray in the time domain.
[0017] According to one embodiment, the event-based detector is an event camera, and the prediction is an optical flow prediction.
[0018] According to one embodiment, the data processing device is configured to perform real-time processing of the new event, the data processing device being, for example, configured to generate the prediction within a processing time of less than 100 microseconds, and preferably less than five microseconds, after an instant of the new event.
[0019] According to one embodiment, the data processing device is configured to generate the prediction by passing the event subgraph through a graph neural network.
[0020] According to one embodiment, the graph neural network comprises a plurality of layers, and passing the event subgraph into the graph neural network comprises: computing, by the graph neural network, an embedding vector for the first node and for additional nodes of the subgraph using the embeddings of all events connected by said unidirectional edges, the embedding vector comprising a subvector for each of said layers; and applying, by the graph neural network, a transformation to the embedding vector of each node, or to the input event data in the case of a first layer of the graph neural network, to generate a resulting embedding vector.
[0021] According to yet another embodiment, a control system is provided comprising: the previous data processing device; an event-based detector connected to the data processing device and configured to generate the new event and to transmit the new event to the data processing device; and the one or more actuators. Brief description of the drawings
[0022] These characteristics and advantages, as well as others, will be explained in detail in the following description of particular embodiments given without limitation in relation to the attached figures among which:
[0023] [Fig.l] schematically represents an event-based processing and detection system according to an exemplary embodiment of the present description;
[0024] [Fig.2] schematically represents an event processing device of [Fig.l] in more detail according to an exemplary embodiment of the present description;
[0025] [Fig.3] is a graph representing an example of the event cloud generated from an event camera recording;
[0026] [Fig.4] is a graph representing an example of creating an event graph based on k jumps between indirectly connected events;
[0027] [Fig.5] is a flowchart representing an example of steps of a method for determining an optical flow prediction;
[0028] [Fig.6] is a graph representing an example of creating an event graph according to an exemplary embodiment of the present description;
[0029] [Fig.7] is a flowchart representing an example of steps of a method for determining an optical flow prediction according to an embodiment of the present description;
[0030] [Fig.8] represents the generation of an embedding vector in each layer of an event graph according to an exemplary embodiment of the present description;
[0031] [Fig.9] represents the concatenation of embedding vectors and their application to a neural network; and
[0032] [Fig. 10] shows a neural network head for performing a prediction according to an exemplary embodiment of the present disclosure. Description of the embodiments
[0033] The same elements have been designated by the same references in the different figures. In particular, the structural and / or functional elements common to the different embodiments may have the same references and may have identical structural, dimensional and material properties.
[0034] For the sake of clarity, only the steps and elements useful for understanding the described embodiments have been shown and are detailed. In particular, devices and methods for event-based detection are known in the state of the art and have not been described in detail. Furthermore, training a graph neural network and using a graph neural network to generate event-based detection or prediction is well known to those skilled in the art and will not be described in detail.
[0035] Unless otherwise specified, when referring to two elements connected to each other, this means directly connected without intermediate elements other than conductors, and when referring to two elements connected (in English "coupled") to each other, this means that these two elements can be connected or be connected by means of one or more other elements.
[0036] In the following description, when referring to absolute position qualifiers, such as the terms "front", "back", "top", "bottom", "left", "right", etc., or relative position qualifiers, such as the terms "above", "below", "upper", "lower", etc., or to orientation qualifiers, such as the terms "horizontal", "vertical", etc., it is made reference unless otherwise specified to the orientation of the figures.
[0037] Unless otherwise specified, the expressions "about", "approximately", "substantially", and "of the order of" mean to within 10%, preferably to within 5%.
[0038] [Fig. 1] schematically represents an event-based detection and processing system 100 according to an exemplary embodiment of the present disclosure. The system 100 includes an event-based sensor (EVENT-BASED SENSOR) 102 configured to generate temporal event data (TEMPORAL EVENT DATA), and an event processing device (EVENT PROCESSING DEVICE) 104 configured to receive the temporal event data and to process it to generate an event-based prediction (EVENT-BASED PREDICTION).
[0039] For example, the event-based detector 102 is an event-based camera, or event camera, such as a visible light or infrared event camera. Alternatively, the event-based detector could be an event-based audio detector, such as an event-based microphone. Other types of event-based detector would also be possible.
[0040] The temporal event data includes, for example, asynchronous events, each event including, for example, a timestamp and an event data value having one or more dimensions. For example, in the case where the event-based detector 102 is a camera, the event data value could be the x and y address of the pixel that captured the event, and the event is, for example, generated at the time a light level change exceeds a threshold relative to the light level at the time the previous event was generated at the pixel. In addition, a detected light level of the pixel could provide an additional dimension of the event data value, for example, the polarity of the change or the absolute light level at the time the event was generated.In the case where the event-based detector 102 is a microphone, events are generated for example due to relative changes in the analog signal generated by the microphone and the characteristics of each event may also include a timestamp, the absolute value of the signal and / or, if the signal is filtered into several frequency bands, the range of the frequency band of the event.
[0041] The event-based processing device 104 is for example configured to process the time event data in real time or near real time, for example with a processing delay of less than one hundred microseconds, and preferably less than five microseconds.
[0042] In the case where the temporal data is image data, the event-based prediction output from the event processing device 104 is by example an optical flow prediction per generated event, or an object classification for the entire event stream, or an object detection per event. In the case where the temporal data is audio data, the event-based prediction provided as output from the event processing device 104 is for example a keyword identification per group of events or a sound class recognition.
[0043] [Fig.2] schematically represents the event processing device 104 of [Fig.l] in more detail according to an exemplary embodiment of the present description.
[0044] The device 104 comprises for example a processing device (P) 202 comprising one or more processors, and one or more memory devices, such as a volatile memory (RAM) 204 and / or a non-volatile memory (FLASH) 206. In some embodiments, the volatile memory 204 is a random access memory, and / or the non-volatile memory 206 is a Flash memory, although other types of volatile and non-volatile memory may be present.
[0045] The processing device 202 and the memories 204, 206 are for example connected by a bus 208. The non-volatile memory 206 stores for example a software code, which is executed by the processing device 202.
[0046] The event processing device 104 further comprises, for example, a sensor interface (SENSOR INTERFACE) 210 connected to the bus 208, the sensor interface 210 being for example configured to communicate with the event-based detector 102 over a wired or wireless communications interface.
[0047] In some embodiments, the event processing device 104 further comprises a graph neural network (GRAPH NEURAL NETWORK) 212, which is for example also connected to the bus 208, and is configured to generate the event-based prediction (EVENT-BASED PREDICTION) at an output of the event processing device 104. For example, although not shown in [Fig. 2], the event-based prediction is provided by the graph neural network to an output interface of the device 104 connected to the bus 208.
[0048] The graphical neural network 212 is, for example, a hardware circuit configured to implement a method of generating a prediction based on event data captured by the event-based detector 102 described in more detail herein. According to alternative embodiments, the graphical neural network is implemented in software, and the method of generating a prediction based on event data captured by the event-based detector 102 is implemented by software stored in the memory 204 or 206 and executed by the processing device 202. According to one alternative, the processor reads data from the event-based detector, processes this data, and then provides this data to the network neural graph in the appropriate format to perform event-based prediction.
[0049] In some embodiments, the event-based prediction generated by the graph neural network 212 is used to control one or more actuators (ACTUATOR(S)) 214 connected to the device 104. The one or more actuators 214 include, for example, a robotic system, such as a robotic arm trained to pull weeds or pick rotten fruit from a tree, automatic steering or braking systems in a vehicle, or an electronic actuator, which is, for example, configured to control the operation of one or more circuits, such as waking up a circuit from a sleep mode, causing a circuit to enter a sleep mode, causing a circuit to generate a text output, performing a data encoding or decoding operation, etc.In another example, the device 104 is an eye tracking unit that determines the area in a scene that a user is viewing and controls actuators based on the determined area, for example to change the focus of the stereogram.
[0050] [Fig. 3] is a graph representing an example of the event cloud generated by an event camera recording. In particular, the graph of [Fig. 3] is a 3D graph representing, on an x-axis, a pixel coordinate (X PIXEL) in an x-direction of a pixel matrix of the camera, on a y-axis, a pixel coordinate (Y PIXEL) in a y-direction of the pixel matrix and, on a z-axis, the time in seconds (TIME(s)). The spiral of data represented in [Fig. 3] results from a circular movement of a person's hand in the image.
[0051] By analyzing the 3D data structure of the data events of [Fig.3], or similarly by analyzing a 2D, 4D or higher dimensional data structure of event data, it is possible to perform prediction operations, such as classification to determine what type of object created the events, or optical flow prediction, for example to determine the velocity vector of the object, or detection, for example, of whether a certain object is present in the scene.
[0052] [Fig.4] is a graph representing an example of an event graph in which the number of k hops between indirectly connected events is indicated. The creation of the event graph involves, in particular, the generation of a tree structure, followed by a nearest neighbor search in the tree that allows edges between events (i.e., vertices) to be defined. For example, the creation of a KD-tree structure is described in more detail in the publication by Zhou, Kun et al. entitled “Real-time KD-Tree Construction on Graphics Hardware”, ACM Transactions on Graphics (TOG) 27.5 (2008):1-11. A set of events is recorded and the tree structure is created. This tree structure is used to search for a set of nearest neighbors for each event. Edges are formed between these events (i.e., nodes) based on a distance measure in the pixel dimensions (e.g., x,y) and the time dimension (t). For example, a directed edge is connected between the M nearest events within a fixed search radius.
[0053] This method is illustrated in Figure 4, which shows time (TIME) on the x-axis and an x,y position (XY POSITION) of pixel data on the y-axis. A new event O for a given pixel is detected by an event-based detector at a time Other events (EVENTS) are also represented on this graph. A search volume S(rxv, rt) is defined to be within a time distance ft from the time of the new event, and within a spatial distance r*y from the new event, as represented by an ellipse S. Half of this search volume S is before the time of the new event and the other half of the search volume S is after the arrival time of the new event L,. Thus, as represented in Figure 4, the search volume includes events within a radius of the new event C.Among the events that enter the search volume, the K nearest neighbors are connected to the new event e< by edges. These edges can be inherently bidirectional or unidirectional, although often two unidirectional edges will form between pairs of events with such a Euclidean-based search paradigm. Furthermore, importantly, as many events are generated, the edge configurations of existing events in the tree are also updated in the case that new events C become one of the nearest neighbors of existing events. As will be explained in more detail below in relation to Figure 5, the event graph is then updated in view of the new event and the new edges.In particular, the effect of the new event ei on existing event graph node embeddings is evaluated for all L layers of the event graph neural network and possibly a large number of times. Depending on the method of updating the node embeddings, this can be computationally very expensive and involve significant latency. To avoid updating the node embeddings of the entire active event graph every time a new event is generated, a recursive k-hop graph search function hopO can for example be applied to find the sparse subset of impacted nodes in each of the L layers. Only this subset of nodes will be updated at the time of arrival of each new event, which can slightly ease the computational load. As shown in Figure 4, the direct connections with . the new event ei are connected by a single hop, corresponding to k=0, and an integer number K of additional hops is determined by means of intermediate connected nodes. In the example in Figure 4, K=4 between the central event ei and the event in the upper-left corner of the diagram.
[0054] [Fig.5] is a flowchart representing an example of steps of a method for determining an optical flow prediction according to the approach represented by the graph of [Fig.4].
[0055] In a step 501, a new event e' is received.
[0056] In a step 502, a subset of past events eJ is determined in the distance r*y and rt from event ei. In other words, past events that fall within the search volume S of [Fig.4] are identified.
[0057] In a step 503, for each past event ef, a search of the K nearest neighbors is performed, and edges are created. In particular, edges e> are created from each past event eJ of the K nearest neighbors to the new event G. Furthermore, edges e> eJ are created from the new event eî to each of the K nearest neighbors in the past, and the node embeddings of all nodes impacted over an integer number of K hops from an event ei must be updated. The event eJ will subsequently be likely to be modified several times when several new events are generated within a time equal to the time search radius, which forms direct connections K=0 with future events, multiplied by the number of layers (via modifications through nodes connected by a certain number of K hops).
[0058] In a step 504, the effect of a new event ei on existing event graph node embeddings for all L layers of the event graph neural network is evaluated. This involves the recursive k-hop graph search function hopO, which finds the impacted sparse subset of nodes in each layer 1 of the event graph, as shown in [Fig.4].
[0059] In a step 505, embeddings of affected nodes in each layer l are updated using a graph convolution function. The graph convolution function, ¢0, may be for example the spectral graph convolution proposed in the publication "SEMI-SUPERVISED CLASSIFICATION WITH GRAPH CONVOLUTIONAL NETWORKS" Kipf 2017, Z{ = f(A' ■ • ^), where A' — ■ A ■ IJ112, A is the adjacency matrix that stores the 0 or 1 in each matrix cell indicating whether nodes are connected and D is the degree matrix.
[0060] In a step 506, after a delay equal to the radius rt of the search window multiplied by the number of layers of the graphical neural network, time during which new additional events which impact the event e> can to be received and processed, a function is applied to the node embeddings to generate an output prediction V e.g. an optical flow prediction, related to the new event ei.
[0061] The method of [Fig.5] can for example be represented by the following algorithm.
[0062] [Math.l] Algorithm 1 Sparse fully-spherical update Tnput: evt = {.¾ y*. G = {EV, £}, rt, rxy Output: Vj M = EV n v.. rxy, rt) for evm GM do £m = knn(evt^ end for G = ir{G,£rn,evi) for / in range(L) do for ev.j G hop(M, G, 1) do zjj = G, l) end for end for if t > ti + (rt x L) then 14 = a(Z;) end if
[0063] where evi is the new event, xi, are data values of the event data value, h is the timestamp of the event, Pi is the event polarity, G is the event graph, EV is a subset equal to [x, y, t} e R < are the edges of the event graph connecting the EV events, B() is a function used to extract the subset from the previous step 502, knnQ is the K nearest neighbor search, ttÇ) is a function that takes as input the state of the previous graph, the new event, and new and recalculated edges E,„ and outputs an updated event graph, hopÇ} is the recursive k-hop graph search function, and Gi are the node embeddings in a given layer 1.
[0064] A major drawback of the method of Figure 5 is that it results in an excessive number of calculations and a relatively large latency between the time of appearance of the new event and the output of the prediction. This is due in part to the complexity of the processing, and also to the latency of constructing a graph, including the G*L delay, which is typically several hundred milliseconds.
[0065] Figure 6 is a graph representing an example of event graph creation according to an exemplary embodiment of the present disclosure. The graph of Figure 6 is similar to that of Figure 4 except that the search volume S is replaced by a search volume S', and the edges (EDGES) between past events (EVENTS) and the new event ei are strictly unidirectional, as will now be explained in more detail.
[0066] According to the embodiment shown in [Fig.6], the latency of building a graph is reduced by limiting the creation of edges in event graphs so that directed unidirectional edges are formed only from past events to a newly arrived event within a hemispherical or hemi-elliptical search radius S' around the newly arrived event. In other words, when a new event arrives, events with which it will connect and from which it will obtain data are searched only in the past neighborhood. Edges to past events, which were created according to the previous method, are no longer created, and therefore the edges are unidirectional in time, from the past to the future, as represented in [Fig.6] by arrows pointing forward in time only.This allows a prediction for an event to be calculated immediately for a generated event. Indeed, it can be guaranteed that neither the node embeddings of this event, nor its edges, will change in the future. This greatly reduces latency as well as the amount of computation.
[0067] In the case of a graph neural network, these edges are used to immediately calculate node embeddings of only the newly generated event for each layer of the network. Past events now no longer change their embedding once calculated. This differs from previous approaches in which a large number of past events must be recalculated at the time of arrival of each event. Only embeddings from past events within a certain search radius of time and space to which it is connected are used in this method.
[0068] According to the approach of [Fig.6], edges, i.e., the identities of two events connected by the edge as well as any edge properties such as a time difference, are for example not stored explicitly in memory because the information about these edges is implicitly represented by the embeddings of previously generated event nodes. Thus, this approach saves memory storage, such as memory 204 and / or 206 of [Fig.2], which can be considerably smaller. In addition, past events that fall outside the time search radius are for example not stored. This differs from previous approaches in which events within a time window equal to the search radius multiplied by the number of layers of the neural network are stored, and all edges connecting these events are also stored.
[0069] The foregoing factors allow for implementation with a significantly reduced memory footprint compared to other approaches. More particularly, a Real-time implementations of the embodiments described herein, for example, incorporate functionality in which the memory of the system in which the algorithm is incorporated is continuously cleaned so that events older than a certain time are removed.
[0070] In some embodiments, in order to maintain a number of detected events that is relatively similar to the approach of Figure 4, the radius rt is for example increased relative to a typical value used according to this approach. Typically, this is achieved by increasing, for example substantially doubling, the radius} t so that the search volumes are equivalent. For example, the radius rt in the case of Figure 4 is for example between 10 and 20 ms, and the radius rt in the case of [Fig.6] is for example between 20 and 40 ms.
[0071] [Fig.7] is a flowchart of an example of steps of a method for determining an optical flow prediction according to an embodiment of the present disclosure. This method could also be used to determine other types of predictions, such as classification or object detection. In the case of classification, in which an event-by-event output may not be required, the node embedding of all events generated in a certain time window may be aggregated, for example using max or average pooling, and it is this aggregated node embedding that is, for example, periodically given to the output prediction function.
[0072] The method of [Fig.7] is for example implemented by the event processing device 104 of [Fig.l].
[0073] In a step 701, a new event ei received by the event processing device 104, for example from the event-based detector 102.
[0074] In a step 702, a search for the K nearest neighboring events is for example performed in a past search volume of radius ? in the one or more dimensions of the event data value, and of radius C in the time domain. For example, this is achieved by performing a distance measurement between the new event and the past events, the distance calculation being for example based on the L2 norm. It would also be possible to use other distance metrics such as the L1 norm or simply the maximum distance function, corresponding to the use of the maximum of all dimensions (such as the maximum of all three dimensions in the case where there are three).
[0075] In a step 703, a subgraph is created containing the new event ei and, for each neighbor ei among the K nearest neighbors, a unidirectional edge
[0076] In a step 704, L node embeddings are calculated for the event e> using a graphical convolution function ÿ'(). This function 0'0 is by example identical to the function ¢0, except that the input of this function is a past-only graph similar to the one shown in [Fig.6], and thus the edges that are used for the calculation of the L node embeddings consist only of strictly unidirectional edges (past to present).
[0077] In a step 705, a cr'O function is for example applied to the node embeddings Z; to generate an optical flow prediction V, for the new event and. For example, the cr'O function is applied by passing the subgraph through a neural network such as that described in relation to [Fig.6].
[0078] The method of [Fig.7] can for example be represented by the following algorithm.
[0079] [Math.2] Algorithm 2 Hemi-spherical update Input: evi = {^,^,0,^),27^,7^,7^ Output: Vi Ki- Si — knn^ EV. rt, rxy) G4 = {A,.£,} for l in range(L) do end for
[0080] where is the new event, \ are data values of the event data value, t; is the timestamp of the event, Pi is the event polarity, K, are the nearest neighbor events connected by unidirectional edges £., Gj is the subgraph, and knrQ is the search for the K nearest neighbors.
[0081] Figure 8 represents the generation of an embedding vector in each layer of an event graph according to an exemplary embodiment of the present description. In particular, a graph neural network is used to calculate the node embeddings. Figure 8 represents an example in which the event graph neural network, applied to the event graph G^, comprises five layers 11 to 15. In each layer, an embedding vector is calculated for an event using the embeddings of the connected events. For example, Figure 8 represents an example of an embedding matrix for an event j Z / associated with one of the connected events eJ of the new event ei, the embedding matrix Zj comprising sub-vectors Z7 j to Z for each layer 1 to 5.The embedding vector ^jA of the first layer is for example made up of the corresponding event, including for example the temporal information in the form of a timestamp, and the value of the event data having one or more . dimensions. In the example in Figure 8, each embedding vector -Z;,i comprises three values, which may, for example, in the first layer correspond to the timestamp and pixel coordinates in X and Y respectively and in later layers correspond to an abstract vector representation of the event. To obtain the embedding in each of these subsequent layers, in each subsequent layer, a transformation is applied to the embedding vector of each node, or to the input event data in the case of the first layer, to generate a resulting embedding vector. This transformation also takes into account the embeddings of other nodes to which the node is connected by an edge. For example, the transformation could be a sum or an average of the embedding vectors or something more complex based on edge information.The layers of the event graph neural network are for example chained sequentially as is the case for graph neural networks.
[0082] Figure 9 represents the concatenation of embedding vectors and their application to a neural network according to an exemplary embodiment of the present description. In particular, the embedding vectors Za of each layer of the neural network associated with the new event ei are for example processed by passing them into a neural network in order to generate a prediction associated with the new event e>. In the example of Figure 9, this involves a concatenation of the embedding vectors to form a single vector and a processing of this vector by a multi-layer perceptron. For example, in a first layer L1 of the multi-layer perceptron neural network, an instance normalization can be applied in order to produce a first vector representation vl for the event.To generate the prediction Vj, which is for example an optical flow prediction, this first vector representation vl is for example processed by one or more additional layers comprising an output layer (OUTPUT LAYER), which provides two output values and Vy, which correspond for example to the x and y components of the event optical flow vector Vh.
[0083] [Fig. 10] schematically represents a neural network head 1100 for performing a prediction according to an exemplary embodiment of the present disclosure. [Fig. 10] particularly represents an example in which there are five layers 11 to 15 in the graph neural network, and the state of the outputs of each layer, for each node, is provided as input to the neural network head 1100. For layers 12 to 15, the input of each layer is the output of the previous layer, while applying, for example, new weights and new message passing parameters. For example, in each layer, the same shared weight is applied to all nodes. The weight is different for each layer. Passing a message depends on the type of convolution. For many convolution types, there are no parameters. For more complex types, such as a B-spline convolution, there is, for example, the number of basis functions / kernels to be set as a parameter, which could change from layer to layer.
[0084] The graphic neural network 1000 comprises for example, in the example of [Fig. 10], a layer L of four neurons receiving a concatenation of the node embeddings of a given node of each of the layers 11 to 15 of the graphic neural network. For example, the neurons of the layer L are configured to apply a linear rectification activation function. An instance normalization can be applied to the activations of this layer. The outputs of the neurons of the layer L are for example provided to a single output neuron Nout, which is for example configured to generate a prediction corresponding, for example, to a probability between 0 and 1.
[0085] An advantage of the graph creation algorithm described herein is that it achieves considerably lower latency, typically about four orders of magnitude, and a smaller total computational load, e.g., two orders of magnitude, compared to prior approaches. In addition, edges are, for example, no longer stored in memory, and events that are older than a time equal to the search time radius are, for example, removed from memory.
[0086] Various embodiments and variations have been described. Those skilled in the art will understand that certain features of these various embodiments and variations could be combined, and other variations will occur to those skilled in the art.
[0087] For example, in the case of a graph neural network, pooling operations can be used to grow the graph, thereby reducing the number of nodes as well as the amount of computation required. For example, a pooling operation is triggered periodically for given layers of the graph neural network using, for example, a given voxel grid dimension, i.e., a periodic grid in the spatial (xy) and temporal (t) dimensions of the input event cloud. Events in a single voxel for example have their features aggregated into a single super-event and the connections between these super-events are for example based either on retaining previously existing connections between voxels or on searching again, in the past only, and defining a new set of edges.
[0088] Further, although embodiments have been primarily described in the context of camera data, i.e., event data values in two dimensions, x and y, the techniques described herein could be applied to other types of event-based detectors, such as a cochlea or an event-based silicon earpiece. In such a case, rather than encoding events in space and time, the silicon ear encodes events in, for example, time and frequency-band energy—resulting in a 2D point plane instead of a 3D point cloud in the case of event cameras. For example, such a silicon earpiece is described in more detail in the publication by Chan, Vincent, Shih-Chii Liu, and Andr van Schaik. "AER EAR: A matched Silicon cochlea pair with address event representation interface." IEEE Transactions on Circuits and Systems I: Regular Papers 54.1 (2007): 48-59. Furthermore, the embodiments described herein could be applied to any application with a temporal or causal dimension.For example, they could be applied to data structures that can be gradually explored, with the possible limit that no loops are created (transposition of the condition "edges only connect the past to the future", meaning that the resulting graph with unidirectional edges has no loops).
[0089] Furthermore, any step could be applied to the data before and / or after the proposed graph construction. For example, data filtering could be applied, whereby an event at a given pixel of the same polarity emitted within a certain time window will be eliminated before processing. Other options are normalization of input features and time encoding using a single or a pair of periodic functions, such as a sawtooth wave, instead of absolute time which could impose some memory load.
[0090] Furthermore, the distance between nodes can be calculated with any distance metric - L2 norm, L1 norm, also known as Manhattan distance, Euclidean distance, etc.
[0091] In addition, the features contained in each node can, for example, be modified. Some of the possibilities that can be used are time, x and y coordinates of the event, a normalized vector approximated to a local event plane (see the publication by Mitrokhin, Anton, et al. "Learning visual motion segmentation using event surfaces." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020.), the polarity of the event, etc.
[0092] Node embedding normalization schemes can be used to normalize node features in intermediate layers of the graph neural network. The use of instance normalization is particularly interesting because it is a technique using only node-level statistics rather than a batch norm or a layer norm, which require pooling statistics from multiple nodes or even multiple graphs.
[0093] The characteristics stored in the created edges can also be modified. For example, a simple binary flag can be used to indicate a connection or the differences in spatial and temporal coordinates between connected events, which has been determined to be particularly powerful.
[0094] Search radii can be modified, i.e., in the same way for all graphs, or dynamically, for example, according to certain rules. For example, for a high density of events over time, which suggests a fast-moving edge, in a certain region of the scene, the search radius could be reduced in order to preserve spatial relationships between events generated by an object independently of its speed. Alternatively, the approximate normal vector of an event can be used to adapt the search radius - a larger vector in given dimensions can be used to decrease the search radius as long as it also implies a fast-moving object. On the contrary, for a normal vector evaluated as small, the search radius could be enlarged.
[0095] In some embodiments, the number of neighbors may be limited, may be fixed or dynamic, or may not be limited.
[0096] The processing performed is not necessarily a graph neural network, as discussed in the publication titled "Graph Convolutional Neural Network (GCN), with the gconv and spline layers", available at https: / / pytorch-geometric.readthedocs.io / en / latest / modules / nn.html. For example, processing schemes that are not AI-based can be applied, including graph search algorithms.
[0097] Finally, the practical implementation of the embodiments and variants described is within the reach of those skilled in the art from the functional indications given above.
Claims
Claims
1. A method of generating, by a data processing device (104), a prediction based on event data captured by an event-based detector (102), the method comprising: - receiving, by the data processing device (104), a new event (e0 from the event-based detector (102), the new event comprising a timestamp and an event data value having one or more dimensions; - determining, by the data processing device (104) in a list of past events, those of the past events (ej) that fall within a time distance (r0 from the timestamp and within a data value distance (rxy) from the event data value;- constructing, by the data processing device (104), an event subgraph comprising a first node representing the new event (e0, additional nodes representing each of the determined past events (fy), and unidirectional edges connecting each additional node to the first node; - calculating, by the data processing device (104), node embeddings (¾ for the first node based on edges consisting only of said unidirectional edges connecting each additional node to the first node; and - generating the prediction (V,) by applying a function (tf'O) to the node embeddings (¾).;
2. The method of claim 1, further comprising controlling one or more actuators (214) based on the prediction (V;).
3. The method of claim 1 or 2, wherein determining past events (ej) comprises performing a search for K nearest neighboring events in a past search volume having an event data value radius (rxy) in the one or more event data value dimensions, and a time radius (rf) in the time domain.
4. A method according to any one of claims 1 to 3, wherein the event-based detector (102) is an event camera, and wherein the prediction (V,) is an optical flow prediction.
5. A method according to any one of claims 1 to 4, wherein the data processing device (104) is configured to implement implements real-time processing of the new event (ei\ the data processing device is for example configured to generate the prediction (V,) during a processing delay of less than 100 microseconds, and preferably less than 5 microseconds, after a time of the new event (er).
6. The method of any one of claims 1 to 5, wherein generating the prediction (V,) comprises passing the event subgraph through a graph neural network (212).
7. The method of claim 6, wherein the graph neural network (212) comprises a plurality of layers, and passing the event subgraph into the graph neural network (212) comprises: - calculating an embedding vector (Zjj) for the first node and for additional nodes of the subgraph using the embeddings of all events connected by said unidirectional edges, the embedding vector (Z / ) comprising a subvector (Zy i to Zy^) for each of said layers; and - applying a transformation to the embedding vector (Z of each node, or to the input event data in the case of a first layer of the graph neural network, to generate a resulting embedding vector (¾).
8. A data processing device for generating a prediction based on event data captured by an event-based detector (102), the data processing device being configured to: - receive a new event (e0 from the event-based detector (102), the new event comprising a timestamp and an event data value having one or more dimensions; - determine, from a list of past events, those of the past events (fy) that fall within a time distance t) from the timestamp and within a data value distance (r <v) de la valeur de données d'événements ; - construire un sous-graphe d'événements comprenant un premier nœud représentant le nouvel événement (^ / ), des nœuds supplémentaires représentant chacun des événements passés déterminés (Vj), et des bords unidirectionnels reliant chaque nœud supplémentaire au premier nœud ;- calculate node embeddings (Zf) for the first node on the; edge base consisting only of said unidirectional edges connecting each additional node to the first node; and - generate the prediction (Ven apply a function (c'O) to the node embeddings (¾).
9. A data processing device according to claim 8, further configured to control one or more actuators (214) based on the prediction (V,).
10. A data processing device according to claim 8 or 9, configured to determine said past events (fy) by performing a search for the K nearest neighboring events in a search volume having an event data value radius (rxr) in the one or more dimensions of the event data value, and a time radius (r0 in the time domain.
11. A data processing device according to any one of claims 8 to 10, wherein the event-based detector (102) is an event camera, and wherein the prediction (^) is an optical flow prediction.
12. A data processing device according to any one of claims 8 to 11, wherein the data processing device (104) is configured to perform real-time processing of the new event (^), the data processing device being, for example, configured to generate the prediction (V,) within a processing time of less than 100 microseconds, and preferably less than five microseconds, after a time of the new event
13. A data processing device according to any one of claims 8 to 12, configured to generate the prediction (V,) by passing the event subgraph through a graph neural network (212).
14. A data processing device according to claim 13, wherein the graph neural network (212) comprises a plurality of layers, and passing the event subgraph into a graph neural network (212) comprises: - calculating, by the graph neural network (212), an embedding vector for the first node and for additional nodes of the subgraph using the embeddings of all events connected by said unidirectional edges, the embedding vector (Zy) comprising a subvector (Zyj to Zy^) for each of said layers; and - applying, by the graphic neural network (212), a transformation to the embedding vector (Zjj) of each node, or to the input event data in the case of a first layer of the graphic neural network, in order to generate a resulting embedding vector (¾).
15. A control system comprising: - the data processing device according to any one of claims 8 to 14; - an event-based detector (102) connected to the data processing device and configured to generate the new event (ei) and to transmit the new event to the data processing device; and - the one or more actuators (214).
