Low-latency event processing method and device

Abstract

Method and device for low-latency event processing. This description relates to a method for generating, by a data processing device, a prediction based on event data captured by an event-based detector, the method comprising: receiving, from the event-based detector, a new event ( ) comprising a timestamp and an event data value having one or more dimensions; determining past events that fall within a time distance ( ) from the timestamp and within a data value distance ( ) from the event data value; constructing an event subgraph comprising a first node representing the new event ( ), additional nodes each representing the determined past events, and one-way edges connecting each additional node to the first node;the calculation of node embeddings for the first node based on edges consisting solely of said unidirectional edges; and the generation of the prediction by applying a function to the node embeddings. Figure for the abstract: Fig. 6;

Description

Title of the invention: Method and device for low-latency event processing technical field

[0001] This description relates generally to the field of data processors and data processing methods and, in particular, to a data processor for processing event-based data generated by an event-based detector. Previous technique

[0002] Event-based detectors are used in many applications and are particularly well-suited to environments where input data is scarce. Indeed, an event-based detector provides significant energy savings in such environments compared to a synchronous detector that periodically samples input data, even when there is no significant input signal. Furthermore, event-based detectors allow for high data compression ratios.

[0003] For example, event cameras differ from normal frame-based cameras in that, instead of each pixel integrating a charge and recording a light intensity, event pixels generate binary events asynchronously when the light intensity changes. This allows for significant compression of information about movement in a scene. The same principle can also be applied to other sensing modalities—for example, event-based audio.

[0004] Although convolutional neural networks (CNNs) and, more recently, vision transformers are excellent means of solving frame-based computer vision tasks, they are not inherently compatible with event data. In order to process event data using such models, events are often packed into so-called compact frames by, for example, counting the number of events generated per pixel in a given time window. However, this fails to take advantage of the sparse nature and low latency of event data. Furthermore, while these approaches allow the reuse of existing CNN architectures and their optimized hardware implementations, the precise spatiotemporal details captured by event cameras are, in practice, ignored.This can lead to performance degradation, particularly for applications such as optical flow prediction. Furthermore, compact frame CNNs do not readily take advantage of inherent scarcity. event data to reduce computational constraints.

[0005] It has been proposed to use graph neural networks to process asynchronous event data. One such example is the publication by Schaefer, Simon, Daniel Gehrig, and Davide Scaramuzza entitled "AEGNN: Asynchronous Event-based Graph Neural Networks." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022. However, a technical difficulty with this solution is that graph generation is relatively slow, leading to relatively high latency.

[0006] There is a need in the state of the art for an improved device and method that can process event data in real time or near real time to enable processing closely linked to the event detector. Summary of the invention

[0007] According to one embodiment, a method is provided for generating, by a data processing device, a prediction based on event data captured by an event-based detector, the method comprising: the reception, by the data processing device, of a new event from the event-based detector, the new event comprising a timestamp and an event data value having one or more dimensions; the determination, by the data processing device in a list of past events, of those past events which fall within a time distance from the timestamp and within a data value distance from the event data value;the construction, by the data processing device, of a subgraph of events comprising a first node representing the new event, additional nodes representing each of the determined past events, and one-way edges connecting each additional node to the first node; the calculation, by the data processing device, of node embeddings for the first node on the basis of edges consisting solely of said one-way edges connecting each additional node to the first node; and the generation of the prediction by applying a function to the node embeddings.

[0008] According to one embodiment, the method further comprises the control of one or more actuators on the basis of the prediction.

[0009] According to one embodiment, the determination of past events includes implementing a search for the K nearest neighbor events in a volume of past searches having a radius of event data value in one or more dimensions of event data value, and a time radius in the time domain.

[0010] According to one embodiment, the event-based detector is a camera of event, and the prediction is an optical flux prediction.

[0011] According to one embodiment, the data processing device is configured to implement real-time processing of the new event, the data processing device being configured for example to generate the prediction during a processing time of less than 100 microseconds, and preferably less than 5 microseconds, after a time of the new event.

[0012] According to one embodiment, the generation of the prediction includes passing the subgraph of events through a graphical neural network.

[0013] According to one embodiment, the graph neural network comprises a plurality of layers, and the passage of the subgraph of events into the graph neural network comprises: the calculation of an embedding vector for the first node and for the additional nodes of the subgraph using the embeddings of all events connected by said one-way edges, the embedding vector comprising a subvector for each of said layers; and the application of a transformation to the embedding vector of each node, or to the input event data in the case of a first layer of the graph neural network, in order to generate a resulting embedding vector.

[0014] According to a further aspect, a data processing device is provided for generating a prediction based on event data captured by an event-based detector, the data processing device being configured to: receive a new event from the event-based detector, the new event comprising a timestamp and an event data value having one or more dimensions; determine, in a list of past events, those past events that fall within a time distance from the timestamp and within a data value distance from the event data value; construct an event subgraph comprising a first node representing the new event, additional nodes representing each of the determined past events, and one-way edges connecting each additional node to the first node;calculate node embeddings for the first node based on edges consisting solely of said unidirectional edges connecting each additional node to the first node; and generate the prediction by applying a function to the node embeddings.

[0015] According to one embodiment, the data processing device is further configured to control one or more actuators on the basis of the prediction.

[0016] According to one embodiment, the data processing device is configured to determine said past events by performing a search for the K nearest neighboring events in a volume of past searches having a radius of event data value in one or more dimensions of the event data value, and a time radius in the time domain.

[0017] According to one embodiment, the event-based detector is an event camera, and the prediction is an optical flow prediction.

[0018] According to one embodiment, the data processing device is configured to perform real-time processing of the new event, the data processing device being configured for example to generate the prediction in a processing time of less than 100 microseconds, and preferably less than five microseconds, after an instant of the new event.

[0019] According to one embodiment, the data processing device is configured to generate the prediction by passing the subgraph of events through a graphical neural network.

[0020] According to one embodiment, the graph neural network comprises a plurality of layers, and the passage of the subgraph of events into the graph neural network comprises: the computation, by the graph neural network, of an embedding vector for the first node and for the additional nodes of the subgraph using the embeddings of all the events connected by said one-way edges, the embedding vector comprising a subvector for each of said layers; and the application, by the graph neural network, of a transformation to the embedding vector of each node, or to the input event data in the case of a first layer of the graph neural network, in order to generate a resulting embedding vector.

[0021] According to yet another embodiment, a control system is provided comprising: the previous data processing device; an event-based detector connected to the data processing device and configured to generate the new event and to transmit the new event to the data processing device; and one or more actuators. Brief description of the drawings

[0022] These features and advantages, as well as others, will be described in detail in the following description of particular embodiments, given by way of non-limiting example, in relation to the accompanying figures, among which:

[0023] Fig. 1 schematically represents an event-based processing and detection system according to an example embodiment of the present description;

[0024] [Fig.2] schematically represents an event processing device of [Fig.1] in more detail according to an example of an embodiment of the present description;

[0025] [Fig.3] is a graph representing an example of the event cloud generated from an event camera recording;

[0026] [Fig.4] is a graph representing an example of creating an event graph based on k jumps between indirectly connected events;

[0027] [Fig.5] is a flowchart representing an example of steps in a process for determining an optical flux prediction;

[0028] [Fig.6] is a graph representing an example of creating an event graph according to an example embodiment of the present description;

[0029] [Fig.7] is a flowchart representing an example of steps in a process for determining an optical flux prediction according to an embodiment of the present description;

[0030] [Fig.8] represents the generation of an embedding vector in each layer of an event graph according to an example embodiment of the present description;

[0031] Figure 9 represents the concatenation of embedding vectors and their application to a neural network; and

[0032] [Fig. 10] represents a neural network head for making a prediction according to an example of an embodiment of the present description. Description of the implementation methods

[0033] The same elements have been designated by the same reference numerals in the different figures. In particular, the structural and / or functional elements common to the different embodiments may have the same reference numerals and may have identical structural, dimensional and material properties.

[0034] For the sake of clarity, only the steps and elements useful for understanding the described embodiments have been shown and are detailed. In particular, event-based detection devices and methods are known in the prior art and have not been described in detail. Furthermore, the training of a graphical neural network and the use of a graphical neural network to generate event-based detection or prediction are well known to those skilled in the art and will not be described in detail.

[0035] Unless otherwise specified, when referring to two elements connected together, this means directly connected without intermediate elements other than conductors, and when referring to two elements connected (in English "coupled") together, this means that these two elements can be connected or linked through one or more other elements.

[0036] In the following description, when referring to absolute positional qualifiers, such as "front", "back", "top", "bottom", "left", "right", etc., or relative positional qualifiers, such as "above", "below", "superior", "inferior", etc., or to orientational qualifiers, such as "horizontal", "vertical", etc., it is made Reference to the orientation of the figures unless otherwise specified.

[0037] Unless otherwise specified, the expressions "approximately", "roughly", and "in the order of" mean within 10%, preferably within 5%.

[0038] Figure 1 schematically represents an event-based detection and processing system 100 according to an example embodiment of the present description. The system 100 comprises an event-based sensor 102 configured to generate temporal event data, and an event processing device 104 configured to receive the temporal event data and to process it in order to generate an event-based prediction.

[0039] For example, the event-based detector 102 is an event-based camera, or event camera, such as a visible light or infrared event camera. In one embodiment, the event-based detector could be an event-based audio detector, such as an event-based microphone. Other types of event-based detectors would also be possible.

[0040] Temporal event data includes, for example, asynchronous events, each event comprising, for example, a timestamp and an event data value having one or more dimensions. For example, in the case where the event-based detector 102 is a camera, the event data value could be the x and y address of the pixel that captured the event, and the event is, for example, generated when a change in light level exceeds a threshold relative to the light level at the time the previous event was generated at the pixel. In addition, a detected light level of the pixel could provide an additional dimension of the event data value, for example, the polarity of the change or the absolute light level at the time the event was generated.In the case where the event-based detector 102 is a microphone, events are generated, for example, due to relative changes in the analog signal generated by the microphone, and the characteristics of each event may also include a timestamp, the absolute value of the signal, and / or, if the signal is filtered into multiple frequency bands, the frequency band range of the event.

[0041] The event-based processing device 104 is configured, for example, to process time-event data in real time or near real time, for example with a processing delay of less than one hundred microseconds, and preferably less than five microseconds.

[0042] In the case where the temporal data are image data, the event-based prediction provided at the output of the event processing device 104 is by For example, optical flow prediction per generated event, object classification for the entire event stream, or object detection per event. In the case where the time-domain data is audio data, the event-based prediction provided as output by the event processing device 104 is, for example, keyword identification per group of events or sound class recognition.

[0043] Fig. 2 schematically represents the event processing device 104 of Fig. 1 in more detail according to an example of an embodiment of the present description.

[0044] The device 104 includes, for example, a processing device (P) 202 comprising one or more processors, and one or more memory devices, such as volatile memory (RAM) 204 and / or non-volatile memory (FLASH) 206. In some embodiments, the volatile memory 204 is RAM, and / or the non-volatile memory 206 is Flash memory, although other types of volatile and non-volatile memory may be present.

[0045] The processing device 202 and the memories 204, 206 are for example connected by a bus 208. The non-volatile memory 206 stores for example a software code, which is executed by the processing device 202.

[0046] The event processing device 104 further includes, for example, a sensor interface 210 connected to the bus 208, the sensor interface 210 being configured, for example, to communicate with the event-based detector 102 on a wired or wireless communications interface.

[0047] In some embodiments, the event processing device 104 further includes a graph neural network 212, which is for example also connected to the bus 208, and is configured to generate event-based prediction at an output of the event processing device 104. For example, although not shown in [Fig.2], the event-based prediction is provided by the graph neural network to an output interface of the device 104 connected to the bus 208.

[0048] The graphics neural network 212 is, for example, a hardware circuit configured to implement a method for generating a prediction based on event data captured by the event-based detector 102 described in more detail herein. According to alternative embodiments, the graphics neural network is implemented in software, and the method for generating a prediction based on event data captured by the event-based detector 102 is implemented by software stored in memory 204 or 206 and executed by the processing device 202. According to one alternative embodiment, the processor reads data from the event-based detector, processes this data, and then provides this data to the network neural graph in the appropriate format to perform event-based prediction.

[0049] In some embodiments, the event-based prediction generated by the graphic neural network 212 is used to control one or more actuator(s) 214 connected to the device 104. The one or more actuator(s) 214 include, for example, a robotic system, such as a robotic arm trained to pull weeds or to pick rotten fruit from a tree, automatic steering or braking systems in a vehicle, or an electronic actuator, which is, for example, configured to control the operation of one or more circuits, such as waking a circuit from a sleep mode, bringing a circuit into sleep mode, bringing a circuit to generate text output, performing a data encoding or decoding operation, etc.According to another example, device 104 is an eye-tracking unit that determines the area in a scene that a user is looking at and controls the actuators based on the determined area, for example in order to change the focus of the stereogram.

[0050] Figure 3 is a graph representing an example of the event cloud generated by an event camera recording. In particular, the graph in Figure 3 is a 3D graph representing, on an x-axis, a pixel coordinate (X PIXEL) in an x-direction of a pixel matrix of the camera, on a y-axis, a pixel coordinate (Y PIXEL) in a y-direction of the pixel matrix, and, on a z-axis, the time in seconds (TIME(s)). The data spiral shown in Figure 3 results from a circular movement of a person's hand in the image.

[0051] By analyzing the 3D data structure of the data events of [Fig.3], or similarly by analyzing a 2D, 4D or higher dimension data structure of event data, it is possible to perform prediction operations, such as classification to determine what type of object created the events, or optical flow prediction, for example to determine the velocity vector of the object, or detection, for example, of whether a certain object is present in the scene.

[0052] Figure 4 is a graph representing an example of an event graph in which the number of k jumps between indirectly connected events is indicated. The creation of the event graph involves, in particular, the generation of a tree structure, followed by a nearest neighbor search in the tree which allows edges between events (i.e., vertices) to be defined. For example, the creation of a KD tree structure is described in more detail in the publication by Zhou, Kun et al. entitled "Real-time KD-Tree Construction on Graphics Hardware", ACM Transactions on Graphics (TOG) 27.5 (2008): 1-11. A set The number of events is recorded, and a tree structure is created. This tree structure is used to search for a set of nearest neighbors for each event. Edges (i.e., nodes) are formed between these events based on a distance measure in pixel dimensions (e.g., x, y) and the time dimension (t). For example, a directed edge is connected between the M nearest neighbors within a fixed search radius.

[0053] This method is illustrated in Figure 4, which shows time (TIME) on the x-axis and an x,y (XY POSITION) of pixel data on the y-axis. A new event O for a given pixel is detected by an event-based detector at a time t. Other events (EVENTS) are also represented on this graph. A search volume S(rxv, rt) is defined as being within a temporal distance ft from the time of the new event, and within a spatial distance r*y from the new event, as represented by an ellipse S. Half of this search volume S is prior to the time of the new event and the other half of the search volume S is subsequent to the arrival time of the new event L. Thus, as shown in Figure 4, the search volume includes events within a radius of the new event C.Among the events that enter the search volume, the K nearest neighbors are connected to the new event en by edges. These edges can be inherently bidirectional or unidirectional, although often two unidirectional edges will form between pairs of events with such a Euclidean-based search paradigm. Furthermore, and importantly, since many events are generated, the edge configurations of existing events in the tree are also updated when new events C become one of the nearest neighbors of existing events. As will be explained in more detail below in relation to Figure 5, the event graph is then updated with regard to the new event and the new edges.In particular, the effect of the new event ei on existing node embeddings in the event graph is evaluated for all L layers of the event graph neural network, and possibly many times. Depending on the node embedding update method, this can be very computationally expensive and involve significant latency. To avoid updating the node embeddings of the entire active event graph every time a new event is generated, a recursive k-hop graph search function hopO can, for example, be applied to find the sparse subset of impacted nodes in each of the L layers. Only this subset of nodes will be updated upon the arrival of each new event, which can slightly alleviate the computational load. As shown in Figure 4, the direct connections with . The new events ei are connected by a single jump, corresponding to k=0, and an integer number K of additional jumps is determined by means of intermediate connected nodes. In the example in Figure 4, K=4 between the central event ei and the event in the upper left corner of the diagram.

[0054] Fig. 5 is a flowchart representing an example of steps in a process for determining an optical flux prediction according to the approach represented by the graph in Fig. 4.

[0055] In a step 501, a new event e' is received.

[0056] In a step 502, a subset of past events eJ is determined in the distance r*y and rt of the event ei. In other words, past events that fall within the search volume S of [Fig.4] are identified.

[0057] In a step 503, for each past event ef, a search for the K nearest neighbors is performed, and edges are created. In particular, edges e> are created from each past event eJ from the K nearest neighbors to the new event G. In addition, edges e> eJ are created from the new event eî to each of the K nearest neighbors in the past, and the node embeddings of all nodes impacted over an integer number of K hops from an event ei must be updated. The event eJ will subsequently be likely to be modified multiple times when several new events are generated within a time interval equal to the temporal search radius, forming direct connections K=0 with future events, multiplied by the number of layers (through modifications across nodes connected by a certain number of K hops).

[0058] In a step 504, the effect of a new event ei on existing event graph node embeddings for all L layers of the event graph neural network is evaluated. This involves the recursive k-hop graph search function hopO, which finds the sparse subset of impacted nodes in each layer 1 of the event graph, as shown in [Fig. 4].

[0059] In a step 505, the embeddings of affected nodes in each layer l are updated using a graph convolution function. The graph convolution function, ¢0, can be, for example, the spectral graph convolution proposed in the publication "SEMI-SUPERVISED CLASSIFICATION WITH GRAPH CONVOLUTIONAL NETWORKS" Kipf 2017, Z{ = f(A' ■ • ^), where A' — ■ A ■ IJ112, A is the adjacency matrix that stores the 0 or 1 in each matrix cell indicating whether nodes are connected and D is the degree matrix.

[0060] In a step 506, after a delay equal to the radius rt of the search window multiplied by the number of layers of the graph neural network, time during which additional new events that impact event e> can to be received and processed, a function is applied to the node embeddings to generate an output prediction V, for example an optical flux prediction, in relation to the new event ei.

[0061] The process in [Fig. 5] can, for example, be represented by the following algorithm.

[0062] [Math.1] Algorithm 1 Sparse fully-spherical update Tnput: evt = {.¾ y*. G = {EV, £}, rt, rxy Output: Vj M = EV n v.. rxy, rt) for evm GM do £m = knn(evt^ end for G = ir{G,£rn,evi) for / in range(L) do for ev.j G hop(M, G, 1) do zjj = G, l) end for end for if t > ti + (rt x L) then 14 = a(Z;) end if

[0063] where evi is the new event, xi are data values ​​of the event data value, h is the timestamp of the event, Pi is the event polarity, G is the event graph, EV is a subset equal to [x, y, t} and R are the edges of the event graph connecting the EV events, B() is a function used to extract the subset from the previous step 502, knnQ is the search for the K nearest neighbors, ttC) is a function which takes as input the state of the previous graph, the new event, and new and recalculated edges E, and provides as output an updated event graph, hopC} is the recursive k-hop graph search function, and Gi are the node embeddings in a given layer 1.

[0064] A major drawback of the method in Figure 5 is that it results in an excessive number of calculations and a relatively large latency between the time the new event occurs and the output of the prediction. This is due in part to the complexity of the processing, and also to the graph construction latency, including the G*L delay, which is typically several hundred milliseconds.

[0065] Figure 6 is a graph representing an example of event graph creation according to an example embodiment of this description. The graph in Figure 6 is similar to that in Figure 4 except that the search volume S is replaced by a search volume S', and the edges between past events (EVENTS) and the new event ei are strictly unidirectional, as will now be explained in more detail.

[0066] According to the embodiment shown in [Fig. 6], the graph construction latency is reduced by limiting the creation of edges in event graphs so that directed unidirectional edges are formed only from past events to a newly arrived event within a hemispherical or hemi-elliptical search radius S' around the newly arrived event. In other words, when a new event arrives, events with which it will connect and from which it will obtain data are searched only in the past neighborhood. Edges to past events, which were created according to the previous method, are no longer created, and consequently the edges are unidirectional in time, from the past to the future, as shown in [Fig. 6] by arrows pointing forward in time only.This allows for the calculation of a prediction for an event immediately after it is generated. Indeed, it can be guaranteed that neither the node embeddings of this event, nor its boundaries, will change in the future. This greatly reduces latency and the amount of computation.

[0067] In the case of a graph neural network, these edges are used to immediately compute the node embeddings of only the newly generated event for each layer of the network. Past events no longer modify their embedding once computed. This differs from previous approaches in which a large number of past events must be recalculated upon the arrival of each event. Only embeddings from past events within a certain time and space search radius to which it is connected are used in this method.

[0068] According to the approach in [Fig. 6], edges—that is, the identities of two events linked by the edge, as well as all edge properties such as a time difference—are not explicitly stored in memory because information about these edges is implicitly represented by the embeddings of previously generated event nodes. Thus, this approach saves memory storage, such as memory 204 and / or 206 in [Fig. 2], which can be considerably smaller. Furthermore, past events that fall outside the time search radius are not stored. This differs from previous approaches in which events within a time window equal to the search radius multiplied by the number of layers in the neural network are stored, and all edges connecting these events are also stored.

[0069] The preceding factors allow for implementation with a significantly reduced memory footprint compared to other approaches. More specifically, a Real-time implementation of the embodiments described here incorporates, for example, a feature in which the memory of the system in which the algorithm is incorporated is continuously cleaned so that events older than a certain time are removed.

[0070] In certain embodiments, in order to maintain a number of detected events that is relatively similar to the approach in Figure 4, the radius rt is, for example, increased compared to a typical value used according to this approach. Typically, this is achieved by increasing, for example substantially by doubling, the radius}t so that the search volumes are equivalent. For example, the radius rt in the case of Figure 4 is, for example, between 10 and 20 ms, and the radius rt in the case of [Fig. 6] is, for example, between 20 and 40 ms.

[0071] Figure 7 is a flowchart of an example of the steps in a method for determining an optical flux prediction according to an embodiment of the present description. This method could also be used to determine other types of predictions, such as classification or object detection. In the case of classification, where an event-by-event output may not be required, the node embedding of all events generated within a certain time window can be aggregated, for example, using maximum or average pooling, and it is this aggregated node embedding that is, for example, periodically fed to the output prediction function.

[0072] The process of [Fig.7] is for example implemented by the event processing device 104 of [Fig.1].

[0073] In a step 701, a new event ei is received by the event processing device 104, for example from the event-based detector 102.

[0074] In a step 702, a search for the K nearest neighboring events is performed, for example, in a past search volume of radius λ in one or more dimensions of the event data value, and of radius C in the time domain. For example, this is achieved by performing a distance measurement between the new event and past events, the distance calculation being, for example, based on the L2 norm. It would also be possible to use other distance metrics such as the L1 norm or simply the maximum distance function, corresponding to the use of the maximum of all dimensions (such as the maximum of the set of three dimensions in the case where there are three).

[0075] In a step 703, a subgraph is created containing the new event ei and, for each neighbor ei among the K nearest neighbors, a one-way edge

[0076] In a step 704, L node embeddings are computed for the event e> using a graph convolution function ÿ'(). This function 0'0 is by identical example to function ¢0, except that the input of this function is a past-only graph similar to that shown in [Fig.6], and thus the edges which are used for the calculation of the L node embeddings consist only of strictly unidirectional edges (past to present).

[0077] In a step 705, a cr'O function is, for example, applied to the embeddings of nodes Z; to generate an optical flux prediction V, for the new event and. For example, the cr'O function is applied by passing the subgraph through a neural network such as the one described in relation to [Fig. 6].

[0078] The process of [Fig.7] can for example be represented by the following algorithm.

[0079] [Math.2] Algorithm 2 Hemi-spherical update Input: evi = {^,^,0,^),27^,7^,7^ Output: Vi Ki- Si — knn^ EV. rt, rxy) G4 = {A,.£,} for l in range(L) do end for

[0080] where is the new event, \ are data values ​​of the event data value, t; is the timestamp of the event, Pi is the event polarity, K, are the nearest neighbor events connected by one-way edges £., Gj is the subgraph, and knrQ is the search for the K nearest neighbors.

[0081] Figure 8 illustrates the generation of an embedding vector in each layer of an event graph according to an example embodiment of the present description. In particular, a graph neural network is used to compute the embeddings of nodes. Figure 8 shows an example in which the graph neural event network, applied to the event graph G^, comprises five layers 11 to 15. In each layer, an embedding vector is computed for an event using the embeddings of connected events. For example, Figure 8 shows an example of an embedding matrix for an event j Z / associated with one of the connected events eJ of the new event ei, the embedding matrix Zj comprising subvectors Z7 j to Z for each layer 1 to 5.The embedding vector ^jA of the first layer is for example made up of the corresponding event, including for example the time information in the form of a timestamp, and the value of the event data having one or more . dimensions. In the example in Figure 8, each embedding vector -Z;,i comprises three values, which might, for example, in the first layer correspond respectively to the timestamp and pixel coordinates in X and Y, and in subsequent layers correspond to an abstract vector representation of the event. To obtain the embedding in each of these subsequent layers, a transformation is applied to the embedding vector of each node, or to the input event data in the case of the first layer, to generate a resulting embedding vector. This transformation also takes into account the embeddings of other nodes to which the node is connected by an edge. For example, the transformation could be a sum or average of the embedding vectors, or something more complex based on edge information.The layers of the event graph neural network are, for example, chained sequentially, as is the case for graph neural networks.

[0082] Figure 9 illustrates the concatenation of embedding vectors and their application to a neural network according to an example embodiment of the present description. In particular, the embedding vectors Za of each layer of the neural network associated with the new event ei are, for example, processed by passing them through a neural network to generate a prediction associated with the new event e. In the example in Figure 9, this involves concatenating the embedding vectors to form a single vector and processing this vector by a multilayer perceptron. For example, in a first layer L1 of the multilayer perceptron neural network, instance normalization can be applied to produce a first vector representation vl for the event.To generate the prediction Vj, which is for example an optical flux prediction, this first vector representation vl is for example processed by one or more additional layers including an output layer (OUTPUT LAYER), which provides two output values ​​and Vy, which correspond for example to the x and y components of the event optical flux vector Vh.

[0083] Figure 10 schematically represents a neural network head 1100 for making a prediction according to an example embodiment of the present description. Figure 10 specifically represents an example in which there are five layers 11 to 15 in the graphical neural network, and the output state of each layer, for each node, is provided as input to the neural network head 1100. For layers 12 to 15, the input to each layer is the output of the preceding layer, while applying, for example, new weights and new message-passing parameters. For example, in each layer, the same shared weight is applied to all nodes. The weight is different for Each layer. The transmission of a message depends on the type of convolution. For many types of convolution, there are no parameters. For more complex types, such as a B-spline convolution, there is, for example, the number of basic functions / kernels to define as a parameter, which could change from one layer to another.

[0084] The graphic neural network 1000 includes, for example, in the example of [Fig. 10], a layer L of four neurons receiving a concatenation of the node embeddings of a given node from each of the layers 11 to 15 of the graphic neural network. For example, the neurons of layer L are configured to apply a linear activation rectification function. Instance normalization may be applied to the activations of this layer. The outputs of the neurons of layer L are, for example, provided to a single output neuron Nout, which is, for example, configured to generate a prediction corresponding, for example, to a probability between 0 and 1.

[0085] One advantage of the graph creation algorithm described here is that it allows for considerably lower latency, typically by about four orders of magnitude, and a smaller total computational load, for example by two orders of magnitude, compared to previous approaches. Furthermore, edges are no longer stored in memory, and events older than a time equal to the search radius are removed from memory.

[0086] Various embodiments and variations have been described. A person skilled in the art will understand that certain features of these various embodiments and variations could be combined, and other variations will become apparent to a person skilled in the art.

[0087] For example, in the case of a graph neural network, pooling operations can be used to grow the graph, thereby reducing the number of nodes and the amount of computation required. For example, a pooling operation is triggered periodically for given layers of the graph neural network using, for example, a given voxel grid dimension, i.e., a periodic grid in the spatial (xy) and temporal (t) dimensions of the input event cloud. Events in a single voxel, for example, have their features aggregated into a single super-event, and the connections between these super-events are based, for example, either on preserving connections between voxels that existed previously or on performing a new search, in the past only, and defining a new set of edges.

[0088] Furthermore, although embodiments have been essentially described in the context of camera data, i.e., two-dimensional event data values, x and y, the techniques described here could be applied to other Event-based detector types include event-based cochlea or silicone earpieces. In such cases, rather than encoding events in space and time, the silicone ear encodes events in frequency band time and energy—resulting in a 2D point plane instead of a 3D point cloud, as seen in event cameras. For example, such a silicone earpiece is described in more detail in the publication by Chan, Vincent, Shih-Chii Liu, and Andr van Schaik, "AER EAR: A matched Silicon cochlea pair with address event representation interface." IEEE Transactions on Circuits and Systems I: Regular Papers 54.1 (2007): 48–59. Furthermore, the embodiments described here could be applied to any application with a temporal or causal dimension.For example, they could be applied to data structures that can be explored gradually, with the possible limitation that no loops are created (transposition of the condition "edges only connect the past to the future", meaning that the graph obtained with unidirectional edges has no loops).

[0089] Furthermore, any step could be applied to the data before and / or after the proposed graph construction. For example, data filtering could be applied, resulting in an event at a given pixel of the same polarity emitted within a certain time window being eliminated before processing. Other options include normalizing input features and encoding time using a single or pair of periodic functions, such as a sawtooth wave, instead of absolute time, which could impose a certain memory load.

[0090] Moreover, the distance between nodes can be calculated with any distance metric - L2 norm, L1 norm, also known as Manhattan distance, Euclidean distance, etc.

[0091] Furthermore, the features contained in each node can, for example, be modified. Some of the possibilities that can be used are time, the x and y coordinates of the event, a normalized vector approximated to a local event plane (see the publication by Mitrokhin, Anton, et al. "Leaming visual motion segmentation using event surfaces." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020.), the polarity of the event, etc.

[0092] Node embedding normalization schemes can be used to normalize node characteristics in intermediate layers of the graph neural network. The use of instance normalization is particularly attractive because it is a technique that uses only node-level statistics rather than a batch norm or a layer norm, which require the pooling of statistics from multiple nodes or even multiple graphs.

[0093] The features stored in the created edges can also be modified. For example, a simple binary flag can be used to indicate a connection or differences in spatial and temporal coordinates between connected events, which has been determined to be particularly effective.

[0094] Search radii can be modified, either uniformly for all graphs or dynamically, for example, according to certain rules. For instance, for a high density of events over time, suggesting a fast-moving edge, in a certain region of the scene, the search radius could be reduced to preserve spatial relationships between events generated by an object regardless of its speed. Alternatively, the approximated normal vector of an event can be used to adjust the search radius—a larger vector in given dimensions can be used to decrease the search radius insofar as it also implies a fast-moving object. Conversely, for a small estimated normal vector, the search radius could be increased.

[0095] In some embodiments, the number of neighbors may be limited, may be fixed or dynamic, or may not be limited.

[0096] The processing performed is not necessarily a graph neural network, as discussed in the publication entitled "Graph Convolutional Neural Network (GCN), with the gconv and spline layers", available at https: / / pytorch-geometric.readthedocs.io / en / latest / modules / nn.html. For example, processing schemes that are not based on AI can be applied, including graph search algorithms.

[0097] Finally, the practical implementation of the embodiments and variants described is within the reach of a person skilled in the art, based on the functional indications given above.

Claims

Demands

1. Method of generating, by a data processing device (104), a prediction based on event data captured by an event-based detector (102), the method comprising: - the reception, by the data processing device (104), of a new event (e0) from the event-based detector (102), the new event comprising a timestamp and an event data value having one or more dimensions; - the determination, by the data processing device (104) in a list of past events, of those past events (ej) which fall within a time distance (r0) from the timestamp and within a data value distance (rxy) of event data value;- the construction, by the data processing device (104), of a subgraph of events comprising a first node representing the new event (e0), additional nodes representing each of the determined past events (fy), and one-way edges connecting each additional node to the first node; - the calculation, by the data processing device (104), of node embeddings (¾) for the first node on the basis of edges consisting solely of said one-way edges connecting each additional node to the first node; and - the generation of the prediction (V,) by applying a function (tf'O) to the node embeddings (¾).

2. A method according to claim 1, further comprising the control of one or more actuators (214) on the basis of the prediction (V;).

3. A method according to claim 1 or 2, wherein the determination of past events (ej) comprises implementing a search for K nearest neighbor events in a volume of past searches having an event data value radius (rxy) in one or more event data value dimensions, and a time radius (rf) in the time domain.

4. A method according to any one of claims 1 to 3, wherein the event-based detector (102) is an event camera, and wherein the prediction (V,) is an optical flux prediction.

5. A method according to any one of claims 1 to 4, wherein the data processing device (104) is configured to put into implements real-time processing of the new event (ei\ the data processing device is for example configured to generate the prediction (V,) for a processing time of less than 100 microseconds, and preferably less than 5 microseconds, after a time of the new event (er).

6. A method according to any one of claims 1 to 5, wherein the generation of the prediction (V,) comprises passing the subgraph of events through a graphical neural network (212).

7. A method according to claim 6, wherein the graph neural network (212) comprises a plurality of layers, and the passing of the subgraph of events through the graph neural network (212) comprises: - the calculation of an embedding vector (Zjj) for the first node and for the additional nodes of the subgraph using the embeddings of all events connected by said one-way edges, the embedding vector (Z / ) comprising a subvector (Zy i to Zy^) for each of said layers; and - the application of a transformation to the embedding vector (Z) of each node, or to the input event data in the case of a first layer of the graph neural network, in order to generate a resulting embedding vector (¾).

8. A data processing device for generating a prediction based on event data captured by an event-based detector (102), the data processing device being configured to: - receive a new event (e0) from the event-based detector (102), the new event comprising a timestamp and an event data value having one or more dimensions; - determine, in a list of past events, those past events (fy) that fall within a time distance t) with respect to the timestamp and within a data value distance (r <v) de la valeur de données d'événements ; - construire un sous-graphe d'événements comprenant un premier nœud représentant le nouvel événement (^ / ), des nœuds supplémentaires représentant chacun des événements passés déterminés (Vj), et des bords unidirectionnels reliant chaque nœud supplémentaire au premier nœud ;- calculate node embeddings (Zf) for the first node on the; edge base consisting only of said unidirectional edges connecting each additional node to the first node; and - generate the prediction (Ven apply a function (c'O) to the node incorporations (¾).

9. Data processing device according to claim 8, further configured to control one or more actuators (214) on the basis of the prediction (V,).

10. Data processing device according to claim 8 or 9, configured to determine said past events (fy) by performing a search of the K nearest neighbor events in a search volume having an event data value radius (rxr) in one or more event data value dimensions, and a time radius (r0 in the time domain.

11. Data processing device according to any one of claims 8 to 10, wherein the event-based detector (102) is an event camera, and wherein the prediction (^) is an optical flow prediction.

12. A data processing device according to any one of claims 8 to 11, wherein the data processing device (104) is configured to perform real-time processing of the new event (^), the data processing device being configured, for example, to generate the prediction (V,) in a processing time of less than 100 microseconds, and preferably less than five microseconds, after an instant of the new event

13. Data processing device according to any one of claims 8 to 12, configured to generate the prediction (V,) by passing the subgraph of events through a graphical neural network (212).

14. A data processing device according to claim 13, wherein the graphical neural network (212) comprises a plurality of layers, and the passage of the event subgraph through a graphical neural network (212) comprises: - the computation, by the graphical neural network (212), of an embedding vector for the first node and for the additional nodes of the subgraph using the embeddings of all events connected by said one-way edges, the embedding vector (Zy) comprising a subvector (Zyj to Zy^) for each of the said layers; and - the application, by the graphic neural network (212), of a transformation to the embedding vector (Zjj) of each node, or to the input event data in the case of a first layer of the graphic neural network, in order to generate a resulting embedding vector (¾).

15. Control system comprising: - the data processing device according to any one of claims 8 to 14; - an event-based detector (102) connected to the data processing device and configured to generate the new event (ei) and to transmit the new event to the data processing device; and - one or more actuators (214).

Images