An event camera based event-by-event spatio-temporal representation method

Abstract

The application discloses an event camera-based event-by-event space-time representation method. The method comprises the following steps: acquiring original event sequence data collected by an event camera; performing feature embedding on the original event sequence data in the space and time dimensions, and converting the obtained feature embedding into a fixed-dimension embedding tensor containing N tensor elements; inputting the fixed-dimension embedding tensor into a triple attention network, and calculating the feature correlation between the embedding tensor elements; constructing a multi-layer perception machine, taking the local space-time autocorrelation weight, the local space autocorrelation weight and the global space-time autocorrelation weight of each event as inputs, and calculating a complete space-time representation tensor of the event sequence. Compared with the time granularity loss caused by the common event image stacking or voxel aggregation, the application can retain all the space-time information and time granularity of the original event sequence to the greatest extent, which is of great significance for the practical application of the event camera in a high-speed scene.

Description

Technical Field

[0001] This invention relates to the field of event representation technology based on event cameras, and more particularly to an event-by-event spatiotemporal representation method based on event cameras. Background Technology

[0002] Event cameras are a novel type of bio-inspired vision sensor, widely used in various machine vision tasks due to their low latency, low power consumption, and high dynamic range imaging characteristics. When the brightness of a pixel in an event camera exceeds a preset brightness change threshold, an event is immediately triggered at that pixel location, thus enabling the event camera to asynchronously output event sequences.

[0003] Because unprocessed event sequences are discrete, sparse, and non-uniformly distributed, they cannot be directly input into deep neural networks for model training in downstream tasks. Therefore, appropriate event representation is necessary. Common event representation methods include converting event sequences into event images or voxel grids. Event images are formed by stacking events within a certain spatiotemporal range on the same two-dimensional plane, while voxel grids divide the event sequence into multiple voxels in the spatiotemporal dimension, with the state of each voxel encoding the spatiotemporal information of the events occurring within it. However, event representation methods such as event image stacking or voxel grid aggregation will lose some temporal information of the original event sequence, reducing the temporal granularity of the event representation. In many application scenarios that require capturing high-speed information using event cameras, the loss of temporal information will severely affect the performance of event cameras in downstream tasks. Summary of the Invention

[0004] The purpose of this invention is to provide an event-by-event spatiotemporal representation method based on event cameras to replace image stacking or voxel aggregation, which can preserve all the temporal information of the original event sequence to the greatest extent.

[0005] To achieve the above-mentioned objectives, the present invention employs the following technical solution:

[0006] An event-by-event spatiotemporal representation method based on an event camera, the method comprising:

[0007] Acquire raw event sequence data captured by the event camera;

[0008] The original event sequence data is embedded with features in spatial and temporal dimensions, and the resulting feature embeddings are transformed into fixed-dimensional embedding tensors containing multiple tensor elements.

[0009] The fixed-dimensional embedding tensor is input into a triple attention network to calculate the feature correlation between the embedding tensor elements, thereby representing the spatiotemporal correlation between the events corresponding to the embedding tensor elements.

[0010] A multilayer perceptron is constructed, and the local spatiotemporal autocorrelation weight, local spatial autocorrelation weight, and global spatiotemporal autocorrelation weight of each event are used as inputs to calculate the complete spatiotemporal representation tensor of the event sequence.

[0011] Furthermore, methods for acquiring raw event sequence data captured by the event camera include:

[0012] The event camera asynchronously captures the brightness changes of each pixel. Whenever the brightness change of a pixel exceeds a brightness threshold C, an event is triggered at that pixel location. The raw event sequence acquired by the event camera is... Where N represents that the event sequence ξ contains N discrete events, e i This represents the i-th event in the event sequence; each event e i ={x i ,y i ,t i ,p i}, x i It is event e i X coordinate, y i It is the Y coordinate, t i It is the timestamp of the event trigger, p i ∈{-1,1} represents the event polarity, where -1 represents a negative polarity event and 1 represents a positive polarity event.

[0013] Furthermore, methods for embedding spatial and temporal features into the original event sequence data include:

[0014] When embedding spatial features of N discrete events, only the X-coordinate, Y-coordinate, and polarity information of the events are considered. First, the N discrete events are stacked on the same two-dimensional plane to form an event image. Based on the positive and negative polarities of the stacked events, positive and negative feature channels are constructed for the event image. The positive and negative feature channels at each pixel position record the number of events of the corresponding polarity stacked at that pixel position. Then, sparse convolution is used to perform a convolution operation on the event image. The convolution operation is performed only if the pixel with stacked events is located at the center of the convolution kernel. Finally, the pixel feature obtained by convolution is the spatial feature tensor of all stacked events at that pixel position.

[0015] When embedding time features into N discrete events, only the trigger timestamps and event polarity information are considered. First, all data for positive polarity events are retained, and all data for negative polarity events are cleared to zero. A linear layer is constructed for the input of positive polarity events, taking the trigger timestamps and event polarity channels of positive polarity events as input, and a linear transformation is performed to extract high-dimensional time features of positive polarity. Then, all data for negative polarity events are retained, and all data for positive polarity events are cleared to zero. A linear layer is constructed for the input of negative polarity events, taking the trigger timestamps and event polarity channels of negative polarity events as input, and a linear transformation is performed to extract high-dimensional time features of negative polarity. Finally, the two calculated high-dimensional time feature tensors are added together to obtain the time feature tensor of all events.

[0016] Furthermore, methods for transforming the obtained feature embeddings into fixed-dimensional embedding tensors containing N tensor elements include:

[0017] A multilayer perceptron for feature fusion is constructed by concatenating the spatial feature tensor and the temporal feature tensor obtained from spatial feature embedding and temporal feature embedding, and using them as input to the multilayer perceptron for feature fusion. This process calculates a fixed-dimensional embedding tensor containing N tensor elements, with a one-to-one correspondence between the subscripts of the embedding tensor elements and the subscripts of the discrete events. The subscripts of the embedding tensor elements represent the position index of each element in the embedding tensor, and the subscripts of the discrete events represent the event index.

[0018] Furthermore, the triple attention network includes a local spatiotemporal attention module, a local spatial attention module, and a global spatiotemporal attention module; all three modules take N discrete events and a fixed-dimensional embedding tensor containing N tensor elements as input; wherein, the local spatiotemporal attention module is used to calculate the local spatiotemporal autocorrelation weight of each event in the event sequence; the local spatial attention module is used to calculate the local spatial autocorrelation weight of each event in the event sequence; and the global spatiotemporal attention module is used to calculate the global spatiotemporal autocorrelation weight of each event in the event sequence.

[0019] Furthermore, the implementation method of the local spatiotemporal attention module includes: firstly, using the nearest neighbor algorithm to find the k nearest neighbors of each event within the spatiotemporal range. l Other events, and the nearest k is obtained by indexing according to the one-to-one correspondence between the subscripts of the embedded tensor elements and the subscripts of the discrete events. l The embedding tensors of each other event; then, based on each event and its k nearest neighbors in its spatiotemporal domain. l The location of each event is encoded by its spatiotemporal distance to other events; finally, based on the location encoding information and the embedding tensor, the local spatiotemporal autocorrelation weight of each event is obtained using the self-attention calculation formula.

[0020] Furthermore, the implementation method of the local spatial attention module includes: firstly, stacking N discrete events in the same two-dimensional plane to form an event image, and using a local window to find the k of each event in the stacking plane. sc Other events, and obtain k by indexing according to the one-to-one correspondence between the subscripts of the embedded tensor elements and the subscripts of the discrete events. sc The embedding tensors of each of the other events; then, based on each event and its k in the stacking plane... sc The planar distances of each other event are used for position encoding; then the embedding tensor prior of each event is added to the feature channel of the event image and sparse convolution is performed again to obtain the spatial features of the new stacked events; finally, based on the position encoding information and the spatial features of the new stacked events, the local spatial autocorrelation weight of each event is obtained using the self-attention calculation formula.

[0021] Furthermore, the implementation method of the global spatiotemporal attention module includes: firstly, using the farthest point downsampling algorithm to find the range k of each event within the spatiotemporal range. g For each of the farthest events, the nearest neighbor algorithm is used to find the nearest neighbor in the spatiotemporal range. Other events; then first use a multilayer perceptron combined with The spatiotemporal features of each farthest event are calculated using the spatiotemporal coordinates of the other events and their indexes to obtain the embedding vectors. Then, based on each event and its spatiotemporal range k... g The location of each farthest event is encoded by its spatiotemporal distance. Finally, based on the location encoding information, the embedding tensor of each event, and the spatiotemporal features of each farthest event, the global spatiotemporal autocorrelation weight of each event is obtained using the self-attention calculation formula.

[0022] Furthermore, the index of the complete spatiotemporal representation tensor of the event sequence corresponds one-to-one with the event index of the event sequence. It records the spatiotemporal characteristics of each event and the spatiotemporal correlation between each event and other events, that is, it completely represents all the spatiotemporal characteristics of the entire event sequence.

[0023] This invention provides an event-by-event spatiotemporal representation method that represents discrete, sparse, and non-uniformly distributed event sequences as fixed-dimensional tensors that can be learned and trained by a deep neural network model. Each element of this tensor represents the spatiotemporal features of each corresponding event in the event sequence. Furthermore, this representation method uses a single event as the smallest processing unit, and the result preserves the complete spatiotemporal information of the original event sequence to the greatest extent possible. Therefore, compared to the temporal fine-grained loss caused by existing image stacking or voxel aggregation methods, this invention can preserve all the spatiotemporal information and temporal fine-grainedness of the original event sequence to the greatest extent possible. This is of great significance for the practical application of event cameras in high-speed scenes. Attached Figure Description

[0024] Figure 1 This is a flowchart of an event-by-event spatiotemporal representation method based on an event camera according to the present invention;

[0025] Figure 2 (a) A schematic diagram of spatial feature embedding and (b) a schematic diagram of temporal feature embedding provided for embodiments of the present invention;

[0026] Figure 3 (a) A schematic diagram of a triple attention network and (b) A schematic diagram of a self-attention calculation process are provided for embodiments of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] like Figure 1 As shown, this invention provides an event-by-event spatiotemporal representation method based on an event camera, the method comprising the following steps:

[0029] Acquire raw event sequence data captured by the event camera; embed spatial and temporal features into the raw event sequence data and transform them into fixed-dimensional embedding tensors; input the embedding tensors into a triple attention network and calculate the feature correlations between the elements of the embedding tensors; construct a multilayer perceptron and use local spatiotemporal correlation weights, local spatial correlation weights, and global spatiotemporal correlation weights as inputs to calculate the final spatiotemporal representation tensor of the event sequence.

[0030] The acquisition of raw event sequence data collected by the event camera specifically includes:

[0031] The event camera asynchronously captures the brightness changes of each pixel with a microsecond-level response time. Whenever the brightness change of a pixel exceeds a brightness threshold C, an event is triggered at that pixel's location. Therefore, the event camera acquires the raw event sequence. Where N represents that the event sequence ξ contains N discrete events, e i Let represent the i-th event in the event sequence. Each event e... i ={x i ,y i , t i ,p i}, x i It is event e i X coordinate, y i It is the Y coordinate, ti It is the timestamp of the event trigger, p i ∈{-1,1} represents the event polarity, where -1 represents a negative polarity event and 1 represents a positive polarity event.

[0032] like Figure 2 As shown, the process of embedding spatial and temporal features into the original event sequence data and transforming them into fixed-dimensional embedding tensors specifically includes:

[0033] When embedding spatial features into N discrete events, only the X coordinate of the event is considered. Y coordinate and event polarity information As attached Figure 2 As shown in (a), first, N events are... Event images I are formed by stacking events on the same two-dimensional plane, based on the positive and negative polarities of the stacked events. Construct positive and negative feature channels for the event image. The positive and negative feature channels at each pixel location record the number of events of the corresponding polarity stacked at that pixel location, respectively. That is, the feature of each pixel in event image I consists of the number of positive and negative polarity events stacked at that pixel location. Then, perform a sparse convolution operation SConv(·) on the event image, applying the convolution if and only if there are pixels with event stacking. The convolution operation is only performed when the pixel is located at the center of the kernel. The final pixel feature obtained from the convolution is the spatial feature of all stacked events at that pixel location.

[0034] When embedding time features into N discrete events, only the trigger timestamps of the events are considered. and event polarity information First, retain the data for all positive events and clear the data for all negative events. This is achieved by constructing a diagonal matrix. The index (i, i) of its diagonal element is related to the event e. i The subscript i corresponds only to event e. i polarity p i If the input is positive, the diagonal element at (i,i) of the diagonal matrix will be 1; otherwise, it will be 0. Construct a linear layer for positive polarity event inputs, with the weights and biases of the linear layer as follows: and C tp This indicates the number of output channels of the linear layer, and includes the event trigger timestamp. and event polarity Using two channels as input, a linear transformation is used to extract high-dimensional time features of positive polarity. Then, retain the data for all negative events and clear the data for all positive events. This is also achieved by constructing a diagonal matrix. The index (i, i) of its diagonal element is related to the event e. i The subscript i corresponds only to event e. i polarity p i The diagonal element at (i,i) of the diagonal matrix is ​​1 only if the input is negative; otherwise, it is 0. A linear layer is constructed for inputs with negative polarity events. The weights and biases of the linear layer are respectively... and Linear transformation extracts high-dimensional time features of negative polarity Finally, the two calculated high-dimensional time feature tensors and Add them together to obtain the time characteristics of all events.

[0035] Construct a first multilayer perceptron (MLP) for feature fusion, and compute the spatial feature tensor F obtained from spatial feature embedding and temporal feature embedding. xyp With time feature tensor F tp Perform concatenation using Concat(·), and use the concatenation result as input to the first multilayer perceptron to calculate a fixed-dimensional embedding tensor containing N tensor elements. Where C represents the number of feature channels in the embedding tensor, and MLP(·) is the first multilayer perceptron. It is particularly important to emphasize that the calculation process of spatial feature embedding and temporal feature embedding does not change the order of events in the event sequence, so the final calculated embedding tensor F = {f1, f2, ..., f...} N The element index (representing the position index of each element in the embedded tensor) and discrete events There is a one-to-one correspondence between the subscripts (representing event indexes), meaning that event e can be directly retrieved using its subscript. i Its corresponding embedded tensor element f i Cross-indexing.

[0036] like Figure 3 As shown, the step of inputting a fixed-dimensional embedding tensor into a triple attention network and calculating the feature correlations between the elements of the embedding tensor specifically includes:

[0037] The triple attention network consists of a local spatiotemporal attention module, a local spatial attention module, and a global spatiotemporal attention module. Each module takes N discrete events as input and a fixed-dimensional embedding tensor containing N tensor elements. Specifically, the local spatiotemporal attention module calculates the local spatiotemporal autocorrelation weight for each event in the event sequence; the local spatial attention module calculates the local spatial autocorrelation weight for each event in the event sequence; and the global spatiotemporal attention module calculates the global spatiotemporal autocorrelation weight for each event in the event sequence.

[0038] The local spatiotemporal attention module first uses the nearest neighbor algorithm KNN(·) to find the value of each event e. i The k nearest neighbors in the spacetime domain l Other events Then based on each event e i Its nearest neighbor k in its spatiotemporal range l Other events ξ KNN spacetime distance Perform position encoding, where Furthermore, a second multilayer perceptron (MLP) is constructed to compute the position encoding tensor. Where C l This represents the number of channels in the position-encoded tensor. Finally, based on the position-encoded information and the embedding tensor, the self-attention calculation formula is used to obtain the e of each event. i Local spatiotemporal autocorrelation weights

[0039]

[0040] in, Represents each event e i Its nearest neighbor k in its spatiotemporal range l Local spatiotemporal correlation weights of other events, The calculation uses a third-level multilayer perceptron, Q,K,V = MLP q,k,v (F) represent the query, index, and content tensors corresponding to the embedded tensor, respectively, and q i ∈Q, k j ∈K, v j ∈V, all tensors can be retrieved based on the corresponding event index, MLP q,k,v (·) The calculations use the fourth, fifth, and sixth multilayer perceptrons.

[0041] The local spatial attention module first considers N discrete events Event images I are formed by stacking events on the same two-dimensional plane. A 3×3 local window is used to find the k-value of each event within the stacked plane. sc Other events Then based on each event e i k within its planar range sc Other events ξ win planar distance Perform position encoding, where Furthermore, a seventh multilayer perceptron (MLP) is constructed to compute the position encoding tensor. Where C sc This represents the number of channels in the position encoding tensor. Next, the embedding tensor F is added as a priori to the feature channels of the event image I and sparse convolution is performed again to obtain the query, index, and content tensors corresponding to the spatial embedding tensor. Finally, by combining the location encoding information and using the self-attention calculation formula, we obtain the e for each event. i Local spatial autocorrelation weights

[0042]

[0043] in, Represents each event e i k within its planar range sc Local spatial correlation weights of other events, The MLP(·) used in the calculation is the eighth multilayer perceptron.

[0044] The global attention module first uses the farthest point downsampling algorithm FPS(·) to find the value of each event e. i Within the spacetime range k g The farthest event Here, the downsampling rate of the farthest point downsampling algorithm is r∈(0,1], and then the nearest neighbor algorithm KNN(·) is used to find the nearest neighbor of each farthest event in the spatiotemporal range. Other events k g The farthest event Corresponding features Where f r According to event e r The corresponding embedded tensor element is retrieved by index. Then, based on each event e... i Within its spatiotemporal range k g The farthest event ξ g spacetime distance Spatiotemporal location encoding is performed, and the corresponding location encoding tensor is: Where C g This represents the number of channels in the position-encoded tensor. The MLP(·) used for computation is a ninth-layer perceptron. Next, the embedding tensors F and Fembedded are utilized. gCalculate the query tensor Q and the index and content tensors respectively. Specifically, Q = MLP q (F), K,V = MLP k,v (F g ), q i ∈Q, MLP q (·) corresponds to the tenth-level perceptron, MLP k,v (·) corresponds to the eleventh and twelfth multilayer perceptrons. Finally, based on the position encoding information, the self-attention calculation formula is used to obtain the value of each event e. i Global spatiotemporal autocorrelation weights

[0045]

[0046] in, Represents each event e i Within its spatiotemporal range k g The global spatiotemporal correlation weights of the farthest events, The MLP(·) used in the calculation is a thirteenth-layer perceptron.

[0047] The construction of the multilayer perceptron, using the local spatiotemporal autocorrelation weight, local spatial autocorrelation weight, and global spatiotemporal autocorrelation weight of each event as input, to calculate the complete spatiotemporal representation tensor of the event sequence specifically includes:

[0048] Construct a fourteenth-layer perceptron (MLP) for feature fusion, with local spatiotemporal autocorrelation weights for each event. Local spatial autocorrelation weights and global spatiotemporal autocorrelation weights As input, the spatiotemporal representation tensor of the event sequence is computed. Where C represents the number of feature channels of the spatiotemporal representation tensor of the event sequence. The subscripts of the spatiotemporal representation tensor F correspond one-to-one with the event subscripts of the event sequence. It records the spatiotemporal characteristics of each event and the spatiotemporal correlation between each event and other events, thus it can completely represent all the spatiotemporal characteristics of the entire event sequence.

[0049] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An event-by-event spatiotemporal representation method based on an event camera, characterized in that, The method includes: Acquire raw event sequence data captured by the event camera; The original event sequence data is subjected to spatial and temporal feature embedding, and the resulting feature embeddings are transformed into a fixed-dimensional embedding tensor containing multiple tensor elements. The methods for performing spatial and temporal feature embedding on the original event sequence data include: In the When embedding spatial features into discrete events, only the X-coordinate, Y-coordinate, and polarity information of the events are considered; firstly, Discrete events are stacked in the same two-dimensional plane to form an event image. Positive and negative feature channels are constructed for the event image according to the positive and negative polarities of the stacked events. The positive and negative feature channels at each pixel position record the number of events of the corresponding polarity stacked at that pixel position. Then, sparse convolution is used to perform a convolution operation on the event image. The convolution operation is performed only if the pixel with stacked events is located at the center of the convolution kernel. Finally, the pixel feature obtained by convolution is the spatial feature tensor of all stacked events at that pixel position. In the When embedding time features into discrete events, only the trigger timestamp and event polarity information are considered. First, all data for positive polarity events are retained, while all data for negative polarity events are cleared to zero. A linear layer is constructed for the input of positive polarity events, using the trigger timestamp and event polarity channels as input, and a linear transformation is performed to extract high-dimensional time features of positive polarity. Then, all data for negative polarity events are retained, while all data for positive polarity events are cleared to zero. A linear layer is constructed for the input of negative polarity events, using the trigger timestamp and event polarity channels as input, and a linear transformation is performed to extract high-dimensional time features of negative polarity. Finally, the two calculated high-dimensional time feature tensors are added together to obtain the time feature tensor for all events. The fixed-dimensional embedding tensor is input into a triple attention network to calculate the feature correlation between the elements of the embedding tensor, thereby representing the spatiotemporal correlation between the events corresponding to the embedding tensor elements. The triple attention network includes a local spatiotemporal attention module, a local spatial attention module, and a global spatiotemporal attention module. All three modules are based on... A discrete event and containing The fixed-dimensional embedding tensor of each tensor element is used as input; wherein, the local spatiotemporal attention module is used to calculate the local spatiotemporal autocorrelation weight of each event in the event sequence; the local spatial attention module is used to calculate the local spatial autocorrelation weight of each event in the event sequence; and the global spatiotemporal attention module is used to calculate the global spatiotemporal autocorrelation weight of each event in the event sequence. A multilayer perceptron is constructed, and the local spatiotemporal autocorrelation weight, local spatial autocorrelation weight, and global spatiotemporal autocorrelation weight of each event are used as inputs to calculate the complete spatiotemporal representation tensor of the event sequence.

2. The event-by-event spatiotemporal representation method based on an event camera according to claim 1, characterized in that, Methods for obtaining raw event sequence data captured by event cameras include: The event camera asynchronously captures the brightness changes of each pixel, as long as the brightness change value of a pixel exceeds a brightness threshold. This will trigger an event at that pixel location; the original event sequence captured by the event camera is... ,in Represents an event sequence Includes Discrete events Represents the first event in the event sequence One event; each event , It is an event The X coordinate, It is the Y coordinate. It is the timestamp of the event trigger. It represents the event polarity, where -1 indicates a negative polarity event and 1 indicates a positive polarity event.

3. The event-by-event spatiotemporal representation method based on an event camera according to claim 1, characterized in that, The obtained feature embeddings are transformed into those containing Methods for embedding tensors with fixed dimensions of tensor elements include: A multilayer perceptron for feature fusion is constructed by concatenating the spatial feature tensors obtained from spatial feature embedding and temporal feature embedding with the temporal feature tensor, and using this concatenation as the input to the multilayer perceptron for feature fusion. This process calculates the feature tensors containing... An embedded tensor with fixed dimensions for each element, and a one-to-one correspondence between the subscripts of the embedded tensor elements and the subscripts of the discrete events. The subscript of the embedded tensor element represents the position index of each element in the embedded tensor, and the subscript of the discrete event represents the event index.

4. The event-by-event spatiotemporal representation method based on an event camera according to claim 1, characterized in that, The implementation method of the local spatiotemporal attention module includes: firstly, using the nearest neighbor algorithm to find the nearest neighbor of each event in the spatiotemporal range. Other events are identified, and the nearest neighbor is obtained by indexing the embedded tensor element indices and discrete event indices according to the one-to-one correspondence between them. The embedding tensors of each of the other events are then used to determine the embedding tensors of each event and its nearest neighbor in its spatiotemporal domain. The location of each event is encoded by its spatiotemporal distance to other events; finally, based on the location encoding information and the embedding tensor, the local spatiotemporal autocorrelation weight of each event is obtained using the self-attention calculation formula.

5. The event-by-event spatiotemporal representation method based on an event camera according to claim 1, characterized in that, The implementation method of the local spatial attention module includes: firstly, Discrete events are stacked on the same two-dimensional plane to form an event image. Local windows are used to find the position of each event within the stacked plane. Other events are indexed based on the one-to-one correspondence between the embedded tensor element indices and the discrete event indices. The embedding tensors of each of the other events; then, based on each event and its in-plane... The planar distances of each other event are used for position encoding; then the embedding tensor prior of each event is added to the feature channel of the event image and sparse convolution is performed again to obtain the spatial features of the new stacked events; finally, based on the position encoding information and the spatial features of the new stacked events, the local spatial autocorrelation weight of each event is obtained using the self-attention calculation formula.

6. The event-by-event spatiotemporal representation method based on an event camera according to claim 1, characterized in that, The implementation method of the global spatiotemporal attention module includes: firstly, using the farthest point downsampling algorithm to find the spatiotemporal range of each event. For each of the farthest events, the nearest neighbor algorithm is used to find the nearest neighbor in the spatiotemporal range. Other events; then first use a multilayer perceptron combined with The spatiotemporal features of each farthest event are calculated using the spatiotemporal coordinates of the other events and their indexes to obtain the embedding vectors. Then, based on each event and its spatiotemporal range... The location of each farthest event is encoded by its spatiotemporal distance. Finally, based on the location encoding information, the embedding tensor of each event, and the spatiotemporal features of each farthest event, the global spatiotemporal autocorrelation weight of each event is obtained using the self-attention calculation formula.

7. The event-by-event spatiotemporal representation method based on an event camera according to claim 1, characterized in that, The index of the complete spatiotemporal representation tensor of the event sequence corresponds one-to-one with the event index of the event sequence. It records the spatiotemporal characteristics of each event and the spatiotemporal correlation of each event with other events, that is, it completely represents all the spatiotemporal characteristics of the entire event sequence.

Images