Frequency-aware event cloud network

Abstract

The invention discloses an event stream processing method and device, and belongs to the field of computer vision. In order to reduce the calculation amount and improve the precision, the event stream processing method comprises the following steps: sampling an original event stream output by an event camera to obtain a first event cloud, and executing one-dimensional convolution on the first event cloud to obtain a first feature; then, a farthest point sampling method and an event-based KNN are utilized, and the first event cloud and the first feature are combined to obtain grouped event features and a second event cloud; fourier transform, Hadamard product, inverse Fourier transform and nonlinear activation are sequentially carried out on the grouped event features to obtain spatial domain features, grouping features are aggregated in the time dimension to obtain aggregated features, and time domain features are obtained through similar operations. And then extracting the time domain feature to obtain a second feature, updating the first feature and the corresponding event cloud by using the second feature, and pooling the first feature after multi-stage updating to obtain a final output result. The method has the effects of light weight, low calculation amount and high precision.

Description

Technical Field

[0001] The present invention relates to a frequency-aware event cloud network, and in particular to an event stream processing method and device. Background Art

[0002] Event cameras asynchronously detect lighting changes and have the advantages of high dynamic range, low latency, and low power consumption. They perform well in capturing high-speed motion and fast scene dynamics. Their output is a raw event containing spatial coordinates, timestamps, and polarity.

[0003] The current event representation methods mainly include: event-based frame representation, voxel representation, and point cloud representation. Among them, the mainstream method converts the original event into a frame or voxel representation, which has good performance but is time-consuming (see Figure 5 ), the model is huge and the time information is sacrificed. Figure 1 It is a comparative schematic diagram of various different methods of representing events in this field.

[0004] Point cloud representation can retain temporal information and has a lighter network architecture, but it has limitations such as ignoring polarity, insufficient extraction of long-term event features, resulting in performance bottlenecks, and superlinear growth of computational load as the number of input points increases. Summary of the invention

[0005] In order to alleviate or partially alleviate the above technical problems, the solution of the present invention is as follows:

[0006] An event stream processing method includes: downsampling an original event stream output by an event camera to obtain a first event cloud;

[0007] Perform a one-dimensional convolution on the first event cloud to obtain a first feature;

[0008] According to the first event cloud and the first feature, the farthest point sampling method and the event-based KNN are used to obtain the grouped event features and the second event cloud;

[0009] Perform Fourier transform on the grouped event features, then perform Hadamard product with the spatial filter, perform inverse Fourier transform, and then obtain spatial features after non-linear activation;

[0010] For the spatial domain features, aggregate the features of each group in the time dimension to obtain aggregate features;

[0011] Perform Fourier transform on the aggregated features, then perform Hadamard product with the time domain filter, perform inverse Fourier transform, and then obtain the time domain features after nonlinear activation;

[0012] Extract the time domain features to obtain the second features;

[0013] The first feature is updated using the second feature, and the first event cloud is updated using the second event cloud. After several stages of updating, pooling is performed on the first feature to obtain a final output result.

[0014] Furthermore, the method of obtaining the grouped event features and the second event cloud based on the first event cloud and the first feature by using the farthest point sampling method and the event-based KNN specifically includes:

[0015] After multiplying the first event cloud by the scaling factor, the centroid coordinates are obtained according to a farthest point sampling method, wherein the farthest point sampling method is a downsampling method;

[0016] According to the downsampling result in the farthest point sampling method, downsample the first feature to obtain the centroid feature;

[0017] According to the centroid feature and the first feature, the event feature and the coordinates of the first grouped event are obtained by using the event-based KNN, wherein the event-based KNN searches for a neighborhood according to the distance of the event feature;

[0018] Concatenate the post-grouping event coordinates and the first post-grouping event features to obtain the second post-grouping event features;

[0019] The third post-grouping event feature obtained by normalizing the second post-grouping event feature is concatenated with the centroid feature to obtain the post-grouping event feature;

[0020] The coordinates of the events in each group in the grouped event coordinates are averaged to obtain the second event cloud.

[0021] Furthermore, the feature of each group is aggregated in the time dimension for the spatial domain feature to obtain the aggregated feature, which specifically includes:

[0022] After the spatial features pass through the multi-layer perceptron, they are activated by SoftMax to obtain the attention value of the group members, and then multiplied by the spatial features to obtain the aggregated features.

[0023] Further, the pooling includes average pooling and / or maximum pooling; and,

[0024] The final output result is obtained based on the average pooling and / or maximum pooling results.

[0025] Furthermore, the extracting the time domain features to obtain the second features specifically includes: extracting the time domain features through a residual module to obtain the second features.

[0026] Furthermore, the residual module includes one-dimensional convolution, batch normalization and non-linear activation operations.

[0027] Further, the time domain filter and the space domain filter are global filters and learnable filters; and,

[0028] The spatial domain filters in each stage are different from each other, and the temporal domain filters in each stage are also different from each other.

[0029] Furthermore, downsampling the original event stream output by the event camera to obtain the first event cloud specifically includes:

[0030] The event cloud obtained after downsampling the original event stream is normalized to obtain a first event cloud.

[0031] Furthermore, the third post-grouping event feature obtained after standardizing the second post-grouping event feature is specifically obtained by:

[0032] The third post-grouping event feature is obtained by subtracting the mean of the second post-grouping event feature from the second post-grouping event feature and then dividing it by the standard deviation between the second post-grouping event feature and the mean of the second post-grouping event feature.

[0033] In addition, on the other hand, the present invention discloses an event stream processing device, which is configured to execute any of the event stream processing methods described above.

[0034] The technical solution of the present invention has one or more of the following beneficial technical effects:

[0035] 1. Improve point cloud expression, including polar event clouds that are closest to the original event stream.

[0036] 2. Make full use of the event cloud containing 2S-1T-1P information through innovative event-based grouping and sampling modules.

[0037] 3. Use the method of extracting features in the frequency domain to adapt to long sequence event streams.

[0038] 4. The spatial frequency perception module significantly reduces MACs operations by replacing convolution with Hadamard products, reduces power consumption, substantially eliminates or suppresses the problem of a sharp increase in multiplication and accumulation (MACs) operations, and effectively extracts spatial and temporal domain features.

[0039] 5. The time-domain frequency-aware module is good at capturing the long-term dependencies of event clouds through global frequency filters rather than providing local time information.

[0040] 6. The network of the present invention has high accuracy.

[0041] 7. The present invention has a relatively low network scale and is lightweight.

[0042] In addition, other beneficial effects of the present invention will be mentioned in the specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a comparative schematic diagram of various different methods of representing events in this field;

[0044] Figure 2 It is a schematic diagram of the overall framework of the present invention;

[0045] Figure 3 It is a detailed block diagram of the technical solution of the present invention;

[0046] Figure 4 It is a schematic diagram of the dimensional changes of coordinates and features in the hierarchical structure of the present invention;

[0047] Figure 5 These are some performance comparison diagrams of the present invention and the prior art. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] In the description of the present invention, unless otherwise specified, “ / ” indicates that the objects associated with each other are in an “or” relationship, for example, A / B can represent A or B; “and / or” in the present invention is only a description of the association relationship between associated objects, indicating that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural.

[0050] In the description of the present invention, unless otherwise specified, "plurality" means two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0051] In addition, in order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, the words "first", "second", etc. are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not necessarily limit the difference.

[0052] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" and "for example" is intended to present related concepts in a specific way for easy understanding.

[0053] The present invention proposes an event stream processing method, which specifically relates to a frequency-aware event cloud network, and runs in various types of neural network accelerators, such as FPGA, GPU, dedicated AI chip or a combination thereof. In the neural network accelerator, the running frequency-aware event cloud network (referred to as network in the present invention) mainly includes modules such as embedding layer, hierarchical structure, prediction head, etc. These modules in the present invention can be implemented by software methods, or by programming hardware, or by a combination of software and hardware.

[0054] The original event stream generated by the event camera sensing the change in light intensity includes 2D spatial domain, 1D temporal domain, and 1D polarity (2P-1T-1P) information. The original event stream ε can be described as follows:

[0055] ε={e i =(x i ,y i , t i , p i )|i∈I}, I=[1, 2..., n];

[0056] where e i is the ith event in the original event stream, (x i ,y i ) is the spatial coordinate of the pixel that generates the event, t i is a timestamp, and I is a set of n natural numbers.

[0057] Unlike point clouds that ignore polarity information, event clouds are downsampled results of the original event stream and are the closest expression to the original event stream, which helps improve network performance. Event clouds can be described as:

[0058]

[0059] The set J is a sequential subset of the set I, and it is assumed that the length or number of elements of the set J is T, so the dimension of the event cloud is T×4 or [T, 4]. The events in the event cloud are arranged in chronological order. The event cloud obtained after downsampling the original event stream is normalized to obtain the first event cloud (which is denoted as ) to extract time domain features.

[0060] Figure 2 It is a schematic diagram of the overall framework of the present invention. In the present invention, the front end of the network is the embedding layer. The first event cloud After one-dimensional convolution (Conv1D), it becomes the first feature of high dimension Its dimension is T×D, where D is a natural number.

[0061] For example, in the present invention, the value of T can be 1024, 2048, 4096, or 8192. The value of D can be 16, 32, or 64.

[0062] Similar to the frame-based convolution operation, the point-based grouping and sampling module, as the basic operation mode for capturing local features, has a limited receptive field in the point cloud space. Since point clouds and event clouds are not equivalent, the present invention redesigns the event-based grouping and sampling module.

[0063] In the present invention, the event-based grouping and sampling module receives the first feature and First Event Cloud And output the event features after grouping and the grouped event coordinates And the second event cloud

[0064] In the present invention, the number of events in a group is assumed to be K, and K=24 by way of example.

[0065] Event characteristics after the spatial frequency perception module receives the packet And output spatial features Airspace characteristics As the input of the aggregation module, the aggregation module outputs the aggregated features

[0066] Aggregate Features As the input of the time domain frequency perception module, the output of the time domain frequency perception module is the time domain feature

[0067] Then, in the time domain features After processing by an exemplary residual module, the second feature is obtained

[0068] In the present invention, the event-based grouping and sampling module, the spatial frequency perception module, the aggregation module, the time frequency perception module and the exemplary residual module constitute a hierarchical structure. For example, in one embodiment, there are m=3 hierarchical structures, namely, the first hierarchical structure, the second hierarchical structure, ..., the mth hierarchical structure.

[0069] At the end of a hierarchy or at the end of a stage, the second feature is used Update the first feature And use Second Event Cloud Update First Event Cloud Then, move on to the next level of hierarchy.

[0070] In the next level, the first feature of the update is still updated through the event grouping and sampling module Updated First Event Cloud The same or similar operations are performed, and the same or similar operations are performed in the subsequent spatial frequency perception module, aggregation module, time domain frequency perception module and exemplary residual module until entering the next hierarchical structure again. In the last hierarchical structure, its output is used as the input of the prediction head.

[0071] In the present invention, both the spatial frequency perception module and the time domain frequency perception module include filters, and these filters are global filters and are learnable. For example, in the first hierarchical structure, the filters of the spatial frequency perception module and the time domain frequency perception module are respectively the first spatial filter V s1 , the first time domain filter V t1 In the second hierarchical structure, the filters of the spatial frequency perception module and the time domain frequency perception module are respectively the second spatial filter V s2 , the second time domain filter V t2 , and so on.

[0072] For example, the number of levels or stages of the hierarchical structure of the present invention is 3, and it can also be other stage numbers, such as 2, 4, 5, etc.

[0073] In addition, the exemplary residual module in the present invention may be replaced by other network modules that can extract features and increase dimensions, such as transformer, etc. The present invention is not limited to this and is not limited to this.

[0074] The spatial frequency perception module of the present invention is mainly used to reduce the amount of calculation. The traditional single convolution calculation complexity is The present invention uses frequency domain product to replace convolution, and the computational complexity becomes

[0075] In one example, the spatial frequency perception module of the present invention can reduce the amount of computation of the actually constructed network by about 20 times. More importantly, this does not significantly reduce the network performance.

[0076] In the present invention, preferably, with the help of a time domain frequency perception module, this can effectively improve or guarantee the final performance of the network.

[0077] Figure 3 The event-based grouping and sampling module mainly includes feature sampling module, D-FPS, EF-KNN and other modules.

[0078] The event-based grouping and sampling module receives the first feature and First Event Cloud After that, the first event cloud is scaled by the scaling factor α, and then the centroid coordinates are obtained using the farthest point sampling (FPS) method belonging to the downsampling method according to the scaled event cloud. And the centroid coordinates are obtained after downsampling the events.

[0079] For example, after downsampling, half of the events are retained. This FPS can be called Differentiation FPS (D-FPS), which can be described by the following formula:

[0080]

[0081] First Event Cloud The size is T×4. Due to the downsampling in D-FPS, the centroid coordinates The size of becomes T / 2×4. In other words, half of the events are retained or discarded, and the first feature is sampled in the feature sampling module according to the index of the retained event. The same downsampling is also performed to obtain the centroid feature Its dimension is T / 2×D.

[0082] Then, the centroid feature and the first feature As the input of the event-based KNN (EF-KNN) module, it finds the neighborhood based on the distance of event features instead of the traditional coordinate-based method. This step can be described as:

[0083]

[0084] in is the event coordinate after grouping, where is the event feature after the first grouping, and K is the number of events in each group.

[0085] Furthermore, the event coordinates after grouping can be The coordinates of the events in each group are averaged to obtain the second event cloud Its dimension is T / 2×4. This step can be performed in the mean module. This link can be described as:

[0086]

[0087] In the present invention, the second event cloud It is one of the outputs of the event-based sampling and grouping module.

[0088] Event characteristics after the first grouping With the grouped event coordinates Event features after splicing into the second group

[0089]

[0090] in, is the concatenation symbol. Then, the event features after the second grouping Standardize and centroid features In the above standardization process, the event features after the second grouping are involved. The step of averaging the event features in each group in . This step can be described as:

[0091]

[0092] in, represents the mean, Std(.) represents the calculated standard deviation, is the event feature after the third grouping. By combining the event feature after the third grouping with the centroid feature (Copy K copies) for splicing, and finally obtain the event features after grouping It serves as another output of the event-based sampling and grouping module.

[0093] As mentioned above, the grouped event features output by the event-based sampling and grouping module are As the input of the spatial frequency perception module.

[0094] The Spatial Frequency Aware (SFA) module includes Fourier transform (FFT) and inverse Fourier transform (iFFT), and the input post-grouping event features After FFT, in the first stage, V s1 The filter performs a Hadamard product operation (which is then multiplied by V in the second stage). s2 The filter performs Hadamard product operation, and so on), and the result of the operation is then passed through iFFT and then nonlinear activation to obtain spatial domain features.

[0095] exist Figure 3 In the example, the filter is represented by the imaginary number of a+bi. The parameters of the filters in different stages are different, and the filters are also global and learnable.

[0096] In the present invention, filters with the same performance that are mathematically similar to those obtained by back-propagation are identified in the frequency domain, the feasibility of which is based on the fact that convolution in the time domain is equivalent to the Hadamard product in the frequency domain.

[0097] In the present invention, except for the different input data to be processed, the spatial frequency perception module and the time domain frequency perception (FrequencyAware, TFA) module are the same or similar in architecture and can be uniformly expressed by the following formula:

[0098] X = FFT(x),

[0099]

[0100] Where x is the input of the SFA or TFA module, such as the grouped event features Aggregate Features x is also the result after iFFT; X is the result after FFT transformation, V is the filter corresponding to this stage or this level structure, ⊙ represents the Hadamard product, is the result after Hadamard product; y is the activation result after nonlinear activation function σ. In SFA, y corresponds to the spatial domain feature In TFA, y corresponds to the time domain feature

[0101] Fourier transform is well known in the art. Nevertheless, the present invention applies one-dimensional discrete Fourier transform (DFT) to one-dimensional signals such as event clouds, and can convert event cloud features into the frequency domain. The specific formula is as follows:

[0102]

[0103] Among them, j is the imaginary part, x represents the time domain signal, X represents the spectrum about different frequencies, and T is the length of the time domain signal x.

[0104] The inverse discrete Fourier transform (iDFT) can recover the time domain signal from the frequency domain. Specifically, it can be obtained through the following formula:

[0105]

[0106] From a mathematical point of view, That is, the spectrum X has conjugate symmetry. Therefore, in the frequency domain, only data of length 1+T / 2 is needed to restore the original signal.

[0107] Next, a Hadamard product is applied between the spectrum X and the filter V, where the dimension of the filter V matches the dimension of the spectrum X, obtaining In through By performing iFFT transformation, the original time domain signal x can be restored, and finally activated using a nonlinear activation function.

[0108] In the present invention, through the processing of SFA and TFA, FFT operations are performed on the features of the event cloud in the spatial and temporal dimensions to extract spatial and temporal features. The present invention can greatly reduce the amount of calculation by applying SFA, and in one example it has been proven that a 20-fold reduction in the amount of calculation can be achieved. In one example, compared to a frame-based solution, the present invention only requires 1 / 1000 of the MACs calculation amount.

[0109] In addition, the present invention can apply several times the number of input events of various previous point cloud network-based solutions, and long-term events include important global and local contextual relationships. The commonly used LSTM and attention mechanisms cannot adapt to capturing such relationships and will introduce excessive computational complexity, which has been proven in some experimental data. The TFA of the present invention plays an important role in improving network performance. For example, the verification results of the DHP19 dataset show that if TFA is removed, the network performance will be greatly reduced; using LSTM instead of TFA to extract time domain features, it is difficult to capture long-term dependencies between a large amount of time. Compared with PointNET++ and TTPOINT, which do not have explicit time domain feature extraction, the accuracy of the present invention is much higher than these two.

[0110] Spatial characteristics of SFA output It is processed in the aggregation module (AGG). The aggregation module includes a linear layer (specifically a multi-layer perceptron MLP) and a SoftMax activation, specifically:

[0111]

[0112] That is, the spatial characteristics After MLP, SoftMax activation is used to obtain the attention value A of the group members, and the attention value A of the group members is then combined with the spatial features Multiply to get aggregate features In the present invention, the aggregation module uses a soft attention mechanism in the neighborhood dimension to aggregate the features of one / each group in the time dimension.

[0113] Aggregate Features As the input of TFA, after processing, TFA outputs the time domain features TFA has been introduced in detail above and will not be described again here. In the present invention, the filter in TFA can cover various frequencies and can capture short-term and long-term interactions.

[0114] Time domain characteristics It is processed in the residual module (RES) and outputs the second feature The residual module is a module known in the art, and the residual module here can also be replaced by any other module that can extract features, such as dimensionality increase, and the present invention is not limited to this. The purpose of setting the residual module is to extract higher-dimensional features from the network.

[0115] As an example of the present invention, the residual module includes a one-dimensional convolution Conv1D and batch normalization (BatchNormal, BN). After one-dimensional convolution, batch normalization, and nonlinear activation σ, it undergoes one-dimensional convolution, batch normalization again, and then is combined with the time domain features. After addition, it is activated again through nonlinear activation σ and the second feature is obtained

[0116] In the present invention, the nonlinear activation σ may be a Gaussian Error Linear Unit (GELU) activation.

[0117] At the end of a hierarchy, through the second feature Update (i.e. replace) the first feature By Second Event Cloud Update (i.e. replace) the first event cloud Then move on to the next level of the structure.

[0118] Finally, for the first feature Pooling is performed, and then a final output result is obtained according to the pooling result. For example, the pooling includes average pooling and / or maximum pooling.

[0119] In one example, after multiple hierarchical structures or multiple stages of processing, finally in the prediction head, preferably by taking the mean or average pooling of the first feature and the first feature of the maximum value (max) or maximum pooling Perform concatenation to obtain the final features. Perform classification and / or regression operations based on the final features. For example, obtain the classification result Get 13 points of articulation Finally, the above results are output.

[0120] In summary, the present invention completes three tasks by processing the event cloud into a series of different modules: embedding layer, hierarchy and prediction head. In more detail, preferably, the hierarchy contains five different modules: event-based sampling and grouping (G&S) module captures local neighborhood relationships, SFA module extracts explicit spatial features and implicit temporal features, aggregation module groups and aggregates features, TFA module captures global explicit temporal features, and residual module is responsible for feature abstraction and high-level representation, and the final result is obtained by maximum pooling features or / and average pooling features.

[0121] In addition, according to the technical concept of the present invention, this article will also disclose a pseudo code for a human posture recognition embodiment to supplement or explain the technical details or logic of the present invention.

[0122]

[0123] In this example, G&S(.) represents an event-based grouping and sampling module, the input is the first event cloud of the event camera for sensing human motion, and the output is the classification result and the person's posture (specifically 13 joint points). Obviously, for different technical purposes, the output results may be different, such as target classification, behavior recognition, person's posture, etc., and the present invention is not limited to this and is not limited to this.

[0124] Figure 4 It is a visualization of the event cloud coordinates and feature dimensions in the hierarchical structure. It is not difficult to see that the closer to the end of the cycle, the fewer events there are and the larger the feature dimensions.

[0125] Figure 5 These are some performance comparison diagrams of the present invention and the prior art. Obviously, the present invention achieves better accuracy with a smaller amount of calculation. The MACs calculation amount represented by the horizontal axis is the result after taking the logarithm. In addition, the preprocessing calculation amount of the present invention is also the smallest or minimum level compared to other event expressions.

[0126] After verification on N-MNIST, N-CARS, CIFAR10-DVS, N-Caltech101, and ALS-DVS datasets, the accuracy of the proposed method in target recognition tasks has been improved by 26% compared to the state-of-the-art (SOTA) solution based on point cloud. The proposed method has achieved comparable performance to the SOTA solution based on voxel expression.

[0127] After verification on the DVSGesture, DailyDVS, and UCF101-DVS datasets, the performance of the present invention is improved by 16% compared with the best SOTA solution VMST-Net.

[0128] In the DVS128 dataset, we verified and compared the number of MACs operations and network parameters. Compared with the frame-based solution, the present invention only requires one thousandth of the MACs operations.

[0129] In the task of character pose estimation, the present invention utilizes the original unprocessed event cloud to achieve the SOTA results of point-based and voxel-based methods. At the same time, the present invention has the smallest network size, with only 0.12G MACs operations, which is 2% of Point Transformer.

[0130] In summary, the present invention can effectively extract spatiotemporal features while greatly suppressing the sharp increase of multiplication-accumulation operations (MACs).

[0131] In addition, the present invention also discloses an event stream processing device, which is configured to execute any of the event stream processing methods described above. The event stream processing device can be an FPGA, a GPU or an ASIC AI chip.

[0132] In order to better illustrate the present invention, numerous specific details are provided in the above specific embodiments. It should be understood by those skilled in the art that the present invention can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present invention.

[0133] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An event stream processing method, characterized in that: include: Downsampling a raw event stream output by the event camera to obtain a first event cloud; Perform a one-dimensional convolution on the first event cloud to obtain a first feature; According to the first event cloud and the first feature, the farthest point sampling method and the event-based KNN are used to obtain the grouped event features and the second event cloud; Perform Fourier transform on the grouped event features, then perform Hadamard product with the spatial filter, perform inverse Fourier transform, and then obtain spatial features after non-linear activation; For the spatial domain features, aggregate the features of each group in the time dimension to obtain aggregate features; Perform Fourier transform on the aggregated features, then perform Hadamard product with the time domain filter, perform inverse Fourier transform, and then obtain the time domain features after nonlinear activation; Extract the time domain features to obtain the second features; The first feature is updated using the second feature, and the first event cloud is updated using the second event cloud. After several stages of updating, pooling is performed on the first feature to obtain a final output result.

2. The event stream processing method according to claim 1, characterized in that: The method of obtaining the grouped event features and the second event cloud according to the first event cloud and the first feature by using the farthest point sampling method and the event-based KNN specifically includes: After multiplying the first event cloud by the scaling factor, the centroid coordinates are obtained according to a farthest point sampling method, wherein the farthest point sampling method is a downsampling method; According to the downsampling result in the farthest point sampling method, downsample the first feature to obtain the centroid feature; According to the centroid feature and the first feature, the event feature and the coordinates of the first grouped event are obtained by using the event-based KNN, wherein the event-based KNN searches for a neighborhood according to the distance of the event feature; Concatenate the post-grouping event coordinates and the first post-grouping event features to obtain the second post-grouping event features; The third post-grouping event feature obtained by normalizing the second post-grouping event feature is concatenated with the centroid feature to obtain the post-grouping event feature; The coordinates of the events in each group in the grouped event coordinates are averaged to obtain the second event cloud.

3. The event stream processing method according to claim 2, characterized in that: The method of aggregating the features of each group in the time dimension for the spatial domain features to obtain aggregated features specifically includes: After the spatial features pass through the multi-layer perceptron, they are activated by SoftMax to obtain the attention value of the group members, and then multiplied by the spatial features to obtain the aggregated features.

4. The event stream processing method according to claim 3, characterized in that: The pooling includes average pooling and / or maximum pooling; and, The final output result is obtained based on the average pooling and / or maximum pooling results.

5. The event stream processing method according to claim 4, characterized in that: The extracting the time domain feature to obtain the second feature specifically includes: The time domain features are extracted through the residual module to obtain the second feature.

6. The event stream processing method according to claim 5, characterized in that: The residual module includes one-dimensional convolution, batch normalization and non-linear activation operations.

7. The event stream processing method according to claim 6, characterized in that: The time domain filter and the space domain filter are global filters and learnable filters; and, The spatial domain filters in each stage are different from each other, and the temporal domain filters in each stage are also different from each other.

8. The event stream processing method according to claim 1 or 7, characterized in that: The downsampling of the original event stream output by the event camera to obtain a first event cloud specifically includes: The event cloud obtained after downsampling the original event stream is normalized to obtain a first event cloud.

9. The event stream processing method according to claim 2, characterized in that: The third post-grouping event feature obtained by normalizing the second post-grouping event feature is specifically obtained by: The third post-grouping event feature is obtained by subtracting the mean of the second post-grouping event feature from the second post-grouping event feature and then dividing it by the standard deviation between the second post-grouping event feature and the mean of the second post-grouping event feature.

10. An event stream processing device, characterized in that: The event stream processing device is configured to execute the event stream processing method described in any one of claims 1-9.

Images