A method for action classification for dynamic vision sensors

Abstract

The application discloses a motion classification method for a dynamic vision sensor and relates to the field of motion classification. The application first adopts an event index plane, considers the space-time information in the cross neighborhood of each event in an event stream, calculates the motion gradient direction of each event as the local motion feature of each event in the event stream, then does pooling and pulse coding in the space-time dimension, and finally inputs the pulse sequence into a pulse neural network to output the classification result of the motion in the scene. The application introduces the event index plane with speed invariance to record the global motion history information of the motion, introduces the motion history information and the gradient direction calculation into the processing of the event stream, and proposes a novel motion gradient direction feature expression method based on events, which can efficiently extract the motion information of the motion and improve the motion recognition accuracy. In addition, compared with the optical flow motion feature expression, the proposed feature expression method avoids the complicated optical flow estimation process and saves the calculation energy consumption.

Description

Technical Field

[0001] This invention relates to the field of motion classification, and in particular to a motion classification method for dynamic vision sensors. Background Technology

[0002] Inspired by the light intensity sensitivity of the retina, event cameras capture changes in scene light intensity and output spatiotemporal event streams in the form of Address Event Representation (AER). Traditional algorithms cannot directly process this asynchronous sparse event stream data with high temporal resolution, while biomimetic spiking neural networks (SNNs) are naturally compatible with this type of event data and have attracted widespread attention from academia and industry.

[0003] Currently, research combining event cameras and SNNs primarily focuses on object classification tasks. One approach is based on separate feature extraction, with the most representative method being the Surface of Activated Event (SAE) feature representation, which proposes that precise temporal information in an event stream can be represented by the timestamp of the latest event for each pixel. To capture more potential spatiotemporal information in the event stream, some researchers have introduced normalization processes into SAE computation. For example, the Hierarchy of Time-Surfaces (HOTS) method uses an exponential kernel to normalize the SAE within the local neighborhood of the activated event. However, normalizing the SAE only within the local neighborhood is still susceptible to interference from noisy events, thus hindering the feature representation process. The Histograms of Averaged Time-Surfaces (HATS) method proposes to grid the entire pixel region and perform regularization on each grid, thereby improving the robustness of SAEs. Afshar S et al. argued that feature representations should possess velocity invariance, meaning that the numerical range of the feature representation should remain stable as the object's velocity changes. They proposed that using event indices instead of timestamps to calculate SAEs would be more beneficial for stable classifier recognition. Another type of feature representation method simulates the feedforward information processing of the ventral flow in the visual cortex. It typically employs a set of direction-selective Gabor filters to extract edge direction features from the event stream in an event-driven computational manner, and a time-based max-pooling method based on a competition mechanism to ensure feature invariance, reduce the number of events, and accelerate computation. Based on this feature representation, effective classification can be achieved using only a single-layer SNN. The second mainstream method is based on end-to-end deep SNNs. The event stream is directly input into a deep SNN after preprocessing steps such as temporal resolution adjustment, and then feature extraction and learning are performed end-to-end using surrogate gradient techniques.

[0004] Event cameras, with their high temporal resolution and high dynamic range, naturally avoid motion blur, enabling them to capture human motion changes more efficiently. While existing event stream classification models have achieved excellent object classification performance, they often suffer performance losses for more spatiotemporally dynamic action classification tasks. Therefore, event-based action classification has gradually become a research hotspot in recent years. A popular solution is to convert event streams into frame sequences using methods such as indexed event planes, and then use computer vision methods to achieve event-based action recognition. However, this frame sequence representation loses the computational advantage of the original high temporal resolution of events. Another approach is to directly utilize event streams for action recognition. Maro et al. proposed a dynamic background suppression method based on the HOTS framework to remove noisy events in the background and combined it with a k-nearest neighbor classifier to achieve gesture classification. In recent years, some SNN-based action classification works have also emerged. George A M et al. proposed an SNN architecture combining convolution and reservoir computation for spatiotemporal feature extraction and classification of gesture actions. Based on surrogate gradient techniques, various supervised learning algorithms applicable to deep convolutional SNNs can learn the feature extraction and classification of event streams end-to-end, such as Spatio-Temporal Backpropagation (STBP) and Spike Layer Error Reassignment in Time (SLAYER).

[0005] Inspired by the extraction and processing of motion information in the dorsal flow of the visual cortex, Liu Q et al. proposed a hierarchical SNN model utilizing event motion information. This model introduces a motion-sensitive neuron with synaptic delay to estimate optical flow, thereby extracting motion direction and velocity features to achieve event-based action classification. However, this model has two significant problems: first, utilizing synaptic delay introduces additional computational latency, which is detrimental to real-time action recognition applications; second, motion velocity information may interfere with action recognition results. Research indicates that event flow recognition should possess velocity invariance, meaning that the model should ensure stable recognition regardless of the motion velocity. Summary of the Invention

[0006] The purpose of this invention is to provide a motion classification method for dynamic vision sensors, so as to improve motion recognition accuracy and save computational energy consumption.

[0007] To achieve the above objectives, the present invention provides the following solution:

[0008] A motion classification method for dynamic vision sensors includes:

[0009] The event camera is used to read the dynamic information of the actions in the scene in real time, and the dynamic information is converted into an event stream;

[0010] Using an event index plane and considering the spatiotemporal information within the cross-shaped neighborhood of each event in the event stream, the motion gradient direction of each event is calculated as the local motion feature of each event in the event stream;

[0011] Pool the local motion features of each event in the event stream along the spatiotemporal dimension to obtain a sparse spatiotemporal feature event stream;

[0012] Pulse coding is performed on sparse spatiotemporal feature event streams to obtain pulse sequences;

[0013] The pulse sequence is input into a spiking neural network, which outputs the classification results of actions in the scene.

[0014] Optionally, the dynamic information is converted into an event stream, specifically including:

[0015] Automatically capture dynamic information from the event camera using Dynamic Vision Viewer software;

[0016] Enable local buffer pool caching of the dynamic information;

[0017] The dv-python interface is called to establish TCP communication with the Dynamic Vision Viewer software server, and dynamic information within a 10000us time window is read from the local buffer pool to form an event stream.

[0018] Optionally, the step of employing an event index plane and considering the spatiotemporal information within the cross-shaped neighborhood of each event in the event stream to calculate the motion gradient direction of each event as a local motion feature of each event in the event stream specifically includes:

[0019] Update the event index plane with the index of the current event to record historical motion information of the action by using the latest index at each position in the event index plane;

[0020] Calculate the value in the cross-shaped neighborhood with radius r centered on the current event's position on the updated event index plane;

[0021] Based on the values ​​in the cross-shaped neighborhood of the current event, calculate the motion gradient direction of the current event, which serves as the local motion feature of the current event.

[0022] Optionally, the formula for calculating the value on the cross-shaped neighborhood is as follows:

[0023] U(e i ,r)={X right ,X left ,Yup ,Y down}

[0024]

[0025] In the formula, U(e i (r) represents event e i The value of the cross neighborhood, e i =[x i ,y i ,t i ,p i ],(x i ,y i ) for event e i The coordinates, t i For event e i timestamp, p i For event e i The polarity of Y; up Y down X left X right , respectively, for event e i The values ​​of the top, bottom, left, and right sides in the cross-shaped neighborhood, where τ is the time decay constant and IS is the event index plane.

[0026] Optionally, the formula for calculating the direction of the motion gradient is as follows:

[0027] ΔX=X right -X left ; ΔY = Y up -Y down ;

[0028]

[0029] In the formula, For event e i The directions of the motion gradients are given, where ΔX is the horizontal gradient and ΔY is the vertical gradient.

[0030] Optionally, the step of pooling the local motion features of each event in the event stream along the spatiotemporal dimension to obtain a sparse spatiotemporal feature event stream specifically includes:

[0031] The spatiotemporal pooling method is used to pool the local motion features of each event in the event stream, reducing the spatial information of the event stream to 1 / (c×c) of the original, and removing redundant information within each refractory period time window on the time scale, thus obtaining a sparse spatiotemporal feature event stream.

[0032] Optionally, the step of pulse encoding the sparse spatiotemporal feature event stream to obtain a pulse sequence specifically includes:

[0033] Merge two polarity events in a sparse spatiotemporal feature event stream to obtain a merged event;

[0034] Based on the synaptic location, using formula e j =[t j addr j ] and addr j =(f j -1)×m×n+(y j -1)×m+x j The merged events are encoded as pulse events, and all pulse events constitute a pulse sequence; where, e j For the j-th impulse event, t j For the timestamp of the j-th merge event, addr j Let x be the synaptic location index of the j-th merging event, (x i ,y i Let f be the coordinate of the j-th merging event. j Let m and n represent the features of the j-th merged event, and m and n be the input resolution sizes of the sparse spatiotemporal feature event stream.

[0035] Optionally, the pulse sequence is input into a spiking neural network to output a classification result of actions in the scene, which further includes:

[0036] The event-driven Tempotron learning algorithm is used to train the spiking neural network.

[0037] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0038] This invention discloses an action classification method for dynamic visual sensors. First, it employs an event index plane and considers the spatiotemporal information within the cross-shaped neighborhood of each event in the event stream, calculating the motion gradient direction of each event as its local motion feature. Then, pooling and pulse coding are performed in the spatiotemporal dimension. Finally, the pulse sequence is input into a spiking neural network to output the action classification result in the scene. This invention introduces a velocity-invariant event index plane to record the global motion history information of actions, incorporating motion history information and gradient direction calculation into the event stream processing. It proposes a novel event-based motion gradient direction feature representation method, which can efficiently extract motion information and improve action recognition accuracy. Furthermore, compared to optical flow motion feature representation, the proposed feature representation method avoids the computationally complex optical flow estimation process, saving computational energy. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A flowchart of an action classification method for dynamic vision sensors provided in an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of a motion classification method for dynamic vision sensors provided in an embodiment of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] The purpose of this invention is to provide a motion classification method for dynamic vision sensors, so as to improve motion recognition accuracy and save computational energy consumption.

[0044] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] This invention provides a method for action classification using dynamic visual sensors, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0046] Step S1: Use an event camera to read the dynamic information of actions in the scene in real time, and convert the dynamic information into an event stream.

[0047] The DV system reads the output information of the event camera in real time, performs preprocessing and noise reduction, and then uses it as the input of the entire system, including the following process:

[0048] This invention uses Dynamic Vision Viewer software from iniVation AG to read and display data from event cameras in real time. The read data is transmitted to the system for processing in real time by calling its open-source dv-python interface.

[0049] First, connect the event camera to the PC via USB. The Dynamic Vision Viewer software can automatically capture the connected event camera device and display the output.

[0050] Next, configure the module connection of the Dynamic Vision Viewer software, add the Dv net output tcpserver module, start the Dynamic Vision Viewer software server, set the local address (127.0.0.1) and port number (7777), and enable the local buffer pool to cache events.

[0051] Finally, the dv-python interface is invoked to establish TCP communication with the Dynamic Vision Viewer software server, reading the event stream within a 10000us time window from the local buffer pool. The event stream consists of a series of events in the form of quadruples, with the data format ['x','y','t','p'], where:

[0052] 1.t represents a timestamp: the data type is int64, and it is assigned according to the event trigger time.

[0053] 2. x and y represent coordinates: the data type is int32, and the values ​​are determined by the position of the event in the camera coordinates. x and y are related to the resolution of the DVS.

[0054] 3. p indicates polarity: the data type is int8, and it is determined by the change in light intensity of the event. When the light intensity increases from weak to strong, it is 1, and vice versa.

[0055] Step S2: Using the event index plane and considering the spatiotemporal information within the cross-shaped neighborhood of each event in the event stream, calculate the motion gradient direction of each event as the local motion feature of each event in the event stream.

[0056] The process of encoding local action features of the input event stream using the event index plane and the arctangent function includes the following steps:

[0057] Step 2.1: Record global motion information of the event. After receiving the event input, update the event index plane IS with the index of the current event, and record the historical motion information of the action by using the latest index for each position. Whenever an event e i =[x i ,y i ,t i ,p i When it arrives, that is

[0058] IS(x i ,y i ,p i ) = i

[0059] Here, IS is the event index plane, set to have the same resolution as the camera, and i is the index of the current event. Furthermore, to eliminate the influence of positive and negative polarities, the event index plane records events of both polarities separately.

[0060] Step 2.2: Obtain local motion information of the current event. To further characterize the local motion information of the action, four values ​​on a cross-shaped neighborhood with the current event's position as the center and r as the radius are selected on the event index plane to calculate the motion direction gradient information within the local neighborhood:

[0061]

[0062]

[0063] Here, τ is the time decay constant, to avoid motion blur. The value in the cross-shaped neighborhood is denoted as U(e i ,r)={X right ,X left ,Y up ,Y down U characterizes the spatiotemporal relationship between the current event and historical events in the local neighborhood by calculating the distance between the surrounding events and the current event index.

[0064] Step 2.3: In order to further obtain local motion information, the motion gradient direction of the current event is calculated using this correlation, and the motion direction information is used as a feature of the event.

[0065]

[0066] ΔX=X right -X left

[0067] ΔY=Y up -Y down

[0068] Here, arctan2(ΔY,ΔX) is the arctangent function, returning an azimuth angle in the range (-π,π). Its calculation formula is:

[0069] Step S3: Pool the local motion features of each event in the event stream in the spatiotemporal dimension to obtain a sparse spatiotemporal feature event stream.

[0070] The spatiotemporal pooling method is used to pool the encoded event stream, which includes the following steps:

[0071] Step 3.1: Spatial Pooling. The pooling layer consists of... It consists of 10 neurons, where m and n are the input resolution size of the event stream, c is the size of the pooling space region, and f is the total number of features (i.e., the number of categories in the azimuth angle θ).

[0072] The pooling layer performs pooling operations on c×c pixel regions that do not overlap with the feature event stream. Each c×c neuron in the feature event stream is connected to a corresponding neuron in the pooling layer. The neurons in the pooling layer are set with very low thresholds to ensure that a single event can activate the neuron.

[0073] Step 3.2: Temporal pooling. To obtain sparse feature events on a temporal scale, the refractory period is used to filter the event stream.

[0074] Each neuron in the pooling layer fires a pulse after receiving an input event, and then enters a refractory period t. ref During this refractory period, neurons do not receive any input and do not fire impulses. Only after the refractory period do neurons integrate the input information.

[0075] After spatiotemporal pooling, the spatial information of the feature event stream is scaled down to 1 / (c×c) of the original value, and the time scale is reduced by removing the value for each t. ref Redundant information within the time window is extracted, thus yielding sparse and crucial spatiotemporal features, reducing the computational burden on subsequent spiking neural networks.

[0076] Step S4: Pulse coding is performed on the sparse spatiotemporal feature event stream to obtain a pulse sequence.

[0077] Encoding a sparse stream of feature events into a pulse sequence involves the following steps:

[0078] Step 4.1: Merge the two polarity events. Each event encoded by the feature is denoted as e. j =[x j ,y j ,t j ,p j ,f j ].

[0079] By merging events of two polarities in chronological order, we obtain {e} j =[x j ,y j ,t j ,f j ]}, where f j This represents the j-th feature.

[0080] Step 4.2: Obtain the pulse sequence using synaptic position encoding. (The event e is then encoded.) i Encoded as pulse e i The process is as follows:

[0081] addrj =(f j -1)×m×n+(y j -1)×m+x j

[0082] e j =[x j ,y j ,t j ,f j ]→e j =[t j addr j ]

[0083] Among them, addr j Let e ​​be the synaptic location index of the j-th event. j For the j-th impulse event, t j Let x be the timestamp of the j-th merge event, (x j ,y j Let f be the coordinate of the j-th merging event. j Let m and n represent the features of the j-th merged event, and m and n be the input resolution sizes of the sparse spatiotemporal feature event stream.

[0084] Step S5: Input the pulse sequence into the spiking neural network and output the classification results of the actions in the scene.

[0085] Training an SNN network based on the event-driven Tempotron learning algorithm and performing real-time action classification includes the following steps:

[0086] Step 5.1: Feedforward Calculation. The neuron model used in the network is the LIF neuron model. The membrane voltage of the LIF neuron is obtained by the weighted sum of the postsynaptic potentials of all input pulses. The typical time-driven calculation formula is as follows:

[0087]

[0088] Among them, V rest It is the resting potential, t i w represents the pulse time of the last pulse of the i-th input neuron. i It is the synaptic weight, K(tt) i τ is the normalized postsynaptic potential contributed by the input pulse. V0 is the normalization coefficient, ensuring that the increase in postsynaptic voltage caused by each input pulse does not exceed 1mV. m and τ s Let V(t) represent the membrane and synapse time constants, respectively. When the membrane voltage V(t) exceeds the threshold V... thr At this time, the neuron will fire a pulse, and then the membrane voltage will be reset to the resting potential.

[0089] To accommodate asynchronous pulse computation, an event-driven membrane voltage update process for LIF neurons is employed:

[0090] V(t i ) = V m (t i )-V s (t i )

[0091]

[0092] Wherein, V(t) i ) is the membrane voltage when the i-th event occurs, Δt = t i -t i-1 V is the time interval between the current event and the previous event at the same location. m and V s The initial value is 0. i Synaptic weights are determined by the synaptic location index addr. i Find the synaptic weight w at the corresponding location i .

[0093] Step 5.2: Action Feature Learning. The Tempotron learning rule aims to train LIF neurons to fire a pulse (called P) when receiving target input. + (State), otherwise remain silent (called P) - (State). Once a neuron fires in the wrong state, the Tempotron learning rule is triggered. The error backpropagation algorithm is used to adjust the synaptic weights, and the loss function is defined as the distance between the maximum value of the membrane voltage and the threshold.

[0094]

[0095] Among them, V tmax It is the maximum membrane voltage of the neuron, t max It is the moment when the neuron reaches its maximum membrane voltage, "P". + "Error" refers to something that should have been issued but wasn't; "P" - "Error" refers to an event that should not have been issued but was. The synaptic update rules using Tempotron are as follows:

[0096] w i =w i +Δw i

[0097] According to the gradient descent algorithm, we know that... Right now

[0098]

[0099] Where η is the learning rate, and the synaptic weights are initialized to a normal distribution with a mean of 0 and a standard deviation of 0.1.

[0100] This invention employs the following key technologies: First, an index surface (IS) with velocity invariance is introduced to record the global motion history information of actions. Second, based on this motion history information, the local motion history information within the cross-shaped neighborhood of the input event is calculated using the arctangent function, thereby obtaining the motion gradient direction features of the event. Finally, mimicking the brain's spiking neuron model LIF (leaky integrate-and-fire), a robust group neural network structure, and the event-driven Tempotron algorithm, the encoded features are used for action learning and classification.

[0101] This invention proposes a real-time motion classification system based on an event camera, which has the following advantages:

[0102] (1) High temporal resolution and low latency: The system employs an event-driven motion feature description operator, which can efficiently pulse-encode high temporal resolution (nanosecond level) event streams, preserving accurate spatiotemporal information during motion. This feature can be fully utilized by the backend spiking neural network classifier for efficient learning and online real-time classification. The following are comparisons of the processing speed and classification performance of this invention with different feature description operators on the benchmark action event stream classification dataset DVS128 Gesture:

[0103] Table 1 Performance comparison of different feature representation methods

[0104]

[0105] Among them, Comparative Example 1: Orchard G, Meyer C, Etienne-Cummings R, Posch C, Thakor Nand Benosman R. 2015. HFirst: A Temporal Approach to Object Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(10): 2028-2040.

[0106] Comparative Example 2: Lagorce

[0107] Comparative Example 3: Liu Q, Xing D, Tang H, Ma D andPan G. 2021. Event-based ActionRecognition Using Motion Information and Spiking Neural Networks\Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. Montreal, Canada: International Joint Conferences on Artificial Intelligence Organization: 743-1749.

[0108] (2) Low redundancy and high energy efficiency: The system adopts an asynchronous event-driven computing paradigm from feature extraction to spiking neural network classification. It only processes the event points that respond to changes in light intensity, avoiding the computational redundancy caused by traditional frame-by-frame processing, and greatly saving the computational energy consumption of the model.

[0109] (3) High precision and robustness: The system employs an event-driven motion feature description operator, which can effectively extract potential spatiotemporal motion information from the event stream and provides a pulse code representation with separable motion features. This representation allows the backend spiking neural network classifier to more easily learn the spatiotemporal differences of the action event stream, thus achieving high recognition accuracy. The performance comparison results of this model with several existing optimal action event stream recognition methods on several benchmark event stream datasets are as follows:

[0110] Table 2 Comparison of Event-Based Action Recognition Algorithms

[0111]

[0112] Among them, Comparative Example 1: Shrestha S B and Orchard G. 2018. SLAYER: Spike Layer Error Reassignment in Time / / Advances in Neural Information Processing Systems 31. Montréal, Canada: MIT Press: 1412 - 1421.

[0113] Comparative Example 2: He W, Wu Y, Deng L, Li G, Wang H, Tian Y, Ding W, Wang W and Xie Y. 2020. Comparing SNNs and RNNs on neuromorphic vision datasets: Similarities and differences. Neural Networks, 132: 108 - 120. [DOI: 10.1016 / j.neunet.2020.08.001].

[0114] Comparative Example 3: Kaiser J, Mostafa H and Neftci E. 2020. Synaptic Plasticity Dynamics for Deep Continuous Local Learning (DECOLLE). Frontiers in Neuroscience, 14: 424.

[0115] Comparative Example 4: Xing Y, Di Caterina G and Soraghan, J. 2020. A New Spiking Convolutional Recurrent Neural Network (SCRNN) With Applications to Event - Based Hand Gesture Recognition. Frontiers in Neuroscience, 14: 1143. [DOI: 10.3389 / fnins.2020.590164].

[0116] Comparative Example 5: George A M, Banerjee D, Dey S, Mukherjee A and Balamurali P. 2020. A Reservoir-based Convolutional Spiking Neural Network for Gesture Recognition from DVS Input / / 2020 International Joint Conference on Neural Networks (IJCNN). Glasgow, United Kingdom: IEEE: 1-9.

[0117] Comparative Example 6: Xiao R, Tang H, Ma Y, Yan R and Orchard G. 2020. An Event-Driven Categorization Model for AER Image Sensors Using Multispike Encoding and Learning. IEEE Transactions on Neural Networks and Learning Systems, 31(9): 3649-3657.

[0118] Comparative Example 7: Liu Q, Ruan H, Xing D, Tang H and Pan G. 2020. Effective AER Object Classification Using Segmented Probability-Maximization Learning in Spiking Neural Networks / / Proceedings of the AAAI Conference on Artificial Intelligence, New York, USA: AAAI: 1308-1315.

[0119] Comparative Example 8:Liu Q,

[0120] This invention introduces motion history information and gradient direction calculation into event flow processing, proposing a novel event-based motion gradient direction feature representation method that can efficiently extract motion information of actions. Furthermore, compared to optical flow motion feature representation, the proposed feature representation method avoids the computationally complex optical flow estimation process, offering a significant advantage in processing speed.

[0121] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0122] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method of action classification for a dynamic vision sensor, characterized by, The application relates to a method for classifying actions in a scene, comprising: reading dynamic information of actions in a scene in real time using an event camera, and converting the dynamic information into an event stream; adopting an event index plane, and considering spatiotemporal information in a cross-neighborhood of each event in the event stream, calculating a motion gradient direction of each event as a local motion feature of each event in the event stream; specifically comprising: updating the event index plane with an index of a current event to record historical motion information of actions by using the latest index in each position of the event index plane to record the historical motion information of the actions; calculating values in a cross-neighborhood with a position of the current event as a center and r as a radius on the updated event index plane; calculating a motion gradient direction of the current event as a local motion feature of the current event according to the values in the cross-neighborhood of the current event; pooling the local motion features of each event in the event stream in a spatiotemporal dimension to obtain a sparse spatiotemporal feature event stream; Pulse coding is performed on the sparse spatiotemporal feature event stream to obtain a pulse sequence; specifically comprising: merging two polarity events in the sparse spatiotemporal feature event stream to obtain a merged event; based on a synapse position, using the formula e j =[t j ,addr j ] and addr j =(f j -1)×m×n+(y j -1)×m+x j Encode the merged event into a pulse event, and all pulse events constitute a pulse sequence; in the formula, e j is the jth pulse event, t j is the time stamp of the jth merged event, addr j is the synapse position index of the jth merged event, (x j , y j ) is the coordinate of the jth merged event, f j indicates the feature of the jth merged event, and m and n are the input resolution size of the sparse spatiotemporal feature event stream; inputting the pulse sequence into a pulse neural network to output a classification result of actions in the scene.

2. The dynamic vision sensor oriented action classification method of claim 1, wherein, The converting the dynamic information into an event stream specifically comprises: automatically capturing dynamic information in the event camera through Dynamic Vision Viewer software; enabling a local buffer pool to cache the dynamic information; calling a dv-python interface to establish TCP communication with a Dynamic Vision Viewer software server, reading dynamic information in a 10000us time window from the local buffer pool to form an event stream. 3.The dynamic vision sensor oriented action classification method of claim 1, wherein, The calculation formula of the values in the cross-neighborhood is U(e i r) = {X right , X left , Y up , Y down} where U(e i ,r) is the value on the cross-neighborhood of event e i , e i =[x i ,y i ,t i ,p i ], (x i ,y i ) is the coordinate of event e i , t i is the timestamp of event e i , p i is the polarity of event e i ; Y up , Y down , X left , X right are the values of the upper, lower, left, right of the cross-neighborhood of event e i , respectively, τ is the time decay constant, and IS is the event index plane.

4. The method of claim 3, wherein, The calculation formula of the motion gradient direction is ΔX = X right - X left ΔY = Y up - Y down wherein is the motion gradient direction of event e i ΔX is the horizontal gradient and ΔY is the vertical gradient.

5. The dynamic vision sensor oriented action classification method of claim 1, wherein, The pooling the local motion features of each event in the event stream in a spatiotemporal dimension to obtain a sparse spatiotemporal feature event stream specifically comprises: pooling the local motion features of each event in the event stream by using a spatiotemporal pooling method, so that the spatial information of the event stream is reduced to 1 / (cxc) of the original, and redundant information in each refractory period time window is removed in the time scale, and a sparse spatiotemporal feature event stream is obtained.

6. The dynamic vision sensor oriented action classification method of claim 1, wherein, The inputting the pulse sequence into a pulse neural network to output a classification result of actions in the scene further comprises: training the pulse neural network by using an event-driven Tempotron learning algorithm.

Images