Event flow action identification method and system based on frame-point synergistic effect

Abstract

The invention provides an event flow action identification method and system based on a frame-point synergistic effect, and relates to the technical field of computer vision. The method comprises the following steps: extracting context event points from an original event stream by using a pulse type context learning device, and adaptively selecting a plurality of key event points from the context event points; sorting the key event points according to a Hilbert curve to reserve a spatial topological structure, and modeling long space-time dependence of the key event points to obtain event point features; converting the original event stream into a structured event frame through a time accumulation window; performing feature extraction on the event frame based on a Transform network to obtain an event frame feature; and fusing event point features and event frame features, introducing a CLIP text encoder to generate frame / point related semantic prompts, and finishing action recognition by comparing the consistency of loss constraint event features and text semantics.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an event flow action recognition method and system based on frame-point collaborative interaction. Background Technology

[0002] Event-based motion recognition leverages the high temporal resolution (microsecond-level accuracy) and dynamic-only response characteristics of event cameras, demonstrating significant advantages in high-speed motion capture, low-power computing, and privacy protection (by not recording static visual content).

[0003] Current mainstream methods face fundamental challenges:

[0004] One type of research involves forcibly converting asynchronous event streams into dense event frames and then processing them using traditional CNNs. While this utilizes a mature architecture, it results in the loss of fine-grained temporal information in the event data and introduces redundant computations.

[0005] Another approach is to directly use point cloud networks to process raw event points. While this preserves sparsity, it is difficult to effectively model the unique "dense time-sparse space" coupling characteristics of event flows.

[0006] Neither of the two existing technologies can fully adapt to the essential characteristics of event data—the dynamic balance between high-resolution asynchronous signals in the time dimension and extreme sparsity in the spatial dimension—leading to limited spatiotemporal modeling efficiency. Summary of the Invention

[0007] Purpose of the invention: In order to overcome the problems existing in the prior art, this invention proposes an event flow action recognition method and system based on frame-point collaboration, which achieves the optimal balance between computational efficiency and model performance, thereby promoting the practical deployment of event cameras in real-world scenarios.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, an event stream action recognition method based on frame-point collaboration is proposed, including the following steps: S1. Use a pulsed context learner to extract context event points from the original event stream, and adaptively select several key event points from all context event points through recursive convolution and membrane potential accumulation mechanism. S2. Encode the key event points and sort the encoded key event points according to the Hilbert curve to preserve the spatial topology. At the same time, model the long-term spatiotemporal dependence of the key event points to obtain event point features. S3. Convert the raw event stream into structured event frames using a time accumulation window. S4. Based on the Transformer network, feature extraction is performed on the event frame obtained in step S3 to capture the static contour information of the event frame and obtain the event frame features. S5. The event point features obtained in step S2 and the event frame features obtained in step S4 are fused together, and the CLIP text encoder is introduced to generate frame / point related semantic prompts. By comparing the consistency between the loss constraint event features and the text semantics, the action recognition is completed.

[0009] As a preferred embodiment of the first aspect, a pulsed context learner is used to extract context event points from the raw event stream, specifically including: By using recurrent convolutions of SRRNNs, and combining the temporal density and spatial sparsity of the original event stream, a continuous and context-rich sequence of event points is generated, satisfying the following formula:

[0010] In the formula, This represents the convolution operation; t represents the time step of the event point. express The event points between; Represents the sigmoid function; For residual weights; , These are the recursive convolution weights for event points and peak discharges, respectively. , These are the recursive convolution weights used to update the decay factor for event points and peak discharges, respectively. It is the attenuation factor; For the first A spike discharge with a time step.

[0011] As a preferred approach to the first aspect, several key event points are adaptively selected from all context event points through recursive convolution and membrane potential accumulation mechanisms, specifically including: A LIF-like neuron model is adopted, and key event points are dynamically selected through membrane potential accumulation and threshold triggering mechanisms to filter redundant noise. The charging process of the LIF-like neuron model satisfies the following formula:

[0012] In the formula, and This represents the membrane potential of the i-th neuron in the l-th layer before and after charging; For input current; It is the output of the spike neuron; and These represent the membrane time constant and the discharge threshold, respectively. It is the Heaviside function; This indicates a reset potential.

[0013] As a preferred approach to the first aspect, during the extraction of context event points from the original event stream, the decay factor controlled by the Sigmoid function is dynamically adjusted. This enables sampling at non-fixed time intervals, resulting in event slices. satisfy:

[0014] In the formula, Index representing the sampling interval, Indicates the sampling time step; Represents a sequence of context event points. , The spatial coordinates of the event point, For the polarity of the event, The time when the event occurred; , These are the start and end times of the k-th sampling interval, respectively; N is the number of event points; From event slice collection Aggregate context event points Where H and W represent the length and width of the space, respectively, and C is the number of feature channels. Indicates the sampling time step.

[0015] As a preferred approach to the first aspect, the encoded key event points are sorted according to Hilbert curves to preserve the spatial topology, while the long-term spatiotemporal dependencies of the key event points are modeled to obtain event point features, specifically including: The context event points represented in three dimensions are converted into topology-preserving one-dimensional serialized event points by using Hilbert space-filling curves; The Spiking Mamba block is used to perform long-term time-series modeling of one-dimensional serialized event points. The Spiking Mamba block contains a selective state-space model (SSM) and a spurious activation function (SN). Local spatiotemporal features are extracted using grouped convolution (SGC) and pointwise convolution (SPC).

[0016] As a preferred embodiment of the first aspect, the context event points represented in three dimensions are converted into topology-preserving one-dimensional serialized event points using Hilbert space-filling curves, specifically including: context event point Spatial and temporal information is constructed into a three-dimensional representation, reindexed, and then processed through three-dimensional convolution. Projecting to non-overlapping patches In the process, learnable spatial location embeddings are merged. and time location embedding Obtain the serialization point token .

[0017] As a preferred embodiment of the first aspect, a Spiking Mamba block is used to perform long-term time-series modeling of serialized event points. The Spiking Mamba block includes a Selective State-Space Model (SSM) and a Impulse Activation Function (SN). The SSM is used to extract the long-term temporal correlation of the event sequence, satisfying… In the formula This represents the input features of the l-th Spiking Mamba block. This represents the features after SMAMBA processing and the addition of residual connections. This represents the feature extraction operation of a selective state-space model.

[0018] As a preferred approach in the first aspect, local spatiotemporal features are extracted through grouped convolution (SGC) and pointwise convolution (SPC). The grouped convolution and pointwise convolution satisfy the following formula:

[0019] In the formula, This indicates a peak-based pointwise convolutional layer; This represents pointwise convolution; This indicates a peak-by-peak convolutional layer; This indicates group-by-group convolution; This indicates that stacked SGC and SPC layers extract local features.

[0020] As a preferred embodiment of the first aspect, in step S5, the event point features... With event frame features Enhanced dynamic motion features are generated through element-wise multiplication and fusion. ; Generate semantic cue features related to event frames and event points using a pre-trained CLIP text encoder. and Where Q is the number of action categories and D is the feature dimension; By comparing the loss functions, the event frame features are constrained respectively. Semantic cue features of event frames Event point characteristics Semantic cue features of event points The semantic consistency, the contrastive loss function is defined as:

[0021] In the formula, This represents a feature vector extracted from event data; This represents the text prompt feature; k represents the index of all possible categories. q represents the index of the positive sample class being calculated; This represents the q-th text prompt feature; The text prompt feature representing the k-th action category; This represents the temperature coefficient, which controls the sharpness of the similarity distribution.

[0022] A second aspect of the present invention provides an event flow action recognition system, the system comprising: The event data preprocessing module is used to receive the raw event stream and process it in parallel into two data streams. One stream stacks the data at fixed time intervals or with a certain number of events to generate a dense sequence of event frames, while the other stream outputs the original sparse set of event points. The pulsed context learner is connected to the event data preprocessing module at its input end to receive the sparse event point set. The pulsed context learner is based on a pulsed residual recurrent neural network, and its core is a leaky integral firing neuron model. By simulating the accumulation and firing mechanism of neural pulses, it performs nonlinear sampling on the continuous input event points and outputs a series of context event point sets containing context information and non-uniform intervals. An event point encoder, whose input is connected to the output of the pulsed context learner, is used to receive the set of context event points and output event point features. An event frame encoder, whose input is connected to the event data preprocessing module, is used to receive the event frame sequence and output event frame features; The text prompt encoder is used to receive two sets of prompt texts, namely the prompt text related to the event frame and the prompt text related to the event point, and encode the two sets of prompt texts into event frame semantic prompt features and event point semantic prompt features, respectively; The feature fusion and action recognition module has its input terminals connected to the output terminals of the event point encoder, event frame encoder, and text prompt encoder, respectively. The feature fusion and action recognition module fuses the event point features and event frame features through a residual connection method to generate the final comprehensive event features. By comparing the consistency between the event features and the text semantics, the action recognition is completed.

[0023] As a preferred embodiment of the second aspect, the event point encoder includes: The Hilbert scan unit is used to traverse the set of context event points in the spatiotemporal dimension using Hilbert curves, serializing them into one-dimensional serialized event points while preserving the spatial proximity between event points. The embedded projection unit uses a 3D convolutional layer to project one-dimensional serialized event points into a series of embedding vectors, forming a non-overlapping patch. The location encoding unit is used to add spatial and temporal location embeddings to non-overlapping patches to generate the final serialized point token; The pulse mamba block stack is composed of multiple identical pulse mamba blocks cascaded together. Each pulse mamba block executes sequentially on the input serialized point tokens: Pulse Mamba operation based on selective state-space model to capture long temporal dependencies; Feedforward operations based on pulse grouped convolution are used to extract local features; Output the feature vector of the event point.

[0024] Compared with the prior art, the present invention has the following beneficial effects: First, this invention adopts a frame-point collaborative representation learning framework, which achieves complementary advantages between event frame branches and event point branches, and can significantly reduce computational overhead while ensuring recognition accuracy, thus balancing efficiency and performance.

[0025] Secondly, this invention introduces LIF-like neurons and spiking residual recurrent networks (SRRNN) into event point processing, which can dynamically filter key event points, effectively remove redundant events and noise, and improve the quality and stability of feature extraction.

[0026] Furthermore, this invention combines Hilbert space-filling curves with Spiking Mamba state-space modeling to capture long-term temporal dependency information while maintaining spatial topology, thereby enhancing the ability to model fast motion and complex actions.

[0027] Furthermore, by fusing event point features and event frame features, this invention takes into account both static contour information and dynamic motion information, making the action representation more comprehensive and detailed.

[0028] Furthermore, this invention utilizes action semantic cues generated by a text encoder and achieves cross-modal semantic alignment by comparing the loss constraint event features with the semantic space. This helps the model distinguish subtle action differences and improves generalization ability.

[0029] The method of this invention features low latency and low power consumption, making it suitable for various real-time applications such as autonomous driving, security monitoring, augmented reality and virtual reality interaction, and robot perception. Experimental results show that this invention achieves significant performance improvements on multiple public datasets, verifying its superior accuracy, robustness, and scalability. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the event flow action recognition framework of the present invention.

[0031] Figure 2This is a flowchart of the event flow action recognition method based on frame-point collaboration of the present invention.

[0032] Figure 3 Visualization of events before and after processing by the pulsed context processor for the SeAct dataset.

[0033] Figure 4 This is a visualization of the top 3 prediction results of this invention in the SeAct dataset.

[0034] Figure 5 This is a schematic diagram illustrating the fusion of event points and event frame features in this invention.

[0035] Figure 6 This is a comparison chart showing the computational efficiency of the present invention and the optimal framework. Detailed Implementation

[0036] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0037] Example 1 Figure 1 This invention demonstrates the structure of an event stream action recognition framework based on frame-point collaboration. The framework mainly includes: an event data preprocessing module, a pulsed context learner (SCL), an event point encoder, an event frame encoder, a text prompt encoder, and a feature fusion and action recognition module.

[0038] For event point embedding, a pulsed context learner (SCL) and a Hilbert scanned pulse mamba (HSM) module for event point encoding were designed to extract contextual event points and mine the spatiotemporal correlation of asynchronous event points, respectively. Meanwhile, for event frame embedding, event frames stacked from the event stream are fed to an event frame encoder (e.g., a Transformer) to obtain event frame features. Similarly, frame cue features (guided by frame-related cueing generated by the CLIP text encoder) also play a guiding role.

[0039] The event data preprocessing module receives the raw event stream data. The raw event stream data is a set of asynchronous event points output by the event camera, denoted as... ,in Represents the spatial coordinates of the event point. Indicates the polarity of the event. This represents the event timestamp. This module processes the raw event stream E in parallel into two data streams: one stream stacks the events at fixed time intervals or based on the number of events, generating a dense sequence of event frames. Where H and W are the spatial resolutions. One output is the frame count; the other output directly shows the original set of sparse event points. .

[0040] The input of the pulsed context learner is connected to the event data preprocessing module to receive a set of sparse event points. The learner is internally implemented based on a pulse residual recurrent neural network, with a leaky integral firing neuron model at its core. By simulating the accumulation and firing mechanism of neural pulses, the learner nonlinearly samples continuous input event points, outputting a series of contextual event points with non-uniform intervals containing contextual information. ,in Let C be the number of time steps after sampling, and C be the number of feature channels. The data flow relationship is as follows: .

[0041] The input of the event point encoder is connected to the output of the pulsed context learner to receive the set of context event points. The event point encoder consists of several units that sequentially perform the following sub-operations: 1. Hilbert scanning unit: Traversing the spatiotemporal dimension using Hilbert curves serializes the event into a one-dimensional sequence while preserving the spatial proximity between event points.

[0042] 2. Embedded Projection Unit: A 3D convolutional layer is used to project the serialized event point data into a series of embedding vectors, forming point tokens. .

[0043] 3. Position encoding unit: for Add learnable temporal and spatial location codes to generate the final serialized point token. .

[0044] 4. Pulse Mamba Block Stack: Consists of L identical pulse mamba blocks cascaded together. Each pulse mamba block corresponds to the input... The stack executes the following operations sequentially: 1) a pulsed Mamba operation based on a selective state-space model to capture long-term temporal dependencies; and 2) a feedforward operation based on pulsed grouped convolutions to extract local features. Finally, the stack outputs event point feature vectors. The data flow relationship is as follows: .

[0045] The input of the event frame encoder is connected to the event data preprocessing module to receive event frame sequences. The encoder is a lightweight visual Transformer network that uses a multi-layer self-attention mechanism to... Perform feature extraction and output event frame feature vectors. The data flow relationship is as follows: .

[0046] The text cue encoder employs a pre-trained CLIP text encoder. This encoder receives two distinct text cue templates: one cue associated with an event frame and one cue associated with an event point (e.g., "waving gesture"). The encoder encodes these two sets of cue texts into frame cue feature matrices, respectively. and point hint feature matrix , where Q is the total number of action categories.

[0047] The input terminals of the feature fusion and recognition module are connected to the output terminals of the event point encoder, event frame encoder, and text prompt encoder, respectively. This module performs the following operations: 1. Feature Fusion: Combining event point features With event frame features The final comprehensive event features are generated by fusing data using residual joins. Specifically, the fusion formula is: , where ⊙ represents element-wise multiplication.

[0048] 2. Comparative Learning and Recognition: During the training phase, comprehensive event features are calculated. With frame cue features Comparison loss between Simultaneously calculate event point features With dot hint features Comparison loss between Total loss function .in To balance hyperparameters, during the testing or inference phase, comprehensive event features are calculated. With frame cue features of all categories The similarity is used to obtain the final classification probability distribution through the Softmax function, thus completing the behavior recognition.

[0049] In a further embodiment, the pulsating context learner (SCL) extracts context event points from the redundant raw event stream, specifically including: This invention introduces a LIF (Leaky Integrate-and-Fire) neuron model for event point processing. By utilizing the accumulation of membrane potential and threshold triggering mechanism, it achieves dynamic screening of key event points, thereby effectively removing redundant noise.

[0050] Based on this, the proposed Spiking Residual Recurrent Neural Network (SRRNN) can combine the high temporal density and spatial sparsity of event streams to generate continuous event point sequences with contextual information through recurrent convolution.

[0051] Meanwhile, the present invention also adaptively adjusts the attenuation factor (controlled by the Sigmoid function) according to the dynamic changes of event points, thereby realizing a sampling method with non-fixed time intervals and preserving long-term temporal dependencies in the event stream.

[0052] Through the above design, the present invention can reduce redundancy while ensuring information integrity, and improve the efficiency of event point feature extraction and the accuracy of recognition.

[0053] In a further embodiment, the Hilbert Scanning Pulse Mamba (HSM) module sorts event points according to Hilbert curves, preserving spatial topology and efficiently modeling long-term spatiotemporal dependencies of event points, specifically including: This invention designs a Hilbert Scan Pulse Mamba (HSM) module in the event point encoding stage to solve the problem that existing methods cannot simultaneously maintain spatial topology and long temporal dependencies.

[0054] First, this invention introduces a spatial scanning method based on Hilbert curves, which rearranges context event points into a serialized representation according to spatial proximity and temporal order, thereby preserving the spatiotemporal topological relationships of event points during the serialization process. Through this method, a three-dimensional spatiotemporal distribution is mapped to a one-dimensional sequence, while avoiding the high complexity associated with point cloud or graph structure modeling.

[0055] Building upon this foundation, this invention proposes a selective state-space model (Spiking Mamba) incorporating a spiking neuron mechanism. This structure introduces spiking activation functions and grouped convolutions during state-space modeling, enabling the model to effectively capture long-term temporal dependencies of event points while maintaining low computational overhead. This module can extract local spatiotemporal features and model motion information over long time spans, thereby forming stable and discriminative feature representations of event points.

[0056] Through the above design, the Hilbert Scan Pulse Mamba (HSM) module can improve the recognition performance of complex dynamic actions while ensuring efficiency, effectively alleviating the performance bottleneck of event data in large-scale scenarios. Furthermore, the spatial scanning method is not limited to Hilbert curves, and other scanning strategies that preserve neighborhood topology can also be used; the state space modeling method can also be replaced with other forms of temporal modeling units to adapt to different application requirements.

[0057] In a further embodiment, event point features and event frame features are fused through element-wise multiplication to enhance dynamic motion information, specifically including: In the fusion process of event point features and event frame features, this invention adopts a residual fusion strategy based on element-wise multiplication, which enables the two types of features to interact at a fine-grained level. This not only preserves the static contour information contained in the frame features, but also further enhances the dynamic motion information carried by the event point features.

[0058] This fusion and semantic constraint design enables the invention not only to capture dynamic and static information during the action process, but also to enhance the model's ability to distinguish subtle differences in actions by utilizing prior linguistic knowledge, thus significantly improving the accuracy and robustness in complex action recognition tasks.

[0059] Example 2 Based on the event stream action recognition framework based on frame-point collaboration disclosed in Embodiment 1 above, this embodiment discloses the specific implementation process of the event stream action recognition method based on frame-point collaboration, see [link to embodiment]. Figure 2 .

[0060] Step S1, Event Point Extraction: Considering the high temporal resolution of event cameras, the massive raw event streams recorded at the microsecond level pose a challenge to most temporal resolution methods. For these methods, sparse and asynchronous event streams are sampled and aggregated through a sliding window of fixed size. Therefore, such sampling strategies not only destroy the temporal correlation between events but also become independent of subsequent feature extraction operations. Research has found that pulsed discharges in Spike Neural Networks (SNNs) are highly consistent with event-based sampling. Therefore, this invention addresses the above challenges by introducing a Pulsating Context Learner (SCL), which can extract context event points from redundant raw event streams using a Spiking Residual Recurrent Neural Network (SRRNN) and effectively combine them with subsequent feature exploration.

[0061] The method of this invention is based on the leakage integral and firing neurons used in neuromorphic computing. Its charging process is described by the following equation:

[0062] and This represents the membrane potential of the i-th neuron in the l-th layer before and after charging. This is the input current. It is the output of the spike neuron. and These represent the membrane time constant and the discharge threshold, respectively. It's the heaviside function. When it receives a spike input, the neuron will input current... Accumulated to its membrane potential In this state, the potential decays at a constant rate. When the membrane potential... Exceeding the threshold At that time, the launch spike will... Reset to ,in This indicates a reset potential.

[0063] The mechanism of membrane potential accumulation in spike neural networks is highly consistent with the underlying principle of event-based sampling. Once the accumulated event information exceeds a predefined threshold, events from the asynchronous stream should be selected. Therefore, this invention utilizes the functionality of leaked integrals and activated neuron models, and fully considers the contextual information between events, to learn the sampling of original event points. Considering the sparsity of event data in the spatial dimension and the density in the temporal dimension, this invention employs recursive synaptic connections to extract a continuous and information-dense subset of event points embedded with contextual information, as shown in the following formula:

[0064] In the above formula, This represents the convolution operation, where t represents the time step of the event. [t] Event points between [1, t] This represents the sigmoid function. This is the residual weight, which defaults to 0.5. It represents the event point at time step t. and at the t Spike discharge s[t] with a time step 1] Recursive convolution and right Make a contribution. Similarly, the decay factor γ[t] at time step t is obtained through recursive convolution. and Depend on and Update. Based on the above process, the previous pulse emission time is obtained. and the current peak distribution time Event slices between ( Interval sampling is implemented using an index representing the sampling interval, as shown below:

[0065] Finally, from Aggregate context event points Where H and W represent the size of the space, This represents the sampling time step, and C is the number of feature channels. The visualization of events before and after processing by the Pulsed Context Learner (SCL) on the SeAct dataset is visible. Figure 3 .

[0066] Step S2, Event Point Mining: Previous schemes for event point encoding typically fall into two distinct categories: (1) the direct use of spiking neural networks consistent with the characteristics of event cameras; and (2) the utilization of point clouds or graph structures initially intended for processing 3D data to handle event points. However, these schemes have not adequately addressed the inherent spatiotemporal nuances of event data and the long-term interdependencies of asynchronous event points, both of which are critical for accurate identification tasks. To address these issues, this invention proposes a novel Hilbert Scan Pulse Mamba Event Point Encoder (HSM) that reconsiders the nature of event points. Specifically, the Hilbert Scan Pulse Mamba Event Point Encoder mainly comprises a Hilbert scan and a spiked Mamba block.

[0067] To preserve the spatiotemporal distribution characteristics of event points, this invention first uses a space-filling curve—the Hilbert curve—to maintain the spatial topology without repeatedly traversing all event points. First, the 2D spatial and 1D temporal event information of each event point is constructed into a 3D representation for processing within the forward and backward Hilbert curves. Second, the context event points are re-indexed. and using 3D convolution to Projected onto a non-overlapping patch with D dimensions In the context, P is the size of the patch. Third, by merging learnable spatial location embeddings... and time location embedding To obtain serialized dot tokens .

[0068] Integrating a spike neural network into the Mamba block leverages its spike-based asynchronous event processing capabilities to analyze contextual event points in long-term series. This ensures low power consumption while fully utilizing the high temporal resolution information of event points. First, the Mamba block efficiently parallelizes and captures long-term temporal correlations within event sequences through a selective state-space model.

[0069] Based on the spike-feedforward network, the spike-group convolutional layers are merged into the spike-point convolutional layers to extract local features while reducing computational overhead. The formula is as follows:

[0070] In the formula, This indicates a peak-based pointwise convolutional layer; This represents pointwise convolution; This indicates a peak-by-peak convolutional layer; This indicates group-by-group convolution; This indicates that stacked SGC and SPC layers extract local features. When... When the l-th pulse Mamba block is input, the event point feature is obtained. It can be expressed as follows:

[0071] in This represents the output feature of the l-th spike Mamba block, and the output of the last block is identified as the final event point feature. .

[0072] Step S3, Event Frame Stacking: By accumulating asynchronous event data within a continuous time window and projecting it onto a regular spatiotemporal grid (such as a 3D voxel or a 2D image), structured data representation is generated.

[0073] like Figure 4 As shown, the visualization of four sets of event actions is presented in the form of event frames, along with a comparison of the top three recognition results of the two methods on the SeAct dataset. As shown in the first and third rows, rapid movement causes blurring of stacked event frames, which hinders the recognition ability of frame-based methods, specifically "running with a ball" vs. "walking with a box". The second and fourth rows illustrate how event actions with subtle inter-class differences (e.g., "walking" vs. "interlacing") can mislead the ExACT frame method when stacked in event frames. These results demonstrate that the method of this invention cleverly captures the essence of motion dynamics by combining the spatial contour information of event frames with the temporal trajectory of event points, thereby mitigating the impact of motion blur.

[0074] Step S4, Event Frame Embedding: such as Figure 1 As shown, it stacks spaces of size (H, W). event frames The input is fed into an event frame encoder (e.g., a Transformer), which outputs event frame features. Furthermore, it is also related to the final frame features. The event points are connected by their characteristics.

[0075] Step S5, Feature Fusion: Event-text consistency is constrained by the contrast loss between event frame features and event point features. During the fusion process, this invention employs an element-wise multiplication-based feature interaction method, multiplying event point features and event frame features element-wise to highlight the consistent parts between the two types of features, thereby enhancing dynamic motion information while preserving static appearance information. To further improve the expressive power of the fused features, the fusion result can be added to the original features using a residual method to avoid information loss caused by a single fusion method. The specific formula is shown below:

[0076] Subscript , The temperature coefficient is used. Specific contrast losses La and Lo were obtained for the event frame embedding and event point embedding branches, respectively. The final overall confirmation loss consists of La and Lo.

[0077] In terms of form, such as Figure 5 Represent the frame text features of all class labels as ,in Represents the text features of the q-th class. Given an event frame feature... It performs residual fusion with the event point features to obtain the classification of the q-th class, as shown in the following formula:

[0078] Furthermore, this invention utilizes a pre-trained text encoding model (such as the CLIP text encoder) to generate semantic cues related to action categories, such as "waving" and "squatting." These semantic cues are encoded and mapped to the same embedding space as the event features, and a contrastive loss function constrains the event features and text features. This design ensures that the fused event features maintain consistency with the corresponding action category in the semantic space, thereby achieving cross-modal alignment.

[0079] The superiority of the proposed Spiking Context Learner (SCL) method over traditional sliding window sampling strategies and basic spike neural network sampling methods was verified. As shown in Table 1, the comparison results show that the sampling of the basic spiking neural network conforms to the asynchronous event sampling principle, which is beneficial for event sampling, and improves performance by 0.98% (Top-1) compared to sliding window sampling on PAF. From the basic sampling of the spiking neural network, the Spiking Context Learner (SCL) of this invention takes into account important contextual information and exhibits better performance.

[0080] Table 1. Comparison of the method of the present invention with several traditional methods

[0081] As shown in Table 2, the proposed method was evaluated based on current representative methods on the PAF, SeAct, HARDVS, and DVS 128 Gesture datasets. The proposed cooperative perception framework demonstrates superior performance on these datasets. Specifically, the frame-point cooperative perception framework achieves improvements of 5.17% (Top-1) and 14.65% (Top-5) on the SeAct dataset with 58 dynamic actions, respectively. On the HARDVS dataset containing 300 action classes, the frame-point cooperative perception framework delivers improvements of 7.01% (Top-1) and 2.79% (Top-5), significantly outperforming the ExACT framework. Simultaneously, the frame-point cooperative perception framework achieves an accuracy of 96.49% on the PAF dataset (containing 10 different categories) and 98.80% on the DVS 128 Gesture dataset (spanning 11 different gesture categories), indicating that its performance is comparable to the ExACT framework. These results demonstrate that while the ExACT framework performs well on small datasets, it faces challenges on large datasets containing multiple categories. The frame-point collaborative sensing framework proposed in this invention fully utilizes the spatiotemporal information in the asynchronous event stream from the perspective of event frames and event points, effectively solving the challenges brought about by complex, large-scale datasets.

[0082] Table 2 Evaluation results for each dataset

[0083] like Figure 6 This paper compares the collaborative sensing framework with the current best framework (ExAct) in terms of computational efficiency. The framework of this invention demonstrates a significant advantage in computational efficiency, specifically in the following aspects: First, in terms of floating-point operations, this method achieves lower computational complexity across multiple tasks, significantly reducing computational resource consumption compared to the best framework (ExAct). Second, in terms of computational throughput efficiency, the framework of this invention achieves the highest value of 0.24, indicating that it is more efficient in parallel processing and resource utilization. In summary, the framework of this invention provides a superior solution for real-time event processing tasks with lower computational cost, higher processing efficiency, and stronger stability.

[0084] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. An event stream action recognition method based on frame-point collaborative interaction, characterized in that, Includes the following steps: S1. Use a pulsed context learner to extract context event points from the original event stream, and adaptively select several key event points from all context event points through recursive convolution and membrane potential accumulation mechanism. S2. Encode the key event points and sort the encoded key event points according to the Hilbert curve to preserve the spatial topology. At the same time, model the long-term spatiotemporal dependence of the key event points to obtain event point features. S3. Convert the raw event stream into structured event frames using a time accumulation window. S4. Based on the Transformer network, feature extraction is performed on the event frame obtained in step S3 to capture the static contour information of the event frame and obtain the event frame features. S5. The event point features obtained in step S2 and the event frame features obtained in step S4 are fused together, and the CLIP text encoder is introduced to generate frame / point related semantic prompts. By comparing the consistency between the loss constraint event features and the text semantics, the action recognition is completed.

2. The event stream action recognition method based on frame-point collaborative interaction according to claim 1, characterized in that, Extracting context event points from the raw event stream using a pulsed context learner, specifically including: By using recurrent convolutions of SRRNNs, and combining the temporal density and spatial sparsity of the original event stream, a continuous and context-rich sequence of event points is generated, satisfying the following formula: In the formula, This represents the convolution operation; t represents the time step of the event point. express The event points between; Represents the sigmoid function; For residual weights; , These are the recursive convolution weights for event points and peak discharges, respectively. , These are the recursive convolution weights used to update the decay factor for event points and peak discharges, respectively. It is the attenuation factor; For the first A spike discharge with a time step.

3. The event stream action recognition method based on frame-point collaboration according to claim 1, characterized in that, Several key event points are adaptively selected from all context event points using recursive convolution and membrane potential accumulation mechanisms, specifically including: A LIF-like neuron model is adopted, and key event points are dynamically selected through membrane potential accumulation and threshold triggering mechanisms to filter redundant noise. The charging process of the LIF-like neuron model satisfies the following formula: In the formula, and This represents the membrane potential of the i-th neuron in the l-th layer before and after charging; For input current; It is the output of the spike neuron; and These represent the membrane time constant and the discharge threshold, respectively. It is the Heaviside function; This indicates a reset potential.

4. The event stream action recognition method based on frame-point collaboration according to claim 1, characterized in that, During the extraction of context event points from the original event stream, the decay factor controlled by the Sigmoid function is dynamically adjusted. This enables sampling at non-fixed time intervals, resulting in event slices. satisfy: In the formula, Index representing the sampling interval, Indicates the sampling time step; Represents a sequence of context event points. , The spatial coordinates of the event point, For the polarity of the event, The time when the event occurred; , These are the start and end times of the k-th sampling interval, respectively; N is the number of event points; From event slice collection Aggregate context event points Where H and W represent the length and width of the space, respectively, and C is the number of feature channels. Indicates the sampling time step.

5. The event stream action recognition method based on frame-point collaboration according to claim 1, characterized in that, The encoded key event points are sorted according to Hilbert curves to preserve the spatial topology, and the long-term spatiotemporal dependencies of the key event points are modeled to obtain event point features, specifically including: The context event points represented in three dimensions are converted into one-dimensional serialized event points by using Hilbert space filling curves; The Spiking Mamba block is used to perform long-term time-series modeling of one-dimensional serialized event points. The Spiking Mamba block contains a selective state-space model (SSM) and a spurious activation function (SN). Local spatiotemporal features are extracted using grouped convolution (SGC) and pointwise convolution (SPC).

6. The event stream action recognition method based on frame-point collaboration according to claim 5, characterized in that, Spiking Mamba blocks are used to perform long-term time-series modeling of serialized event points. These blocks include a Selective State-Space Model (SSM) and a Impulse Activation Function (SN). The SSM is used to extract the long-term temporal correlation of the event sequences, satisfying the following conditions: In the formula This represents the input features of the l-th Spiking Mamba block. This represents the features after SMAMBA processing and the addition of residual connections. This represents the feature extraction operation of a selective state-space model.

7. The event stream action recognition method based on frame-point collaboration according to claim 5, characterized in that, Local spatiotemporal features are extracted using grouped convolution (SGC) and pointwise convolution (SPC). The grouped convolution and pointwise convolution satisfy the following formula: In the formula, This indicates a peak-based pointwise convolutional layer; This represents pointwise convolution; This indicates a peak-by-peak convolutional layer; This indicates group-by-group convolution; This indicates that stacked SGC and SPC layers extract local features.

8. The event stream action recognition method based on frame-point collaboration according to claim 1, characterized in that, In step S5, event point features Event frame features Enhanced dynamic motion features are generated through element-wise multiplication and fusion. ; Generate semantic cue features related to event frames and event points using a pre-trained CLIP text encoder. and Where Q is the number of action categories and D is the feature dimension; By comparing the loss functions, the event frame features are constrained respectively. Semantic cue features of event frames Event point characteristics Semantic cue features of event points The semantic consistency, the contrastive loss function is defined as: In the formula, This represents a feature vector extracted from event data; represents the text prompt features; k represents the index of all possible categories; q represents the index of the positive sample category being calculated. This represents the q-th text prompt feature; The text prompt feature representing the k-th action category; This represents the temperature coefficient, which controls the sharpness of the similarity distribution.

9. An event flow action recognition system, characterized in that, include: The event data preprocessing module is used to receive the raw event stream and process it in parallel into two data streams. One stream stacks the data at fixed time intervals or with a certain number of events to generate a dense sequence of event frames, while the other stream outputs the original sparse set of event points. The pulsed context learner is connected to the event data preprocessing module at its input end to receive the sparse event point set. The pulsed context learner is based on a pulsed residual recurrent neural network, and its core is a leaky integral firing neuron model. By simulating the accumulation and firing mechanism of neural pulses, it performs nonlinear sampling on the continuous input event points and outputs a series of context event point sets containing context information and non-uniform intervals. An event point encoder, whose input is connected to the output of the pulsed context learner, is used to receive the set of context event points and output event point features. An event frame encoder, whose input is connected to the event data preprocessing module, is used to receive the event frame sequence and output event frame features; The text prompt encoder is used to receive two sets of prompt texts, namely the prompt text related to the event frame and the prompt text related to the event point, and encode the two sets of prompt texts into event frame semantic prompt features and event point semantic prompt features, respectively; The feature fusion and action recognition module has its input terminals connected to the output terminals of the event point encoder, event frame encoder, and text prompt encoder, respectively. The feature fusion and action recognition module fuses the event point features and event frame features through a residual connection method to generate the final comprehensive event features. By comparing the consistency between the event features and the text semantics, the action recognition is completed.

10. An event flow action recognition system according to claim 9, characterized in that, The event point encoder includes: The Hilbert scan unit is used to traverse the set of context event points in the spatiotemporal dimension using Hilbert curves, serializing them into one-dimensional serialized event points while preserving the spatial proximity between event points. The embedded projection unit uses a 3D convolutional layer to project one-dimensional serialized event points into a series of embedding vectors, forming a non-overlapping patch. The location encoding unit is used to add spatial and temporal location embeddings to non-overlapping patches to generate the final serialized point token; The pulse mamba block stack is composed of multiple identical pulse mamba blocks cascaded together. Each pulse mamba block executes sequentially on the input serialized point tokens: Pulse Mamba operation based on selective state-space model to capture long temporal dependencies; Feedforward operations based on pulse grouped convolution are used to extract local features; Output the feature vector of the event point.

Images