Event camera optical flow estimation method based on motion perception and spatiotemporal collaborative modeling

Abstract

The present invention proposes an event camera optical flow estimation method based on motion perception and spatiotemporal collaborative modeling, comprising the following steps: preparing a data set; dividing and encoding the event data into a spatiotemporal voxel representation; constructing a motion perception spatiotemporal aggregation model, including a bidirectional correlation volume module, a spatial feature enhancement module, and a spatiotemporal motion collaborative module; training the model using a training set, and continuously optimizing the model parameters through a backpropagation algorithm to generate a corresponding high-resolution dense optical flow map. The method of the present invention can effectively alleviate the problem of missing spatial information caused by the sparsity of event data, and utilizes temporal consistency to enhance the stable perception of motion trajectories, thereby significantly improving the accuracy of optical flow estimation. It is suitable for task scenarios with high requirements for precise motion perception and real-time responsiveness in highly dynamic environments such as autonomous driving, robot navigation, and intelligent monitoring.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an event camera optical flow estimation method based on motion perception and spatiotemporal collaborative modeling. Background Art

[0002] Optical flow estimation, a core task in computer vision, aims to capture the spatiotemporal correlations between consecutive images through pixel-by-pixel motion vectors, providing key representations for dynamic scene understanding. This technology plays an important role in applications such as video reconstruction, robot navigation, autonomous driving, and target tracking. In recent years, deep learning-based optical flow estimation has become a mainstream research paradigm. By constructing a global correlation volume and combining iterative optimization mechanisms, this method has made significant progress in motion feature regression and adaptive modeling of complex scenes. However, traditional frame cameras are limited by fixed sampling frequency and limited dynamic range. In scenes with rapid motion or strong lighting changes, they are prone to motion blur and exposure distortion, which seriously affects the accuracy of optical flow estimation.

[0003] To address the perception limitations of frame-based cameras, event cameras are increasingly being used for optical flow estimation. These sensors have microsecond temporal resolution, can asynchronously perceive brightness changes, and offer high dynamic range, low latency, and extremely high temporal accuracy, giving them a natural advantage in highly dynamic scenes. As a new type of sensor, event cameras offer new possibilities for more robust and efficient optical flow estimation, driving the development of a large number of event-based optical flow estimation methods. Current mainstream methods typically encode event streams into voxel grids to adapt to image domain network architectures for optical flow prediction.

[0004] While existing methods have made some progress in event modeling, they still face three key challenges. First, event data is spatially sparse and irregular, and is often contaminated by pseudo-noise due to the asynchronous nature of event generation and the high sensitivity of sensors. These characteristics result in the constructed correlation volume lacking sufficient motion cues, which is particularly evident in low-texture or static areas. Second, existing methods generally rely on local modeling between adjacent time steps, failing to effectively exploit the high-frequency temporal consistency inherent in the event stream. Third, related work has attempted to expand the temporal window to enhance motion perception (such as the TMA method, which introduces multi-frame forward modeling). However, these methods are often limited to modeling unidirectional temporal dependencies and ignore the critical bidirectional temporal consistency in event data. Furthermore, the fixed voxel partitioning strategy employed makes it difficult to effectively capture events near temporal boundaries, further weakening the integrity of spatiotemporal modeling. In short, existing methods have significant shortcomings in modeling the spatial context and temporal consistency of event streams. In complex dynamic scenes, these methods struggle to cope with challenges such as noise interference in the spatial dimension and missing trajectories in the temporal dimension. The above limitations reveal a core problem: how to comprehensively explore the complementary characteristics of event data in the temporal and spatial dimensions to build an optical flow estimation framework with powerful spatiotemporal modeling capabilities. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of the existing technology and propose an event camera optical flow estimation method based on motion perception and spatiotemporal collaborative modeling. By aggregating motion features and modeling temporal consistency, a unified spatiotemporal perception representation is constructed, thereby significantly improving the stability and accuracy of event optical flow estimation in complex dynamic scenes, thereby solving the problems raised in the above background technology. The technical solutions provided by the present invention are as follows:

[0006] The event camera optical flow estimation method based on motion perception and spatiotemporal collaborative modeling includes the following steps:

[0007] Step 1: Collect the original event data in the real scene, generate the corresponding pseudo-true value optical flow labels through multi-sensor fusion, and construct the event optical flow dataset for training and evaluation;

[0008] Step 2: Discretize the raw event data in the time domain and encode it into an event voxel sequence with a fixed temporal resolution. Using a bidirectional voxelization strategy, the event voxel sequence is divided into two sub-segments, forward and backward, centered at the reference time point, as the input of the subsequent model.

[0009] Step 3: Build a motion perception spatiotemporal aggregation model based on a deep recurrent neural network, including a bidirectional correlation volume module, a state-space feature enhancement module, and a spatiotemporal motion coordination module;

[0010] Step 4: Divide the preprocessed event optical flow dataset into a training set, a validation set, and a test set. The model is trained using the training set. The bidirectional correlation volume module and the state-space feature enhancement module collaborate to extract multi-scale spatiotemporal features from the event voxel sequence. The spatiotemporal motion collaboration module dynamically models and fuses the extracted multi-scale spatiotemporal features during the decoding phase, outputting a high-resolution dense optical flow map. The model parameters are continuously optimized using the backpropagation algorithm to minimize the loss function. The validation set is used to monitor and adjust the training process.

[0011] Step 5: After training, select the model with the best performance on the validation set, perform optical flow estimation on the event voxel sequence of the test set, and generate the corresponding high-resolution dense optical flow map.

[0012] Preferably, step 2 is specifically as follows: for a given time window , divide it into B time windows, and embed the raw event data into voxel units using the following method :

[0013]

[0014]

[0015]

[0016] Where W and H represent the width and height of the acquisition event, represents the normalized time index, x and y represent the event pixel coordinates, b is the time window, Indicates the time when the event occurs. is the polarity of the event, indicating the direction of brightness change, It is a one-dimensional linear interpolation kernel function used to interpolate events in spatial and temporal dimensions.

[0017] Preferably, the bidirectional correlation volume module is constructed as follows: forward time correlation volume calculation reference feature and future frame features Similarity between; Backward time correlation volume calculation reference features With past frame features The similarity between them is obtained, and the calculation formula of the relevant volume is as follows:

[0018]

[0019]

[0020] Where D is the feature dimension;

[0021] For the current optical flow estimate f, it is evenly divided into L segments, and the displacement of each segment is , through a linear search strategy Extract motion-aligned features from the correlation volume as follows:

[0022]

[0023]

[0024] Thus, we can obtain L groups of forward correlation graphs and L-1 group backward correlation diagram .

[0025] Preferably, the spatial feature enhancement module follows the common optical flow estimation architecture, with four additional visual Mamba enhancement modules stacked after the CNN encoder backbone. Each enhancement module contains a Mamba mixer and a multi-head self-attention submodule, which together form a bidirectional compensation structure for local spatial modeling and global context modeling.

[0026] Preferably, the Mamba mixer in the visual Mamba enhancement module is composed of a dual-branch structure, including: a selective scanning branch, which adopts a state-space modeling mechanism, performs one-dimensional convolution processing on the input features in the channel dimension, and then introduces a selective scanning operation to model long-term temporal dependencies, thereby capturing the temporal dynamic characteristics of the event stream; a convolution branch, which retains local spatial structural information through a convolution path with a symmetrical structure and compensates for the sequence order constraints in state-space modeling; the two branches model temporal dependencies and spatial structural information in parallel.

[0027] Preferably, the spatiotemporal motion collaboration module consists of three submodules: a motion feature aggregator, a temporal trajectory modeling module, and an adaptive gated fusion unit, which are used to model spatial alignment, temporal evolution, and spatiotemporal integration, respectively.

[0028] Preferably, the motion feature aggregator first combines the features from the forward correlation graph and backward correlation graph Input into the shared motion encoder to obtain L forward related features and L-1 backward correlation features ; Then, use the target time Movement characteristics on As a guide, the motion features of each intermediate time step are aligned through a single-head crisscross attention mechanism as follows:

[0029]

[0030]

[0031]

[0032] in, 、 and is a learnable linear transformation matrix, D represents the feature dimension, MLP is a two-layer feedforward perceptron, and the final forward space enhancement feature ; By using Replace with , obtain the backward space enhancement feature .

[0033] Preferably, the input of the time trajectory modeling module is a complete motion feature sequence consisting of L forward correlation features and L-1 backward correlation features Composition, all features are aligned to a reference time point;

[0034] The model is built using standard Transformer blocks, including layer normalization, multi-head self-attention mechanism, and feedforward network, to enhance the representation of motion trajectories and temporal modeling. The forward temporal enhancement features are obtained through the above Transformer modules: ; Backward temporal enhancement features In the same way Calculated.

[0035] Preferably, the adaptive gated fusion unit fuses the spatial enhancement features and the temporal enhancement features in two steps:

[0036] Intra-time step fusion: For each time step i, a shared convolutional gating network is used to generate dynamic weights , dynamically fusion forward space enhancement features With forward time enhancement features , and get the forward enhanced feature sequence: ; Perform the same operation on the backward features to obtain the backward enhanced feature sequence ;

[0037] Fusion across time steps: Forward enhancement of feature sequences and backward enhancement feature sequence Normalize the weights using the softmax function Perform weighted aggregation:

[0038]

[0039] Finally, a 1×1 convolution is used to perform residual connection. Input to the decoder for optical flow prediction.

[0040] Preferably, the model is trained with back propagation using a gradient descent algorithm, and the loss function uses the L1 distance between the model-predicted optical flow map and the ground truth optical flow as the supervisory signal. The specific form is as follows:

[0041]

[0042] in represents the ground truth optical flow, represents the j-th predicted optical flow result, N is the total number of prediction iterations, is a hyperparameter for adjusting the loss weights at each stage.

[0043] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: on the one hand, the present invention uses a bidirectional correlation volume module and a visual Mamba spatial feature enhancement module to enhance the inherent sparsity of event streams and the motion representation under noise. The former introduces backward correlation to enhance the temporal consistency of forward estimation and compensate for the motion information of occluded and blurred areas; the latter combines state-space modeling with an attention mechanism to construct a spatial representation that combines local details with global semantics, effectively improving the motion perception and spatial consistency of event data; on the other hand, the spatiotemporal motion collaboration module is introduced to combine the motion feature aggregator with the cross-frame time modeling mechanism, and uniformly model local motion patterns and global time trajectories in the decoding stage, thereby constructing a highly consistent spatiotemporal aggregation representation. The present invention fully exploits the complementary characteristics of event data in time, space, and spatiotemporal dimensions, significantly improving the accuracy and stability of optical flow perception of occluded areas, weakly textured surfaces, and high-speed moving objects in complex dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0045] Figure 1 It is the overall flow chart of the method of the present invention;

[0046] Figure 2 It is the overall network architecture diagram of the present invention;

[0047] Figure 3 is a voxel grid partitioning diagram of the present invention;

[0048] Figure 4 It is a branch architecture diagram of the Mamba mixer of the present invention;

[0049] Figure 5 It is a qualitative comparison result diagram of the embodiment of the present invention on DSEC-FLow. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0051] In order to make the above-mentioned objects, features and effects of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Example 1: Event camera optical flow estimation method based on motion perception and spatiotemporal collaborative modeling, such as Figure 1 The specific steps are as follows.

[0053] Step 1, data set preparation. Collect the original event data in the real scene, and generate the corresponding pseudo-true value optical flow labels through multi-sensor fusion to build the event optical flow dataset for training and evaluation. The original event flow is recorded as , where N represents the total number of events in a given time window, and W, H, and T represent the width, height, and total time step of the collected events, respectively. The format is ,in, represents the event pixel coordinates, Indicates the time when the event occurs. is the polarity of the event, indicating the direction of brightness change.

[0054] In this example, the proposed optical flow estimation method was evaluated on two datasets, DSEC-Flow and MVSEC. Both datasets are collected from real-world scenes, contain frame images and event data, and are widely used for event-driven optical flow estimation tasks. For DSEC-Flow, the model was trained on the official training set and evaluated on its public test set. DSEC-Flow provides 7,800 training samples and 416 test samples with an image resolution of 640×480, covering a variety of extreme driving conditions such as night, sunrise, sunset, and tunnels. It is currently one of the mainstream event optical flow evaluation benchmarks. For MVSEC, training was performed on the outdoor_day2 sequence according to the settings of existing methods. 800 frames of samples from outdoor_day1 and three indoor_flying sequences were evaluated at different time intervals (dt=1 and dt=4).

[0055] Step 2: discretize the original event data in the time domain and encode it into an event voxel sequence with a fixed time resolution as the input of the subsequent model.

[0056] Specifically, for a given time window , divide it into B time windows and embed the events as tensors using :

[0057]

[0058]

[0059]

[0060] in represents the normalized time index, b is the time window, is a one-dimensional linear interpolation kernel function used to interpolate events in space and time dimensions. This embodiment uses a time window , B=3 time windows.

[0061] Step 3: Construct a motion-aware spatiotemporal aggregator. A dense optical flow estimation framework based on a deep recurrent neural network is constructed. The core is a spatiotemporal aggregation module for motion perception, including a bidirectional correlation volume module, a state-space feature enhancement module, and a spatiotemporal motion coordination module.

[0062] The motion-aware spatiotemporal aggregator (MoSTA) based on deep recurrent neural networks consists of two parts, such as Figure 2 As shown in the figure, the first part is a bidirectional motion modeling module based on Mamba, which is used to enhance motion representation capabilities under conditions of event data sparsity and noise interference. This module integrates the following two key designs: the first is a bidirectional correlation volume module, which collaboratively models forward and backward motion information to improve the temporal consistency and information integrity of forward optical flow estimation; the second is a spatial feature enhancement module, which combines selective state space modeling with an attention mechanism to enhance the perception of local structural details and global context. The second part is a spatiotemporal motion collaboration module, which jointly models spatial motion patterns and temporal trajectory evolution during the decoding stage to achieve adaptive aggregation and consistency enhancement of motion features in dynamic scenes.

[0063] The present invention estimates the time from the reference To target time The optical flow is targeted, and forward and reverse temporal event features are used to achieve more stable and fine-grained motion modeling. A bidirectional motion modeling module based on Mamba, which combines feature enhancement based on visual Mamba with bidirectional correlation modeling, and jointly utilizes high-frequency global context and directional temporal cues to extract dense and consistent motion representations. Specifically, its bidirectional design can introduce more temporal information when estimating the forward optical flow, thereby more closely restoring the complete motion field in occluded or blurred areas. The bidirectional correlation volume module contains two key components: one is a bidirectional voxelization strategy for dividing the event stream into time-aligned voxel units; the other is a bidirectional correlation calculation for extracting motion cues from past and future contexts.

[0064] In order to better utilize the edge-perceived event information near the time boundary, the present invention first expands the time window to enhance the context coverage. Based on this extended time interval, the present invention introduces a bidirectional voxelization strategy, which divides the event stream into two sub-segments, forward and backward, centered on the reference time point. Unlike the traditional unidirectional division method, this method introduces time clues from the past and the future at the same time, especially emphasizing the use of reverse information, thereby improving motion perception capabilities. In addition, since the entire time range is expanded, the event density is also increased accordingly, thereby alleviating sparsity and suppressing noise, which helps to obtain a more stable and informative motion representation.

[0065] like Figure 3 As shown, the event stream is first divided into two time periods: from the initial time To reference time , and the reference time To target time Each time segment is evenly divided into L voxel units, and finally a 2L voxel grid is constructed (denoted as ). The first L voxels are derived from Previous events, used for backward modeling; the last L voxels come from the time window , used for forward modeling. Voxels at the reference time point It is also shared by the forward and backward processes and used as a bridge node to connect bidirectional time modeling.

[0066] The above voxel sequence is then input into a weighted feature extraction network, which outputs a sequence of spatiotemporal feature maps. ,in The network adopts a spatial feature enhancement module based on Visual Mamba (the specific structure is shown below), which combines state space modeling with the self-attention mechanism to better capture the temporal consistency and global spatial context information in event data.

[0067] Next, a bidirectional temporal correlation volume module is constructed to encode the pixel-level similarity between the reference frame and its surrounding frames. Among them, the forward temporal correlation calculates the reference feature With the future frame Similarity between; backward temporal correlation calculation With the past frame The similarity between them is obtained. The relevant volume calculation formula is as follows:

[0068]

[0069]

[0070] Where D is the feature dimension.

[0071] In order to extract time-aligned motion cues from the correlation volume, it is assumed that the optical flow is in the interval Given the current optical flow estimate f, it is evenly divided into L segments, and the displacement of each segment is , and then through the linear search strategy Extract motion-aligned features from the correlation volume. For the i-th correlation map, the current optical flow estimate is divided into Scale to determine the sampling offset of the pixel. For each pixel, determine its deformed coordinates in the corresponding frame and define a local grid with a radius of r around it. Then perform bilinear interpolation within the grid to obtain a grid containing The feature patches are as follows:

[0072]

[0073]

[0074] Thus, we can obtain L groups of forward correlation graphs and L-1 group backward correlation diagram ,These features are then fed into the decoder to generate richer and more temporally consistent motion ,features, and are further used for subsequent optical flow fusion and ,prediction.

[0075] This bidirectional correlation modeling mechanism, designed specifically for event data, explicitly integrates motion information in both the forward and backward directions of the reference frame, effectively converting dense temporal cues into spatially aligned feature representations. It demonstrates significant advantages in complex scenarios such as occlusion, motion blur, and event sparseness, while also providing a robust and sophisticated motion encoding foundation for downstream optical flow estimation modules.

[0076] Next, to enhance the encoder's ability to extract robust spatial features from sparse and noisy event inputs, the present invention introduces a spatial feature enhancement module (MFE) based on Visual Mamba on top of a standard CNN backbone. Previous event optical flow methods typically rely on pure convolutional encoders, but their local receptive fields severely limit their ability to model global spatial context, making them susceptible to rapid motion, occlusion, or sparse event scenarios. The present invention develops a hybrid spatial encoder that combines selective state-space modeling (SSM) with a self-attention mechanism. Unlike Transformer-based encoders, the present MFE captures sequence order without explicit positional encoding, resulting in enhanced robustness when processing asynchronous and geometrically independent event data.

[0077] Specifically, the hybrid spatial encoder of the present invention follows the common optical flow estimation architecture, consisting of 6 stacked residual convolution blocks, and gradually downsamples the input feature map to 1 / 8 of the original resolution through strided convolution, and outputs a multi-scale spatial feature representation with a channel dimension of D=256. Although this design helps to capture local spatial patterns, the receptive field of convolution limits its ability to model across frames. To address this problem, the present invention stacks 4 visual Mamba enhancement modules after the CNN encoder backbone. Each enhancement module contains a Mamba mixer and a multi-head self-attention submodule, which work together to form a bidirectional compensation structure for local spatial modeling and global context modeling.

[0078] The Mamba mixer in the visual Mamba enhancement module consists of a dual-branch structure. Figure 4 As shown in the figure, it includes: a selective scanning branch, which adopts the state-space modeling mechanism, performs one-dimensional convolution processing on the input features in the channel dimension, and then introduces a selective scanning (Scan) operation to model long-term time dependencies, thereby capturing the temporal dynamic characteristics of the event stream; a convolution branch, which retains local spatial structure information through a symmetrically structured convolution path and compensates for the sequence order constraints in state-space modeling.

[0079] The two branches model temporal dependency and spatial structure information in parallel. Given the input feature from the bidirectional voxel strategy, it is represented as , the Mamba mixer output is The details are as follows:

[0080]

[0081]

[0082]

[0083] in Denotes the SiLU activation function. The one-dimensional convolution before scanning pre-mixes local spatial information between channels, enabling the model to effectively capture long-range interactions even with a single scan. Furthermore, the SSM branch naturally preserves sequence order, implicitly encoding positional information and eliminating the need for explicit positional embedding in the downstream self-attention module. The enhanced features are then fed into a standard multi-head self-attention module to further aggregate global context across spatial locations.

[0084] By adopting a weight-sharing strategy in the MFE, a solid foundation is laid for bidirectional correlation reasoning in the subsequent bidirectional correlation volume module. At low resolution, the input features are derived from L voxelized event grids, which are first encoded by a CNN. Subsequently, the encoded features of each voxel enter the SSM branch to model the implicit temporal structure implied by the event sequence within the voxel time window—an approach that helps mitigate transient noise artifacts common in sparse event scenarios. Simultaneously, the self-attention module further enhances the spatial representation by capturing long-range dependencies in the two-dimensional feature map.

[0085] Subsequently, the Spatiotemporal Motion Synergy Module (STMS) is used to jointly model local spatial motion patterns and global temporal trajectory evolution during the decoding phase. Specifically, this module introduces a unified spatiotemporal modeling framework that integrates target frame guidance, multi-scale temporal modeling, and an adaptive fusion strategy to simultaneously enhance spatial consistency and temporal coherence. The resulting motion representation helps improve the fine-grained discrimination and dynamic compensation capabilities of optical flow estimation. The detailed structure of STMS consists of three submodules: the Motion Feature Aggregator (MFA), the Temporal Trajectory Module (TTM), and the Adaptive Gate Fusion unit. These three components work in parallel to model spatial alignment, temporal evolution, and spatiotemporal integration, respectively.

[0086] Among them, the motion feature aggregator (MFA) aims to solve the problem of inconsistent representation of motion features in the intermediate time steps and the difficulty of aligning these features with the target frame. Specifically, the forward correlation graph is first and backward correlation graph Input into the shared motion encoder to obtain L forward related features and L-1 backward correlation features ; Then, use the target time Movement characteristics on As a guide, the motion features of each intermediate time step are aligned through a single-head crisscross attention mechanism. The process is as follows:

[0087]

[0088]

[0089]

[0090] in, 、 and is a learnable linear transformation matrix, D represents the feature dimension, MLP is a two-layer feedforward perceptron, and the final enhanced motion feature is recorded as Similarly, by replacing Replace with , we can obtain the backward enhanced motion features The module can explicitly enhance the motion patterns consistent with the target frame while suppressing inconsistent noise, especially in fast motion, occlusion or low texture areas.

[0091] In order to capture the global evolution of motion, we introduce the Temporal Trajectory Modeling Module (TTM), which is a lightweight Transformer-based structure used to model long-range dependencies in motion sequences. Specifically, the input of TTM is a complete motion feature sequence consisting of L forward features. and L-1 backward features Composition, all features are aligned to a reference time point.

[0092] The model is built using standard Transformer blocks, including layer normalization, multi-head self-attention mechanism, and feedforward network, to enhance the representation of motion trajectories and temporal modeling. For the forward flow, the temporally enhanced features are obtained through the above Transformer modules:

[0093]

[0094] For backward flow, the enhanced features In the same way Calculated.

[0095] Finally, given that MFA and TTM model different aspects of motion respectively—the former focuses on spatial consistency and the latter emphasizes temporal coherence—a two-stage adaptive gating mechanism is introduced again to fuse these two types of enhanced features.

[0096] Intra-time step fusion: For each time step i, a shared convolutional gating network is used to generate dynamic weights , used to dynamically fuse spatial consistency modeling results Time trajectory modeling results :

[0097]

[0098] The same operation is performed on the backward features to obtain .

[0099] Fusion across time steps: Forward enhancement of feature sequences and backward enhanced feature sequence Normalize the weights using the softmax function Perform weighted aggregation:

[0100]

[0101] Finally, in order to preserve the original structure of the motion features, a 1×1 convolution is used for residual connection, and Input to the decoder for optical flow prediction.

[0102] The STMS module integrates a target-frame alignment mechanism, cross-frame temporal modeling, and adaptive spatiotemporal feature fusion during the decoding phase. Unlike previous methods that typically process spatial and temporal information independently, STMS explicitly jointly models the spatial pattern and temporal trajectory evolution of motion, significantly improving feature consistency and detail completeness in complex scenes such as occlusion, low-texture, and fast motion.

[0103] Step 4: Model training and parameter optimization. The preprocessed event optical flow dataset is divided into a training set, a validation set, and a test set. The model is trained using the training set. The bidirectional correlation volume module and the state-space feature enhancement module collaborate to extract multi-scale spatiotemporal features from the event voxel sequence. The spatiotemporal motion collaboration module dynamically models and fuses the extracted multi-scale spatiotemporal features during the decoding phase, outputting a high-resolution dense optical flow map. The model parameters are continuously optimized through the backpropagation algorithm to minimize the loss function. The validation set is used to monitor and adjust the training process.

[0104] The model is trained with backpropagation using a gradient descent algorithm, and the optical flow prediction results are supervised and optimized using a stage-by-stage weighted L1 loss function. This example uses the AdamW optimizer with a learning rate of 0.0001. The model is trained for 250K steps on DSEC-Flow and 100K steps on MVSEC. Random cropping and horizontal flipping are used as data augmentation during training. The loss function uses the L1 distance between the model-predicted optical flow map and the ground truth optical flow as the supervisory signal, and its specific form is as follows:

[0105]

[0106] in represents the ground truth optical flow, represents the j-th predicted optical flow result, N is the total number of prediction iterations, is a hyperparameter for adjusting the loss weights at each stage.

[0107] Step 5: Model inference and result generation: After training, the model with the best performance on the validation set is selected to perform optical flow estimation on the event voxel sequence of the test set and generate the corresponding high-resolution dense optical flow map.

[0108] To verify the effectiveness of the proposed method, the method is compared with several state-of-the-art event-based methods, such as Multi-Reference Contrast Maximization (MultiCM), Vector Symbolic Architectures (VSA-Flow), Event-Based Flow Network (Ev-FlowNet), Taming Contrast Maximization (TamingCM), Event-Based Recurrent All-Pairs Field Transforms (E-RAFT), Event-based Anytime Flow (EVA-Flow), Adaptive Density Module Flow (ADMFlow), Event-Based Optical Flow Transformer (E-Flowformer), Bezier Flow (BFlow), and Temporal Motion Aggregation (TMA). The following are some examples of the proposed methods: TMA, Iterative Deblurring / Temporal Iterative Deblurring Network (IDNet / TIDNet), Efficient Event-Based MeshFlow (EEMFlow+), Spike-Based Flow Network (Spike-FlowNet), Spatio-Temporal Event-Based Flow Network (STE-FlowNet), and Dense Continuous Event-Image Fusion Flow (DCEIFlow). This example uses endpoint error (EPE) as the primary evaluation metric for DSEC-Flow and MVSEC. In addition, DSEC-Flow also introduces angular error (AE) and N-pixel error (NPE).For MVSEC, the percentage of pixels with an EPE value greater than 3 and greater than 5% of the true value was used.

[0109] Table 1 compares the performance of various methods on the DSEC-Flow dataset. Model-based (MB) and self-supervised learning (SSL) methods generally underperform supervised learning (SL) methods. MB methods such as MultiCM and VSA-Flow rely on handcrafted constraints, resulting in high EPE errors. SSL methods such as TamingCM and Ev-FlowNet also perform poorly, reflecting their difficulty capturing dense and fine-grained motion information in the absence of supervision. In contrast, SL-based methods generally achieve stronger performance but still struggle in spatiotemporal modeling. EVA-Flow and TIDNet pursue lightweight architectures for real-time inference, but their shallow temporal modeling makes them difficult to handle complex motion scenes, resulting in insufficient performance. ADMFlow introduces an adaptive density module to adapt to event sparsity, while E-Flowformer leverages a Transformer backbone to enhance spatial modeling capabilities. However, both remain weak in spatiotemporal collaborative modeling. BFlow and EEMFlow+ utilize geometric priors to guide optical flow estimation, demonstrating strong robustness in structured motion scenes but struggle to adapt to complex, fine-grained dynamics. TMA enhances local consistency by aggregating motion features, but lacks global temporal modeling and still processes event slices independently. It is worth noting that existing SL methods generally do not explicitly utilize bidirectional temporal cues or build long-term temporal consistency. The model proposed in this invention achieves an EPE of 0.70 on DSEC-Flow, which is about 9% higher than E-RAFT. Compared with existing advanced methods, the method of this invention achieves a good balance between accuracy and efficiency, and even outperforms IDNet running at 1 / 4 resolution, fully demonstrating the effective modeling ability of the model of this invention for the temporal structure of event data.

[0110] Table 1 Performance evaluation on the DSEC-Flow dataset

[0111]

[0112] Table 2 compares typical optical flow methods in terms of accuracy, model size, and inference latency. E-RAFT is used as the baseline method of the present invention, while TMA and BFlow represent high-precision methods recently built under similar architectures; EVA-Flow and TIDNet are lightweight models designed for real-time performance. From the comparison results, although the method proposed in the present invention does not yet have the real-time inference capability of TIDNet, its performance is close to or even slightly better than other recent methods. In particular, compared with the baseline method E-RAFT, the model of the present invention can improve the inference efficiency by up to 7 times while maintaining higher estimation accuracy. It is worth emphasizing that this efficiency improvement is mainly due to the introduction of bidirectional temporal cues, which enriches the motion representation while reducing the dependence on deep iterative updates. The results show that the model of the present invention can significantly improve the inference speed while still maintaining a high optical flow estimation accuracy.

[0113] Table 2 Comparison of optical flow estimation methods in terms of accuracy (EPE), model size, and inference latency

[0114]

[0115] Figure 5 Qualitative results are presented and compared with BFlow and E-RAFT. Existing methods often fail to fully exploit the spatiotemporal characteristics of event data, resulting in blurred edges and insufficient texture representation. In contrast, our method, by fully exploiting the temporal consistency and spatial structure of event streams, can more effectively capture clear edge details and demonstrate excellent handling of occlusion scenarios.

[0116] The present invention is trained on outdoor daytime 2 sequences and tested on indoor flight and outdoor daytime 1 sequences. Table 3 reports the results under different frame interval settings, with dt=1 in the upper half and dt=4 in the lower half. Under the setting of dt=4, the method of the present invention shows the best accuracy among all methods; when dt=1, the performance on outdoor daytime 1 is also close to that of the dual-modal method DCEIFlow. This result shows that the model of the present invention has excellent same-domain performance on outdoor event sequences. However, like other existing advanced supervised models, the performance of the model of the present invention on indoor flight sequences is still inferior to the model-based method MultiCM and most self-supervised or unsupervised methods. The main reason for this gap is that the generalization ability of supervised learning methods is limited, especially when there is a large domain difference between the training set and the test set.

[0117] Table 3 Performance evaluation on the MVSEC dataset

[0118]

[0119] In contrast, model-based and unsupervised methods are less sensitive to domain changes because they do not rely on annotations or have self-guidance mechanisms. Despite this, our method still achieves the best overall performance among all purely event-based supervised methods, highlighting its effectiveness in estimating high-quality, single-modal event optical flow. These findings also suggest that semi-supervised frameworks that incorporate prior knowledge, such as MultiCM, may provide a promising direction for future research, potentially achieving stronger cross-environment generalization while maintaining accuracy in the same domain.

[0120] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein with equivalents. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. An event camera optical flow estimation method based on motion perception and spatiotemporal collaborative modeling, characterized by: The following steps are involved: Step 1: Collect the original event data in the real scene, generate the corresponding pseudo-true value optical flow labels through multi-sensor fusion, and construct the event optical flow dataset for training and evaluation; Step 2: Discretize the raw event data in the time domain and encode it into an event voxel sequence with a fixed temporal resolution. Using a bidirectional voxelization strategy, the event voxel sequence is divided into two sub-segments, forward and backward, centered at the reference time point, as the input of the subsequent model. Step 3: Build a motion perception spatiotemporal aggregation model based on a deep recurrent neural network, including a bidirectional correlation volume module, a state-space feature enhancement module, and a spatiotemporal motion coordination module; Step 4: Divide the preprocessed event optical flow dataset into a training set, a validation set, and a test set. The model is trained using the training set. The bidirectional correlation volume module and the state-space feature enhancement module collaborate to extract multi-scale spatiotemporal features from the event voxel sequence. The spatiotemporal motion collaboration module dynamically models and fuses the extracted multi-scale spatiotemporal features during the decoding phase, outputting a high-resolution dense optical flow map. The model parameters are continuously optimized using the backpropagation algorithm to minimize the loss function. The validation set is used to monitor and adjust the training process. Step 5: After training, select the model with the best performance on the validation set, perform optical flow estimation on the event voxel sequence of the test set, and generate the corresponding high-resolution dense optical flow map.

2. The method for event camera optical flow estimation based on motion perception and spatiotemporal collaborative modeling according to claim 1, characterized in that: Step 2 is as follows: For a given time window , divide it into B time windows, and embed the raw event data into voxel units using the following method : ； ； ； Where W and H represent the width and height of the acquisition event, represents the normalized time index, x and y represent the event pixel coordinates, b is the time window, Indicates the time when the event occurs. is the polarity of the event, indicating the direction of brightness change, It is a one-dimensional linear interpolation kernel function used to interpolate events in spatial and temporal dimensions.

3. The method for event camera optical flow estimation based on motion perception and spatiotemporal collaborative modeling according to claim 2, characterized in that: The bidirectional correlation volume module is constructed as follows: forward time correlation volume calculation reference feature and future frame features similarities between; Backward time-correlated volume calculation reference features With past frame features The similarity between them is obtained, and the calculation formula of the relevant volume is as follows: ； ； Where D is the feature dimension; For the current optical flow estimate f, it is evenly divided into L segments, and the displacement of each segment is , through a linear search strategy Extract motion-aligned features from the correlation volume as follows: ； ； Thus, we can obtain L groups of forward correlation graphs and L-1 group backward correlation diagram .

4. The method for event camera optical flow estimation based on motion perception and spatiotemporal collaborative modeling according to claim 3, characterized in that: The spatial feature enhancement module follows the common optical flow estimation architecture and stacks four additional visual Mamba enhancement modules after the CNN encoder backbone. Each enhancement module contains a Mamba mixer and a multi-head self-attention sub-module, which together form a bidirectional compensation structure for local spatial modeling and global context modeling.

5. The method for event camera optical flow estimation based on motion perception and spatiotemporal collaborative modeling according to claim 4, characterized in that: The Mamba mixer in the visual Mamba enhancement module consists of a dual-branch structure, including: a selective scanning branch that adopts a state-space modeling mechanism to perform one-dimensional convolution processing on the input features in the channel dimension, and then introduces a selective scanning operation to model long-term temporal dependencies, thereby capturing the temporal dynamic characteristics of the event stream; a convolution branch that retains local spatial structural information through a symmetrically structured convolution path and compensates for the sequential order constraints in state-space modeling. The two branches model temporal dependencies and spatial structural information in parallel.

6. The method for event camera optical flow estimation based on motion perception and spatiotemporal collaborative modeling according to claim 4, characterized in that: The spatiotemporal motion collaboration module consists of three sub-modules: a motion feature aggregator, a temporal trajectory modeling module, and an adaptive gated fusion unit, which are used to model spatial alignment, temporal evolution, and spatiotemporal integration, respectively.

7. The method for event camera optical flow estimation based on motion perception and spatiotemporal collaborative modeling according to claim 6, characterized in that: The motion feature aggregator first integrates the features from the forward correlation graph and backward correlation graph Input into the shared motion encoder to obtain L forward related features and L-1 backward correlation features ; Then, use the target time Movement characteristics on As a guide, the motion features of each intermediate time step are aligned through a single-head crisscross attention mechanism as follows: ； ； ； in, 、 and is a learnable linear transformation matrix, D represents the feature dimension, MLP is a two-layer feedforward perceptron, and the final forward space enhancement feature is obtained ; By replacing Replace with , obtain the backward space enhancement feature .

8. The method for event camera optical flow estimation based on motion perception and spatiotemporal collaborative modeling according to claim 6, characterized in that: The input of the temporal trajectory modeling module is a complete motion feature sequence consisting of L forward correlation features and L-1 backward correlation features Composition, all features are aligned to a reference time point; The model is built using standard Transformer blocks, including layer normalization, multi-head self-attention mechanism, and feedforward network, to enhance the representation of motion trajectories and temporal modeling. The forward temporal enhancement features are obtained through the above Transformer modules: ; Backward temporal enhancement features In the same way Calculated.

9. The method for event camera optical flow estimation based on motion perception and spatiotemporal collaborative modeling according to claim 6, characterized in that: The adaptive gated fusion unit fuses spatial enhancement features and temporal enhancement features in two steps: Intra-time step fusion: For each time step i, a shared convolutional gating network is used to generate dynamic weights , dynamically fusion forward space enhancement features With forward time enhancement features , and get the forward enhanced feature sequence: ; Perform the same operation on the backward features to obtain the backward enhanced feature sequence ; Fusion across time steps: Forward enhancement of feature sequences and backward enhancement feature sequence Normalize the weights using the softmax function Perform weighted aggregation: ； Finally, a 1×1 convolution is used to perform residual connection. Input to the decoder for optical flow prediction.

10. The method for event camera optical flow estimation based on motion perception and spatiotemporal collaborative modeling according to any one of claims 1 to 9, characterized in that: The model is trained with back propagation using the gradient descent algorithm. The loss function uses the L1 distance between the model-predicted optical flow map and the ground truth optical flow as the supervisory signal. The specific form is as follows: ； in represents the ground truth optical flow, represents the j-th predicted optical flow result, N is the total number of prediction iterations, is a hyperparameter for adjusting the loss weights at each stage.

Images