A bidirectional unsupervised method and system for calculating optical flow of events from an event camera.

Abstract

This invention relates to a bidirectional unsupervised method and system for calculating optical flow in an event camera. The method includes: acquiring a discrete event stream and converting it into forward and reverse event volumes through discrete sampling; extracting features using an optical flow computation convolutional network and enhancing features using a shift-window attention network; constructing a total loss function by combining bidirectional consistency loss, temporal consistency loss, image consistency loss, and smoothing loss; pre-training the optical flow computation convolutional network in an unsupervised manner using the total loss function, and then performing supervised fine-tuning training on a dataset containing real optical flow labels to obtain the final optical flow computation convolutional network. Based on the bidirectional losses for forward and reverse optical flow computation, unsupervised pre-training is achieved, and fine-tuning training is performed on existing real-world scene datasets, enabling the final optical flow computation convolutional network to achieve high-precision and robust event optical flow calculation.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision technology, and in particular to a bidirectional unsupervised event optical flow calculation method and system for event cameras. Background Technology

[0002] Optical flow computation is a core underlying technology in computer vision. Its core objective is to accurately capture the motion trajectory and variation patterns of image pixels. It plays an irreplaceable supporting role in many key application scenarios such as visual odometry, autonomous driving, high dynamic range (HDR) image synthesis, and action recognition. The accuracy and robustness of optical flow estimation directly affect the performance of upper-layer systems. Traditional optical flow computation methods are usually modeled as an optimization problem, solving for energy minimization by considering constraints such as brightness consistency and smoothness between images. In recent years, with the development of deep learning technology, end-to-end optical flow estimation algorithms based on convolutional neural networks have emerged with their powerful feature learning and fitting capabilities. Several efficient optical flow computation networks have been proposed, realizing the direct calculation of optical flow values ​​and significantly improving the accuracy and efficiency of optical flow computation.

[0003] However, the performance of existing optical flow computing technology is heavily dependent on the output quality of imaging devices. Traditional image frame-based cameras have inherent physical limitations: their exposure intervals are usually in the millisecond range, making it difficult to capture instantaneous changes in high-speed moving scenes and easily causing motion blur; at the same time, there is an upper limit to the dynamic range of pixels, and regional oversaturation is prone to occur under harsh lighting conditions such as strong light or backlight. These problems can lead to large errors in optical flow computing results, which seriously restricts the application of optical flow technology in complex real-world scenarios.

[0004] An event camera, as a novel type of visual sensor, works by mimicking the biological visual system, asynchronously outputting pixel-level brightness change events. When the brightness change of a pixel in a scene exceeds a preset threshold, the camera immediately outputs an event containing the pixel's coordinates, a precise timestamp (microsecond level), and the polarity of the brightness change. Compared to traditional cameras, event cameras offer significant advantages such as microsecond-level response latency, high dynamic range (>120dB), and low power consumption, making them particularly suitable for visual tasks in demanding environments such as high-speed motion, strong light, or low light. Although event cameras provide high temporal resolution and high dynamic range input data for optical flow computation, research progress in event optical flow estimation is still far from reaching the maturity level of image optical flow estimation. There are two main technical bottlenecks: First, existing mainstream deep learning optical flow computation networks are designed for structured image frame inputs and cannot directly adapt to the unstructured asynchronous event streams output by event cameras, making it difficult to effectively utilize event information. Second, the commercialization time of event camera-related technologies is relatively short, and there is currently a lack of large-scale, high-quality real-world event datasets, making it difficult to fully train and validate event optical flow computation networks, severely hindering the iteration and implementation of this technology.

[0005] Therefore, traditional event optical flow calculation methods often suffer from low computational accuracy and performance due to their difficulty in adapting to the unstructured asynchronous event stream input of event cameras and the lack of large-scale, high-quality real event optical flow datasets. Summary of the Invention

[0006] Based on this, in order to solve the above-mentioned technical problems, a bidirectional unsupervised event optical flow calculation method and system for event cameras is provided, which can overcome the lack of data and structural limitations, and achieve high-precision, efficient and robust event optical flow calculation.

[0007] A bidirectional unsupervised event optical flow calculation method for an event camera, the method comprising:

[0008] Acquire a discrete event stream output by the event camera, wherein the discrete event stream includes various events, and each event contains pixel coordinates, timestamps, and polarity information;

[0009] The discrete event stream is converted into a forward event body through discrete sampling; the timestamps and polarity information in the discrete event stream are reversed to obtain a reversed event stream, and the reversed event stream is converted into a reversed event body through discrete sampling.

[0010] The forward event volume and the reverse event volume are respectively input into the optical flow calculation convolutional network. Feature extraction is performed through the optical flow calculation convolutional network, and feature enhancement is performed through the shift window attention network. The correlation volume (CV) of the two features is calculated, and the 2D motion vector is parsed from the CV. Then, the optical flow increment is obtained through the GRU unit and the predicted optical flow value is calculated as the forward optical flow and reverse optical flow output.

[0011] The bidirectional consistency loss is calculated based on the forward and reverse optical flows, and the total loss function is constructed by combining the temporal consistency loss, image consistency loss, and smoothing loss.

[0012] The optical flow computation convolutional network is pre-trained in an unsupervised manner using the total loss function, and then supervised fine-tuning is performed on a dataset containing real optical flow labels to obtain the final optical flow computation convolutional network, thereby realizing optical flow computation.

[0013] In one embodiment, the discrete sampling transformation employs a discrete event volume representation method, including:

[0014] For each event in the discrete event stream, it is distributed in the spatiotemporal grid through bilinear interpolation to form a three-dimensional tensor event volume V(x, y, t);

[0015] in, ; ; B is a constant, and... Timestamp discretization Then the event body at any time t is in Bilinear sampling is performed on the sample. This represents the polarity information of the i-th event; represents the bilinear sampling kernel function; x and y represent the x-coordinate and y-coordinate of the target pixel of the event body, respectively; , Let x and y be the pixel coordinates of the i-th original event, respectively; 'a' represents the independent variable of the bilinear sampling kernel function.

[0016] In one embodiment, the timestamps and polarity information in the discrete event stream are reversed to obtain a reversed event stream, including:

[0017] Traverse all events in the discrete event stream, extract the minimum and maximum timestamp values, and record the pixel coordinates of each event;

[0018] The polarity information of each event in the discrete event stream is inverted while keeping the pixel coordinates unchanged. The time span is calculated based on the minimum and maximum timestamps, and the inversion calculation is performed on each timestamp.

[0019] All events that have undergone polarity and timestamp reversal are reordered in ascending order of their reversed timestamps to form a reversed event stream.

[0020] In one embodiment, the forward event body and the reverse event body are respectively input into an optical flow computation convolutional network, feature extraction is performed through the optical flow computation convolutional network, and feature enhancement is performed through a shift window attention network, including:

[0021] The forward event body and the reverse event body are respectively input into the optical flow calculation convolutional network, and feature extraction is performed through the convolutional layer, normalization layer and ReLU activation function in the optical flow calculation convolutional network to obtain the forward basic feature map and the reverse basic feature map.

[0022] The forward base feature map and the backward base feature map are concatenated with the learnable positional encoding at the corresponding scale to supplement spatial positional information; and the forward base feature map and the backward base feature map are divided into non-overlapping local windows according to the window size through a shifted window attention network. For the feature vectors in each window, the association weight is calculated through a multi-head attention mechanism, and window shifting and cross-window interaction are performed to achieve feature fusion to obtain forward enhanced features and backward enhanced features.

[0023] The forward enhancement features and backward enhancement features are respectively input into the GRU network, and iterative processing is performed through the GRU network. The output optical flow is then subjected to interpolation upsampling and median filtering for noise reduction, and finally outputs the forward optical flow, backward optical flow, forward occlusion map, and backward occlusion map.

[0024] In one embodiment, a bidirectional consistency loss is calculated based on the forward and reverse optical flows, and a total loss function is constructed by combining temporal consistency loss, image consistency loss, and smoothing loss, including:

[0025] The total loss function is calculated based on the forward optical flow, reverse optical flow, forward occlusion map, and reverse occlusion map. This total loss function is then used to optimize the optical flow computation convolutional network, achieving unsupervised training. The total loss function is: ;in, For time consistency loss, For image consistency loss, To smooth out the loss, This is a loss due to bidirectional consistency. , , , These are the weighting coefficients for each loss component.

[0026] In one embodiment, when the optical flow computation convolutional network is pre-trained in an unsupervised manner using the total loss function, a random time interval sampling strategy is adopted to select time windows of different lengths from the event stream for training.

[0027] In one embodiment, the pre-training is performed in an unsupervised manner on a synthetic or real-world event dataset;

[0028] The fine-tuning training was conducted in a supervised manner on a dataset with real optical flow labels.

[0029] In one embodiment, the final optical flow computation convolutional network is obtained to realize optical flow computation, including:

[0030] Acquire the event stream output by the event camera for optical flow calculation, and classify the event stream into target forward event volume and target reverse event volume;

[0031] The target forward event volume and the target backward event volume are input into the final optical flow calculation convolutional network, which outputs the target forward optical flow and the target backward optical flow, respectively.

[0032] A bidirectional unsupervised event optical flow calculation system for an event camera, the system comprising:

[0033] The time stream acquisition module is used to acquire the discrete event stream output by the event camera. The discrete event stream includes various events, and each event contains pixel coordinates, timestamps, and polarity information.

[0034] The event body conversion module is used to convert the discrete event stream into a forward event body through discrete sampling; to reverse the timestamp and polarity information in the discrete event stream to obtain a reversed event stream; and to convert the reversed event stream into a reversed event body through discrete sampling.

[0035] The optical flow calculation module is used to input the forward event volume and the reverse event volume into the optical flow calculation convolutional network, extract features through the optical flow calculation convolutional network, enhance features through the shift window attention network, calculate the correlation volume (CV) of the two features, parse the 2D motion vector from the CV, obtain the optical flow increment through the GRU unit and calculate the predicted optical flow value, which is used as the forward optical flow and reverse optical flow output.

[0036] The loss calculation module is used to calculate the bidirectional consistency loss based on the forward optical flow and the reverse optical flow, and to construct the total loss function by combining the temporal consistency loss, image consistency loss and smoothing loss.

[0037] The training module is used to pre-train the optical flow computation convolutional network in an unsupervised manner using the total loss function, and to perform supervised fine-tuning training on a dataset containing real optical flow labels to obtain the final optical flow computation convolutional network and realize optical flow computation.

[0038] The aforementioned bidirectional unsupervised event optical flow calculation method and system for event cameras reverses the timestamps of the event stream and re-inputs them into the optical flow calculation convolutional network for processing. A shifted window attention network is then introduced for feature enhancement, resulting in more focused and targeted extracted information. Based on bidirectional losses for forward and reverse optical flow calculations, different loss functions constrain different training processes, achieving unsupervised training. Simultaneously, this unsupervised training mode is used as a pre-training method, fine-tuned on existing real-world scene datasets to achieve optimal results. This allows the final trained optical flow calculation convolutional network to overcome data scarcity and structural limitations, achieving high-precision, efficient, and robust event optical flow calculation. Attached Figure Description

[0039] Figure 1 This is a diagram illustrating the application environment of a bidirectional unsupervised event optical flow calculation method for an event camera in one embodiment.

[0040] Figure 2 This is a flowchart illustrating a bidirectional unsupervised event optical flow calculation method for an event camera in one embodiment.

[0041] Figure 3 Here is an example of an event count graph and a timestamp graph;

[0042] Figure 4 This is a schematic diagram of the system architecture of an optical flow computation convolutional network in one embodiment;

[0043] Figure 5 This is a schematic diagram of the framework of a shift window attention network in one embodiment;

[0044] Figure 6 This is a graphical representation of the optical flow calculation results on the outdoor_day1 sequence of the MVSEC dataset in one embodiment.

[0045] Figure 7 This is a block diagram of a bidirectional unsupervised event optical flow computing system for an event camera in one embodiment.

[0046] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] The bidirectional unsupervised event optical flow calculation method for event cameras provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown. For example... Figure 1 As shown, the application environment can be Figure 1 The device shown includes a computer device 110, a mobile robot 120, or an edge device 130. When applied to the computer device 110 (PC), it can perform image data acquisition, network training, and analysis; when applied to the mobile robot 120 or the edge device 130, it can be embedded in real-time into visual tasks such as SLAM, motion analysis, and gesture recognition. In this embodiment, taking the application in the computer device 110 as an example, the computer device 110 can acquire a discrete event stream output by an event camera. The discrete event stream includes various events, and each event contains pixel coordinates, timestamps, and polarity information. The computer device 110 can convert the discrete event stream into a forward event body through discrete sampling; it can invert the timestamps and polarity information in the discrete event stream to obtain an inverted event stream, and then convert the inverted event stream into a reverse event body through discrete sampling. The computer device 110 can input the forward event body and the reverse event body into an optical flow computation convolutional network, respectively, and perform feature extraction through the optical flow computation convolutional network, and then pass the feature extraction through a shift window. An attention network is used for feature enhancement, calculating the correlation volume (CV) of two feature pairs, extracting 2D motion vectors from the CVs, and then obtaining optical flow increments through GRU units to calculate predicted optical flow values, which are used as forward and backward optical flow outputs. The computer device 110 can calculate bidirectional consistency loss based on the forward and backward optical flow, and construct a total loss function by combining temporal consistency loss, image consistency loss, and smoothing loss. The computer device 110 can pre-train the optical flow calculation convolutional network in an unsupervised manner using the total loss function, and then perform supervised fine-tuning training on a dataset containing real optical flow labels to obtain the final optical flow calculation convolutional network, thus realizing optical flow calculation. The computer device 110 can be, but is not limited to, various personal computers, laptops, smartphones, robots, unmanned aerial vehicles, tablets, and other devices.

[0049] In one embodiment, such as Figure 2 As shown, a bidirectional unsupervised event optical flow calculation method for an event camera is provided, including the following steps:

[0050] Step 202: Obtain the discrete event stream output by the event camera. The discrete event stream includes various events, and each event contains pixel coordinates, timestamps, and polarity information.

[0051] The event camera tracks the brightness changes of each pixel. When the brightness change of a pixel compared to the last event exceeds a threshold, an event is triggered. The output of the event camera is a discrete list of events, i.e., a discrete event stream, where each event is defined as follows: ;in, These are pixel coordinates. It is the timestamp of the event. This is the event polarity information, represented by 1 or -1.

[0052] The output of the event camera is a discrete event stream, that is... The set of discrete event streams is used, but convolutional networks only accept and process tensors with image-like structures, so it is necessary to integrate and transform the discrete event streams.

[0053] Step 204: Convert the discrete event stream into a forward event body through discrete sampling; reverse the timestamp and polarity information in the discrete event stream to obtain a reversed event stream, and convert the reversed event stream into a reversed event body through discrete sampling.

[0054] In EV-FlowNet, a self-supervised deep learning model designed specifically for event cameras, it is assumed that... and There are grayscale images at different times. and All events in between are counted in a 4-channel image. The first two channels record the number of positive and negative events, respectively, and the last two channels record the timestamps of the last positive and negative events, respectively. Figure 3 As shown, the left side represents the counting channels for events of positive and negative polarity, while the right side represents the channels for event timestamps. EV-FlowNet is used to estimate optical flow from event stream data without requiring actual optical flow labels.

[0055] When there is dense motion or large displacement motion, event counting and timestamp methods inevitably overwrite the original pixels and lose motion information. Therefore, in this embodiment, the event representation method of discrete event volume is adopted.

[0056] In one embodiment, the provided bidirectional unsupervised event optical flow calculation method for an event camera may further include a process of discrete sampling transformation using a discrete event volume representation method. Specifically, this process includes: distributing each event in the discrete event flow across a spatiotemporal grid using bilinear interpolation to form a three-dimensional tensor event volume V(x, y, t); wherein... ; ; B is a constant, and... Timestamp discretization Then the event body at any time t is in Bilinear sampling is performed on the sample. This represents the polarity information of the i-th event; represents the bilinear sampling kernel function; x and y represent the x-coordinate and y-coordinate of the target pixel of the event body, respectively; , Let x and y be the pixel coordinates of the i-th original event, respectively; 'a' represents the independent variable of the bilinear sampling kernel function.

[0057] Compared to event counting and timestamps, the event body V(x, y, t) retains the temporal information of the event and can be conveniently input into artificial neural networks for feature extraction and optical flow calculation.

[0058] In one embodiment, a bidirectional unsupervised event optical flow calculation method for an event camera may further include a reversal process, specifically including: traversing all events in a discrete event stream, extracting the minimum and maximum timestamp values, and recording the pixel coordinates of each event; performing an inversion operation on the polarity information of each event in the discrete event stream while keeping the pixel coordinates unchanged, and calculating the time span based on the minimum and maximum timestamp values, and performing a reversal calculation on each timestamp; and reordering all events that have undergone polarity information reversal and timestamp reversal processing in ascending order of the reversed timestamps to form a reversed event stream.

[0059] Computer devices can convert discrete event streams into forward event volumes through discrete sampling. The timestamps and polarities of the discrete event stream are reversed to obtain the inverted event stream. The inverted event stream is then converted into a reversed event body through discrete sampling. Specifically, when reversing the timestamps and polarities of a discrete event stream, the computer device can reverse the key attributes of the discrete event stream to generate a reverse event stream that is a time mirror and polarity opposite to the forward event stream, providing input data for subsequent bidirectional optical flow calculations and bidirectional consistency loss constraints.

[0060] Computer devices can process the polarity information of each event in a discrete event stream. The core logic of the opposite transformation is to reverse the positive and negative directions of pixel brightness changes. First, the computer device can traverse each event in the discrete event stream and perform an inversion operation on the polarity of the current event. Since the polarity can only take the value 1 or -1, the inversion operation directly achieves the opposite transformation. Next, the computer device can keep the pixel coordinates of each event unchanged, ensuring that the spatial position of the event is completely consistent with the discrete event stream, and only change the brightness change direction attribute.

[0061] Computer devices can use the minimum value of timestamps in a discrete event stream. Maximum value That is, time boundary ( , Using a preprocessed discrete event stream as a baseline, the timestamps of each event are reverse-mapped to restore the chronological order of the discrete event stream. First, the computer calculates the time span based on the preprocessed time boundaries of the discrete event stream. Then, a reverse calculation is performed on the original timestamps of each event, mapping each point on the original timeline to its corresponding position on the reverse timeline. Finally, all reversed events are reordered in ascending order of their reversed timestamps, forming a structured reversed event stream, which is then converted into a reversed event body through discrete sampling.

[0062] Step 206: Input the forward event volume and the reverse event volume into the optical flow calculation convolutional network, respectively. The optical flow calculation convolutional network is used for feature extraction, and the shift window attention network is used for feature enhancement. The correlation volume (CV) of the two features is calculated, and the 2D motion vector is parsed from the CV. The optical flow increment is obtained through the GRU unit and the predicted optical flow value is calculated as the forward optical flow and reverse optical flow output.

[0063] The discrete event flow is first represented as forward and reverse event volumes through discrete sampling. These are then input into an optical flow computation convolutional network for feature extraction, followed by feature enhancement through a shift-window attention network. Finally, iterative updates are performed using a GRU to obtain the forward and reverse optical flows, forward and reverse occlusion maps, allowing for the calculation of network loss. The correlation volume (CV) is one of the core modules of optical flow computation. It is typically calculated pixel-by-pixel multiplication of the feature maps of two images, recording the similarity of all pixels in dimensions [H, W, H, W].

[0064] In one embodiment, a bidirectional unsupervised event optical flow calculation method for an event camera may further include obtaining forward optical flow, backward optical flow, forward occlusion map, and backward occlusion map through an optical flow calculation convolutional network. Specifically, this process includes: inputting the forward event volume and backward event volume into the optical flow calculation convolutional network, respectively; extracting features through convolutional layers, normalization layers, and ReLU activation functions in the optical flow calculation convolutional network to obtain forward basic feature maps and backward basic feature maps; and concatenating the forward basic feature maps and backward basic feature maps with learnable positional codes of corresponding scales to supplement spatial positions. The system sets information and then divides the forward and backward basic feature maps into non-overlapping local windows according to the window size using a shifted window attention network. For the feature vectors within each window, the system calculates the association weights through a multi-head attention mechanism, performs window shifting and cross-window interaction, and achieves feature fusion to obtain forward and backward enhanced features. The forward and backward enhanced features are then input into a GRU network for iterative processing. The output optical flow is then interpolated, upsampled, and denoised using median filtering. Finally, the system outputs forward optical flow, backward optical flow, forward occlusion map, and backward occlusion map.

[0065] The system architecture of optical flow computation convolutional networks is as follows: Figure 4 As shown, it mainly includes: a feature extraction module, used to extract spatiotemporal features from the event body; a feature enhancement module, which uses a shifted window attention mechanism to globally enhance the extracted features; and an optical flow iterative update module, which uses a GRU structure to progressively optimize the optical flow estimation results. The computer device can input the forward and reverse event bodies into the optical flow computation convolutional network, and extract the spatial-temporal basic features of the event bodies through multi-stage convolutional operations. Specifically, the optical flow computation convolutional network adopts a pyramid-shaped multi-scale feature extraction architecture, including convolutional layers, batch normalization (BN) layers, and activation functions (ReLU). By downsampling and feature aggregation processing on the input forward and reverse event bodies, the forward event body outputs a forward basic feature map, and the reverse event body outputs a reverse basic feature map. The basic feature maps contain core information such as the spatial distribution and temporal changes of the events.

[0066] Due to the locality of features in convolutional networks, the extracted features are limited to the vicinity of the extraction point. When occlusion occurs, the effectiveness of the features obtained through convolution is greatly reduced. Therefore, in this embodiment, the computer device can input the event volume into the Swing window attention network to obtain enhanced features. Compared with traditional convolutional networks, enhanced features can extract more global features, while the computational cost increases only linearly with the input size. The formula can be expressed as: ;in, It is a positional encoding. It is a constant factor. These are the enhanced features obtained.

[0067] Specifically, the framework of the shifted window attention network is as follows: Figure 5 As shown, the event body can be analyzed through a shifted window attention network to obtain more global spatiotemporal information, which lays a good foundation for subsequent optical flow processing. When performing global feature enhancement through a shifted window attention network, the computer device can concatenate the forward / backward base feature maps with the corresponding scale location codes to supplement spatial location information, ensuring that the spatial localization of pixels is preserved during attention calculation. Next, the base feature map is divided into non-overlapping local windows. For the feature vectors within each window, a multi-head attention mechanism is used to calculate the correlation weights between vectors, capturing the local correlations of features within the window. Then, the attention calculation results of each window can be non-linearly transformed through an MLP (Multilayer Perceptron) and residually connected with the original window features to output an enhanced feature map. Finally, the enhanced feature maps are fused to output the forward enhanced features and the backward enhanced features.

[0068] Next, as Figure 4 As shown, the computer device can input the forward enhancement features and the backward enhancement features into the gated recurrent unit (GRU) for iterative updates, and decode to obtain the forward optical flow, the backward optical flow, the forward occlusion map, and the backward occlusion map.

[0069] The GRU network consists of multiple iterative steps, each consisting of an update gate, a reset gate, and candidate hidden states. The input is the enhanced feature map and the optical flow prediction result of the previous iteration, and the output is the optical flow prediction map of the current iteration. After multiple iterations, the GRU network outputs the final optical flow prediction result. The forward enhanced features correspond to the output of the forward optical flow, and the backward enhanced features correspond to the output of the backward optical flow. The isolated noise points in the optical flow field are removed by median filtering to obtain the final forward and backward optical flows.

[0070] like Figure 4 As shown, in order to enhance the learning of occlusion during the attention learning process, the occlusion map is used as a location encoding in the learning process. The occlusion map only uses black and white, where white areas represent occlusion. The occlusion map is represented by a value of 1 for occluded locations and 0 for unoccluded locations. This occlusion map contains crucial motion information and is used to calculate bidirectional loss and image consistency loss during network training. However, it is not needed during inference. In this embodiment, the network first outputs forward and backward optical flow, calculates the bidirectional error map, and marks pixels with errors greater than 1.5 times as occluded. This map is then concatenated with the original event volume as input to Swing, requiring no additional parameters and sharing one forward propagation with optical flow estimation.

[0071] Step 208: Calculate the bidirectional consistency loss based on forward and reverse optical flow, and construct the total loss function by combining temporal consistency loss, image consistency loss, and smoothing loss.

[0072] In one embodiment, a bidirectional unsupervised event optical flow calculation method for an event camera may further include a process of constructing a loss function. Specifically, this process includes: calculating a total loss function based on forward optical flow, backward optical flow, forward occlusion map, and backward occlusion map; optimizing the optical flow calculation convolutional network using the total loss function to achieve unsupervised training; the total loss function is: ;in, For time consistency loss, For image consistency loss, To smooth out the loss, This is a loss due to bidirectional consistency. , , , These are the weighting coefficients for each loss component.

[0073] In this embodiment, the loss during network training includes four parts: time consistency loss. Image consistency loss Smoothing loss and bidirectional consistency loss Drawing inspiration from the iterative incremental optical flow calculation of RAFT networks, the optical flow prediction value at each step... The calculated result is an optical flow sequence, and the final loss can be expressed as:

[0074] .

[0075] In this embodiment, the input is an event stream within any selected time period. Unlike traditional methods that use grayscale images as supervision, after each optical flow calculation, the input event stream is warped using the optical flow, and then the warped event stream is sampled. The calculation formula is as follows:

[0076] ; The final time consistency loss can be expressed as the sum of the losses under both polarities, calculated as follows: To avoid proportional issues, optical flow was... Scale amplification will lead to [a problem] during backpropagation. Events from a more distant time are given greater weight, while those from a less distant time are given less weight. Recent events are very likely to be overlooked. To reduce the problems caused by this proportion, we can calculate simultaneously. =0 and The time consistency loss ensures that all events receive equal weight. The time consistency loss can be expressed as: .

[0077] Image consistency loss utilizes the principle that the brightness of pixels remains consistent before and after the forward (backward) warp operation. The loss can be expressed as:

[0078]

[0079] A smoothing loss function is used to constrain the motion smoothness of pixels. The smoothing loss is shown in the following formula. The motion consistency of forward optical flow and backward optical flow is calculated separately, and the loss can be expressed as:

[0080]

[0081] Bidirectional consistency loss Applied to all non-occluded elements, for non-occluded elements, their forward and backward optical flow enable pixels to accurately backtrack to the source point, and the loss can be expressed as:

[0082]

[0083] in, This represents the forward occlusion map. This represents the back occlusion map. This represents the occlusion region in the forward optical flow calculation for any pixel x. This represents the occlusion region in the reverse optical flow calculation for any pixel x.

[0084] Step 210: Pre-train the optical flow computation convolutional network in an unsupervised manner using the total loss function, and then perform supervised fine-tuning training on a dataset containing real optical flow labels to obtain the final optical flow computation convolutional network and realize optical flow computation.

[0085] In one embodiment, a bidirectional unsupervised event optical flow calculation method for an event camera may further include a pre-training process, which specifically includes: when pre-training the optical flow calculation convolutional network in an unsupervised manner using the total loss function, a random time interval sampling strategy is adopted to select time windows of different lengths from the event flow for training.

[0086] In one embodiment, a bidirectional unsupervised event optical flow calculation method for an event camera may further include a fine-tuning training process, specifically including: fine-tuning training is performed in a supervised manner on a dataset with real optical flow labels.

[0087] In one embodiment, a bidirectional unsupervised event optical flow calculation method for an event camera may further include the process of performing optical flow calculation through a final optical flow calculation convolutional network. The specific process includes: acquiring the event stream to be calculated from the event camera and classifying the event stream as a target forward event volume and a target backward event volume; inputting the target forward event volume and the target backward event volume into the final optical flow calculation convolutional network and outputting the target forward optical flow and the target backward optical flow, respectively.

[0088] In one embodiment, a validation experiment was conducted on the bidirectional unsupervised event optical flow calculation method for event cameras. Specifically, the validation experiment was performed on the MVSEC dataset, and training was performed on the outdoor_day sequences, where the outdoor_day1 sequence contains 12,000 images and the outdoor_day2 sequence contains 26,000 images. These images were captured by a device traveling on public roads, and the scenes of outdoor_day1 and outdoor_day2 differ significantly, including straight-line travel, turning, and the presence of moving objects such as other vehicles and pedestrians. The input images were cropped to a size of 256×256.

[0089] The training process consists of two phases: a pre-training phase and a fine-tuning phase. The pre-training strategy is similar to that of EV-FlowNet, training two models in an unsupervised manner: training on the `outdoor_day2` sequence and validating on the `outdoor_day1` sequence; or training on both `outdoor_day1` and `outdoor_day2` and validating on the first two sequences of `indoor_flying`. The fine-tuning phase uses supervised training, employing ground truth values ​​from the dataset on the `outdoor_day2` sequence and validating on `indoor_flying3`.

[0090] To increase the variation in optical flow amplitude during training, time intervals were randomly selected. One picture, and all the ones located here Events between images, in the experiment 2, 4, 6, 8, 10, 12. Additionally, perform enhanced operations such as random horizontal flipping and random cropping.

[0091] Similar to frame optical flow, the evaluation metrics used in this embodiment are mainly AEPE (Average Endpoint Error) and %Outliner (the percentage of pixels with an endpoint error greater than 3 pixels and greater than 5% of the true amplitude). The experiment was conducted using an i9-13900K, NVIDIA GeForce RTX3060, 64GB of RAM, and based on CUDA 11.8, PyTorch 1.13.1, and Ubuntu 25.04.

[0092] For better comparison, representative event-based optical flow computing networks EV-Flownet, Zhu and STE, Spike-FlowNet, and frame-based unsupervised optical flow computing network UFlow were selected. The results on the MVSEC dataset are compared in the table below:

[0093]

[0094] As can be seen from the table, %Outliner increases significantly when dt=4. dt=4 corresponds to a time interval of approximately 90ms, with the motion amplitude increasing by an average of 3.3 times compared to dt=1, and the maximum displacement reaching 35 pixels. This large displacement leads to an increase in occlusion and out-of-bounds pixels, causing the consistency assumption to fail. Furthermore, a longer time window and faster motion result in more event coverage, further reducing the signal-to-noise ratio in areas with small displacements.

[0095] To reduce calculation errors caused by large displacements, random sampling with a probability of 0.5 can be performed during the pre-training phase. In the inference stage, the result of dt=4 is weighted by a confidence level of 0.75; or the sampling frequency is increased in the event body calculation stage to store more event information in video memory.

[0096] The impact of the loss function on the results is shown in the table below:

[0097]

[0098] The table above shows a comparison of quantization under different loss functions. As can be seen from the table, the bidirectional input loss has a significant impact on the final result.

[0099] The results before and after introducing attention are shown in the table below:

[0100]

[0101] The table above shows a comparison of data before and after the introduction of feature enhancement. It can be seen that EPE is significantly improved after introducing attention for feature enhancement. Meanwhile, the impact of different attention subnetworks on the overall network is not significantly different from the impact of channel attention.

[0102] The comparison table of results before and after introducing attention shows that the number of windows W in Swin has a slight impact on the calculation results. The detailed hyperparameters of Swin are shown in the table below:

[0103]

[0104] Figure 6 This is a graphical display of the optical flow calculation results on the outdoor_day1 sequence of the MVSEC dataset. Five images were selected, including both fast and subtle motion scenarios: the first row is the event flow, the second row is the predicted optical flow, the third row is the ground truth optical flow, and the fourth row is the ground truth optical flow after masking blank events. Figure 6 As can be seen from the comparison, the bidirectional unsupervised event optical flow calculation method of the event camera provided in this application can capture the corresponding motion details during both fast movement and subtle motion.

[0105] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0106] In one embodiment, such as Figure 7 As shown, a bidirectional unsupervised event optical flow calculation system for an event camera is provided, including: a time flow acquisition module 710, an event volume conversion module 720, an optical flow calculation module 730, a loss calculation module 740, and a training module 750, wherein:

[0107] The time stream acquisition module 710 is used to acquire the discrete event stream output by the event camera. The discrete event stream includes various events, and each event contains pixel coordinates, timestamps, and polarity information.

[0108] The event body conversion module 720 is used to convert a discrete event stream into a forward event body through discrete sampling; to reverse the timestamp and polarity information in the discrete event stream to obtain a reversed event stream; and to convert the reversed event stream into a reversed event body through discrete sampling.

[0109] The optical flow calculation module 730 is used to input the forward event volume and the reverse event volume into the optical flow calculation convolutional network, extract features through the optical flow calculation convolutional network, enhance features through the shift window attention network, calculate the correlation volume (CV) of the two features, parse the 2D motion vector from the CV, and then obtain the optical flow increment through the GRU unit and calculate the predicted optical flow value as the forward optical flow and reverse optical flow output.

[0110] The loss calculation module 740 is used to calculate the bidirectional consistency loss based on the forward optical flow and the reverse optical flow, and to construct the total loss function by combining the temporal consistency loss, image consistency loss and smoothing loss.

[0111] Training module 750 is used to pre-train the optical flow computation convolutional network in an unsupervised manner using the total loss function, and then perform supervised fine-tuning training on a dataset containing real optical flow labels to obtain the final optical flow computation convolutional network, thereby realizing optical flow computation.

[0112] In one embodiment, the discrete sampling transformation employs a discrete event volume representation method, including: distributing each event in the discrete event stream across a spatiotemporal grid using bilinear interpolation to form a three-dimensional tensor event volume V(x, y, t); wherein... ; ; B is a constant, and... Timestamp discretization Then the event body at any time t is in Bilinear sampling is performed on the sample. This represents the polarity information of the i-th event; represents the bilinear sampling kernel function; x and y represent the x-coordinate and y-coordinate of the target pixel of the event body, respectively; , Let x and y be the pixel coordinates of the i-th original event, respectively; 'a' represents the independent variable of the bilinear sampling kernel function.

[0113] In one embodiment, the event body conversion module 720 is further configured to traverse all events in the discrete event stream, extract the minimum and maximum timestamp values, and record the pixel coordinates of each event; perform an inversion operation on the polarity information of each event in the discrete event stream while keeping the pixel coordinates unchanged, calculate the time span based on the minimum and maximum timestamp values, and perform a reversal calculation on each timestamp; and reorder all events that have undergone polarity information reversal and timestamp reversal processing according to the ascending order of the reversed timestamps to form a reversed event stream.

[0114] In one embodiment, the optical flow calculation module 730 is further configured to input the forward event volume and the reverse event volume into the optical flow calculation convolutional network, respectively, and extract features through the convolutional layers, normalization layers, and ReLU activation functions in the optical flow calculation convolutional network to obtain the forward basic feature map and the reverse basic feature map; concatenate the forward basic feature map and the reverse basic feature map with the learnable positional encoding of the corresponding scale to supplement spatial positional information; and divide the forward basic feature map and the reverse basic feature map into non-overlapping local windows according to the window size through a shifted window attention network, calculate the association weight of the feature vector in each window through a multi-head attention mechanism, perform window shifting and cross-window interaction, realize feature fusion to obtain the forward enhanced feature and the reverse enhanced feature; input the forward enhanced feature and the reverse enhanced feature into the GRU network, respectively, perform iterative processing through the GRU network, and perform interpolation upsampling and median filtering denoising on the output optical flow, finally outputting the forward optical flow, the reverse optical flow, the forward occlusion map, and the reverse occlusion map.

[0115] In one embodiment, the loss calculation module 740 is further configured to calculate a total loss function based on forward optical flow, reverse optical flow, forward occlusion map, and reverse occlusion map, and optimize the optical flow calculation convolutional network through the total loss function to achieve unsupervised training; the total loss function is: ;in, For time consistency loss, For image consistency loss, To smooth out the loss, This is a loss due to bidirectional consistency. , , , These are the weighting coefficients for each loss component.

[0116] In one embodiment, the training module 750 is further configured to employ a random time interval sampling strategy to select time windows of different lengths from the event stream for training when pre-training the optical flow computation convolutional network in an unsupervised manner using the total loss function.

[0117] In one embodiment, the training module 750 is also used to perform supervised training on a dataset with real optical flow labels.

[0118] In one embodiment, the optical flow calculation module 730 is further configured to acquire the event stream to be calculated by the event camera, and to divide the event stream into a target forward event body and a target backward event body; input the target forward event body and the target backward event body into the final optical flow calculation convolutional network, and output the target forward optical flow and the target backward optical flow respectively.

[0119] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a bidirectional unsupervised event optical flow calculation method for an event camera. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0120] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0121] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a bidirectional unsupervised event optical flow calculation method for an event camera.

[0122] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being executed by a processor to implement the steps of a bidirectional unsupervised event optical flow calculation method for an event camera.

[0123] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0124] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A bidirectional unsupervised method for calculating optical flow from an event camera, characterized in that, The method includes: Acquire a discrete event stream output by the event camera, wherein the discrete event stream includes various events, and each event contains pixel coordinates, timestamps, and polarity information; The discrete event stream is converted into a forward event body through discrete sampling; the timestamps and polarity information in the discrete event stream are reversed to obtain a reversed event stream, and the reversed event stream is converted into a reversed event body through discrete sampling. The forward event volume and the reverse event volume are respectively input into the optical flow calculation convolutional network. Feature extraction is performed through the optical flow calculation convolutional network, and feature enhancement is performed through the shift window attention network. The correlation volume (CV) of the two features is calculated, and the 2D motion vector is parsed from the CV. Then, the optical flow increment is obtained through the GRU unit and the predicted optical flow value is calculated as the forward optical flow and reverse optical flow output. The bidirectional consistency loss is calculated based on the forward and reverse optical flows, and the total loss function is constructed by combining the temporal consistency loss, image consistency loss, and smoothing loss. The optical flow computation convolutional network is pre-trained in an unsupervised manner using the total loss function, and then supervised fine-tuning is performed on a dataset containing real optical flow labels to obtain the final optical flow computation convolutional network, thereby realizing optical flow computation.

2. The bidirectional unsupervised event optical flow calculation method for an event camera according to claim 1, characterized in that, The discrete sampling transformation employs a discrete event volume representation method, including: For each event in the discrete event stream, it is distributed in the spatiotemporal grid through bilinear interpolation to form a three-dimensional tensor event volume V(x, y, t); in, ; ; B is a constant, and... Timestamp discretization Then the event body at any time t is in Bilinear sampling is performed on the sample. This represents the polarity information of the i-th event; represents the bilinear sampling kernel function; x and y represent the x-coordinate and y-coordinate of the target pixel of the event body, respectively; , Let x and y be the pixel coordinates of the i-th original event, respectively; 'a' represents the independent variable of the bilinear sampling kernel function.

3. The bidirectional unsupervised event optical flow calculation method for an event camera according to claim 1, characterized in that, The timestamps and polarity information in the discrete event stream are reversed to obtain a reversed event stream, including: Traverse all events in the discrete event stream, extract the minimum and maximum timestamp values, and record the pixel coordinates of each event; The polarity information of each event in the discrete event stream is inverted while keeping the pixel coordinates unchanged. The time span is calculated based on the minimum and maximum timestamps, and the inversion calculation is performed on each timestamp. All events that have undergone polarity and timestamp reversal are reordered in ascending order of their reversed timestamps to form a reversed event stream.

4. The bidirectional unsupervised event optical flow calculation method for an event camera according to claim 1, characterized in that, The forward and reverse event volumes are respectively input into an optical flow computation convolutional network, and features are extracted through the optical flow computation convolutional network and enhanced through a shift window attention network, including: The forward event body and the reverse event body are respectively input into the optical flow calculation convolutional network, and feature extraction is performed through the convolutional layer, normalization layer and ReLU activation function in the optical flow calculation convolutional network to obtain the forward basic feature map and the reverse basic feature map. The forward base feature map and the backward base feature map are concatenated with the learnable positional encoding at the corresponding scale to supplement spatial positional information; and the forward base feature map and the backward base feature map are divided into non-overlapping local windows according to the window size through a shifted window attention network. For the feature vectors in each window, the association weight is calculated through a multi-head attention mechanism, and window shifting and cross-window interaction are performed to achieve feature fusion to obtain forward enhanced features and backward enhanced features. The forward enhancement features and backward enhancement features are respectively input into the GRU network, and iterative processing is performed through the GRU network. The output optical flow is then subjected to interpolation upsampling and median filtering for noise reduction, and finally outputs the forward optical flow, backward optical flow, forward occlusion map, and backward occlusion map.

5. The bidirectional unsupervised event optical flow calculation method for an event camera according to claim 4, characterized in that, Based on the forward and reverse optical flow, the bidirectional consistency loss is calculated, and combined with the temporal consistency loss, image consistency loss, and smoothing loss, a total loss function is constructed, including: The total loss function is calculated based on the forward optical flow, reverse optical flow, forward occlusion map, and reverse occlusion map. This total loss function is then used to optimize the optical flow computation convolutional network, achieving unsupervised training. The total loss function is: ;in, For time consistency loss, For image consistency loss, To smooth out the loss, This is a loss due to bidirectional consistency. , , , These are the weighting coefficients for each loss component.

6. The bidirectional unsupervised event optical flow calculation method for an event camera according to claim 1, characterized in that, When pre-training the optical flow computation convolutional network in an unsupervised manner using the total loss function, a random time interval sampling strategy is adopted to select time windows of different lengths from the event stream for training.

7. The bidirectional unsupervised event optical flow calculation method for an event camera according to claim 1, characterized in that, The pre-training is performed in an unsupervised manner on synthetic or real event datasets; The fine-tuning training was conducted in a supervised manner on a dataset with real optical flow labels.

8. The bidirectional unsupervised event optical flow calculation method for an event camera according to claim 1, characterized in that, The final optical flow computation convolutional network is obtained, and optical flow computation is implemented, including: Acquire the event stream output by the event camera for optical flow calculation, and classify the event stream into target forward event volume and target reverse event volume; The target forward event volume and the target backward event volume are input into the final optical flow calculation convolutional network, which outputs the target forward optical flow and the target backward optical flow, respectively.

9. A bidirectional unsupervised event optical flow calculation system for an event camera, characterized in that, The system includes: The time stream acquisition module is used to acquire the discrete event stream output by the event camera. The discrete event stream includes various events, and each event contains pixel coordinates, timestamps, and polarity information. The event body conversion module is used to convert the discrete event stream into a forward event body through discrete sampling; to reverse the timestamp and polarity information in the discrete event stream to obtain a reversed event stream; and to convert the reversed event stream into a reversed event body through discrete sampling. The optical flow calculation module is used to input the forward event volume and the reverse event volume into the optical flow calculation convolutional network, extract features through the optical flow calculation convolutional network, enhance features through the shift window attention network, calculate the correlation volume (CV) of the two features, parse the 2D motion vector from the CV, obtain the optical flow increment through the GRU unit and calculate the predicted optical flow value, which is used as the forward optical flow and reverse optical flow output. The loss calculation module is used to calculate the bidirectional consistency loss based on the forward optical flow and the reverse optical flow, and to construct the total loss function by combining the temporal consistency loss, image consistency loss and smoothing loss. The training module is used to pre-train the optical flow computation convolutional network in an unsupervised manner using the total loss function, and to perform supervised fine-tuning training on a dataset containing real optical flow labels to obtain the final optical flow computation convolutional network and realize optical flow computation.

Images