A Multimodal Optical Flow Estimation Method Based on Event Cameras

Abstract

This invention discloses a multimodal optical flow estimation method based on event cameras, belonging to the field of optical flow estimation. It uses both event and image modal data as input, leveraging the high temporal resolution, low latency, and high dynamic range of events to improve the performance of optical flow estimation algorithms in low-light and high-speed scenes. By converting the event data stream into event voxels, neural networks can be used for event data processing; through recurrent neural networks and feature residual connections, event features and image features are fused and enhanced; through association layers and a flow-based feature query mechanism, motion information of adjacent frames is encoded; and through iterative refinement of optical flow, the error of the final estimation result is reduced. This invention is applicable to the field of optical flow estimation, fully utilizing both event and image modal data to accurately predict the optical flow between adjacent frames, and improving the robustness of optical flow estimation in extreme scenes such as low light and high speed.

Description

Technical Field

[0001] This invention relates to a multimodal optical flow estimation method based on an event camera, belonging to the field of optical flow estimation. Background Technology

[0002] Optical flow estimation is an important research branch in computer vision and is fundamental to many downstream vision tasks. It has wide applications in autonomous driving, UAV visual navigation, and target tracking. Optical flow is the instantaneous velocity of pixels on the imaging plane of a moving object in space. It is a method for calculating the motion information of an object between adjacent frames. It is generated by the relative motion between the object and the camera and reflects the velocity of the corresponding imaging pixels of the object in a short period of time.

[0003] Event cameras are an emerging type of visual sensor that offers advantages over traditional RGB cameras, including higher temporal resolution, higher dynamic range, and lower latency. These advantages stem from their unique working principle. Unlike RGB cameras, which record light intensity at fixed time intervals within an exposure, event cameras capture changes in light intensity with microsecond-level reaction times. When the light intensity change of a pixel exceeds a threshold, the event camera records an event, and all events are ultimately compiled into a stream of event data as the result. Currently, event cameras are being researched and applied in areas such as object detection, deblurring, and depth estimation, and their application prospects in computer vision are very broad.

[0004] Traditional optical flow estimation methods use RGB image frames as input for computation. In recent years, deep learning-based algorithms have continuously iterated, improving the accuracy of optical flow estimation. However, under extreme conditions such as low light and high motion speed, the imaging quality of RGB images is poor, resulting in information loss. This low-quality imaging significantly reduces the accuracy of optical flow estimation. The limited sampling frequency and dynamic range of RGB cameras restrict the application of optical flow estimation in extreme scenarios. Therefore, exploring the use of emerging visual sensors to complete optical flow estimation tasks in extreme scenarios has become a research approach. Summary of the Invention

[0005] To address the issue that existing optical flow estimation methods perform poorly in extreme scenarios such as low light and high speed, the main objective of this invention is to provide a multimodal optical flow estimation method based on event cameras. This method estimates optical flow in a multimodal manner by using asynchronous, sparse, and high temporal resolution event data, as well as synchronous, dense, and low temporal resolution image data, thereby improving the robustness of optical flow estimation in extreme scenarios such as low light and high speed.

[0006] To achieve the above objectives, the present invention adopts the following technical solution.

[0007] This invention discloses a multimodal optical flow estimation method based on event cameras, which improves the performance of optical flow estimation in extreme scenes by utilizing multimodal information from event data and image data. By converting the event data stream into event voxels, neural networks can be used to process the event data; through recurrent neural networks and feature residual connections, the fusion and enhancement of event features and image features are achieved; through association layers and a flow-based feature query mechanism, motion information of adjacent frames is encoded; and through iterative refinement of the optical flow, the error of the final estimation result is reduced. This invention fully utilizes data from both event and image modalities, accurately predicts the optical flow between adjacent frames, and exhibits strong anti-interference capabilities in low-light and high-speed scenes.

[0008] This invention discloses a multimodal optical flow estimation method based on an event camera, comprising the following steps:

[0009] Step 101: Extract the event data stream E based on the start and end timestamps of adjacent image frames I1 and I2, and convert the event data stream E into event voxels V according to the event voxel conversion relationship.

[0010] The image frames I1 and I2 are H×W×C multi-channel images, where H, W, and C represent the image height, width, and number of channels, respectively. The event data stream E is an N×4 continuous event sequence, where N is the number of events. Each event is a quadruple (x, y, t, p), where t is the timestamp when the event occurs; p∈{1,-1} represents the event polarity, characterizing the increase or decrease in light intensity; 1≤x≤W and 1≤y≤H are the spatial positions of the event on the imaging plane.

[0011] To adapt to neural network computation, after acquiring event input, the event stream in pulse form is first organized into a three-dimensional spatiotemporal distribution matrix in array form. The number of time channels of a voxel is defined as B. The event timestamp t is first normalized to (0, B), and then the event data stream E is aggregated into event voxels V according to spatiotemporal relationships.

[0012]

[0013] Where e k =(x,y,t) k ,p k ), representing k events at coordinates (x, y), t k and p k The timestamp and polarity of the event are respectively, and the final size of the event voxel V is (H, W, B).

[0014] Event data is aligned with image data in both time and space. Specifically, in terms of time, the start and end timestamps of the event stream are aligned with the timestamps of the preceding and following image frames; in terms of space, the event voxels maintain the same resolution as the image, and the spatial positions of the pixels correspond one-to-one.

[0015] Step 102: Extract image features F1 and F2 from image frames I1 and I2 using a feature extraction network, and extract image context features F from image data I1. C Extracting event features from event voxel V and

[0016] The feature extraction networks include, but are not limited to: fully connected networks, convolutional neural networks, residual neural networks, and recurrent neural networks.

[0017] For image data, feature extraction is divided into three paths: two paths use feature extraction networks to extract features from I1 and I2 respectively, for the purpose of extracting motion features between adjacent frames, and finally calculating the initial image features F1 and F2; the other path uses feature extraction networks to extract features from I1, with the purpose of obtaining context features, building the correlation between pixels, and finally calculating the image context feature F. C The process of extracting features using a feature extraction network (Extractor) can be represented as follows:

[0018] F1 = Extractor_1(I1) (2)

[0019] F2 = Extractor_1(I2) (3)

[0020] F C =Extractor_2(I1) (4)

[0021] For event data, feature extraction is divided into three paths: using a feature extraction network (Extractor) to extract event features from V. and The three features were used for enhancement and fusion with image features from different subsequent stages. This process is represented as:

[0022]

[0023]

[0024]

[0025] Step 103: Utilize event features through recurrent neural networks and feature residual connections. Image features F1 and F2 are enhanced and fused to obtain the fused features. Utilizing event characteristics For image context features F C Enhanced feature fusion is performed to obtain the fused features.

[0026] The recurrent neural networks mentioned include, but are not limited to, RNN, LSTM, and GRU.

[0027] Using recurrent neural networks to fuse image features and event features: using image features F1 as the hidden state, and event features... As the input state; image features F2 as the hidden state; event features... As input state; image features F C As a hidden state, event characteristics As input states, the above combination is used as input state x0 and hidden state h0 respectively, and then input into the recurrent neural network for k iterations of fusion to obtain the feature fusion result h. k .

[0028] The characteristic residual connection methods include, but are not limited to, summation connection, channel connection, and weighted connection.

[0029] h based on the feature fusion result of recurrent neural network k The initial image features h0, which serve as the hidden states of the recurrent neural network, are used to enhance the features through residual connections. The aim is to retain more effective information and reduce information loss during multimodal feature fusion. Finally, the enhanced feature fusion result is calculated.

[0030] Therefore, by utilizing a feature enhancement fusion mechanism based on recurrent neural networks and feature residual connections, it is possible to achieve feature-based F1 and... Determine fusion characteristics Based on feature F2 and Determine fusion characteristics Based on feature F C and Determine fusion characteristics

[0031] Step 104: Based on the current optical flow f, use the association layer and local window query mechanism. i (Initially set to 0) for features After processing, the motion features F of adjacent frames are calculated. motion Utilizing event characteristics For motion characteristics F motion Enhanced feature fusion is performed to obtain the fused features.

[0032] The association layer processes the initial features of the image after event feature enhancement and fusion through dot product and downsampling operations. and Four layers of image association features at different scales, C0, C1, C2, and C3, were calculated, where the size of the k-th layer association feature is H×W×H / 2. k ×W / 2 k .

[0033] First, calculate the smallest correlation matrix C0, then input the image features. and To encode the spatial similarity between pixels, take Any pixel position (x1, y1) and The channel data at any pixel position (x2, y2) is processed sequentially by vector dot product to obtain the H×W×H×W image association feature C0.

[0034] Secondly, for the association features in the k≥1 layer, the association feature C0 is downsampled k times by pooling to reduce it to H×W×H / 2. k ×W / 2 k The scale is then used to determine the correlation feature C. k This feature encodes spatial similarity relationships between feature points on a larger scale.

[0035] The local window query mechanism is based on the current optical flow f i (Initially set to 0), retrieve a more accurate and concise association feature map C from the association features C0, C1, C2, and C3. look This reduces the complexity of subsequent calculations.

[0036] For the associated feature C k The query will apply to a region of scale H×W, with coordinates x=(i,j) selected, and the coordinate offset x'=x+f determined based on optical flow. i After (x), x' is determined as the midpoint of the query range. Data within a square region of (2r+1)×(2r+1) is extracted and used as the associated feature of the pixel at coordinate x, where r is the search range. The query operation is performed on all coordinate points within the H×W region, and the calculation formula is as follows:

[0037]

[0038] Where δ satisfies the condition ||δ||1≤r, corresponding to a query region of (2r+1)×(2r+1), and finally a feature map of size H×W×D is generated. Where D = (2r + 1) 2 .

[0039] The multi-layer feature maps are then concatenated to obtain the final feature map C. look The formula is shown below, where the operation [·] is for channel connection.

[0040]

[0041] optical flow f i and feature map C look After channel concatenation, convolution is performed to encode and obtain motion features F. motion :

[0042] F motion =CNNs(Concat(f i C look (10)

[0043] Motion characteristics F motion The information in the middle also comes from optical flow f i and feature map C look This preserves the correct information of the current optical flow and allows for the retrieval of the new feature map C. look New information is obtained to correct errors in optical flow.

[0044] Finally, based on the feature enhancement and fusion mechanism described in step 103, event features are utilized. For image features F motion Perform feature enhancement fusion to obtain the fused features.

[0045] Step 105: Extract motion features from recurrent neural networks and context features Information was extracted and the optical flow change Δf was calculated.

[0046] The recurrent neural networks mentioned include, but are not limited to, RNN, LSTM, and GRU.

[0047] Motion characteristics and context features Channel connections are established, and features are extracted from them using convolution operations as the input state of the recurrent neural network. After calculating and updating the hidden state (initialized to 0), the optical flow change Δf is calculated from the updated hidden state again using convolution operations. This process is represented as:

[0048]

[0049] Step 106: Update the optical flow f based on the optical flow change Δf i The optical flow result f is obtained. i+1 If the iteration number i+1 < I, then jump to step 104 for iteration; if the iteration number i+1 ≥ I, then the optical flow result f i+1 This is the converged estimated optical flow, thus obtaining the optical flow estimation result f between adjacent frames. I .

[0050] The mathematical formula for optical flow update is expressed as:

[0051] f i+1 =f i +Δf (12)

[0052] After I iterations, the optical flow results converge. The number of iterations here is a constant based on experience and set manually. The output result f after the I-th iteration is... I This is the estimated optical flow between adjacent frames.

[0053] Beneficial effects:

[0054] 1. The present invention discloses a multimodal optical flow estimation method based on an event camera, which uses Event events and RGB images to estimate optical flow, and leverages the advantages of multimodal input to predict more accurate dense optical flow results in extreme scenarios such as low light and high speed.

[0055] 2. The present invention discloses a multimodal optical flow estimation method based on an event camera, which utilizes a recurrent neural network and feature residual connections to achieve enhanced fusion of event features and image features. While preserving dense image information, it integrates event information into image features, thereby achieving effective fusion of two modalities.

[0056] 3. This invention discloses a multimodal optical flow estimation method based on an event camera. It fuses multimodal features of images and events in a multi-stage manner, enabling event features to play an effective role in each stage of optical flow estimation and fully utilizing the information provided by event data from multiple perspectives. This improves the robustness of optical flow estimation in extreme scenarios such as low light and high speed.

[0057] 4. The multimodal optical flow estimation method based on event cameras disclosed in this invention fully leverages the advantages of event data, resulting in more accurate and detailed optical flow estimation results, stronger robustness of the model algorithm, applicability to various scenarios, and broadening the application scope of the optical flow estimation field. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the multimodal optical flow estimation method based on an event camera disclosed in this invention.

[0059] Figure 2 This is a system framework diagram of the multimodal optical flow estimation method based on an event camera disclosed in this embodiment;

[0060] Figure 3 This is a comparison of the results of the multimodal optical flow estimation method based on event cameras disclosed in this embodiment in a general scene;

[0061] Figure 4This is a comparison of the results of the multimodal optical flow estimation method based on event cameras disclosed in this embodiment in dark scenes. Detailed Implementation

[0062] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. The technical problems solved by the present invention and its beneficial effects are also described. It should be noted that the described embodiments are only intended to facilitate understanding of the present invention and do not constitute any limitation thereof.

[0063] This embodiment discloses a multimodal optical flow estimation method based on an event camera, such as... Figure 1 As shown, it includes the following steps:

[0064] Step 101: Extract the event data stream E based on the start and end timestamps of adjacent image frames I1 and I2, and convert the event data stream E into event voxels V according to the event voxel conversion relationship.

[0065] The image frames I1 and I2 are H×W×C multi-channel images, where H, W, and C represent the height, width, and number of channels of the image, respectively; the event data stream E is an N×4 continuous event sequence, where N is the number of events, each event is a quadruple (x,y,t,p), where t is the timestamp when the event occurs, p∈{1,-1} is the event polarity, representing the increase and decrease of light intensity, and 1≤x≤W and 1≤y≤H are the spatial positions of the events on the imaging plane.

[0066] To adapt to neural network computation, after acquiring event input, the event stream in pulse form is first organized into a three-dimensional spatiotemporal distribution matrix in array form. The number of time channels of a voxel is defined as B. The event timestamp t is first normalized to (0, B), and then the event data stream E is aggregated into event voxels V according to spatiotemporal relationships.

[0067]

[0068] Where e k =(x,y,t) k ,p k ), representing k events at coordinates (x, y), t k and p k The timestamp and polarity of the event are respectively, and the final size of the event voxel V is (H, W, B).

[0069] Event data is aligned with image data in both time and space. Specifically, in terms of time, the start and end timestamps of the event stream are aligned with the timestamps of the preceding and following image frames; in terms of space, the event voxels maintain the same resolution as the image, and the spatial positions of the pixels correspond one-to-one.

[0070] Step 102: Extract image features F1 and F2 from image frames I1 and I2 using a feature extraction network, and extract image context features F from image data I1. C Extracting event features from event voxel V and

[0071] In this embodiment, a residual neural network ResNet is used for feature extraction. The input data is processed through six residual blocks to generate a feature matrix.

[0072] For image data, feature extraction is divided into three paths: two paths use feature extraction networks to extract features from I1 and I2 respectively, for the purpose of extracting motion features between adjacent frames, and finally calculating the initial image features F1 and F2; the other path uses feature extraction networks to extract features from I1, with the purpose of obtaining context features, building the correlation between pixels, and finally calculating the image context feature F. C The process of extracting features using the ResNet feature extraction network can be represented as follows:

[0073] F1 = ResNet_1(I1) (2)

[0074] F2 = ResNet_1(I2) (3)

[0075] F C =ResNet_2(I1) (4)

[0076] For event data, feature extraction is divided into three paths: using the ResNet feature extraction network to extract event features from V. and The three features were used for enhancement and fusion with image features from different subsequent stages. This process is represented as:

[0077]

[0078]

[0079]

[0080] The input images I1 and I2 are of size H×W×3, the input event V is of size H×W×B, and the output feature size is H×W×D. In this embodiment, B=8 and D=256 are set.

[0081] Step 103: Utilize event features through recurrent neural networks and feature residual connections. Image features F1 and F2 are enhanced and fused to obtain the fused features. Utilizing event characteristics For image context features F C Enhanced feature fusion is performed to obtain the fused features.

[0082] The recurrent neural network for feature fusion uses a gated recurrent unit (GRU).

[0083] Image features and event features are fused using a gated recurrent unit (GRU): image features F1 are used as the hidden state, and event features... As the input state; image features F2 as the hidden state; event features... As input state; image features F C As a hidden state, event characteristics As input states, the above combination is used as input state x0 and hidden state h0 respectively, and then input into the recurrent neural network for k iterations of fusion to obtain the feature fusion result h. k The calculation formula is shown below:

[0084] z t =sigmoid(W z ·[h t-1 ,x t (8)

[0085] r t =sigmoid(W r ·[h t-1 ,x t ]) (9)

[0086]

[0087]

[0088] The feature residual connections are weighted connections, and the weights are calculated using selective kernel units (SKUnits).

[0089] Feature fusion results based on gated recurrent unit (GRU) h k The initial image features h0, which serve as the hidden states of the Gated Recurrent Unit (GRU), are used for weighted feature fusion using Selective Kernel Units (SKUnits). The aim is to retain more effective information and reduce information loss during multimodal feature fusion. The weighting weights W are calculated using the Selective Kernel Units, as shown in the following formula:

[0090] z = Linear C→C / r (P(h0+h k (12)

[0091] W = softmax(Linear) C / r→C×2 (z)) (13)

[0092] Where P is the global average pooling operation, Linear is the fully connected layer, and the weights W are two-channel matrices, which can be represented as W0 and W1 respectively, and satisfy W0+W1=1.

[0093] Finally, the fused features are obtained using weights W0 and W1.

[0094]

[0095] Therefore, the feature of residual connections based on gated recurrent units (GRUs) and selective kernel units (SKUnits) is utilized.

[0096] Enhanced fusion mechanisms can be based on feature F1 and Determine fusion characteristics Based on feature F2 and Determine fusion characteristics Based on feature F C and Determine fusion characteristics

[0097] Step 104: Based on the current optical flow f, use the association layer and local window query mechanism. i (Initially set to 0) for features After processing, the motion features F of adjacent frames are calculated. motion Utilizing event characteristics For image features F motion Enhanced feature fusion is performed to obtain the fused features.

[0098] The association layer processes the initial features of the image after event feature enhancement and fusion through dot product and downsampling operations. and Four layers of image association features at different scales, C0, C1, C2, and C3, were calculated, where the size of the k-th layer association feature is H×W×H / 2. k ×W / 2 k .

[0099] First, calculate the smallest correlation matrix C0, then input the image features. and To encode the spatial similarity between pixels, take Any pixel position (x1, y1) and The channel data at any pixel position (x2, y2) is processed sequentially by vector dot product to obtain the H×W×H×W image association feature C0.

[0100] Secondly, for the association features in the k≥1 layer, the association feature C0 is downsampled k times by pooling to reduce it to H×W×H / 2. k ×W / 2 k The scale is then used to determine the correlation feature C. k This feature encodes spatial similarity relationships between feature points on a larger scale.

[0101] The local window query mechanism is based on the current optical flow f i (Initially set to 0), retrieve a more accurate and concise association feature map C from the association features C0, C1, C2, and C3. look This reduces the complexity of subsequent calculations.

[0102] For the associated feature C k The query will apply to a region of scale H×W, with coordinates x=(i,j) selected, and the coordinate offset x'=x+f determined based on optical flow. i After (x), x' is determined as the midpoint of the query range. Data within a square region of (2r+1)×(2r+1) is extracted and used as the associated feature of the pixel at coordinate x, where r is the search range. The query operation is performed on all coordinate points within the H×W region, and the calculation formula is as follows:

[0103]

[0104] Where δ satisfies the condition ||δ||1≤r, corresponding to a query region of (2r+1)×(2r+1), and finally a feature map of size H×W×D is generated. Where D = (2r + 1) 2 .

[0105] The multi-layer feature maps are then concatenated to obtain the final feature map C. look The formula is shown below, where the operation [·] is for channel connection.

[0106]

[0107] optical flow f i and feature map C look After channel concatenation, convolution is performed to encode and obtain motion features F. motion :

[0108] F motion =CNNs(Concat(f i C look (17)

[0109] Motion characteristics F motion The information in the middle also comes from optical flow f i and feature map C lookThis preserves the correct information of the current optical flow and allows for the retrieval of the new feature map C. look New information is obtained to correct errors in optical flow.

[0110] Finally, based on the feature enhancement and fusion mechanism described in step 103, event features are utilized. For image features F motion Enhanced feature fusion is performed to obtain the fused features.

[0111] Step 105: Extract motion features from recurrent neural networks and context features Information was extracted and the optical flow change Δf was calculated.

[0112] The recurrent neural network used for feature extraction is GRU, which extracts motion features. and context features Channel concatenation is performed, and features are extracted from these features using convolution operations as the input state of the GRU. After calculating and updating the hidden state (initialized to 0), the optical flow change Δf is calculated from the updated hidden state again using convolution operations. This process can be represented as:

[0113]

[0114] Step 106: Update the optical flow f based on the optical flow change Δf i The optical flow result f is obtained. i+1 If the iteration number i+1 < I, then jump to step 104 for iteration; if the iteration number i+1 ≥ I, then the optical flow result f i+1 This is the converged estimated optical flow, thus obtaining the optical flow estimation result f between adjacent frames. I .

[0115] The mathematical formula for optical flow update is expressed as:

[0116] f i+1 =f i +Δf (19)

[0117] After I iterations, the optical flow results converge. The number of iterations here is a constant based on experience and set manually. The output result f after the I-th iteration is... I This is the estimated optical flow between adjacent frames.

[0118] This embodiment discloses a multimodal optical flow estimation system based on an event camera, such as... Figure 2 As shown, it includes the following modules:

[0119] The input module acquires image frames I1 and I2 and event data stream E. The module preprocesses the input data, including data alignment and data normalization. At the same time, according to the event voxel transformation relationship, the pulse-form event stream E is organized into an array-form three-dimensional spatiotemporal distribution matrix V.

[0120] The feature extraction module is used to extract features from the input data, including extracting primary visual features F1 and F2 and contextual features F from the image data. C And extracting event enhancement features from event data. and

[0121] The feature enhancement and fusion module enhances and fuses image features using event features. Based on the GRU recurrent neural network and SKUnit feature residual connections, it integrates event features into image features. This module plays a role at key nodes in a multi-stage manner.

[0122] The association layer utilizes dot product and downsampling operations to construct a multi-level association matrix graph for the input visual feature map. Pixel-by-pixel encoding is performed to generate multi-scale image association feature matrices C0, C1, C2, and C3.

[0123] The feature query module utilizes a local window query mechanism based on the current optical flow f. i Feature map C is extracted from multi-scale correlation features C0, C1, C2, and C3. look Based on the current optical flow state and query features, a new motion feature F is encoded using a convolutional neural network. motion .

[0124] The optical flow iteration module updates the optical flow state through a GRU recurrent neural network, based on motion features. and context features The optical flow is updated in the iteration. After a specified number of iterations, the optical flow is sufficiently refined, and finally an accurate optical flow estimation result is obtained.

[0125] This embodiment will illustrate the effectiveness of the present invention through the results of optical flow estimation.

[0126] 1. Experimental conditions

[0127] The hardware testing conditions for this experiment were: Intel Xeon Gold 6240C, 32GB RAM, NVIDIA GeForce RTX 3090 GPU with 24GB VRAM and CUDA version 11.0; the camera used was a DAVIS 346 event camera. The optical flow iteration count was set to I=12, and the test data consisted of simulated data based on the FlyingChairs-Dark dataset, real data from the MVSEC dataset, and data captured in real dark scenes.

[0128] 2. Experimental Results

[0129] To quantitatively evaluate the quality of optical flow results, mean endpoint error (EPE) and outlier are used as evaluation metrics to measure the accuracy of optical flow estimation results.

[0130] This experiment uses the FlyingChairs-Dark simulation dataset for testing. This dataset is a simulation dataset based on the FlyingChairs optical flow dataset. The specific steps involved in simulating the data distribution under dark conditions are as follows:

[0131] For image data, firstly, the image data is transformed into a data distribution form under dark scenes using formula (20), and then the image noise distribution under dark scenes is simulated using formula (21), finally generating simulated image data under dark scenes.

[0132] I out =A×(I in ) γ (20)

[0133] z(x)~N(μ=x,σ=ax+b) (21)

[0134] For event data, simulated event data in dark scenes is generated using an event simulator based on existing image data. Event simulators include, but are not limited to, ESIM, ICNS, and v2e. During the event simulation generation process, the noise distribution in dark scenes is simulated by adjusting parameters, and finally, simulated event data in dark scenes is generated. In this experiment, v2e is used to simulate and generate event data.

[0135] Based on the above steps, the FlyingChair-Dark optical flow dataset, which contains image and event data in a dark environment, is finally generated using the FlyingChairs optical flow dataset. This dataset is used for subsequent testing.

[0136] To verify the effectiveness of this invention, it was compared with other optical flow estimation methods on the FlyingChairs-Dark simulation dataset. The results are shown in Table 1. To ensure the fairness of the experiment, all methods were retrained on the FlyingChairs-Dark training set.

[0137] Table 1. Comparison of optical flow indices of different methods on the FlyingChairs-Dark test set.

[0138] method enter EPE %Outlier RAFT Image 2.33 10.85 FlowFormer Image 2.05 9.13 E-RAFT Event - - DCEI Image + Event 3.53 20.40 This invention Image + Event 1.90 7.75

[0139] As can be seen from Table 1, the optical flow estimation results of the method disclosed in this invention are significantly better than those of existing methods. Compared with image-based methods, the method disclosed in this invention is superior in accuracy; compared with event-based methods, the method disclosed in this invention can output denser optical flow results and is superior in stability; compared with event-based multimodal methods, the method disclosed in this invention achieves a significant performance improvement and has practical value.

[0140] To verify the performance of this invention in real-world scenarios, comparative tests were conducted on the MVSEC dataset, and the results are shown in Tables 2 and 3.

[0141] Table 2 Comparison of EPE results for different methods on the MVSEC dataset.

[0142] method indoor1 indoor2 indoor3 outdoor1 average EV-FlowNet 0.58 1.02 0.87 0.32 0.70 E-RAFT 0.70 0.94 0.82 0.95 0.85 STE-FlowNet 0.57 0.79 0.72 0.42 0.63 PWC-Net 1.57 3.62 1.55 1.83 1.64 RAFT 0.44 0.54 0.50 0.86 0.59 FlowFormer 0.42 0.50 0.46 0.81 0.55 Fusion-FlowNet 0.62 0.89 0.85 1.02 085 DCEI 0.56 0.64 0.57 0.91 0.67 This invention 0.42 0.51 0.46 0.77 0.54

[0143] Table 3. %Outlier Comparison Results of Different Methods on the MVSEC Dataset

[0144] method indoor1 indoor2 indoor3 outdoor1 average EV-FlowNet 0.0 4.0 3.0 0.0 1.75 E-RAFT 0.16 2.97 1.48 4.55 2.29 STE-FlowNet 0.1 1.6 1.3 0.0 0.75 PWC-Net 3.11 3.29 2.70 11.50 5.15 RAFT 0.13 0.05 0.00 0.21 0.10 FlowFormer 0.13 0.04 0.00 0.12 0.07 Fusion-FlowNet - - - - - DCEI 0.28 0.16 0.12 0.71 0.32 This invention 0.14 0.02 0.00 0.10 0.06

[0145] As can be seen from Tables 2 and 3, on the MVSEC dataset, the method disclosed in this invention outperforms existing optical flow estimation methods on average, proving that the method disclosed in this invention still has good performance in general scenarios.

[0146] Figure 3 The optical flow estimation results of the method disclosed in this invention in a general scenario are shown. Figure 4 The optical flow estimation results of the method disclosed in this invention in dark scenes are shown. It can be seen that the method disclosed in this invention can generate relatively accurate optical flow estimation results in both scenarios, proving that the method disclosed in this invention has strong anti-interference ability in extreme scenarios and can adapt well to low-light and high-speed environments.

[0147] In summary, this example presents a multimodal optical flow estimation method based on event cameras. By converting event data streams into voxels, it successfully introduces event data into the field of deep learning-based optical flow estimation. The method disclosed in this example utilizes a multi-stage feature enhancement and fusion mechanism to process event data, and employs recurrent neural networks and feature residual connections to fuse and enhance event features and image features. This allows the advantages of event data—high temporal resolution, low latency, and high dynamic range—to be fully utilized, enabling the final optical flow estimation method to adapt to low-light, high-speed scenes and broadening the application scope of optical flow estimation algorithms. Therefore, the method disclosed in this invention has very high practical value and important guiding significance.

[0148] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal optical flow estimation method based on an event camera, characterized in that: Includes the following steps, Step 101: Based on adjacent image frames and Extracting event data streams from start and end timestamps Based on the event voxel transformation relationship, the event data stream is... Transformed into event voxels ; Step 102: Extracting features from image frames using a feature extraction network and Extracting image features and From image data Extracting image context features From the event voxels Extracting event features , and ; Step 103: Utilize event features through recurrent neural networks and feature residual connections. Image features , Enhanced feature fusion is performed to obtain the fused features. , ; Utilizing event characteristics Image context features Enhanced feature fusion is performed to obtain the fused features. ; The implementation method for step 103 is as follows: Using recurrent neural networks to fuse image features and event features: combining image features... As a hidden state, event characteristics As input state; Image features As a hidden state, event characteristics As input state; Image features As a hidden state, event characteristics As input states; the above combinations are respectively used as input states. and hidden state Input recurrent neural network The next iteration of fusion yields the result after feature fusion. ; Based on feature fusion results of recurrent neural network and initial image features as hidden states of recurrent neural networks Feature enhancement is achieved by utilizing feature residual connections to retain more effective information and reduce information loss during multimodal feature fusion. The final result of feature fusion enhancement is then calculated. ; Therefore, by utilizing a feature enhancement fusion mechanism based on recurrent neural networks and feature residual connections, it is possible to enhance features... and Determine fusion characteristics Based on features and Determine fusion characteristics Based on features and Determine fusion characteristics ; Step 104: Based on the current optical flow, use the association layer and local window query mechanism. Initially set to 0, for features , After processing, the motion features of adjacent frames are calculated. Utilizing event characteristics Motion characteristics Enhanced feature fusion is performed to obtain the fused features. ; The implementation method for step 104 is as follows: The association layer processes the initial features of the image after event feature enhancement and fusion through dot product and downsampling operations. and Four layers of image association features at different scales were calculated. , , and , of which The size of the layer association feature is ; First, calculate the smallest scale incidence matrix. Input image features and To encode the spatial similarity relationship between pixels, take any pixel position and any pixel position The channel data is processed sequentially by vector dot product to obtain the final result. Image association features ; Secondly, regarding the first Layer association features, for association features conduct Secondary pooling downsampling reduces it to The scale is then used to determine the correlation features. This feature encodes spatial similarity relationships between feature points on a larger scale; The local window query mechanism is based on the current optical flow. Initially set to 0, based on associated features , , and Querying in the middle yields more accurate and concise association feature maps. This reduces the complexity of subsequent calculations; For association features The query will apply to scales of Select the coordinates within the region. Determine coordinate offset based on optical flow After that, it was determined As the midpoint of the query range, the extraction range is The data within the square area is used as coordinates. The associated features of the pixels, where For the search range; Perform a query operation on all coordinate points within the region, using the following formula: in Meet the conditions ,correspond The query range will eventually generate a size of Feature map ,in ; The multi-layer feature maps are then concatenated to obtain the final feature map. The formula is shown below, where the operation... Connect the channels; Optical flow and feature map After concatenating the channels, convolution is performed to encode and obtain motion features. : Motion characteristics The information also originates from optical flow. and feature map This preserves the correct information of the current optical flow while also allowing for the retrieval of new feature maps. New information is obtained to correct errors in optical flow; Finally, based on the feature enhancement and fusion mechanism described in step 103, event features are utilized. Image features Perform feature enhancement fusion to obtain the fused features. ; Step 105: Extract motion features from recurrent neural networks and context features Information was extracted and optical flow changes were calculated. ; Step 106: Based on changes in optical flow Update optical flow The optical flow results were obtained. If the number of iterations If the iteration count is not met, proceed to step 104 for iteration; if the iteration count is not met, proceed to step 104 for iteration. Then the optical flow result This is the converged estimated optical flow, thus obtaining the optical flow estimation results between adjacent frames. .

2. The multimodal optical flow estimation method based on an event camera as described in claim 1, characterized in that: The implementation method for step 101 is as follows: The image frame and for Multi-channel images, in which , , These represent the image's height, width, and number of channels, respectively; the event data stream... for A sequence of consecutive events, where The number of events; each event is a quadruple. ,in It is the timestamp when the event occurred; The polarity of the event represents the increase and decrease in light intensity; and It is the spatial location of the event on the imaging plane; To adapt to neural network computation, after acquiring event input, the event stream in pulse form is first organized into a three-dimensional spatiotemporal distribution matrix in array form, and the number of time channels of the voxel is specified as follows: First, set the event timestamp. Normalization to Then the event data stream Aggregates into event voxels according to spatiotemporal relationships. : , in Characterizing coordinates place One event, and The event voxels are generated by assigning the event's timestamp and polarity. The size is ; Event data is aligned with image data in time and space. Specifically, in time, the start and end timestamps of the event stream are aligned with the timestamps of the preceding and following image frames. In space, the event voxels maintain the same resolution as the image, and the spatial positions of the pixels correspond one-to-one.

3. The multimodal optical flow estimation method based on an event camera as described in claim 2, characterized in that: The implementation method for step 102 is as follows: For image data, feature extraction is divided into three paths: two of these paths utilize feature extraction networks to extract features from... and Features are extracted for use in the subsequent extraction of motion features between adjacent frames, ultimately calculating the initial-order features of the image. and Another approach utilizes a feature extraction network from... The purpose of extracting features is to obtain contextual features, build relationships between pixels, and ultimately calculate the image contextual features. ; For event data, feature extraction is divided into three paths: using a feature extraction network from... Extracting event features , and The three features were used to enhance and fuse with image features from different subsequent stages.

4. The multimodal optical flow estimation method based on an event camera as described in claim 3, characterized in that: The implementation method for step 105 is as follows: Motion characteristics and context features Channel connections are established, and features are extracted from them using convolution operations as the input state of the recurrent neural network. After calculating and updating the hidden state, the optical flow change is calculated from the updated hidden state again using convolution operations. .

5. The multimodal optical flow estimation method based on an event camera as described in claim 4, characterized in that: The implementation method for step 106 is as follows: The mathematical formula for optical flow update is expressed as follows: Iteration After this, the optical flow results converge, and the number of iterations at this point is [number missing]. Human-defined, experience-based constants, the first Output result after the second iteration This is the estimated optical flow between adjacent frames.

Images