An event camera image reconstruction method based on a multi-frame fusion network guided by optical flow

Abstract

An event camera image reconstruction method based on an optical flow-guided multi-frame fusion network belongs to the field of computer vision and image processing technology. Step 1: Acquire event data and convert it into continuous, smooth spatiotemporal voxels using a Gaussian distribution-based full-time-interval voxel encoding scheme. Step 2: Construct an optical flow-guided FMF-Net network. Step 3: Design a comprehensive loss function for training the FMF-Net network. Step 4: Supervised training of the FMF-Net network using a publicly available event dataset. Step 5: Utilize the trained FMF-Net network, input spatiotemporal voxels, and output a high-fidelity reconstructed image. This invention fully utilizes long-term motion cues and temporal dynamic features to achieve high-fidelity image reconstruction under high-speed motion and sparse event input, solving the problems of detail loss and poor consistency in reconstruction under dynamic scenes in existing methods, and significantly improving the structural similarity and visual quality of the reconstructed image.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, and relates to an event camera image reconstruction method based on an optical flow-guided multi-frame fusion network. Background Technology

[0002] With the rapid development of machine vision, autonomous driving, and robotics, higher demands are being placed on the perception capabilities of visual sensors in extreme environments. Traditional frame-based cameras record the absolute brightness of a scene at a fixed frame rate, which easily leads to motion blur in high-speed motion scenes and is limited by dynamic range in scenes with drastic lighting changes, resulting in overexposure or underexposure, making it difficult to meet the stringent requirements of real-time performance and robustness. In contrast, event cameras, as a neuromorphic visual sensor inspired by the biological retina, employ a novel asynchronous perception paradigm. They do not record redundant background information but independently detect brightness changes of each pixel with microsecond-level temporal resolution, outputting an event when the brightness change exceeds a set threshold. This working mechanism endows event cameras with significant advantages such as no motion blur, high dynamic range, low latency, and low power consumption, making them suitable for high-speed and high-dynamic vision tasks.

[0003] However, the data output by event cameras is a sparse, asynchronous stream of events (typically represented as a quadruple of spatiotemporal coordinates, timestamps, and polarity), rather than the dense intensity images relied upon by traditional vision algorithms. While this sparse format is advantageous for certain event-driven algorithms, the vast majority of existing mature computer vision tasks (such as object recognition, object tracking, semantic segmentation, and 3D reconstruction) still heavily rely on textured, structurally complete grayscale or color images. Therefore, reconstructing high-fidelity intensity images from unstructured sparse event streams has become a crucial bridge connecting novel event vision sensors with standard vision processing systems.

[0004] Existing methods for reconstructing event camera images can be mainly divided into three categories: photometric constraint-based methods, integral-based methods, and deep learning-based methods. Details are as follows:

[0005] (1) Photometric constraint-based methods: These methods are usually based on the "constant brightness assumption," which assumes that the brightness of moving points in the scene remains constant over time. Early studies used this assumption to build optimization frameworks to jointly estimate optical flow and intensity maps. However, this strong assumption often fails in real-world scenarios with complex textures and non-uniform lighting changes, resulting in severe artifacts in the reconstructed images and extremely high computational complexity, making it difficult to meet real-time requirements.

[0006] (2) Integration-based methods: These methods treat events as temporal samples of image gradients and recover intensity images by accumulating or integrating events within a time window. For example, manifold regularization can be used for real-time temporal accumulation, or complementary filters can be used to fuse high-frequency event data with low-frequency frame data. However, when faced with high-speed motion, these methods are prone to significant edge blurring and artifacts in the reconstructed image due to motion misalignment within the integration window, making it difficult to retain clear details.

[0007] (3) Deep learning-based methods: In view of the limitations of the above traditional methods, deep learning-based methods use convolutional neural networks (CNNs) to fit the complex nonlinear mapping relationship between event data and intensity images, and have gradually become the mainstream.

[0008] Despite significant advancements in reconstruction quality achieved by existing deep learning-based methods, two serious technical limitations remain when handling complex dynamic scenes, hindering further improvements in the fidelity of reconstructed images:

[0009] (1) Lack of effective utilization of long-term temporal motion cues. Most existing mainstream methods adopt feedforward mechanisms or rely solely on short-term memory, mainly focusing on the local event aggregation at the current moment. They ignore the rich texture information and clear motion trends contained in the historical reconstructed frame sequence. Especially when the input event stream is sparse (such as when the camera is stationary or in a region with simple texture) or when the object undergoes high-speed nonlinear motion, the lack of explicit motion guidance from historical frames makes it difficult for the network to infer the correct texture structure, resulting in low-fidelity issues in the reconstructed image.

[0010] (2) Input data representation suffers from temporal precision loss and aliasing. Deep networks typically require input data to be in regular tensor form, thus necessitating the conversion of continuous asynchronous event streams into discrete voxel grids. The commonly used bilinear interpolation method only linearly assigns the polarity of events to the two most recent time intervals. This approach is essentially a rigid linear approximation, artificially introducing discrete boundaries into the continuous time domain, resulting in temporal quantization errors and preventing the model from reconstructing the image.

[0011] Several deep learning-based event camera image reconstruction methods have been proposed. For example, in the deep learning-based event camera video reconstruction method proposed in Chinese invention patent CN115484410A, although the Transformer module and ConvLSTM are introduced to try to capture long-range spatiotemporal dependencies, it mainly relies on implicit attention mechanisms to mine inter-frame correlations. It lacks explicit alignment and guidance of pixel-level motion trajectories of historical reconstructed frames, which makes it difficult for the network to accurately infer pixel positions in areas with large and rapid object movement or complex textures. Moreover, the high memory usage caused by complex attention calculations limits its real-time application capability on edge devices. In the method for reconstructing latent images based on dynamic visual sensors proposed in Chinese invention patent CN113837938B, although the image quality is improved by fusing information from multiple frames, its data preprocessing stage still uses traditional bilinear interpolation or simple discrete accumulation to construct the input tensor, ignoring the microsecond-level continuous distribution characteristics of events in the time domain. This inevitably introduces temporal quantization errors (aliasing noise) and lacks an adaptive reliability evaluation mechanism for the quality of historical frames. As a result, when the input data is noisy or misaligned, the accumulation of errors will seriously reduce the signal-to-noise ratio and clarity of the reconstructed image. Summary of the Invention

[0012] To address the problem that high-fidelity image reconstruction is impossible in scenarios with high-speed motion, drastic lighting changes, or extremely sparse event streams due to the sparse and asynchronous nature of event camera data and the reliance of existing methods on only short-term local information, this invention proposes an event camera image reconstruction method based on an optical flow-guided multi-frame fusion network. This method constructs a deep neural network (FMF-Net) containing an adaptive guidance module, explicitly capturing pixel-level long-term motion cues between historical reconstruction frames and the current frame using optical flow estimation technology. By introducing motion reliability scoring and an adaptive weighting mechanism, the model's motion alignment capability in complex dynamic scenes is effectively improved, thereby accurately guiding the reconstruction of the current frame using historical temporal information. Simultaneously, to solve the problems of temporal aliasing and quantization errors caused by discrete event data, this invention proposes a Gaussian distribution-based full-time-interval voxel encoding scheme, transforming discrete events into continuous and smooth spatiotemporal voxel representations. Furthermore, a comprehensive objective function incorporating pixel-level L1 loss, structural similarity-based SSIM loss, and VGG perceptual loss is designed for model training. Through collaborative innovation in data representation and network architecture, the model is able to reconstruct high-fidelity images in extremely challenging dynamic scenarios.

[0013] To achieve the above objectives, the technical solution of the present invention is as follows:

[0014] An event camera image reconstruction method based on an optical flow-guided multi-frame fusion network, the event camera image reconstruction method comprising the following steps:

[0015] Step 1: Acquire event data and convert it into continuous and smooth spatiotemporal voxels using a Gaussian distribution-based voxel encoding scheme for the entire time interval. Specifically:

[0016] Step 11: Set the time window Each event data within is represented as follows:

[0017] (1)

[0018] in, This represents the data for each event; The pixel coordinates for each event data; The timestamp of the event data; This represents the polarity of the event data.

[0019] Step 12: Combine consecutive timestamps Normalized to discrete time interval coordinates :

[0020] (2)

[0021] in, The total number of preset time intervals; For this time window The timestamp of the first event data within.

[0022] Step 13: Use a Gaussian kernel function to distribute the polarity of the event data across all time intervals. In the process of constructing spatiotemporal voxels :

[0023] (3)

[0024] in, Represents spacetime voxels; To control the hyperparameters of the time-domain diffusion range, the smoothness of the motion is simulated using a Gaussian distribution, with values ​​ranging from 0.1 to 1.0; subscript Indicates the event index; Indicates the total number of events.

[0025] Step 2: Construct an optical flow-guided multi-frame fusion network, abbreviated as FMF-Net network. Its components include a U-Net backbone module for image reconstruction and an adaptive guidance module that extracts motion cues using historical reconstructed frames. The FMF-Net network operates within the current time window. Generated spacetime voxels As input to the U-Net backbone module, and using the previous three time windows , , Three reconstructed historical frames As input to the adaptive bootstrapping module, where As a reference frame As auxiliary frames, the current time window is eventually reconstructed. Reconstructed image Specifically:

[0026] Step 21: Construct the image reconstruction backbone module based on the U-Net architecture, referred to as the U-Net backbone module. This U-Net backbone module adopts an encoder-decoder structure, consisting of a head feature extraction layer, a two-stage encoder, a two-stage residual bottleneck block, a two-stage upsampling decoder, and an end prediction layer cascaded sequentially. The specific structure and functions are as follows:

[0027] (1) Feature fusion and preliminary extraction stage: First, the current time window is... Generated spacetime voxels The input header feature extraction layer extracts shallow features through two convolutional layers; during this process, external guidance features from the adaptive guidance module are also incorporated. (See step 22 for the generation process) It is fused with shallow features in the channel dimension to introduce historical time series information and generate a hybrid feature map containing motion cues;

[0028] (2) Spatiotemporal feature encoding stage: The hybrid feature map is fed into the two-level encoder in sequence. Each level encoder first extracts spatial texture information through the convolutional layer, and then passes through the Long Short-Term Memory (ConvLSTM) network layer. The memory state is updated by its gating mechanism to capture local temporal dependencies. After the two-level encoding process, the spatial resolution of the hybrid feature map is gradually reduced, generating encoded features rich in spatiotemporal semantics.

[0029] (3) Deep feature extraction stage: The encoded features are fed into the concatenated two-level residual bottleneck block. The encoded features are connected to the convolutional layer and the residual convolutional layer through the path. While preventing gradient vanishing, the feature representation is further deepened and deep features are extracted.

[0030] (4) Multi-scale decoding and recovery stage: Deep features are fed into two-level upsampling decoders. Each level of decoder first recovers the spatial size of the mixed feature map through upsampling, and then performs fine structure recovery processing through convolutional layers. Subsequently, it receives and splices the encoded features of the corresponding encoder level through skip connections to supplement the lost spatial details.

[0031] (5) Image prediction output stage: The decoded mixed feature map is fed into the final prediction layer, and the high-dimensional features are mapped into single-channel gray values ​​through convolution operation. Finally, after processing by Sigmoid operation, the current time window is output. Reconstructed image The reconstructed image is a high-fidelity image.

[0032] Step 22: Build the adaptive bootstrap module, based on the previous three time windows. , , Three reconstructed historical frames As input, and As a reference frame This serves as an auxiliary frame; subsequently, external guiding features generated through motion estimation and reliability scoring will be used. Input the U-Net backbone module to guide the current time window. The reconstruction yields a reconstructed image. Specifically:

[0033] Step 221, Motion estimation: Using the open-source depth optical flow estimation model RAFT, cross-frame motion alignment is performed on the historical reconstructed frames. The motion is calculated for each auxiliary frame. With reference frame The optical flow, among which, The time index of the auxiliary frame is represented by the following process:

[0034] Step 2211, Feature Extraction and Correlation Volume Construction: Extract reference frames respectively With auxiliary frames The feature maps are used to construct a 4D related volume by calculating the electroproduct similarity between all pixel pairs of the two feature maps.

[0035] Step 2212, Iterative update of optical flow: Initialize the optical flow to zero and enter the iterative optimization loop. In each iteration, based on the optical flow estimated in the current iteration, local correlation features are retrieved from the 4D correlated volume through linear sampling, and these correlation features, along with the current optical flow, are input into the update module based on the gated loop unit;

[0036] Step 2213, Residual Prediction and Overlay: The update module outputs the optical flow residual of the current step and adds it to the current optical flow to obtain the updated optical flow estimate;

[0037] Step 2214: The above update and prediction steps are repeated, that is, steps 2212-2213 are repeated until the preset number of iterations is reached, and finally the refined optical flow is output. .

[0038] Step 222, perform motion reliability assessment: Since optical flow estimation may fail in occluded areas or under rapid motion, the FMF-Net network assesses optical flow through the following three dimensions. The quality is quantified in real time to obtain a reliability score, specifically:

[0039] Step 2221, Spatial Smoothness Evaluation: Calculate Optical Flow Spatial gradient Norm, defined as smoothness loss This smoothness loss is used to constrain the spatial continuity of optical flow, preventing abrupt motion from distorting the image.

[0040] Step 2222, Motion sparsity evaluation: Calculate the sum of the optical flow vector magnitudes, defined as the sparsity loss. In regions with significant noise interference, this sparsity loss suppresses unstable overcompensation by penalizing larger displacements.

[0041] Step 2223, Photometric Consistency Reprojection Error: Using the estimated optical flow The auxiliary frame is processed by bilinear interpolation. Perform reverse deformation to obtain the aligned auxiliary frame. Calculate the alignment auxiliary frame and the reference frame. The pixel-level residual between them is defined as photometric loss. As the most crucial indicator for measuring the accuracy of motion estimation.

[0042] Step 223, Adaptive Weight Calculation Based on Inverse Proportional Mapping: The reliability score obtained in Step 222 is converted into weight factors, and a standardized mapping is performed using an inverse proportional function. Specifically:

[0043] First, calculate the auxiliary frame. Reliability rating :

[0044] (4)

[0045] Among them, subscript Indicates the time index of the auxiliary frame; To prevent hyperparameters with a denominator of zero; Indicates auxiliary frame Smoothness loss; Indicates auxiliary frame The sparsity loss; Indicates auxiliary frame Light intensity loss.

[0046] Subsequently, an asymmetric weight allocation strategy is adopted: the reference frame is assigned weights. Fixed weights And auxiliary frames weight , Based on reliability score Dynamic allocation is defined as follows:

[0047] (5)

[0048] in, Indicates assigning a reference frame Fixed weights; Indicates the assignment of auxiliary frames The corresponding weights; Indicates the assignment of auxiliary frames The corresponding weights; Indicates auxiliary frame Reliability score; Indicates auxiliary frame Reliability score.

[0049] Step 224, Feature-level deep fusion: Generating external guiding features through adaptive weights Specifically:

[0050] Step 2241: Extract historical reconstructed frames using a combination of convolution operations, batch normalization, and activation functions. Feature map ;

[0051] Step 2242, use the three adaptive weights calculated in step 223. , , Acting on feature map Obtain external guidance features As shown in formula (6):

[0052] (6)

[0053] Step 2243: Use a combination of convolution operations, batch normalization, and activation functions to guide external features. After smoothing, the input is fed into the head feature extraction layer of the U-Net backbone module to guide the current time window. The reconstruction yields a reconstructed image. .

[0054] Step 3: Design a comprehensive loss function for training the FMF-Net network, including a pixel-level L1 loss function, a structural reliability loss function, and a perceptual style loss function; specifically:

[0055] Step 31: Calculate the L1 loss function to constrain the reconstructed image. with truth value Pixel-level absolute error between:

[0056] (7)

[0057] in, Represents the L1 loss function; Indicates the total number of pixels; Indicates the index of a pixel; Represents the reconstructed image In pixels Pixel value at; Representing truth value In pixels The pixel value at that location.

[0058] Step 32: Calculate the SSIM loss function The local structural consistency of an image is maintained by evaluating brightness, contrast, and structural similarity.

[0059] Step 33: Calculate the perceptual style loss function based on the VGG network to enhance the fine-grained texture of the image by matching the statistical distribution of high-level feature maps:

[0060] (8)

[0061] in, This represents the perceptual style loss function based on the VGG network; This indicates that the VGG network reconstructs images The extracted first Layer feature map; This indicates that the VGG network is based on the true value. The extracted first Layer feature map; This represents the total number of layers in the VGG network, typically ranging from 1 to 5. This indicates a specific layer in a VGG network.

[0062] Step 34: Sum the above three losses to construct the total loss function. This is used to guide the network towards convergence towards higher fidelity.

[0063] (9)

[0064] Step 4: Supervised training of the FMF-Net network using a publicly available event dataset; specifically:

[0065] Step 41: The spatiotemporal voxels obtained by the Gaussian distribution full-time interval voxel encoding scheme Input the FMF-Net network and perform forward propagation to generate a high-fidelity reconstructed image;

[0066] Step 42: Calculate the reconstructed image output by the FMF-Net network using the total loss function defined in Step 3. with truth value The loss between the two is calculated by using the Adam optimizer to update all learnable parameters of the U-Net backbone module and the adaptive guidance module in the FMF-Net network through backpropagation. After several rounds of iteration, the trained FMF-Net network is finally obtained.

[0067] Step 5: Using the trained FMF-Net network, input spatiotemporal voxels and output a high-fidelity reconstructed image.

[0068] The beneficial effects of this invention are as follows:

[0069] (1) This invention can significantly improve the image reconstruction fidelity and structural integrity in high-speed motion and sparse event scenarios; specifically:

[0070] This invention innovatively constructs an adaptive guidance module in the network architecture (step 22). Unlike existing technologies that rely solely on the local short-term event window at the current moment, this invention explicitly establishes a reference frame using optical flow estimation technology. With auxiliary frames The pixel-level correspondence between them. An external guiding feature is generated by warping and aligning the rich textures and sharp edge information contained in the auxiliary frame to the viewpoint of the reference frame. By inputting the U-Net backbone module, the problem of information loss caused by event-triggered sparsity in areas with high-speed object movement or smooth textures is effectively solved. This enables the model to reconstruct images even in extremely dynamic scenes.

[0071] (2) This invention effectively eliminates temporal aliasing and quantization noise in the input data representation stage, ensuring the temporal smoothness of the reconstructed video stream. Specifically:

[0072] This invention proposes a voxel encoding scheme based on Gaussian distribution across all time intervals (step 1). Existing techniques typically use bilinear interpolation to discretize continuous event timestamps, which artificially introduces jagged quantization errors (aliasing). This invention utilizes the smoothing properties of the Gaussian kernel function to diffuse the polarity of each event across all time intervals according to a probability distribution. This nonlinear soft-assignment strategy better reflects the continuous nature of motion in the physical world. This encoding scheme preserves the microsecond-level temporal resolution advantage of event cameras, providing a high-quality data foundation for network-reconstructed images.

[0073] In summary, this invention overcomes the bottleneck in existing event camera image reconstruction technology by deeply collaborating with the FMF-Net network and the Gaussian-distributed voxel encoding scheme across the entire time interval; it provides high-fidelity image data support for the practical application of event cameras in extreme environments such as autonomous driving and high-speed industrial inspection. Attached Figure Description

[0074] Figure 1 This diagram illustrates the implementation steps of the method of the present invention.

[0075] Figure 2 This is a diagram of the Gaussian distribution voxel encoding scheme for the entire time interval proposed in this embodiment of the invention;

[0076] Figure 3 This is a network structure diagram of the optical flow-guided multi-frame fusion network proposed in the embodiments of the present invention. Detailed Implementation

[0077] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0078] This embodiment proposes an event camera image reconstruction method based on an optical flow-guided multi-frame fusion network, such as... Figure 1 As shown, it includes the following steps:

[0079] Step 1: Acquire event data and convert it into continuous and smooth spatiotemporal voxels using a Gaussian distribution-based voxel encoding scheme for the entire time interval. ;

[0080] Step 2: Construct an optical flow-guided multi-frame fusion network, abbreviated as FMF-Net network. Its components include a U-Net backbone module for image reconstruction and an adaptive guidance module that extracts motion cues using historical reconstructed frames. The FMF-Net network operates within the current time window. Generated spacetime voxels As input to the U-Net backbone module, and using the previous three time windows , , Three reconstructed historical frames As input to the adaptive bootstrapping module, where As a reference frame As auxiliary frames, the current time window is eventually reconstructed. Reconstructed image ;

[0081] Step 3: Design a comprehensive loss function for training the FMF-Net network, including a pixel-level L1 loss function, a structural reliability loss function, and a perceptual style loss function;

[0082] Step 4: Supervised training of the FMF-Net network using a publicly available event dataset;

[0083] Step 5: Using the trained FMF-Net network, input spatiotemporal voxels and output a high-fidelity reconstructed image.

[0084] Specifically, step 1 of this embodiment includes the following steps:

[0085] Step 11: Set the time window Each event data within is represented as follows:

[0086] (1)

[0087] in, This represents the data for each event; The pixel coordinates for each event data; The timestamp of the event data; This represents the polarity of the event data.

[0088] Step 12: Combine consecutive timestamps Normalized to discrete time interval coordinates :

[0089] (2)

[0090] in, The total number of preset time intervals is taken as an empirical value of 5; For this time window The timestamp of the first event data within.

[0091] Step 13: As Figure 2 As shown, first set the time window Divided into The window is divided into several time intervals; then, a Gaussian kernel function is used to smoothly distribute the polarity of each event within the window to all time intervals. Finally, the data from each interval are aggregated to construct the spatiotemporal voxel for the current moment. The specific calculation formula is as follows:

[0092] (3)

[0093] in, Represents spacetime voxels; To control the hyperparameters of the time-domain diffusion range, the smoothness of the Gaussian distribution is used to simulate the continuity of motion, with a smaller... This will lead to the generation of spacetime voxels Degenerates into discrete sampling, larger The resulting spacetime voxels Over-smoothing on the time axis causes motion blur in the reconstructed image, so an empirical value of 0.5 is used here; subscript Indicates the event index; Indicates the total number of events.

[0094] Specifically, step 2 in this embodiment is: constructing an optical flow-guided multi-frame fusion network, abbreviated as FMF-Net network, such as... Figure 3 As shown, its components include a U-Net backbone module for image reconstruction and an adaptive guidance module that extracts motion cues using historical reconstruction frames; the FMF-Net network operates within the current time window. Generated spacetime voxels As input to the U-Net backbone module, and using the previous three time windows , , Three reconstructed historical frames As input to the adaptive bootstrapping module, where As a reference frame As auxiliary frames, the current time window is eventually reconstructed. Reconstructed image Specifically:

[0095] Step 21: Construct the image reconstruction backbone module based on the U-Net architecture, referred to as the U-Net backbone module. This U-Net backbone module adopts an encoder-decoder structure, consisting of a head feature extraction layer, a two-stage encoder, a two-stage residual bottleneck block, a two-stage upsampling decoder, and an end prediction layer cascaded sequentially. The specific structure and functions are as follows:

[0096] 1) Feature fusion and preliminary extraction stage: First, the current time window... Generated spacetime voxels The input header feature extraction layer extracts shallow features through two convolutional layers; during this process, external guidance features from the adaptive guidance module are also incorporated. (See step 22 for the generation process) It is fused with shallow features in the channel dimension to introduce historical time series information and generate a hybrid feature map containing motion cues;

[0097] 2) Spatiotemporal feature encoding stage: The hybrid feature map is fed into two levels of encoders in sequence. Each level of encoder first extracts spatial texture information through a convolutional layer, and then passes it through a Long Short-Term Memory (ConvLSTM) network layer. The memory state is updated using its gating mechanism to capture local temporal dependencies. After two levels of encoding, the spatial resolution of the hybrid feature map is gradually reduced, generating encoded features rich in spatiotemporal semantics.

[0098] 3) Deep feature extraction stage: The encoded features are fed into the concatenated two-level residual bottleneck block. The encoded features are connected to the convolutional layer and the residual convolutional layer to further deepen the feature representation while preventing gradient vanishing and extracting deep features.

[0099] 4) Multi-scale decoding and recovery stage: Deep features are fed into a two-stage upsampling decoder. Each stage of the decoder first recovers the spatial size of the mixed feature map through upsampling, and then performs fine structure recovery processing through a convolutional layer. Subsequently, it receives and splices the encoded features of the corresponding encoder level through skip connections to supplement the lost spatial details.

[0100] 5) Image prediction output stage: The decoded mixed feature map is fed into the final prediction layer. A convolution operation maps the high-dimensional features into single-channel grayscale values. Finally, after processing with a Sigmoid operation, the current time window is output. Reconstructed image .

[0101] Step 22: Build the adaptive bootstrap module, with the following structure: Figure 3 As shown. Based on the previous three time windows. , , Three reconstructed historical frames As input, and As a reference frame This serves as an auxiliary frame; subsequently, external guiding features generated through motion estimation and reliability scoring will be used. Input the U-Net backbone module to guide the current time window. The reconstruction yields a reconstructed image. Specifically:

[0102] Step 221, Motion estimation: Using the open-source depth optical flow estimation model RAFT, cross-frame motion alignment is performed on the historical reconstructed frames. The motion is calculated for each auxiliary frame. With reference frame The optical flow, among which, The time index of the auxiliary frame is represented by the following process:

[0103] Step 2211, Feature Extraction and Correlation Volume Construction: Extract reference frames respectively With auxiliary frames The feature maps are used to construct a 4D related volume by calculating the electroproduct similarity between all pixel pairs of the two feature maps.

[0104] Step 2212, Iterative update of optical flow: Initialize the optical flow to zero and enter the iterative optimization loop. In each iteration, based on the optical flow estimated in the current iteration, local correlation features are retrieved from the 4D correlated volume through linear sampling, and these correlation features, along with the current optical flow, are input into the update module based on the gated loop unit;

[0105] Step 2213, Residual Prediction and Overlay: The update module outputs the optical flow residual of the current step and adds it to the current optical flow to obtain the updated optical flow estimate;

[0106] Step 2214: The above update and prediction steps are repeated, that is, steps 2212-2213 are repeated until the preset number of iterations is reached (an empirical value of 12), and finally the refined optical flow is output. .

[0107] Step 222, perform motion reliability assessment: Since optical flow estimation may fail in occluded areas or under rapid motion, the FMF-Net network assesses optical flow through the following three dimensions. The quality is quantified in real time to obtain a reliability score, specifically:

[0108] Step 2221, Spatial Smoothness Evaluation: Calculate Optical Flow Spatial gradient Norm, defined as smoothness loss This smoothness loss is used to constrain the spatial continuity of optical flow, preventing abrupt motion from distorting the image.

[0109] Step 2222, Motion sparsity evaluation: Calculate the sum of the optical flow vector magnitudes, defined as the sparsity loss. In regions with significant noise interference, this sparsity loss suppresses unstable overcompensation by penalizing larger displacements.

[0110] Step 2223, Photometric Consistency Reprojection Error: Using the estimated optical flow The auxiliary frame is processed by bilinear interpolation. Perform reverse deformation to obtain the aligned auxiliary frame. Calculate the alignment auxiliary frame and the reference frame. The pixel-level residual between them is defined as photometric loss. As the most crucial indicator for measuring the accuracy of motion estimation.

[0111] Step 223, Adaptive Weight Calculation Based on Inverse Proportional Mapping: The reliability score obtained in Step 222 is converted into weight factors, and a standardized mapping is performed using an inverse proportional function. Specifically:

[0112] First, calculate the auxiliary frame. Reliability rating :

[0113] (4)

[0114] Among them, subscript Indicates the time index of the auxiliary frame; To prevent hyperparameters with a denominator of zero; Indicates auxiliary frame Smoothness loss; Indicates auxiliary frame The sparsity loss; Indicates auxiliary frame Light intensity loss.

[0115] Subsequently, an asymmetric weight allocation strategy is adopted: the reference frame is assigned weights. Fixed weights (Taking an empirical value of 0.7), while auxiliary frames weight , Based on reliability score Dynamic allocation is defined as follows:

[0116] (5)

[0117] in, Indicates assigning a reference frame Fixed weights; Indicates the assignment of auxiliary frames The corresponding weights; Indicates the assignment of auxiliary frames The corresponding weights; Indicates auxiliary frame Reliability score; Indicates auxiliary frame Reliability score.

[0118] Step 224, Feature-level deep fusion: Generating external guiding features through adaptive weights Specifically:

[0119] Step 2241: Extract historical reconstructed frames using a combination of convolution operations, batch normalization, and activation functions. Feature map ;

[0120] Step 2242, use the three adaptive weights calculated in step 223. , , Acting on feature map Obtain external guidance features As shown in formula (6):

[0121] (6)

[0122] Step 2243: Use a combination of convolution operations, batch normalization, and activation functions to guide external features. After smoothing, the input is fed into the head feature extraction layer of the U-Net backbone module to guide the current time window. The reconstruction yields a reconstructed image. .

[0123] Step 3 of this embodiment: Design a comprehensive loss function for training the FMF-Net network, including a pixel-level L1 loss function, a structural reliability loss function, and a perceptual style loss function; specifically:

[0124] Step 31: Calculate the L1 loss function to constrain the reconstructed image. with truth value Pixel-level absolute error between:

[0125] (7)

[0126] in, Represents the L1 loss function; Indicates the total number of pixels; Indicates the index of a pixel; Represents the reconstructed image In pixels Pixel value at; Representing truth value In pixels The pixel value at that location.

[0127] Step 32: Calculate the SSIM loss function The local structural consistency of an image is maintained by evaluating brightness, contrast, and structural similarity.

[0128] Step 33: Calculate the perceptual style loss function based on the VGG network to enhance the fine-grained texture of the image by matching the statistical distribution of high-level feature maps:

[0129] (8)

[0130] in, This represents the perceptual style loss function based on the VGG network; This indicates that the VGG network reconstructs images The extracted first Layer feature map; This indicates that the VGG network is based on the true value. The extracted first Layer feature map; This represents the total number of layers in the VGG network, with a value of 5. This indicates a specific layer in a VGG network.

[0131] Step 34: Sum the above three losses to construct the total loss function. This is used to guide the network towards convergence towards higher fidelity.

[0132] (9)

[0133] Step 4 of this embodiment: Supervised training of the FMF-Net network using a public event dataset; specifically:

[0134] Step 41: The spatiotemporal voxels obtained by the Gaussian distribution full-time interval voxel encoding scheme Input the FMF-Net network and perform forward propagation to generate a high-fidelity reconstructed image;

[0135] Step 42: Calculate the reconstructed image output by the FMF-Net network using the total loss function defined in Step 3. with truth value The loss between the two is calculated by using the Adam optimizer to update all learnable parameters of the U-Net backbone module and the adaptive guidance module in the FMF-Net network through backpropagation. After several rounds of iteration, the trained FMF-Net network is finally obtained.

[0136] Step 5 of this embodiment: Using the trained FMF-Net network, input spatiotemporal voxels and output a high-fidelity reconstructed image.

[0137] In summary, this invention overcomes the bottleneck in existing event camera image reconstruction technology by deeply collaborating with the FMF-Net network and the Gaussian-distributed voxel encoding scheme across the entire time interval; it provides high-fidelity image data support for the practical application of event cameras in extreme environments such as autonomous driving and high-speed industrial inspection.

Claims

1. An event camera image reconstruction method based on an optical flow-guided multi-frame fusion network, characterized in that, The event camera image reconstruction method includes the following steps: Step 1: Acquire event data and convert it into continuous and smooth spatiotemporal voxels using a Gaussian distribution-based voxel encoding scheme for the entire time interval. Specifically: Step 11: Set the time window Each event data within is represented as follows: （1） in, This represents the data for each event; The pixel coordinates for each event data; The timestamp of the event data; The polarity of the event data; Step 12: Combine consecutive timestamps Normalized to discrete time interval coordinates : （2） in, The total number of preset time intervals; For this time window The timestamp of the first event data within; Step 13: Use a Gaussian kernel function to distribute the polarity of the event data across all time intervals. In the process of constructing spatiotemporal voxels : （3） in, Represents spacetime voxels; To control the hyperparameters of the time-domain diffusion range, the continuity of motion is simulated using the smoothness of a Gaussian distribution; subscript Indicates the event index; Represents the total number of events Step 2: Construct an optical flow-guided multi-frame fusion network, abbreviated as FMF-Net network, which consists of a U-Net backbone module for image reconstruction and an adaptive guidance module that extracts motion cues using historical reconstructed frames. The FMF-Net network uses the current time window Generated spacetime voxels As input to the U-Net backbone module, and using the previous three time windows , , Three reconstructed historical frames As input to the adaptive guidance module, the current time window is ultimately reconstructed. Reconstructed image ; Step 21: Construct an image reconstruction backbone module based on the U-Net architecture, referred to as the U-Net backbone module. The U-Net backbone module adopts an encoder-decoder structure and is composed of a head feature extraction layer, a two-stage encoder, a two-stage residual bottleneck block, a two-stage upsampling decoder, and an end prediction layer cascaded in sequence. Step 22: Build the adaptive bootstrap module, based on the previous three time windows. , , Three reconstructed historical frames As input, and As a reference frame This serves as an auxiliary frame; subsequently, external guiding features generated through motion estimation and reliability scoring will be used. Input the U-Net backbone module to guide the current time window. The reconstruction yields a reconstructed image. ; Step 3: Design a comprehensive loss function for training the FMF-Net network, including a pixel-level L1 loss function, a structural reliability loss function, and a perceptual style loss function; Step 4: Supervised training of the FMF-Net network using a publicly available event dataset; Step 5: Using the trained FMF-Net network, input spatiotemporal voxels and output a high-fidelity reconstructed image.

2. The event camera image reconstruction method based on an optical flow-guided multi-frame fusion network according to claim 1, characterized in that, In step 13 The value range is 0.1-1.

0.

3. The event camera image reconstruction method based on an optical flow-guided multi-frame fusion network according to claim 2, characterized in that, The specific structure and function of step 21 are as follows: Feature fusion and preliminary extraction stage: The current time window... Generated spacetime voxels The input header feature extraction layer extracts shallow features through two convolutional layers; during this process, the external guidance features from the adaptive guidance module are incorporated. It is fused with shallow features in the channel dimension to introduce historical time series information and generate a hybrid feature map containing motion cues; Spatiotemporal feature encoding stage: The hybrid feature map is fed into two-stage encoders in sequence. Each encoder first extracts spatial texture information through a convolutional layer, and then updates the memory state through a long short-term memory network layer. After two-stage encoding, the spatial resolution of the hybrid feature map is gradually reduced, generating encoded features rich in spatiotemporal semantics. Deep feature extraction stage: The encoded features are fed into the concatenated two-level residual bottleneck block. The encoded features are then connected to the convolutional layer and the residual convolutional layer to extract deep features. Multi-scale decoding and recovery stage: Deep features are fed into two-stage upsampling decoders. Each decoder recovers the spatial size of the mixed feature map through upsampling, and then performs fine structure recovery processing through convolutional layers. Finally, it receives and splices the encoded features of the corresponding encoder level through skip connections. Image prediction output stage: The decoded mixed feature map is fed into the final prediction layer, where convolutional operations map the high-dimensional features into single-channel grayscale values. Finally, after processing with a sigmoid operation, the current time window is output. Reconstructed image .

4. The event camera image reconstruction method based on an optical flow-guided multi-frame fusion network according to claim 3, characterized in that, Specifically, step 22 is as follows: Step 221, perform motion estimation: use the open-source depth optical flow estimation model RAFT to perform cross-frame motion alignment on historical reconstructed frames; Calculate auxiliary frames separately With reference frame The optical flow, among which, The time index of the auxiliary frame is represented by the following process: Step 2211, Feature Extraction and Correlation Volume Construction: Extract reference frames respectively With auxiliary frames The feature maps are used to construct a 4D related volume by calculating the electroproduct similarity between all pixel pairs of the two feature maps. Step 2212, Iterative update of optical flow: Initialize the optical flow to zero and enter the iterative optimization loop; In each iteration, based on the optical flow estimated in the current iteration, retrieve local correlation features from the 4D correlated volume through linear sampling, and input the correlation features along with the current optical flow into the update module based on the gated loop unit; Step 2213, Residual Prediction and Overlay: The update module outputs the optical flow residual of the current step and adds it to the current optical flow to obtain the updated optical flow estimate; Step 2214: The above update and prediction steps are repeated, that is, steps 2212-2213 are repeated until the preset number of iterations is reached, and finally the refined optical flow is output. ; Step 222, perform motion reliability assessment: The FMF-Net network assesses optical flow through the following three dimensions. The quality is quantified in real time to obtain a reliability score, specifically: Step 2221, Spatial Smoothness Evaluation: Calculate Optical Flow Spatial gradient Norm, defined as smoothness loss ; Step 2222, Motion sparsity evaluation: Calculate the sum of the optical flow vector magnitudes, defined as the sparsity loss. ; Step 2223, Photometric Consistency Reprojection Error: Using the estimated optical flow The auxiliary frame is processed by bilinear interpolation. Perform reverse deformation to obtain the aligned auxiliary frame. ; Calculate the alignment auxiliary frame and the reference frame The pixel-level residual between them is defined as photometric loss. As the most core indicator for measuring the accuracy of motion estimation; Step 223, Adaptive weight calculation based on inverse proportional mapping: The reliability score obtained in step 222 is converted into weight factors, and a standardized mapping is performed using an inverse proportional function; specifically: First, calculate the auxiliary frame. Reliability rating : （4） Among them, subscript Indicates the time index of the auxiliary frame; To prevent hyperparameters with a denominator of zero; Indicates auxiliary frame Smoothness loss; Indicates auxiliary frame The sparsity loss; Indicates auxiliary frame Light intensity loss; Subsequently, an asymmetric weight allocation strategy is adopted: the reference frame is assigned weights. Fixed weights And auxiliary frames weight , Based on reliability score Dynamic allocation is defined as follows: （5） in, Indicates assigning a reference frame Fixed weights; Indicates the assignment of auxiliary frames The corresponding weights; Indicates the assignment of auxiliary frames The corresponding weights; Indicates auxiliary frame Reliability score; Indicates auxiliary frame Reliability score; Step 224, Feature-level deep fusion: Generating external guiding features through adaptive weights Specifically: Step 2241: Extract historical reconstructed frames using a combination of convolution operations, batch normalization, and activation functions. Feature map ; Step 2242, use the three adaptive weights calculated in step 223. , , Acting on feature map Obtain external guidance features As shown in formula (6): （6） Step 2243: Use a combination of convolution operations, batch normalization, and activation functions to guide external features. After smoothing, the input is fed into the head feature extraction layer of the U-Net backbone module to guide the current time window. The reconstruction yields a reconstructed image. .

5. The event camera image reconstruction method based on an optical flow-guided multi-frame fusion network according to claim 4, characterized in that, Specifically, step 3 is as follows: Step 31: Calculate the L1 loss function to constrain the reconstructed image. with truth value Pixel-level absolute error between: （7） in, Represents the L1 loss function; Indicates the total number of pixels; Indicates the index of a pixel; Represents the reconstructed image In pixels Pixel value at; Representing truth value In pixels Pixel value at; Step 32: Calculate the SSIM loss function Local structural consistency is maintained by evaluating brightness, contrast, and structural similarity. Step 33: Calculate the perceptual style loss function based on the VGG network to enhance the fine-grained texture of the image by matching the statistical distribution of high-level feature maps: （8） in, This represents the perceptual style loss function based on the VGG network; This indicates that the VGG network reconstructs images The extracted first Layer feature map; This indicates that the VGG network is based on the true value. The extracted first Layer feature map; Indicates the total number of layers in the VGG network; This indicates a specific layer in a VGG network. Step 34: Sum the above three losses to construct the total loss function. This is used to guide the network towards convergence towards higher fidelity. （9）。 6. The event camera image reconstruction method based on an optical flow-guided multi-frame fusion network according to claim 5, characterized in that, In step 33, the total number of levels The value range is 1-5.

7. The event camera image reconstruction method based on an optical flow-guided multi-frame fusion network according to claim 6, characterized in that, Step 4 specifically includes: Step 41: The spatiotemporal voxels obtained by the Gaussian distribution full-time interval voxel encoding scheme Input the FMF-Net network and perform forward propagation to generate a high-fidelity reconstructed image; Step 42: Calculate the reconstructed image output by the FMF-Net network using the total loss function defined in Step 3. with truth value The loss between them is used to update all learnable parameters of the U-Net backbone module and the adaptive guidance module in the FMF-Net network through the backpropagation algorithm. After several rounds of iteration, the trained FMF-Net network is obtained.

8. The event camera image reconstruction method based on an optical flow-guided multi-frame fusion network according to claim 7, characterized in that, In step 42, the Adam optimizer is used to update all learnable parameters through the backpropagation algorithm.

Images