Event camera video reconstruction method and system based on active aperture modulation

Abstract

The invention discloses an event camera video reconstruction method and system based on active aperture modulation, and the method comprises the steps: introducing an aperture modulation strategy for the first time, and reconstructing an initial frame with good quality by periodically adjusting the opening and closing of an aperture and actively triggering a dense global event signal; the problem of low background reconstruction quality caused by sparse events in a static region in the prior art is solved, and good prior information is provided for subsequent dynamic scene reconstruction. By constructing a forward-reverse bidirectional network, rich intensity information is provided for a static scene, and the problems of background disappearance and error accumulation in long-time operation in a traditional method are effectively solved. And meanwhile, high-time-resolution capture of a dynamic region is kept, and high-fidelity and high-dynamic-range video reconstruction is realized.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for video reconstruction from an event camera based on active aperture modulation. Background Technology

[0002] An event camera is a bio-inspired visual sensor. Unlike traditional cameras, it doesn't capture a global image at a fixed frame rate. Instead, it asynchronously records pixel-level logarithmic changes in brightness within a scene. Each pixel operates independently, generating an event containing a timestamp, spatial location, and polarity (indicating an increase or decrease in brightness) when a local brightness change exceeds a preset threshold. This approach endows event cameras with microsecond-level temporal resolution, high dynamic range (typically exceeding 120dB), and low latency, making them uniquely advantageous in high-speed motion capture, extreme lighting conditions, and low-power scenarios. Therefore, event cameras have broad application prospects in cutting-edge fields such as robot navigation, autonomous driving, and high-speed target tracking.

[0003] To bridge the gap between event data and traditional vision algorithms, event-to-video reconstruction technology has become a research hotspot in recent years. This technology aims to convert asynchronous event streams into continuous video frame sequences, making event camera data compatible with existing computer vision algorithm frameworks. Currently, most mainstream methods employ recurrent neural network architectures, predicting video frames by accumulating event information over time. However, these methods face a fundamental challenge: the event triggering mechanism dictates that event signals are only generated when the scene changes, resulting in extremely sparse events in static or low-motion areas. This sparsity makes the reconstruction process highly ill-conditioned, especially during long-term operation, where small prediction errors accumulate, ultimately leading to a severe deterioration in the reconstruction quality of static background areas, manifesting as structural blurring, loss of detail, or drift artifacts.

[0004] To address this issue, existing solutions can be broadly categorized into three types: The first type attempts to introduce an additional traditional camera as auxiliary input, enhancing reconstruction quality by fusing data from two modalities. However, this approach not only increases system complexity and cost but also faces challenges in cross-modal data alignment. The second type utilizes active illumination modulation technology to trigger global events in a short period to acquire dense scene information. However, this method is difficult to implement outdoors or in uncontrolled environments, limiting its application scope. The third type attempts to model noise events in a static background to enhance the perception of static scenes. Although this type of method introduces separation of dynamic and static events and two neural network branches, its static scene reconstruction heavily relies on noise events in the scene. In complex and ever-changing real-world environments, noise events often lack distinctiveness, resulting in limited reconstruction effectiveness.

[0005] As a fundamental component of imaging systems, the aperture controls the intensity of light received by the sensor by adjusting the amount of light entering the camera. This physical characteristic offers a new approach to solving the problem of static scene reconstruction in event cameras, but no research has yet systematically explored the application potential of aperture modulation in event video reconstruction. In particular, how to design efficient aperture modulation strategies and how to fuse aperture-modulated events with motion-triggered events remain key issues that urgently need to be addressed. Summary of the Invention

[0006] This invention addresses the shortcomings of existing technologies by introducing an aperture modulation strategy for the first time. It proposes a method and system for video reconstruction from an event camera based on active aperture modulation. By periodically adjusting the aperture opening and closing, it actively triggers dense global event signals, fundamentally solving the problem of sparse events in static areas while maintaining high temporal resolution capture of dynamic areas. This achieves unified high-fidelity and high dynamic range reconstruction of both static backgrounds and dynamic foregrounds. To achieve the above objectives, this invention provides the following technical solution:

[0007] In a first aspect, the present invention proposes an event-based video reconstruction method (AE2VID) based on active aperture modulation, comprising the following steps:

[0008] S1, Aperture Period Modulation:

[0009] Control the aperture of the event camera in advance according to a preset cycle. Perform periodic modulation of opening the aperture, holding it open, and then closing it; where opening and closing the aperture are symmetrical processes, each requiring a time of... The duration for which the aperture remains open is ;

[0010] S2, Event Stream Segmentation:

[0011] With the aperture completely closed at consecutive time points and Divide the event stream into time windows as boundaries. Furthermore, the aperture modulation trigger event during the aperture opening phase was segmented. Motion-triggered events during the aperture-open phase ;

[0012] S3, Endpoint Frame Estimation:

[0013] The aperture modulation trigger event Input the aperture modulation event subnetwork to estimate the endpoint frame reference intensity for each time window. and hidden state ;

[0014] S4. Dynamic Scene Reconstruction:

[0015] The motion trigger event The endpoint frame reference strength and hidden state Input the motion-triggered event subnetwork to reconstruct a series of high frame rate image frames within the time window.

[0016] Furthermore, in step S2, after segmenting the motion trigger event... Then, the corresponding time interval is determined based on the target frame rate. Cut evenly into It calculates the event voxels of the event flow within each sub-interval.

[0017] Further, in step S3, the aperture modulation event sub-network includes:

[0018] The intensity reconstruction module based on the first positive event is used to... The events are counted pixel by pixel, recording the timestamp of the first positive polarity event triggered by each pixel during the aperture opening process. Construct the time matrix and estimate the steady-state irradiance of the scene based on the following physical model. This yields the initial intensity image;

[0019]

[0020] in, For the event camera threshold, Dark current, For an aperture-modulated optical system, is the transmittance function.

[0021] The image denoising module, connected to the intensity reconstruction module based on the first positive event, performs end-to-end denoising and detail restoration on the initial intensity image, and outputs the denoised endpoint frame reference intensity. ;

[0022] The hidden state generation module, connected to the image denoising module, is used to... The processed input is a convolutional network with the same forward branch structure as the dynamic scene reconstruction module, generating hidden states compatible with the dynamic scene reconstruction module. .

[0023] Furthermore, the image denoising module employs a SwinIR neural network based on convolution and self-attention mechanisms.

[0024] Furthermore, the motion-triggered event subnetwork adopts a forward-reverse bidirectional reconstruction structure and includes a pixel-level fusion module, specifically used for:

[0025] Forward branch, in hidden state In the initial state, the motion-triggered event voxel sequence is processed in chronological order to recursively reconstruct all positive candidate frames within the time window;

[0026] Backward branch, in hidden state Starting from the initial state, the motion-triggered event voxel sequence is processed in reverse chronological order to recursively reconstruct all reverse candidate frames within the time window.

[0027] Through a pixel-level fusion module, for each reconstruction time point, the corresponding forward candidate frame, reverse candidate frame, and reference strength of the preceding and following endpoint frames are combined. and The corresponding event voxel grid is input into a lightweight convolutional network to generate a pixel-level normalized fusion weight map, which is then used to fuse the four frames to obtain the final reconstructed frame.

[0028] Secondly, the present invention also provides an event camera video reconstruction system based on active aperture modulation, comprising the following modules to implement the method described in any of the above:

[0029] An aperture modulation controller, located at the front of the event camera, is used to adjust the aperture according to a preset period. Perform a periodic modulation process of opening the aperture, holding it open, and then closing it, where the opening and closing of the aperture are symmetrical processes, and the required time is... The duration for which the aperture remains open is ;

[0030] Event stream segmentation module, used to segment the event stream at consecutive aperture full-closing time points. and Divide the event stream into time windows as boundaries. For each time window, the event stream is further divided into: aperture opening phase, based on the aperture state. Internally triggered events As an aperture modulation trigger event; aperture holding open phase Events triggered by internal factors such as scene motion or changes in light intensity , as a motion trigger event;

[0031] The endpoint frame estimation module, connected to the event stream segmentation module, is used to receive the aperture modulation trigger event. The endpoint frame reference intensity for each time window is estimated using an aperture modulation event subnetwork. and hidden state ;

[0032] The dynamic scene reconstruction module, connecting the event stream segmentation module and the endpoint frame estimation module, is used to receive the motion-triggered events. The endpoint frame reference strength and hidden state A series of high frame rate image frames within a time window are reconstructed through a motion-triggered event subnetwork.

[0033] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, performs the steps of the method as described in any of the preceding claims.

[0034] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method as described in any of the preceding claims.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0036] 1. This invention provides an event camera video reconstruction method based on active aperture modulation. It introduces an aperture modulation strategy for the first time, actively triggering dense global event signals by periodically adjusting the aperture opening and closing, thus reconstructing a high-quality initial frame. This solves the problem of poor background reconstruction quality in static areas caused by sparse events in existing methods, and also provides good prior information for subsequent dynamic scene reconstruction. By constructing a forward-backward bidirectional network, rich intensity information is provided for static scenes, effectively overcoming the problems of background disappearance and error accumulation during long-term operation in traditional methods.

[0037] 2. This invention designs a unified AE2VID framework, comprising two sub-networks: AENet, which handles aperture modulation triggered events, and MENet, which handles motion triggered events. This enables efficient fusion and collaborative processing of two different types of event data. This dual-path architecture can simultaneously handle the detailed reconstruction of static backgrounds and the high temporal resolution capture of dynamic targets, solving the technical problem of low quality often encountered in existing methods when processing dynamic-static regions under the same framework.

[0038] 3. This invention constructs a real-world aperture modulation and motion-triggered event dataset (AMED). This dataset is collected using effective aperture control parameters (including aperture diameter, aperture speed, and modulation interval) and covers a variety of scenarios. This dataset not only verifies the effectiveness of this method in real-world environments but also provides a benchmark for subsequent research. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0040] Figure 1 This is a flowchart of an event camera video reconstruction method based on active aperture modulation, provided as an embodiment of the present invention.

[0041] Figure 2 The system architecture diagram and effect comparison diagram provided for the embodiments of the present invention are as follows: (a) is a physical diagram of the event camera system with adjustable aperture built according to the present invention; (b) is the event stream of the existing method without aperture modulation; (c) is the event stream of the method of the present invention with aperture modulation; (d) is the comparison result of the method of the present invention with other methods. Detailed Implementation

[0042] To better understand this technical solution, the method of the present invention will be described in detail below with reference to the accompanying drawings.

[0043] This invention provides a method for reconstructing event camera video based on active aperture modulation, such as... Figure 1 As shown, the overall technical solution mainly consists of four steps: aperture period modulation, event stream segmentation, endpoint frame estimation, and dynamic scene reconstruction, as detailed below:

[0044] S1, Aperture Periodic Modulation: A motorized lens is configured at the front of the event camera as an aperture modulation controller, which operates according to a preset period. The process involves a cyclical sequence of "opening the aperture—keeping it open—closing the aperture," where opening and closing the aperture are symmetrical. The time required for this sequence is denoted as... Thus, the aperture remains "open" continuously. According to experimental calibration, the time required for aperture change is much shorter than the time required for aperture to remain open.

[0045] S2, Event Stream Segmentation: During the shooting process, we segment the event stream according to continuous time windows. The event stream is segmented, with each time window corresponding to a complete process of "opening the aperture - keeping it open - closing the aperture". and All of these are the moments when the aperture is completely closed. Then, based on the aperture state and the time of state change, we further divide the event flow into three segments:

[0046] Aperture Opening Stage The triggered event is denoted as Used for endpoint frame estimation;

[0047] During aperture stays open Events triggered by the movement of objects within the scene or changes in ambient light are denoted as... Used for dynamic scene reconstruction;

[0048] Aperture closed phase The triggered event is denoted as Experiments show that it has a lot of noise, which is not conducive to estimating endpoint frames and is difficult to provide motion spatiotemporal information. Therefore, in the present invention, it is discarded and only aperture modulation trigger events and motion trigger events are retained to participate in subsequent algorithms.

[0049] For each time window, we will trigger the aperture modulation event. The endpoint frame intensity information is directly fed into AENet for estimation, without further segmentation. For motion-triggered events... Based on the target frame rate, the time interval corresponding to the aperture-open phase is determined. Cut evenly into The system calculates the event voxels of each event stream as input to the dynamic scene reconstruction module.

[0050] S3. Endpoint Frame Estimation: The endpoint frame estimation step utilizes the Aperture Modulation Event Subnetwork (AENet). The AENet architecture diagram is shown below. Figure 1 (b). Specifically, AENet consists of three modules: an intensity reconstruction module (FIR) based on the first positive event, an image denoising module (IDN), and a hidden state generation module (HSG). The specific algorithm is as follows:

[0051] FIR module: First, for The events are statistically analyzed according to pixel coordinates, recording the timestamp of the first positive polarity event triggered by each pixel during the aperture opening process. A time matrix is ​​constructed. Based on the physical model of event triggering, this invention establishes a time matrix and a relationship between the steady-state irradiance of the scene. Relationship:

[0052]

[0053] in, For the event camera threshold, For dark current term, Let be the transmittance function of the aperture-modulated optical system. Therefore, the technique of this invention can estimate the intensity distribution of the current scene when the aperture is fully open using the time matrix.

[0054] IDN Module: Considering the noise introduced by time matrix statistics and physical model approximation, this invention introduces a neural network to perform end-to-end denoising and detail restoration on the initial image estimate, based on the output of the FIR module. This invention employs a SwinIR neural network based on convolution and self-attention mechanisms to perform multi-scale feature extraction and reconstruction of the input image. The network output ensures consistency with the original spatial resolution of the event camera, ultimately obtaining the denoised endpoint frame reference strength. .

[0055] HSG Module: To provide MENet with an initial hidden state compatible with its architecture over time, this invention introduces the HSG module. The HSG module references the endpoint frame strength. Replication in the channel dimension Next, frame voxels with similar voxel structures were obtained. It is then fed into a convolutional network with the same structure as the MENet forward branch, and the hidden states corresponding to the time window are output. It can also output an additional pseudo-reconstructed frame. This serves as an auxiliary supervision signal during training. Over the entire time series, AENet performs training on each window... The above processing is performed independently to obtain the reference frame sequence and the hidden state sequence, which are used as the prior inputs of MENet.

[0056] S4. Dynamic Scene Reconstruction: The dynamic scene reconstruction step utilizes a motion-triggered event subnetwork (MENet). The core steps include three parts: forward reference frame reconstruction, backward reference frame reconstruction, and pixel-level fusion. The MENet structure diagram is shown below. Figure 2 (b). This invention utilizes a forward-backward bidirectional reconstruction structure to fully leverage temporal reference information, effectively mitigating the error accumulation problem in recurrent neural networks. The specific algorithm is as follows:

[0057] Forward reference frame reconstruction: The sequence of motion event voxels in each time window is input into the forward branch of MENet in chronological order, and the forward branch is in a hidden state. As the initial hidden state, spatiotemporal features are extracted using convolutional LSTM recursive units, and for each time window... Output positive candidate frames and updated hidden states, and then reconstruct all positive candidate frames within the time window in a loop.

[0058] Reverse reference frame reconstruction: To fully utilize temporal information, this invention performs time-domain inversion on motion event voxels and uses... As the initial hidden state, it drives the reverse branch of MENet to perform reverse reconstruction, obtaining all reverse candidate frames within the reconstruction time window.

[0059] Pixel-level fusion: This invention designs a pixel-level fusion module. For each time window, the forward candidate frame, the reverse candidate frame, the two endpoint reference frames, and the event voxel grid corresponding to the time window are input into a lightweight convolutional network. The network outputs a mixed weight map of four channels. Finally, the four frames are fused according to the pixel-level normalized weights of the four channels, so as to ensure the stability of the static background while obtaining good motion details and temporal consistency.

[0060] The method of this invention is trained on a synthetic dataset, and the specific training process is as follows:

[0061] Synthetic training data:

[0062] The rendering process includes scenes with various geometric objects. To enhance data diversity, both the scene background and the geometric objects use textures from the MS-COCO dataset. This invention utilizes 1000 existing 2-second scene datasets, where the motion in each scene is a global random motion of the camera. Additionally, this invention uses Blender software to render 500 1-second scene datasets, each with a fixed background and random motion added only to the foreground, thus ensuring the motion diversity of the training data.

[0063] Using the ESIM event simulator to simulate events, this invention randomly samples thresholds in a Gaussian distribution to simulate threshold noise in real-world event cameras.

[0064] Data augmentation was applied to the synthetic data, the image resolution was adjusted, and random 128×128 pixel image patches were cropped and randomly rotated. ) and horizontal / vertical flipping to enhance data diversity.

[0065] Loss function design:

[0066] Hidden State Supervision Loss: To constrain the alignment of the features output by the hidden state generation module HSG with the intensity information recovered by AENet, this invention uses pseudo-frames output by HSG... Denoising reference frames generated by AENet Apply between loss: This loss encourages HSG to learn representations consistent with the reference image in the feature space, thereby providing a stable initial hidden state for subsequent temporal reconstruction.

[0067] Reconstruction quality loss: For video sequences output by MENet, this invention employs... A combination of loss and perceptual loss. Specifically, for each frame... ,definition ,in For LPIPS perceptual loss, it can be used to measure the consistency between high-level semantics and texture details, thereby improving the performance of reconstructed video in terms of subjective visual quality.

[0068] Temporal consistency loss: To suppress flickering and artifacts in the reconstructed video over time, this invention further introduces temporal consistency loss. This is used to constrain the differences between adjacent frames after motion compensation. Considering that applying temporal consistency loss to all frames may introduce "dirty window" artifacts, this invention only applies it to the latter half of each segment's frames (…). This loss is applied to balance smoothness with the accuracy of detail reconstruction.

[0069] Overall Loss Function: Combining the above loss function terms, this invention ultimately adopts the following target loss function. Supervision will be conducted. Among these... This represents the total number of reconstructed frames. For indexes where time consistency penalty is first applied, This refers to the weighting coefficient for time consistency loss. This invention selects... , as well as As a parameter, it can maintain a high level of detail fidelity while ensuring timing smoothness.

[0070] Training Strategy and Optimization Process: This invention is implemented using the deep learning framework PyTorch, and the AdamW optimizer is selected to update the network parameters through backpropagation. This invention employs a two-stage training strategy to improve overall stability, specifically:

[0071] Phase 1: Freeze MENet parameters and update HSG module parameters for 10 epochs. Optimize HSG separately to make its output hidden state match the fixed MENet structure as closely as possible, thereby avoiding instability caused by gradient interference in the early stages of joint training.

[0072] Phase 2: Unfreeze the entire framework and jointly train AENet and MENet for 10 epochs. This phase utilizes the total loss function. The system is optimized as a whole to enable aperture modulation events and motion events to work together in a unified network.

[0073] In both phases, the training employs a cosine annealing strategy, with an initial learning rate of... The learning rate eventually decays to Furthermore, gamma correction (gamma value of 1.2) is applied to the reconstructed frames at the network output to compensate for the nonlinearity of brightness distribution during event reconstruction and improve the visual effect of the final video.

[0074] Secondly, the present invention also provides an event camera video reconstruction system based on active aperture modulation, comprising the following modules to implement the method described in any of the above:

[0075] An aperture modulation controller, located at the front of the event camera, is used to adjust the aperture according to a preset period. Perform a periodic modulation process of opening the aperture, holding it open, and then closing it, where the opening and closing of the aperture are symmetrical processes, and the required time is... The duration for which the aperture remains open is ;

[0076] Event stream segmentation module, used to segment the event stream at consecutive aperture full-closing time points. and Divide the event stream into time windows as boundaries. For each time window, the event stream is further divided into: aperture opening phase, based on the aperture state. Internally triggered events As an aperture modulation trigger event; aperture holding open phase Events triggered by internal factors such as scene motion or changes in light intensity , as a motion trigger event;

[0077] The endpoint frame estimation module, connected to the event stream segmentation module, is used to receive the aperture modulation trigger event. The endpoint frame reference intensity for each time window is estimated using an aperture modulation event subnetwork. and hidden state ;

[0078] The dynamic scene reconstruction module, connecting the event stream segmentation module and the endpoint frame estimation module, is used to receive the motion-triggered events. The endpoint frame reference strength and hidden state A series of high frame rate image frames within a time window are reconstructed through a motion-triggered event subnetwork.

[0079] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, performs the steps of the method as described in any of the preceding claims.

[0080] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method as described in any of the preceding claims.

[0081] Example

[0082] like Figure 2As shown.

[0083] Camera system setup: The camera system consists of a Prophesee EVK4 event camera and a ComputarLensConnect BH Series Variable Focal Length Lens motorized lens. This lens has an adjustable aperture function, which can periodically change the aperture diameter according to a preset aperture modulation curve, so that the aperture goes through the process of "opening the aperture - keeping it open - closing the aperture" in one cycle.

[0084] Endpoint Frame Estimation: Following the above process, the original event stream is segmented according to the observation time window, divided into aperture modulation triggered events and motion triggered events. Furthermore, based on the target frame rate, the event stream corresponding to the "keep on" phase is further evenly divided. The aperture modulation triggered events are then input into the AENet network. Specifically, the events are first input into the FIR module to establish a temporal matrix, then the IDN module is used to obtain the denoised endpoint frame estimate, and finally the HSG module is used to obtain the hidden state corresponding to the endpoint frame. This ultimately yields accurate endpoint frame estimates and hidden state estimates, facilitating subsequent dynamic scene reconstruction.

[0085] Dynamic scene reconstruction: Motion-triggered events within the same time window are voxelized in chronological order to obtain a set of event voxel sequences corresponding to the target frame rate. These sequences, along with the two reference frames and hidden states output by AENet, are input into the motion event sub-network MENet. Finally, for each observation window, the algorithm solves for the two endpoint frames, the forward candidate frame sequence, and the reverse candidate frame sequence. These sequences are then fused using a pixel-level fusion processor to obtain the final reconstructed video. Because the algorithm does not utilize aperture closure trigger events, a frame missing problem occurs in the temporal sequence. This invention uses the RIFE algorithm to interpolate frames and obtain the corresponding frames, ensuring temporal consistency and video reconstruction quality.

[0086] Compared with other existing methods (such as FireNet, V2V-E2VID, BDE2VID, ETNet, etc.), the method of this invention significantly improves the visual quality of the final video (e.g., Figure 2(As shown). These comparison methods are all relatively classic or recent event-based video reconstruction methods, including FireNet published at WACV'20, ETNet at ECCV'24, BDE2VID at TMM'25, and V2V-E2VID at NeurIPS'25. This invention uses the corresponding pre-trained models for performance comparison. Regarding the test dataset, this invention uses the EvAid and HQF event-based video reconstruction datasets to synthesize two semi-realistic datasets, while also using the AMED dataset captured in this invention. To maintain consistency with real data when synthesizing the semi-realistic datasets, this invention selects every 5 seconds as a time window in the sequence. Within each time window, the first frame is considered as the image captured when the aperture is open, and the last frame corresponds to the aperture being closed. Since the EvAid and HQF datasets only contain motion-triggered events, this invention synthesizes each frame obtained when the aperture is open based on the degradation model proposed in existing work. The last frame is discarded. After processing each sequence, RIFE is used to interpolate missing frames using adjacent frames. The comparison methods directly use motion-triggered events from the dataset as input. On all datasets, this invention demonstrates best-in-class quantitative and qualitative results to date. Figure 2 The results shown use the EvAid dataset, and it can be seen that the method proposed in this invention has better background reconstruction quality and overall effect.

[0087] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for reconstructing video from an event camera based on active aperture modulation, characterized in that, Includes the following steps: S1, Aperture Period Modulation: Control the aperture of the event camera in advance according to a preset cycle. Perform periodic modulation of opening the aperture, holding it open, and then closing it; where opening and closing the aperture are symmetrical processes, each requiring a time of... The duration for which the aperture remains open is ; S2, Event Stream Segmentation: With the aperture completely closed at consecutive time points and Divide the event stream into time windows as boundaries. Furthermore, the aperture modulation trigger event during the aperture opening phase was segmented. Motion-triggered events during the aperture-open phase ; S3, Endpoint Frame Estimation: The aperture modulation trigger event Input the aperture modulation event subnetwork to estimate the endpoint frame reference intensity for each time window. and hidden state ; S4. Dynamic Scene Reconstruction: The motion trigger event The endpoint frame reference strength and hidden state Input the motion-triggered event subnetwork to reconstruct a series of high frame rate image frames within the time window.

2. The event camera video reconstruction method based on active aperture modulation according to claim 1, characterized in that, In step S2, after segmenting the motion trigger event... Then, the corresponding time interval is determined based on the target frame rate. Cut evenly into It calculates the event voxels of the event flow within each sub-interval.

3. The event camera video reconstruction method based on active aperture modulation according to claim 1, characterized in that, In step S3, the aperture modulation event subnetwork includes: The intensity reconstruction module based on the first positive event is used to... The events are counted pixel by pixel, recording the timestamp of the first positive polarity event triggered by each pixel during the aperture opening process. Construct the time matrix and estimate the steady-state irradiance of the scene based on the following physical model. This yields the initial intensity image; ， in, For the event camera threshold, Dark current, For an aperture-modulated optical system, is the transmittance function. The image denoising module, connected to the intensity reconstruction module based on the first positive event, performs end-to-end denoising and detail restoration on the initial intensity image, and outputs the denoised endpoint frame reference intensity. ; The hidden state generation module, connected to the image denoising module, is used to... The processed input is a convolutional network with the same forward branch structure as the dynamic scene reconstruction module, generating hidden states compatible with the dynamic scene reconstruction module. .

4. The event camera video reconstruction method based on active aperture modulation according to claim 3, characterized in that, The image denoising module employs a SwinIR neural network based on convolution and self-attention mechanisms.

5. The event camera video reconstruction method based on active aperture modulation according to claim 1, characterized in that, In step S4, the motion-triggered event subnetwork adopts a forward-backward bidirectional reconstruction structure and includes a pixel-level fusion module, specifically used for: Forward branch, in hidden state In the initial state, the motion-triggered event voxel sequence is processed in chronological order to recursively reconstruct all positive candidate frames within the time window; Backward branch, in hidden state Starting from the initial state, the motion-triggered event voxel sequence is processed in reverse chronological order to recursively reconstruct all reverse candidate frames within the time window. Through a pixel-level fusion module, for each reconstruction time point, the corresponding forward candidate frame, reverse candidate frame, and reference strength of the preceding and following endpoint frames are combined. and The corresponding event voxel grid is input into a lightweight convolutional network to generate a pixel-level normalized fusion weight map, which is then used to fuse the four frames to obtain the final reconstructed frame.

6. An event camera video reconstruction system based on active aperture modulation, characterized in that, The following modules are included to implement the method of any one of claims 1-5: An aperture modulation controller, located at the front of the event camera, is used to adjust the aperture according to a preset period. Perform a periodic modulation process of opening the aperture, holding it open, and then closing it, where the opening and closing of the aperture are symmetrical processes, and the required time is... The duration for which the aperture remains open is ; Event stream segmentation module, used to segment the event stream at consecutive aperture full-closing time points. and Divide the event stream into time windows as boundaries. For each time window, the event stream is further divided into: aperture opening phase, based on the aperture state. Internally triggered events As an aperture modulation trigger event; aperture holding open phase Events triggered by internal factors such as scene motion or changes in light intensity , as a motion trigger event; The endpoint frame estimation module, connected to the event stream segmentation module, is used to receive the aperture modulation trigger event. The endpoint frame reference intensity for each time window is estimated using an aperture modulation event subnetwork. and hidden state ; The dynamic scene reconstruction module, connecting the event stream segmentation module and the endpoint frame estimation module, is used to receive the motion-triggered events. The endpoint frame reference strength and hidden state A series of high frame rate image frames within a time window are reconstructed through a motion-triggered event subnetwork.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1-5.

Images