Event-based sensor and spatio-temporal fusion aberration recovery algorithm-based minimal imaging system

Abstract

The application discloses a kind of based on event sensor and space-time fusion aberration recovery algorithm's minimal imaging system, belong to aberration recovery field.It includes minimal optical device, event camera imaging sensor and computing processing device, minimal optical device is used to capture surrounding scene visual information with aberration, event camera imaging sensor is used to collect scene dynamic visual information with aberration, while outputting digital image and event information;Computing processor is equipped with space-time fusion aberration recovery algorithm based on event auxiliary, while inputting digital image with aberration and event information, output clear digital image.The aberration recovery system of the application is built by minimal optical system, reduces system weight, volume and cost, solves the problem of quality decline in minimal optical system, by introducing additional event data as time sequence information supplement, fusion with original aberration recovery spatial information processing procedure, realizes high-quality aberration recovery task.

Description

Technical Field

[0001] This invention belongs to the field of aberration recovery, specifically relating to a minimalist imaging system based on an event sensor and a spatiotemporal fusion aberration recovery algorithm. Background Technology

[0002] With the continuous advancement and development of optical technology, the demand for high image quality, large field of view, small size, and light weight is gradually increasing, which has become a new development trend in optical system design. Traditional optical systems, while pursuing high image quality, often occupy a large volume and space, hindering integration and portability. This presents new challenges for some special applications, such as assistive devices for the blind, drones, virtual reality glasses, and robots.

[0003] With the increasing demands of mobile devices, minimalist optical systems are gradually gaining popularity as an alternative imaging system. Specifically, minimalist optical systems typically consist of only 1-3 spherical lenses. Therefore, the advantages of this imaging system are obvious: it is lightweight, has low load, and is easy to manufacture. Furthermore, due to the small number of lenses and simple structure, it is almost unaffected by eccentricity or tilting caused by collisions or movement, making it suitable for deployment in mobile devices, including drones and robots. However, because of the use of simple and few optical components to maintain a compact structure, there are few parameters that can be optimized during optical design. Minimalist optical systems often suffer from significant aberrations, leading to decreased image quality. This degraded image quality poses a challenge to downstream applications.

[0004] The current mainstream solution is to utilize computational imaging algorithms, combined with aberration distribution of minimalist optical systems and data-driven post-processing image restoration algorithms, to restore images with aberrations. However, in current computational imaging frameworks for minimalist optical systems, when faced with systems with large aberrations like these, existing data-driven methods fail to reconstruct high-quality, clear images due to significant information loss, resulting in artifacts and blurred areas in the reconstruction results. Therefore, to achieve better restoration results, the algorithm needs to be supplemented with information about the shooting scene. Meanwhile, temporal information is the easiest additional information to obtain; it can be acquired simply by moving the camera to capture the surrounding scene. Therefore, this invention proposes using an event camera with extremely high temporal resolution to assist the minimalist optical system in imaging. Summary of the Invention

[0005] To address the problems in existing technologies, this invention proposes a minimalist optical system based on an event sensor and a spatiotemporal fusion aberration recovery algorithm. The event camera is a neuromorphic-inspired sensing device. Unlike traditional cameras that capture synchronized frames at a fixed frame rate, the event camera asynchronously generates event output with a microsecond-level delay at the precise time of event occurrence. It features extremely low latency (~1μs), extremely high temporal resolution, significantly increased dynamic range (140dB), and extremely low power consumption. The event camera can provide more temporal information, which can help the minimalist optical system recover aberrations in extreme environments.

[0006] This invention proposes a simplified imaging system based on an event sensor and a spatiotemporal fusion aberration recovery algorithm. The aberration recovery system built using a simplified optical system reduces system weight, size, and cost, and addresses the image quality degradation problem in simplified optical systems. By introducing additional event data as temporal information supplementation, it is integrated with the original aberration recovery spatial information processing flow to achieve high-quality aberration recovery.

[0007] This invention is achieved through the following method:

[0008] A minimalist imaging system based on an event sensor and a spatiotemporal fusion aberration recovery algorithm, comprising:

[0009] A minimalist optical device for capturing visual information about the surrounding scene with aberrations;

[0010] An event camera imaging sensor is used to collect dynamic visual information of a scene with aberrations, while outputting digital images and event information.

[0011] The computing processor is equipped with an event-assisted spatiotemporal fusion aberration recovery unit. The computing processing device simultaneously inputs a digital image with aberrations and event information, and outputs a clear digital image.

[0012] The event-assisted spatiotemporal fusion aberration recovery unit uses a spatiotemporal fusion aberration recovery algorithm for aberration recovery. The spatiotemporal fusion aberration recovery algorithm includes the following steps:

[0013] 1) Obtain clear, aberration-free data I gt Based on the parameters of a minimalist optical device, aberration-affected data I is obtained through aberration simulation. aber ;

[0014] 2) For clear, aberration-free data I gt Frame interpolation and aberration simulation are performed sequentially. The datasets after frame interpolation and aberration simulation are then fed into the event simulator one by one to generate the corresponding original event data, denoted as I. event ;

[0015] 3) Will Igt I aber and I event The data in the dataset are paired one by one to form training data pairs, that is, multiple training data pairs constitute the dataset for the aberration recovery algorithm.

[0016] 4) Train the spatiotemporal fusion aberration recovery model and use the trained spatiotemporal fusion aberration recovery model to perform aberration recovery.

[0017] According to a preferred embodiment of the present invention, the spatiotemporal fusion aberration restoration model in step 4) includes an event representation module, a spatiotemporal fusion module, a feature processing unit, and an image reconstruction unit; the event representation module processes the original event data I event The event representation is standardized and normalized; the spatiotemporal fusion module includes a temporal fusion module and a spatial fusion module; the temporal fusion module extracts additional temporal information from the event data and transforms it into additional spatial information; the spatial fusion module combines the spatial information transformed by the temporal fusion module with the blurred, aberration-laden data after feature processing. aber To achieve spatial integration.

[0018] The event representation module describes the raw event data I. event The event representation is standardized and normalized, specifically as follows:

[0019] a) Transfer the original event data I event Slice the data according to a fixed timestamp length;

[0020] b) Aggregate the spatial information within a fixed timestamp of the event data in a slice using 3D convolution, effectively reducing the memory consumption of the event data without losing much of the event's temporal information;

[0021] c) Using a multilayer perceptron, the events aggregated through 3D convolution in step b) are mapped to a high-dimensional space to extract effective spatiotemporal information from the event data, thereby effectively representing the event information, denoted as I. event f eature .

[0022] According to a preferred embodiment of the present invention, step 4), the training steps of the spatiotemporal fusion aberration restoration model include:

[0023] 4.1) Training of Feature Processing Unit and Image Reconstruction Unit

[0024] 4.1.1) Batch load training data, including blurry data with aberrations. aber Clear, aberration-free data I gt ;

[0025] 4.1.2) For data I that is blurred and has aberrations... aberThe data is input into the feature processing unit and the image reconstruction unit to obtain the prediction data I. pred ;

[0026] 4.1.3) Predict the data I pred and clear, aberration-free data I gt Calculate the absolute value loss of pixels between them;

[0027] 4.1.4) Based on the pixel absolute value loss calculated in 4.1.3), perform backward gradient optimization and update the parameters of the two units mentioned above; determine whether the current number of training iterations has reached the expected number of training iterations. If it has, end the training; if it has not, return to step 4.1.1) and continue training.

[0028] 4.2) Training of the event representation module and the spatiotemporal fusion module

[0029] 4.2.1) Batch load training data, including blurry data with aberrations. aber Clear, aberration-free data I gt and raw event data I event ;

[0030] 4.2.2) For data I that is blurred and has aberrations... abert Input the already trained feature processing unit in 4.1) to obtain a high-dimensional feature map, denoted as I. aberfeature ;

[0031] 4.2.3) Transfer the original event data I event The input is fed into the event representation module, resulting in a high-dimensional event feature map, denoted as I. eventfeature ;

[0032] 4.2.3) The time fusion module integrates high-dimensional event features rich in temporal information. event feature The spatial fusion module converts the spatial information converted by the temporal fusion module into spatial information, and then combines this spatial information with the I obtained in 4.2.2). aber featur Spatial fusion is performed to obtain a fused feature map, denoted as I. fusion feature ;

[0033] 4.2.4) Obtain clear, aberration-free data I gt Input the already trained feature processing unit in 4.1) to obtain a high-dimensional feature map, denoted as I. gtfeature ;

[0034] 4.2.5) The I obtained in 4.2.3) fusion feature The I obtained from (and 4.2.4) gt feature Constraints are applied using a fusion loss function, the specific expression of which is shown below:

[0035] Lossfusion =αMSE(I fusion feature I gt featur )βMSE(Gram(I fusion feature ), Gram(I gt feature ))

[0036] Where α and β are the loss weights, and the MSE function represents the calculation of the mean squared error, MSE(I fusion feature I gt featur ) for I fusion feature and I gt feature The pixel mean square error between them, the Gram function represents the calculation of feature I fusion feature and I gt feature The Gram matrix;

[0037] 4.2.6) Based on the fusion loss calculated in 4.2.5), perform inverse gradient optimization, only update the network parameters of the event flag module, spatial fusion and temporal fusion module, and keep the parameters of the feature extraction module unchanged; determine whether the current number of training iterations has reached the expected number of training iterations. If it has, end the training. If it has not, return to step 4.2.1) and continue training.

[0038] Furthermore, when using the trained spatiotemporal fusion aberration recovery model for aberration recovery, the specific steps are as follows: The trained model is loaded into a computing processor, and the entire minimalist imaging system is first made to move relative to the imaging scene. The event camera imaging sensor will record the blurred, aberrated RGB image or video and the corresponding original event data. Then, the RGB image or video and the original event data are input into the trained spatiotemporal fusion aberration recovery model. This model uses the rich temporal information in the event data to effectively recover the blurred, aberrated RGB image or video, and finally obtain a clear RGB image or video.

[0039] This invention utilizes event data as an auxiliary modality to mine potential temporal information, and combines it with a spatiotemporal fusion aberration recovery algorithm to solve the problem of image quality degradation in minimalist optical imaging systems. It includes the following beneficial effects:

[0040] (1) This invention uses a minimalist optical imaging system to perceive the surrounding environment, which greatly simplifies the weight and volume of the system, promotes the development of optical systems towards lightweight and miniaturization, and is more suitable for applications in some mobile terminals and wearable devices, such as intelligent detection drones and intelligent detection robots; at the same time, the imaging system is not easily affected by factors such as collision and eccentricity, and has a stronger ability to adapt to the environment.

[0041] (2) Improves image restoration quality for systems with large aberrations: Traditional data-driven image restoration algorithms face difficulties when processing images from minimalist optical systems. Due to significant information loss, the reconstruction results may suffer from artifacts and blurriness. By introducing an event camera, additional temporal information can be provided, which can be used to supplement the details of the shooting scene. Through the designed spatiotemporal fusion module, the additional temporal information in the event data can be fused with the original spatial information for aberration restoration, thereby improving the aberration restoration quality of the image. Attached Figure Description

[0042] Figure 1 It is a minimalist imaging system based on event sensors and spatiotemporal fusion aberration recovery algorithms;

[0043] Figure 2 This is a schematic diagram of the training of the feature extraction unit and the image reconstruction unit;

[0044] Figure 3 This is a flowchart of the training process for the feature extraction unit and the image reconstruction unit;

[0045] Figure 4 This is a schematic diagram of the training of an event-assisted spatiotemporal fusion aberration recovery algorithm model;

[0046] Figure 5 This is a schematic diagram of the event representation module;

[0047] Figure 6 This is a flowchart of the training process for an event-assisted spatiotemporal fusion aberration recovery algorithm model.

[0048] Figure 7 This is a schematic diagram of the evaluation of an event-assisted spatiotemporal fusion aberration recovery algorithm model;

[0049] Figure 8 These are before-and-after comparison images. Detailed Implementation

[0050] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0051] Figure 1It is a minimalist imaging system based on an event sensor and a spatiotemporal fusion aberration recovery algorithm. The hardware device includes a minimalist optical device, an event camera sensor, and a computing processor. The minimalist optical device is used to acquire visual information of the surrounding environment with aberrations. The event camera sensor is used to collect visual information with aberrations and simultaneously output digital images and event information. The computing processor is equipped with an event-assisted spatiotemporal fusion aberration recovery unit. The computing processing device simultaneously inputs digital images with aberrations and event information and outputs clear digital images.

[0052] The minimalist optical device can be a single spherical lens or a single Fresnel lens. The former has a simple structure, low cost, and is easy to mass-produce; the latter has more optimization variables and can provide better aberration correction and balance, but it is more difficult to manufacture and is suitable for scenarios with high requirements for accuracy and robustness. The event camera sensor is a commercial event camera sensor, the computing processor is a GPU computing unit, and it is equipped with an event-assisted spatiotemporal fusion aberration recovery unit (which contains a spatiotemporal fusion aberration recovery algorithm).

[0053] The present invention utilizes an event-assisted spatiotemporal fusion aberration recovery unit to train a spatiotemporal fusion aberration recovery model using a spatiotemporal fusion aberration recovery algorithm, thereby performing aberration recovery. The spatiotemporal fusion aberration recovery algorithm includes the following steps:

[0054] 1) Obtain clear, aberration-free data I gt Based on the parameters of a minimalist optical device, aberration-affected data I is obtained through aberration simulation. aber ;

[0055] 2) For clear, aberration-free data I gt Frame interpolation and aberration simulation are performed sequentially. The datasets after frame interpolation and aberration simulation are then fed into the event simulator one by one to generate the corresponding original event data, denoted as I. event ;

[0056] 3) Will I gt I aber and I event The data in the dataset are paired one by one to form training data pairs, that is, multiple training data pairs constitute the dataset for the aberration recovery algorithm.

[0057] 4) Train the spatiotemporal fusion aberration recovery model and use the trained spatiotemporal fusion aberration recovery model to perform aberration recovery.

[0058] In step 4), the training process of the spatiotemporal aberration recovery model is divided into two steps.

[0059] The first step is the training of the feature extraction unit and the image reconstruction unit, as shown in the framework diagram below. Figure 2 As shown. I aber : Blurry, aberrant data, Igt Clear, aberration-free data, I pred Predicted data for feature extraction unit and image reconstruction unit. Figure 3 yes Figure 2 The training process diagrams for the feature extraction unit and the image reconstruction unit are shown below.

[0060] 4.1.1) Batch load training data, including blurry data with aberrations. aber Clear, aberration-free data I gt ;

[0061] 4.1.2) For data I that is blurred and has aberrations... aber The data is input into the feature processing unit and the image reconstruction unit to obtain the prediction data I. pred ;

[0062] 4.1.3) Predict the data I pred and clear, aberration-free data I gt Calculate the absolute value loss of pixels between them;

[0063] 4.1.4) Based on the pixel absolute value loss calculated in 4.1.3), perform backward gradient optimization and update the parameters of the two units mentioned above; determine whether the current number of training iterations has reached the expected number of training iterations. If it has, end the training; if it has not, return to step 4.1.1) and continue training.

[0064] The second step involves training the event representation module and the spatiotemporal fusion module. The framework diagram is as follows: Figure 4 As shown. I aber : Blurry, aberrant data, I gt Clear, aberration-free data, I event Raw event data, T i : timestamp, I event feature : High-dimensional feature map of event data, I fusion feature : High-dimensional features of fused data, I gt feature High-dimensional features of clear, aberration-free data, Loss fusion : Fusion loss function. The spatiotemporal fusion aberration restoration model mainly includes an event representation module, a spatiotemporal fusion module, a feature processing unit, and an image reconstruction unit. Figure 5 It is the specific structure of the event representation module, capable of processing raw data I event After normalization and unification of the representation, I is obtained. event feature This helps subsequent modules to extract rich temporal information from the event data. The feature processing unit can convert I... aber Data is mapped to a high-dimensional space to obtain I aber feature The time fusion module uses input I... eventfeature, dig I event feature The event data contains potential temporal information, which is then transformed into spatial information. The spatial fusion module combines the transformed spatial information with I... aber feature The data undergoes spatial fusion and is then input into the subsequent image reconstruction unit. The temporal fusion module employs an information fusion structure based on ResBlock or Transformer, capable of extracting additional temporal information from the event data and transforming it into additional spatial information. The spatial fusion module is a feature alignment and fusion module based on optical flow or deformable convolutional structures, capable of integrating spatial information and inputting it into the subsequent aberration recovery network. The feature processing unit can consist of modules such as RRDB or ResBlock to further process the fused features. The image reconstruction unit can consist of modules such as convolutional layers and upsampling layers to reconstruct the image from the processed features. The key to this invention is proposing the use of event data with richer temporal information, fused into the original aberration recovery process, achieving good aberration recovery results even in extreme environments.

[0065] The training scheme for the spatiotemporal fusion aberration restoration model requires the following data for training:

[0066] (1) Clear, aberration-free datasets: Directly use a large number of existing public datasets, such as REDS or Vimeo-90k datasets, where clear, aberration-free data are used as ground truth.

[0067] (2) Blurred and aberration-laden dataset: Input the parameters of the simplified optical device with aberrations (including simulation image size, sensor size, lens surface definition parameters and material parameters) into the optical simulation model (including ray tracing simulation model and wave optical simulation model), input the clear and aberration-free data into the optical simulation model one by one, and output the blurred and aberration-laden dataset.

[0068] (3) Simulated event data: After the clear, aberration-free data is processed by frame interpolation and aberration simulation, the high frame rate blurry, aberration-affected data is obtained and input into the event simulator to obtain blurry, aberration-affected event data.

[0069] Figure 6 This demonstrates the specific training process for the event representation module and the spatiotemporal fusion module:

[0070] 4.2.1) Batch load training data, including blurry data with aberrations. aber Clear, aberration-free data I g t and raw event data I event ;

[0071] 4.2.2) For data I that is blurred and has aberrations... abertInput the already trained feature processing unit in 4.1) to obtain a high-dimensional feature map, denoted as I. aber feature ;

[0072] 4.2.3) Transfer the original event data I event The input is fed into the event representation module, resulting in a high-dimensional event feature map, denoted as I. event feature ;

[0073] 4.2.3) The time fusion module integrates high-dimensional event features rich in temporal information. event feature The spatial fusion module converts the spatial information converted by the temporal fusion module into spatial information, and then combines this spatial information with the I obtained in 4.2.2). aber featur Spatial fusion is performed to obtain a fused feature map, denoted as I. fusion feature ;

[0074] 4.2.4) Obtain clear, aberration-free data I gt Input the already trained feature processing unit in 4.1) to obtain a high-dimensional feature map, denoted as I. gt feature ;

[0075] 4.2.5) The I obtained in 4.2.3) fusion feature The I obtained from (and 4.2.4) gt feature Constraints are applied using a fusion loss function, the specific expression of which is shown below:

[0076] Loss fusion =αMSE(I fusion feature I gt featur )βMSE(Gram(I fusion feature ), Gram(I gt feature ))

[0077] Where α and β are the loss weights, and the MSE function represents the calculation of the mean squared error, MSE(I fusion feature I gt featur ) for I fusion feature and I gt feature The pixel mean square error between them, the Gram function represents the calculation of feature I fusion feature and I gt feature The Gram matrix;

[0078] 4.2.6) Based on the fusion loss calculated in 4.2.5), perform inverse gradient optimization, only update the network parameters of the event flag module, spatial fusion and temporal fusion module, and keep the parameters of the feature extraction module unchanged; determine whether the current number of training iterations has reached the expected number of training iterations. If it has, end the training. If it has not, return to step 4.2.1) and continue training.

[0079] The event simulator described in this invention can be ESIM (Rebecq H, Gehrig D, Scaramuzza D. ESIM: an open event camera simulator [C] / / Conference on robot learning.PMLR,2018:969-982.) or V2E (Hu Y, Liu SC, Delbruck T. v2e: From video frames to realistic DVSevents [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2021:1312-1321.) or DVS-Voltmeter (Lin S, Ma Y, Guo Z, et al. DVS-Voltmeter: Stochastic process-based event simulator for dynamic visionsensors [C] / / European Conference on Computer Vision.Cham:Springer NatureSwitzerland,2022:578-593.).

[0080] In application, the trained model is loaded into the computing unit. The complete structure of the model is as follows: Figure 7 As shown, when using this imaging device, a relative motion is first generated between the device and the imaging scene. The sensor records a blurred, aberrant RGB image or video and the corresponding original event data. Then, the RGB image or video and the original event data are input into a trained spatiotemporal fusion aberration recovery model. The spatiotemporal fusion aberration recovery model includes five parts: feature processing unit, event representation module, spatial fusion module, temporal fusion module, and image reconstruction unit. This model utilizes the rich temporal information in the event data to effectively recover the blurred, aberrant RGB image or video, and finally obtains a clear RGB image or video.

[0081] Figure 8The presentation shows a comparison between this method and a pure RGB-based method. The backbone network can be any restoration network based on a pure RGB image; this patent uses the EDVR network as an example. Aber: blurred data with aberrations; Event: raw event data generated by the event simulator; GT: clear, aberration-free data; RGB: restoration result based on the pure RGB model; RGB+Event: restoration result of the spatiotemporal fusion aberration restoration model. It is evident that the restoration result based on the pure RGB model is almost unable to distinguish subtle data points. This method effectively improves aberration restoration performance, achieving results essentially consistent with GT.

[0082] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A simplified imaging system based on an event sensor and a spatiotemporal fusion aberration recovery algorithm, characterized in that, include: A minimalist optical device for capturing visual information about the surrounding scene with aberrations; An event camera imaging sensor is used to collect dynamic visual information of a scene with aberrations, while outputting digital images and event information. The computing processor is equipped with an event-assisted spatiotemporal fusion aberration recovery unit. The computing processor simultaneously inputs a digital image with aberrations and event information, and outputs a clear digital image. The event-assisted spatiotemporal fusion aberration recovery unit uses a spatiotemporal fusion aberration recovery algorithm for aberration recovery. The spatiotemporal fusion aberration recovery algorithm includes the following steps: 1) Obtain clear, aberration-free data I gt Based on the parameters of a minimalist optical device, aberration-affected data I is obtained through aberration simulation. aber ; 2) For clear, aberration-free data I gt Frame interpolation and aberration simulation are performed sequentially. The datasets after frame interpolation and aberration simulation are then fed into the event simulator one by one to generate the corresponding original event data, denoted as I. event ; 3) Will I gt I aber and I event The data in the dataset are paired one by one to form training data pairs, that is, multiple training data pairs constitute the dataset for the aberration recovery algorithm. 4) Train the spatiotemporal fusion aberration recovery unit and use the trained spatiotemporal fusion aberration recovery unit to perform aberration recovery.

2. The simplified imaging system based on an event sensor and a spatiotemporal fusion aberration recovery algorithm according to claim 1, characterized in that, In step 1), clear, aberration-free data I is obtained. gt For example: directly use existing public datasets, where the clear, aberration-free data is used as the ground truth, denoted as I. gt .

3. The simplified imaging system based on an event sensor and a spatiotemporal fusion aberration recovery algorithm according to claim 1, characterized in that, In step 1), the blurred data I with aberrations aber The method for obtaining I is as follows: input the parameters of the simplified optical device with aberrations into the optical simulation model, and then input I... gt Each input is fed into the optical simulation model, and the output is a blurred dataset with aberrations, denoted as I. aber The parameters of the simplified optical device with aberrations include the simulated image size, sensor size, lens surface definition parameters, and material parameters. The optical simulation model is either a ray tracing simulation model or a wave optical simulation model.

4. The simplified imaging system based on an event sensor and a spatiotemporal fusion aberration recovery algorithm according to claim 1, characterized in that, Step 2) specifically refers to: 2-1) Utilize frame interpolation algorithms to improve the clarity of aberration-free data I gt The frame rate is set to meet the usage requirements of the event simulator. 2-2) Input the parameters of the simplified optical device with aberrations into the optical simulation model, and input the high frame rate data processed by frame interpolation in 2-1) into the optical simulation model one by one to perform aberration simulation; 2-3) Input the high frame rate, blurred and aberration-laden datasets obtained in 2-2) one by one into the event simulator to generate the corresponding raw event data, denoted as I. event .

5. The simplified imaging system based on an event sensor and a spatiotemporal fusion aberration recovery algorithm according to claim 1 or 4, characterized in that, The event simulator is ESIM, V2E, or DVS-Voltmeter.

6. The simplified imaging system based on an event sensor and a spatiotemporal fusion aberration recovery algorithm according to claim 1, characterized in that, Step 4) The spatiotemporal fusion aberration restoration unit includes an event representation module, a spatiotemporal fusion module, a feature processing unit, and an image reconstruction unit; the event representation module processes the original event data I event To achieve unified and normalized event representation; the spatiotemporal fusion module includes a time fusion module and a space fusion module; The time fusion module extracts additional temporal information contained in the event data and transforms it into additional spatial information; The spatial fusion module combines the spatial information transformed by the temporal fusion module with the blurred, aberration-laden data after feature processing. aber To achieve spatial integration.

7. The simplified imaging system based on an event sensor and a spatiotemporal fusion aberration recovery algorithm according to claim 6, characterized in that, The event representation module describes the raw event data I. event The event representation is standardized and normalized, specifically as follows: a) Transfer the original event data I event Slice the data according to a fixed timestamp length; b) Aggregate spatial information within a fixed timestamp by using 3D convolution to aggregate event data within a slice, effectively reducing memory consumption of event data without losing much of the event's temporal information; c) Using a multilayer perceptron, the events aggregated through 3D convolution in step b) are mapped to a high-dimensional space to extract effective spatiotemporal information from the event data, thereby effectively representing the event information, denoted as I. event feature .

8. The simplified imaging system based on an event sensor and a spatiotemporal fusion aberration recovery algorithm according to claim 6, characterized in that, The temporal fusion module is a feature extraction and transformation module based on ResBlock or Transformer structures; the spatial fusion module is a feature alignment and fusion module based on optical flow or deformable convolutional structures; the feature processing unit and the image reconstruction unit are aberration recovery network structures based on pure RGB images, wherein the feature processing unit is regarded as an encoder of the pure RGB aberration recovery network, and the image reconstruction unit is regarded as a decoder of the pure RGB aberration recovery network. The feature processing unit is composed of RRDB or ResBlock modules, which further extract and process the features after spatiotemporal fusion. The image reconstruction unit is composed of convolutional layers and upsampling layers, which reconstruct the image from the features processed by the feature processing unit.

9. The simplified imaging system based on an event sensor and a spatiotemporal fusion aberration recovery algorithm according to claim 6, characterized in that, In step 4), the training steps for the spatiotemporal fusion aberration recovery unit include: 4-1) Training of the feature processing unit and the image reconstruction unit: 4-1-1) Batch loading of training data, including blurry data with aberrations. aber Clear, aberration-free data I gt ; 4-1-2) I of the blurred and aberrated data aber The data is input into the feature processing unit and the image reconstruction unit to obtain the prediction data I. pred ; 4-1-3) Predict the data I pred and clear, aberration-free data I gt Calculate the absolute value loss of pixels between them; 4-1-4) Based on the pixel absolute value loss calculated in 4-1-3), perform inverse gradient optimization to update the parameters of the feature processing unit and the image reconstruction unit; determine whether the current number of training iterations has reached the expected number of training iterations. If it has, end the training; if it has not, return to step 4-1-1) and continue training. 4-2) Training of the event representation module and the spatiotemporal fusion module: 4-2-1) Batch loading of training data, including blurry data with aberrations. aber Clear, aberration-free data I gt and raw event data I event ; 4-2-2) I of the blurred and aberrated data abert Input the already trained feature processing unit from section 4.1) to obtain a high-dimensional feature map, denoted as I. aber feature ; 4-2-3) Transfer the original event data I event The input is fed into the event representation module, resulting in a high-dimensional event feature map, denoted as I. event feature ; 4-2-4) The time fusion module will integrate high-dimensional event feature maps rich in temporal information into I. event feature The spatial fusion module converts the spatial information converted by the temporal fusion module into I obtained in 4-2-2). aber feature Spatial fusion is performed to obtain a fused feature map, denoted as I. fusion feature ; 4-2-5) Obtain clear, aberration-free data I gt Input the feature processing unit that has already been trained in 4-1) to obtain a high-dimensional feature map, denoted as I. gt feature ; 4-2-6) The I obtained from 4-2-4) fusion feature I obtained from (and 4-2-5) gt feature Constraints are applied using a fusion loss function, the specific expression of which is shown below: ； in, and It is loss weight. The function represents the calculation of the mean square error. Indicate I fusion feature and I gt feature The pixel mean square error between them, the Gram function represents the calculation of feature I fusion feature Or I gt feature The Gram matrix; 4-2-7) Based on the fusion loss calculated in 4-2-6), perform reverse gradient optimization, only update the network parameters of the event representation module, spatial fusion module and temporal fusion module, and keep the feature processing unit parameters unchanged; determine whether the current number of training iterations has reached the expected number of training iterations. If it has, end the training. If it has not, return to step 4-2-1) and continue training.

10. The simplified imaging system based on an event sensor and a spatiotemporal fusion aberration recovery algorithm according to claim 1, characterized in that, The aforementioned minimalist optical device is a single spherical lens or a single Fresnel lens; the computing processor is a GPU computing unit.

Images