Dynamic event light field acquisition apparatus and method

Abstract

The application relates to an event light field acquisition device and method, belonging to the technical field of optical imaging and machine vision. The device comprises an optical imaging module, a high-speed modulation module, an event sensing module, a control module and a domain adaptive reconstruction module; the optical imaging module collects light field light rays and guides the light field light rays to the high-speed modulation module; the high-speed modulation module performs high-speed space-time modulation on the light field light rays; the event sensing module captures the light intensity change of the modulated light field light rays and outputs an event light field pseudo image; the domain adaptive reconstruction module receives the event light field pseudo image, recovers high-fidelity 5D dynamic event light field data from compressed and encoded 4D event light field pseudo image data through a multi-stage domain adaptive depth equalization model. The application realizes high-speed, high-fidelity and high-robustness dynamic event light field acquisition, reduces the hardware complexity, improves the light efficiency, and solves the problems of resolution trade-off, poor low-light robustness, low dynamic scene adaptability and insufficient reconstruction precision.

Description

Technical Field

[0001] This invention relates to the fields of computational optical imaging and machine vision technology, and specifically to a device and method for acquiring dynamic event light fields. Background Technology

[0002] Light field imaging can capture the spatial, angular, and radiative distribution information of light, enabling computational optics operations such as post-focusing, view synthesis, and depth estimation. It is a core technology in the field of 3D visual perception. Traditional light field cameras mostly rely on microlens arrays or camera arrays to acquire light fields, which suffers from inherent trade-offs in spatial-angular-temporal resolution. The frame rate of consumer-grade light field cameras is usually limited to single digits, making it difficult to effectively capture fast-moving scenes. At the same time, traditional light field cameras are based on complementary metal-oxide-semiconductor (CMOS) intensity sensors to detect light signals, which suffers from low signal-to-noise ratio in low-light, high dynamic range scenes, poor light efficiency, and a bulky hardware architecture with low integration, making it difficult to meet the deployment requirements of embedded devices.

[0003] To improve the temporal resolution of light field cameras, existing technologies have proposed snapshot compression acquisition methods based on digital micromirror devices (DMDs). These methods modulate and compress multiple 5D dynamic light field frames into a single 4D measurement value using a DMD, and combine this with a depth equalization reconstruction algorithm to increase the frame rate. However, this method still relies on traditional CMOS intensity sensors and suffers from problems such as insufficient robustness in low-light scenes, high light energy loss due to DMD modulation and beam splitting structures, low hardware integration, and high costs associated with adapting the domain gap between simulation and real-world scenes. Furthermore, the high complexity of the reconstruction algorithm model makes it difficult to deploy in embedded devices. Event cameras, as a novel type of visual sensor, can asynchronously capture changes in light intensity and generate event streams. They offer advantages such as high temporal resolution, high dynamic range, and low data volume, and their sensing performance in low-light, fast-moving scenes far surpasses that of traditional CMOS intensity sensors. Existing technologies have proposed light field reconstruction methods based on coded aperture and event cameras, which eliminate the dependence on intensity images and achieve pixel-level reconstruction of 4D light fields. However, this method can only be adapted to low-speed dynamic scenes, and artifacts are easily generated in the reconstruction of high-speed dynamic scenes. Furthermore, the modulation flexibility of the coded aperture is low, and it cannot achieve adaptive compression acquisition of light field data. The spatiotemporal-angular resolution control capability of the light field is also limited.

[0004] In summary, existing light field imaging technologies cannot simultaneously meet the practical application requirements of high-speed dynamic capture, low-light robustness, highly integrated hardware, high-fidelity reconstruction, and lightweight deployment. There is an urgent need for a light field acquisition technology that integrates the advantages of event cameras and DMD snapshot compression technology to solve the above-mentioned defects of existing technologies.

[0005] This invention relates to the fields of computational optical imaging and machine vision, and in particular to an event light field acquisition device and method based on event cameras, digital micromirror devices (DMDs), snapshot compression, and domain adaptive reconstruction. It can be applied to fields such as dynamic 3D scene perception, autonomous driving, augmented reality / virtual reality (AR / VR), and biological microscopy imaging. Summary of the Invention

[0006] The present invention aims to provide an event light field acquisition device and method, belonging to the field of computational optical imaging and machine vision technology. It includes an optical imaging module, a high-speed modulation module, an event sensing module, a control module, and a domain adaptive reconstruction module. The optical imaging module collects light field rays from the target scene and guides them to the high-speed modulation module. The high-speed modulation module performs high-speed temporal modulation on the light field rays transmitted by the optical imaging module. The event sensing module captures the intensity changes of the modulated light field rays and outputs an event light field pseudo-image. The domain adaptive reconstruction module receives the compressed and encoded event light field pseudo-image data and recovers high-fidelity 5D dynamic event light field data from the compressed and encoded 4D event light field pseudo-image data using a multi-stage domain adaptive depth equalization model. This invention integrates the high temporal resolution and high dynamic range advantages of an event camera with the high-speed snapshot compression modulation advantages of a DMD, combined with a domain adaptive depth equalization reconstruction algorithm, to achieve high-speed, high-fidelity, and highly robust dynamic event light field acquisition. Simultaneously, it reduces hardware architecture complexity, improves light efficiency, and solves the technical problems of existing light field imaging technologies, such as resolution trade-offs, poor robustness in low-light conditions, low adaptability to dynamic scenes, and insufficient reconstruction accuracy.

[0007] An event light field acquisition device, such as Figure 1 As shown, it includes an optical imaging module (1), a high-speed modulation module (2), an event sensing module (3), a control module (4), and a domain adaptive reconstruction module (5); the optical imaging module (1) is connected to the high-speed modulation module (2), the high-speed modulation module (2) is connected to the control module (4), and the control module (4) is connected to the event sensing module (3); the event sensing module (3) is connected to the domain adaptive reconstruction module (5); the domain adaptive reconstruction module (5) is connected to the control module (4); the control module (4) is the core control unit to coordinate the synchronization of each module.

[0008] Furthermore, the connection is an electrical connection.

[0009] Furthermore, the connection is an optical connection.

[0010] Furthermore, the optical imaging module (1) sequentially includes an imaging lens (11), an aperture (12), and a 4f relay optical system (13); the imaging lens (11), the aperture (12), and the 4f relay optical system are optically connected.

[0011] The imaging lens (11) collects the light field and transmits it to the aperture (12). The aperture (12) then transmits the light field to the high-speed modulation module (2) and the event sensing module (3) respectively through aberration-free transmission, so that the modulation surface of the high-speed modulation module (2) and the detection surface of the event sensing module (3) form an optical conjugate.

[0012] The optical imaging module (1) is used to collect the light field rays of the target scene and guide the light field rays to the high-speed modulation module (2).

[0013] Furthermore, the high-speed modulation module (2) includes a digital micromirror device (DMD) (21) and a DMD driving unit (22); the digital micromirror device (DMD) (21) and the DMD driving unit (22) are electrically connected;

[0014] The digital micromirror device (DMD) (21) is electrically connected to the optical imaging module (1) and receives the light field transmitted by the optical imaging module (1); the DMD driving unit (22) is electrically connected to the control module (4) and generates a dynamic binary encoding mask under the instruction of the control module (4) to perform high-speed time-varying control on the light field transmitted by the optical imaging module (1); and compresses and encodes the 5D dynamic light field information of multiple time frames into a single 4D light field measurement value;

[0015] The dynamic binary encoding mask generated by the high-speed modulation module (2) includes a black mask pattern; the black mask pattern is fixed at the beginning of the encoding mask sequence to form a black priority encoding sequence, which is used to realize the mutual conversion between the event light field pseudo image and the light field intensity information, reduce the number of events generated, and improve the light field acquisition efficiency.

[0016] The digital micromirror device (DMD) (21) has a refresh rate greater than or equal to 20kHz and a resolution greater than or equal to 1920×1080.

[0017] Furthermore, the event perception module (3) is an event camera, which is electrically connected to the control module (4) to capture the light intensity changes of the light field light after being modulated by the high-speed modulation module (2), generate an asynchronous event stream, and convert the asynchronous event stream into an event light field pseudo image;

[0018] The event perception module (3) has a built-in reference perception event generation unit, which is used to dynamically update the light intensity reference value, generate an asynchronous event stream in accordance with the triggering mechanism of the event camera, and improve the mapping accuracy between the event light field pseudo image and the light field information.

[0019] The event camera has a pixel resolution of 1280×720 or higher, an event throughput of 1 Gevents / sec or higher, and a dynamic range of 120dB or higher.

[0020] Furthermore, the control module (4) includes a main control chip, a synchronization triggering unit, and a data cache unit; the main control chip is electrically connected to the synchronization triggering unit and the data cache unit respectively;

[0021] The control module (4) is used to send an encoding mask generation instruction to the high-speed modulation module (2) and a synchronization acquisition instruction to the event sensing module (3) to achieve precise synchronization between the modulation of the digital micromirror device (DMD) (21) and the event acquisition. At the same time, it buffers the event light field pseudo image data output by the event sensing module (3) and transmits the event light field pseudo image data to the domain adaptive reconstruction module (5).

[0022] Furthermore, the domain adaptive reconstruction module (5) is an embedded computing unit or a cloud computing unit, which is electrically connected to the control module (4); it has a pre-stored multi-stage domain adaptive depth equalization (domain adaptive depth equalization model) reconstruction model, which is used to receive compressed and encoded event light field pseudo image data and recover high-fidelity 5D dynamic event light field data from compressed and encoded 4D event light field pseudo image data through the multi-stage domain adaptive depth equalization model.

[0023] The domain adaptive reconstruction module (5) includes three sub-modules in sequence: a spatial domain depth equalization model sub-module (51), an angle domain depth equalization model sub-module (52), and a multi-domain fusion depth equalization model sub-module (53). The three sub-modules are executed sequentially, with the output of the previous sub-module serving as the initialization value for the next sub-module, such as... Figure 2 As shown;

[0024] The spatial domain deep equalization model submodule (51) is used to restore the high-fidelity spatial structure of the light field. It includes an integrated scale-aware context (ISC) module, a layer scaling and skip connection path module, and a first spectral normalized convolution (LipCNN-s) module. The ISC module captures the multi-scale spatial context features of the light field, and the LipCNN-s module refines the multi-scale spatial context features. It is used to restore the high-fidelity spatial structure of the light field. Wherein, s represents space.

[0025] The angle domain depth equalization model submodule (52) is used to enhance the angle coherence of the light field and maintain the consistency of light flux between light field viewpoints; it includes layer scaling and skip connection paths and second spectrum normalized convolution (LipCNN-a) to enhance the angle coherence of the light field and maintain the consistency of light flux between light field viewpoints; the low-frequency angle features of the light field are captured through the LipCNN-a module; where a represents the angle;

[0026] The multi-domain fusion deep equalization model submodule (53) is a domain adaptive correction unit, which includes three parallel feature extraction branches: spatial, angular, and epipolar. The features of the three parallel feature extraction branches are fused through learnable adaptive weights, and the spatiotemporal coherence of the light field is enhanced by combining spatiotemporal convolution. At the same time, it compensates for the errors caused by hardware non-ideals, and realizes the domain gap adaptation from simulation to real scene. The hardware non-ideals include optical aberrations, DMD mask and event camera pixel misalignment, and light energy loss.

[0027] A method for acquiring an event light field using the aforementioned event light field acquisition device includes the following steps:

[0028] Step 1, System Initialization:

[0029] Start the control module (4) to complete the parameter configuration and synchronous calibration of the optical imaging module (1), high-speed modulation module (2) and event sensing module (3); load the preset black-first dynamic binary encoding mask sequence through the DMD driving unit (22); initialize the light intensity reference value through the event sensing module (3); load the pre-trained domain adaptive depth equalization model to reconstruct the model through the domain adaptive reconstruction module (5);

[0030] Step 2, Light Field Acquisition and High-Speed ​​Modulation:

[0031] The optical imaging module (1) collects light rays from the target scene and transmits them aberration-free to the high-speed modulation module (2) via the 4f relay optical system (13). The control module (4) sends a synchronous trigger command, and the digital micromirror device (DMD) (21), under the control of the DMD driving unit (22), performs high-speed time-varying modulation on the light rays according to the black-first coding mask sequence to obtain the encoded light field data. :

[0032] ;

[0033] Among them, L k Let be the light field of the k-th frame; For the Hadama product operator, For noise; M k For binary encoding mask; k is the DMD-encoded optical field frame number;

[0034] Step 3, Event Awareness and Data Caching:

[0035] The event camera in the event perception module (3) is precisely synchronized with the modulation process of the high-speed modulation module (2). The event camera captures the light intensity change of the light field after modulation. The light intensity reference value is dynamically updated by the reference perception event generation unit to generate an asynchronous event stream. The asynchronous event stream is then converted into a structured event light field pseudo image.

[0036] ;

[0037] Where Y is the event light field pseudo-image obtained by the event sensor; B is the number of encoded frames of the event light field pseudo-image. Here, C is the threshold operator, C is the event trigger threshold, and X is the single-frame event light field pseudo-image.

[0038] The formula for obtaining the pseudo-image Y of the event light field can be expressed as a linear function:

[0039] ;

[0040] ;

[0041] Where (p,q) represents the spatial dimension of the light field; (u,v) represents the angular dimension; These represent the resolution values ​​for the four dimensions of the light field: p, q, u, and v. It is a noise term; It is the set of real numbers; denoted as , where is the dimension of the vectorized single-frame event light field pseudo-image; denoted as x, where x is the vector representation of the vectorized B-frame event light field pseudo-image; and H is the transformation matrix that maps the coded light field sequence to the event light field pseudo-image. i Let i be the vector of the light field pseudo-image of the event in the i-th frame, where i ranges from 1 to B.

[0042] The optimization problem to be solved Represented as:

[0043] ;

[0044] in, It's noise. The corresponding regularization term is B≥4;

[0045] The event light field pseudo-image data is cached in real time by the data caching unit of the control module (4), and the event light field pseudo-image data is transmitted to the domain adaptive reconstruction module (5) for reconstruction.

[0046] Step 4, Domain Adaptive High-Fidelity Reconstruction:

[0047] The domain adaptive reconstruction module (5) receives the event light field pseudo-image transmitted by the control module (4), inputs the event light field pseudo-image into the pre-trained domain adaptive depth equalization model, and sequentially reconstructs it through the spatial domain depth equalization model sub-module, the angle domain depth equalization model sub-module, and the multi-domain fusion depth equalization model sub-module to obtain dynamic event light field data. ;

[0048] Step 5, Output the results:

[0049] The domain adaptive reconstruction module (5) will recover the dynamic event light field data. Output to terminal devices to achieve 3D event light field perception of the target scene. At the same time, perform subsequent computational optical operations such as light field post-focusing, view synthesis and depth estimation on dynamic event light field data according to actual needs.

[0050] Furthermore, in step 1, the pre-training process of the domain adaptive deep equalization model is as follows:

[0051] Step 1.1, construct the simulation dataset;

[0052] The simulation dataset includes a synthetic event light field dataset, a real event light field dataset, and a measured dataset collected by a hardware prototype.

[0053] Step 1.2: Preprocess the simulation dataset to unify the spatial-angular resolution of the dataset;

[0054] Step 1.3: Using the decoupled optimization paradigm, the spatial domain depth equalization model submodule and the angle domain depth equalization model submodule are pre-trained on the simulation dataset to establish a hardware-independent event light field feature manifold.

[0055] Step 1.4: Fine-tune the multi-domain fusion deep equalization model submodule using the measured dataset collected from the hardware prototype, so that the multi-domain fusion deep equalization model submodule becomes a correction operator for hardware non-idealism, and realizes the domain gap adaptation from simulation to real scene.

[0056] Step 1.5: The Anderson acceleration method is used to solve the fixed point of the domain adaptive deep equilibrium model. The gradient is calculated by implicit differentiation to ensure the constant memory usage and convergence of the domain adaptive deep equilibrium model.

[0057] Furthermore, in step 4, the sequential reconstruction process is as follows:

[0058] Step 4.1, the spatial domain depth equalization model submodule (51) reshapes the spatial dimension of the 4D event light field pseudo-image, captures the multi-scale spatial context features of the light field through the ISC module, refines the multi-scale spatial context features through the LipCNN-s module, and outputs spatially balanced light field data. :

[0059] ;

[0060] in, It is a pre-trained spatial domain depth equalization model submodule. These are the network parameters of the spatial domain depth equilibrium model submodule. It is the step size value. Represents the matrix transpose operation;

[0061] Step 4.2, the angle domain depth equalization model submodule (52) uses the spatial equilibrium light field data as the initial value, reshapes the angle dimension of the initial value, captures the low-frequency angle features of the light field through the LipCNN-a module, enhances the angle coherence of the light field, and outputs the angle equilibrium light field data. :

[0062] ;

[0063] in, It is a pre-trained angle-domain depth equalization model submodule. These are the network parameters of the angle domain depth equalization model submodule. It is the step size value.

[0064] Step 4.3, the multi-domain fusion deep equalization model submodule (53) uses the angle equilibrium state light field data as the initial value, extracts light field features through three parallel branches: space, angle and epipolar, and uses learnable adaptive weights to fuse multi-domain features. It combines spatiotemporal convolution to enhance the spatiotemporal coherence of the light field, while compensating for errors caused by hardware non-ideality, and outputs high-fidelity dynamic event light field data. :

[0065] ;

[0066] in, It is a pre-trained multi-domain fusion deep equalization model submodule. These are the parameters of the sub-module network of the multi-domain fusion deep equilibrium model. It is the step size value.

[0067] Compared with the prior art, the technical effects of the present invention are as follows:

[0068] (1) Integrating the advantages of dual technologies to break through the bottleneck of resolution trade-off: Combining the advantages of high temporal resolution and high dynamic range of the event camera with the advantages of high-speed snapshot compression modulation of DMD, it not only achieves the frame rate improvement of 5D dynamic light field, but also decouples the inherent trade-off of spatial-angle-temporal resolution through the asynchronous sensing characteristics of the event camera, so as to improve temporal resolution without sacrificing spatial and angle resolution.

[0069] (2) Strong robustness in low light / high dynamic range: Using an event camera as the light field sensing unit, it does not need to rely on the absolute light intensity of the intensity image, but only captures the relative information of light intensity changes. Combined with the black-first coding mask design of DMD, the light efficiency is improved. The sensing performance in low light, high contrast and high dynamic range scenarios far exceeds that of traditional CMOS light field cameras.

[0070] (3) High hardware integration and strong engineering feasibility: It abandons the microlens array and hybrid camera architecture of traditional light field cameras and adopts a minimalist hardware architecture of "imaging lens + DMD + event camera". It realizes aberration-free transmission of light field rays through a 4f relay optical system, without the need for precise calibration of multiple optical components. The hardware is small in size and light in weight, and can be adapted to the deployment requirements of embedded devices.

[0071] (4) High reconstruction accuracy and strong robustness: The multi-stage domain adaptive depth equalization reconstruction model is adopted, which decomposes the light field reconstruction into three sequential stages: space, angle and multi-domain fusion. The spatial, angle and epipolar geometric priors of the light field are explicitly used. At the same time, the multi-domain fusion sub-module is used as a correction operator for hardware non-ideal, which can effectively compensate for errors such as optical aberration and mask misalignment, and realize the domain gapless adaptation from simulation to real scene. The peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) of the reconstructed light field are improved.

[0072] (5) Lightweight model with high inference efficiency: The domain adaptive deep equalization model achieves infinite effective network depth through implicit layer formula while maintaining constant memory usage (O(1)). Combined with the lightweight modules of ISC and LipCNN, the number of model parameters and computational cost are greatly reduced. Compared with traditional deep unfolded networks, the inference efficiency is improved by more than 50%, and real-time reconstruction can be achieved in embedded computing units.

[0073] (6) Wide adaptability to dynamic scenes: The high-speed modulation of DMD (refresh rate not less than 20kHz) combined with the high temporal resolution of the event camera can shorten the light field measurement time to less than 30ms. It can not only adapt to linear motion scenes, but also achieve artifact-free reconstruction of complex dynamic scenes such as low-speed rotation and slight non-rigid motion. Through hardware upgrades, it can be further adapted to high-speed dynamic scenes.

[0074] (7) Flexible coding and modulation with strong scalability: The dynamic binary coding mask sequence of DMD can be flexibly configured by software, supports adaptive adjustment of compression ratio, and can adjust the temporal resolution of light field according to actual scene requirements. At the same time, the permutation invariance of coding mask makes the model cross-hardware migration cost low, and only a few hyperparameters need to be fine-tuned to adapt to different types of event cameras and DMD. Attached Figure Description

[0075] Figure 1 This is a schematic diagram of the overall structure of the event light field acquisition device of the present invention;

[0076] Figure 2 This is a schematic diagram of the network structure of the domain adaptive reconstruction module of the present invention;

[0077] 1 is the optical imaging module, 11 is the imaging lens, 12 is the aperture stop, and 13 is the 4f relay optical system; 2 is the high-speed modulation module, 21 is the DMD, and 22 is the DMD driving unit; 3 is the event sensing module; 4 is the control module; and 5 is the domain adaptive reconstruction module. Detailed Implementation

[0078] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0079] Example 1 Event Light Field Acquisition Device

[0080] like Figure 1 As shown, this embodiment provides an event light field acquisition device, including an optical imaging module 1, a high-speed modulation module 2, an event sensing module 3, a control module 4, and a domain adaptive reconstruction module 5. The modules are electrically or optically connected. The control module 4 is the core control unit of the entire device, used to coordinate the synchronous operation of the modules. The terminal device 7 is electrically connected to the domain adaptive reconstruction module 5 and is used to receive and display the 5D dynamic event light field data output by the domain adaptive reconstruction module 5.

[0081] The optical imaging module 1 includes an imaging lens 11, an aperture 12, and a 4f relay optical system 13 connected in sequence. The imaging lens 11 is a plano-convex lens with f=100mm, and the parameters of the 4f relay optical system 13 are f1=125mm and f2=60mm. The imaging lens 11 is used to collect the light field rays of the target scene 6, the aperture 12 is used to adjust the amount of light entering, and the 4f relay optical system 13 is used to realize the aberration-free transmission of the light field rays, so that the modulation surface of the DMD21 of the high-speed modulation module 2 and the detection surface of the event camera 31 of the event sensing module 3 form an optical conjugate, ensuring that the modulated light field rays are accurately projected onto the pixel surface of the event camera 31.

[0082] The high-speed modulation module 2 includes a DMD 21 and a DMD driver unit 22. The DMD 21 uses a UPOLabs HDSLM108D95-Smart digital micromirror device with a refresh rate of 22kHz, a resolution of 1920×1080, and a micromirror spacing of 10.8μm. The DMD driver unit 22 is electrically connected to the synchronization trigger unit 42 of the control module 4, and generates a black-first dynamic binary encoded mask sequence (e.g., ...) under the instructions of the control module 4. Figure 2 As shown, the first part of the sequence is a completely black mask, and the rest are random binary masks (the number of masks is B=4). High-speed time-varying modulation is applied to the light field transmitted by the optical imaging module 1 to compress and encode the 5D dynamic light field information of 4 time frames into a single 4D light field measurement value.

[0083] The event sensing module 3 includes an event camera, which is a Prophesee EVK4 event camera with a resolution of 1280×720, an event throughput of 1.066G events / sec, and a dynamic range of 140dB. The event camera is electrically connected to the synchronization trigger unit 42 of the control module 4 to achieve precise synchronization with the DMD21 modulation process.

[0084] The control module 4 includes a main control chip, a synchronization trigger unit, and a data cache unit. The main control chip is an XC7Z045 field-programmable gate array (FPGA), and the data cache unit is a 16G DDR4 memory. The main control chip 41 is the core unit of the control module 4. The synchronization trigger unit is used to generate a precise synchronization trigger signal to ensure that the time synchronization error between DMD21 modulation and event camera acquisition is ≤1μs. The data cache unit is used to cache the event light field pseudo image data output by the event camera in real time and transmit the event light field pseudo image data to the domain adaptive reconstruction module 5 via Gigabit Ethernet.

[0085] Domain adaptive reconstruction module 5 is an NVIDIA Jetson AGX Orin type embedded computing unit, which internally stores a pre-trained domain adaptive deep equalization model, the model structure of which is as follows. Figure 2 As shown, it includes a spatial domain depth equalization model submodule 51, an angle domain depth equalization model submodule 52, and a multi-domain fusion depth equalization model submodule 53, with the three submodules executing sequentially. Among them, the spatial domain depth equalization model submodule 51 integrates the ISC module and the LipCNN_s module, the angle domain depth equalization model submodule 52 integrates the LipCNN_a module, and the multi-domain fusion depth equalization model submodule 53 integrates three parallel light field feature extraction branches for space, angle, and epipolar lines, as well as a multi-space LipCNN module. The features of the three parallel branches are fused through learnable adaptive weights to compensate for errors caused by hardware non-ideals.

[0086] Example 2: Method for Acquiring Event Light Field

[0087] Based on the event light field acquisition device of Embodiment 1, this embodiment provides an event light field acquisition method, including the following steps:

[0088] System initialization: Start the main control chip 41 of control module 4 and complete the parameter configuration of each module: adjust the aperture of imaging lens 11 to F1.4, adjust the light intake of aperture 12 to 80%, and complete the optical calibration of 4f relay optical system 13; load the preset black-priority dynamic binary encoded mask sequence (B=4, the first bit is a full black mask, and the rest are random binary masks) through DMD drive unit 22; initialize the light intensity reference value through event camera. The contrast threshold τ=0.30 and the dark current bias ε=0.01 were set. The pre-trained domain adaptive depth equalization model was loaded and reconstructed through the domain adaptive reconstruction module 5. The spatial resolution of the model is 456×241 and the angular resolution is 8×8.

[0089] The pre-training process of the domain adaptive deep equalization model is as follows:

[0090] Step 1.1: Construct a simulation dataset, which includes the Sintel synthetic light field dataset, the Lytro real light field dataset, and the measured dataset collected by the hardware prototype in this embodiment. All datasets are preprocessed to a spatial resolution of 456×241 and an angular resolution of 8×8.

[0091] Step 1.2: Using the decoupled optimization paradigm, the spatial domain depth equalization model submodule and the angular domain depth equalization model submodule are pre-trained on the simulation dataset. The Adam optimizer is selected, the initial learning rate is set to 1×10⁻³, the training epochs are 20, and a hardware-independent light field feature manifold is established.

[0092] Step 1.3: Fine-tune the multi-domain fusion deep equalization model submodule using the measured dataset collected from the hardware prototype, so that the multi-domain fusion deep equalization model submodule becomes a correction operator for hardware non-idealism, and realizes the domain gap adaptation from simulation to real scene.

[0093] Step 1.4: During the training process, the Anderson acceleration method is used to solve the fixed point of the deep equilibrium model. The acceleration coefficient is set to 0.8, and the gradient is calculated by implicit differentiation to ensure the constant memory usage of the model.

[0094] Light field acquisition and high-speed modulation: The light field rays of the target scene are collected by the imaging lens 11 of the optical imaging module 1. The target scene is a 3D model rotating at a low speed of 1.75 rpm. The light field rays are transmitted to the DMD 21 without aberration through the aperture 12 and the 4f relay optical system 13. The synchronization trigger unit 42 of the control module 4 sends a synchronization trigger signal. Under the control of the DMD drive unit 22, the DMD 21 performs high-speed time-varying modulation on the light field rays according to the black priority coding mask sequence, with a modulation frequency of 20 kHz.

[0095] Step 3, Domain Adaptive High-Fidelity Reconstruction: The domain adaptive reconstruction module 5 receives the event light field pseudo-image data transmitted by the control module 4 and inputs it into the domain adaptive depth equalization model reconstruction model. Sequential reconstruction is then performed through the spatial domain depth equalization model submodule 51, the angle domain depth equalization model submodule 52, and the multi-domain fusion depth equalization model submodule 53.

[0096] Step 3.1, the spatial domain depth equalization model submodule 51 reshapes the spatial dimension of the event light field pseudo-image, transforming it into a tensor of (B×U×V)×C×H×W. The ISC module captures the multi-scale spatial context features of the light field, LayerScale and DropPath ensure the stability of model training, and the LipCNN_s module refines the features. The solution is iteratively solved until the spatial equilibrium state is reached, and the spatial equilibrium state light field data is output.

[0097] Step 3.2: The angle domain depth equalization model submodule 52 uses the spatial equilibrium state light field data as the initial value, reshapes its angle dimension, and transforms it into a tensor of (B×H×W)×C×U×V. The low-frequency angle features of the light field are captured by the LipCNN_a module to enhance the consistency of light flux between viewpoints. The solution is iteratively solved to the angle equilibrium state and the angle equilibrium state light field data is output.

[0098] Step 3.3: The multi-domain fusion deep equalization model submodule 53 uses the angle equilibrium state light field data as the initial value, extracts the spatial, angular and epipolar features of the light field through three parallel branches: space, angle and epipolar. It uses learnable adaptive weights to fuse the multi-domain features, and then uses the multi-space LipCNN module combined with the ISC module to enhance the spatiotemporal coherence of the light field. At the same time, it compensates for the errors caused by DMD mask inaccuracy and optical aberrations. It iteratively solves to the global equilibrium state and outputs dynamic event light field data of 456×241×8×8×4.

[0099] Step 4, Result Output: The domain adaptive reconstruction module 5 transmits the recovered dynamic event light field data to the terminal device via an HDMI interface. The terminal device is a 4K display. The terminal device realizes real-time display of the light field and can perform operations such as post-focusing, view compositing, and depth estimation on the light field data according to actual needs. According to the test, the average PSNR of the reconstructed dynamic event light field in this embodiment is 30.52dB and SSIM is 0.845. Compared with the existing pure event light field reconstruction method, the PSNR is improved by 2.69dB and the SSIM is improved by 0.059, and there are no motion artifacts or aberration artifacts.

Claims

1. An event light field acquisition device, characterized in that, It includes an optical imaging module (1), a high-speed modulation module (2), an event sensing module (3), a control module (4), and a domain adaptive reconstruction module (5); the optical imaging module (1) is connected to the high-speed modulation module (2), the high-speed modulation module (2) is connected to the control module (4), the control module (4) is connected to the event sensing module (3); the event sensing module (3) is connected to the domain adaptive reconstruction module (5); the domain adaptive reconstruction module (5) is connected to the control module (4).

2. The event light field acquisition device according to claim 1, characterized in that, All connections are either electrical or optical.

3. The event light field acquisition device according to claim 1, characterized in that, The optical imaging module (1) includes, in sequence, an imaging lens (11), an aperture (12), and a 4f relay optical system (13); the imaging lens (11), the aperture (12), and the 4f relay optical system are optically connected. The imaging lens (11) collects the light field and transmits it to the aperture (12). The aperture (12) then transmits the light field to the high-speed modulation module (2) and the event sensing module (3) respectively through aberration-free transmission, so that the modulation surface of the high-speed modulation module (2) and the detection surface of the event sensing module (3) form an optical conjugate.

4. The event light field acquisition device according to claim 1, characterized in that, The high-speed modulation module (2) includes a digital micromirror device (DMD) (21) and a DMD driving unit (22); the digital micromirror device (DMD) (21) and the DMD driving unit (22) are electrically connected; The digital micromirror device (DMD) (21) is electrically connected to the optical imaging module (1) and receives the light field transmitted by the optical imaging module (1); the DMD driving unit (22) is electrically connected to the control module (4) and generates a dynamic binary encoding mask under the instruction of the control module (4) to perform high-speed time-varying control on the light field transmitted by the optical imaging module (1); and compresses and encodes the 5D dynamic light field information of multiple time frames into a single 4D light field measurement value; The dynamic binary coding mask generated by the high-speed modulation module (2) contains a black mask pattern; the black mask pattern is fixed at the beginning of the coding mask sequence to form a black priority coding sequence.

5. The event light field acquisition device according to claim 1, characterized in that, The event sensing module (3) is an event camera. The event camera is electrically connected to the control module (4) and is used to capture the light intensity changes of the light field light after being modulated by the high-speed modulation module (2), generate an asynchronous event stream, and convert the asynchronous event stream into an event light field pseudo image.

6. The event light field acquisition device according to claim 1, characterized in that, The control module (4) includes a main control chip, a synchronization triggering unit, and a data cache unit; the main control chip is electrically connected to the synchronization triggering unit and the data cache unit respectively; The control module (4) is used to send an encoding mask generation instruction to the high-speed modulation module (2) and a synchronization acquisition instruction to the event sensing module (3) to achieve precise synchronization between the modulation of the digital micromirror device (DMD) (21) and the event acquisition. At the same time, it buffers the event light field pseudo image data output by the event sensing module (3) and transmits the event light field pseudo image data to the domain adaptive reconstruction module (5).

7. The event light field acquisition device according to claim 1, characterized in that, The domain adaptive reconstruction module (5) is an embedded computing unit or a cloud computing unit, which is electrically connected to the control module (4); it has a pre-stored multi-stage domain adaptive depth equalization reconstruction model, which is used to receive compressed and encoded event light field pseudo image data, and recover high-fidelity 5D dynamic event light field data from compressed and encoded 4D event light field pseudo image data through the multi-stage domain adaptive depth equalization model. The domain adaptive reconstruction module (5) includes three sub-modules in sequence: spatial domain depth equalization model sub-module (51), angle domain depth equalization model sub-module (52), and multi-domain fusion depth equalization model sub-module (53). The three sub-modules are executed sequentially, and the output of the previous sub-module is used as the initialization value of the next sub-module. The spatial domain deep equalization model submodule (51) is used to recover the high-fidelity spatial structure of the light field. It includes the ISC module, layer scaling and skip connection path and LipCNN-s module in sequence. The ISC module captures the multi-scale spatial context features of the light field, and the LipCNN-s module refines the multi-scale spatial context features. The angle domain depth equalization model submodule (52) is used to enhance the angle coherence of the light field and maintain the consistency of light flux between light field viewpoints; it includes layer scaling and jump connection path and LipCNN-a module, which are used to enhance the angle coherence of the light field and maintain the consistency of light flux between light field viewpoints; the LipCNN-a module captures the low-frequency angle features of the light field; The multi-domain fusion deep equalization model submodule (53) is a domain adaptive correction unit, which includes three parallel feature extraction branches: space, angle and epipolar. The features of the three parallel feature extraction branches are fused by learnable adaptive weights, and the spatiotemporal coherence of the light field is enhanced by spatiotemporal convolution, while compensating for errors caused by hardware.

8. A method for acquiring an event light field using the event light field acquisition device according to any one of claims 1 to 7, comprising the following steps: Step 1, System Initialization: Start the control module (4) to complete the parameter configuration and synchronous calibration of the optical imaging module (1), high-speed modulation module (2) and event sensing module (3); load the preset black-first dynamic binary encoding mask sequence through the DMD driving unit (22); initialize the light intensity reference value through the event sensing module (3); load the pre-trained domain adaptive depth equalization model to reconstruct the model through the domain adaptive reconstruction module (5); Step 2, Light Field Acquisition and High-Speed ​​Modulation: The optical imaging module (1) collects light rays from the target scene and transmits them aberration-free to the high-speed modulation module (2) via the 4f relay optical system (13). The control module (4) sends a synchronous trigger command, and the digital micromirror device (DMD) (21), under the control of the DMD driving unit (22), performs high-speed time-varying modulation on the light rays according to the black-first coding mask sequence to obtain the encoded light field data. : ； in, L k Let be the light field of the k-th frame; For the Hadama product operator, For noise; M k For binary encoding mask; k is the DMD-encoded optical field frame number; Step 3, Event Awareness and Data Caching: The event camera in the event perception module (3) is precisely synchronized with the modulation process of the high-speed modulation module (2). The event camera captures the light intensity change of the light field after modulation. The light intensity reference value is dynamically updated by the reference perception event generation unit to generate an asynchronous event stream. The asynchronous event stream is then converted into a structured event light field pseudo image. ； Where Y is the event light field pseudo-image obtained by the event sensor; B is the number of encoded frames of the event light field pseudo-image. Here, C is the threshold operator, C is the event trigger threshold, and X is the single-frame event light field pseudo-image. The formula for obtaining the pseudo-image Y of the event light field can be expressed as a linear function: ； ； Where (p,q) represents the spatial dimension of the light field; (u,v) represents the angular dimension; These represent the resolution values ​​for the four dimensions of the light field: p, q, u, and v. It is a noise term; It is the set of real numbers; denoted as , where is the dimension of the vectorized single-frame event light field pseudo-image; denoted as x, where x is the vector representation of the vectorized B-frame event light field pseudo-image; and H is the transformation matrix that maps the coded light field sequence to the event light field pseudo-image. i Let i be the vector of the light field pseudo-image of the event in the i-th frame, where i ranges from 1 to B. The optimization problem to be solved Represented as: ； in, It's noise. The corresponding regularization term is B≥4; The event light field pseudo-image data is cached in real time by the data caching unit of the control module (4), and the event light field pseudo-image data is transmitted to the domain adaptive reconstruction module (5) for reconstruction. Step 4, Domain Adaptive High-Fidelity Reconstruction: The domain adaptive reconstruction module (5) receives the event light field pseudo-image transmitted by the control module (4), inputs the event light field pseudo-image into the pre-trained domain adaptive depth equalization model, and sequentially reconstructs it through the spatial domain depth equalization model sub-module, the angle domain depth equalization model sub-module, and the multi-domain fusion depth equalization model sub-module to obtain dynamic event light field data. ; Step 5, Output the results: The domain adaptive reconstruction module (5) will recover the dynamic event light field data. Output to terminal devices.

9. The event light field acquisition method according to claim 8, in step 1, the pre-training process of the domain adaptive depth equalization model is as follows: Step 1.1, construct the simulation dataset; The simulation dataset includes a synthetic event light field dataset, a real event light field dataset, and a measured dataset collected by a hardware prototype. Step 1.2: Preprocess the simulation dataset to unify the spatial-angular resolution of the dataset; Step 1.3: Using the decoupled optimization paradigm, the spatial domain depth equalization model submodule and the angle domain depth equalization model submodule are pre-trained on the simulation dataset to establish a hardware-independent event light field feature manifold. Step 1.4: Fine-tune the multi-domain fusion deep equalization model submodule using the measured dataset collected from the hardware prototype, so that the multi-domain fusion deep equalization model submodule becomes a hardware correction operator. Step 1.5: Solve for the fixed point of the domain adaptive deep equilibrium model using the Anderson acceleration method, and calculate the gradient using the implicit differential method to ensure the constant memory usage and convergence of the domain adaptive deep equilibrium model.

10. The event light field acquisition method according to claim 8, wherein in step 4, the sequential reconstruction process is as follows: Step 4.1, the spatial domain depth equalization model submodule (51) reshapes the spatial dimension of the 4D event light field pseudo-image, captures the multi-scale spatial context features of the light field through the ISC module, refines the multi-scale spatial context features through the LipCNN-s module, and outputs spatially balanced light field data. : ； in, It is a pre-trained spatial domain depth equalization model submodule. These are the network parameters of the spatial domain depth equilibrium model submodule. It is the step size value. Represents the matrix transpose operation; Step 4.2, the angle domain depth equalization model submodule (52) uses the spatial equilibrium light field data as the initial value, reshapes the angle dimension of the initial value, captures the low-frequency angle features of the light field through the LipCNN-a module, enhances the angle coherence of the light field, and outputs the angle equilibrium light field data. : ； in, It is a pre-trained angle-domain depth equalization model submodule. These are the network parameters of the angle domain depth equalization model submodule. It is the step size value; Step 4.3, the multi-domain fusion deep equalization model submodule (53) uses the angle equilibrium state light field data as the initial value, extracts light field features through three parallel branches: space, angle and epipolar, and uses learnable adaptive weights to fuse multi-domain features. It combines spatiotemporal convolution to enhance the spatiotemporal coherence of the light field, while compensating for errors caused by hardware non-ideality, and outputs high-fidelity dynamic event light field data. : ； in, It is a pre-trained multi-domain fusion deep equalization model submodule. These are the parameters of the sub-module network of the multi-domain fusion deep equilibrium model. It is the step size value.

Images