Three-dimensional imaging method and system in extreme illumination environment

Abstract

The invention discloses a three-dimensional imaging method and system in an extreme illumination environment, and the system comprises an EvHGen model, and the EvHGen model comprises an importance sampling and dynamic embedding module, a space-time structure estimator, and a hierarchical attention generator. According to the invention, the problems of limitation and non-generalization during 3D reconstruction in an extreme illumination scene at present are solved. The method can still effectively generate Gaussian primitives and reconstruct scenes in too bright and too dark scenes, and can be generalized to untrained scenes, a comprehensive experiment of a reference data set shows that EvHGen obtains optimal performance, and EvHGen is the first method capable of performing real-time (Real-Time, > 30FPS) three-dimensional imaging from an unfamiliar scene event stream in real time. And the performance and the efficiency reach the optimal level at present.

Description

Technical Field

[0001] The present invention relates to the field of imaging, and in particular to a three-dimensional imaging method in extreme lighting environments. Background Art

[0002] Existing 3D imaging methods rely on traditional cameras capturing stable, ideal frame sequences under static conditions. However, in environments with rapid motion (such as cars and drones) and extreme lighting (excessive light / darkness), traditional camera image quality degrades significantly, impairing the quality of 3D images. Unlike traditional cameras, which capture images at a constant rate, event sensors, an emerging class of neuromorphic vision sensors, asynchronously measure brightness changes at each pixel, encoding the timing, location, and polarity of these changes into an event stream. Event sensors offer microsecond-level temporal resolution and a dynamic range of up to 140 dB. In comparison, traditional cameras typically achieve only millisecond-level temporal resolution and a dynamic range of approximately 60 dB. Event sensors, with their advantages of high temporal resolution and wide dynamic range, have sparked significant research interest. However, event streams differ fundamentally from traditional frame-based intensity images not only in terms of data type but also in their processing methods, making direct transfer of existing frame-based 3D reconstruction methods difficult. To address this, several cutting-edge research approaches have proposed novel approaches. Some leverage event streams to effectively eliminate motion blur in traditional RGB imaging, while others go further and reconstruct the 3D shape of objects directly from event data. However, these methods, like many existing 3D imaging methods, are based on optimization and cannot be generalized to unseen scenes. The difficulty lies in learning how to learn the necessary representations for 3D reconstruction from sparse inputs and reconstruct the 3D shape in the forward process. Some new works have proposed generalizable feedforward methods based on RGB frames and events, which use multi-branch or hierarchical structures to learn point cloud matching from images, estimate structural features, and then perform 3D imaging. Event-based generalizable methods still refer to the RGB frame-based paradigm, accumulating events into frames in the time dimension, discarding the rich temporal information contained in the events.

[0003] In general, the existing technology has two flaws: in extreme lighting and fast-motion scenes, frame-based camera imaging blur will weaken the three-dimensional imaging quality; each scene must be optimized separately and cannot be generalized to unseen scenes.

[0004] Glossary:

[0005] Selective State Space Model Mamba: selectively processes information based on the current input through a recurrent model, thereby focusing on relevant data and discarding unimportant information. Summary of the Invention

[0006] In order to solve the above problems, the present invention discloses a three-dimensional imaging method under extreme lighting environments.

[0007] To achieve the above object, the technical solution of the present invention is:

[0008] A 3D imaging system for extreme lighting environments, including an EvHGen model, which includes an importance sampling and dynamic embedding module, a spatiotemporal structure estimator, and a hierarchical attention generator;

[0009] The importance sampling and dynamic embedding module is used to input the spatiotemporal event sequence E N Obtain spatiotemporal importance embedding χ st ;

[0010] The spatiotemporal structure estimator is used to calculate the event frame E according to the input f , voxel grid E vg and spatiotemporal importance embedding χ st Obtain the spatiotemporal structure characteristics χ st-e ;

[0011] The hierarchical attention generator is used to generate the event frame E according to the input f , event voxel grid E vg and spatiotemporal structural characteristics χ st-e Get parameter group Then, differentiable rendering is performed by parameter group Synthesize new views

[0012] in is the depth map mapped by the Gaussian primitives, is the surface intensity of the Gaussian element, is the rotation of the Gaussian basis element; is the scaling of the Gaussian basis, is the opacity matrix of the Gaussian basis element, is a real number.

[0013] As a further improvement, the input of the importance sampling and dynamic embedding module is a spatiotemporal event sequence of length N. Importance sampling and dynamic embedding module through the importance function module E N Encode into dense representation and calculate density probability, then take the event sequence corresponding to the top k items with the largest density probability to form the sampled event E N Input the multi-resolution hash encoder to get E N Hash feature, projection module E k The polarity part of the projected into the polarity part hash feature, the polarity part hash feature and E N The hash feature dimensions of E are the same;N The hash feature and the polarity part hash feature are concatenated and used Mapping to spatiotemporal importance embedding The importance function module According to the data processing direction, it includes normalization layer, linear layer, ReLu activation layer, linear layer and Softmax layer; projection module Consists of a linear layer Linear;

[0014] in, The multi-resolution hash encoder process is as follows:

[0015]

[0016] Splicing to get χ st The method is as follows:

[0017]

[0018] e k' is the k'th event, e k' =(x k' ,t k' ,p k' ), x k is the k'th event coordinate, consisting of x, y, t k is the time when the k'th event occurs, p k is the polarity of the k'th event, e j Indicates E k The jth element in represents the multi-resolution hash encoder, E k|x,y,t Represents the x-element set, y-element set, and t-element set in k events, where x represents the x-coordinate of the event, y represents the y-coordinate of the event, t represents the time when the event occurs, and s l represents the scaling factor of the lth layer, E k,l|x represents the x element of the lth layer in event k, p1 represents the first element of the prime number set p, E k,l|y represents the y element of the lth layer in event k, p2 represents the second element of the prime number set p, E k,l|t Indicates the t element of the lth layer in event k, p3 represents the third element of the prime number set p, T represents the size of the hash table, mod represents modulus; cat() represents splicing, p3 represents the third element of the prime number set p, Represents a mapping module.

[0019] A further improvement is that the input of the spatiotemporal structure estimator is the spatiotemporal event sequence E N The corresponding event frame E f , event voxel grid Evg and spatiotemporal importance embedding χ st ; The spatiotemporal structure estimator first projects the spatial embedding The event frame E f With voxel grid E vg Projected from pixel space into spatiotemporal importance embedding χ st Latent space embedding χ of the same dimension ep , and then embed the latent space into χ ep Added 2D sine-cosine position encoding Marked with χ ep The corresponding position of each value in the original space; then the event space-time feature χ st Fusion is performed to obtain fusion features; token-level masks are calculated through event frames Actively shield the non-event areas of the fused features to obtain a representation of the temporal-spatial information focused on the effective events Then, the selective mechanism state space model is used to model the scene space latent state into a time-varying model, and the spatiotemporal structural characteristics of the scene χ are learned. st-e ;

[0020]

[0021] Mamba[] represents the state space model of the selective mechanism;

[0022] The data processing method of the core layer SSM of the selective mechanism state space model is as follows:

[0023]

[0024] C are all learnable parameters, u(t) represents the input at the current moment, is the potential state of the scene space at the current moment, is the time-varying state transfer matrix, C represents the observation matrix; y(t) is the observation matrix C that maps the current state observation value from the scene space latent state.

[0025] As a further improvement, the EvHGen model is trained through a joint spatiotemporal and structural supervision mechanism;

[0026] For imaging frames Use L1 loss and structural similarity index loss SSIM to synthesize supervision to approximate the true value I;

[0027]

[0028] For surface strength use Loss to supervise to approximate the true value I

[0029]

[0030] For point cloud locations, L1 loss and SSIM are used to synthesize the depth map of the supervised point cloud. To approximate the true value D.

[0031]

[0032] Represents imaging frame The loss function, λ is the balance coefficient, set to 0.3, Surface strength The loss function is Depth map The loss function of

[0033] The spatiotemporal perception loss is as follows:

[0034]

[0035] φ μ and φ σ Respectively represent the image mean feature and variance feature calculated using vgg16; use cos() to calculate the cosine similarity of the mean feature, constrain the directional consistency of the feature vector, and reflect the similarity of the main spatiotemporal structure; use D KL (‖) Calculate the KL divergence of the variance feature to constrain the diversity of the spatiotemporal distribution of features and avoid excessive concentration of the generated results on the features at a certain moment; Indicates the calculation of rendering frame and the event time surface E suf The average of cosine similarity and KL divergence in feature space, E suf represents an event-time surface;

[0036] Training makes The smallest one is the trained EvHGen model.

[0037] For further improvement, the value of λ is set to 0.3.

[0038] A 3D imaging method in extreme lighting environments, which converts the spatiotemporal event sequence E N , event frame E f and the voxel grid E vg Input the above trained EvHGen model to get a new view

[0039] Advantages of the present invention:

[0040] Unlike input methods that only use event frames / voxel grids, which lack sufficient temporal information, we directly sample from the raw event stream of variable length, learning fixed-dimensional spatiotemporal importance embeddings from the temporally rich spatiotemporal structural signals. We then learn spatial embeddings with dense spatial features from the spatially dense event frames and interpolated voxel grids. The spatiotemporal importance embeddings are integrated into the spatial embeddings to form features rich in spatiotemporal information. A time-varying state-space mechanism is used to more flexibly capture the dynamic evolution of event data, from which state-space features are learned. Through hierarchical attention, point clouds, surface colors, and other Gaussian parameters are sequentially generated from event features. Spatiotemporal structural supervision is then used to supervise the learning of point clouds, surface colors, and rendered images for joint training. EvHGen can generate Gaussian primitives from input events in unseen scenarios online in real time (>30 FPS) and render them into new views, outperforming existing methods in both performance and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is the framework diagram of EvHGen.

[0042] Figure 2 This is a visual comparison of EvHGen and EvGGS (Reference 3) in complex lighting scenes. EvHGen's imaging effect is closer to the real situation.

[0043] Figure 3 A visual comparison of EvHGen and EvGGS in a scene with complex lighting and object textures. EvGGS performs very poorly in this scenario, while EvHGen can still generate stable images, and its results are very close to the real situation. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0045] The EvHGen proposed in the present invention is mainly composed of three major components: Importance Sampling and Dynamic Embedding Module (ISDE), Spatiotemporal Structure Estimator (STSE) and Hierarchical Attention Generator (HAGen). Importance Sampling and Dynamic Embedding directly learn rich temporal features from the original event stream, and map the variable-length event sequence into a fixed-length spatiotemporal importance embedding. The spatiotemporal structure estimator first learns the spatial features of the event frame and event voxel grid, and then fuses them with the spatiotemporal importance embedding, learns the state space expression of the scene from the fused features, and outputs the spatiotemporal structure features. Finally, using hierarchical attention, queries with different Gaussian parameters are calculated in turn, and the point cloud position, surface intensity, rotation, scale and opacity matrix are estimated step by step from the original event to form a Gaussian basis tuple. Three-dimensional imaging is completed through the above process.

[0046] The training dataset of the present invention is created in a poorly illuminated environment with a high motion speed. A portion of the dataset is captured by an event sensor (model DVXplore) to obtain a spatiotemporal event sequence E N , event frame E f , event voxel grid E vg and the event time surface E suf The other part is built and rendered using Blender software, which contains a large number of realistic objects with complex geometric diversity and texture information. The scene is placed at the origin of the coordinate system, and a virtual camera is placed around the scene and circled to collect each frame. The rendered image is converted into event data using v2e software under the default configuration to obtain the spatiotemporal event sequence E. N , event frame E f , event voxel grid E vg and the event time surface E suf The data volume ratio of these two parts is 1:1.

[0047] (1) Importance Sampling and Dynamic Embedding Module (ISDE)

[0048] Compared to dense intensity frames, sparse event data only expresses changes in brightness. Existing event preprocessing methods mainly include event frames and event voxel grids. Events are accumulated in a spatiotemporal neighborhood to form an event frame, but this folds all events into a 2D grid, discarding temporal information. Voxel grids, on the other hand, bucket the event stream by time and independently aggregate events within each time slice to form a 3D tensor similar to a video frame, retaining some temporal information. The event data processed by these two processing methods can be directly applied to existing visual methods, but still cannot fully retain the rich temporal information in the event.

[0049] To solve this problem, this paper proposes an Importance Sampling and Dynamic Embedding (ISDE) mechanism to learn and sample the most important parts from any event sequence of variable length and encode them into fixed-length embeddings. As shown in Formula 1, for a spatiotemporal event sequence of length N, Learnable Importance Function Module Encode it into a dense representation and calculate the density probability. Then take the event sequence corresponding to the first k items with the largest density probability to form a new sequence k is 1 / 256 of the total number of pixels in the picture, representing the most important part of this sequence of events.

[0050]

[0051] Since event data is sparse in space and dense in time, processing requires both efficient data storage and capturing information at different time and spatial scales. We use a multi-resolution hash encoder to address these challenges. Specifically, as shown in Formula 2, an 8-level hash parameter mapping is used to encode the coordinates and time of the input event. For each level l, the coordinates and time are mixed by a prime number set p and modulo the hash table size T. l , generate a hash index to ensure that the mapping is within the valid range. The size of the hash table at each level increases with the resolution. After extracting the feature from the hash table according to the index, multiply it by the level-related scaling factor S l , and then fuse 8 levels of hash features.

[0052]

[0053] Among them, the prime number set p={1,2654435761,805459861}, Indicates exclusive OR.

[0054] As shown in Formula 3, The sampled event E k The polar part is projected into the same dimension as the hash feature and concatenated using Mapping to spatiotemporal importance embedding Characterize the most important parts of input events and their rich spatiotemporal importance features.

[0055]

[0056] (2) Spatiotemporal Structure Estimator (STSE)

[0057] Event frames and event voxel grids contain rich spatial structural features of the scene, and mining this information is a key step in high-quality scene reconstruction. However, event data has significant spatial sparsity: traditional convolutional neural network-based methods extract features in pixel space through sliding windows, which not only forces the calculation of event-free regions, but also makes it difficult to model long-range spatiotemporal correlations between events. To this end, this paper proposes a novel spatiotemporal structure estimator (STSE) - unlike the traditional pixel space learning paradigm, we transfer feature learning to a more semantically dense token latent space. This representation method can unify the spatiotemporal dimensions and is more suitable for capturing wide-area dependencies. By modeling the state space of the scene, spatiotemporal structural features are estimated from it.

[0058] First, the spatial embedding projection The event frame E f With voxel grid E vg Projected from pixel space into st Latent space embeddings of the same dimension This process will lose the spatial position information, and the mapping in the latent space is not sensitive to the spatial position perception. ep Added 2D sine-cosine position encoding To mark the corresponding position of each element in the original space. Then, with the event spatiotemporal feature χ st Fusion is performed to introduce spatiotemporal importance embedding. The spatial sparsity of event data will result in a large number of embeddings corresponding to event-free areas. These invalid areas not only increase the computational burden but also interfere with the extraction of key features. To this end, token-level masks are calculated through event frames. Actively shield non-event areas to obtain a representation of time-space information focused on effective events

[0059]

[0060] In order to fully explore the temporal and spatial characteristics of the scene, the selective mechanism state space model (Mamba) is used to model the scene spatial latent state into a time-varying model. The core layer SSM of Mamba is shown in Formula 5. is the potential state of the scene space at the current moment, is the time-varying state transfer matrix, u(t) represents the current input, and the time-varying Map the input to the latent state, and the observation matrix C maps the current state observation value y(t) from the scene space latent state. Finally, the spatiotemporal structural characteristics of the scene are learned from it. C are all learnable parameters and are obtained through optimization.

[0061]

[0062] (3) Hierarchical Attention Generator (HAGen)

[0063] To address the instability of directly regressing Gaussian parameters, this paper proposes a hierarchical progressive generation strategy, the Hierarchical Attention Generator (HAGen). This strategy uses three attention layers to gradually refine the prediction of Gaussian parameters. It first predicts the structural position of the scene, then the surface intensity, and finally generates the geometry of each Gaussian primitive to fine-tune rotation, scale, and opacity. This layered attention approach prioritizes global spatiotemporal structure before focusing on local details, avoiding overcomputation in irrelevant areas and significantly improving the robustness of the generation process.

[0064]

[0065] At the beginning of each level prediction, the event frame and event voxel grid are projected into k and v, and the features of the previous level are used as the query q of the next level. For the scene structure position (i.e., the coordinates of the point cloud), the spatiotemporal structure feature χ st-e As the query, k and v are fused through self-attention to generate scene structure features. Through the point cloud head The center point coordinates of each Gaussian primitive in space are obtained by prediction and perspective back projection, and the depth is express.

[0066] Scene structure feature χ struc As the query of the scene surface, k and v are fused through self-attention to generate the scene surface features By strength head Predict the surface intensity for each Gaussian primitive

[0067] Surface feature χ surf The query as the rotation, scaling and opacity parameters is fused with k and v through self-attention to obtain Gaussian features. Through Gaussian head Predict the rotation of each Gaussian primitive Zoom and the opacity matrix

[0068]

[0069] Finally, differentiable rendering is performed by parameter group Synthesize new views

[0070] (3) Joint supervision of spatiotemporal structure

[0071] To ensure more stable training of EvHGen, this paper proposes a joint spatiotemporal and structural supervision mechanism. At the spatial level, losses are calculated for point cloud positions, surface intensities, and imaged images, respectively, compared to the ground truth. This constrains the consistency of the reconstructed structure and surface. At the temporal level, a time-aware loss is calculated between the rendered image and the event-time surface. This constraint ensures the temporal consistency of the rendered results, ensuring that the synthesized view temporally conforms to the dynamic changes of the real scene captured by the event sensor. This automatically focuses on the latest events and suppresses the interference of historical noise on the reconstruction, thereby improving the spatiotemporal coherence of dynamic scene reconstruction.

[0072] As shown in formula 8, for the imaging frame L1 loss and structural similarity index loss (SSIM) are used to synthesize supervision to approximate the ground truth I.

[0073]

[0074] As shown in formula 9, for the surface strength L2 loss is used for supervision to approximate the true value I.

[0075]

[0076] As shown in Formula 10, for point cloud locations, L1 loss and SSIM are used to synthesize the depth map of the supervised point cloud To approximate the true value D.

[0077]

[0078] The above balance coefficients λ are all set to 0.3.

[0079] The temporal surface encodes the timestamp information of each event as a local surface value (exponentially decaying timestamp), forming a continuous spatiotemporal representation. This better reflects the original asynchrony and continuity of events than discrete event frames or voxel grids. Its gradient direction and intensity directly correspond to the speed and direction of the object's movement. The spatiotemporal perception loss is as follows:

[0080]

[0081] Among them, φ μ and φ σ Represents the image mean feature and variance feature calculated using vgg16. Use cos(·,·) to calculate the cosine similarity of the mean feature, constrain the directional consistency of the feature vector, and reflect the similarity of the main spatiotemporal structure. Use D KL (·‖·) calculates the KL divergence of the variance feature, constrains the diversity of the feature's spatiotemporal distribution, and avoids the generation results from being overly concentrated on the features at a certain moment. The total loss function used to train EvHGen is:

[0082]

[0083] EvHGen addresses the current limitations and non-generalization issues of 3D reconstruction in extreme lighting scenarios. It can effectively generate Gaussian primitives and reconstruct scenes in both extremely bright and dark scenes, and can generalize to untrained scenes. Comprehensive experiments on benchmark datasets show that EvHGen achieves state-of-the-art performance. Furthermore, EvHGen is the first method capable of real-time (>30 FPS) 3D imaging from unfamiliar scene event streams. It achieves current state-of-the-art performance and efficiency.

[0084] Document 1: B.Kerbl, G.Kopanas, T.Leimkühler, and G.Drettakis, "3d gaussiansplatting for real-time radiance field rendering." ACM Trans.Graph., vol.42, no.4, pp.139–1, 2023.

[0085] Document 2: H.Lin, S.Peng, Z.Xu, Y.Yan, Q.Shuai, H.Bao, and X.Zhou, "Efficientneural radiance fields for interactive free-viewpoint video," in SIGGRAPH Asia2022Conference Papers, 2022, pp.1–9.

[0086] Document 3: J.Wang, J.He, Z.Zhang, M.Sun, J.Sun, and R.Xu, "Evggs: A collaborative learning framework for event-based generalizable gaussian splatting," arXivpreprint arXiv:2405.14959, 2024.

[0087] The parameters used in References 1, 2, and 3 are guaranteed to be the optimal parameters for the model.

[0088] Table 1 summarizes the experimental data from References 1, 2, and 3, and the example, on the mixed dataset EV3D-Mix. The example achieves the best results to date. References 1 and 2 use optimization methods that are incapable of online scene imaging, making the FPS metric meaningless for these methods.

[0089] Table 1 Quantitative comparison results

[0090]

[0091] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and the embodiments. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and shown here.

Claims

1. A three-dimensional imaging system for extreme lighting environments, characterized in that: Includes the EvHGen model, which includes an importance sampling and dynamic embedding module, a spatiotemporal structure estimator, and a hierarchical attention generator; The importance sampling and dynamic embedding module is used to input the spatiotemporal event sequence E N Obtain spatiotemporal importance embedding χ st ; The spatiotemporal structure estimator is used to calculate the event frame E according to the input f , event voxel grid E vg and spatiotemporal importance embedding χ st Obtain the spatiotemporal structure characteristics χ st-e ; The hierarchical attention generator is used to generate the event frame E according to the input f , event voxel grid E vg and spatiotemporal structural characteristics χ st-e Get parameter group Then, differentiable rendering is performed by parameter group Synthesize new views in is the depth map mapped by the Gaussian primitives, is the surface intensity of the Gaussian element, is the rotation of the Gaussian basis element; is the scaling of the Gaussian basis, is the opacity matrix of the Gaussian basis element, is a real number.

2. The three-dimensional imaging system for extreme lighting environments according to claim 1, wherein: The input of the importance sampling and dynamic embedding module is a spatiotemporal event sequence of length N. Importance sampling and dynamic embedding module through the importance function module E N Encode into dense representation and calculate density probability, then take the event sequence corresponding to the top k items with the largest density probability to form the sampled event E N Input the multi-resolution hash encoder to get E N Hash feature, projection module E k The polarity part of the projected into the polarity part hash feature, the polarity part hash feature and E N The hash feature dimensions are the same; E N The hash feature and the polarity part hash feature are concatenated and used Mapping to spatiotemporal importance embedding The importance function module According to the data processing direction, it includes normalization layer, linear layer, ReLu activation layer, linear layer and Softmax layer; projection module It is composed of a linear layer Linear; The multi-resolution hash encoder process is as follows: Splicing to get χ st The method is as follows: e k' is the k'th event, e k' =(x k' ,t k' ,p k' ), x k is the k'th event coordinate, consisting of x, y, t k is the time when the k'th event occurs, p k is the polarity of the k'th event, e j Indicates E k The jth element in represents the multi-resolution hash encoder, E k|x,y,t Represents the x-element set, y-element set, and t-element set in k events, where x represents the x-coordinate of the event, y represents the y-coordinate of the event, t represents the time when the event occurs, and s l represents the scaling factor of the lth layer, E k,l|x represents the x element of the lth layer in event k, p1 represents the first element of the prime number set p, E k,l|y represents the y element of the lth layer in event k, p2 represents the second element of the prime number set p, E k,l|t Indicates the t element of the lth layer in event k, p3 represents the third element of the prime number set p, T represents the size of the hash table, mod represents modulus; cat() represents splicing, p3 represents the third element of the prime number set p, Represents a mapping module.

3. The three-dimensional imaging system for extreme lighting environments according to claim 1, wherein: The input of the spatiotemporal structure estimator is the spatiotemporal event sequence E N The corresponding event frame E f , event voxel grid E vg and spatiotemporal importance embedding χ st ; The spatiotemporal structure estimator first projects the spatial embedding The event frame E f With voxel grid E vg Projected from pixel space into spatiotemporal importance embedding χ st Latent space embedding χ of the same dimension ep , and then embed the latent space into χ ep Added 2D sine-cosine position encoding Marked with χ ep The corresponding position of each value in the original space; then the event space-time feature χ st Fusion is performed to obtain fusion features; token-level masks are calculated through event frames Actively shield the non-event areas of the fused features to obtain a representation of the temporal-spatial information focused on the effective events Then, the selective mechanism state space model is used to model the scene space latent state into a time-varying model, and the spatiotemporal structural characteristics of the scene χ are learned. st-e ; Mamba[] represents the state space model of the selective mechanism; The data processing method of the core layer SSM of the selective mechanism state space model is as follows: C are all learnable parameters, u(t) represents the input at the current moment, is the potential state of the scene space at the current moment, is the time-varying state transfer matrix, C represents the observation matrix; y(t) is the observation matrix C that maps the current state observation value from the scene space latent state.

4. The 3D imaging system for extreme lighting environments according to claim 1, wherein: The EvHGen model is trained through a joint spatiotemporal structural supervision mechanism; For imaging frames Use L1 loss and structural similarity index loss SSIM to synthesize supervision to approximate the true value I; For surface strength use Loss to supervise to approximate the true value I For point cloud locations, L1 loss and SSIM are used to synthesize the depth map of the supervised point cloud. To approximate the true value D. Represents imaging frame The loss function, λ is the balance coefficient, set to 0.3, Surface strength The loss function is Depth map The loss function of The spatiotemporal perception loss is as follows: φ μ and φ σ Respectively represent the image mean feature and variance feature calculated using vgg16; use cos() to calculate the cosine similarity of the mean feature, constrain the directional consistency of the feature vector, and reflect the similarity of the main spatiotemporal structure; use D KL (‖) Calculate the KL divergence of the variance feature to constrain the diversity of the spatiotemporal distribution of features and avoid excessive concentration of the generated results on the features at a certain moment; Indicates the calculation of rendering frame and the event time surface E suf The average of cosine similarity and KL divergence in feature space, E suf represents an event-time surface; Training makes The smallest one is the trained EvHGen model.

5. The three-dimensional imaging system for extreme lighting environments according to claim 4, wherein: The value of λ is 0.

3.

6. A three-dimensional imaging method in extreme lighting environments, characterized in that: The spatiotemporal event sequence E N , event frame E f and the voxel grid E vg Input the trained EvHGen model described in claim 4 or 5 to obtain a new view

Images