Feedforward event camera three-dimensional reconstruction method and system based on spatiotemporal feature aggregation

Abstract

The application discloses a kind of feedforward event camera three-dimensional reconstruction method and system based on space-time feature aggregation, comprising: obtaining at least two asynchronous event streams, convert each event stream into space-time voxel tensor, space-time voxel tensor includes multiple time boxes, each time box corresponds to the event accumulation in a short time fixed time slice;Space-time voxel tensor is input into time attention encoder, the feature of each spatial position is aggregated in time sequence on different time boxes by self-attention mechanism, obtain space-time feature map rich in time context information;Space-time feature map is input into the space encoder-decoder based on feedforward architecture, and the globally aligned three-dimensional point graph is generated by regression.The method and system can effectively extract the motion information and geometric clues in event data, can accurately predict three-dimensional point graph, and are significantly better than existing event camera methods in depth estimation, camera pose estimation and three-dimensional reconstruction tasks.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D reconstruction technology, specifically relating to a feedforward event camera 3D reconstruction method and system based on spatiotemporal feature aggregation. Background Technology

[0002] Robust 3D reconstruction is fundamental to robotics, augmented reality and virtual reality (AR / VR) systems, and autonomous driving systems, providing dense geometric information for navigation, interaction, and scene understanding. In practical applications, 3D reconstruction requires not only high geometric accuracy but also robustness and real-time performance. Traditional optimization-based methods, such as Structure-from-Motion (SfM) and Multi-View Stereo (MVS), recover scene structure through multi-view geometric constraints. While achieving accurate reconstruction results, these methods typically rely on feature matching and multiple rounds of iterative optimization, resulting in significant computational overhead and difficulty meeting real-time requirements. Furthermore, these methods are prone to feature degradation or matching failures in dynamic scenes, weakly textured regions, and under conditions of drastic lighting changes, thus affecting reconstruction quality.

[0003] With the development of deep learning, methods based on Neural Radiation Field (NeRF) and 3D Gaussian Splatting (3DGS) have gradually emerged. These methods achieve photorealistic 3D reconstruction results by modeling the radiation information of a scene, and excel in novel perspective synthesis. However, these methods typically require scene-by-scene optimization training, resulting in long training times, strong dependence on computational resources, and limited generalization ability. In large-scale or online applications, their scene-by-scene optimization characteristic significantly limits their practical application value.

[0004] In recent years, feedforward 3D reconstruction methods, represented by DUSt3R, have made significant progress. These methods predict globally consistent 3D structures directly from RGB images using an end-to-end network, eliminating the need for iterative optimization, significantly improving inference efficiency, and exhibiting a certain degree of cross-scene generalization capability. However, these methods rely on traditional frame-based cameras. Under conditions of rapid motion or low light, traditional cameras suffer from motion blur or reduced signal-to-noise ratio during exposure, leading to the loss of crucial information and significantly degrading reconstruction performance.

[0005] Event cameras, as a novel type of visual sensor, offer a potential solution to the aforementioned problems. They asynchronously record pixel brightness changes with microsecond-level temporal resolution, offering advantages such as high dynamic range, low latency, and no motion blur, making them particularly suitable for high-speed motion and complex lighting scenes. However, event data is characterized by high sparsity, a lack of absolute intensity information, and complex noise, making it difficult to directly apply to traditional image-based reconstruction methods.

[0006] Existing 3D reconstruction methods for event cameras mostly focus on local depth estimation or reconstruction frameworks combined with optimization. These methods typically have high computational complexity and lack effective modeling of global structural consistency, making it difficult to achieve efficient and stable 3D reconstruction. Therefore, how to extend feedforward reconstruction paradigms such as DUSt3R to event data, design representation and modeling methods suitable for event streams, and achieve globally consistent 3D reconstruction in the absence of large-scale, high-quality labeled data remains a critical problem that urgently needs to be solved. Summary of the Invention

[0007] In view of the above-mentioned technical problems, the purpose of this invention is to provide a feedforward event camera 3D reconstruction method and system based on spatiotemporal feature aggregation. By reconstructing asynchronous event streams into structured space-time voxel tensors and designing a temporal attention mechanism to model temporal information, the method and system finally inputs the data into the feedforward 3D reconstruction backbone network to directly predict globally aligned 3D point clouds.

[0008] To achieve the above-mentioned objectives, an embodiment provides a feedforward event camera 3D reconstruction method based on spatiotemporal feature aggregation, comprising the following steps: Acquire at least two asynchronous event streams, and convert each event stream into a space-time voxel tensor. The space-time voxel tensor contains multiple time bins, and each time bin corresponds to the event accumulation within a short fixed-time slice. The space-time voxel tensor is input into the temporal attention encoder, and the features of each spatial location in different time boxes are temporally aggregated through the self-attention mechanism to obtain a spatiotemporal feature map rich in temporal context information. The spatiotemporal feature map is input into a spatial encoder-decoder based on a feedforward architecture, and a globally aligned 3D point map is generated directly by regression.

[0009] Preferably, each event stream is converted into a space-time voxel tensor, including: Each event in the event stream is ,in For the first k One event, For pixel coordinates, For timestamps, To determine the polarity, trilinear interpolation is used to assign it to adjacent time bins, calculated as follows: in, This is the normalized timestamp. For the index of the timebox, To find the function with the maximum value, For the first Each timebox at pixel coordinates The corresponding cumulative event results have a spatial resolution of ; For a total of B timeboxes, the resulting size is... Space-time voxel tensors.

[0010] Preferably, the space-time voxel tensor is input into a temporal attention encoder, and the features of each spatial location in different timeboxes are temporally aggregated through a self-attention mechanism to obtain a spatiotemporal feature map rich in temporal context information, including: The two-dimensional event accumulation graph corresponding to each time box in the space-time voxel tensor is divided into non-overlapping spatial blocks, and each spatial block is mapped to a feature token through a convolutional embedding layer. After stacking the feature tokens of the same spatial location across all time boxes along the time dimension, the stacked feature tokens of the center time box of a single spatial location are input into the time attention encoder. The feature tokens of the center time box of a single spatial location are used as queries, and the feature tokens of all time boxes of a single spatial location are used as keys and values. Multi-head self-attention calculation is performed on the feature tokens of all time boxes, and the aggregated center time box feature tokens are output. The central timebox feature tokens of all spatial locations are recombined into a two-dimensional feature map, which serves as the spatiotemporal feature map rich in temporal context information.

[0011] Preferably, the temporal attention encoder and spatial encoder-decoder are optimized for parameters using a two-stage training strategy before application, including: The first stage employs self-supervised pre-training of the temporal attention encoder for a masking box modeling task; The second stage employs end-to-end supervised joint fine-tuning of the pre-trained temporal attention encoder and spatial encoder-decoder.

[0012] Preferably, the temporal attention encoder undergoes self-supervised pre-training based on a masking box modeling task, including: Randomly mask one or more time bins in the input space-temporal voxel tensor, and input the remaining visible time bins into the temporal attention encoder to obtain a partial feature map; A lightweight reconstruction decoder is used to decode the partial feature map to obtain the predicted features of the masked time box, and the reconstruction loss is calculated based on the predicted features of the masked time box and its original features. By introducing contrast loss, we can bring the feature distance of the same spatial location closer together under different time boxes or augmented views, and push the feature distance of different spatial locations further apart. A consistency loss is introduced to constrain the original features of the time box and its data augmentation features to remain spatially consistent. The parameters of the temporal attention encoder are optimized based on reconstruction loss, contrastive learning loss, and consistency loss.

[0013] Preferably, end-to-end supervised joint fine-tuning of the pre-trained temporal attention encoder and spatial encoder-decoder includes: The pre-trained temporal attention encoder is combined with the spatial encoder-decoder and then fine-tuned under supervision using an event dataset labeled with depth and pose. The total loss function for fine-tuning includes the 3D regression loss of the spatial encoder-decoder, the confidence loss, and the auxiliary loss related to the masking box modeling task. The auxiliary loss includes the reconstruction loss with balanced weights, the contrastive learning loss, and the consistency loss.

[0014] Preferably, the method further includes: the generated 3D point map is used for downstream tasks, including dense depth estimation, multi-view 3D reconstruction and camera pose estimation, wherein the depth map and pose map are both aligned to the reference frame of the central timebox.

[0015] To achieve the above-mentioned objectives, the embodiments also provide a feedforward event camera 3D reconstruction system based on spatiotemporal feature aggregation, comprising: The event stream acquisition and voxelization module is used to acquire at least two asynchronous event streams and convert each event stream into a space-time voxel tensor. The space-time voxel tensor contains multiple time boxes, and each time box corresponds to the event accumulation within a short fixed time slice. The temporal attention encoding module is used to input the space-time voxel tensor into the temporal attention encoder, and to perform temporal aggregation of the features of each spatial location in different time boxes through a self-attention mechanism to obtain a spatiotemporal feature map rich in temporal context information. The spatial encoder-decoder module is used to input the spatiotemporal feature map into the spatial encoder-decoder based on a feedforward architecture, and directly regress to generate a globally aligned 3D point map.

[0016] To achieve the above-mentioned objectives, the embodiments also provide a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-mentioned feedforward event camera 3D reconstruction method based on spatiotemporal feature aggregation.

[0017] To achieve the above-mentioned objectives, the embodiments also provide a computer-readable storage medium storing a program that, when executed by a processor, implements the above-mentioned feedforward event camera 3D reconstruction method based on spatiotemporal feature aggregation.

[0018] Compared with the prior art, the beneficial effects of the present invention include at least the following: This invention proposes the first feedforward 3D reconstruction framework based entirely on event camera input. By transforming asynchronous event streams into space-time voxel tensors and designing a temporal attention encoder for feature aggregation, this invention can effectively extract motion information and geometric cues from event data.

[0019] Furthermore, the introduced masking box modeling self-supervised learning strategy significantly reduces the reliance on large-scale labeled data and enhances the model's ability to model temporal information. Extensive experiments on synthetic and real-world datasets demonstrate that this invention significantly outperforms existing event camera methods in depth estimation, camera pose estimation, and 3D reconstruction tasks. This provides an efficient, scalable, and data-efficient solution for the application of event cameras in robotics, autonomous navigation, and real-time scene understanding. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of the feedforward event camera 3D reconstruction method based on spatiotemporal feature aggregation provided in the embodiment; Figure 2 This is a schematic diagram of the temporal attention encoder provided in the embodiment; Figure 3 This is a schematic diagram of the Masked Bin Modeling (MBM) pre-training strategy provided in the embodiment; Figure 4 This is a qualitative comparison of the method of the present invention and existing event methods in the depth estimation task provided in the embodiments. Figure 5 This is a comparison of the depth estimation results of the method of the present invention provided in the embodiment and the RGB-based feedforward method under challenging lighting conditions; Figure 6 This is a schematic diagram of the structure of the feedforward event camera 3D reconstruction system based on spatiotemporal feature aggregation provided in the embodiment. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0023] like Figure 1 As shown in the embodiment, a feedforward event camera 3D reconstruction method based on spatiotemporal feature aggregation is provided, including the following steps: S1, obtain at least two asynchronous event streams, and convert each event stream into a space-time voxel tensor. The space-time voxel tensor contains multiple time boxes, and each time box corresponds to the event accumulation within a short fixed time slice.

[0024] In this embodiment, at least two asynchronous event streams are first acquired. The time window for each event stream is... (i.e., the time interval between the last event and the first event); discretize each event stream into a segment of size. The voxel mesh, in which Indicates the number of time bins. Spatial resolution of the cumulative results of events corresponding to each timebox.

[0025] Each event is ,in For the first k One event, For pixel coordinates, For timestamps, The value is polarity, where +1 represents an increase in brightness compared to the previous time step, and -1 represents a decrease in brightness. It is then assigned to its adjacent time blocks using trilinear interpolation, calculated as follows: in, for Normalized timestamp The initial time, For the index of the timebox, To find the function with the maximum value, For the t-th timebox at pixel coordinates The corresponding cumulative results of events.

[0026] The multiple timeboxes corresponding to the event stream are arranged in chronological order to form a dense space-time voxel tensor that retains the temporal structure. ,in This represents the number of timeboxes. In this embodiment, to balance performance and computational overhead, we take... That is, including time boxes b 1, b 2, b 3, b 4, b 5, and set The event block length is odd to ensure the existence of a definite central timebox as a reference frame for subsequent 3D point cloud output. Fixed at 50 ms.

[0027] S2, input the space-time voxel tensor into the temporal attention encoder, and use the self-attention mechanism to perform temporal aggregation of the features of each spatial location in different time boxes to obtain a spatiotemporal feature map rich in temporal context information.

[0028] like Figure 2 As shown, the space-time voxel tensor The input is fed into a temporal attention encoder (TE) to extract its spatiotemporal feature map. For example... Figure 2 As shown, the TE processing flow is as follows: First, time-box feature extraction: extracting the space-time voxel tensor... Two-dimensional event accumulation graph corresponding to each timebox The data is divided into non-overlapping spatial patches, and each patch is mapped to a feature token using a convolutional embedding layer. In this embodiment, a size of [size missing] is used. The spatial blocks are embedded into 32-dimensional feature tokens using convolutional layers with a stride and kernel size of 8. : in, This represents the spatial embedding achieved by the convolutional layer, thus, each timebox Represented as a by A feature graph composed of feature tokens.

[0029] Secondly, temporal attention aggregation: In order to model the temporal information of the same spatial location, the same spatial location is aggregated across all... B The feature tokens from each timebox are stacked along the time dimension to form a feature map sequence. The input is fed into the temporal attention encoder, so that each spatial location of Each feature token is processed independently.

[0030] For spatial location Its corresponding feature token sequence , For the feature space, based on spatial location Feature token corresponding to the central timebox For querying, by spatial location All timebox feature tokens Using these as keys and values, multi-head self-attention computation is performed on the feature tokens of all timeboxes, outputting the aggregated central timebox feature token, for example... Figure 2 Central Time Box Feature Token .

[0031] In this embodiment, the temporal attention encoder consists of two layers of multi-head self-attention and feedforward network (MLP), with four attention heads and an MLP scaling factor of 4.0. In this way, the token in the central timebox aggregates information from neighboring timeboxes, thereby encoding rich motion information.

[0032] Finally, feature combination: the central timebox feature tokens of all spatial locations are recombined into a two-dimensional feature map, which serves as the spatiotemporal feature map rich in temporal context information. .

[0033] S3 inputs the spatiotemporal feature map into a spatial encoder-decoder based on a feedforward architecture, and directly regresses to generate a globally aligned 3D point map.

[0034] In the embodiment, the spatiotemporal feature map The input is fed into a DUST3R-based spatial encoder-decoder, which employs a ViT-Base architecture to process multi-view features and perform cross-view information interaction, ultimately directly regressing a globally aligned 3D point map. This 3D point map corresponds to the camera viewpoint of the central timebox of the input event stream, ensuring spatiotemporal consistency. The entire inference process can be completed in approximately 0.2 seconds.

[0035] To enhance the temporal attention encoder and spatial encoder-decoder's ability to represent temporal information and reduce their dependence on labeled data, this embodiment adopts a two-stage training strategy to optimize parameters: the first stage is to perform self-supervised pre-training of the temporal attention encoder based on the Masked Bin Modeling (MBM) task, and the second stage is to perform end-to-end supervised fine-tuning of the pre-trained temporal attention encoder and spatial encoder-decoder.

[0036] For the first stage, a self-supervised pre-training of the temporal attention encoder is performed on a large amount of unlabeled event data (such as MatrixCity and TUM-VIE). For example... Figure 3 As shown, for the input space-time voxel tensor, one or more timeboxes are randomly masked, such as masking timeboxes. b2. Then, the remaining visible time bins are input into the temporal attention encoder to obtain partial feature maps. Then, a lightweight reconstruction decoder is used to process some feature maps. Decoding yields the predicted features of the masked timeboxes. And based on the prediction features of the masked timebox. and its original characteristics Calculate reconstruction loss : Meanwhile, to enhance the discriminativeness and robustness of the features, a contrastive loss is introduced. and consistency loss Among them, comparative loss To bring features closer together at the same spatial location in different timeboxes or augmented views, and to push features further apart at different spatial locations: in, and Indexing different spatial locations, and Indexing for different time boxes Indicates the first The spatial location is at the first Feature vectors corresponding to each time bin; Indicates and The corresponding positive sample features originate from features of the same spatial location in different time bins or under different data augmentation views; the rest... (when or ) as negative samples; For cosine similarity, In this embodiment, a temperature parameter is set. .

[0037] Consistency loss The original features used to constrain the time box are spatially consistent with their data augmentation features, represented as follows: in, Indicates the first The spatial location is at the first The feature vectors corresponding to each time bin To enhance the features of the input at the same location in the corresponding data, The loss is the L2 norm.

[0038] The total pre-training loss function is: In this embodiment, a loss weight is set. , Pre-training uses the Adam optimizer with a learning rate of [missing information]. With a weight decay of 0.05 and a batch size of 4, the training was conducted for 300 epochs on 8 NVIDIA RTX A100 GPUs.

[0039] For the second stage, on datasets with depth and pose annotations (such as TartanAir and MVSEC), the pre-trained temporal attention encoder and spatial encoder-decoder are jointly fine-tuned end-to-end. The overall loss function is: in, For the 3D point cloud regression loss of the spatial encoder-decoder, For confidence loss, , , This is an auxiliary loss term used to continuously strengthen the overall model's modeling of temporal dependencies. In this embodiment, balanced weights are set. .

[0040] Fine-tuning using the Adam optimizer, learning rate The cosine annealing strategy was used, with the maximum norm of gradient clipping set to 1.0, and the training was performed for 300 epochs. The entire training process took approximately 72 hours on 8 GPUs.

[0041] After training, the method of this invention receives an asynchronous event stream as input and directly outputs a globally aligned 3D point map. This output can be used for various downstream tasks, such as dense depth estimation, multi-view 3D reconstruction, and camera pose estimation. All prediction results (depth, pose, point coordinates) correspond to the center timebox of the input event voxel grid, ensuring temporal consistency.

[0042] Figure 4 This is a qualitative comparison of the method of the present invention and existing event-based methods in the depth estimation task. From left to right: the true depth map (i.e., the true value), the depth estimation result of the method of the present invention, and the depth estimation results of existing comparison methods (EvGGS and E2Depth, etc.). The results show that the method of the present invention can generate a clearer and more complete depth map.

[0043] Figure 5This image shows a comparison of depth estimation results between the method of this invention and an RGB-based feedforward method (such as DUSt3R) under challenging lighting conditions. From left to right: event camera, RGB camera, depth estimation result of the method of this invention, and depth estimation result of the comparison method (DUSt3R). Experiments show that under challenging lighting conditions, such as low-light and high dynamic range scenes, the RGB-based method produces incorrect depth estimations due to image quality degradation, while the method of this invention maintains accurate geometric reconstruction. This fully verifies the advantages of the event camera in high dynamic range scenes and the robustness of the method of this invention.

[0044] The experiments in summary show that the method of this invention outperforms existing event camera methods in depth estimation and camera pose estimation tasks, and can effectively handle challenging scenarios such as rapid motion and changes in lighting.

[0045] like Figure 6 As shown in the embodiment, a feedforward event camera 3D reconstruction system 60 based on spatiotemporal feature aggregation is also provided. This system includes an event stream acquisition and voxelization module 61, a temporal attention encoding module 62, and a spatial encoding-decoding module 63. The event stream acquisition and voxelization module 61 acquires at least two asynchronous event streams and converts each event stream into a spatial-temporal voxel tensor. Each spatial-temporal voxel tensor contains multiple timeboxes, each corresponding to the accumulation of events within a short fixed-time slice. The temporal attention encoding module 62 inputs the spatial-temporal voxel tensor into a temporal attention encoder, using a self-attention mechanism to temporally aggregate features of each spatial location across different timeboxes, resulting in a spatiotemporal feature map rich in temporal context information. The spatial encoding-decoding module 63 inputs the spatiotemporal feature map into a feedforward-based spatial encoder-decoder system, directly regressing to generate a globally aligned 3D point map.

[0046] It should be noted that the feedforward event camera 3D reconstruction system based on spatiotemporal feature aggregation provided in the above embodiments should be illustrated using the above-described functional module division as an example when performing feedforward event camera 3D reconstruction. The functions described above can be assigned to different functional modules as needed, i.e., the internal structure of the terminal or server can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the feedforward event camera 3D reconstruction system based on spatiotemporal feature aggregation provided in the above embodiments and the feedforward event camera 3D reconstruction method embodiment based on spatiotemporal feature aggregation belong to the same concept. For details of its implementation process, please refer to the feedforward event camera 3D reconstruction method embodiment based on spatiotemporal feature aggregation, which will not be repeated here.

[0047] Based on the same inventive concept, the embodiment also provides a computing device, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-described feedforward event camera 3D reconstruction method based on spatiotemporal feature aggregation, specifically including the following steps: S1, acquire at least two asynchronous event streams, and convert each event stream into a space-time voxel tensor. The space-time voxel tensor contains multiple time boxes, and each time box corresponds to the event accumulation within a short fixed time slice. S2, the space-time voxel tensor is input into the temporal attention encoder, and the features of each spatial location in different time boxes are temporally aggregated through the self-attention mechanism to obtain a spatiotemporal feature map rich in temporal context information; S3, the spatiotemporal feature map is input into a spatial encoder-decoder based on a feedforward architecture to directly regress and generate a globally aligned 3D point map.

[0048] The computing device provided in this embodiment, at the hardware level, includes not only a processor and memory, but also internal buses, network interfaces, memory, and other hardware required for business operations. The memory is non-volatile memory. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the feedforward event camera 3D reconstruction method based on spatiotemporal feature aggregation described in S1-S3 above. Of course, besides software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0049] Based on the same inventive concept, the embodiments also provide a computer-readable storage medium storing a program that, when executed by a processor, implements the above-described feedforward event camera 3D reconstruction method based on spatiotemporal feature aggregation, specifically including the following steps: S1, acquire at least two asynchronous event streams, and convert each event stream into a space-time voxel tensor. The space-time voxel tensor contains multiple time boxes, and each time box corresponds to the event accumulation within a short fixed time slice. S2, the space-time voxel tensor is input into the temporal attention encoder, and the features of each spatial location in different time boxes are temporally aggregated through the self-attention mechanism to obtain a spatiotemporal feature map rich in temporal context information; S3, the spatiotemporal feature map is input into a spatial encoder-decoder based on a feedforward architecture to directly regress and generate a globally aligned 3D point map.

[0050] In this embodiment, computer-readable media includes both permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0051] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A feedforward event camera 3D reconstruction method based on spatiotemporal feature aggregation, characterized in that, Includes the following steps: Acquire at least two asynchronous event streams, and convert each event stream into a space-time voxel tensor. The space-time voxel tensor contains multiple time bins, and each time bin corresponds to the event accumulation within a short fixed-time slice. The space-time voxel tensor is input into the temporal attention encoder, and the features of each spatial location in different time boxes are temporally aggregated through the self-attention mechanism to obtain a spatiotemporal feature map rich in temporal context information. The spatiotemporal feature map is input into a spatial encoder-decoder based on a feedforward architecture, and a globally aligned 3D point map is generated directly by regression.

2. The feedforward event camera 3D reconstruction method based on spatiotemporal feature aggregation according to claim 1, characterized in that, Each event stream is converted into a space-time voxel tensor, including: Each event in the event stream is ,in For the first k One event, For pixel coordinates, For timestamps, To determine the polarity, trilinear interpolation is used to assign it to adjacent time bins, calculated as follows: in, This is the normalized timestamp. For the index of the timebox, To find the function with the maximum value, For the first Each timebox at pixel coordinates The corresponding cumulative event results have a spatial resolution of ; For a total of B timeboxes, the resulting size is... Space-time voxel tensors.

3. The feedforward event camera 3D reconstruction method based on spatiotemporal feature aggregation according to claim 1, characterized in that, The space-time voxel tensor is input into a temporal attention encoder, and the features of each spatial location in different timeboxes are temporally aggregated through a self-attention mechanism to obtain a spatiotemporal feature map rich in temporal context information, including: The two-dimensional event accumulation graph corresponding to each time box in the space-time voxel tensor is divided into non-overlapping spatial blocks, and each spatial block is mapped to a feature token through a convolutional embedding layer. After stacking the feature tokens of the same spatial location across all time boxes along the time dimension, the stacked feature tokens of the center time box of a single spatial location are input into the time attention encoder. The feature tokens of the center time box of a single spatial location are used as queries, and the feature tokens of all time boxes of a single spatial location are used as keys and values. Multi-head self-attention calculation is performed on the feature tokens of all time boxes, and the aggregated center time box feature tokens are output. The central timebox feature tokens of all spatial locations are recombined into a two-dimensional feature map, which serves as the spatiotemporal feature map rich in temporal context information.

4. The feedforward event camera 3D reconstruction method based on spatiotemporal feature aggregation according to claim 1, characterized in that, Before being applied, the temporal attention encoder and spatial encoder-decoder employ a two-stage training strategy to optimize their parameters, including: The first stage employs self-supervised pre-training of the temporal attention encoder for a masking box modeling task; The second stage employs end-to-end supervised joint fine-tuning of the pre-trained temporal attention encoder and spatial encoder-decoder.

5. The feedforward event camera 3D reconstruction method based on spatiotemporal feature aggregation according to claim 4, characterized in that, Self-supervised pre-training of the temporal attention encoder for a masked box modeling task includes: Randomly mask one or more time bins in the input space-temporal voxel tensor, and input the remaining visible time bins into the temporal attention encoder to obtain a partial feature map; A lightweight reconstruction decoder is used to decode the partial feature map to obtain the predicted features of the masked time box, and the reconstruction loss is calculated based on the predicted features of the masked time box and its original features. By introducing contrast loss, we can bring the feature distance of the same spatial location closer together under different time boxes or augmented views, and push the feature distance of different spatial locations further apart. A consistency loss is introduced to constrain the original features of the time box and its data augmentation features to remain spatially consistent. The parameters of the temporal attention encoder are optimized based on reconstruction loss, contrastive learning loss, and consistency loss.

6. The feedforward event camera 3D reconstruction method based on spatiotemporal feature aggregation according to claim 5, characterized in that, End-to-end supervised joint fine-tuning of the pre-trained temporal attention encoder and spatial encoder-decoder includes: The pre-trained temporal attention encoder is combined with the spatial encoder-decoder and then fine-tuned under supervision using an event dataset labeled with depth and pose. The total loss function for fine-tuning includes the 3D regression loss of the spatial encoder-decoder, the confidence loss, and the auxiliary loss related to the masking box modeling task. The auxiliary loss includes the reconstruction loss with balanced weights, the contrastive learning loss, and the consistency loss.

7. The feedforward event camera 3D reconstruction method based on spatiotemporal feature aggregation according to claim 1, characterized in that, The method also includes: the generated 3D point map is used for downstream tasks, including dense depth estimation, multi-view 3D reconstruction and camera pose estimation, wherein the depth map and pose map are both aligned to the reference frame of the central timebox.

8. A feedforward event camera 3D reconstruction system based on spatiotemporal feature aggregation, characterized in that, include: The event stream acquisition and voxelization module is used to acquire at least two asynchronous event streams and convert each event stream into a space-time voxel tensor. The space-time voxel tensor contains multiple time boxes, and each time box corresponds to the event accumulation within a short fixed time slice. The temporal attention encoding module is used to input the space-time voxel tensor into the temporal attention encoder, and to perform temporal aggregation of the features of each spatial location in different time boxes through a self-attention mechanism to obtain a spatiotemporal feature map rich in temporal context information. The spatial encoder-decoder module is used to input the spatiotemporal feature map into the spatial encoder-decoder based on a feedforward architecture, and directly regress to generate a globally aligned 3D point map.

9. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the one or more processors execute the executable code, they are used to implement the feedforward event camera 3D reconstruction method based on spatiotemporal feature aggregation as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the feedforward event camera 3D reconstruction method based on spatiotemporal feature aggregation as described in any one of claims 1-7.

Images