A Condition-Constrained 3D Gaussian Adaptive Reconstruction Method

Abstract

This invention relates to the fields of computer graphics and 3D reconstruction technology, proposing a conditional adaptive 3D Gaussian reconstruction method. Addressing the problems of overfitting, reliance on external priors, and high computational cost in existing 3D Gaussian reconstruction methods under conditions, this invention introduces spiking neurons to perform time-cumulative evaluation and adaptive selection of Gaussian primitive importance. Specifically, the radius and opacity are mapped to input potentials, and activation and discard markers are generated through leakage integration and firing. During training, primitives participating in rendering and optimization are dynamically selected, retaining only those with significant contributions. No fixed discard rate is required; the participation ratio can be adaptively adjusted according to the scene and stage, suppressing redundancy expansion, reducing memory and rendering overhead, and improving generalization to unseen viewpoints. Experiments on LLFF and Mip-NeRF360 show that 50%–70% of primitives can be discarded without quality degradation. The device is used to acquire and output time-aligned multi-view images, camera intrinsics, and poses.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and 3D reconstruction technology, and in particular to a 3D Gaussian adaptive reconstruction method under certain conditions. Background Technology

[0002] 3D reconstruction and novel perspective synthesis, as fundamental capabilities in applications such as virtual reality, augmented reality, robot navigation, and digital twins, aim to recover the geometric structure of a scene from the observer's perspective and generate realistic images from any viewpoint. As applications move from offline displays to interactive and real-time systems, the comprehensive requirements for reconstruction quality, computational efficiency, and deployment costs are constantly increasing, prompting continuous exploration of more efficient 3D representation and rendering methods.

[0003] In traditional methods, Neural Radiance Fields (NeRF) has achieved high-fidelity reconstruction of complex scenes through implicit field representation and differentiable volume rendering, driving the development of this field. However, NeRF relies on dense viewpoint acquisition and a large amount of volume sampling, resulting in long training time and slow inference speed. To alleviate these bottlenecks, 3DGaussian Splatting (3DGS) has attracted attention as an efficient and differentiable explicit representation. This method models the scene as a set of 3D Gaussian primitives and jointly optimizes the position, covariance, opacity, and color attributes of the primitives. It can be initialized from sparse point clouds and achieves high-efficiency rendering using rasterization, achieving higher rendering efficiency and lower computational load, showing significant advantages in high-resolution visualization and interactive applications.

[0004] However, the methods described above all assume that the training data has sufficient viewpoint coverage and good image quality. In real-world environments, training data often suffers from limitations such as sparse viewpoints, incomplete coverage, noise blurring, or exposure variations due to equipment layout, acquisition costs, and imaging conditions. Under these constraints, insufficient input observations amplify geometric and appearance ambiguities, leading to problems such as slow convergence, degenerate reconstruction, and insufficient generalization ability for unseen viewpoints in NeRF. While 3DGS offers greater efficiency, it is still prone to overfitting to the visible area under constraints, resulting in floating artifacts for unseen viewpoints. Furthermore, the number of 3D Gaussian primitives rapidly increases during optimization, leading to significant rendering and memory overhead, which contradicts the resource-constrained deployment requirements.

[0005] To address the aforementioned issues and improve generalization ability and computational efficiency under constraints, existing technologies mainly fall into two categories: First, introducing external prior constraints such as monocular depth, generative models, or optical flow to enhance geometric consistency or appearance constraints. However, these methods typically rely on prior accuracy and introduce additional models or computational processes, thereby increasing system complexity and computational overhead. Second, drawing on the idea of ​​dropout, randomly masking some 3D Gaussian primitives during training to increase attention to low-visibility primitives. However, these methods are sensitive to hyperparameters such as dropout rate and lack differentiation of primitive importance, easily leading to the problem of mistakenly deleting key structures or retaining redundant primitives.

[0006] Therefore, there is an urgent need for a method that adaptively discards primitives based on their importance without relying on external priors or introducing complex network structures. This method would effectively suppress primitive redundancy and computational overhead while improving the generalization ability of unseen perspectives, thereby achieving efficient, robust, and realistic adaptive reconstruction of 3D Gaussian under constraints. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and propose a condition-constrained 3D Gaussian adaptive reconstruction technique. By introducing a spiking neuron model, it performs temporal accumulation importance assessment and adaptive screening of 3D Gaussian neurons, achieving efficient, robust, and realistic 3D Gaussian adaptive reconstruction under conditional constraints without the need for additional prior knowledge or manual adjustment of the dropout rate. Simultaneously, this invention also provides a dataset construction device for building the training dataset of the method, used to acquire and output time-aligned multi-view images, camera intra-parameters, and camera poses under real-world scene constraints, providing data input for the method.

[0008] The technical solution of the present invention:

[0009] A three-dimensional Gaussian adaptive reconstruction method under certain constraints includes the following steps:

[0010] S1. Obtain multi-view images of the scene to be reconstructed and the corresponding camera poses. Estimate the camera intrinsic and extrinsic parameters of the multi-view images to obtain the pose information of each multi-view image. Construct the initial point cloud of the scene based on the structure-of-motion (SfM) or multi-view stereo (MVS) method. With the initial point cloud of the scene as the center, arrange several three-dimensional Gaussian primitives in three-dimensional space and initialize the parameters of the three-dimensional Gaussian primitives, including spatial position, scale, color and opacity.

[0011] S2. Construct each three-dimensional Gaussian element in the scene to be reconstructed as a spiking neuron unit. The spiking neuron unit maintains the state variable of the membrane potential and the firing threshold. The membrane potential is initialized to zero or a predetermined resting potential, and the firing threshold is set according to experience.

[0012] S3. In each training iteration, for each 3D Gaussian, calculate the importance input of the 3D Gaussian based on the current opacity and projection radius, and update the membrane potential of the corresponding spiking neuron unit according to the importance input.

[0013] S4. When the membrane potential corresponding to a certain 3D Gaussian element reaches or exceeds the firing threshold, a pulse firing signal is generated, and the 3D Gaussian element is marked as active; 3D Gaussian elements that do not exceed the firing threshold are marked as inactive and do not participate in rendering and parameter optimization in the current iteration.

[0014] S5. Perform 3D Gaussian Splatting (3DGS) rendering using only the activated 3D Gaussian primitives to obtain the reconstructed image from the training viewpoint, and calculate the reconstruction loss between the reconstructed image and the real image.

[0015] S6. Based on the reconstruction loss, the parameters of the activated 3D Gaussian units and the firing threshold of the spiking neuron units are backpropagated and updated. During multiple iterations, the activation ratio and spatial distribution of the 3D Gaussian units are adaptively adjusted, thereby completing the 3D reconstruction of the scene under multi-view conditions.

[0016] Further, in step S3, the calculation of the importance input includes: using the opacity of three-dimensional Gaussian elements. Its projected radius The product of the squares is used as the original importance quantity. Then, the original importance quantity is logarithmically mapped and normalized to obtain the normalized importance input. Finally, the normalized importance input is used as the input current of the corresponding spiking neuron unit to update the membrane potential.

[0017] Furthermore, the original importance quantity The calculation formula is: Normalized Importance Input The calculation formula is: ,in, , Let represent the original importance quantity of the spiking neuron unit corresponding to the i-th 3D Gaussian element in the t-th iteration. Let represent the original importance quantity of the spiking neuron unit corresponding to the i-th three-dimensional Gaussian element after logarithmic mapping at the t-th iteration; This is the mean value used for normalization in the t-th iteration; This is the standard deviation used for normalization in the t-th iteration; To be placed in the denominator to prevent =0 is a positive number that divides by zero to improve numerical stability; i represents the spiking neuron unit corresponding to the i-th three-dimensional Gaussian element; t represents the t-th iteration.

[0018] Furthermore, the membrane potential update in step S3 satisfies the leakage-integral-release model (LIF):

[0019]

[0020] in, Let be the membrane potential of the spiking neuron unit corresponding to the i-th three-dimensional Gaussian cell at the t-th iteration. The attenuation coefficient is... Let $\frac{i}{t-1}$ represent the firing state of the spiking neuron unit corresponding to the $i$-th three-dimensional Gaussian element during the (t-1)th iteration. This is the normalized importance input for the t-th iteration.

[0021] Further, in step S4, the issuance state satisfies:

[0022] when hour, The corresponding three-dimensional Gaussian primitive is marked as active;

[0023] when hour, The corresponding three-dimensional Gaussian elements are marked as inactive;

[0024] in, The firing threshold of the membrane potential is a learnable parameter that is adaptively updated during training via gradient descent.

[0025] Furthermore, the reconstruction loss in step S5 includes a distance-based pixel-level error term. and perceptual error term based on structural similarity The two are then summed using a weighted average based on preset weights: where, Indicates in and A weighting factor that balances the contributions of each factor. Indicates the reconstructed image. The formula for representing a real image is as follows:

[0026] .

[0027] Furthermore, the training process also includes periodically performing densification and merging operations on the 3D Gaussian units based on the current density distribution and reconstruction error. After performing densification or merging operations, the states of the corresponding spiking neuron units of the 3D Gaussian units are reinitialized or inherited to ensure that the adaptive selection process remains stable on the new set of 3D Gaussian units.

[0028] Furthermore, the device used in step S1 to obtain multi-view images and corresponding camera poses includes:

[0029] Multi-view acquisition module: used to acquire multi-view images from different perspectives of the scene to be acquired, wherein the multi-view images include at least RGB images;

[0030] Synchronization trigger module: used to provide clock synchronization signal and trigger signal to the multi-view acquisition module to achieve time alignment of the acquired multi-view images;

[0031] Calibration and pose solving module: used to obtain the intrinsic parameters of cameras from each viewpoint, and perform camera extrinsic parameter estimation based on the multi-view images to obtain the corresponding camera pose;

[0032] Data alignment processing module: used to align, crop or extract frames of multi-view images based on timestamp information to generate multi-view image sequences;

[0033] Data storage and output module: used to organize the multi-view image sequence, camera intrinsic parameters and camera pose into a training dataset and output it.

[0034] Compared with the prior art, the present invention has at least the following beneficial effects:

[0035] (1) High-quality 3D Gaussian reconstruction under conditional constraints: This invention can maintain efficiency and robustness under conditional constraints, while taking into account high-quality rendering and geometric consistency, thereby achieving efficient, robust and realistic 3D Gaussian adaptive reconstruction under conditional constraints.

[0036] (2) Strong Adaptability: This invention introduces spiking neuron units with time accumulation and leakage characteristics, and adaptively evaluates their long-term importance based on the radius and opacity of the three-dimensional Gaussian units, dynamically deciding whether to participate in training and reconstruction. The firing threshold of the spiking neuron unit is a learnable parameter, which can automatically adjust the actual discard ratio under different scene conditions, maintaining a lower screening intensity for complex structural regions and more aggressively sparsifying large areas of uniform texture, without the need to manually set a fixed discard rate, thus achieving stable adaptive three-dimensional reconstruction.

[0037] (3) Significant sparsity and lightweighting: On public datasets such as LLFF and Mip-NeRF360, this invention can discard more than 50% to 70% of the three-dimensional Gaussian primitives during training, greatly reducing the number of primitives involved in rendering and optimization. While maintaining or exceeding the reconstruction quality of existing methods, it effectively reduces memory usage and computation, and improves deployability under conditions of constraint.

[0038] (4) No external priors required and easy to integrate and deploy: This invention relies solely on the parameters of the 3D Gaussian primitives themselves for importance judgment, without relying on external depth, semantics, or generation priors, thus avoiding the overhead and risks caused by unstable prior quality and additional model inference. At the same time, this method can be implemented on the existing 3DGS framework by only adding a lightweight neuron module, which is compatible with the original rendering pipeline and facilitates integration and deployment in existing systems. Attached Figure Description

[0039] To more clearly illustrate the technical solution of the present invention, the accompanying drawings are briefly described below:

[0040] Figure 1 This is a schematic diagram of the overall process of three-dimensional Gaussian adaptive reconstruction under certain conditions according to the present invention;

[0041] Figure 2 This is a schematic diagram comparing the PSNR of the present invention with existing state-of-the-art methods with different 3D Gaussian skeletal drop ratios on the Mip-NeRF360 dataset and the LLFF dataset; where (a) is the Mip-NeRF360 dataset and (b) is the LLFF dataset.

[0042] Figure 3 This is a schematic diagram illustrating the adaptive adjustment of the 3D Gaussian pixel discard ratio based on scene complexity in this invention.

[0043] Figure 4 This diagram illustrates the reconstruction quality of our invention on the LLFF dataset compared to other existing methods. The columns for 3DGS, FSGS, CoR-GS, and DropGaussian represent other existing methods, the column for Ours represents our method, and the column for GT represents the real images.

[0044] Figure 5 This diagram illustrates the reconstruction quality of our invention on the Mip-NeRF360 dataset compared to other existing methods. The columns for 3DGS, FSGS, CoR-GS, and DropGaussian represent other existing methods, the column for Ours represents our method, and the column for GT represents the real images.

[0045] Figure 6 This is a schematic diagram of the overall architecture of the condition-constrained dataset construction device of the present invention;

[0046] Figure 7 This is a schematic diagram of the clock synchronization scheme of the condition-constrained dataset construction device of the present invention. Detailed Implementation

[0047] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0048] A three-dimensional Gaussian adaptive reconstruction method under certain constraints includes the following steps:

[0049] S1. Obtain multi-view images of the scene to be reconstructed and the corresponding camera poses. Estimate the camera intrinsic and extrinsic parameters of the multi-view images to obtain the pose information of each multi-view image. Construct the initial point cloud of the scene based on the structure-of-motion (SfM) or multi-view stereo (MVS) method. With the initial point cloud of the scene as the center, arrange several three-dimensional Gaussian primitives in three-dimensional space and initialize the parameters of the three-dimensional Gaussian primitives, including spatial position, scale, color and opacity.

[0050] S2. Construct each three-dimensional Gaussian element in the scene to be reconstructed as a spiking neuron unit. The spiking neuron unit maintains the state variable of the membrane potential and the firing threshold. The membrane potential is initialized to zero or a predetermined resting potential, and the firing threshold is set according to experience.

[0051] S3. In each training iteration, for each 3D Gaussian, calculate the importance input of the 3D Gaussian based on the current opacity and projection radius, and update the membrane potential of the corresponding spiking neuron unit according to the importance input.

[0052] S4. When the membrane potential corresponding to a certain 3D Gaussian element reaches or exceeds the firing threshold, a pulse firing signal is generated, and the 3D Gaussian element is marked as active; 3D Gaussian elements that do not exceed the firing threshold are marked as inactive and do not participate in rendering and parameter optimization in the current iteration.

[0053] S5. Perform 3D Gaussian Splatting (3DGS) rendering using only the activated 3D Gaussian primitives to obtain the reconstructed image from the training viewpoint, and calculate the reconstruction loss between the reconstructed image and the real image.

[0054] S6. Based on the reconstruction loss, the parameters of the activated 3D Gaussian units and the firing threshold of the spiking neuron units are backpropagated and updated. During multiple iterations, the activation ratio and spatial distribution of the 3D Gaussian units are adaptively adjusted, thereby completing the 3D reconstruction of the scene under multi-view conditions.

[0055] In step S3, projection radius and opacity are key indicators of the importance of 3D Gaussian primitives because they characterize the "degree" and "intensity" of their contribution to 3D Gaussian splash rendering, effectively reflecting the impact of each 3D Gaussian primitive on reconstruction quality. The projection radius determines the projection coverage of the 3D Gaussian primitive on the image plane; a larger projection radius affects a wider area and has a stronger impact on the rendered appearance. Opacity controls light transmission during rendering; higher opacity indicates dominance in the visible area and a stronger impact on local color consistency and structural integrity. Therefore, in each iteration, the opacity of the current 3D Gaussian primitive is read from its parameters. With projection radius The calculation of the importance input includes: using the opacity of three-dimensional Gaussian elements. Its projected radius The product of the squares is used as the original importance quantity. Then, the original importance quantity is logarithmically mapped and normalized to obtain the normalized importance input. Finally, the normalized importance input is used as the input current of the corresponding spiking neuron unit to update the membrane potential.

[0056] Original importance quantity The calculation formula is: Normalized Importance Input The calculation formula is: ,in, , Let represent the original importance quantity of the spiking neuron unit corresponding to the i-th 3D Gaussian element in the t-th iteration. Let represent the original importance quantity of the spiking neuron unit corresponding to the i-th three-dimensional Gaussian element after logarithmic mapping at the t-th iteration; This is the mean value used for normalization in the t-th iteration; This is the standard deviation used for normalization in the t-th iteration; To be placed in the denominator to prevent =0 is a positive number that divides by zero to improve numerical stability; i represents the spiking neuron unit corresponding to the i-th three-dimensional Gaussian element; t represents the t-th iteration.

[0057] The membrane potential update in step S3 satisfies the leakage-integral-release model (LIF):

[0058]

[0059] in, Let be the membrane potential of the spiking neuron unit corresponding to the i-th three-dimensional Gaussian cell at the t-th iteration. The attenuation coefficient is... Let $\frac{i}{t-1}$ represent the firing state of the spiking neuron unit corresponding to the $i$-th three-dimensional Gaussian element during the (t-1)th iteration. This is the normalized importance input for the t-th iteration.

[0060] In step S4, the issuance status satisfies:

[0061] when hour, The corresponding three-dimensional Gaussian primitive is marked as active;

[0062] when hour, The corresponding three-dimensional Gaussian elements are marked as inactive;

[0063] in, The learnable firing threshold for the membrane potential is adaptively updated during training via gradient descent. Based on the activation vector, only activated 3D Gaussian primitives are retained for this round of rendering and gradient updates, while inactive 3D Gaussian primitives are not included in the calculation, but their spiking neuron units continue to receive input in subsequent iterations and may be reactivated. 3D Gaussian splash rendering is performed using the activated 3D Gaussian primitives from the training perspective to obtain the reconstructed image.

[0064] The reconstruction loss in step S5 includes a distance-based pixel-level error term. and perceptual error term based on structural similarity The two are then summed using a weighted average based on preset weights: where, Indicates in and A weighting factor that balances the contributions of each factor. Indicates the reconstructed image. The formula for representing a real image is as follows:

[0065] .

[0066] The training process also includes periodically performing densification and merging operations on the 3D Gaussian units based on the current density distribution and reconstruction error. After performing densification or merging operations, the states of the corresponding 3D Gaussian spiking neuron units are reinitialized or inherited to ensure that the adaptive selection process remains stable on the new set of 3D Gaussian units.

[0067] The apparatus used to obtain multi-view images and corresponding camera poses in step S1 includes:

[0068] A multi-view acquisition module is used to acquire multi-view images from different perspectives of the scene to be acquired. These multi-view images include at least RGB images. The multi-view acquisition module includes at least a frame camera array, which is used to simultaneously acquire multi-view images from multiple perspectives at the same acquisition time. Optionally, the multi-view acquisition module may also include an event camera array, used to acquire corresponding event data near the frame camera acquisition time. The frame camera array consists of multiple industrial RGB frame cameras, supporting configuration of different perspective cameras according to acquisition requirements. During acquisition, the host computer sends acquisition requests to multiple slave computers to drive each slave computer to control its corresponding frame camera to complete exposure and output images at the same time, thereby forming multi-view frame images at the same moment. When an event camera array is included, a data alignment processing module performs time-trimming, noise reduction, and frame conversion on the event data to generate event visualization images or event frames that correspond to the multi-view frame images in time, serving as supplementary data to the frame image set. Specifically, the multi-view acquisition module includes 8 RTX 3090 GPU computers (including 1 host computer and 7 slave computers), a switch group, a server, a main control board, and 44 industrial RGB cameras (2448 resolution). 2048 pixels, 8-bit pixel depth, maximum frame rate 79fps), 10 event cameras (camera resolution 1280). 720), several trigger boards, several LED lights, several bracket control boxes, brackets, and cabinets.

[0069] Synchronization trigger module: This module provides clock synchronization and trigger signals to the multi-view acquisition module to achieve time alignment of the acquired multi-view images. Specifically, the synchronization trigger module uses a master-slave architecture to synchronize and unify the time reference of the frame camera array. When the host computer sends frame image acquisition requests to different slave computers, the master camera under each slave computer outputs a clock synchronization pulse signal to the slave camera through a time synchronization line to achieve time reference consistency among multiple frame cameras. This ensures synchronized exposure at the same acquisition time under a unified time reference, thereby guaranteeing the time consistency of the multi-view frame image set. When the multi-view acquisition module optionally includes an event camera array, the synchronization trigger module sends a trigger signal to the event camera array. The event camera writes the trigger signal into the event data stream as a special event marker. The data alignment processing module performs synchronous cropping of the event data stream based on the special event marker, and uses the acquisition time of the frame camera as the alignment reference to obtain event data corresponding one-to-one with the multi-view frame image set.

[0070] The calibration and pose estimation module is used to acquire the intrinsic parameters of cameras from various viewpoints and perform extrinsic parameter estimation based on the multi-view images to obtain the corresponding camera pose. Specifically, the module uses a method of calibrating intrinsic and extrinsic parameters separately. Intrinsic parameters are obtained by taking approximately 10 images per camera using a checkerboard pattern, achieving a standard deviation of 0.1 pixels. After intrinsic parameter calibration, the lens is locked. With the intrinsic parameters known, all camera extrinsic parameters are calibrated using a custom extrinsic parameter calibration board. After calibration, the support remains fixed. This module acquires the camera intrinsic parameters and estimates the camera extrinsic parameters based on the acquired data to obtain the camera pose.

[0071] Data alignment processing module: used to align, crop or extract frames of multi-view images based on timestamp information to generate multi-view image sequences;

[0072] Data storage and output module: used to organize the multi-view image sequence, camera intrinsic parameters and camera pose into a training dataset and output it.

[0073] This experiment was run on an RTX 3090 GPU with 24GB of memory. All model parameters were trained using the Adam optimizer, with momentum factors set to 0.9 and 0.999, respectively. The proposed method was trained for 10,000 iterations, with densification performed every 100 iterations. Simultaneously, to ensure stable adaptation of the firing threshold of the spiking neuron units, we introduced an auxiliary optimizer specifically for the spiking neuron units. The firing threshold parameters were trained using Adam with a learning rate of 5 × 10⁻⁶. -4 Regarding datasets, we evaluated our experimental method on the LLFF and Mip-NeRF360 datasets.

[0074] Example 1: 3D Gaussian adaptive reconstruction process under constraints;

[0075] (1) Data preparation and initialization:

[0076] The SfM tool is used to perform feature matching and camera pose estimation on the input multi-view image to obtain a sparse point cloud. Then, a dense point cloud is generated based on dense matching, and the positions of three-dimensional Gaussian elements are initialized in three-dimensional space accordingly. The color of each three-dimensional Gaussian element is initialized to the average color of its neighboring pixels projected onto the image. The opacity is initialized to 0.1, and the scale and rotation are initialized to zero or unit values.

[0077] (2) Initialization of spiking neuron units:

[0078] A spiking neuron unit was constructed for each 3D Gaussian cell, with the initial membrane potential set to 0, the firing threshold initialized to 0.8, and the attenuation coefficient set to... Set to 0.5, and the total time step T (number of iterations) to 20; in the implementation, the membrane potentials corresponding to all three-dimensional Gaussian elements can be vectorized and stored and updated in parallel on the GPU.

[0079] (3) Training iteration:

[0080] In each iteration, a batch of views is sampled from the training set, and the following is performed on each 3D Gaussian primitive:

[0081] 1) Calculate its projection radius on the image plane based on the current scale and camera parameters. ;

[0082] 2) Read the current opacity ,calculate and through And batch normalization to obtain input ;

[0083] 3) Update membrane potential according to LIF model And generate the distribution status based on the distribution threshold. After dispensing, the membrane potential will be decayed or reset according to a preset ratio;

[0084] 4) If If the condition is met, the 3D Gaussian primitive is included in the current rendering set; otherwise, it is excluded from the current rendering set.

[0085] For the activated 3D Gaussian primitive set, a reconstructed image from the training viewpoint is generated using the 3DGS renderer, and then compared with the real image. The loss is then backpropagated to update the parameters of the three-dimensional Gaussian cells and the firing threshold of the membrane potential of the spiking neuron units.

[0086] (4) Training termination and reasoning:

[0087] When the iteration reaches the predetermined number of rounds or the loss converges, the final three-dimensional Gaussian set and the state of the spiking neuron unit are obtained;

[0088] During the inference phase, only the three-dimensional Gaussian units with an average activation frequency higher than the firing threshold during training are used as the final representation to achieve the effect of model compression.

[0089] Example 2: Scene-Adaptive Drop Rate Control

[0090] In this embodiment, the firing threshold of the spiking neuron unit As learnable parameters, they are updated through a separate optimizer to adapt to the complexity and sparsity of different scenarios. Specifically, this includes:

[0091] For regions with uniform texture, the importance input is relatively small, the membrane potential is difficult to accumulate over a long period, the firing frequency of spiking neurons is low, and the corresponding 3D Gaussian neurons are more easily discarded, thus increasing the sparsity. For structurally complex or edge regions, the radius and opacity of the 3D Gaussian neurons can continuously provide high importance input from multiple perspectives, and the membrane potential accumulates beyond the threshold in multiple iterations, thus maintaining a high activation rate. See attached diagram. Figure 4 , Figure 5 As shown.

[0092] Experimental results show that our proposed method dynamically adjusts the dropout rate based on scene complexity under given constraints, achieving a higher dropout rate in simpler areas and a lower dropout rate in more complex areas. For example, the dropout rate is 70.9% in a room scene on the Mip-NeRF360 dataset and 88.7% in a tree stump scene. In the room scene, more objects are concentrated in the foreground region, requiring dense representations to preserve fine details. Therefore, our method uses a relatively low dropout rate to avoid dropping primitives. In contrast, in the tree stump scene, textures are mainly concentrated in the background region, allowing our method to effectively sparse the scene. On different scenes of the Mip-NeRF360 dataset, the actual dropout rate generated by our proposed method fluctuates between approximately 70% and 88%, but it still maintains stable or even better PSNR and perceptual quality compared to other state-of-the-art methods, validating the effectiveness of the scene adaptation mechanism.

[0093] Example 3: Dataset building apparatus for constructing conditionally constrained datasets:

[0094] The training and testing datasets required in this embodiment. The device may include a camera array, a synchronization triggering unit, data acquisition and processing software, and a data storage unit.

[0095] (1) Multi-view acquisition and synchronization:

[0096] In one example implementation, the camera array comprises multiple frame cameras arranged in a circular or arc topology around the target at different azimuth angles and heights, forming a multi-view acquisition set covering the target. This array can be flexibly configured with the number of cameras and the field of view range according to acquisition needs, supporting data acquisition from 1 to 44 industrial RGB cameras at different viewpoints. A synchronization trigger unit provides a unified clock and trigger signal to each camera, enabling multi-view images to be acquired and timestamped at the same time reference. See attached diagram. Figure 6 As shown.

[0097] In another example implementation, the camera array can also include an event camera array. This event camera array also employs a master-slave architecture, allowing for flexible configuration of 1 to 10 event cameras to acquire data from different perspectives, depending on the acquisition needs. During acquisition, the master camera outputs a clock pulse signal via a time synchronization line, which is received by the other cameras to unify the time base; the frame cameras and event cameras can achieve soft synchronization by writing special event markers into the event stream using trigger signals. See the attached diagram. Figure 7 As shown.

[0098] (2) Calibration and pose determination:

[0099] Intrinsic parameter calibration is performed on the frame camera array, and extrinsic parameters for each viewpoint are estimated through multi-view feature matching and motion structure recovery to obtain the camera pose. When the scene is a dynamic object, the pose can be estimated separately for each sampling time or each time window, or an approximately invariant pose configuration within the time window can be used.

[0100] (3) Data alignment and dataset organization:

[0101] Based on timestamps and synchronization markers, images from various perspectives are aligned, cropped, or frames are extracted to generate a time-aligned multi-view image sequence. When event data is included, denoising, hot pixel masking, time cropping, and frame conversion are performed on the event stream to obtain event frames or event visualization images aligned with the frame images. Finally, the "multi-view images under constraints + camera intrinsics + camera pose" are output in a preset directory structure to form a dataset, which is used as the data input for the method of this invention.

Claims

1. A three-dimensional Gaussian adaptive reconstruction method under certain constraints, characterized in that, Includes the following steps: S1. Obtain multi-view images of the scene to be reconstructed and the corresponding camera poses. Estimate the camera intrinsic and extrinsic parameters of the multi-view images to obtain the pose information of each multi-view image. Construct the initial point cloud of the scene based on the structure of motion recovery (SfM) or the multi-view stereo method (MVS). With the initial point cloud of the scene as the center, arrange several three-dimensional Gaussian primitives in three-dimensional space and initialize the parameters of the three-dimensional Gaussian primitives, including spatial position, scale, color and opacity. S2. Construct each three-dimensional Gaussian element in the scene to be reconstructed as a spiking neuron unit. The spiking neuron unit maintains the state variable of the membrane potential and the firing threshold. The membrane potential is initialized to zero or a predetermined resting potential, and the firing threshold is set according to experience. S3. In each training iteration, for each 3D Gaussian, calculate the importance input of the 3D Gaussian based on the current opacity and projection radius, and update the membrane potential of the corresponding spiking neuron unit according to the importance input. S4. When the membrane potential corresponding to a certain 3D Gaussian element reaches or exceeds the firing threshold, a pulse firing signal is generated, and the 3D Gaussian element is marked as active; 3D Gaussian elements that do not exceed the firing threshold are marked as inactive and do not participate in rendering and parameter optimization in the current iteration. S5. Perform 3D Gaussian splash rendering using only the activated 3D Gaussian primitives to obtain the reconstructed image from the training viewpoint, and calculate the reconstruction loss between the reconstructed image and the real image. S6. Based on the reconstruction loss, the parameters of the activated 3D Gaussian units and the firing threshold of the spiking neuron units are backpropagated and updated. During multiple iterations, the activation ratio and spatial distribution of the 3D Gaussian units are adaptively adjusted, thereby completing the 3D reconstruction of the scene under multi-view conditions.

2. The three-dimensional Gaussian adaptive reconstruction method under conditional constraints according to claim 1, characterized in that, In step S3, the calculation of the importance input includes: using the opacity of three-dimensional Gaussian elements. Its projected radius The product of the squares is used as the original importance quantity. Then, the original importance quantity is logarithmically mapped and normalized to obtain the normalized importance input. Finally, the normalized importance input is used as the input current of the corresponding spiking neuron unit to update the membrane potential.

3. The three-dimensional Gaussian adaptive reconstruction method under conditional constraints according to claim 2, characterized in that, Original importance quantity The calculation formula is: Normalized Importance Input The calculation formula is: ,in, , Let represent the original importance quantity of the spiking neuron unit corresponding to the i-th 3D Gaussian element in the t-th iteration. Let represent the original importance quantity of the spiking neuron unit corresponding to the i-th three-dimensional Gaussian element after logarithmic mapping at the t-th iteration; This is the mean value used for normalization in the t-th iteration; This is the standard deviation used for normalization in the t-th iteration; To be placed in the denominator to prevent =0 is a positive number that divides by zero to improve numerical stability; i represents the spiking neuron unit corresponding to the i-th three-dimensional Gaussian element; t represents the t-th iteration.

4. The three-dimensional Gaussian adaptive reconstruction method under certain conditions according to claim 3, characterized in that, The membrane potential update in step S3 satisfies the Leakage-Integral-Discharge (LIF) model: in, Let be the membrane potential of the spiking neuron unit corresponding to the i-th three-dimensional Gaussian cell at the t-th iteration. The attenuation coefficient is... Let $\frac{i}{t-1}$ represent the firing state of the spiking neuron unit corresponding to the $i$-th three-dimensional Gaussian element during the (t-1)th iteration. This is the normalized importance input for the t-th iteration.

5. The three-dimensional Gaussian adaptive reconstruction method under certain conditions according to claim 4, characterized in that, In step S4, the issuance status satisfies: when hour, The corresponding three-dimensional Gaussian primitive is marked as active; when hour, The corresponding three-dimensional Gaussian elements are marked as inactive; in, The firing threshold of the membrane potential is a learnable parameter that is adaptively updated during training via gradient descent.

6. The three-dimensional Gaussian adaptive reconstruction method under conditional constraints according to claim 5, characterized in that, The reconstruction loss in step S5 includes a distance-based pixel-level error term. and perceptual error term based on structural similarity The two are then summed using a weighted average based on preset weights: where, Indicates in and A weighting factor that balances the contributions of each factor. Indicates the reconstructed image. The formula for representing a real image is as follows: 。 7. The three-dimensional Gaussian adaptive reconstruction method under conditional constraints according to claim 6, characterized in that, The training process also includes: periodically performing densification and merging operations on the three-dimensional Gaussian units based on the current density distribution and reconstruction error; after performing densification or merging operations, re-initializing or inheriting the state of the spiking neuron units of the corresponding three-dimensional Gaussian units to ensure that the adaptive selection process remains stable on the new set of three-dimensional Gaussian units.

8. The three-dimensional Gaussian adaptive reconstruction method under certain conditions according to claim 6, characterized in that, The apparatus used to obtain multi-view images and corresponding camera poses in step S1 includes: Multi-view acquisition module: used to acquire multi-view images from different perspectives of the scene to be acquired, wherein the multi-view images include at least RGB images; Synchronization trigger module: used to provide clock synchronization signal and trigger signal to the multi-view acquisition module to achieve time alignment of the acquired multi-view images; Calibration and pose solving module: used to obtain the intrinsic parameters of cameras from each viewpoint, and perform camera extrinsic parameter estimation based on the multi-view images to obtain the corresponding camera pose; Data alignment processing module: used to align, crop or extract frames of multi-view images based on timestamp information to generate multi-view image sequences; Data storage and output module: used to organize the multi-view image sequence, camera intrinsic parameters and camera pose into a training dataset and output it.

Images