Three-dimensional Gaussian adaptive reconstruction method under condition limitation

Abstract

The invention relates to the technical field of computer graphics and three-dimensional reconstruction, and provides a three-dimensional Gaussian adaptive reconstruction method under condition limitation. Aiming at the problems that the existing three-dimensional Gaussian reconstruction is easy to overfit under the condition limitation, depends on external prior, is high in calculation overhead and the like, pulse neurons are introduced to carry out time accumulation evaluation and adaptive screening on the importance of Gaussian primitives. The radius and opacity are mapped into input potentials, activation and discard marks are generated through leakage integration and distribution, primitives participating in rendering and optimization are dynamically selected in training, and only the primitives with significant contributions are reserved. A fixed discarding rate is not needed, the participation proportion can be adaptively adjusted along with scenes and stages, redundancy expansion is inhibited, video memory and rendering overhead is reduced, and generalization of unseen view angles is improved. According to experiments, 50%-70% of elements can be discarded on LLFF and Mip-NeRF360, and the quality is not reduced. The device is used for collecting and outputting time-aligned multi-view images, camera internal reference and poses.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer graphics and three-dimensional reconstruction technology, and particularly relates to a three-dimensional Gaussian adaptive reconstruction method under conditional constraints. BACKGROUND

[0002] Three-dimensional reconstruction and novel view synthesis are basic capabilities in virtual reality, augmented reality, robot navigation, digital twin and other applications, aiming to recover the scene geometry from the observation view and generate realistic images under arbitrary views. With the application moving from offline display to interactive and real-time systems, the comprehensive requirements for reconstruction quality, computational efficiency and deployment cost are constantly improving, triggering continuous exploration of more efficient three-dimensional representation and rendering methods.

[0003] In traditional methods, Neural Radiance Fields (NeRF) realizes high-fidelity reconstruction of complex scenes through implicit field representation and differentiable volume rendering, promoting the development of the field. However, NeRF relies on dense view acquisition and a large number of volume sampling, with long training time and slow inference speed. To alleviate the above bottlenecks, three-dimensional Gaussian splatting (3D Gaussian Splatting, 3DGS) as an efficient and differentiable explicit representation has attracted attention. This method models the scene as a set of 3D Gaussian primitives and jointly optimizes the position, covariance, opacity and color of the primitives. It can be initialized by sparse point clouds and achieve high efficiency rendering using rasterization, achieving higher rendering efficiency and lower computational load, and showing obvious advantages in high-resolution visualization and interactive applications.

[0004] However, the above methods all assume that the training data has sufficient view coverage and good imaging quality. In real environments, affected by device layout, acquisition cost and imaging conditions, training data often presents conditions such as sparse views, incomplete coverage, noise blur or exposure changes. Under the condition of conditional constraints, insufficient input observations will amplify geometric and appearance ambiguities, causing NeRF to have slow convergence, degenerative reconstruction and insufficient generalization ability for unseen views. Although 3DGS has stronger efficiency, it is still prone to overfitting to visible areas under conditional constraints, producing floating artifacts in unseen views, and the number of three-dimensional Gaussian primitives rapidly expands during optimization, resulting in significant rendering and video memory overhead, which contradicts the resource-constrained deployment requirements.

[0005] To solve the above problems, the existing technologies mainly include two categories to improve the generalization ability and computational efficiency under the condition restriction: one is to introduce external prior constraints such as monocular depth, generative model or optical flow to enhance geometric consistency or appearance constraints, but this method usually relies on prior accuracy and introduces additional models or computational processes, thereby increasing system complexity and computational overhead; the other is to refer to the random dropout idea, and randomly mask part of the three-dimensional Gaussian primitives during the training process to improve the attention to low visibility primitives, but this method is sensitive to hyperparameters such as dropout rate, and lacks distinction of the importance of primitives, which is prone to the problem of deleting key structures or retaining redundant primitives.

[0006] Therefore, there is an urgent need for a three-dimensional Gaussian adaptive reconstruction method under the condition restriction, which does not rely on external priori and does not introduce complex network structure, and selects the three-dimensional Gaussian primitive importance for adaptive screening, effectively suppresses the redundancy expansion of primitives and computational overhead, and improves the generalization ability of unobserved view angles, thereby realizing efficient and robust realistic three-dimensional Gaussian adaptive reconstruction under the condition restriction. SUMMARY

[0007] The purpose of the present application is to overcome the shortcomings of the prior art, and to provide a three-dimensional Gaussian adaptive reconstruction technology under the condition restriction, which realizes efficient and robust realistic three-dimensional Gaussian adaptive reconstruction under the condition restriction by introducing a pulse neuron model to evaluate and adaptively screen the importance of three-dimensional Gaussian primitives without additional priori and manual adjustment of the dropout rate. Meanwhile, the present application also provides a data set construction device for constructing the training data set of the method, which is used to collect and output time-aligned multi-view images, camera intrinsic parameters and camera poses under the condition restriction of real scenes, so as to provide data input for the method.

[0008] The technical scheme of the present application is as follows: A three-dimensional Gaussian adaptive reconstruction method under the condition restriction, comprising the following steps: S1, obtaining multi-view images and corresponding camera poses of a scene to be reconstructed, estimating camera intrinsic and extrinsic parameters of the multi-view images, obtaining pose information of each multi-view image, constructing an initial point cloud of the scene based on structure from motion (SfM) or multi-view stereo (MVS), and arranging a plurality of three-dimensional Gaussian primitives in the three-dimensional space based on the initial point cloud of the scene, and initializing parameters of the three-dimensional Gaussian primitives, including spatial position, scale, color and opacity; S2, constructing each three-dimensional Gaussian primitive in the scene to be reconstructed as a pulse neuron unit, the pulse neuron unit internally maintaining a state variable of membrane potential and a firing threshold, initializing the membrane potential to zero or a predetermined resting potential, and setting the firing threshold according to experience; S3. In each training iteration, for each 3D Gaussian, calculate the importance input of the 3D Gaussian based on the current opacity and projection radius, and update the membrane potential of the corresponding spiking neuron unit according to the importance input. S4. When the membrane potential corresponding to a certain 3D Gaussian element reaches or exceeds the firing threshold, a pulse firing signal is generated, and the 3D Gaussian element is marked as active; 3D Gaussian elements that do not exceed the firing threshold are marked as inactive and do not participate in rendering and parameter optimization in the current iteration. S5. Perform 3D Gaussian Splatting (3DGS) rendering using only the activated 3D Gaussian primitives to obtain the reconstructed image from the training viewpoint, and calculate the reconstruction loss between the reconstructed image and the real image. S6. Based on the reconstruction loss, the parameters of the activated 3D Gaussian units and the firing threshold of the spiking neuron units are backpropagated and updated. During multiple iterations, the activation ratio and spatial distribution of the 3D Gaussian units are adaptively adjusted, thereby completing the 3D reconstruction of the scene under multi-view conditions.

[0009] Further, in step S3, the calculation of the importance input includes: using the opacity of three-dimensional Gaussian elements. Its projected radius The product of the squares is used as the original importance quantity. Then, the original importance quantity is logarithmically mapped and normalized to obtain the normalized importance input. Finally, the normalized importance input is used as the input current of the corresponding spiking neuron unit to update the membrane potential.

[0010] Furthermore, the original importance quantity The calculation formula is: Normalized Importance Input The calculation formula is: ,in, , Let represent the original importance quantity of the spiking neuron unit corresponding to the i-th 3D Gaussian element in the t-th iteration. Let represent the original importance quantity of the spiking neuron unit corresponding to the i-th three-dimensional Gaussian element after logarithmic mapping at the t-th iteration; This is the mean value used for normalization in the t-th iteration; This is the standard deviation used for normalization in the t-th iteration; To be placed in the denominator to prevent =0 is a positive number that divides by zero to improve numerical stability; i represents the spiking neuron unit corresponding to the i-th three-dimensional Gaussian element; t represents the t-th iteration.

[0011] Furthermore, the membrane potential update in step S3 satisfies the leakage-integral-release model (LIF): in, Let be the membrane potential of the spiking neuron unit corresponding to the i-th three-dimensional Gaussian cell at the t-th iteration. The attenuation coefficient is... Let $\frac{i}{t-1}$ represent the firing state of the spiking neuron unit corresponding to the $i$-th three-dimensional Gaussian element during the (t-1)th iteration. This is the normalized importance input for the t-th iteration.

[0012] Further, in step S4, the issuance state satisfies: when hour, The corresponding three-dimensional Gaussian primitive is marked as active; when hour, The corresponding three-dimensional Gaussian elements are marked as inactive; in, The firing threshold of the membrane potential is a learnable parameter that is adaptively updated during training via gradient descent.

[0013] Furthermore, the reconstruction loss in step S5 includes a distance-based pixel-level error term. and perceptual error term based on structural similarity The two are then summed using a weighted average based on preset weights: where, Indicates in and A weighting factor that balances the contributions of each factor. Indicates the reconstructed image. The formula for representing a real image is as follows: .

[0014] Furthermore, the training process also includes periodically performing densification and merging operations on the 3D Gaussian units based on the current density distribution and reconstruction error. After performing densification or merging operations, the states of the corresponding spiking neuron units of the 3D Gaussian units are reinitialized or inherited to ensure that the adaptive selection process remains stable on the new set of 3D Gaussian units.

[0015] Furthermore, the device used in step S1 to obtain multi-view images and corresponding camera poses includes: Multi-view acquisition module: used to acquire multi-view images from different perspectives of the scene to be acquired, wherein the multi-view images include at least RGB images; Synchronization trigger module: used to provide clock synchronization signal and trigger signal to the multi-view acquisition module to achieve time alignment of the acquired multi-view images; Calibration and pose solving module: used to obtain the intrinsic parameters of cameras from each viewpoint, and perform camera extrinsic parameter estimation based on the multi-view images to obtain the corresponding camera pose; Data alignment processing module: used to align, crop or extract frames of multi-view images based on timestamp information to generate multi-view image sequences; Data storage and output module: used to organize the multi-view image sequence, camera intrinsic parameters and camera pose into a training dataset and output it.

[0016] Compared with the prior art, the present invention has at least the following beneficial effects: (1) High-quality 3D Gaussian reconstruction under conditional constraints: This invention can maintain efficiency and robustness under conditional constraints, while taking into account high-quality rendering and geometric consistency, thereby achieving efficient, robust and realistic 3D Gaussian adaptive reconstruction under conditional constraints.

[0017] (2) Strong Adaptability: This invention introduces spiking neuron units with time accumulation and leakage characteristics, and adaptively evaluates their long-term importance based on the radius and opacity of the three-dimensional Gaussian units, dynamically deciding whether to participate in training and reconstruction. The firing threshold of the spiking neuron unit is a learnable parameter, which can automatically adjust the actual discard ratio under different scene conditions, maintaining a lower screening intensity for complex structural regions and more aggressively sparsifying large areas of uniform texture, without the need to manually set a fixed discard rate, thus achieving stable adaptive three-dimensional reconstruction.

[0018] (3) Significant sparsity and lightweighting: On public datasets such as LLFF and Mip-NeRF360, this invention can discard more than 50% to 70% of the three-dimensional Gaussian primitives during training, greatly reducing the number of primitives involved in rendering and optimization. While maintaining or exceeding the reconstruction quality of existing methods, it effectively reduces memory usage and computation, and improves deployability under conditions of constraint.

[0019] (4) No external priors required and easy to integrate and deploy: This invention relies solely on the parameters of the 3D Gaussian primitives themselves for importance judgment, without relying on external depth, semantics, or generation priors, thus avoiding the overhead and risks caused by unstable prior quality and additional model inference. At the same time, this method can be implemented on the existing 3DGS framework by only adding a lightweight neuron module, which is compatible with the original rendering pipeline and facilitates integration and deployment in existing systems. Attached Figure Description

[0020] To more clearly illustrate the technical solution of the present invention, the accompanying drawings are briefly described below: Figure 1 This is a schematic diagram of the overall process of three-dimensional Gaussian adaptive reconstruction under certain conditions according to the present invention; Figure 2This is a schematic diagram comparing the PSNR of the present invention with existing state-of-the-art methods with different 3D Gaussian skeletal drop ratios on the Mip-NeRF360 dataset and the LLFF dataset; where (a) is the Mip-NeRF360 dataset and (b) is the LLFF dataset. Figure 3 This is a schematic diagram illustrating the adaptive adjustment of the 3D Gaussian pixel discard ratio based on scene complexity in this invention. Figure 4 This diagram illustrates the reconstruction quality of our invention on the LLFF dataset compared to other existing methods. The columns for 3DGS, FSGS, CoR-GS, and DropGaussian represent other existing methods, the column for Ours represents our method, and the column for GT represents the real images. Figure 5 This diagram illustrates the reconstruction quality of our invention on the Mip-NeRF360 dataset compared to other existing methods. The columns for 3DGS, FSGS, CoR-GS, and DropGaussian represent other existing methods, the column for Ours represents our method, and the column for GT represents the real images. Figure 6 This is a schematic diagram of the overall architecture of the condition-constrained dataset construction device of the present invention; Figure 7 This is a schematic diagram of the clock synchronization scheme of the condition-constrained dataset construction device of the present invention. Detailed Implementation

[0021] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0022] A three-dimensional Gaussian adaptive reconstruction method under certain constraints includes the following steps: S1. Obtain multi-view images of the scene to be reconstructed and the corresponding camera poses. Estimate the camera intrinsic and extrinsic parameters of the multi-view images to obtain the pose information of each multi-view image. Construct the initial point cloud of the scene based on the structure-of-motion (SfM) or multi-view stereo (MVS) method. With the initial point cloud of the scene as the center, arrange several three-dimensional Gaussian primitives in three-dimensional space and initialize the parameters of the three-dimensional Gaussian primitives, including spatial position, scale, color and opacity. S2. Construct each three-dimensional Gaussian element in the scene to be reconstructed as a spiking neuron unit. The spiking neuron unit maintains the state variable of the membrane potential and the firing threshold. The membrane potential is initialized to zero or a predetermined resting potential, and the firing threshold is set according to experience. S3. In each training iteration, for each 3D Gaussian, calculate the importance input of the 3D Gaussian based on the current opacity and projection radius, and update the membrane potential of the corresponding spiking neuron unit according to the importance input. S4. When the membrane potential corresponding to a certain 3D Gaussian element reaches or exceeds the firing threshold, a pulse firing signal is generated, and the 3D Gaussian element is marked as active; 3D Gaussian elements that do not exceed the firing threshold are marked as inactive and do not participate in rendering and parameter optimization in the current iteration. S5. Perform 3D Gaussian Splatting (3DGS) rendering using only the activated 3D Gaussian primitives to obtain the reconstructed image from the training viewpoint, and calculate the reconstruction loss between the reconstructed image and the real image. S6. Based on the reconstruction loss, the parameters of the activated 3D Gaussian units and the firing threshold of the spiking neuron units are backpropagated and updated. During multiple iterations, the activation ratio and spatial distribution of the 3D Gaussian units are adaptively adjusted, thereby completing the 3D reconstruction of the scene under multi-view conditions.

[0023] In step S3, projection radius and opacity are key indicators of the importance of 3D Gaussian primitives because they characterize the "degree" and "intensity" of their contribution to 3D Gaussian splash rendering, effectively reflecting the impact of each 3D Gaussian primitive on reconstruction quality. The projection radius determines the projection coverage of the 3D Gaussian primitive on the image plane; a larger projection radius affects a wider area and has a stronger impact on the rendered appearance. Opacity controls light transmission during rendering; higher opacity indicates dominance in the visible area and a stronger impact on local color consistency and structural integrity. Therefore, in each iteration, the opacity of the current 3D Gaussian primitive is read from its parameters. With projection radius The calculation of the importance input includes: using the opacity of three-dimensional Gaussian elements. Its projected radius The product of the squares is used as the original importance quantity. Then, the original importance quantity is logarithmically mapped and normalized to obtain the normalized importance input. Finally, the normalized importance input is used as the input current of the corresponding spiking neuron unit to update the membrane potential.

[0024] Original importance quantity The calculation formula is: Normalized Importance Input The calculation formula is: ,in, , Let represent the original importance quantity of the spiking neuron unit corresponding to the i-th 3D Gaussian element in the t-th iteration. Let represent the original importance quantity of the spiking neuron unit corresponding to the i-th three-dimensional Gaussian element after logarithmic mapping at the t-th iteration; This is the mean value used for normalization in the t-th iteration; This is the standard deviation used for normalization in the t-th iteration; To be placed in the denominator to prevent =0 is a positive number that divides by zero to improve numerical stability; i represents the spiking neuron unit corresponding to the i-th three-dimensional Gaussian element; t represents the t-th iteration.

[0025] The membrane potential update in step S3 satisfies the leakage-integral-release model (LIF): in, Let be the membrane potential of the spiking neuron unit corresponding to the i-th three-dimensional Gaussian cell at the t-th iteration. The attenuation coefficient is... Let $\frac{i}{t-1}$ represent the firing state of the spiking neuron unit corresponding to the $i$-th three-dimensional Gaussian element during the (t-1)th iteration. This is the normalized importance input for the t-th iteration.

[0026] In step S4, the issuance status satisfies: when hour, The corresponding three-dimensional Gaussian primitive is marked as active; when hour, The corresponding three-dimensional Gaussian elements are marked as inactive; in, The learnable firing threshold for the membrane potential is adaptively updated during training via gradient descent. Based on the activation vector, only activated 3D Gaussian primitives are retained for this round of rendering and gradient updates, while inactive 3D Gaussian primitives are not included in the calculation, but their spiking neuron units continue to receive input in subsequent iterations and may be reactivated. 3D Gaussian splash rendering is performed using the activated 3D Gaussian primitives from the training perspective to obtain the reconstructed image.

[0027] The reconstruction loss in step S5 includes a distance-based pixel-level error term. and perceptual error term based on structural similarity The two are then summed using a weighted average based on preset weights: where, Indicates in and A weighting factor that balances the contributions of each factor. Indicates the reconstructed image. The formula for representing a real image is as follows: .

[0028] The training process also includes periodically performing densification and merging operations on the 3D Gaussian units based on the current density distribution and reconstruction error. After performing densification or merging operations, the states of the corresponding 3D Gaussian spiking neuron units are reinitialized or inherited to ensure that the adaptive selection process remains stable on the new set of 3D Gaussian units.

[0029] The apparatus used to obtain multi-view images and corresponding camera poses in step S1 includes: A multi-view acquisition module is used to acquire multi-view images from different perspectives of the scene to be acquired. These multi-view images include at least RGB images. The multi-view acquisition module includes at least a frame camera array, which is used to simultaneously acquire multi-view images from multiple perspectives at the same acquisition time. Optionally, the multi-view acquisition module may also include an event camera array, used to acquire corresponding event data near the frame camera acquisition time. The frame camera array consists of multiple industrial RGB frame cameras, supporting configuration of different perspective cameras according to acquisition requirements. During acquisition, the host computer sends acquisition requests to multiple slave computers to drive each slave computer to control its corresponding frame camera to complete exposure and output images at the same time, thereby forming multi-view frame images at the same moment. When an event camera array is included, a data alignment processing module performs time-trimming, noise reduction, and frame conversion on the event data to generate event visualization images or event frames that correspond to the multi-view frame images in time, serving as supplementary data to the frame image set. Specifically, the multi-view acquisition module includes 8 RTX 3090 GPU computers (including 1 host computer and 7 slave computers), a switch group, a server, a main control board, and 44 industrial RGB cameras (2448 resolution). 2048 pixels, 8-bit pixel depth, maximum frame rate 79fps), 10 event cameras (camera resolution 1280). 720), several trigger boards, several LED lights, several bracket control boxes, brackets, and cabinets.

[0030] Synchronization trigger module: This module provides clock synchronization and trigger signals to the multi-view acquisition module to achieve time alignment of the acquired multi-view images. Specifically, the synchronization trigger module uses a master-slave architecture to synchronize and unify the time reference of the frame camera array. When the host computer sends frame image acquisition requests to different slave computers, the master camera under each slave computer outputs a clock synchronization pulse signal to the slave camera through a time synchronization line to achieve time reference consistency among multiple frame cameras. This ensures synchronized exposure at the same acquisition time under a unified time reference, thereby guaranteeing the time consistency of the multi-view frame image set. When the multi-view acquisition module optionally includes an event camera array, the synchronization trigger module sends a trigger signal to the event camera array. The event camera writes the trigger signal into the event data stream as a special event marker. The data alignment processing module performs synchronous cropping of the event data stream based on the special event marker, and uses the acquisition time of the frame camera as the alignment reference to obtain event data corresponding one-to-one with the multi-view frame image set.

[0031] The calibration and pose estimation module is used to acquire the intrinsic parameters of cameras from various viewpoints and perform extrinsic parameter estimation based on the multi-view images to obtain the corresponding camera pose. Specifically, the module uses a method of calibrating intrinsic and extrinsic parameters separately. Intrinsic parameters are obtained by taking approximately 10 images per camera using a checkerboard pattern, achieving a standard deviation of 0.1 pixels. After intrinsic parameter calibration, the lens is locked. With the intrinsic parameters known, all camera extrinsic parameters are calibrated using a custom extrinsic parameter calibration board. After calibration, the support remains fixed. This module acquires the camera intrinsic parameters and estimates the camera extrinsic parameters based on the acquired data to obtain the camera pose.

[0032] Data alignment processing module: used to align, crop or extract frames of multi-view images based on timestamp information to generate multi-view image sequences; Data storage and output module: used to organize the multi-view image sequence, camera intrinsic parameters and camera pose into a training dataset and output it.

[0033] This experiment was run on an RTX 3090 GPU with 24GB of memory. All model parameters were trained using the Adam optimizer, with momentum factors set to 0.9 and 0.999, respectively. The proposed method was trained for 10,000 iterations, with densification performed every 100 iterations. Simultaneously, to ensure stable adaptation of the firing threshold of the spiking neuron units, we introduced an auxiliary optimizer specifically for the spiking neuron units. The firing threshold parameters were trained using Adam with a learning rate of 5 × 10⁻⁶. -4 Regarding datasets, we evaluated our experimental method on the LLFF and Mip-NeRF360 datasets.

[0034] Example 1: 3D Gaussian adaptive reconstruction process under constraints; (1) Data preparation and initialization: The SfM tool is used to perform feature matching and camera pose estimation on the input multi-view image to obtain a sparse point cloud. Then, a dense point cloud is generated based on dense matching, and the positions of three-dimensional Gaussian elements are initialized in three-dimensional space accordingly. The color of each three-dimensional Gaussian element is initialized to the average color of its neighboring pixels projected onto the image. The opacity is initialized to 0.1, and the scale and rotation are initialized to zero or unit values.

[0035] (2) Initialization of spiking neuron units: A spiking neuron unit was constructed for each 3D Gaussian cell, with the initial membrane potential set to 0, the firing threshold initialized to 0.8, and the attenuation coefficient set to... Set to 0.5, and the total time step T (number of iterations) to 20; in the implementation, the membrane potentials corresponding to all three-dimensional Gaussian elements can be vectorized and stored and updated in parallel on the GPU.

[0036] (3) Training iteration: In each iteration, a batch of views is sampled from the training set, and the following is performed on each 3D Gaussian primitive: 1) Calculate its projection radius on the image plane based on the current scale and camera parameters. ; 2) Read the current opacity ,calculate and through And batch normalization to obtain input ; 3) Update membrane potential according to LIF model And generate the distribution status based on the distribution threshold. After dispensing, the membrane potential will be decayed or reset according to a preset ratio; 4) If If the condition is met, the 3D Gaussian primitive is included in the current rendering set; otherwise, it is excluded from the current rendering set.

[0037] For the activated 3D Gaussian primitive set, a reconstructed image from the training viewpoint is generated using the 3DGS renderer, and then compared with the real image. The loss is then backpropagated to update the parameters of the three-dimensional Gaussian cells and the firing threshold of the membrane potential of the spiking neuron units.

[0038] (4) Training termination and reasoning: When the iteration reaches the predetermined number of rounds or the loss converges, the final three-dimensional Gaussian set and the state of the spiking neuron unit are obtained; During the inference phase, only the three-dimensional Gaussian units with an average activation frequency higher than the firing threshold during training are used as the final representation to achieve the effect of model compression.

[0039] Example 2: Scene-Adaptive Drop Rate Control In this embodiment, the firing threshold of the spiking neuron unit As learnable parameters, they are updated through a separate optimizer to adapt to the complexity and sparsity of different scenarios. Specifically, this includes: For regions with uniform texture, the importance input is relatively small, the membrane potential is difficult to accumulate over a long period, the firing frequency of spiking neurons is low, and the corresponding 3D Gaussian neurons are more easily discarded, thus increasing the sparsity. For structurally complex or edge regions, the radius and opacity of the 3D Gaussian neurons can continuously provide high importance input from multiple perspectives, and the membrane potential accumulates beyond the threshold in multiple iterations, thus maintaining a high activation rate. See attached diagram. Figure 4 , Figure 5 As shown.

[0040] Experimental results show that our proposed method dynamically adjusts the dropout rate based on scene complexity under given constraints, achieving a higher dropout rate in simpler areas and a lower dropout rate in more complex areas. For example, the dropout rate is 70.9% in a room scene on the Mip-NeRF360 dataset and 88.7% in a tree stump scene. In the room scene, more objects are concentrated in the foreground region, requiring dense representations to preserve fine details. Therefore, our method uses a relatively low dropout rate to avoid dropping primitives. In contrast, in the tree stump scene, textures are mainly concentrated in the background region, allowing our method to effectively sparse the scene. On different scenes of the Mip-NeRF360 dataset, the actual dropout rate generated by our proposed method fluctuates between approximately 70% and 88%, but it still maintains stable or even better PSNR and perceptual quality compared to other state-of-the-art methods, validating the effectiveness of the scene adaptation mechanism.

[0041] Example 3: Dataset building apparatus for constructing conditionally constrained datasets: The training and testing datasets required in this embodiment. The device may include a camera array, a synchronization triggering unit, data acquisition and processing software, and a data storage unit.

[0042] (1) Multi-view acquisition and synchronization: In one example implementation, the camera array comprises multiple frame cameras arranged in a circular or arc topology around the target at different azimuth angles and heights, forming a multi-view acquisition set covering the target. This array can be flexibly configured with the number of cameras and the field of view range according to acquisition needs, supporting data acquisition from 1 to 44 industrial RGB cameras at different viewpoints. A synchronization trigger unit provides a unified clock and trigger signal to each camera, enabling multi-view images to be acquired and timestamped at the same time reference. See attached diagram. Figure 6 As shown.

[0043] In another example implementation, the camera array can also include an event camera array. This event camera array also employs a master-slave architecture, allowing for flexible configuration of 1 to 10 event cameras to acquire data from different perspectives, depending on the acquisition needs. During acquisition, the master camera outputs a clock pulse signal via a time synchronization line, which is received by the other cameras to unify the time base; the frame cameras and event cameras can achieve soft synchronization by writing special event markers into the event stream using trigger signals. See the attached diagram. Figure 7 As shown.

[0044] (2) Calibration and pose determination: Intrinsic parameter calibration is performed on the frame camera array, and extrinsic parameters for each viewpoint are estimated through multi-view feature matching and motion structure recovery to obtain the camera pose. When the scene is a dynamic object, the pose can be estimated separately for each sampling time or each time window, or an approximately invariant pose configuration within the time window can be used.

[0045] (3) Data alignment and dataset organization: Based on timestamps and synchronization markers, images from various perspectives are aligned, cropped, or frames are extracted to generate a time-aligned multi-view image sequence. When event data is included, denoising, hot pixel masking, time cropping, and frame conversion are performed on the event stream to obtain event frames or event visualization images aligned with the frame images. Finally, the "multi-view images under constraints + camera intrinsics + camera pose" are output in a preset directory structure to form a dataset, which is used as the data input for the method of this invention.

Claims

1. A method of 3D Gaussian adaptive reconstruction under condition constraints, characterized in that, The method comprises the following steps: S1, obtaining multi-view images of a scene to be reconstructed and corresponding camera poses, performing camera internal and external parameter estimation on the multi-view images to obtain pose information of each multi-view image, constructing an initial point cloud of the scene based on a motion structure recovery SfM or a multi-view stereo method MVS, and arranging a plurality of three-dimensional Gaussian primitives in a three-dimensional space with the initial point cloud of the scene as the center, and initializing parameters of the three-dimensional Gaussian primitives, the parameters including a spatial position, a scale, a color and an opacity; S2, constructing each three-dimensional Gaussian primitive in the scene to be reconstructed into a pulse neuron unit, the pulse neuron unit internally maintaining a state variable of a membrane potential and a firing threshold, initializing the membrane potential to zero or a predetermined resting potential, and setting the firing threshold according to experience; S3, in each training iteration, for each three-dimensional Gaussian primitive, calculating an importance input of the three-dimensional Gaussian primitive according to a current opacity and a projection radius, and updating a membrane potential of a corresponding pulse neuron unit according to the importance input; S4, when the membrane potential of a certain three-dimensional Gaussian primitive reaches or exceeds the firing threshold, a pulse firing signal is generated, and the three-dimensional Gaussian primitive is marked as an activated state; three-dimensional Gaussian primitives that do not exceed the firing threshold are marked as non-activated states and do not participate in rendering and parameter optimization in the current iteration; S5, performing three-dimensional Gaussian splatting rendering only by using the three-dimensional Gaussian primitives in the activated state to obtain a reconstructed image of a training view, and calculating a reconstruction loss between the reconstructed image and a real image; S6, based on the reconstruction loss, performing back propagation and updating on parameters of the activated three-dimensional Gaussian primitives and the firing threshold of the pulse neuron unit, and adaptively adjusting an activation proportion and a spatial distribution of the three-dimensional Gaussian primitives in a plurality of iterations, so as to complete three-dimensional reconstruction of the scene under multi-view conditions.

2. The method of claim 1, wherein the method is a 3D Gaussian adaptive reconstruction method under condition restriction, characterized in that, In step S3, the calculation of the importance input includes: using the opacity of three-dimensional Gaussian elements. Its projected radius The product of the squares is used as the original importance quantity. Then, the original importance quantity is logarithmically mapped and normalized to obtain the normalized importance input. Finally, the normalized importance input is used as the input current of the corresponding spiking neuron unit to update the membrane potential.

3. The method of claim 2, wherein the method is a 3D Gaussian adaptive reconstruction method under condition restriction, characterized in that, Original importance quantity The calculation formula is: Normalized Importance Input The calculation formula is: ,in, , Let represent the original importance quantity of the spiking neuron unit corresponding to the i-th 3D Gaussian element in the t-th iteration. Let represent the original importance quantity of the spiking neuron unit corresponding to the i-th three-dimensional Gaussian element after logarithmic mapping at the t-th iteration; This is the mean value used for normalization in the t-th iteration; This is the standard deviation used for normalization in the t-th iteration; To be placed in the denominator to prevent =0 is a positive number that divides by zero to improve numerical stability; i represents the spiking neuron unit corresponding to the i-th three-dimensional Gaussian element; t represents the t-th iteration.

4. The method of claim 3, wherein the method is a 3D Gaussian adaptive reconstruction method under condition restriction, characterized in that, The updating of the membrane potential in step S3 satisfies a leaky-integrate-fire model LIF: wherein, is the membrane potential of the i-th three-dimensional Gaussian basis function corresponding pulse neuron unit at the t-th iteration, is the decay coefficient, is the firing state of the i-th three-dimensional Gaussian basis function corresponding pulse neuron unit at the t-1-th iteration, is the normalized importance input at the t-th iteration.

5. The method of claim 4, wherein the method is a 3D Gaussian adaptive reconstruction method under condition restriction, characterized in that, In step S4, the firing state satisfies: When the corresponding three-dimensional Gaussian cell is marked as active; , the corresponding three-dimensional Gaussian cell is marked as active; When time, corresponding three-dimensional Gaussian cell is marked as an inactive state; wherein, is the learnable firing threshold of the membrane potential, which is updated adaptively by gradient descent during training.

6. The method of claim 5, wherein the method is a 3D Gaussian adaptive reconstruction method under condition restriction, characterized in that, The reconstruction loss in step S5 comprises a distance-based pixel-wise error term and a structure similarity-based perceptual error term both weighted by a pre-set weight: where, denotes a weight factor balancing the contributions of and , denotes the reconstructed image, denotes the real image and is formulated as follows: 。 7. The method of claim 6, wherein the method is a 3D Gaussian adaptive reconstruction method under condition restriction, characterized in that, In the training process, it also includes periodically performing a densification and merging operation of the three-dimensional Gaussian primitives according to a density distribution of the current three-dimensional Gaussian primitives and a reconstruction error; after performing the densification or merging operation, the state of the pulse neuron unit of the corresponding three-dimensional Gaussian primitive is reinitialized or inherited to ensure that the adaptive screening process remains stable on a new set of three-dimensional Gaussian primitives.

8. The method of claim 6, wherein the method is a 3D Gaussian adaptive reconstruction method under condition restriction, characterized in that, The device used to obtain the multi-view images and the corresponding camera poses in step S1 comprises: A multi-view acquisition module for acquiring multi-view images from different views of a scene to be acquired, the multi-view images at least including RGB images; A synchronous triggering module for providing a clock synchronization signal and a triggering signal to the multi-view acquisition module to realize time alignment of the acquired multi-view images; A calibration and pose solving module for obtaining internal parameters of each view camera, and performing camera external parameter estimation based on the multi-view images to obtain corresponding camera poses; A data alignment processing module for aligning, cropping or frame extracting the multi-view images according to timestamp information to generate a multi-view image sequence; A data storage and output module for organizing the multi-view image sequence, the internal parameters of the cameras and the camera poses into a training data set and outputting.

Images