A method and system for dynamic three-dimensional reconstruction of Gaussian splashes based on spiking neurons

Abstract

The application discloses a Gaussian splatting dynamic three-dimensional reconstruction method and system based on a pulse neuron, and comprises the following steps: video recording of a reconstruction target is performed from multiple perspectives to obtain a two-dimensional RGB image sequence and time dimension data; based on the two-dimensional RGB image sequence, sparse point cloud data and camera poses are calculated by using a COLMAP algorithm, and a three-dimensional Gaussian ellipsoid is initialized based on the sparse point cloud data; a Gaussian deformation field network is constructed, time dimension data and position information of the three-dimensional Gaussian ellipsoid are input, and the Gaussian deformation field network is used for training the deformation field of the Gaussian ellipsoid changing with time; based on the camera poses and the two-dimensional RGB image sequence, parameters of the three-dimensional Gaussian ellipsoid and the deformation field are optimized by using a Gaussian splatting algorithm based on the pulse neuron, the optimized three-dimensional Gaussian ellipsoid and the deformation field are output, an RGB image under a specific pose state and time is rendered, and dynamic three-dimensional reconstruction is realized. The application significantly reduces Gaussian storage redundancy, suppresses the generation of reconstruction artifacts, and improves the rendering and training efficiency.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction in computer vision, and in particular to a Gaussian splash dynamic 3D reconstruction method and system based on spiking neurons. Background Technology

[0002] A spiking neuron is a biologically inspired neuron model that mimics the way neurons work in the brain. Unlike traditional artificial neurons, spiking neurons transmit signals in the form of pulses only when their input reaches a certain threshold, rather than continuously outputting numerical values ​​like traditional neurons. Due to their energy efficiency advantages, they have been widely used in recent years in fields such as video or event signal processing, classification tasks, and adversarial attacks.

[0003] 3D reconstruction in computer vision and graphics refers to acquiring images using devices such as digital cameras, and then using algorithms to estimate the 3D information of the captured scene or object for related applications. Currently, 3D reconstruction is widely used in fields such as autonomous driving, computational graphics, computer animation, and robot navigation. Deep learning-based 3D reconstruction methods mainly include methods based on implicit field representations and methods based on 3D Gaussian representations. For example, Chinese patent document CN120198592A discloses a 3D image reconstruction method based on a large model; Chinese patent document CN120411419A discloses an interpretable 3D reconstruction method based on decoupled representation learning and Gaussian splashing.

[0004] 3D Gaussian Splatting (3DGS) is an emerging 3D reconstruction method that represents a 3D scene by optimizing a set of parameters of a 3D Gaussian ellipsoid. Existing 3DGS-based 3D reconstruction methods often involve a large number of low-opacity Gaussians (LOG) during the dynamic reconstruction process, which can easily lead to motion artifacts in the reconstructed objects. Furthermore, 3DGS-based 3D reconstruction methods often require a large number of Gaussians for computation and scene representation, consuming significant reconstruction time and storage space, making them particularly unsuitable for resource-constrained applications. Summary of the Invention

[0005] This invention provides a dynamic three-dimensional reconstruction method and system based on Gaussian splashing of spiking neurons, which can solve the problems of motion artifacts and high reconstruction costs in the dynamic reconstruction process of existing technologies.

[0006] A dynamic 3D reconstruction method based on Gaussian splashing using spiking neurons includes:

[0007] (1) Record video of the reconstructed target from multiple perspectives to obtain two-dimensional RGB image sequences and time dimension data;

[0008] (2) Based on the two-dimensional RGB image sequence, the sparse point cloud data and camera pose are calculated using the COLMAP algorithm, and a set of three-dimensional Gaussian ellipsoids are initialized based on the sparse point cloud data.

[0009] (3) Construct a Gaussian deformation field network, input time dimension data and the position information of the three-dimensional Gaussian ellipsoid, and use it to train the deformation field of the Gaussian ellipsoid as it changes over time;

[0010] (4) Based on the camera pose and two-dimensional RGB image sequence, the parameters of the three-dimensional Gaussian ellipsoid and deformation field are optimized using the Gaussian splashing algorithm based on spiking neurons. After iterative optimization, the optimized three-dimensional Gaussian ellipsoid and deformation field are output.

[0011] (5) Dynamic three-dimensional reconstruction is achieved by rendering specific pose states and RGB images at specific times based on the optimized three-dimensional Gaussian ellipsoid and deformation field.

[0012] In step (2), based on the two-dimensional RGB image sequence, the sparse point cloud data and camera pose are calculated using the COLMAP algorithm. The specific process is as follows:

[0013] (2-1) Perform SIFT feature extraction based on the image sequence, and perform feature matching based on the feature extraction results;

[0014] (2-2) Filter out incorrect matches using geometric consistency to obtain the verified feature point set;

[0015] (2-3) The least squares method is used to globally optimize the camera pose and the 3D sparse point cloud based on the matched feature point set.

[0016] In step (2), the parameters of each three-dimensional Gaussian ellipsoid include opacity. Center point location Scale vector and quaternions Parametrically represented rotation matrix ; Represents the space of real numbers.

[0017] In step (3), the Gaussian deformation field network is an MLP neural network, which contains 8 fully connected neural network layers and 3 linear layers;

[0018] The first eight fully connected neural network layers are activated by ReLU layers and output a 256-dimensional feature vector; the output of the fourth layer is concatenated with the input of the first layer as the input of the fifth layer; the output of the eighth layer is input to three different linear layers respectively; finally, the output is the deformation field of a three-dimensional Gaussian ellipsoid that varies with time.

[0019] In step (3), the deformation field includes position change, rotation change and scaling change.

[0020] The specific process of step (4) is as follows:

[0021] (4-1) Construct a global FIF spiking neuron and initialize the spiking threshold. for ;

[0022] (4-2) The opacity information of the Gaussian ellipsoid is passed through a global FIF spiking neuron, and values ​​less than the spiking threshold are processed. Gaussian ellipsoidal opacity Set it to 0, and this part of the Gaussian ellipsoid will not be displayed in subsequent calculations;

[0023] (4-3) Input the position and time information of the three-dimensional Gaussian ellipsoid into the Gaussian deformation field network, and the deformation field network outputs the deformation field of the three-dimensional Gaussian ellipsoid as it changes over time;

[0024] (4-4) Add the deformation field of the output three-dimensional Gaussian ellipsoid over time to the original parameters corresponding to the three-dimensional Gaussian ellipsoid to obtain the deformed three-dimensional Gaussian ellipsoid.

[0025] (4-5) Based on the camera pose in the current iteration, perform a projection operation on the deformed 3D Gaussian ellipsoid to obtain the 2D representation of the 3D Gaussian ellipsoid in the image plane;

[0026] (4-6) The 2D representation is rasterized into a differentiable form. Alpha blending is performed according to the spatial depth order of the three-dimensional Gaussian ellipsoid corresponding to each 2D representation. Finally, the rendered image of the dynamic scene at the current time and viewpoint is obtained.

[0027] (4-7) Perform loss calculation on the obtained rendered image and the original RGB image, and perform gradient backpropagation to optimize the parameters of the three-dimensional Gaussian ellipsoid and deformation field;

[0028] (4-8) Density control of the Gaussian ellipsoid is performed based on the gradient calculation results.

[0029] In steps (4-6), the formula for calculating alpha mixing is as follows:

[0030] ;

[0031] ;

[0032] ;

[0033] in, It is a pixel. The value, This indicates the number of Gaussian ellipsoids involved in the rendering. It is the RGB color of the Gaussian ellipsoid. It is the Gaussian distribution after the splash. It is the first Opacity after Gaussian weighting Indicates the rendered number of times. Opacity after Gaussian weighting Indicates the first Gaussian weighted pre-opacity; by traversing pixels You will get the entire rendered RGB image.

[0034] In steps (4-7), the expression for calculating the loss is as follows:

[0035] ;

[0036] in, For luminance loss, For D-SSIM loss, For pulse threshold loss, and This is a hyperparameter.

[0037] Pulse threshold loss The expression is:

[0038] ;

[0039] in, The opacity value of the Gaussian ellipsoid after processing by a global FIF spiking neuron.

[0040] A Gaussian splash dynamic 3D reconstruction device based on spiking neurons includes a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned Gaussian splash dynamic 3D reconstruction method.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] This invention utilizes multi-view imagery to acquire 2D RGB image sequences and pose information of a target scene, and constructs a deformation field that varies over time. Subsequently, the parameters of the 3D Gaussian ellipsoid and deformation field are optimized using a Gaussian splashing algorithm based on spiking neurons, and iteratively updated using photometric loss and pulse thresholding loss. Ultimately, high-quality RGB video data can be reconstructed under specific poses and times. Compared with existing methods, this invention significantly reduces Gaussian storage redundancy, suppresses reconstruction artifacts, improves rendering and training efficiency, and achieves efficient, low-energy dynamic 3D reconstruction. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of a Gaussian splash dynamic three-dimensional reconstruction method based on spiking neurons according to an embodiment of the present invention.

[0045] Figure 2 This is a comparison chart of experimental results using the method of the present invention and existing methods.

[0046] Figure 3 This is a comparison chart of experimental results using the method of the present invention and existing methods. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0049] like Figure 1 As shown, a Gaussian splash dynamic 3D reconstruction method based on spiking neurons includes the following steps:

[0050] S1, record video of the reconstructed target from multiple perspectives to obtain a two-dimensional RGB image sequence and temporal data. .

[0051] S2, based on a two-dimensional RGB image sequence, uses the COLMAP algorithm to calculate sparse point cloud data and camera pose, and initializes a set of three-dimensional Gaussian ellipsoids based on the sparse point cloud data.

[0052] The sparse point cloud data and camera pose are calculated using the COLMAP algorithm. The specific process is as follows:

[0053] (2-1) Perform SIFT feature extraction based on the image sequence, and perform feature matching based on the feature extraction results;

[0054] (2-2) Filter out incorrect matches using geometric consistency to obtain the verified feature point set;

[0055] (2-3) The least squares method is used to globally optimize the camera pose and the 3D sparse point cloud based on the matched feature point set.

[0056] The parameters of each three-dimensional Gaussian ellipsoid include opacity. Center point location Scale vector and quaternions Parametrically represented rotation matrix Define each Gaussian distribution as:

[0057] ;

[0058] in, In three-dimensional coordinates, It is the covariance matrix, expressed as .

[0059] S3, Construct a Gaussian deformation field network, inputting time-dimensional data. and the position information of the three-dimensional Gaussian ellipsoid And obtained through position encoding and It is used to train the deformation field of the Gaussian ellipsoid as it changes over time.

[0060] The Gaussian deformation field network is an MLP neural network consisting of 8 fully connected layers and 3 linear layers. The first 8 fully connected layers are activated by ReLU layers and output 256-dimensional feature vectors. The output of the fourth layer is concatenated with the input of the first layer as the input of the fifth layer. The output of the eighth layer is input to the three different linear layers. The final output is the time-varying deformation field of a three-dimensional Gaussian ellipsoid. The deformation field includes changes in position, rotation, and scaling.

[0061] S4, based on camera pose and 2D RGB image sequence, uses a Gaussian splashing algorithm based on spiking neurons to optimize the parameters of the 3D Gaussian ellipsoid and deformation field. After iterative optimization, the optimized 3D Gaussian ellipsoid and deformation field are output.

[0062] The specific process of step S4 is as follows:

[0063] (4-1) Construct a global FIF spiking neuron and initialize the spiking threshold. for .

[0064] (4-2) The opacity information of the Gaussian ellipsoid is passed through a global FIF spiking neuron, and values ​​less than the spiking threshold are processed. Gaussian ellipsoidal opacity If set to 0, this part of the Gaussian ellipsoid will not be displayed in subsequent calculations.

[0065] All Gaussian ellipsoids share a single FIF spiking neuron. Applying the global FIF spiking neuron to the opacity parameter of all Gaussian ellipsoids yields pulsed opacity values. To reduce low-opacity Gaussian, the calculation process is as follows:

[0066] ;

[0067] Among them, pulse threshold It is shared by all Gaussian ellipsoids and dynamically updated by an alternative gradient method during algorithm execution.

[0068] (4-3) Input the position and time information of the three-dimensional Gaussian ellipsoid into the Gaussian deformation field network, and the deformation field network outputs the deformation field of the three-dimensional Gaussian ellipsoid as it changes over time.

[0069] (4-4) Add the deformation field of the output three-dimensional Gaussian ellipsoid to the original parameters corresponding to the three-dimensional Gaussian ellipsoid to obtain the deformed three-dimensional Gaussian ellipsoid.

[0070] (4-5) Based on the camera pose in the current iteration, perform a projection operation on the deformed 3D Gaussian ellipsoid to obtain the 2D representation of the 3D Gaussian ellipsoid in the image plane.

[0071] Transform the Gaussian ellipsoid through the world-camera transformation matrix Transform to camera coordinates and use an affine approximation matrix. The projection onto the image plane, i.e., the three-dimensional Gaussian ellipsoid splashing process, is calculated as follows:

[0072] ;

[0073] The third row and third column of the third-order covariance matrix were skipped during the calculation.

[0074] (4-6) The 2D representation is rasterized into a differentiable form. Alpha blending is then performed according to the spatial depth order of the three-dimensional Gaussian ellipsoid corresponding to each 2D representation, ultimately yielding the rendered image of the dynamic scene at the current time and viewpoint. The formula for alpha blending is as follows:

[0075] ;

[0076] ;

[0077] ;

[0078] in, It is a pixel. The value, This indicates the number of Gaussian ellipsoids involved in the rendering. It is the RGB color of the Gaussian ellipsoid. It is the Gaussian distribution after the splash. It is the first Opacity after Gaussian weighting Indicates the rendered number of times. Opacity after Gaussian weighting Indicates the first Gaussian weighted pre-opacity; by traversing pixels The entire rendered RGB image is obtained. The whole process is run in a custom CUDA program similar to 3DGS rasterization calculations to ensure fast rendering.

[0079] (4-7) Perform loss calculations on the obtained rendered image and the original RGB image (real image), and perform gradient backpropagation to optimize the parameters of the 3D Gaussian ellipsoid and deformation field; the expression for loss calculation is as follows:

[0080] ;

[0081] in, For luminance loss, For D-SSIM loss, For pulse threshold loss, and For hyperparameters. Pulse threshold loss. The expression is:

[0082]

[0083] in, The opacity value of the Gaussian ellipsoid after processing by a global FIF spiking neuron.

[0084] (4-8) Density control of the Gaussian ellipsoid is performed based on the gradient calculation results.

[0085] The details of the Gaussian splashing algorithm based on spiking neurons in step S4 are as follows: the spiking threshold learning rate of the spiking neurons is set to 0.0003; only the Gaussian ellipsoid parameters are optimized in the initial 3000 iterations of training, and a deformation field is added for co-optimization in subsequent iterations; the Adam optimizer is used in the training process, and its β value gradually increases from 0.9 to 0.999 during the training process; the code framework of this algorithm is implemented using PyTorch; every 100 iterations, the Gaussian ellipsoids that are not activated by the spiking neurons are deleted.

[0086] S5, based on the optimized 3D Gaussian ellipsoid and deformation field rendering of specific pose states and RGB images at specific times, achieves dynamic 3D reconstruction.

[0087] Based on the same inventive principle, this invention also provides a Gaussian splash dynamic three-dimensional reconstruction device based on spiking neurons, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the Gaussian splash dynamic three-dimensional reconstruction method mentioned in the above embodiment.

[0088] To verify the effectiveness of this invention, detailed experiments were conducted on the mainstream dynamic scene reconstruction dataset NeRF-DS, including seven dynamic scenes. Tables 1-4 below compare the numerical results of this method and existing methods across four evaluation dimensions: PSNR (decibels), Gaussian count, FPS (frames per second), and time (minutes). PSNR evaluates the rendering quality of the method, while Gaussian count, FPS, and time evaluate the resource consumption and efficiency. The number of Gaussian counts is inversely proportional to the storage efficiency of the method. As can be seen from the tables, this invention can significantly improve the storage efficiency, inference speed, and time efficiency of reconstruction.

[0089] Table 1

[0090]

[0091] Table 2

[0092]

[0093] Table 3

[0094]

[0095] Table 4

[0096]

[0097] To address the artifact problem in complex dynamic scenes, this invention further tests the method on the HyperNeRF dataset. The HyperNeRF dataset contains complex real-world lighting variations and inaccurate camera poses, making the reconstruction results more prone to artifacts. Figure 2 and Figure 3 As shown, it can be seen that the present invention significantly reduces artifacts compared to the original method, such as Figure 2 The bending phenomenon of the spray gun has been reduced. Figure 3 The surface of the middle finger is smoother.

[0098] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic three-dimensional reconstruction method based on Gaussian splashing of spiking neurons, characterized in that, include: (1) Record video of the reconstructed target from multiple perspectives to obtain two-dimensional RGB image sequences and time dimension data; (2) Based on the two-dimensional RGB image sequence, the sparse point cloud data and camera pose are calculated using the COLMAP algorithm, and a set of three-dimensional Gaussian ellipsoids are initialized based on the sparse point cloud data. (3) Construct a Gaussian deformation field network, input time dimension data and the position information of the three-dimensional Gaussian ellipsoid, and use it to train the deformation field of the Gaussian ellipsoid as it changes over time; (4) Based on the camera pose and the two-dimensional RGB image sequence, the parameters of the three-dimensional Gaussian ellipsoid and deformation field are optimized using the Gaussian splashing algorithm based on spiking neurons. After iterative optimization, the optimized three-dimensional Gaussian ellipsoid and deformation field are output. The specific process is as follows: (4-1) Construct a global FIF spiking neuron and initialize the spiking threshold. for ; (4-2) The opacity information of the Gaussian ellipsoid is passed through a global FIF spiking neuron, and values ​​less than the spiking threshold are processed. Gaussian ellipsoidal opacity Set it to 0, and this part of the Gaussian ellipsoid will not be displayed in subsequent calculations; (4-3) Input the position and time information of the three-dimensional Gaussian ellipsoid into the Gaussian deformation field network, and the deformation field network outputs the deformation field of the three-dimensional Gaussian ellipsoid as it changes over time; (4-4) Add the deformation field of the output three-dimensional Gaussian ellipsoid over time to the original parameters corresponding to the three-dimensional Gaussian ellipsoid to obtain the deformed three-dimensional Gaussian ellipsoid. (4-5) Based on the camera pose in the current iteration, perform a projection operation on the deformed 3D Gaussian ellipsoid to obtain the 2D representation of the 3D Gaussian ellipsoid in the image plane; (4-6) The 2D representation is rasterized into a differentiable form. Alpha blending is performed according to the spatial depth order of the three-dimensional Gaussian ellipsoid corresponding to each 2D representation. Finally, the rendered image of the dynamic scene at the current time and viewpoint is obtained. (4-7) Perform loss calculation on the obtained rendered image and the original RGB image, and perform gradient backpropagation to optimize the parameters of the three-dimensional Gaussian ellipsoid and deformation field; (4-8) Density control of the Gaussian ellipsoid is performed based on the gradient calculation results; (5) Dynamic three-dimensional reconstruction is achieved by rendering specific pose states and RGB images at specific times based on the optimized three-dimensional Gaussian ellipsoid and deformation field.

2. The Gaussian splash dynamic three-dimensional reconstruction method based on spiking neurons according to claim 1, characterized in that, In step (2), based on the two-dimensional RGB image sequence, the sparse point cloud data and camera pose are calculated using the COLMAP algorithm. The specific process is as follows: (2-1) Perform SIFT feature extraction based on the image sequence, and perform feature matching based on the feature extraction results; (2-2) Filter out incorrect matches using geometric consistency to obtain the verified feature point set; (2-3) The least squares method is used to globally optimize the camera pose and the 3D sparse point cloud based on the matched feature point set.

3. The Gaussian splash dynamic three-dimensional reconstruction method based on spiking neurons according to claim 1, characterized in that, In step (2), the parameters of each three-dimensional Gaussian ellipsoid include opacity. Center point location Scale vector and quaternions Parametrically represented rotation matrix ; Represents the space of real numbers.

4. The Gaussian splash dynamic three-dimensional reconstruction method based on spiking neurons according to claim 1, characterized in that, In step (3), the Gaussian deformation field network is an MLP neural network, which contains 8 fully connected neural network layers and 3 linear layers; The first eight fully connected neural network layers are activated by ReLU layers and output a 256-dimensional feature vector; the output of the fourth layer is concatenated with the input of the first layer as the input of the fifth layer; the output of the eighth layer is input to three different linear layers respectively; finally, the output is the deformation field of a three-dimensional Gaussian ellipsoid that varies with time.

5. The Gaussian splash dynamic three-dimensional reconstruction method based on spiking neurons according to claim 1 or 4, characterized in that, In step (3), the deformation field includes position change, rotation change and scaling change.

6. The Gaussian splash dynamic three-dimensional reconstruction method based on spiking neurons according to claim 1, characterized in that, In steps (4-6), the formula for calculating alpha mixing is as follows: ； ； ； in, It is a pixel. The value, This indicates the number of Gaussian ellipsoids involved in the rendering. It is the RGB color of the Gaussian ellipsoid. It is the Gaussian distribution after the splash. It is the first Opacity after Gaussian weighting Indicates the rendered number of times. Opacity after Gaussian weighting Indicates the first Gaussian weighted pre-opacity; by traversing pixels You will get the entire rendered RGB image.

7. The Gaussian splash dynamic three-dimensional reconstruction method based on spiking neurons according to claim 1, characterized in that, In steps (4-7), the expression for calculating the loss is as follows: ； in, For luminance loss, For D-SSIM loss, For pulse threshold loss, and This is a hyperparameter.

8. The Gaussian splash dynamic three-dimensional reconstruction method based on spiking neurons according to claim 7, characterized in that, Pulse threshold loss The expression is: ； in, The opacity value of the Gaussian ellipsoid after processing by a global FIF spiking neuron.

9. A Gaussian splash dynamic three-dimensional reconstruction device based on spiking neurons, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the Gaussian splash dynamic three-dimensional reconstruction method according to any one of claims 1-8.

Images