An implicit three-dimensional reconstruction method and system based on a hybrid ANN-SNN architecture

Abstract

The application discloses a kind of implicit three-dimensional reconstruction method and system based on hybrid ANN-SNN architecture, in the three-dimensional reconstruction process using RGB image, using the following steps: (1) using the network of hybrid ANN-SNN architecture to build neural radiation field, using the way of voxel rendering using density field and color field;(2) construct bounded full-precision IF neuron model as the neuron model of SNN, the upper limit of output is constrained, to reduce three-dimensional reconstruction loss;(3) in the training process, on the basis of luminosity loss function, using additional loss function, start training from low-frequency surface, while increasing pulse threshold, to reduce three-dimensional reconstruction error;(4) finally, using the neural radiation field obtained from trained new view image and three-dimensional geometry.The application is used, in the case where guaranteeing general scene three-dimensional reconstruction precision, also improve the reconstruction precision of special scene.

Description

Technical Field

[0001] This invention belongs to the field of neuromorphic computing, and in particular relates to an implicit 3D reconstruction method and system based on a hybrid ANN-SNN architecture. Background Technology

[0002] 3D reconstruction from RGB images is a complex and challenging task in computer vision. Neural radiation fields (NeRF) are a promising recent solution for implicit novel view synthesis, achieving very competitive results in 3D reconstruction.

[0003] For example, Chinese patent document CN116051740A discloses a method and system for three-dimensional reconstruction of outdoor unbounded scenes based on neural radiation fields. The method reconstructs the implicit neural radiation field model of the scene based on neural radiation field technology, and obtains the foreground object model, background environment model and full scene model respectively. Finally, the implicit three-dimensional model is visualized by volume rendering technology.

[0004] Chinese patent document CN116416375A discloses a three-dimensional reconstruction method and system based on deep learning, including: acquiring scene image data; building an improved neural radiation field three-dimensional reconstruction model; training the improved neural radiation field three-dimensional reconstruction model to obtain reconstruction results; and finally evaluating the reconstruction results.

[0005] One reason for NeRF's success is its ability to implicitly represent geometric information using neural networks based on traditional continuous multilayer perceptrons (MLPs). However, real-world geometric information is discontinuous, which is inconsistent with the computational representation in traditional MLPs. This inconsistency presents three problems. First, existing methods require post-processing (i.e., filtering with a threshold to extract discontinuous geometric representations from the learned continuous field). However, determining the optimal threshold requires experience and tedious tuning, and many methods rely on manually selecting the threshold for different scenes. Second, NeRF-based methods optimize the network per view, resulting in different optimal thresholds for surfaces at different viewpoints. However, existing methods use the same threshold to filter output values, failing to obtain accurate geometric information due to the different optimal filtering thresholds across different views. Finally, this inconsistency can lead to larger errors in specific scenarios. Because continuous functions like traditional MLPs struggle to produce completely different densities (i.e., significantly different outputs) for points that are close to each other (i.e., similar inputs), objects may disappear. Summary of the Invention

[0006] This invention provides an implicit 3D reconstruction method and system based on a hybrid ANN-SNN architecture, which improves the reconstruction accuracy of special scenes while ensuring the accuracy of 3D reconstruction in general scenes.

[0007] An implicit 3D reconstruction method based on a hybrid ANN-SNN architecture employs the following steps in the 3D reconstruction process using RGB images:

[0008] (1) A neural radiation field is constructed using a hybrid ANN-SNN architecture, and density field and color field are used by voxel rendering.

[0009] (2) Construct a bounded full-precision IF neuron model as the neuron model of SNN, and constrain the upper bound of the output to reduce the loss of 3D reconstruction.

[0010] (3) During the training process, an additional loss function is used on the basis of the photometric loss function, and training starts from the low-frequency surface. At the same time, the pulse threshold is increased to reduce the three-dimensional reconstruction error.

[0011] (4) Finally, the trained neural radiation field is used to obtain new perspective images and three-dimensional geometry.

[0012] Furthermore, in step (1), the neural radiation field is constructed using a hybrid ANN-SNN architecture as follows:

[0013] Spatial coordinates are encoded by constructing an ANN, and the output is processed by an SNN to perform discontinuity processing; the discontinuity processing includes: discontinuing the density and normalizing the color from 0 to 1.

[0014] In step (1), the pixel values ​​are obtained using voxel rendering.

[0015]

[0016]

[0017] Where σ(t) represents the voxel density at sampling point t, c(t,d) represents the color along the d direction at sampling point t, and w(t) represents the color based on... The weight of each sampling point is defined, and T represents the sampling endpoint.

[0018] In step (2), the specific steps for constructing the bounded full-precision IF neuron model are as follows:

[0019] The membrane potential equation for an IF neuron is defined as follows:

[0020] u (t+1) =u t +I (t+1) -V th ·Θ(u (t+1) -V th )

[0021] I (t+1) =Wx (t+1)

[0022] Where W represents the connection weights, t represents the t-th time step, and Θ is a step function defined as:

[0023]

[0024] The spiking neurons used in the final layer of the bounded full-precision IF neuron model are:

[0025]

[0026] Here, k and r represent the learnable spiking neuron parameters. Increasing k allows for a larger threshold, and increasing r makes the function closer to a full-precision IF neuron.

[0027] In step (2), the constraint on the upper bound of the output is specifically as follows:

[0028] Where o is the output of the density field of the last layer.

[0029] In step (3), the additional loss function includes:

[0030] The threshold loss function reduces reconstruction loss and improves reconstruction accuracy by gradually increasing the threshold;

[0031] The smooth loss function allows the network to be trained starting from a smooth surface with low frequencies, ensuring the accuracy of the final high-frequency information and improving reconstruction accuracy.

[0032] Threshold loss is defined as:

[0033]

[0034] In the formula, V th This represents the impulse threshold of a neuron.

[0035] Smoothing loss is defined as:

[0036]

[0037] In the formula, p refers to each defined pixel, i represents each sampling point, and w i This represents the weight of each sampling point. Density σ i The gradient with respect to the input coordinates.

[0038] An implicit 3D reconstruction system based on a hybrid ANN-SNN architecture includes a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they implement the aforementioned implicit 3D reconstruction method.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. This invention employs a hybrid ANN-SNN architecture to represent discontinuous geometric information in the real world. Based on this discontinuous representation, it demonstrates that the reconstruction loss can be continuously reduced by increasing the threshold, thereby improving reconstruction accuracy.

[0041] 2. This invention uses a bounded full-precision IF neuron model to constrain the upper bound of the output in order to reduce the loss in 3D reconstruction.

[0042] 3. This invention uses a threshold loss function to gradually increase the pulse threshold, reducing reconstruction loss, and uses a smooth loss function to train the network from a low-frequency smooth surface, ensuring the correctness of the final high-frequency information, thereby further improving reconstruction accuracy. Attached Figure Description

[0043] Figure 1 This is a framework diagram of an implicit 3D reconstruction method based on a hybrid ANN-SNN architecture according to the present invention.

[0044] Figure 2 This relates to the upper bound of error and the average depth error during the training process of this invention.

[0045] Figure 3 This is a diagram illustrating the effects of the present invention. Detailed Implementation

[0046] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.

[0047] like Figure 1 As shown on the left, an implicit 3D reconstruction method based on a hybrid ANN-SNN architecture is presented. The input consists of position coordinates and orientation vectors, which are passed through a position encoding layer and then fed into a multilayer perceptron (MLP) of the ANN. The ANN encodes the spatial coordinates and orientation vectors. Finally, the input passes through a bounded impulse multilayer perceptron (BLP) of the SNN, which performs discontinuity processing on the output to obtain the final color and density. This method improves the reconstruction accuracy for special scenes while maintaining the accuracy of 3D reconstruction in general scenes. The specific steps include:

[0048] S01, by constructing a network representation of the density field and color field based on a hybrid ANN-SNN architecture.

[0049] In the field of voxel-based 3D reconstruction, the network needs to represent the color field and density field. In this embodiment of the invention, the pixel values ​​obtained after voxel rendering are:

[0050]

[0051]

[0052] Where σ(t) represents the voxel density at sampling point t, and c(t,d) represents the color along the d direction at sampling point t.

[0053] This model models the density field σ(t) and color field c(t,d) using a neural network. The photometric loss used is:

[0054]

[0055] in, This represents the actual pixel value corresponding to the image.

[0056] The membrane potential equation for an IF neuron is defined as follows:

[0057] u (t+1) =u t +I (t+1) -V th ·Θ(u (t+1) -V th )

[0058] I (t+1) =Wx (t+1) .

[0059] Where W represents the connection weights, t represents the t-th time step, and Θ is a step function defined as:

[0060]

[0061] The spiking neurons used in the final layer of the bounded full-precision IF neuron model are:

[0062]

[0063] Where k and r represent the parameters of the learnable spiking neuron. For example... Figure 1 The right side illustrates the difference between a general spiking neuron and a bounded full-precision spiking neuron. A general spiking neuron, for any input, either produces an unbounded output activation value or only a finite number of values. In contrast, the bounded full-precision spiking neuron proposed in this invention guarantees full precision while ensuring bounded output activation values, thus reducing reconstruction loss. Furthermore, increasing k allows for a larger threshold, and increasing r makes the function closer to a full-precision IF neuron.

[0064] In this invention, d is defined v It is the estimated depth, d v This refers to the accurate depth. T represents the length of the sampling range, and Δt represents the sampling interval. Vth It is the pulse threshold, V max This represents the maximum density value. For a well-trained NeRF, V... th It is also the non-zero minimum of its density field, and the density value of the first non-zero density point encountered by the ray is V. th When σ used for voxel rendering is a discontinuous function, we have:

[0065]

[0066] This is because m is defined as the index of the first non-zero σ (i.e., the extracted depth d). v =t m ), where m′ is the index of the largest σ within m+1 and N. Δt i It is the sampling interval between sampling point i-1 and point i. i This refers to the weight of each sampling point defined in the literature "Mildenhall, Ben, et al. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65.1(2021):99-106.". Then there is:

[0067]

[0068]

[0069] Furthermore, due to:

[0070]

[0071] Since this invention does not consider the sampling interval, the conclusion is valid.

[0072] Based on this relationship, when V th When large enough, dd v It is small enough, thus resulting in a sufficiently small error. Meanwhile, for a fixed V... th Smaller V max This can reduce errors.

[0073] S02 employs threshold loss to gradually increase the threshold, thereby reducing reconstruction loss. For example... Figure 2 As shown, based on discontinuous representation, the reconstruction loss can be continuously reduced by increasing the threshold, thereby improving the reconstruction accuracy.

[0074] Threshold loss is defined as:

[0075]

[0076] In the initial stages of network training, the density field exhibits significant bias. A large impulse threshold will prevent the network from training because most membrane potentials cannot reach the impulse threshold and are outside the range of non-zero alternative gradients. Therefore, this invention sets the initial impulse threshold to a small value. Furthermore, this invention initially sets the coefficients of the threshold loss function to a small value and continues to increase them as the training process progresses.

[0077] S03 employs a smooth loss function to train the network starting from a low-frequency smooth surface, ensuring the accuracy of the final high-frequency information and thus further improving reconstruction accuracy.

[0078] Smoothing loss is defined as:

[0079]

[0080] Here, p refers to each pixel as defined in "Guo, Yuan-Chen, et al. Nerfren: Neural radiance fields with reflections. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022." i represents each sampling point, and w... i This represents the weight of each sampling point.

[0081] like Figure 3 As shown, the number in the lower left corner of each image represents the chamfer distance. It can be seen that by using this invention, the reconstruction accuracy of 3D geometry is improved, and while maintaining the 3D reconstruction accuracy for general scenes, the reconstruction accuracy for special scenes is also improved.

[0082] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An implicit 3D reconstruction method based on a hybrid ANN-SNN architecture, characterized in that, The following steps are used in the 3D reconstruction process using RGB images: (1) A neural radiation field is constructed using a hybrid ANN-SNN architecture, and density field and color field are used by voxel rendering. (2) Construct a bounded full-precision IF neuron model as the neuron model of the SNN, and constrain the upper bound of the output to reduce the 3D reconstruction loss; the specific steps for constructing the bounded full-precision IF neuron model are as follows: The membrane potential equation for an IF neuron is defined as follows: u (t+1) =u t +I (t+1) -V th ·Θ(u (t+1) -V th ) I (t+1) = Wx (t+1) where W denotes the weights of the connections, t denotes the t-th time step, V th denotes the pulse threshold of the neuron, Θ is a step function defined as: The spiking neurons used in the final layer of the bounded full-precision IF neuron model are: Where k and r represent the parameters of the learnable spiking neuron, increasing k allows for a larger threshold, and increasing r makes the function closer to a full-precision IF neuron; The specific steps for constraining the upper bound of the output are as follows: Where o is the output of the density field of the last layer; (3) During training, based on the photometric loss function, an additional loss function is used to train from low-frequency surfaces, while increasing the pulse threshold to reduce 3D reconstruction error; the additional loss function includes: The threshold loss function reduces reconstruction loss and improves reconstruction accuracy by gradually increasing the threshold; the threshold loss is defined as: The smooth loss function allows the network to start training from a low-frequency smooth surface, ensuring the accuracy of the final high-frequency information and improving reconstruction accuracy. (4) Finally, the trained neural radiation field is used to obtain new perspective images and three-dimensional geometry.

2. The method of claim 1, wherein the hybrid ANN-SNN architecture-based implicit three-dimensional reconstruction method is characterized by, In step (1), the neural radiation field is constructed using a hybrid ANN-SNN architecture as follows: Spatial coordinates are encoded by constructing an ANN, and the output is processed by an SNN to perform discontinuity processing; the discontinuity processing includes: discontinuing the density and normalizing the color from 0 to 1.

3. The method of claim 1, wherein the hybrid ANN-SNN architecture-based implicit three-dimensional reconstruction method is characterized by, In step (1), the pixel values ​​are obtained using voxel rendering. C = ∫0 T w(t) · c(t, d) dt w(t) = σ(t) - exp(-∫0 t σ(t)dt) Where σ(t) represents the voxel density at sampling point t, c(t,d) represents the color along the d direction at sampling point t, w(t) represents the weight defined for each sampling point, and T represents the sampling endpoint.

4. The method of claim 1, wherein the hybrid ANN-SNN architecture-based implicit three-dimensional reconstruction method is characterized by, Smoothing loss is defined as: where p refers to each pixel defined, i represents each sample point, w i represents the weight of each sample point, represents the density σ i the gradient of the input coordinates.

5. An implicit 3D reconstruction system based on a hybrid ANN-SNN architecture, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the implicit three-dimensional reconstruction method according to any one of claims 1-4.

Images