Universal shell-shaped compression method based on voxel radiation field
By introducing shell-like compression constraints and adaptive pruning technology into the voxel radiation field, the problems of long training time of neural radiation field and high storage cost of voxel radiation field are solved, and fast rendering and efficient compression are achieved, suitable for a variety of purposes voxel method.
Patent Information
- Application Number
- CN202411049269.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-12
AI Technical Summary
The problems of low neural radiation field time efficiency and low voxel radiation field memory efficiency, the existing compression methods are complex and inconvenient for general applications.
A general shell-like compression method based on voxel radiation field is proposed. Through surface distillation constraints, the surface voxel-dominated rendering of objects is encouraged, redundant voxels are pruned, and cross-scene adaptive thresholds and dynamic pruning strategies are adopted to optimize voxels. Mesh structure to reduce storage requirements.
Fast rendering and efficient compression are achieved, reducing model size, improving compression efficiency and reconstruction accuracy, and eliminating the need for additional tags and complex training processes, suitable for all voxel methods.
Smart Images

Figure CN118941687A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of three-dimensional reconstruction, and in particular to a universal shell compression method based on voxel radiation field. Background Art
[0002] With the rapid development of computer vision and graphics, 3D reconstruction and rendering technology have become indispensable tools in various fields. 3D reconstruction generates realistic 3D models by scanning and modeling objects, scenes or human bodies, which can be used for environmental simulation in technologies such as virtual reality, augmented reality, game development, multimedia education and medical treatment.
[0003] Current 3D reconstruction technology can be mainly divided into two categories: based on traditional multi-view geometry and based on deep learning.
[0004] Methods based on traditional multi-view geometry include structured light, TOF laser time of flight, triangulation, monocular / multi-camera vision, etc., but these traditional methods have problems such as high reconstruction complexity and high hardware cost.
[0005] In contrast, deep learning-based 3D reconstruction algorithms provide new ideas for optimizing the performance of traditional methods. Most deep learning-based 3D reconstruction algorithms rely only on images and the camera parameters corresponding to the images (camera parameters indicate the position of the corresponding 2D image in 3D space), without the need for cumbersome data acquisition and feature matching processes in traditional methods. In terms of hardware, high-precision sensors or laser cameras are not required. Only a series of 2D images carrying camera parameters are needed to achieve 3D object reconstruction. This 3D reconstruction task that only relies on images and camera parameters can be called a new perspective synthesis task. A new perspective task is to predict or generate new perspective images of the remaining perspectives of a 3D scene based on 2D images of multiple perspectives.
[0006] Among a series of deep learning methods for solving new view synthesis tasks, Neural Radiance Field (NeRF) is a representative method that has received widespread attention. The key idea of NeRF is to train a multilayer perceptron (MLP) to predict and map the color and density corresponding to the sampling points at different viewpoints and spatial positions, and combine it with the volume rendering process to achieve high-quality 3D scene reconstruction. Specifically, NeRF generates new view images through an implicit five-dimensional neural radiance field construction process, which is specifically expressed as: F θ (x, d) = (c, σ), where x = (x, y, z) represents the three-dimensional position; The output c represents the RGB value; σ represents the volume density, which can be understood as the probability that a ray terminates at a given particle. The neural radiation field generates the final target two-dimensional image through volume rendering:
[0007]
[0008] Where c(r(t), d) and σ(r(t)) represent the color and volume density of the sampling point at distance t along the target ray r; T(t) represents the n The cumulative transmittance along the ray from t to t, that is, the ray from t n The probability of not encountering any other particles by t. The design of this item shows that the farther the sampling point is from the pixel to be rendered, the smaller the contribution of the sampling point to the color information of the pixel.
[0009] The neural radiance field is optimized by minimizing the mean square error between the predicted pixel color and the true pixel color obtained by rendering:
[0010]
[0011] in, is the set of all camera rays of the target view P. NeRF has the advantages of excellent performance, strong spatial continuity, and convenient training. However, several major problems need to be solved to realize its practical application, including how to implement faster or even real-time rendering algorithms, how to compress the model for easy use on mobile devices, etc.
[0012] Although the reconstruction performance of neural radiance field is very good, the training time required by the original NeRF is tens of hours, which is unacceptable for its practical application in real-world scenarios. Secondly, since each sampling point of NeRF needs to visit the entire radiance field network (which is essentially a multi-layer perceptron with a large number of layers) forward during inference, this also causes the problem of low inference time efficiency. In order to improve the accuracy and speed of reconstruction, previous methods have tried to combine various explicit representations with implicit representations, such as point clouds, 3D mesh patches, voxels, etc. Among them, the voxel grid-based method regards the 3D scene as composed of countless small blocks / voxels. Each 3D block contains the geometry and appearance information of the local space of the 3D scene in the block. The intuitiveness of voxel modeling makes the training and rendering speeds of these methods significantly faster than the purely implicit neural radiance field method. However, the voxel modeling-based method often has a large room for improvement in reconstruction performance.
[0013] In order to achieve a balance between reconstruction quality and training speed, the latest method proposes a voxel radiance field (VRF) that combines voxels with NeRF. The voxel radiance field uses two different voxel grids to represent the geometry and appearance information of the three-dimensional scene respectively. For this explicit representation, as long as the location of the sampling point or the three-dimensional coordinate x to be queried is known, trilinear interpolation can be conveniently used to retrieve features for any sampling point. Unlike the pure voxel modeling method, the color c and volume density σ information of the sampling point are not directly obtained through feature retrieval, but the implicit features f that indirectly contain the geometry and appearance information of the sampling point. These implicit features will be further sent to a shallow neural radiance field (i.e., a multi-layer perceptron with a shallow number of layers) to obtain the final predicted color c and σ. However, since the original deep network is replaced by a very shallow network, the voxel radiance field is much faster than the purely implicit neural radiance field in both training and reasoning. The process of obtaining the final two-dimensional image of a new perspective from the voxel radiation field is similar to the original NeRF, that is, the color and volume density of all sampling points on the target ray are integrated through the volume rendering equation in formula (1).
[0014] Compared with neural radiation fields, voxel radiation fields are much faster, and can generally complete scene training and rendering within ten minutes. And because of its combination of explicit and implicit characteristics, it can accelerate training and rendering while ensuring reconstruction accuracy. However, using voxel grids to represent three-dimensional scenes takes up a lot of space, because voxel grids are a dense storage method, and their nodes do not directly store the corresponding spatial positions, but rely on adjacent nodes for position inference. Therefore, storing a scene usually requires hundreds of megabytes of space. In short, although there are many solutions that combine voxel representation to accelerate model training, how to ensure reconstruction accuracy under fast rendering and the high storage cost caused by display representation have yet to be solved.
[0015] Currently, some mainstream voxel compression methods are inspired by neural network compression technology.
[0016] The first category of methods focuses on post-processing the data by applying techniques such as weight quantization or low-rank decomposition. For example, weight quantization is based on weight clustering, which discretizes the continuously distributed weights to reduce the number of weights that need to be stored. Low-rank decomposition, on the other hand, uses CP decomposition to decompose the 3D voxel grid features into the sum of the outer products of vectors, or VM decomposition to decompose the 3D voxel grid features into the sum of the outer products of vectors and matrices.
[0017] Another class of methods emphasizes data pruning during voxel-wise training, including removing unnecessary layers or meshes with low volumetric density.
[0018] The third category of methods uses knowledge distillation to distill the performance of a larger teacher model into a smaller and more compact student model. In summary, previous methods either still retain a large number of redundant voxels or require a lot of post-processing or additional training to obtain a compressed version of the trained model. There is still a lot of room for improvement in both compression ratio and compression convenience. Summary of the invention
[0019] In view of the shortcomings of the background technology, the purpose of the present invention - the technical problems to be solved are:
[0020] (1) Low time efficiency of neural radiation fields: Traditional neural radiation fields rely on purely implicit representation methods, which leads to high time costs for training models. In order to improve the accuracy and speed of reconstruction, previous methods have tried to combine various explicit representations with implicit representations. However, it remains to be studied what kind of representation method to choose and how to design the network to model 3D scenes to achieve excellent results in both speed and accuracy.
[0021] (2) Low memory efficiency of voxel radiation field: Using voxel grids to represent three-dimensional scenes takes up a lot of space. Storing a scene usually requires hundreds of megabytes of space. How to design the model to keep the memory usage within an acceptable range to meet practical application requirements such as convenient local storage and online transmission? At the same time, achieve a balance between model accuracy and memory efficiency, and compress the model as much as possible without losing reconstruction accuracy.
[0022] (3) The generality and convenience of voxel compression methods: How to design a compression method so that it can be easily used in all voxel methods to meet the generality of the compression method. And avoid complicated and tedious additional training or post-processing processes, while avoiding the use of additional supervision signals to meet the convenience of the model compression method.
[0023] Technical Solution
[0024] The present invention proposes a general shell compression method based on voxel radiance field (VRF).
[0025] The method mainly includes a new perspective reconstruction method based on voxel radiation field and surface distillation constraints for voxel radiation field, which has two main purposes:
[0026] One is to combine the purely implicit Neural Radiance Field (NeRF) with the explicit representation of voxels to improve the training and rendering speed of the original NeRF.
[0027] The other is to optimize the voxel grid in the voxel radiation field into a shell-like structure to reduce storage cost while maintaining rendering accuracy.
[0028] Specifically, the present invention first proposes a shell-shaped compression constraint, and the proposed constraint is carried out from two aspects:
[0029] 1) Encourage voxels around the object surface to play a dominant role in determining the rendering results;
[0030] 2) Accelerate the pruning of other redundant voxels inside and outside the object.
[0031] In addition, the present invention introduces cross-scene adaptive thresholds to ensure that there are appropriate pruning standards for different 3D scenes. In order to prevent the erroneous removal of key parts of 3D objects during the compression process, the present invention further adopts a dynamic pruning strategy to perform smooth and accurate model pruning during the training process. The self-supervised constraints proposed by the present invention do not require additional labels and can be seamlessly integrated into any voxel grid-based method to achieve model compression.
[0032] Technical solution of the present invention:
[0033] A general shell compression method based on voxel radiation field, comprising the steps of:
[0034] Step 1: Modeling voxel radiation field and model link
[0035] The scene is modeled using volume density meshes, appearance meshes, and shallow neural radiance fields, and detailed model links of the modeled network are trained and optimized.
[0036] 1.1 Voxel radiation field modeling
[0037] First, use an explicit volume density grid V σ ∈R H×W×D And the appearance grid Two different voxel grids represent the geometry and appearance information of the 3D scene, where H, W, and D represent the number of voxels in the length, width, and height of the 3D scene, respectively. c The two networks are composed of many small voxel grids in a way similar to building blocks, and each different voxel grid contains scene information at different three-dimensional spatial locations. c What is stored is the appearance feature f of the object c , and to obtain the RGB value corresponding to the voxel grid at that location, it is necessary to transform the appearance feature f c Sending a superficial neural radiation field F θ Making further predictions:
[0038] c=F θ(f c ,d) (4) Here c represents the RGB value of the object's color, and d represents the target viewing direction at this time.
[0039] 1.2 Target ray sampling
[0040] The training subject of the radiation field is set as a discrete target ray. Using the parameters of the target camera, a series of target rays r in the three-dimensional space are randomly sampled: r = o + td, where o represents the origin of the target camera, d represents the direction of the target ray in the three-dimensional space, and t represents the distance from the camera origin.
[0041] 1.3 Feature Extraction
[0042] Based on the sampled target ray and the modeled voxel grid, trilinear interpolation is used to retrieve features for all voxels x that the target ray r passes through: f = interp(x, V), where interp(·) represents a trilinear interpolation operation, that is, finding the 8 voxel grid nodes closest to x, and performing linear interpolation based on the distance between each grid node and x as a weight. f represents the feature corresponding to the spatial position x, and V represents the voxel. Color feature f c By voxel x for V c In the model link, the shallow neural radiation field F is sent to θ The color feature f c , and the predicted volume density value is obtained directly by trilinear interpolation of the volume density grid: σ = interp(x,V σ ).
[0043] 1.4 Rendering based on voxel radiation field
[0044] For a target ray r, extract the appearance features of all voxels it passes through Then it can be sent into the superficial neural radiation field F θ Predict the final color All volume density values It is obtained directly by trilinear interpolation of the volume density grid. The two-dimensional pixel RGB value of the target viewing angle can be obtained through the integration process of formula (1).
[0045] Previous work approximated the integration process in formula (1) as a discrete summation process and added the background color to the final rendering result. The updated rendering process is as follows:
[0046]
[0047]
[0048] T i represents the cumulative transmittance along the ray from the camera origin to the i-th voxel, α i represents the probability of ray termination at the i-th voxel, c bg It represents the predefined background color, δ i It represents the distance between adjacent voxels.
[0049] The mean square error loss between the rendered color and the true color (Formula (3)) is used as the reconstruction loss for training.
[0050] Step 2: Surface voxel positioning
[0051] 2.1 Using the volume density values of all sampling points on the target ray obtained in step 1 Depth rendering is performed using formula (7) to obtain the estimated target ray depth:
[0052]
[0053] where t i Represents the distance from the i-th voxel to the ray origin o.
[0054] 2.2 When locating the surface of an object, the voxels that intersect with the target ray r are screened to find the surface voxels corresponding to r. Specifically, formula (8) is used to find Compare the voxel with the smallest depth difference:
[0055]
[0056] Where V(r) represents all voxels intersected by ray r; D(v) represents the depth distance from voxel v to the ray origin o.
[0057] If ray r does not intersect any object, the predicted Often at the scene boundary min These rays are excluded from the depth-guided surface constraint.
[0058] Step 3: Depth-guided shell compression
[0059] First, a surface distillation constraint is used to compress the voxel grid. In step 2, the estimated voxels around the object surface v are obtained. surf After that, you can query its corresponding appearance feature c surf And the geometric characteristics σ surf The voxel feature is used to render the formula (5), but for a single element, the rendering process is simplified to:
[0060] Csurf (r) = c surf ·T surf α surf , (9)
[0061] Next, let's add a surface distillation loss L d To achieve the effect of encouraging surface voxels to dominate the rendering results:
[0062]
[0063] Here R(P) represents the set of all camera rays for the camera with posture P, N r refers to the total number of all target rays, and C(r) refers to the true value of the pixel for ray r.
[0064] In addition, in order to accelerate the pruning of other redundant voxels inside and outside the object, the non-surface voxel density value is selected to be close to 0, and the volume density value of all voxels on a ray is obtained. And the corresponding opacity value is obtained by formula (6): After that, the volume density probability density value of each voxel in this ray is calculated And regard this value as the probability that this is the surface of the object. To speed up the pruning of redundant voxels, it is necessary to expect that the volume density probability density value corresponding to the surface voxel is close to 1, and the volume density probability density value of the rest should be close to 0. Therefore, the problem is simplified as a binary classification problem, and the estimated surface voxels obtained in step 2 are used to create corresponding pseudo labels for the classification task.
[0065]
[0066] The designed shell-guided loss is defined as the binary cross entropy loss:
[0067]
[0068] Here N r represents the total number of all training rays, and N represents the total number of sampling points on a target ray.
[0069] Step 4: Adaptive dynamic pruning
[0070] A method is designed to make the pruning threshold τ standard adaptively change between different scenarios.
[0071] When the volume density value of a voxel is less than a certain threshold τ, the corresponding position of the voxel is considered to be an empty area. At this time, the voxel should be pruned or covered, and no sampling or optimization is performed:
[0072]
[0073] Here, M(v) represents a mask indicating whether voxel v is retained. The present invention designs a method for calculating the importance score, which takes into account the volume density value of the voxel, the number of times the voxel is traversed by the target ray, and the distance of the voxel from the camera origin:
[0074]
[0075] Here R(v) represents the set of all training rays that pass through voxel v, T i represents the cumulative transmittance along the ray from the camera origin to the i-th voxel, α i represents the probability of ray termination at the i-th voxel.
[0076] Next, calculate the importance score ratio of each voxel to the entire scene:
[0077]
[0078] Here R(M(v)) represents all the unpruned voxels, and P v Sort from high to low to get And calculate a cumulative importance score according to the ranking
[0079]
[0080] Determine a fixed important voxel ratio β applicable to various scenarios to represent the most critical voxel ratio that needs to be retained, so that adaptive pruning thresholds for different scenarios can be achieved; the cumulative importance score S v Voxels smaller than this voxel ratio will be pruned away.
[0081] Compared with the prior art, the present invention has the following beneficial effects:
[0082] (1) Excellent compression efficiency: The shell compression constraint proposed in the present invention is based on the characteristics of the voxel grid itself, and maximizes the removal of redundant voxels without any additional labels. The proposed surface distillation constraint encourages the voxel grid to be modeled as a shell with both inside and outside being empty, so as to maximize the reduction of redundancy.
[0083] (2) More accurate pruning: The adaptive dynamic pruning strategy proposed in the present invention can realize adaptive pruning standard decisions between different three-dimensional scenes to achieve a balance between accuracy and compression ratio, and maximize compression efficiency without losing accuracy.
[0084] (3) Convenience and versatility: It can be easily used in any voxel method without any additional labels. Experiments have shown that the present invention reduces the number of original voxel grids by more than 50% while ensuring the quality of reconstruction, and also achieves a good balance between rendering quality and compression efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0085] Figure 1 It is the overall process framework diagram of the present invention.
[0086] Figure 2 is the target ray sampling map.
[0087] Figure 3 The visual comparison results with DVGO on four data sets: The visual comparison results of the present invention and previous methods on the new perspective synthesis task.
[0088] Figure 4 The visualization comparison results on the LLFF dataset are as follows: a visualization comparison result diagram of the model geometry reconstruction between the present invention and the previous method. DETAILED DESCRIPTION
[0089] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0090] right Figure 1 The overall process framework is briefly introduced, and the process and logical relationship are as follows:
[0091] In step 1, in order to achieve fast and high-quality new perspective synthesis, the present invention first proposes the modeling of voxel radiation field, combining the explicit voxel grid with the implicit neural radiation field to build a complete new perspective reconstruction link. This involves the target ray sampling process, feature extraction process, network prediction process and final rendering process in the modeled link. In this process, V represents the geometric information of the 3D scene. σ Voxel grid, V representing appearance information c The voxel grid and the shallow neural radiance field F for further implicitly modeling appearance information θ is initialized, the training ray r is sampled and sent into the voxel grid to query the corresponding voxel grid features and
[0092] In step 2, in order to locate the voxels around the surface of the object, it is necessary to use the voxel grid features obtained in step 1 Perform depth rendering and predict the depth of ray r And locate the surface voxel v according to the predicted depth surfTo construct a depth-guided label
[0093] In step 3, the depth of the surface voxel v is located surf The surface voxel features obtained after feature extraction (c surf ,σ surf ) and the depth-guided label obtained in step 2 They are used to perform the main surface distillation constraint and the depth-guided shell compression constraint respectively to compress the voxel grid to improve memory efficiency.
[0094] In step 4, an adaptive density threshold that can be transformed in different scenes is designed to help the voxel grid to be accurately pruned, and a dynamic pruning process is designed to further compress the voxel grid while ensuring that important parts of the voxel grid are not removed, that is, to maximize the reconstruction accuracy of the retained model. Finally, a model with a geometric shape similar to a thin shell is obtained.
[0095] Technical solution of the present invention:
[0096] A general shell compression method based on voxel radiation field, comprising the steps of:
[0097] Step 1: Modeling voxel radiation field and model link
[0098] Note: After the scene is modeled using the volume density mesh, appearance mesh, and shallow neural radiance field in step 1.1, steps 1.2, 1.3, and 1.4 will train and optimize the detailed model links of the modeled network.
[0099] 1.1 Voxel radiation field modeling
[0100] First, use an explicit volume density grid V σ ∈R H×W×D And the appearance grid Two different voxel grids represent the geometry and appearance information of the 3D scene, where H, W, and D represent the number of voxels in the length, width, and height of the 3D scene, respectively. c It represents the eigenvalue dimension contained in the appearance voxel grid node. The two networks are composed of many small voxel grids in a way similar to building blocks. Each different voxel grid contains scene information at different three-dimensional spatial positions. In addition to the voxel grid, the present invention also combines the implicit neural radiation field to accelerate training while maintaining the accuracy of the model reconstruction result. Specifically, the appearance grid V c What is stored is not directly the RGB value representing the color of the object, but the appearance characteristics of the object. c , and to obtain the RGB value corresponding to the voxel grid at that location, it is necessary to transform the appearance feature f c Sending a superficial neural radiation field Fθ Making further predictions:
[0101] c=F θ (f c ,d) (4)
[0102] Here c represents the RGB value of the object's color, and d represents the target viewing direction at this time. Due to factors such as reflection, the color output corresponding to different target viewing directions is also different.
[0103] 1.2 Target ray sampling, provided to step 1.3, step 2
[0104] First, it is necessary to determine the subject of the training process. Since the model of the present invention is oriented to the task of synthesizing new perspectives, the final goal of the reconstruction is also a two-dimensional image of the new perspective, and the two-dimensional image of the new perspective is composed of a series of new perspective pixels. Therefore, the new perspective synthesis task generally uses the discrete pixel values of the synthesized new perspective image as the reconstruction object, and a two-dimensional pixel of a target perspective corresponds to a target ray passing through the origin of the target camera in three-dimensional space. Therefore, the training subject of the radiation field can be set as discrete target rays, and using the parameters of the target camera, a series of target rays r in the three-dimensional space can be randomly sampled: r=o+td, where o represents the origin of the target camera, d represents the direction of the target ray in the three-dimensional space, and t represents the distance from the camera origin. ( Figure 2 Shown: Target ray sampling)
[0105] 1.3 Feature Extraction
[0106] Based on the sampled target ray and the modeled voxel grid, trilinear interpolation is used to retrieve features for all voxels x that the target ray r passes through: f = interp(x, V), where interp(·) represents a trilinear interpolation operation, that is, finding the 8 voxel grid nodes closest to x, and performing linear interpolation based on the distance between each grid node and x as a weight. f represents the feature corresponding to the spatial position x, and V represents the voxel. Color feature f c By voxel x for V c It should be noted that in the model link designed by the present invention, the shallow neural radiation field F θ The only feature of f c , and the predicted volume density value is obtained directly by trilinear interpolation of the volume density grid: σ = interp(x,V σ ).
[0107] 1.4 Rendering based on voxel radiation field
[0108] For a target ray r, extract the appearance features of all voxels it passes through Then it can be sent into the superficial neural radiation field F θ Predict the final color All volume density values It is obtained directly by trilinear interpolation of the volume density grid. The 2D pixel RGB value of the target viewing angle can be obtained through the integration process of formula (1). However, in practical applications, previous work approximates the integration process in formula (1) as a discrete summation process and adds the background color to the final rendering result. Therefore, the updated rendering process is as follows:
[0109]
[0110] T i represents the cumulative transmittance along the ray from the camera origin to the i-th voxel, α i represents the probability of ray termination at the i-th voxel, c bg It represents the predefined background color, δ i Represents the distance between adjacent voxels. Similar to the original NeRF, the present invention also uses the mean square error loss between the rendered color and the true value color (Formula (3)) as the reconstruction loss for training.
[0111] Step 2: Surface voxel positioning
[0112] 2.1 Using the volume density values of all sampling points on the target ray obtained in step 1 Depth rendering is performed using formula (7) to obtain the estimated target ray depth:
[0113]
[0114] where t i Represents the distance from the i-th voxel to the ray origin o.
[0115] 2.2 When locating the surface of an object, the voxels that intersect with the target ray r are screened to find the surface voxels corresponding to r. Specifically, formula (8) is used to find Compare the voxel with the smallest depth difference:
[0116]
[0117] Where V(r) represents all voxels intersected by ray r; D(v) represents the depth distance from voxel v to the ray origin o.
[0118] It is worth noting that if the ray r does not intersect any object, the predicted Often at the scene boundary minThese rays are excluded from depth-guided surface constraints to ensure that only meaningful surfaces are constrained.
[0119] Step 3: Depth-guided shell compression
[0120] This step is constrained from two aspects:
[0121] 1) Encourage voxels around the object surface to play a dominant role in determining the rendering results;
[0122] 2) Accelerate the pruning of other redundant voxels inside and outside the object.
[0123] First, a surface distillation constraint is used to implement the first aspect. In step 2, the estimated surface surrounding voxels v are obtained. surf After that, you can query its corresponding appearance feature c surf And the geometric characteristics σ surf The voxel feature is used for rendering similar to formula (5), but for a single element, the rendering process can be simplified to:
[0124] C surf (r) = c surf ·T surf α surf , (9)
[0125] Next, add a surface distillation loss L d To achieve the effect of encouraging surface voxels to dominate the rendering results:
[0126]
[0127] Here R(P) represents the set of all camera rays for the camera with posture P, N r Refers to the total number of all target rays, and C(r) refers to the true pixel value for ray r. And a surface distillation loss is added to encourage surface voxels to dominate the rendering results.
[0128] As for aspect 2), in order to accelerate the pruning of other redundant voxels inside and outside the object, it is generally necessary to encourage the density value of non-surface voxels to approach 0, so as to accelerate their pruning and obtain the volume density value of all voxels on a ray. And the corresponding opacity value is obtained by formula (6): After that, the volume density probability density value of each voxel on this ray can be calculated And regard this value as the probability that this is the surface of the object. To accelerate the pruning of redundant voxels, it is necessary to expect that the volume density probability density value corresponding to the surface voxels should be close to 1, and the volume density probability density value of the rest should be close to 0. Therefore, this problem can be simplified as a binary classification problem. The estimated surface voxels obtained in step 2 can be used to create corresponding pseudo labels for the classification task.
[0129]
[0130] Therefore, the shell-shaped guided loss designed by the present invention can be defined as the binary cross entropy loss:
[0131]
[0132] Here N r represents the total number of all training rays, and N represents the total number of sample points on a target ray. Although during training, a training ray usually intersects at least two surfaces of the object, this problem can be naturally solved by the training strategy used. During training, training rays are randomly emitted in multiple 3D directions. Some voxels on the surface may be penalized when the ray is emitted from behind them, but they will be encouraged when the ray is emitted from the front. Eventually, only those voxels that have never received encouragement will gradually decrease in voxel density value, and finally be considered redundant and removed.
[0133] Step 4: Adaptive dynamic pruning
[0134] Whether a voxel grid needs to be pruned during training is generally determined by its corresponding volume density value. When the volume density value is less than a certain threshold τ, it is considered empty and the voxel is pruned or covered, and no longer sampled or optimized:
[0135]
[0136] The setting of this threshold τ is undoubtedly very important. A too strict threshold may cause many important voxels to be cut, while a too loose threshold cannot achieve the compression effect. It is also unreasonable to set a unified threshold for each different scene. Therefore, the present invention designs a method to make the threshold standard of pruning adaptively change between different scenes. Specifically, a calculation method of importance score is designed, which takes into account factors such as the volume density value of the voxel, the number of times the voxel is traversed by the target ray, and the distance of the voxel from the camera origin:
[0137]
[0138] Here R(v) represents the set of all training rays that pass through voxel v, Ti represents the cumulative transmittance along the ray from the camera origin to the i-th voxel, α i represents the probability of ray termination at the i-th voxel. Next, we calculate the importance score ratio of each voxel to the entire scene:
[0139]
[0140] Here R(M(v)) represents all the unpruned voxels, and we will P v Sort from high to low to get And calculate a cumulative importance score according to the ranking
[0141]
[0142] Now we only need to determine a fixed important voxel ratio β that is applicable to various scenarios to represent the most critical voxel ratio that needs to be retained, and we can achieve adaptive pruning thresholds for different scenarios. v Voxels smaller than this voxel ratio will be pruned away.
[0143] In addition, in order to better achieve a balance between reconstruction accuracy and compression efficiency, the present invention introduces smooth dynamic pruning during the training process. Previous methods usually choose to prune the voxel grid after the voxel grid has completed training, which will cause additional time costs. In contrast, the present invention proposes a dynamic pruning method that integrates adaptive thresholds into the training process. Considering that the geometric accuracy of the model is insufficient at the beginning of training, overly strict pruning at this time may affect the final effect of the model. Therefore, the present invention adopts a smoother pruning method during the training process. As described above, by determining a fixed important voxel ratio β, the less important voxels after the i-th can be identified and pruned, where is less than β, Now we introduce a dynamic pruning strategy, first setting a desired final important voxel ratio β n , and set a relatively large initial important voxel ratio β 0 , each pruning cycle gradually reduces the current cumulative important voxel ratio: β i =β 0 -(β 0 -β n )·i / n, where i represents the i-th pruning cycle and n represents the total number of pruning cycles. In addition, to prevent important voxels from being mistakenly pruned at the beginning of training, N is set before switching to dynamic pruning mode. staticIn the iteration, a fixed threshold for α is still used for pruning. After switching to dynamic pruning mode, similar to dynamically reducing the proportion of accumulated important voxels, the threshold for α is gradually increased from τ to τ during training. 0 Increase to τ n .
[0144] experiment
[0145] Dataset
[0146] The experiments are performed on five publicly available datasets for performance evaluation: Synthetic-NeRF, Synthetic-NSVF, BlendedMVS, and Tanks&Temples. The first three datasets are synthetic, while the last one is a real-world dataset.
[0147] Method application
[0148] The present invention adopts a two-stage training process. The purpose of the coarse stage is to roughly explore the scene geometry to guide effective sampling and promote faster training in the subsequent fine stage. In the coarse stage, the resolution of the voxel is set to a low level, and the appearance voxel grid directly stores the RGB values of the three-dimensional object. The color features obtained by trilinear interpolation are directly used for volume rendering without further processing by MLP. Considering the significant geometric errors inherent in this stage, surface distillation constraints or adaptive dynamic pruning are not used at this time, so the process of surface positioning and depth estimation is skipped. This strategy enables SVRF to complete the training of the coarse stage in one minute. Subsequently, the fine stage is the main training stage, which is characterized by a higher resolution setting and a multi-level step-by-step training strategy. Using the geometric results obtained from the coarse stage, the external blank space can be directly bypassed in subsequent training. At this stage, object surface distillation constraints and adaptive dynamic pruning are adopted. In order to alleviate the negative impact on network training caused by the depth estimation error affecting geometric training, object surface distillation constraints are intermittently applied within a certain number of iteration intervals. At the same time, the present invention begins to implement a dynamic pruning strategy in the refinement stage, and gradually tightens the voxel pruning standard as the number of training iterations increases.
[0149] Comparison of implementation effects
[0150] In order to make a fair comparison in new view synthesis, comparison and testing are carried out from two perspectives. First, the present invention shows the comparison results with several classic methods in the field of new view reconstruction, such as the purely implicit methods NeRF and SRN, and shows the comparison results with several grid-based methods, including NV, NSVF, KiloNeRF, FastNeRF, PlenOctrees, Plenoxels and the most relevant baseline DVGO of the present invention. Secondly, in order to verify the effectiveness of the constraints proposed by the present invention in model compression, we further show the comparison results of the method of the present invention with several classic grid-based new view synthesis methods and the latest methods specifically for voxel compression. These methods include CCNeRF, TensoRF, TinyNeRF, Re:NeRF and VQRF. The same test method is used to conduct experiments on the compared methods and the method of the present invention, and the reconstruction quality of different methods is compared. The results are shown in the following table.
[0151] Table 1 Quantitative comparative experimental results of the present invention and other models in the new perspective synthesis task
[0152]
[0153] Table 2 Quantitative comparison experimental results of the present invention and other models in model compression efficiency
[0154]
[0155] Table 1 gives a quantitative comparison on the new perspective synthesis task. Across the five datasets, the method of the present invention consistently outperforms most existing methods. It is worth noting that purely implicit models such as NeRF require a lot of training time, and relying solely on implicit embeddings to predict appearance and geometry is inherently limited. In contrast, the total training and testing time for the model of the present invention is only about 5 minutes per scene, and it shows excellent results. Compared with the original NeRF, integrating explicit representations such as voxels significantly improves interpretability and training speed. However, dense voxel representations incur significant memory overhead, but by integrating the shell compression constraints proposed by the present invention, the model outperforms DVGO while keeping the number of remaining voxels unchanged.
[0156] Table 2 shows the powerful ability of combining the shell compression proposed in this invention with existing methods to achieve memory saving. These results are compared with the results of the original NeRF and its various extensions. From the results, it can be observed that the method proposed in this invention can be flexibly applied to sparse grid representations of 3D scenes, achieving significant model compression while ensuring excellent reconstruction quality and speed, and showing an excellent trade-off between model size and reconstruction quality. "Shell-" in the table means that our proposed shell compression constraint is applied on the corresponding different baselines.
[0157] Figure 3 The visual comparison results of the present invention and the previous method DVGO on four data sets are given.
[0158] Figure 4 This is to verify that the final model geometry modeled by the proposed method is shell-like. In order to intuitively demonstrate the impact of constraints on the model geometry, we show the geometric results obtained by cutting the final reconstructed model in half. Obviously, the cutting result of DVGO can be seen, and its object contains a large number of redundant voxels, while the result obtained by the method of the present invention is cleaner and clearer.
[0159] like Figure 4 As shown, the method of the present invention shows significant advantages in reducing floating artifacts, thanks to the elimination of most of the redundant voxels, including "ghosts" in external blank areas and invisible voxels inside objects. In addition, the method proposed in the present invention also achieves more detailed rendering in challenging areas. This detailed result can be attributed to the fact that the proposed compression method allows the use of larger voxel resolution in the model setting. In addition, due to the reduction of redundant voxels, more attention is paid to optimizing truly meaningful areas during training, including areas with more complex textures, resulting in more detailed results.
Claims
1. A general shell compression method based on voxel radiation fields, It is characterized in that Includes steps: Step 1: Modeling voxel radiation field and model link; Model the scene using volume density meshes, appearance meshes, and shallow neural radiance fields, and train and optimize detailed model links of the modeled network; Step 2: Surface voxel positioning; Use the predicted depth during training to locate the voxels around the object surface to provide guidance for voxel compression in step 3; Step 3: Depth-guided shell compression; Constrain the voxel grid based on the deep pseudo-label constructed in step 2, so that the voxel grid can model the shape of the object as a shell with both inside and outside being hollow; Step 4: Adaptive dynamic pruning; The voxel grid optimized in the previous step is pruned to increase the model memory efficiency, and a method is designed to make the pruning threshold τ standard adaptively change between different scenes.
2. A general shell compression method based on voxel radiation field according to claim 1, It is characterized in that The specific process of step 1 is: 1.1 Voxel radiation field modeling First, use an explicit volume density grid V σ ∈R H×W×D And the appearance grid Two different voxel grids represent the geometry and appearance information of the 3D scene, where H, W, and D represent the number of voxels in the length, width, and height of the 3D scene, respectively. c It represents the eigenvalue dimension contained in the appearance voxel grid node; these two networks are composed of many small voxel grids in a way similar to building blocks, and each different voxel grid contains scene information at different three-dimensional spatial positions; the appearance grid V c What is stored is the appearance feature f of the object c , and to obtain the RGB value corresponding to the voxel grid at that location, it is necessary to transform the appearance feature f c Sending a superficial neural radiation field F θ Making further predictions: c=F θ (f c ,d) (4) Here c represents the RGB value of the object's color, and d represents the target viewing direction at this time; 1.2 Target ray sampling The training subject of the radiation field is set as a discrete target ray. Using the parameters of the target camera, a series of target rays r in the three-dimensional space are randomly sampled: r = o + td, where o represents the origin of the target camera, d represents the direction of the target ray in the three-dimensional space, and t represents the distance from the origin of the camera. 1.3 Feature Extraction Based on the sampled target ray and the modeled voxel grid, trilinear interpolation is used to retrieve features for all voxels x that the target ray r passes through: f = interp(x, V), where interp(·) represents a trilinear interpolation operation, that is, finding the 8 voxel grid nodes closest to x, and performing linear interpolation based on the distance between each grid node and x as a weight; f represents the feature corresponding to the spatial position x, V represents the voxel; the color feature f c By voxel x for V c Interpolation is obtained; in the model link, the shallow neural radiation field F is sent to θ The color feature f c , and the predicted volume density value is obtained directly by trilinear interpolation of the volume density grid: σ = interp(x,V σ ); 1.4 Rendering based on voxel radiation field For a target ray r, extract the appearance features of all voxels it passes through Then it can be sent into the superficial neural radiation field F θ Predict the final color All volume density values Directly obtained by trilinear interpolation of the volume density grid; using The two-dimensional pixel RGB value of the target viewing angle can be obtained through the integration process of formula (1); Previous work approximated the integration process in formula (1) as a discrete summation process and added a background color to the final rendering result; the updated rendering process is as follows: T i represents the cumulative transmittance along the ray from the camera origin to the i-th voxel, α i represents the probability of ray termination at the i-th voxel, c bg It represents the predefined background color, δ i It represents the distance between adjacent voxels; The mean square error loss between the rendered color and the true color (Formula (3)) is used as the reconstruction loss for training.
3. A general shell compression method based on voxel radiation field according to claim 1, It is characterized in that The specific process of step 2 is: 2.1 Using the volume density values of all sampling points on the target ray obtained in step 1 Depth rendering is performed using formula (7) to obtain the estimated target ray depth: where t i represents the distance from the i-th voxel to the ray origin o; 2.2 When locating the surface of an object, the voxels that intersect with the target ray r are screened to find the surface voxels corresponding to r. Specifically, formula (8) is used to find Compare the voxel with the smallest depth difference: Where V(r) represents all voxels intersecting with ray r; D(v) represents the depth distance from voxel v to the ray origin o; If ray r does not intersect any object, the predicted Often at the scene boundary min outside and close to zero; exclude these rays from the depth-guided surface constraint.
4. A general shell compression method based on voxel radiation field according to claim 1, It is characterized in that The specific process of step 3 is: First, a surface distillation constraint is used to compress the voxel grid. In step 2, the estimated voxels around the object surface v are obtained. surf After that, you can query its corresponding appearance feature c surf And the geometric characteristics σ surf ; The voxel feature is used to render formula (5), but for a single element, the rendering process is simplified to: C surf (r)=c surf ·T surf ·α surf , (9) Next, let's add a surface distillation loss L d To achieve the effect of encouraging surface voxels to dominate the rendering results: Here R(P) represents the set of all camera rays for the camera with posture P, N r refers to the total number of all target rays, and C(r) refers to the true value of the pixel for ray r; In addition, in order to accelerate the pruning of other redundant voxels inside and outside the object, non-surface voxels are selected to make their density values close to 0, and the volume density values of all voxels on a ray are obtained. And the corresponding opacity value is obtained by formula (6): After that, the volume density probability density value of each voxel on this ray is calculated And regard this value as the probability that this is the surface of the object. To speed up the pruning of redundant voxels, it is necessary to expect that the volume density probability density value corresponding to the surface voxel is close to 1, and the volume density probability density value of the rest should be close to 0. Therefore, the problem is simplified as a binary classification problem, and the estimated surface voxels obtained in step 2 are used to create corresponding pseudo labels for the classification task. The designed shell-guided loss is defined as the binary cross entropy loss: Here N r represents the total number of all training rays, and N represents the total number of sampling points on a target ray.
5. A general shell compression method based on voxel radiation field according to claim 1, It is characterized in that The specific process of step 4 is: When the volume density value of a voxel is less than a certain threshold τ, the corresponding position of the voxel is considered to be an empty area. At this time, the voxel should be pruned or covered, and no sampling or optimization is performed: Design a way to calculate the importance score that takes into account the volume density value of the voxel, the number of times the voxel is traversed by the target ray, and the distance of the voxel from the camera origin: Here R(v) represents the set of all training rays that pass through voxel v, T i represents the cumulative transmittance along the ray from the camera origin to the i-th voxel, α i represents the probability of ray termination at the i-th voxel; Next, calculate the importance score ratio of each voxel to the entire scene: Here R(M(v)) represents all the unpruned voxels, and P v Sort from high to low to get And calculate a cumulative importance score according to the ranking Determine a fixed important voxel ratio β applicable to various scenarios to represent the most critical voxel ratio that needs to be retained, so that adaptive pruning thresholds for different scenarios can be achieved; the cumulative importance score S v Voxels smaller than this voxel ratio will be pruned away.
