Context model based instant-ngp model compression method
By introducing contextual models and arithmetic coding techniques, the hash representation in the Instant-NGP model is parameter estimated and fused, which solves the problem of large model storage space and achieves efficient compression.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2024-06-03
- Publication Date
- 2026-05-26
AI Technical Summary
The existing Instant-NGP model has a large storage space due to the introduction of explicit hash representation, and further compression is needed.
A context model is introduced to estimate the parameter values in the hash representation. Context information is provided through inter-layer and inter-dimensional context models, and model compression is achieved by hash fusion and arithmetic coding techniques.
It effectively reduced the information entropy of the model, achieving a compression effect of more than 100 times, while maintaining or improving the rendering quality.
Smart Images
Figure CN118586471B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a technology in the field of neural network applications, specifically an Instant-NGP model compression method based on a context model. Background Technology
[0002] Instant-NGP is a model for synthesizing 3D scenes using multi-view image rendering ("SIGGRAPHInstant Neural Graphics Primitives with a Multiresolution Hash Encoding"). Due to the introduction of explicit hash representation, this model has a large storage space. BiRF addresses this by introducing a hybrid 3D and 2D hash representation ("Binary radiance fields") and binarizing the parameters of the hash representation to +1 or -1, thus achieving initial compression of this hash representation. This invention further compresses the hybrid hash representation of Instant-NGP based on this BiRF approach. Summary of the Invention
[0003] To address the aforementioned shortcomings of existing technologies, this invention proposes an Instant-NGP model compression method based on a context model. This method introduces a context model to estimate the binarized parameter values in the hash representation to reduce its information entropy, and finally uses arithmetic coding techniques to achieve compression.
[0004] This invention is achieved through the following technical solution:
[0005] This invention relates to an Instant-NGP model compression method based on a context model. By constructing and training a context model customized for hash representations of three-dimensional and two-dimensional hybrid hashes, the method performs probability estimation on the parameter values in the hash table, hashes the information of all index nodes that generate hash collisions in the estimated parameter values, and then uses arithmetic coding techniques to achieve entropy coding, thereby achieving the purpose of model compression.
[0006] The aforementioned context model includes: an inter-layer context model and an inter-dimensional context model, wherein: the inter-layer context model, based on the multi-layer and multi-resolution characteristics of hash representation, uses shallow hash representation to provide inter-layer context information for deep hash representation through linear interpolation; the inter-dimensional context model projects the reconstructed three-dimensional hash representation onto two dimensions and then provides inter-dimensional context information for the two-dimensional hash representation through linear interpolation.
[0007] The interlayer context information is obtained in the following way: the shallow layer is first decoded and used to provide context for estimating the parameter values of the deep layer: for a parameter in the deep layer that needs to be probabilistically estimated, the position of its corresponding grid point is found through hash index, and the position of the grid point is used to perform linear interpolation in the hash representation of the shallow layer to obtain context information. After inputting this context information into a multilayer perceptron (MLP), the probability prediction of the parameter value is realized.
[0008] The probability prediction uses the Bernoulli probability model. For a predicted value p, if the true value of the current parameter is +1, then its probability is p; otherwise, if the true value of the current parameter is -1, then its probability is 1-p.
[0009] The linear interpolation mentioned refers to the step of using the feature values of neighboring points and performing a weighted summation based on the distances between the current grid point and its neighboring points. Taking one-dimensional linear interpolation as an example, for point C located between line segments AB, the specific steps are as follows: Where: f A and f B These are the characteristic values of points A and B, respectively, f C The characteristic value of point C obtained by interpolation, D AC and D CB These are the lengths of line segments AC and CB, respectively, and D... AB It is the length of line segment AB.
[0010] The aforementioned multilayer perceptron (MLP) is a neural network module composed of several neural network units.
[0011] The aforementioned inter-dimensional context information is obtained in the following way: the three-dimensional hash representation is decoded using the above-mentioned inter-layer context model, and after being dimensionality reduced and projected onto two dimensions, context information is provided for the two-dimensional hash representation using the same inter-layer context model.
[0012] The method of dimensionality reduction projection is to calculate the proportion of the +1 parameter to all parameters on the axis that needs to be dimensionality reduced.
[0013] The hash fusion refers to: after obtaining the predicted probability values for all indexed grid points using the inter-layer context model and the inter-dimensional context model, using normalized AOE weights to perform a weighted sum of these probability values to obtain the final probability estimate for the parameter, specifically: in: p is the probability value at the grid point predicted by the above inter-layer and inter-dimensional context model. i For the current parameter θ i The final fused probability prediction value, normalized AOE weights K is the value for the current parameter θ. i The number of grid points indexed.
[0014] The AOE mentioned is an intersecting volume, numerically calculated as the intersection volume between the interpolation range of the grid points and the effective occupied grid in the Instant-NGP model. The definition of the occupied grid can be found in the Instant-NGP model.
[0015] When parameter θ i If the AOE of all corresponding grid points is 0, it means that this parameter is not applied to the rendering and can be directly removed to reduce the model size.
[0016] This invention relates to a system for implementing the above method, comprising: an inter-layer context module, an inter-dimensional context module, and a hash fusion module, wherein: the inter-layer context module predicts the numerical probability of a parameter based on the multi-layer, multi-resolution characteristics of the hash representation and through context prediction relationships from shallow to deep layers; the inter-dimensional context module uses a three-dimensional hash representation to provide contextual information for a two-dimensional hash representation, thereby improving the latter's parameter prediction accuracy; the hash fusion module utilizes the characteristics of hash indexing to fuse the prediction probabilities at all grid points indexed by the current parameter, thereby improving the prediction accuracy of the current parameter.
[0017] Technical effect
[0018] This invention utilizes a multi-layer, multi-resolution inter-layer context model designed with hash representations, effectively removing structural redundancy and reducing information entropy. An inter-dimensional context model designed using the dimensional relationships between three-dimensional and two-dimensional hash representations effectively removes inter-dimensional redundancy, achieving model compression. Through a hash fusion module, hash collisions are effectively utilized to improve the accuracy of parameter value prediction, thus ensuring the effectiveness of compression. Compared to existing technologies, this invention can predict parameter values in hash representations, uses a Bernoulli mathematical model to obtain the probability of prediction accuracy, and uses arithmetic coding for entropy encoding compression based on this probability, thereby reducing bit consumption during encoding and saving storage space. Attached Figure Description
[0019] Figure 1 This is the main flowchart of the present invention;
[0020] Figure 2 This is a schematic diagram illustrating the relationship between the present invention and the benchmark model Instant-NGP;
[0021] In the figure: the upper left part is the process of the baseline model Instant-NGP, the lower left part is a schematic diagram of the scope of the proposed inter-layer and inter-dimensional context model; the right part is a schematic diagram of the reduction in spatial size after compression. In the illustrated example, a compression effect of more than 100 times is achieved.
[0022] Figure 3 System module block diagram: A schematic diagram of the inter-layer context model of hash representation;
[0023] Figure 4 System module block diagram: A schematic diagram of the inter-dimensional context model of hash representation;
[0024] Figure 5 System module block diagram: Schematic diagram of the hash fusion module;
[0025] Figure 6 This is a subjective effect diagram of the present invention compared to the benchmark methods Instant-NGP and BiRF. Detailed Implementation
[0026] like Figure 2 As shown, this embodiment relates to an Instant-NGP model compression method based on a context model, including:
[0027] Step 1: Acquire multi-view images and their corresponding view parameters, specifically including:
[0028] 1.1 Arrange multiple identical cameras in a hemispherical shape, place the object to be photographed in the center, and simultaneously press the camera shutters to obtain multi-view images of the object. Alternatively, use a single camera device to capture multi-view images of a scene from multiple different perspectives.
[0029] 1.2 Obtain the viewing angle parameters by collecting the intrinsic parameters of the camera and the pose of the camera arrangement.
[0030] Step 2: Construct an end-to-end network including Instant-NGP and the corresponding compression module, specifically including:
[0031] 2.1 Construct a hybrid three-dimensional and two-dimensional hash representation, and use BiRF technology to binarize the parameters in the hash representation into +1 and -1.
[0032] 2.2 Based on the multi-layered and multi-resolution characteristics of hash representation, an inter-layer context model is first constructed. Shallow hash representations are then used to provide inter-layer context information to deeper hash representations through linear interpolation. Specifically, as follows... Figure 3 As shown: After recovering the shallow hash representation through decoding, context information is provided to the corresponding grid points of the current layer's hash representation through linear interpolation, combined with the frequency f of +1 in the current layer.G A common input multilayer perceptron (MLP) is used to obtain the predicted value p for the current grid point. i The probability that the point is accurately predicted is calculated using Bernoulli's distribution.
[0033] 2.3 Constructing an Inter-Dimensional Context Model: After mapping the three-dimensional hash representation obtained in step 2.2 to two dimensions, inter-dimensional context information is provided to the two-dimensional hash representation through linear interpolation. For example... Figure 4 As shown, the inter-dimensional context model obtains the three-dimensional part of the hash representation by decoding, and then projects the three-dimensional part into the corresponding two-dimensional representation by dimensionality reduction along any one of the xyz axes. When decoding the two-dimensional part of the hash representation, the context information is obtained by linear interpolation of the grid point positions on the representation after dimensionality reduction and projection.
[0034] 2.4 A hash fusion module is introduced to fuse all the grid points to which the parameters are hashed, such as... Figure 5 As shown: For a parameter θ i The predicted probabilities of all indexed grid points are first calculated using steps 2.2 and 2.3. Then through Weighted summation is used to obtain the parameter θ. i The final predicted probability The number of bits consumed is then calculated by calculating information entropy. Among them, weight The method of obtaining it is through AOE normalization.
[0035] 2.5 After obtaining the probability of each grid point through steps 2.2 and 2.3, in step 2.4, the probabilities of these grid points are fused into the corresponding parameter θ. i And obtain the final predicted probability p i Then, entropy coding can be performed using arithmetic coding to achieve model compression.
[0036] Step 3: Input the multi-view images and view parameters acquired in Step 1 into the end-to-end network constructed in Step 2 for training. Then, perform arithmetic encoding on the trained network parameters, using the loss function L = L... mse +λL entropy / M, where: L mse It is the mean squared error function, originally used in Instant-NGP and BiRF technologies for rendering quality loss, measuring the similarity between the rendered image and the ground truth image; L entropyThis is the loss component proposed in this invention, used to measure the number of bits consumed by the hash representation, directly reflecting the model size. λ is a trade-off parameter between the two components; a larger λ results in a smaller compressed model but lower rendering quality, and vice versa. M is the number of all parameters in the hash representation, and L... entropy The calculation method is the sum of the number of bits consumed by all parameters in the hash representation: Where L represents the number of layers in the hash representation, and I represents the number of parameters in the current layer.
[0037] The arithmetic encoding refers to representing a given numerical value and its corresponding predicted probability using a bit stream of "01".
[0038] Through specific practical experiments, using an NVIDIA 3090 GPU, the initial learning rate was set to 0.01, and the training steps were 20,000 epochs. For the 3D hash representation, 12 layers were used, with a shallowest layer resolution of 16 and a deepest layer resolution of 512; for the 2D hash representation, 4 layers were used, with a shallowest layer resolution of 128 and a deepest layer resolution of 1024. For the 3D hash representation, the maximum hash table length per layer was 2. 19 For a two-dimensional hash representation, the maximum length of the hash table at each level is 2. 17 In the inter-layer context, the number of shallow layers providing context information to deeper layers is 3. The dimension F of the parameters in the hash representation is set to 8. λ is set to 0.7e-3 to 4e-3, and the results obtained through the above method are shown in Tables 1-3.
[0039] Table 1. Experimental results of this invention on the Synthetic-NeRF dataset (F = 8, λ = 1e-3)
[0040]
[0041]
[0042] Table 2. Experimental results of BiRF on the Synthetic-NeRF dataset (F=8)
[0043] Scene PSNR↑ SSIM↑ LPIPS↓ Model size (MB) ↓ chair 36.17 0.986 0.016 5.6 drums 26.05 0.937 0.063 5.7 ficus 34.71 0.984 0.018 5.8 hotdog 37.51 0.981 0.028 5.8 lego 36.48 0.982 0.015 5.8 materials 30.09 0.951 0.051 5.7 mic 37.44 0.992 0.009 5.6 ship 30.27 0.897 0.112 6.0 average 33.59 0.964 0.039 5.8
[0044] Table 3. Experimental results of Instant-NGP on the Synthetic-NeRF dataset (F=8)
[0045] Scene PSNR↑ SSIM↑ LPIPS↓ Model size (MB) ↓ chair 35.91 0.986 0.021 45.56 drums 25.18 0.933 0.092 45.56 ficus 33.76 0.983 0.024 45.56 hotdog 37.48 0.983 0.034 45.56 lego 35.86 0.981 0.022 45.56 materials 29.65 0.950 0.069 45.56 mic 36.98 0.992 0.014 45.56 ship 30.93 0.896 0.138 45.56 average 33.22 0.963 0.052 45.56
[0046] Compared with existing technologies, this invention achieves a smaller model while maintaining the same or higher rendering quality by accurately predicting the parameter values in the hash representation.
[0047] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.
Claims
1. A context-based Instant-NGP model compression method, characterized in that, By constructing and training a context model customized for hash representations of three-dimensional and two-dimensional hybrid hashes, the model is used to perform probability estimation of parameter values in the hash table. Then, the information of all index nodes that generate hash collisions in the estimated parameter values is hashed and fused. Finally, entropy coding is achieved through arithmetic coding techniques to achieve the purpose of model compression. The aforementioned context model includes: an inter-layer context model and an inter-dimensional context model, wherein: the inter-layer context model, based on the multi-layer and multi-resolution characteristics of hash representation, uses shallow hash representation to provide inter-layer context information for deep hash representation through linear interpolation; the inter-dimensional context model projects the reconstructed three-dimensional hash representation onto two dimensions and then provides inter-dimensional context information for the two-dimensional hash representation through linear interpolation. The input to the Instant-NGP model is a multi-view image and view parameters.
2. The Instant-NGP model compression method based on a context model according to claim 1, characterized in that, The interlayer context information is obtained in the following way: the shallow layer is first decoded and used to provide context for estimating the parameter values of the deep layer: for a parameter in the deep layer that needs to be probabilistically estimated, the position of its corresponding grid point is found through hash index, and the position of the grid point is used to perform linear interpolation in the hash representation of the shallow layer to obtain context information. After inputting this context information into a multilayer perceptron (MLP), the probability prediction of the parameter value is realized.
3. The Instant-NGP model compression method based on a context model according to claim 2, characterized in that, The aforementioned probability prediction employs a Bernoulli probability model, for a given predicted value... If the true value of the current parameter is +1, then its probability is... Conversely, if the true value of the current parameter is -1, then its probability is... .
4. The Instant-NGP model compression method based on a context model according to claim 2, characterized in that, The linear interpolation mentioned refers to the step of using the feature values of neighboring points and performing a weighted summation based on the distances between the current grid point and its neighboring points. Taking one-dimensional linear interpolation as an example, for point C located between line segments AB, the specific steps are as follows: ,in: and These are the characteristic values of points A and B, respectively. These are the characteristic values of point C obtained through interpolation. and These are the lengths of line segments AC and CB, respectively. It is the length of line segment AB.
5. The Instant-NGP model compression method based on a context model according to claim 1, characterized in that, The aforementioned inter-dimensional context information is obtained in the following way: the three-dimensional hash representation is decoded using the above-mentioned inter-layer context model, and after being dimensionality reduced and projected onto two dimensions, context information is provided for the two-dimensional hash representation using the same inter-layer context model.
6. The Instant-NGP model compression method based on a context model according to claim 5, characterized in that, The method of dimensionality reduction projection is to calculate the proportion of the +1 parameter to all parameters on the axis that needs to be dimensionality reduced.
7. The Instant-NGP model compression method based on a context model according to claim 1, characterized in that, The hash fusion refers to: after obtaining the predicted probability values for all indexed grid points using the inter-layer context model and the inter-dimensional context model, using normalized AOE weights to perform a weighted sum of these probability values to obtain the final probability estimate for the parameter, specifically: ,in: These are the probability values at the grid points predicted through the aforementioned inter-layer and inter-dimensional context models. This is for the current parameter The final fused probability prediction value, normalized AOE weights , This is for the current parameter The number of grid points indexed.
8. The Instant-NGP model compression method based on a context model according to claim 7, characterized in that, When parameter If the AOE of all corresponding grid points is 0, it means that this parameter is not applied to the rendering and can be directly removed to reduce the model size.
9. An Instant-NGP model compression system based on a context model that implements the method of any one of claims 1-8, characterized in that, include: The system includes an inter-layer context module, an inter-dimensional context module, and a hash fusion module. The inter-layer context module predicts the numerical probability of parameters based on the multi-layer and multi-resolution characteristics of hash representations and the context prediction relationship from shallow to deep layers. The inter-dimensional context module uses three-dimensional hash representations to provide context information for two-dimensional hash representations, thereby improving the latter's parameter prediction accuracy. The hash fusion module utilizes the characteristics of hash indexes to merge the predicted probabilities of all grid points indexed by the current parameter, thereby improving the prediction accuracy of the current parameter.