Neural radiation field compression rendering method based on decomposition expression

Abstract

The invention provides a neural radiation field compression rendering method based on decomposition expression, and relates to the technical field of computer graphics, and the method comprises the steps: inputting a multi-view image and camera internal and external parameters, carrying out the ray sampling, generating a space sampling point, carrying out the mixed feature coding of the space sampling point, and obtaining a multi-view image; obtaining three-dimensional voxel features, aligned and fused three-plane features and position codes, and splicing the three features to form fused features; inputting the fusion features into a factorization neural BRDF rendering network to predict volume density, geometric latent features, material parameters and reflection features, and improving the view angle color under the compression condition through a BRDF modulator; a ray weight is predicted through a plane-ray combined modeling module, and a compressed neural radiation field model is obtained through combined optimization of miniaturized body rendering, weighted reconstruction loss and compression constraint and is used for target view angle image rendering; according to the method, the storage overhead of the neural radiation field model is reduced, and meanwhile, the synthesis quality and rendering consistency of the new view angle under different compression ratios are improved.

Description

Technical Field

[0001] This invention relates to the field of computer graphics technology, and in particular to a neural radiation field compression rendering method based on decomposition representation. Background Technology

[0002] With the rapid development of applications such as virtual reality, augmented reality, and digital content creation, higher demands are being placed on the high-fidelity reconstruction and efficient rendering of 3D scenes. Neural Radiation Field (NeRF) implicitly models the volume density and view-dependent color distribution in continuous 3D space using a multilayer perceptron. It can achieve high-quality synthesis of new viewpoint images relying only on multi-view images and their camera parameters, and has significant advantages over traditional mesh or point cloud methods in terms of detail fidelity and view continuity.

[0003] However, existing NeRF methods, even with high compression ratios, often suffer from issues such as loss of high-frequency geometric details, texture blurring, and view-dependent appearance degradation. Furthermore, existing methods typically employ uniform color regression or simple orientation modeling, which struggles to stably characterize view-dependent reflection properties and easily leads to phenomena like specular flicker and inconsistent appearance across different viewpoints. Therefore, how to improve high-frequency detail representation, view-dependent appearance stability, and overall rate-distortion performance while reducing the storage size and computational complexity of neural radiation field models remains a pressing problem to be solved in current technologies. Summary of the Invention

[0004] This invention proposes a neural radiation field compression rendering method based on decomposition representation to solve the problems of high-frequency detail loss and unstable appearance of specular highlights and viewpoint-dependent appearance under high compression ratio conditions in existing technologies.

[0005] This invention provides a neural radiation field compression rendering method based on decomposition representation, the method comprising the following steps: Step S1: Based on the input multi-view images and corresponding camera parameters, perform ray sampling to generate spatial sampling points and their corresponding direction vectors; Step S2: Perform hybrid feature encoding on the spatial sampling points to obtain three-dimensional voxel features, aligned and fused three-plane features, and positional encoding, and then stitch the three together to form a fused feature; Step S3: Input the fused features into the factorized neural BRDF rendering network to predict the volume density, geometric latent features and color of each sampling point, and obtain the predicted color of the pixel through differentiable volume rendering; Step S4: Based on the ray origin, direction, and geometric latent feature sequence along the ray, predict the reconstruction difficulty weight of each ray through the plane-ray joint modeling module; Step S5: Construct a rate-distortion joint loss function, which includes a weighted reconstruction loss between the predicted color and the true color obtained in step S3, and a bitrate regularization term used to constrain the model's storage overhead; wherein the weighted reconstruction loss is weighted using the ray weights obtained in step S4. Step S6: Use the rate-distortion joint loss function to train the network end-to-end to obtain a compressed neural radiation field model for rendering new perspective images.

[0006] Furthermore, the specific method for performing hybrid feature encoding on the spatial sampling points in step S2 to obtain three-dimensional voxel features, aligned and fused three-plane features, and positional encoding, and then concatenating the three to form a fused feature, includes: S2-1: Input spatial sampling points into the 3D hash grid encoder Three-dimensional voxel features were obtained. ; S2-2: Project the spatial sampling points onto three mutually orthogonal feature planes XY, XZ, and YZ, and encode the projected coordinates to obtain the three-plane features. And through a deformable interactive module, the three-plane features are aligned and fused across planes; S2-3: Input the coordinates of the spatial sampling points into the position encoder To obtain the position code ； S2-4: 3D voxel features Three-plane features after alignment and fusion and position encoding By concatenating along the channel dimension, the fused features are obtained. The formula is: .

[0007] Furthermore, in step S2-2, the spatial sampling points are projected onto three mutually orthogonal feature planes XY, XZ, and YZ, and the projected coordinates are encoded to obtain the three-plane features. The specific method for cross-plane alignment and fusion of the three-plane features through a deformable interactive module includes: S2-2-1: Perform layer normalization on the three-plane features to obtain normalized plane features. ; S2-2-2: Normalized planar features Linear mappings are applied separately to predict the channel offset vectors used for cross-plane alignment. ; S2-2-3: The channel offset vector obtained from any plane prediction When applied additively to the other two planes simultaneously, we obtain... ; S2-2-4: Employing an interactive fusion mapping layer Features Compress and generate interactive increments ; S2-2-5: Update the planar features using the ReLU activation function and residual method to obtain the aligned and fused three-plane features. .

[0008] Furthermore, the specific method for inputting the fused features into the factorized neural BRDF rendering network in step S3 to predict the volume density, geometric latent features, and color of each sampling point, and obtaining the predicted color of the pixel through differentiable volume rendering includes: The fusion features obtained in step S2 are input into the factorized neural BRDF rendering network. The volume density and geometric latent features of the spatial sampling points are predicted by the density geometric perceptron. Under the condition of the geometric latent features, the direction vector obtained in step S1 is introduced. Color is generated by the RA-BRDF module. During the training phase, the BRDF-guided adaptive ray sampling strategy is introduced to dynamically adjust the sampling budget. The volume rendering weight of each spatial sampling point along the ray is calculated based on the volume density. The colors of the sampling points are weighted and summed according to the volume rendering weights to obtain the pixel prediction color.

[0009] Furthermore, the factorized neural BRDF rendering network in step S3 includes a density geometry perceptron, which is used to predict the volume density and geometric latent features of spatial sampling points after inputting the fused features obtained in step S2. Specifically, it includes: The spatial sampling points are linearly normalized to the unit cube coordinate system using the scene bounding box, and an effective domain indicator function is set. To suppress the contribution of sampling points outside the bounding box, the volume density value obtained by fusing the feature h input density and the geometric multilayer perceptron is obtained. and geometric latent features Volume density Non-negative density is obtained by using exponential activation, and boundary clipping is completed by combining it with the effective domain indicator function, as shown in the following formula: .

[0010] Furthermore, in step S3, appearance modeling is performed by introducing viewpoint direction information based on the aforementioned geometric latent features, and color generation is achieved through the RA-BRDF module, specifically including: (1) Normalize the direction vector d and obtain the direction features through the direction encoder. The directional features and geometric latent features are concatenated and then linearly projected onto the core feature space. ,in, The linear mapping matrix is ​​input to a two-layer gated residual channel hybrid module (RCMU) for updating. The update form for each layer is shown in the following formula:

[0011] in, It is the ReLU activation function. This indicates channel-by-channel multiplication. Used to generate gated vectors Intermediate representations used to generate hidden channels are then subjected to ReLU nonlinear channel transformation. The intermediate representation is mapped back to the core channel dimension and multiplied channel-by-channel with the gated vector to form the residual increment, thus obtaining the updated core features. LN is the normalization layer; (2) Based on the updated core features, albedo is predicted by material branch and BRDF branch respectively. Roughness With BRDF implicit reflection characteristics The formula is:

[0012]

[0013] Among them, material branches Output RGB diffuse albedo With roughness BRDF branch Output potential high-dimensional reflection feature vector ; (3) Geometric latent features albedo With roughness Together they constitute the modulation input The channel modulation vector is calculated by a BRDF modulator based on the Rank-Aware modulation mechanism. The formula is:

[0014] in, Let be the weight matrix of the two fully connected layers of the modulator network. For bias; The modulation scaling vector is obtained using the hyperbolic tangent function. and BRDF features Perform channel-by-channel modulation to obtain the modulated BRDF features, as shown in the formula:

[0015] Modulated BRDF features After linear mapping, the color is output at the i-th sampling point. And it is mapped to the interval [0,1] using tanh, as shown in the formula:

[0016] in, This is the linear mapping matrix of the BRDF output header, where clip represents the upper and lower bound clipping.

[0017] Furthermore, step S3, which introduces a BRDF-guided adaptive ray sampling strategy to dynamically adjust the sampling budget during the training phase, specifically includes: (1) Based on core feature space and roughness Both are used as input, and the mapping function in the BRDF branch is used to predict the mirror intensity. Then, through Sigmoid mapping, we obtain The formula is:

[0018] Further take the batch average As a global specular sensitivity indicator; (2) The average value of the batch The relative scale factor (scale) is constructed through a normalized mapping, as shown in the formula:

[0019] in, The numerically stable term is obtained by cropping its upper and lower bounds to obtain the original sampling gain. Then through the strength coefficient Interpolation yields the final sampling gain.

[0020] (3) Update the number of rays in each training step according to the sampling gain. Let the current number of rays be... The actual number of sampling points generated during the rendering process is The target sampling batch size is B Then the updated ray number As in the formula:

[0021] The lower bound is 32 to avoid instability during mini-batch training, and the upper bound is 2. 20 To ensure that memory overhead is controllable and to avoid abnormal increases in ray budget, after the ray count is updated, ray sampling is performed according to the updated ray count and volume rendering optimization is executed.

[0022] Further, in step S3, calculating the volume rendering weight of each spatial sampling point along the ray based on the volume density, and then weighting and summing the colors of the sampling points according to the volume rendering weights to obtain the pixel prediction color specifically includes: (1) First, calculate the distance between sampling points along the sampling ray. The expression is as follows:

[0023] in, and These are the depth values ​​of consecutive sampling points i and i+1; (2) Based on the volume density value With distance Calculate cumulative transparency The formula is:

[0024] in, Sampling points The distance value, Sampling points The volume density value; (3) Calculate the volume rendering weights of each sampling point The formula is:

[0025] in, Sampling points The distance value, Sampling points The volume density value, Sampling points Cumulative transparency; (4) The pixel color is obtained by weighted summation of the color values ​​of the sampled points according to the volume rendering weight. The formula is:

[0026] in, It is the weight value of sampling point i. It is the color value of the i-th sampling point.

[0027] Furthermore, the step S4, which involves predicting the reconstruction difficulty weight of each ray based on its origin, direction, and geometric latent feature sequence along the ray using the plane-ray joint modeling module, specifically includes: S4-1: Obtain geometric condition features from the set of geometric latent features obtained by ray sampling through aggregation operators. and the ray origin ray direction Geometric condition features The ray input vector is obtained by concatenation. ; S4-2: Using a linear embedding layer to... Mapped to a fixed-dimensional ray sequence vector:

[0028] in, For embedding matrices of light rays, Here, D is the bias term, and D is the model dimension of the Transformer. All ray sequence vectors form a ray sequence of length N. ; S4-3: Transmit the ray sequence Inputting a Transformer encoder, the query Q, key K, and value V are obtained through linear transformation, using the following formula:

[0029] in, To obtain a learnable parameter matrix, a multi-head self-attention layer is used to aggregate global context information along the ray dimension, and then a context-enhanced ray-level latent representation is obtained via a feedforward network and residual connections. ; S4-4: Further... Input the weighted prediction head and obtain the ray reconstruction difficulty weights via Sigmoid mapping. The formula is:

[0030] in, and These are the learnable weight vector and bias of the ray weight prediction head, respectively; S4-5: Let the first The predicted color and the actual color of the ray are respectively and The weighted reconstruction loss is obtained by weighting the ray-level reconstruction error based on the ray weights, and the formula is as follows: .

[0031] Further, the construction of the rate-distortion joint loss function in step S5 includes a weighted reconstruction loss between the predicted color and the true color obtained in step S3, and a bitrate regularization term used to constrain model storage overhead; wherein, the specific method for weighting the weighted reconstruction loss using the ray weights obtained in step S4 includes: The overall objective function consists of a color reconstruction term and a compression ratio regularization term, as shown in the following formula:

[0032] in, Used to constrain the consistency between rendered colors and real pixels; when ray weights are not introduced, mean squared error is used. As a color reconstruction loss; when ray weights are introduced... The weighted reconstruction loss obtained in step S4 is used for weighted calculation; For rate-distortion tradeoff coefficients, The code rate overhead used to measure parametric coding is defined as the average information content of the quantized parametric symbols under the entropy model, thereby achieving a joint trade-off between reconstruction quality and compression efficiency, as shown in the following formula:

[0033] in, To quantize the total number of parameter symbols, Indicates the first One quantization parameter symbol, For its corresponding context conditions, It is a learnable conditional probability model.

[0034] Compared with the prior art, the present invention has the following advantages: 1. This invention introduces a neural radiation field compression rendering method based on decomposition representation. By hybridizing 3D / 2D feature encoding and introducing a deformable interactive module with three-plane feature domains, cross-plane feature alignment and fusion are achieved, enhancing the ability to express high-frequency geometric structures and detailed information. Through a factorized neural BRDF rendering network, volume density and appearance modeling are decoupled. A density geometry perceptron is used to predict volume density and latent geometric features, and directional features are introduced for appearance modeling under the condition of latent geometric features. The RA-BRDF module decomposes and expresses albedo, roughness, and view-dependent reflection components. Combined with a BRDF modulator, channel adaptive modulation of reflection features is performed, effectively suppressing the amplification of specular highlight region errors under high compression ratios and improving view-dependent appearance consistency and stability. Furthermore, a BRDF-guided adaptive ray sampling strategy is introduced, dynamically adjusting the ray sampling budget according to specular sensitivity. A contextual modeling mechanism is introduced in the ray dimension through a plane-ray joint modeling module to estimate the ray reconstruction difficulty. During rate-distortion joint optimization, the reconstruction loss is weighted, thereby improving the overall reconstruction quality and generalization performance under high compression ratios, while balancing storage overhead and rendering efficiency.

[0035] 2. The method of this invention achieves efficient modeling and rendering of neural radiation fields under high compression ratio constraints, and solves the problems of high frequency detail loss, specular instability and insufficient learning in difficult regions that neural radiation fields are prone to under high compression ratio conditions. It improves viewpoint consistency while significantly reducing model storage and computational overhead.

[0036] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0037] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, with the same or corresponding reference numerals denoteing the same or corresponding parts, wherein: Figure 1 A flowchart of a neural radiation field compression rendering method based on decomposition representation provided by the present invention; Figure 2 A schematic diagram of the network structure for a neural radiation field compression rendering method based on decomposition representation provided by the present invention; Figure 3 A schematic diagram of the three-plane feature encoding and three-plane feature domain deformable interaction module provided by the present invention; Figure 4 A schematic diagram of the factorization rendering module provided by the present invention; Figure 5 This is a structural diagram of the RCMU module provided by the present invention; Figure 6 A schematic diagram of the planar-ray joint modeling module provided by the present invention; Figure 7 This is a first schematic diagram of the neural radiation field compression rendering result based on decomposition representation provided by the present invention; Figure 8 This is a second schematic diagram of the neural radiation field compression rendering result based on decomposition representation provided by the present invention. Detailed Implementation

[0038] The exemplary embodiments disclosed in this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art. Unless otherwise specified, the technical means used in the embodiments are conventional means well known to those skilled in the art.

[0039] Combination Figures 1 to 8 As shown, an optional embodiment of the present invention provides a neural radiation field compression rendering method based on decomposition representation, which significantly reduces model storage overhead while improving the modeling stability and cross-view consistency of specular and complex reflection regions. This makes it suitable for application scenarios with limited bandwidth, limited storage, or requiring efficient rendering. The specific steps of this method include: First, based on the different input viewpoint images and their corresponding camera intrinsic and extrinsic parameters, ray sampling is performed on the images, and multiple spatial sampling points are generated in space along each sampling ray. p(x,y,z) The spatial sampling points and their corresponding direction vectors d are input into a 3D hash grid encoder, a three-plane encoder, and a position encoder to obtain 3D voxel features, three-plane features, and position codes. The three-plane features are aligned and fused through a deformable interactive module of the three-plane feature domain. Then, the aligned and fused three-plane features, 3D voxel features, and position codes are concatenated to construct the fused features of the spatial sampling points. The fused features are input into a factorized neural BRDF rendering network. The density geometry perceptron predicts the volume density and geometric latent features of the spatial sampling points. Under the condition of geometric latent features, the direction vector is introduced for appearance modeling. The sampling point color is generated through the RA-BRDF module. At the same time, the BRDF-guided adaptive ray sampling strategy is introduced during the training phase to dynamically adjust the sampling budget. The color of the sampling points along the ray is integrated by differentiable volume rendering based on the volume density to obtain the pixel predicted color. The ray parameters and the geometric latent feature sequence obtained by sampling along the ray are further input into the plane-ray joint modeling module to jointly model the ray reconstruction difficulty and obtain the ray weights. Finally, the network is jointly optimized for rate distortion based on the weighted reconstruction loss and the code rate regularization term to obtain a compressed neural radiation field model for efficient rendering generation of the target viewpoint. The overall processing flow of the method is as follows: Figure 1 As shown in the diagram, the overall network structure is as follows: Figure 2 As shown.

[0040] Step 1: Ray Sampling and Generation of Spatial Sampling Points: Input multiple images from different viewpoints and their camera intrinsic and extrinsic parameters. Perform ray sampling based on the camera parameters and generate spatial sampling points along the sampling rays. p(x,y,z) and its corresponding direction vector d.

[0041] Step 2: Hybrid 3D / 2D Feature Encoding: Spatial Sampling Points p(x,y,z) Apply 3D hash grid encoders respectively E xyz Three-plane encoder E xy 、E xz 、E yz and position encoder E p Three-dimensional voxel features were obtained. Three-plane features and position encoding γ(p) The three-plane features are input into the deformable interactive module of the three-plane feature domain to realize the alignment and fusion of the three-plane features, and obtain the aligned and fused three-plane features.

[0042] Spatial sampling points p Projecting onto the feature plane, the correlation between different planes is explicitly modeled to achieve cross-plane feature alignment and interactive fusion. The specific implementation process is as follows: Figure 3 As shown.

[0043] Spatial sampling points p Input to 3D hash grid encoder Three-dimensional voxel features were obtained. The spatial sampling points are projected onto three mutually orthogonal feature planes (XY, XZ, YZ), and then passed through a three-plane encoder. To obtain planar features .

[0044] Although the three planes have a geometric correspondence, the different hash mappings and effective resolutions of the encoders in each plane result in inconsistent value distributions of features obtained from the same spatial point on different planes in the feature vector space, thus limiting direct interaction and fusion of cross-plane information. To achieve cross-plane alignment and information interaction, the features of the three planes are first processed by layer normalization LN(·) to obtain... The normalized features are then used to predict the cross-plane feature domain offset vectors by applying linear mappings. ,in, A linear layer is used to predict the offset for the corresponding plane. The cross-plane feature domain offset vector predicted for any plane is then used. Similarly, by applying additive action to the other two planes simultaneously, we obtain... Specifically, it can be expressed as:

[0045] Adopting an interactive fusion mapping layer Features Compression and generation of interactive increments can be specifically represented as follows:

[0046] Similarly, we can obtain Planar features are updated using the ReLU activation function and residual method. This allows us to obtain the aligned and fused three-plane features.

[0047] The aforementioned three-plane feature domain deformable interaction module alleviates the differences between cross-plane feature domains through offset alignment and integrates the interaction increments from other plane features in the form of residuals, thereby improving the consistency and stability of the three-plane features in rendering.

[0048] Furthermore, the coordinates of the spatial sampling points are input into the position encoder. E p To obtain the position code γ(p) .

[0049] Step 3: Fusion Characterization: Integrating 3D Voxel Features Three-plane features after alignment and fusion and position encoding By concatenating along the channel dimension, the fused features are obtained. The formula is:

[0050] Step 4: Factorized Neural BRDF Rendering: Render the fused features obtained in Step 3. Input factorized neural BRDF rendering network, predict volume density of spatial sampling points through density geometry perceptron. and geometric latent features Under the aforementioned geometric latent feature conditions, a directional feature d is introduced, and color is generated through the RA-BRDF module. During the training phase, a BRDF-guided adaptive ray sampling strategy is introduced to dynamically adjust the sampling budget. Based on the volume density, the volume rendering weight of each spatial sampling point along the ray is calculated, and the colors of the sampling points are weighted and summed according to the volume rendering weights to obtain the pixel prediction color. .

[0051] The specific implementation process of the factorized neural BRDF rendering network is as follows: Figure 4 As shown.

[0052] To unify the spatial scale of different scenes, limit the effective modeling range of the network, and suppress the interference of invalid sampling points outside the scene bounding box on training and rendering, scene bounding boxes are used. ,in , Let represent the minimum and maximum corner points of the bounding box in the 3D world coordinate system, respectively. For any spatial sampling point, linear normalization to the unit cube coordinate system is performed, and an effective domain indicator function is defined. , Suppress the contribution of AABB out-sampling points. Input the fused feature h into the density geometry perceptron to predict the volume density and geometric latent features. Among them, volume density Non-negative density is obtained by using exponential activation, and boundary clipping is achieved by combining it with an effective domain indicator function, as shown in the formula:

[0053] The viewpoint direction d is L2 normalized, and the orientation feature is obtained through a spherical harmonic encoder. And then with geometric latent features e As a conditional input, it is passed through a linear projection matrix. Projected onto the core feature space The data is then fed into a two-layer gated residual channel-mixing unit (RCMU) for updating. The overall design of the RCMU module is as follows: Figure 5 As shown. The left branch of RCMU. Used to generate gated vectors, right branch Intermediate representations used to generate hidden channels are then subjected to ReLU nonlinear channel transformation. The intermediate representation is mapped back to the core channel dimension and multiplied channel-by-channel with the gated vector to form the residual increment, which is used to update the core features. The update format for each layer is shown in the formula:

[0054] in, For activation function, This indicates channel-by-channel multiplication.

[0055] Furthermore, material branches and BRDF branches are used to update core features. The formulas for predicting material parameters and reflection features are as follows:

[0056]

[0057] Among them, material branches Output RGB diffuse albedo With roughness BRDF branch Output potential high-dimensional reflection feature vector .

[0058] Because BRDF has strong directionality and a large dynamic range, it is prone to noise amplification during compression and quantization. Therefore, a BRDF modulator based on a Rank-Aware modulation mechanism is introduced to address the geometric latent features. albedo With roughness The modulation input formed by the combination Calculate the channel modulation vector The formula is:

[0059] in, Let be the weight matrix of the two fully connected layers of the modulator network. For bias. The modulation scaling vector is obtained using the hyperbolic tangent function. By modulating the scaling vector BRDF features Perform channel-by-channel modulation to obtain the modulated BRDF features. Modulated BRDF features After linear mapping, the color is output at the i-th sampling point. And it is mapped to the interval [0,1] using tanh, as shown in the formula: in, This is the linear mapping matrix of the BRDF output header, where clip represents the upper and lower bound clipping.

[0060] Furthermore, during the training phase, a BRDF-guided adaptive ray sampling strategy is introduced to dynamically adjust the sampling budget. Based on the volume density, the volume rendering weight of each spatial sampling point along the ray is calculated, and the colors of the sampling points are weighted and summed according to the volume rendering weights to obtain the pixel prediction color. .

[0061] For a ray with an effective depth, let the origin and direction of the ray be respectively... o and d, depth is d i Then the candidate surface points are Construct high light intensity at each candidate point As a proxy for mirror sensitivity from the current perspective, it is derived from the core feature space. and roughness r Both are used as input, and the mapping function in the BRDF branch is used to predict the mirror intensity. Then obtained through Sigmoid mapping As shown in the formula: This design allows areas with stronger mirror surfaces to correspond to higher elevations. This allows for a higher weighting in the sampling budget allocation. Further, the batch mean is taken. As a global specular sensitivity indicator, as shown in the formula:

[0062] Due to different scenarios and training stages The statistical distribution varies, so the batch high light sensitivity is... It is converted into a controllable sampling and adjustment signal, using a power-law approach. s The normalized mapping is greater than 0, and a relative scale factor is constructed using 0.5 as the baseline intensity, specifically expressed as: in, This is a numerically stable term. To prevent large fluctuations in the relative scale factor during early training or under extreme mirror conditions, its upper and lower bounds are cropped to obtain the original sampling gain. and through strength coefficient Interpolation yields the final sampling gain. .

[0063] While ensuring the overall sampling scale remains stable, the gain will be... The number of rays applied in each training step. Let the current number of rays be... The actual number of sampling points generated during the rendering process is The target sampling batch size is B Then the updated ray number The formula is:

[0064] The lower bound is 32 to avoid instability during mini-batch training, and the upper bound is 2. 20 This is used to ensure that memory overhead is controllable and to avoid abnormal increases in the ray budget.

[0065] During the training phase Adjust the number of rays sampled in each step and optimize volume rendering. The BRDF-guided adaptive ray sampling strategy only changes the number and allocation of training samples, without changing the rendering equation or loss form. It can be seamlessly embedded into the training framework, thereby reconstructing specular details more stably under compression conditions.

[0066] After completing the volume density and color prediction, a differentiable volume rendering integral is performed on the spatial sampling points along each ray. First, the distance between sampling points along the sampling ray is calculated. The expression is as follows:

[0067] in, and These are the depth values ​​of consecutive sampling points i and i+1. Subsequently, based on the body density value With distance Calculate cumulative transparency The formula is:

[0068] in, Sampling points The distance value, Sampling points The volume density value; Furthermore, the volume rendering weights of each sampling point are calculated. The formula is:

[0069] in, Sampling points Cumulative transparency Sampling points The distance value, Sampling points The volume density value; Finally, the color values ​​of the sampled points are adjusted according to the volume rendering weights. The pixel color is obtained by performing a weighted summation. The formula is:

[0070] Step 5: Plane-Light Joint Modeling: Identify the origin of each ray. ray direction and the geometric latent feature sequence obtained by ray sampling The ray sequence is encoded and input into the plane-ray joint modeling module for joint modeling to obtain the ray weight of each ray.

[0071] For each sampled ray, the contextual dependencies between rays are modeled using the plane-ray co-modeling module, such as... Figure 6 As shown.

[0072] The geometric latent feature set is used to obtain geometric condition features through an aggregation operator. The ray parameters and geometric conditions are then concatenated to form the ray input vector. Using a linear embedding layer to Mapped to a fixed-dimensional ray sequence vector ,in, For embedding matrices of light rays, Here, D is the bias term, and D is the model dimension of the Transformer. All ray sequence vectors constitute a ray sequence of length N. For ray sequences Execution layer normalization, obtaining the query through linear transformation. ,key Sum : ,in, This is the learnable parameter matrix.

[0073] Then, a multi-head self-attention layer is used to calculate the correlation weight between any two rays in the ray dimension, with the specific expression being:

[0074] in, The scaling factor is used. After attention aggregation, the context-enhanced ray features are obtained. Feedforward networks (FFNs) and residual connections utilize ray characteristics. Obtain context-enhanced ray-level hidden representation The formula is:

[0075] Further Input a single-layer linear weighted prediction head and obtain ray reconstruction difficulty weights via Sigmoid mapping. This weight, acting as a ray importance factor, is used to reweight ray-level reconstruction errors, thereby suppressing the dominance of rays with lower reconstruction difficulty and strengthening the supervision of complex regions. The formula is as follows:

[0076] in, and These are the learnable weight vector and bias of the ray weight prediction head, respectively.

[0077] Let the first The predicted color and the actual color of the ray are respectively and After incorporating the ray difficulty weights, the weighted ray-level reconstruction loss is obtained, as shown in the formula:

[0078] The aforementioned planar-ray joint modeling module can model the correlation between rays from different viewpoints globally, adaptively highlighting rays that are more difficult to reconstruct, and improving the learning efficiency of difficult and high-frequency regions. This module is only introduced during the training phase and does not calculate weights during the inference phase, thus not increasing rendering overhead.

[0079] Step Six: Rate-Distortion Joint Loss Optimization: The network is trained end-to-end using rate-distortion joint optimization. The reconstruction loss is constructed based on the reconstruction error between the pixel predicted color obtained in Step Four and the real image, and the reconstruction loss is weighted using the ray weights obtained in Step Five. At the same time, a rate regularization term is introduced to constrain the compression overhead of the network parameters. The network parameters are optimized using the joint objective function of the weighted reconstruction loss and the rate regularization term to obtain a compressed neural radiation field model.

[0080] The overall objective function consists of a color reconstruction term and a compression ratio regularization term, as shown in the formula:

[0081] in, Used to constrain the consistency between rendered colors and real pixels. When ray weights are not introduced, mean squared error is used. As a color reconstruction loss, when ray weights are introduced, The weighted calculation is performed based on the weighted ray-level reconstruction loss obtained above. For rate-distortion tradeoff coefficients, The code rate overhead used to measure parametric coding is defined as the average information content of the quantized parametric symbols under the entropy model, thereby achieving a joint trade-off between reconstruction quality and compression efficiency, as shown in the formula:

[0082] in, To quantize the total number of parameter symbols, Indicates the first One quantization parameter symbol, For its corresponding context conditions, It is a learnable conditional probability model.

[0083] Step 7: Output Compressed Neural Radiation Field Model: Output the compressed neural radiation field model obtained in Step 6, and use the compressed neural radiation field model to perform volume rendering on the target viewpoint to generate a rendered image of the target viewpoint.

[0084] The neural radiation field compression rendering network based on decomposition representation designed in this invention requires 20,000 iterations to train a single scene, and the reconstruction loss is constructed by using the reconstruction error between pixel predicted color and the real image. Furthermore, a ray-level weighting method is used to weight the reconstruction loss ray by ray, enhancing the learning ability for high-reflectivity regions, geometrically complex regions, and boundary detail regions, thereby improving the reconstruction quality and cross-view consistency of complex regions. In addition, a rate-distortion constraint term is further introduced. Joint constraints are applied to the bit overhead of model storage to maintain a good balance between visual quality and rate-distortion performance under compression. Overall optimization employs an improved Adam optimization method, an enhancement of gradient descent, to update network parameters. A learning rate scheduling strategy is used to gradually adjust the training step size, ensuring stable convergence and obtaining a high-quality compressed radiation field model under compression conditions. Example

[0085] This invention inputs the scenes (chair, lego, ficus, hotdog, drums, materials, mic, ship) from the NeRF-Synthetic dataset into a neural radiation field based on cross-plane deformable interactive fusion and factorized neural BRDF rendering for training, and outputs the corresponding compressed neural radiation field model. The compressed rendering result is as follows: Figure 7 , Figure 8 As shown, from left to right, the images are the original images of eight objects and the composite images generated from the new perspective of the compressed model.

[0086] Experiments on neural radiation field compression in multiple typical scenes on the NeRF-Synthetic dataset demonstrate that our method exhibits significant advantages in image quality and geometric consistency. Specifically, in Figure 7 In the showcased chair scene, the model size is 0.508 MB with a PSNR of 35.98 dB, accurately reproducing high-frequency details such as the fabric texture and wood carvings on the chair surface. In the lego scene, the model size is 0.379 MB with a PSNR of 36.78 dB, and the high-frequency structures such as the edges of the blocks are clear and sharp, with ample detail retention. For the ficus scene, which includes small branches and thin structures, the model is only 0.391 MB with a PSNR of 34.88 dB, with clear edges and contours, no obvious noise or artifacts in the background area, and good overall geometric consistency. In the hotdog scene, the model size is 0.226 MB with a PSNR of 38.12 dB, maintaining rich details and clear boundaries with minimal storage overhead, demonstrating strong compression efficiency. Figure 8 In the drums and materials scenes, which feature strong specular reflection and complex specular distribution, the model sizes are 0.768 MB and 0.622 MB, respectively, with PSNRs of 26.59 dB and 31.40 dB. The specular and reflection changes are continuous and stable across multiple viewpoints, without noticeable flickering or breaks, validating the consistency and robustness of the method in modeling specular highlights and viewpoint-dependent appearance. In the mic scene, the model size is 0.286 MB with a PSNR of 38.31 dB. Thin structures, contour edges, and the boundary between objects and the background exhibit better alignment consistency across different viewpoints, reducing contour drift, edge breaks, and geometric jitter commonly seen under compression conditions. In the ship scene, the model size is 0.778 MB with a PSNR of 32.57 dB. Specular and reflection changes are continuous across multiple viewpoints, with good appearance consistency.

[0087] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A neural radiation field compression rendering method based on decomposition representation, characterized in that, The method includes the following steps: Step S1: Based on the input multi-view images and corresponding camera parameters, perform ray sampling to generate spatial sampling points and their corresponding direction vectors; Step S2: Perform hybrid feature encoding on the spatial sampling points to obtain three-dimensional voxel features, aligned and fused three-plane features, and positional encoding, and then stitch the three together to form a fused feature; Step S3: Input the fused features into the factorized neural BRDF rendering network to predict the volume density, geometric latent features and color of each sampling point, and obtain the predicted color of the pixel through differentiable volume rendering; Step S4: Based on the ray origin, direction, and geometric latent feature sequence along the ray, predict the reconstruction difficulty weight of each ray through the plane-ray joint modeling module; Step S5: Construct a rate-distortion joint loss function, which includes a weighted reconstruction loss between the predicted color and the true color obtained in step S3, and a bitrate regularization term used to constrain the model's storage overhead; wherein the weighted reconstruction loss is weighted using the ray weights obtained in step S4. Step S6: Use the rate-distortion joint loss function to train the network end-to-end to obtain a compressed neural radiation field model for rendering new perspective images.

2. The neural radiation field compression rendering method based on decomposition representation according to claim 1, characterized in that, The specific method for performing hybrid feature encoding on the spatial sampling points in step S2 to obtain three-dimensional voxel features, aligned and fused three-plane features, and positional encoding, and then concatenating the three to form a fused feature, includes: S2-1: Input spatial sampling points into the 3D hash grid encoder Three-dimensional voxel features were obtained. ; S2-2: Project the spatial sampling points onto three mutually orthogonal feature planes XY, XZ, and YZ, and encode the projected coordinates to obtain the three-plane features. And through a deformable interactive module, the three-plane features are aligned and fused across planes; S2-3: Input the coordinates of the spatial sampling points into the position encoder To obtain the position code ； S2-4: 3D voxel features Three-plane features after alignment and fusion and position encoding By concatenating along the channel dimension, the fused features are obtained. The formula is: 。 3. The neural radiation field compression rendering method based on decomposition representation according to claim 2, characterized in that, In step S2-2, the spatial sampling points are projected onto three mutually orthogonal feature planes XY, XZ, and YZ, and the projected coordinates are encoded to obtain the three-plane features. The specific method for cross-plane alignment and fusion of the three-plane features through a deformable interactive module includes: S2-2-1: Perform layer normalization on the three-plane features to obtain normalized plane features. ; S2-2-2: Normalized planar features Linear mappings are applied separately to predict the channel offset vectors used for cross-plane alignment. ; S2-2-3: The channel offset vector obtained from any plane prediction When applied additively to the other two planes simultaneously, we obtain... ; S2-2-4: Employing an interactive fusion mapping layer Features Compress and generate interactive increments ; S2-2-5: Update the planar features using the ReLU activation function and residual method to obtain the aligned and fused three-plane features. .

4. The neural radiation field compression rendering method based on decomposition representation according to claim 1, characterized in that, The specific method for inputting the fused features into the factorized neural BRDF rendering network in step S3 to predict the volume density, geometric latent features, and color of each sampling point, and obtaining the predicted color of the pixel through differentiable volume rendering, includes: The fusion features obtained in step S2 are input into the factorized neural BRDF rendering network. The volume density and geometric latent features of the spatial sampling points are predicted by the density geometric perceptron. Under the condition of the geometric latent features, the direction vector obtained in step S1 is introduced. Color is generated by the RA-BRDF module. During the training phase, the BRDF-guided adaptive ray sampling strategy is introduced to dynamically adjust the sampling budget. The volume rendering weight of each spatial sampling point along the ray is calculated based on the volume density. The colors of the sampling points are weighted and summed according to the volume rendering weights to obtain the pixel prediction color.

5. The neural radiation field compression rendering method based on decomposition representation according to claim 4, characterized in that, In step S3, the factorized neural BRDF rendering network includes a density geometry perceptron, which is used to predict the volume density and geometric latent features of spatial sampling points after inputting the fused features obtained in step S2. Specifically, it includes: The spatial sampling points are linearly normalized to the unit cube coordinate system using the scene bounding box, and an effective domain indicator function is set. To suppress the contribution of sampling points outside the bounding box, the volume density value obtained by fusing the feature h input density and the geometric multilayer perceptron is obtained. and geometric latent features Volume density Non-negative density is obtained by using exponential activation, and boundary clipping is completed by combining it with the effective domain indicator function, as shown in the following formula: 。 6. The neural radiation field compression rendering method based on decomposition representation according to claim 4, characterized in that, Step S3, which introduces viewpoint direction information based on the geometric latent features to perform appearance modeling and generates colors through the RA-BRDF module, specifically includes: (1) Normalize the direction vector d and obtain the direction features through the direction encoder. The directional features and geometric latent features are concatenated and then linearly projected onto the core feature space. ,in, The linear mapping matrix is ​​input to a two-layer gated residual channel hybrid module (RCMU) for updating. The update form for each layer is shown in the following formula: in, It is the ReLU activation function. This indicates channel-by-channel multiplication. Used to generate gated vectors Intermediate representations used to generate hidden channels are then subjected to ReLU nonlinear channel transformation. The intermediate representation is mapped back to the core channel dimension and multiplied channel-by-channel with the gated vector to form the residual increment, thus obtaining the updated core features. LN is the normalization layer; (2) Based on the updated core features, albedo is predicted by material branch and BRDF branch respectively. Roughness With BRDF implicit reflection characteristics The formula is: Among them, material branches Output RGB diffuse albedo With roughness BRDF branch Output potential high-dimensional reflection feature vector ; (3) Geometric latent features albedo With roughness Together they constitute the modulation input The channel modulation vector is calculated by a BRDF modulator based on the Rank-Aware modulation mechanism. The formula is: in, Let be the weight matrix of the two fully connected layers of the modulator network. For bias; The modulation scaling vector is obtained using the hyperbolic tangent function. and BRDF features Perform channel-by-channel modulation to obtain the modulated BRDF features, as shown in the formula: Modulated BRDF features After linear mapping, the color is output at the i-th sampling point. And it is mapped to the interval [0,1] using tanh, as shown in the formula: in, This is the linear mapping matrix of the BRDF output header, where clip represents the upper and lower bound clipping.

7. The neural radiation field compression rendering method based on decomposition representation according to claim 4, characterized in that, Step S3, which introduces a BRDF-guided adaptive ray sampling strategy to dynamically adjust the sampling budget during the training phase, specifically includes: (1) Based on core feature space and roughness Both are used as input, and the mapping function in the BRDF branch is used to predict the mirror intensity. Then, through Sigmoid mapping, we obtain The formula is: Further take the batch average As a global specular sensitivity indicator; (2) The average value of the batch The relative scale factor (scale) is constructed through a normalized mapping, as shown in the formula: in, The numerically stable term is obtained by cropping its upper and lower bounds to obtain the original sampling gain. Then through the strength coefficient Interpolation yields the final sampling gain. (3) Update the number of rays in each training step according to the sampling gain. Let the current number of rays be... The actual number of sampling points generated during the rendering process is The target sampling batch size is B Then the updated ray number As in the formula: The lower bound is 32 to avoid instability during mini-batch training, and the upper bound is 2. 20 To ensure that memory overhead is controllable and to avoid abnormal increases in ray budget, after the ray count is updated, ray sampling is performed according to the updated ray count and volume rendering optimization is executed.

8. The neural radiation field compression rendering method based on decomposition representation according to claim 4, characterized in that, Step S3, which calculates the volume rendering weight of each spatial sampling point along the ray based on the volume density and then weights and sums the colors of the sampling points according to the volume rendering weight to obtain the pixel prediction color, specifically includes: (1) First, calculate the distance between sampling points along the sampling ray. The expression is as follows: in, and These are the depth values ​​of consecutive sampling points i and i+1; (2) Based on the volume density value With distance Calculate cumulative transparency The formula is: in, Sampling points The distance value, Sampling points The volume density value; (3) Calculate the volume rendering weights of each sampling point The formula is: in, Sampling points The distance value, Sampling points The volume density value, Sampling points Cumulative transparency; (4) The pixel color is obtained by weighted summation of the color values ​​of the sampled points according to the volume rendering weight. The formula is: in, It is the weight value of sampling point i. It is the color value of the i-th sampling point.

9. The neural radiation field compression rendering method based on decomposition representation according to claim 1, characterized in that, Step S4, which involves predicting the reconstruction difficulty weight of each ray based on its origin, direction, and geometric latent feature sequence along the ray using a plane-ray joint modeling module, specifically includes: S4-1: Obtain geometric condition features from the set of geometric latent features obtained by ray sampling through aggregation operators. and the ray origin ray direction Geometric condition features The ray input vector is obtained by concatenation. ; S4-2: Using a linear embedding layer to... Mapped to a fixed-dimensional ray sequence vector: in, For embedding matrices of light rays, Here, D is the bias term, and D is the model dimension of the Transformer. All ray sequence vectors form a ray sequence of length N. ; S4-3: Transmit the ray sequence Inputting a Transformer encoder, the query Q, key K, and value V are obtained through linear transformation, using the following formula: in, To obtain a learnable parameter matrix, a multi-head self-attention layer is used to aggregate global context information along the ray dimension, and then a context-enhanced ray-level latent representation is obtained via a feedforward network and residual connections. ; S4-4: Further... Input the weighted prediction head and obtain the ray reconstruction difficulty weights via Sigmoid mapping. The formula is: in, and These are the learnable weight vector and bias of the ray weight prediction head, respectively; S4-5: Let the first The predicted color and the actual color of the ray are respectively and The weighted reconstruction loss is obtained by weighting the ray-level reconstruction error based on the ray weights, and the formula is as follows: 。 10. The neural radiation field compression rendering method based on decomposition representation according to claim 1, characterized in that, The rate-distortion joint loss function constructed in step S5 includes a weighted reconstruction loss between the predicted color and the true color obtained in step S3, and a bitrate regularization term used to constrain model storage overhead; wherein, the specific method for weighting the weighted reconstruction loss using the ray weights obtained in step S4 includes: The overall objective function consists of a color reconstruction term and a compression ratio regularization term, as shown in the following formula: in, Used to constrain the consistency between rendered colors and real pixels; when ray weights are not introduced, mean squared error is used. As a color reconstruction loss; when ray weights are introduced... The weighted reconstruction loss obtained in step S4 is used for weighted calculation; For rate-distortion tradeoff coefficients, The code rate overhead used to measure parametric coding is defined as the average information content of the quantized parametric symbols under the entropy model, thereby achieving a joint trade-off between reconstruction quality and compression efficiency, as shown in the following formula: in, To quantize the total number of parameter symbols, Indicates the first One quantization parameter symbol, For its corresponding context conditions, It is a learnable conditional probability model.

Images