Explicit volume rendering primitive densification method based on detail perception gradient

Abstract

The invention belongs to the field of computer graphics and computer vision, and provides an explicit volume rendering primitive densification method based on detail perception gradient, which is suitable for a three-dimensional reconstruction task under a sparse view angle or sparse point cloud condition. According to the method, a detail perception gradient index is introduced, the fitting capability of a current model to a fuzzy region in an image is dynamically evaluated, and whether point refinement operation is executed or not is judged according to the fitting capability. Compared with a traditional densification strategy depending on position gradient, the index provided by the invention can more accurately identify a detail missing region, and the rationality of density distribution is improved. Furthermore, according to the method, visual angle correlation and gradient normalization information are combined, a threshold function is set to split and reconstruct Gaussian primitives, and low-overhead and controllably-distributed point set enhancement is achieved. While the compactness of the whole model is kept, the reconstruction quality of a complex structure and an edge region is effectively improved, and the method can be widely applied to various application scenes such as real-time rendering, underwater perception and sparse reconstruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and graphics, and particularly relates to an explicit volume rendering primitive densification method based on detail perception gradient. BACKGROUND

[0002] Explicit volume rendering is a three-dimensional modeling and image synthesis method based on spatial parameterization primitives, which has the characteristics of high efficiency, continuity and differentiability, and is widely used in tasks such as novel view synthesis, virtual reality and three-dimensional reconstruction. Three-dimensional Gaussian splatting (3DGS) is a representative technology, which models the scene through a set of Gaussian primitives with attributes such as position, scale, direction and color, and realizes high-quality and real-time rendering.

[0003] However, such traditional densification strategies have the following problems:

[0004] Ignoring blurred areas: Under conditions such as underwater images, low-texture areas or distant objects, although these areas exhibit low gradients or low residuals, their structural information is still important, and traditional methods are difficult to accurately identify.

[0005] Misjudgment of boundaries or detailed structures: Simply relying on amplitude indicators can lead to redundant primitives in non-structural areas such as brightness mutations and occluded boundaries, increasing storage and computing burden.

[0006] Lack of controllability and adaptability: Existing methods usually use fixed thresholds or global rules, lacking the ability to adjust in detail for different scenes, perspectives or feature areas.

[0007] Therefore, there is an urgent need for a Gaussian primitive densification method that is more detail-perception-capable, more controllable and adaptable to complex perception scenarios, to improve rendering quality, mapping accuracy and computing efficiency, especially in resource-constrained scenarios such as sparse perspectives, sparse point clouds and underwater imaging. SUMMARY

[0008] The present application aims to overcome the deficiencies of existing three-dimensional volume rendering methods in the primitive densification phase, and proposes an explicit rendering primitive densification method based on detail perception gradient, which is used to improve the three-dimensional reconstruction accuracy and image synthesis quality of the model in sparse perspective, high noise or low texture scenes, and is particularly suitable for underwater perception, virtual reality and sparse mapping applications in resource-constrained application scenarios.

[0009] The "splatting" provided by the present application is an image synthesis method based on spatial explicit distribution function, and the rendering unit (i.e. "explicit primitive") refers to a volume distribution unit with explicit spatial parameters (including position, size, direction) and attribute parameters (including density, transparency, color or features, etc.). The distribution function of the unit is not limited to the standard Gaussian function, but also includes truncated Gaussian, exponential decay function, radial basis function, generalized Gaussian kernel function or other spatial distribution models with continuous differentiable characteristics.

[0010] The technical solutions of the present application are as follows:

[0011] As shown in Figure 1 , an explicit volume rendering primitive densification method based on detail perception gradient, the steps are as follows:

[0012] First step: based on the input rendering image or projection image, the projection of each explicit primitive on the image is obtained, and the gradient information about the position parameter is calculated Wherein is the rendering loss function, p i is the i-th pixel where the primitive is projected, and x is the spatial position of the primitive;

[0013] Second step: introduce a detail perception gradient index Q, measure the ratio of the total projection gradient norm M of each pixel to the overall gradient information G, which is used to indicate the underfitting area of the model in the image detail expression; wherein η is a small number close to 0 to prevent illegal numerical operation;

[0014] Third step: based on G and Q, the primitives are sorted and screened, and the subset that needs to perform densification operation is selected;

[0015] All primitives are sorted according to the size of G index and Q index respectively. The union of the top r percent of each primitive set is taken as the primitive to be densified in this round. Wherein r is equal to Where S tar represents the target model size, S init represents the initial model size, and S cur represents the current model size. Here S tar is set to the

[0016] Fourth step: perform densification operations such as splitting and cloning on the selected subset to increase the number of primitives of the model and enhance the representation ability of the model;

[0017] Step 5: Continue training model parameters, continue to execute the first to fifth steps of the densification process after the rated number of steps, until the stop condition is reached.

[0018] Compared with the existing densification methods based on position gradient amplitude or image residual, the present application has the following beneficial effects:

[0019] 1. Stronger detail recognition ability: the normalized detail perception index of the present application can effectively capture low-contrast, blurred but structurally important areas;

[0020] 2. More reasonable primitive resource allocation: by calculating resources in response to intensity, the structure area is accurately enhanced rather than globally expanded redundantly;

[0021] 3. Wider adaptability: the present method is suitable for various types of explicit volume rendering methods, including but not limited to standard 3D Gaussian splatting, truncated Gaussian model, Bezier boundary Gaussian model or other image space distribution projection methods

[0022] In summary, the present application provides a high-robustness densification method suitable for general explicit volume rendering systems, which significantly improves the expression ability of high-frequency details and complex shapes while balancing model compactness and rendering efficiency, and has good engineering practical value and theoretical popularization potential. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a densification flowchart;

[0024] Figure 2 is an embodiment flowchart;

[0025] Figure 3 is a primitive distribution map densified using different densification indicators;

[0026] Figure 4 is a local comparison chart of the reconstruction results of scenes scan24 under the DTU dataset;

[0027] Figure 5 is a bubble chart of the average reconstruction accuracy-speed-model size of 10 scenes under the DTU dataset. DETAILED DESCRIPTION

[0028] In order to more clearly illustrate the technical solutions of the present application, the following will combine specific embodiments to describe in detail a Gaussian primitive densification method for three-dimensional reconstruction and image rendering proposed by the present application.

[0029] In this embodiment, the explicit rendering primitive is initialized with a three-dimensional Gaussian kernel, but is not limited to a standard Gaussian function, and can be extended to a truncated Gaussian, a generalized exponential kernel, and other differentiable distribution functions with spatially controllable decay. Some scenes in the DTU dataset are selected for testing, and the experimental platform is a high-performance workstation equipped with an NVIDIA RTX 3090 graphics card.

[0030] In order to more clearly show the implementation process and implementation effect of the present application, as shown in Figure 2 , the embodiment applies the densification method proposed by the present application to a three-dimensional Gaussian splatting reconstruction process comprising multiple steps. The specific implementation process is as follows, wherein the first to fourth steps and the sixth and seventh steps are the reconstruction process of the three-dimensional Gaussian splatting, and the fifth step is the densification method proposed by the present application.

[0031] Step 1: Primitive initialization: use COLMAP to generate an initial point cloud and initialize a three-dimensional Gaussian primitive accordingly;

[0032] Step 2: Forward rendering and loss construction: forward rendering is performed under multiple viewpoints to obtain rendered images , which are compared with corresponding reference images I to construct a loss function The loss function can be an L1 loss or a structural similarity index (SSIM);

[0033] Step 3: Training and storing gradient information: using the constructed loss function , the parameters of the Gaussian primitive are optimized by gradient descent, and the L j of the jth time is calculated The Gaussian position gradient of the Gaussian primitive k and the norm sum of the projection gradient of each pixel k is updated. If the current viewpoint is , it is considered that the primitive is activated, and its appearance frequency N k is updated to N k +1. The overall training step number j = j + 1;

[0034] Step 4: Training and densification control: when the training step number j satisfies 500 < j < 15000 and j mod 100 == = 0, perform a densification operation according to step 6 once; when the training step number j > 30000, jump to step 11, otherwise jump to step 2;

[0035] Step 5: Densification method

[0036] 5.1 Calculate the average gradient and index: calculate the average Gaussian position gradient and the norm sum of the average projection gradient of each pixel wherein η is an infinitesimal quantity to prevent numerical errors, and is taken as 1 x 10 -7 ;

[0037] 5.2 Calculation of the detail-aware gradient indicator As shown in Figure 3 , the leftmost image is the model without sufficient training, and the right side is the normalized graph according to different densification indicators. The brighter the color, the larger the value. It can be found that, compared with the mainstream 3DGS / GOF and other SOTA methods, the use of our indicators can find the primitives of the under-fitting area for the model without sufficient training;

[0038] 5.3 Sort all primitives according to the size of the G indicator and the Q indicator respectively. Take the union of the top r percentile of the primitive set as the primitive to be densified in this round. Wherein r is equal to wherein S tar represents the target model size, S init represents the initial model size, and S cur represents the current model size. Here, S tar is set to 80% of the SOTA result (GES) in the current scene.

[0039] 5.4 Perform the densification operation: for the primitives in step 5.3, clear their Gaussian position gradient G, average densification indicator Q, and occurrence frequency N. According to the size, perform cloning (copying) or splitting operation to enhance the detail representation ability;

[0040] Step 6: Jump to Step 2 and continue training;

[0041] Step 7: Terminate the training, save the Gaussian parameters, and end the training process.

[0042] The final training effect and the comparison results with other methods are shown in Figure 4 and Figure 5 , which verifies that the method of the present application has a positive effect on improving image quality and reducing model size.

Claims

1. A method for explicit volume rendering primitive densification based on detail-aware gradients, characterized in that, The steps are as follows: Step 1: Based on the input rendered or projected image, obtain the projected response of each explicit primitive on the image and calculate its gradient information with respect to the position parameters. in It is the rendering loss function, p i is the i-th pixel where the primitive projection is located, and x is the spatial position of the primitive; The second step involves introducing a detail-aware gradient metric, Q, which measures the ratio of the sum of the projected gradient norms M of each pixel to the overall gradient information G. This metric indicates areas where the model is underfitting in terms of image detail representation. η is a decimal close to 0 used to prevent illegal numerical operations; Step 3: Sort and filter the primitives based on G and Q to select the subset that needs to be compacted. Step 4: Perform a compaction operation on the selected subset to increase the number of primitives in the model and enhance its representational power; Step 5: Continue training the model parameters. After the set number of steps, continue the densification process from step 1 to step 5 until the stopping condition is met.

2. The explicit volume rendering primitive densification method based on detail-aware gradient as described in claim 1, characterized in that, The third step, as described above, is performed as follows: Sort all primitives according to the magnitude of the G and Q indices respectively; take the union of the sets of primitives in the top r percent of each index as the primitives to be compacted in this round; where r equals Where S tar S represents the size of the target model. init S represents the initial model size. cur This represents the current model size; here S tar Set as the state-of-the-art (GES) result for the current scenario.

3. The explicit volume rendering primitive densification method based on detail-aware gradient as described in claim 1, characterized in that, In the fourth step, the densification operation employs splitting and cloning operations.

Images