Layered densification Gaussian sputtering method based on visibility
By constructing a multidimensional observable parameter set and using a hierarchical clustering Gaussian sputtering method, the problem of insufficient reconstruction accuracy under sparse views is solved, achieving efficient optimization of geometric continuity and radiation consistency, and improving rendering quality and efficiency.
Patent Information
- Application Number
- CN202511646602.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-11
- Publication Date
- 2026-02-27
AI Technical Summary
Existing 3D Gaussian scattering methods lack sufficient reconstruction accuracy under sparse view conditions, cannot effectively distinguish the differences in visibility between different regions, resulting in improper allocation of computational resources and affecting efficiency and quality.
A visibility-based hierarchical densification Gaussian sputtering method is adopted. By constructing a multidimensional observable parameter set, weighted fusion and hierarchical clustering of Gaussian elements are performed to dynamically adjust the densification range and density, thereby optimizing the Gaussian structure of the salient layer, transition layer and background layer.
Under sparse view conditions, the continuity of scene geometry and radiation consistency are maintained, rendering quality and computational efficiency are improved, redundancy is reduced, and local texture detail reconstruction is enhanced.
Smart Images

Figure CN121582428A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and more specifically to a visibility-based layered densification Gaussian sputtering method. Background Technology
[0002] Novel perspective synthesis techniques aim to generate realistic images from any viewpoint using images with limited perspectives, and are a core research direction in computer vision and graphics. Existing novel perspective synthesis methods mainly fall into two categories: explicit geometric modeling and implicit neural field representation. The former relies on polygonal meshes or point cloud structures, achieving rendering by explicitly encoding geometric and color information, but suffers from high computational and storage overhead in complex scenes. The latter, represented by Neural Radiation Field (NeRF), achieves high-fidelity synthesis by implicitly modeling the radiative properties of 3D space; however, it is highly dependent on multi-viewpoint input and its performance degrades under sparse views. In recent years, 3D Gaussian Splatting, as an efficient scheme combining explicit representation and differentiable rendering, can generate high-quality images in real-time, but still relies on sufficient viewpoint coverage and struggles to maintain geometric integrity and texture accuracy under sparse observation conditions.
[0003] Existing 3D Gaussian scattering and its improved methods primarily enhance reconstruction accuracy under sparse view conditions through global optimization or error-driven densification strategies. However, these methods generally fail to adequately consider the differences in visibility across regions, often resulting in an even distribution of computational resources. This leads to insufficient reconstruction of critical geometric regions while over-computing low-visibility areas, impacting both efficiency and quality. Furthermore, existing schemes often employ fixed sampling or global error-driven mechanisms, failing to dynamically adjust the densification range and density based on scene characteristics, resulting in local texture blurring or boundary discontinuities. Simultaneously, fixed visibility thresholds struggle to cope with varying scene complexity, causing unstable layering and inconsistencies with optimization objectives. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this invention discloses a visibility-based hierarchical densification Gaussian sputtering method, which aims to maintain computational stability while comprehensively optimizing geometric continuity and radiation consistency in sparse view scenes, thereby improving the problem of decreased reconstruction accuracy caused by observation sparsity.
[0005] To achieve the above-mentioned technical effects, the present invention adopts the following technical solution: A visibility-based layered densification Gaussian sputtering method includes: Step 1: Obtain sparse multi-view image data, construct the initial 3D Gaussian representation of the scene, and determine the spatial location, shape parameters, and radiation properties of each Gaussian element; Step 2: Calculate a multidimensional observability parameter set based on the multi-view projection data of each Gaussian element in the initial three-dimensional Gaussian representation. The multidimensional observability parameter set includes visibility parameters, radiometric consistency parameters, reprojection error parameters, and local texture complexity parameters. Step 3: Perform weighted fusion processing on each Gaussian element to generate comprehensive observability index data; Step 4: Based on the comprehensive observability index data, perform multi-level structural clustering on all Gaussian units to obtain a hierarchical Gaussian structure set containing a salient layer, a transition layer, and a background layer. Step 5: Input the hierarchical Gaussian structure set into the cross-layer control module, perform densification processing on the salient layer based on the comprehensive observability distribution between each layer, perform continuity constraint processing on the transition layer, and update the Gaussian parameters of the background layer based on the error distribution between the salient layer and the transition layer. Step 6: Perform volumetric ray propagation and transparency blending on the Gaussian structure updated by the cross-layer control module to obtain the rendered image data corresponding to the target viewpoint.
[0006] Based on the above technical solution, the positive and beneficial effects of the present invention are as follows: 1. This invention constructs a Gaussian representation system containing multi-dimensional observability parameters, and then performs fusion processing and hierarchical clustering on this basis. This allows the geometric structure and radiation distribution of each region in the scene to be dynamically distinguished based on their observability differences under multiple views. Therefore, under sparse view input conditions, it can formulate differentiated structure densification strategies for the uneven observation characteristics of different regions, thereby maintaining structural continuity at the global level while compensating for reconstruction biases caused by the lack of geometric and texture information at the local level.
[0007] 2. This invention employs cross-layer collaborative control processing based on multi-dimensional parameter fusion. The optimization results between the salient layer and the transition layer are mutually constrained, ensuring that the spatial update of the Gaussian distribution is dependent on overall observation consistency rather than single-point error correction. This allows the Gaussian structure to maintain geometric consistency and radiative information stability during iterative convergence. This logical path avoids the structural fragmentation and texture blurring phenomena caused by existing strategies based on fixed thresholds or unidirectional densification in low-view coverage scenes. Furthermore, in the unified rendering process that blends volumetric ray travel and transparency, the hierarchically adaptively updated Gaussian structure can express complex geometry and multi-scale radiative information with lower redundancy, ensuring that the distribution of computational resources in the rendering process matches the scene information density. Attached Figure Description
[0008] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 This is a schematic diagram of the steps of the present invention; Figure 2 This is a schematic diagram of the architecture of the present invention; Figure 3 This is a schematic diagram illustrating the continuity constraint and sampling generation of the transition layer in this invention; Figure 4 This is a comparison chart of the rendering effects of the present invention and existing methods on the LLFF and MipNeRF360 datasets; Figure 5 This is a comparison diagram of the hierarchical and non-hierarchical Gaussian representations of the present invention; Figure 6 This is a comparative study of different densification strategies of the present invention. Detailed Implementation
[0009] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0010] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0011] Unless otherwise defined, all techniques and scientific methods used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The descriptions herein are for the purpose of illustrating particular embodiments only and are not intended to limit the invention. The terms "and / or" as used herein include any and all combinations of one or more of the associated listed items.
[0012] In one possible embodiment, this method is implemented on a workstation equipped with an NVIDIA RTX 4090 GPU and an Intel i9 processor, running on Ubuntu 22.04, Python 3.10, and the CUDA 12.2 parallel computing framework. The entire method is implemented within a separate rendering pipeline system, consisting of a data loading module, a 3D Gaussian initialization module, a multidimensional observability calculation module, a hierarchical clustering module, a cross-layer densification and manipulation module, and a volumetric ray-traveling rendering module. Data is exchanged between these modules through a unified cache management interface.
[0013] Please see Figure 2 The overall architecture diagram shown illustrates that the system first reads the input sparse multi-view image dataset in the data loading module. In this embodiment, five-view indoor scene images are used as input data. Each image has a corresponding camera intrinsic and extrinsic parameter matrix, which is stored in a JSON file and converted into a 4×4 homogeneous matrix representation after loading. The data loading module normalizes all images and depth data to a unified world coordinate system and passes them to the 3D Gaussian initialization module via a GPU memory buffer.
[0014] In the 3D Gaussian initialization module, the system employs multi-viewpoint initialization based on the sparse depth map and viewpoint correspondence. Figure 3 An initial sparse point cloud is generated through angle measurement. Each point in this point cloud is used as an initial Gaussian element center, and its covariance matrix is estimated from the distribution of local neighborhood points. The covariance is calculated using the PCA method, specifically by constructing the covariance matrix from each point's eight neighboring points and determining the three principal axis directions and their scales through eigenvalue decomposition. The system scales the principal axis lengths according to the local point cloud density to form an ellipsoidal Gaussian distribution. Each Gaussian element also records color information; the color values are obtained by multi-view color fusion of the input image and weighted averaging across the visible viewpoints through ray backprojection. The final initial 3D Gaussian set contains approximately 120,000 Gaussian elements, each including attributes such as position, covariance matrix, color vector, and visibility initialization value.
[0015] Next, multidimensional observability computation is implemented using CUDA parallel threads, with each thread responsible for updating the parameters of one Gaussian primitive. The system first constructs a unified projection cache structure in the GPU for fast computation of the projected visibility of each Gaussian primitive across multiple viewpoints. For each Gaussian primitive, in the sparse layer, the system calculates its visible area ratio in different camera projection planes to obtain the visibility parameter α. Then, in the same thread block, the radiometric consistency parameter β is calculated by comparing the standard deviation of the color channels under different viewpoints. Next, the reprojection error parameter is calculated in the intermediate layer, its value given by the root mean square of the Euclidean distance between the reprojection point of the Gaussian primitive at different viewpoints and the real image pixels. Finally, the local texture complexity parameter ζ is calculated. The system extracts a 3×3 pixel neighborhood within the image region corresponding to each Gaussian primitive, calculates the variance of the brightness gradient magnitude, and normalizes it to the [0,1] interval. These four parameters together constitute the multidimensional observability parameter set of the Gaussian primitive.
[0016] These parameters, after synchronization, are transmitted to the comprehensive observability fusion module, i.e., into the dense layer, according to the formula. Calculate the overall index, where each weight is initialized to 0.25 based on empirical values and can be adaptively adjusted during subsequent training. Overall Index It is stored in a video memory array and called in the next clustering and hierarchical step.
[0017] The multi-level clustering module is executed using GPU parallel computation. The system calculates the joint distance to neighboring primitives for each Gaussian primitive, taking into account both spatial proximity and observability differences. To reduce computational burden, the system employs an octree spatial index structure to partition Gaussian primitives according to spatial hierarchy. The clustering process starts from the median of the global observability distribution. Starting point, observability is higher than The Gausky units are divided into salient layers; then, the observable gradient changes between adjacent units are calculated, and when the gradient magnitude falls into the middle interval, the corresponding unit is marked as a transition layer; the remaining low-observable regions are marked as background layers. The system continuously calculates the overall variance during the clustering process. And dynamically update the threshold based on this value. This ensures the stable convergence of the hierarchical structure. The clustering results form three index tables in the GPU memory, pointing to the indices of primitives in the salient, transition, and background layers, respectively.
[0018] After clustering is completed, the cross-layer control module begins to perform densification and parameter optimization. This module consists of two sub-processes: Color-Depth Difference Driven Density Boosting (CDD) and Normal Orthogonal Plane Density Boosting (NOD). In the salient layer, the system calculates the color difference between the previous rendering result and the actual observed image. Differences in depth For those that simultaneously exceed the threshold The system back-projects the pixels onto three-dimensional space to obtain new sampling point locations. And generate a new Gaussian primitive at that position. The covariance matrix of the new primitive is obtained by... Calculation, where The rotation matrix is determined by constructing the local surface normals. , , The visibility of the new primitive is then adjusted linearly based on the difference amplitude. Depend on Calculate and truncate to [0,1]. Increase density at boundaries and in high-frequency regions to repair missing textures.
[0019] In the transition layer, the system performs local compaction based on a plane with orthogonal normal vectors. First, the normal vectors are extracted from the Gaussian covariance matrix. Two tangent vectors are obtained through Gram-Schmidt orthogonalization. The system is based on Within a local circular region centered at radius R, new Gaussian cell positions are generated by sampling in polar coordinates. Where r and θ are random variables. Newly generated primitives inherit the covariance and visibility of their parent primitives to ensure local geometric continuity and smooth transition. For the background layer, the system re-estimates the radiation intensity and visibility based on the error distribution of the salient layer and the transition layer, and adjusts the color and density parameters of the background layer primitives through global averaging to balance the overall energy distribution.
[0020] After densification, the system enters the volumetric ray travel and transparency blending rendering module. In this module, the system calculates the ray direction vector for each pixel based on the input camera viewpoint parameters and samples the ray at equal intervals in world space. Each sampling point searches for its neighboring Gaussian units based on its spatial coordinates, calculates the local density contribution value of that point, and infers the transparency decay rate based on density and visibility. The system uses a forward accumulation method to calculate transmittance and color overlay: as the ray travels from near to far, transparency decays cumulatively with depth, while color values are weighted and overlaid based on the current point's radiance intensity. To improve efficiency, the system sets an early termination threshold, stopping sampling when the cumulative transparency of the ray falls below 0.05. To further improve accuracy, the system dynamically reduces the sampling step size in areas with high visibility gradients (such as object edges or high-frequency texture areas), decreasing it from the default 0.005 units to 0.001 units, while increasing it to 0.01 units in flat areas, thereby improving the quality of detail reconstruction while ensuring real-time performance.
[0021] The entire rendering process is executed in parallel by CUDA kernel functions, with each ray corresponding to a thread. Threads share memory to cache local Gaussian meta-parameters to reduce memory access latency. After rendering, the system outputs an RGB image and a depth map, and calculates PSNR, SSIM, and LPIPS metrics by comparing them with real images for performance evaluation. Experimental results show that, under the same five-view input conditions, the method of this invention improves PSNR by approximately 2.5 dB, structural similarity index by 0.04, and achieves a rendering frame rate of 45 FPS, which can meet the application requirements of interactive view composition.
[0022] In terms of system architecture, each module achieves asynchronous scheduling through a unified CUDA stream. Data is stored in GPU memory as a structured array (SoA) to reduce cross-thread memory access conflicts. The 3D Gaussian set is indexed using an octree, reducing the neighborhood lookup complexity during rendering to O(log N). Dynamic updates to visibility parameters are performed periodically by a background thread to maintain convergence stability during long training or optimization phases. The entire system's training process can achieve global convergence in approximately 12 hours on a single GPU, and this time can be reduced to less than 3 hours using a multi-GPU distributed strategy.
[0023] To verify the effectiveness of this invention, it was evaluated on two datasets suitable for sparse view object reconstruction: LLFF and MipNeRF360. Please refer to [link / reference]. Figure 4 The rendering performance of this invention compared to existing methods on the LLFF and MipNeRF360 datasets is shown in the comparison chart. Experimental results demonstrate that OHDGaussian consistently achieves state-of-the-art results across all datasets, particularly excelling in the perceptual quality metric LPIPS. Notably, OHDGaussian performs exceptionally well even with only 5 views, improving LPIPS from 0.18 to 0.139 on the LLFF dataset compared to Mip-Splatting. The performance advantage of OHDGaussian becomes even more pronounced as the number of input views increases. With a 9-view configuration, OHDGaussian achieves a PSNR of 25.13 dB and an SSIM of 0.891 on the LLFF dataset, and a PSNR of 24.07 dB and an SSIM of 0.870 on the MipNeRF360 dataset, showcasing a significant quantization advantage over all baseline methods.
[0024] Further ablation experiments demonstrated the effectiveness of the hierarchical strategy and densification method of this invention. Results show that the adaptive hierarchical strategy significantly improves scene rendering quality, while removing this strategy leads to degraded results. The non-hierarchical model uses a globally uniform strategy to optimize all Gaussian primitives, failing to effectively distinguish changes in geometric importance and resulting in improper allocation of computational resources. In contrast, the adaptive hierarchical strategy of this invention, through a dynamic adjustment mechanism, concentrates newly generated Gaussian ellipsoids in geometrically important and high-frequency texture regions, effectively balancing the preservation of critical details and global consistency. This method improves scene rendering quality while reducing the number of Gaussian ellipsoids by 42%.
[0025] Similarly, ablation experiments also verified the effectiveness of the densification method of this invention. When only the color-depth difference method is used, high-frequency textures are enhanced but accompanied by obvious artifacts; when only the normal vector orthogonal plane method is used, although geometric consistency is effectively maintained and surface noise is reduced, the rendering results are slightly insufficient in detail, and texture details appear somewhat blurry. The integrated strategy combining these two methods overcomes the limitations of a single method, not only enhancing texture details but also effectively mitigating artifacts, showing a significant improvement across all metrics.
[0026] To facilitate a deeper understanding of the technology in this invention, a visibility-based layered densification Gaussian sputtering method disclosed in the embodiments of this application will be described in detail below. Please refer to [link to relevant documentation]. Figure 1 The schematic diagram shown illustrates the steps of the invention, which include: Step 1: Acquire sparse multi-view image data, construct an initial 3D Gaussian representation of the scene, and determine the spatial location, shape parameters, and radiative properties of each Gaussian element. Specifically, the system first acquires color image data and corresponding camera parameters from several viewpoints during the multi-view acquisition phase. The number of viewpoints is typically between 3 and 8, with fixed baselines between viewpoints, and the extrinsic parameter matrix is solved using a set of calibration plates. All images undergo uniform exposure and white balance correction to avoid deviations in subsequent color consistency calculations. It should be noted that "sparse multi-view" in this application refers to a type of input dataset where there is significant parallax between adjacent viewpoints, but the total number of viewpoints is insufficient to cover the complete 360° observation range. Its characteristic is that the same spatial point may be observed by only a small number of cameras.
[0027] After obtaining the input data, the system initiates the multi-baseline stereo matching module. Simultaneously considering the projection consistency between multiple camera pairs, a cost volume is constructed to measure the matching reliability of each pixel under different depth assumptions. Each voxel in the cost volume corresponds to a disparity assumption, the cost of which is calculated from the multi-view color residuals. To suppress noise, the system performs 3D Gaussian smoothing on the cost volume and employs a soft-argmin strategy to extract depth values, thus obtaining a depth confidence map for each pixel. This confidence value reflects the degree of uncertainty in depth estimation; a value closer to 1 indicates greater reliability, while a value closer to 0 indicates difficulty in matching the pixel across different views or the presence of occlusion.
[0028] Subsequently, the system performs multi-channel residual calculation using the depth confidence map and the color gradient information of the input image. Specifically, for each pixel, the system backprojects the pixel into 3D space based on its depth value to obtain the corresponding 3D point coordinates. Then, using camera extrinsic parameters, the 3D point is projected onto other viewpoints, and the difference between the pixel color intensity in the target viewpoint and the color intensity in the current viewpoint is compared. This difference is calculated not only for the color difference of the RGB channels but also for the geometric difference of the projection depth, thus obtaining a four-dimensional residual vector. Unlike traditional pixel errors, this residual vector can express the combined relationship between color and geometric offset in 3D space, used to describe inconsistencies between observations.
[0029] The system projects the residual vectors of all pixels back into 3D space, forming a discrete "projected residual field." To extract stable geometric structure information from this field, the system performs a least-squares-based Gaussian fitting in each local neighborhood. The fitting objective is to find a Gaussian function that minimizes the local residual variance, with its center representing the location in space where the error is minimized and stability is highest. Each fitting operation outputs the Gaussian center and a covariance matrix. The principal axis of the covariance matrix aligns with the principal direction of the error distribution, reflecting the spatial uncertainty distribution of that region. For example, in regions with large errors along the surface normal direction, the component of the covariance in that direction will be amplified to soften edges and occluded areas in subsequent rendering.
[0030] After fitting, the system solves for the radiative properties of each initial Gaussian primitive. The radiative property is not a simple color average, but a vector structure describing the energy distribution characteristics of the primitive. The system constructs a weighted optimization model on color observations from multiple perspectives, where the weights are determined by the depth confidence value. High-confidence perspectives have a greater impact on the results, while low-confidence perspectives are suppressed. The goal of the optimization model is to minimize the weighted sum of squares of the color brightness residuals. The solution is the radiative property vector of that Gaussian primitive, which contains three main components: reflectance component, color response component, and perspective dependency.
[0031] During the calculation of radiation properties, the system simultaneously estimates the initial visibility parameter for each Gaussian element. Unlike traditional visibility, the "visibility" in this application refers to the projection coverage ratio of the Gaussian element within the current camera's view frustum. This parameter is calculated from the overlap area of the Gaussian ellipsoid and the camera's view frustum. To simplify the calculation, the system uses a principal axis approximation model, projecting the Gaussian ellipsoid as a two-dimensional ellipse and calculating its coverage ratio on the image plane. This ratio varies between [0,1], with a larger value indicating that the Gaussian element is directly observed from more viewpoints.
[0032] To reduce the impact of outliers caused by noise, the system performs density regularization after generating the Gaussian element set. Density regularization controls the spatial distribution of Gaussian elements, ensuring a balanced number within local regions. If a region has an excessively dense Gaussian set, the system merges adjacent Gaussian elements based on the volume of the covariance matrix; if it is too sparse, new Gaussian points are added based on the gradient direction of the residual field. After regularization, the initial 3D Gaussian set obtained by the system is geometrically uniform and exhibits continuous radiation properties.
[0033] It should be noted that the "Gaussian unit" in this application is not a pixel in the traditional sense, but a parametric entity containing spatial geometric and radiative features. Essentially, it is a high-dimensional parametric point with a central location, covariance matrix, and radiative vector in three-dimensional space. Unlike point clouds or voxels, it can represent more continuous scene surfaces with a lower number of elements.
[0034] Step 2: Calculate a multidimensional observability parameter set based on the multi-view projection data of each Gaussian element in the initial three-dimensional Gaussian representation. The multidimensional observability parameter set includes visibility parameters, radiometric consistency parameters, reprojection error parameters, and local texture complexity parameters. In practical implementation, the system first aligns the sparse multi-view data with the spatial coordinates of the 3D Gaussian primitives using a unified coordinate mapping module. The 3D position of each Gaussian primitive... The covariance matrix has been obtained from the fitting in step 1. The spatial distribution range of the primitives is described. The system utilizes the intrinsic and extrinsic parameter matrices of each camera. Projecting the Gaussian element, we obtain its position in the th order. The corresponding two-dimensional pixel coordinates from each perspective These pixel locations constitute the set of observation projections of Gaussian elements from different viewpoints, serving as the fundamental input for subsequent calculations of observability metrics across various dimensions.
[0035] During the construction of the multidimensional observability parameter set, the system sequentially calculates four types of indicators: visibility parameters. The joint observability index after fusing radiation consistency parameters and reprojection error parameters and local texture complexity parameters The following section provides a detailed explanation based on the actual calculation logic.
[0036] When calculating visibility parameters, the system considers arbitrary Gaussian elements. Iterate through all its visible view sets. Extract the set of pixels corresponding to the primitive projection from each viewpoint. The camera ray direction vector for each viewpoint is denoted as... And the normal vector of the Gaussian element It can be derived from its covariance matrix and principal direction vector. The system calculates the angle between the two. This angle is used to measure the visibility tendency of the primitive at that viewpoint. Direction weighting coefficient. Given by the exponential decay function: ,in, This is the directional dispersion control coefficient, defined as positive real numbers, and its typical value range is... Radius; a larger value indicates a higher tolerance for viewing angles deviating from the normal direction. This value is used to reduce the impact of high-angle viewing angles on overall visibility estimation and ensure directional stability of visibility.
[0037] At each viewpoint, the system calculates the transmission probability based on the projection depth and transparency of the primitive in the depth buffer. The transmission probability represents the probability that light rays originating from the camera center and passing through this Gaussian element are blocked or penetrated, with a value ranging from [0,1]. The final visibility parameter is calculated using the following formula: In the formula, the numerator represents the weighted sum of transmission probabilities across all viewing angles, and the denominator is the direction-weighted normalization factor. Visibility parameter The value range is also [0,1]. When its value is close to 1, it means that the primitive can be directly observed from most viewpoints with almost no occlusion; conversely, a lower value means that the primitive is located in an occluded area or has poor projection stability. This visibility parameter is the one that most directly reflects the spatial occlusion relationship in the entire multidimensional parameter set. Its design purpose is to establish a directional estimate of the exposure degree of three-dimensional points in sparse view scenes through a statistical direction weighting mechanism.
[0038] In calculating the joint observability index At that time, the system adopts a fusion strategy based on two-dimensional consistency constraints of color and depth. For each Gaussian element... First, from various perspectives Extract the color information at its projection location. and depth value Color information is typically presented as an RGB three-channel vector, while depth values are expressed in camera coordinate units. The system calculates the color values at each viewpoint and their multi-view average. The Euclidean distance squared average yields the color consistency cost. ,in, The larger the value, the more drastic the color change of the primitive under different viewing angles, indicating the presence of radiometric inconsistency. Since color changes may be affected by differences in illumination, this embodiment performs histogram normalization preprocessing based on grayscale distribution on the images from each viewing angle in actual implementation to reduce the interference of external illumination changes.
[0039] Geometric consistency cost The calculation is used to measure the depth consistency of the primitive under different viewpoints. Specifically: in, This represents the average depth value across all viewpoints. By introducing a weighted denominator, the system maintains numerical stability in terms of error comparison across different depth scales. Next, by fusing the color consistency cost and the geometric consistency cost, a joint observability index can be obtained: in, These are the color and geometric consistency weight coefficients, respectively, both defined in the domain [0,1]. In this embodiment, if the illumination of the input image varies significantly, the sensitivity will be appropriately increased. To strengthen the stability weights of the geometric dimension; conversely, to increase . The value range is still [0,1], and the higher the value, the better the color and geometric stability of the Gaussian element across different viewing angles.
[0040] For local texture complexity parameters This embodiment characterizes local detail complexity by statistically analyzing the gradient direction distribution entropy of the primitive within the multi-view image projection region. Specifically, the system first establishes a local window (e.g., a 9×9 pixel region) around the Gaussian primitive projection center in each viewpoint image, and performs histogram statistics on the gradient directions within the window. The gradient directions are typically divided into N=8 or N=16 direction intervals, and the normalized probability of the primitive in the d-th direction is calculated. Then, the information entropy is calculated: in, The value range of is [0,1] and satisfies . When the texture direction distribution is more uniform, A higher value indicates more dramatic changes in surface texture and greater detail; conversely, if a certain direction accounts for a significant portion, the area is likely smooth or flat. In practical implementations, to reduce the impact of noise, the system performs Gaussian blur preprocessing (typically with a kernel size of 1.0 to 1.5 pixels) before calculating the gradient direction histogram to avoid statistical bias in direction caused by single-pixel noise.
[0041] In summary, the system forms a multidimensional parameter vector at each Gaussian level. ,in This is the radiation attribute vector of the primitive, primarily used for lighting estimation during subsequent rendering. All The data is stored in sorted order according to spatial indexes so that subsequent hierarchical clustering and densification modules can access it by region. The domains and relationships of these parameters are as follows: Indicates multi-directional visibility. Indicates multi-view stability, This represents the local texture complexity. These parameters are stored in a unified observability feature matrix, where row indices correspond to primitive numbers and columns correspond to parameter dimensions. This matrix will serve as the input for subsequent hierarchical Gaussian clustering and densification adjustments.
[0042] It should be noted that visibility is used to compensate for missing angular information under sparse multi-view conditions; the joint observability parameter in this application integrates the multi-view stability of the two physical dimensions of color and depth, and is a cross-modal fusion index; local texture complexity is a local detail measure directly bound to Gaussian meta-projection, and can be combined with the visibility parameter for densification weight allocation.
[0043] As one possible implementation, this embodiment employs a Gaussian primitive-level parallel computation approach to complete the aforementioned parameter calculation process. Under a GPU architecture, each thread is responsible for processing one Gaussian primitive, accessing its pixel data from different viewpoints through a unified memory structure. The calculation of orientation weighting, color consistency, depth error, and gradient entropy is performed using CUDA kernel functions, significantly improving computational efficiency. After parameter calculation, the system aggregates all results into a unified feature cache for subsequent steps. Through this method, the system can obtain a multidimensional observable parameter set for each Gaussian primitive based solely on sparse multi-view image data, without relying on external templates or complete geometric models. These parameters can fully describe the characteristics of each primitive in the three dimensions of visibility, color consistency, and texture complexity, providing a reliable quantitative basis for subsequent hierarchical Gaussian clustering and densification control. This significantly improves the rendering results under sparse views in terms of both structural integrity and visual consistency.
[0044] Step 3: Perform weighted fusion processing on each Gaussian unit to generate comprehensive observability index data. Specifically, after standardization, Step 3 determines adaptive weight coefficients by calculating the statistical characteristics between parameters. The principle is as follows: First, calculate the statistical variance of each parameter dimension. This is used to characterize the stability of the parameter in the global sample; the larger the variance, the higher the variability and the lower the reliability of the parameter. The system then calculates the Pearson correlation coefficient between each parameter component. Its expression is: ,in This represents the covariance function. By analyzing the correlation coefficient matrix, the degree of redundancy between different parameter dimensions can be identified. For example, when there is a strong correlation between color consistency and texture complexity, the system will appropriately reduce the weights of both to minimize the interference of redundant information on the fusion result.
[0045] Based on the above statistics, the system generates a set of weight coefficients. The rules for determining it are as follows: in, This represents the average correlation coefficient between this parameter dimension and other dimensions. It can be seen that the weights... Simultaneously, they are subject to the combined constraints of inverse variance and correlation penalty: when parameter noise is high or highly correlated with other dimensions, its weight is automatically reduced. All weights are then processed through normalization constraints to ensure... .
[0046] After the weight set is determined, the system performs a weighted fusion operation to obtain the initial comprehensive observability response: in, This represents the unregularized initial fusion result. Considering that the observation environments of different primitives in the local space may vary significantly, this embodiment introduces a regularization mechanism based on the inverse variance function to suppress the contribution of high-noise parameters. This regularization function is defined as: ,in, This is the regularization strength coefficient, which controls the penalty intensity; a typical value is 0.5 to 1.5. Indicated in Gaussian Yuan The local observability variance within the neighborhood reflects the degree of spatial consistency in that area. When the neighborhood variance is high, the system automatically reduces the fusion response at that point to prevent local noise from propagating to the global metrics.
[0047] It should be noted that the "variance inverse ratio function regularization" in this application differs from traditional linear smoothing. Its core is not to directly average the fusion results in space, but to adaptively adjust the amplitude of the fusion response of each primitive through the variance inverse ratio constraint, thereby stabilizing the distribution of observability indices without destroying local details.
[0048] To further ensure spatial continuity during the fusion process, this embodiment also implements dynamic updates to the weight coefficient set for local spatial consistency. Specifically, in each optimization iteration, the system updates the weight set of adjacent Gaussian elements. Performing a smoothing operation based on Euclidean distance ensures that the weight changes between adjacent primitives remain continuous, thus preventing false boundaries with observable abrupt changes. This process is typically implemented using a 3D mesh topology or adjacency list as the index structure and can be efficiently executed on large-scale primitive sets through GPU parallel computation.
[0049] Finally, the system integrates all Gaussian elements' observability indices. It is stored in a structured feature table and used as one of the input feature dimensions for hierarchical clustering in subsequent steps. Mathematically, this metric combines orientation sensitivity and noise suppression, and physically reflects the cross-view stability, illumination consistency, and local spatial reliability of Gaussian elements.
[0050] It should be further explained that the "weighted fusion processing" in this application is not a simple averaging of multidimensional parameters, but rather an adjustment of the importance of feature dimensions through a dynamic adaptive statistical modeling mechanism, so that the fusion result has interpretability and adjustability; the "comprehensive observability index data" is an important input parameter for subsequent structural clustering and densification control, which can directly affect the convergence characteristics and reconstruction accuracy of Gaussian hierarchical partitioning.
[0051] Step 4: Perform multi-level structural clustering on all Gaussian units to obtain a hierarchical Gaussian structure set containing salient, transition, and background layers; please refer to [link / reference]. Figure 3 The hierarchical structure diagram shown illustrates that, specifically, the entire process is based on the comprehensive observability index of Gaussian elements. Its three-dimensional spatial coordinate vector Using this as input, a joint distance metric function is constructed to simultaneously measure the differences between two primitives in terms of observability and spatial location. It should be noted that the "joint distance function" in this application differs from common spatial clustering distances that primarily consider geometric location. It can simultaneously reflect the differences in observability reliability and spatial geometric relationships of primitives, thus more accurately characterizing the hierarchical features of scene structure in the observability dimension. This distance function is defined as: in, Weights for observability differences, Let be the spatial adjacency weights, both of which are non-negative real numbers, satisfying . In one possible implementation, The value can be set from 0.4 to 0.7, and should be dynamically adjusted based on the proportion of observation noise in the scene. When the number of views is small and lighting changes are significant, A larger value can be chosen to increase the weight of the attribute layer; however, in geometrically dense scenes, This can relatively improve the spatial continuity and make it more prominent.
[0052] The clustering process consists of three phases: initialization, intermediate stages, and termination. In the initial stage, the system calculates the mean and standard deviation of all Gaussian unit indices based on the distribution of comprehensive observability indicators, and uses this to determine the adaptive threshold. Generally, when the index value If the primitive is considered to belong to a salient region, it is classified into the primary salient cluster; otherwise, it is classified into the primary background cluster. Here, "salient" does not refer to the salientity of visual brightness, but rather to the region in which the Gaussian primitive exhibits high observation reliability and color geometric consistency under multi-view conditions.
[0053] In the intermediate stage, to avoid abrupt discontinuities between the salient layer and the background layer, this embodiment performs gradient analysis on the boundary region. The system calculates the observable gradient between each Gaussian primitive and its neighboring primitives. The gradient is defined as the average magnitude of the differences between the primitive and its neighboring indices. For a gradient magnitude to fall within the interval... The Gaussian set is systematically defined as a "transition layer". It should be noted that the "transition layer" in this application is different from the salient layer and the background layer. Its function is to connect the former two and avoid structural breaks between layers. It is particularly suitable for scenes with occlusion boundaries or abrupt depth changes.
[0054] When clustering reaches the termination phase, the system will use the global variance of the current iteration. For threshold set Dynamic updates will be performed. The update rules are as follows: in, The adaptive learning rate is used to control the speed of threshold adjustment, and is generally set to a certain value. A preset variance reference value is used to maintain cluster stability. If the current variance is greater than the reference value, it indicates that the clustering stratification is too discrete, and the system will increase the threshold accordingly to reduce excessive separation between strata; conversely, it will decrease the threshold to make the stratification more refined. Through this adaptive mechanism, the system can automatically adjust the clustering granularity according to different scenario complexities, achieving a balance between global and local clustering.
[0055] In the implementation of this invention, to verify the effectiveness of the hierarchical Gaussian structure under view sparsity conditions, a comparative experiment was conducted between the hierarchical clustering strategy (i.e., the salient layer, transition layer, and background layer structure proposed in this invention) and the traditional non-hierarchical densification method. Please refer to [link / reference]. Figure 5 The comparison diagram shown illustrates the difference between hierarchical and non-hierarchical Gaussian representations. The non-hierarchical method, under the same sparse viewpoint input, generates a uniformly diffused Gaussian distribution, with a near-average density distribution between salient and background regions. This results in blurred foreground contours, unnatural transitions in boundary transparency, and noticeable ghosting, particularly in reflective or semi-transparent areas. In contrast, the hierarchical structure of this invention automatically creates primary and secondary layers in spatial distribution. The salient layer concentrates Gaussian elements on the main visible target surface, the transition layer forms density buffers in contour and transparent transition areas, and the background layer carries low-frequency environmental information. This layered organization makes the visibility accumulation during rendering more consistent with realistic optical occlusion characteristics, thereby achieving higher edge sharpness and lower global error with the same sampling rate.
[0056] In practical implementation, to improve clustering efficiency, the system generally adopts a neighborhood retrieval method based on a spatial index structure. For example, Gaussian elements can be used to build a KD-tree or octree structure according to their three-dimensional coordinates, so as to quickly query the neighborhood element set when calculating the joint distance, thereby significantly reducing the global indexing efficiency. Complexity.
[0057] It should be further explained that in the "hierarchical Gaussian structure set" of this application, the salient layer contains Gaussian primitives with high reliability, which typically correspond to the main structural regions in the scene; the transition layer contains boundary regions with gentle observable gradients; and the background layer is mainly composed of primitives of low observability, high occlusion, or distant background regions. This hierarchical structure not only guides subsequent densification strategies but is also used in the rendering stage to dynamically adjust the weight distribution of Gaussian sputtering, achieving a visually smoother interlayer transition.
[0058] Step 5: Input the hierarchical Gaussian structure set into the cross-layer control module, perform densification processing on the salient layer based on the comprehensive observability distribution between each layer, perform continuity constraint processing on the transition layer, and update the Gaussian parameters of the background layer based on the error distribution between the salient layer and the transition layer. Specifically, for the densification of salient layers, the system first analyzes the difference between the rendered result and the actual observed image pixel by pixel in screen space to determine candidate regions that need densification. That is, for each pixel in the rendered view image and the actual observed image... The system reads the rendered color value and the actual observed color value of the pixel, respectively. Both sets of values are RGB three-channel vectors, denoted as... At the same time, read the depth value obtained from the rendering. With true depth All of this data can be obtained directly from the output buffer or depth map cache of the rendering pipeline. It should be noted that the "rendering depth" in this application is different from the grayscale depth of a normal image. It is the average depth estimate obtained by the Gaussian sputtering model after the accumulation of volumetric rays in the view frustum, which represents the geometric center position of the pixel along the ray direction.
[0059] The system first calculates the color difference. This value reflects the degree of deviation between the rendered image and the real image in terms of radiometric consistency, and its calculation formula can be expressed as a three-channel weighted sum of squares: in, These are color channel weighting coefficients, typically set based on human visual perception characteristics or sensor sensitivity. For example, they can be... This weighting reduces the bias caused by non-uniform illumination, making densification judgment more stable.
[0060] Then the depth difference was calculated. This value describes the magnitude of the error in geometric reconstruction for that pixel. If both conditions are met... Exceeding the first threshold ,and Exceeding the second threshold If the condition is not met, it is determined that the spatial point corresponding to the pixel has texture or geometric defects in the salient layer, and densification needs to be performed. The two thresholds can be dynamically adjusted according to the noise level of the current scene, and are generally set to a certain value. (Normalized color space) (in units of camera normalized depth).
[0061] Pixels identified as candidates for densification will have their corresponding 3D spatial positions obtained through backprojection. The system uses the camera intrinsic matrix K and extrinsic matrix of this pixel. Calculate the back-projected coordinates: This operation restores the two-dimensional pixel to its three-dimensional position in the camera coordinate system, serving as the spatial center of the new Gaussian unit. This process is typically executed in parallel via CUDA to support large-scale densification point generation.
[0062] After obtaining the 3D coordinates of the candidate points, the system estimates the local surface normal vector based on the covariance matrix of adjacent Gaussian elements. In one possible implementation, the eigenvector corresponding to the minimum eigenvalue of the local neighborhood point set can be extracted using PCA or SVD methods and used as the normal vector. This normal vector is then used to construct the rotation matrix. , representing the principal axis directions of the newly generated Gaussian ellipsoid. Based on the magnitude of the color and depth differences, the system uses a weighted linear combination to determine the scale coefficients of the Gaussian ellipsoid along the three principal axes. The scale calculation formula is as follows: in, This represents the reference scale (i.e., the average semi-axis length of the original Gaussian cells in this region). These are the color and depth response weights, used to adjust the intensity of the effects of various differences on scale changes; their values typically range from 0 to 1. Larger values... A larger value makes the model more sensitive to depth errors, thus generating more primitives in geometrically misaligned regions; while a larger value... The value helps to enhance the high-frequency detail at texture edges. This scale vector With rotation matrix Together they form the covariance matrix of the new Gaussian element: here The shape and orientation of the Gaussian ellipsoid in three-dimensional space are determined by this parameter, which is a core parameter of the sputtering distribution function during rendering. By dynamically generating new Gaussian primitives in this manner within salient layers, adaptive densification can be achieved while maintaining local geometric consistency, effectively enhancing object boundaries and texture details.
[0063] Meanwhile, to ensure that the newly generated Gaussian elements can correctly participate in transmission and occlusion calculations during subsequent ray propagation, the system assigns them visibility parameters. The optical visibility of this primitive is described by the following calculation relationship: in, These are the visibility weighting coefficients corresponding to color and depth differences, reflecting the contributions of color and geometric differences to the probability of light transmission, respectively. Function This is used to limit the results to the interval [0,1] to avoid oversaturation values caused by abnormal errors. Larger values... This indicates that the primitive is more prominent in the viewing direction, meaning it should have a higher weight in rendering composition.
[0064] Through the aforementioned significant layer densification process, the system adaptively generates multiple new Gaussian primitive groups in three-dimensional space. The spatial distribution and morphology of these primitives are jointly constrained by the original Gaussian structure and multimodal differences, thereby forming a high-density, structurally continuous Gaussian distribution field in the visually salient region.
[0065] For the continuity constraint handling of the transition layer, the system's goal is not to increase density, but to maintain a smooth transition of the geometry between the salient and background layers. See also... Figure 3 The diagram illustrates the continuity constraints of the transition layer and the sampled generation structure. In the figure, the ellipsoid at the center represents the parent Gaussian element, with its center point being... The principal axis direction of this primitive is determined by the eigenvector corresponding to the smallest eigenvalue of the covariance matrix, denoted as the local normal vector n. In the plane perpendicular to this normal vector direction, two mutually perpendicular tangent vectors are constructed using the Gram-Schmidt orthogonalization method. and This forms a locally orthogonal coordinate system. Radial sampling is performed within this plane, centered on the target plane.
[0066] As shown in the figure, multiple small dots represent new Gaussian sub-basic units generated according to the above formula. These sub-basic units are distributed within the tangent plane neighborhood of the parent basic unit and maintain a certain geometric continuity along the principal axis. The area encircled by the dashed line represents sampling points that are too close to the Euclidean distance of adjacent Gaussian basic units. These points are suppressed during the generation process to avoid structural overlap caused by excessive local density. The newly generated Gaussian sub-basic units inherit the visibility parameters and principal axis direction of the parent basic unit and adjust the axial proportion of the covariance matrix according to the local surface curvature, achieving a smooth geometric transition from the salient layer to the background layer.
[0067] Specifically, firstly, the covariance matrix of each transition layer Gaussian element... Perform eigenvalue decomposition to extract the eigenvector corresponding to the smallest eigenvalue. This vector represents the normal direction of the local surface. To construct a local sampling basis on this normal plane, the Gram–Schmidt orthogonalization method is employed, based on... Construct two mutually perpendicular tangent vectors Subsequently, random radial sampling is performed within this local orthogonal plane to generate compensation points for the local structure. The sampling point locations are calculated using the following relationship: in, The central coordinates of the father Gorsky unit, This represents the radial distance from the center to the sampling point. The sampling angle is defined in the interval. , Radial distance It is not a uniform distribution, but rather follows a density decay function. , where parameters This is called the sampling attenuation coefficient, used to control the concentration of sampling points. When When the value is large, the sampling points are more concentrated at the center of the primitive, which is suitable for fine texture areas; when When the sample size is smaller, the sampling distribution is more uniform, which is suitable for the smooth expansion of the transition layer.
[0068] The newly generated Gaussian primitives inherit the visibility parameters and principal axis directions of the covariance matrix from their parent primitives. Simultaneously, the system adjusts the principal axis length ratio of the covariance matrix based on local curvature characteristics, thereby creating a continuous, dense surface in the 3D structure. To prevent redundant overlap in the sampling results, the system calculates the Euclidean distance between each new primitive and existing primitives in its neighborhood after generation; if the distance is less than a preset overlap threshold... When writing to this primitive, write operations are suppressed to avoid redundant densification between structures.
[0069] To enable the lighting and geometric distribution of the background layer to adaptively respond to the local reconstruction deviations of the salient and transition layers, thereby forming an overall distribution with continuous energy and consistent depth during multi-layer rendering, the rendering residuals in the salient and transition layers are statistically aggregated and fed back to the Gaussian meta-parameters of the background layer. This allows the background layer to automatically adjust its visibility and covariance matrix structure in subsequent iterations, compensating for geometric mismatches and photometric deviations between multi-layer structures.
[0070] Specifically, the system first targets each Gaussian unit in the salient layer and the transition layer. Calculate the residual values in the color and depth dimensions. Here, "residual" refers to the magnitude of the difference between the pixel values generated after the primitive participates in rendering and the actual observed image within the same viewport. Color residual. It can be calculated at the pixel level and is usually expressed as a weighted average of the three-channel color errors; depth residual This represents the geometric distance error between the primitive element and the actual surface along the depth projection direction. The residual can be calculated using the formula: The definitions of each symbol are consistent with the aforementioned salient layer densification process. After the calculation is completed, the system stores the residual information of each primitive as a composite structure containing color error, depth error, and spatial coordinates, forming the so-called "Gaussian error dataset".
[0071] Subsequently, the system aggregates these residual information according to their three-dimensional spatial locations to construct a comprehensive error matrix for the salient and transition layers. This error matrix is a statistical distribution structure in three-dimensional space, where each cell corresponds to a local voxel region, recording the average residual magnitude and variance of all salient and transition layer primitives within that region. This spatial mapping allows for a direct description of the reconstruction bias intensity in different regions.
[0072] After obtaining the comprehensive error matrix, the system then processes the update of the background layer. For each Gaussian element in the background layer... The system first spatially retrieves the set of salient and transitional layer primitives within its neighborhood, denoted as . This neighborhood retrieval is implemented based on Euclidean distance or a spatial octree structure, and is used to quickly find neighborhoods within a certain radius. The neighborhood radius is defined as all Gaussians adjacent to the background primitive within the specified range. The radius can be dynamically adjusted based on the average spacing of the background layer, typically 1.5 to 2 times the average spacing between adjacent primitives.
[0073] For each background layer primitive Count all neighboring primitives in its neighborhood. Color Residual With depth residual And based on these values, a weighted average error response is calculated. The weights consist of two coefficients. This joint control balances the contribution of color and depth errors to feedback updates. Generally, if there are significant lighting variations or reflective surfaces in the scene, the control should be increased. The value should be increased if the scene is mainly characterized by geometric errors; if the scene is characterized by geometric errors, then the value should be increased. .
[0074] After obtaining the average error response, the system employs a dual-channel iterative update strategy to update the visibility parameters of the background layer high-level pixels. With covariance matrix Synchronized adjustments. The specific update relationships are as follows: in, Let represent the visibility parameter and covariance matrix of the j-th primitive in the background layer at the t-th iteration, respectively; These are the update rates for visibility and covariance, respectively. Let be a neighborhood set, and the number of its elements is . This is the weighting factor for the color and depth residuals; This represents the spatial positional deviation vector between adjacent primitives. The core idea of this update mechanism is to use the residual information of adjacent layers as a spatial gradient signal to backpropagate the deviation trend between the salient layer and the transition layer to the background layer, so that the statistical morphology of the background layer gradually adapts to the correction behavior of the first two layers over time.
[0075] From a physical perspective, the first equation controls the change in the "optical saliency" of the background primitives. When the average residual within the neighborhood is large, The weighted result increases, making This means that the background primitive is given a higher visibility weight in rendering to enhance the luminous flux compensation in areas occluded by the foreground or with high error; conversely, when the residual is small, It will decrease slowly, reducing its contribution during rendering and compositing, thus maintaining global energy balance. Parameters This determines the update speed, which is generally between 0.05 and 0.2, to ensure that the system gradually converges after several iterations rather than oscillating.
[0076] The second equation controls the adaptive adjustment of the covariance matrix. (Term) The variance matrix represents the spatial direction, while the preceding weighting terms... This is equivalent to the error intensity factor. Therefore, when the error is concentrated in a certain direction in the neighborhood, The corresponding principal axis direction will increase, thus "stretching" the spatial shape of the background element to compensate for the structural gap in the foreground layer; conversely, when the local error is isotropic, The updates tend to be more even, allowing the background layer to remain smooth in this area. Update rate The typical value is 0.01 to 0.1, and its magnitude determines the sensitivity of covariance adjustment.
[0077] It should be noted that the "visibility parameter" in this application not only reflects the transmission ratio of the element in optical imaging, but also implicitly implies the element's modulating effect on the global light field energy distribution. Its numerical range is [0,1]. A value close to 1 indicates a strong optical contribution from the region, requiring more detail to be preserved; a value close to 0 indicates a weak illumination contribution, which can be suppressed during rendering blending. Similarly, the "covariance matrix" here represents not only a shape parameter but also the volumetric distribution characteristics of the Gaussian element in 3D voxel space, with its principal axis squared values... It directly corresponds to the spatial scale, and its domain is a positive definite matrix space.
[0078] In the specific implementation, to avoid numerical drift, the system performs positive definite constraints and normalization operations on the covariance matrix after each round of updates, namely: This operation ensures that the updated covariance matrix retains its original scale energy, only adjusting its directional distribution characteristics.
[0079] Through the aforementioned multi-layered linked updates based on error feedback, the Gaussian distribution of the background layer can adaptively absorb error information from the salient and transition layers, thereby achieving cross-layer illumination energy compensation and depth field smoothing in subsequent rendering. As a result, when the densification operation of the salient layer causes local expansion of the foreground structure, the visibility of the background layer is automatically increased to maintain overall brightness; when the curvature constraint of the transition layer causes excessive smoothing in a certain area, the covariance of the background layer automatically shrinks to restore a distinct sense of volume. After multiple iterations, the system forms a dynamic equilibrium state based on error drive among the salient, transition, and background layers, significantly improving the accuracy and rendering stability of sparse view reconstruction based on Gaussian sputtering.
[0080] To verify the combined effect of the salient layer densification, transition layer continuity constraint, and background layer error feedback mechanism in step 5, simulation experiments were conducted on the method of this invention. The experimental environment was a dual-GPU platform (RTX4090 ×2). The publicly available 3D sparse multi-view datasets Family and Truck were selected as test scenes. The number of input views for each scene was controlled to be between 10 and 12 to ensure the stability of the algorithm under sparse conditions. All initial Gaussian elements were approximately 400,000. The color threshold for salient layer densification was set to 0.08, the depth threshold to 0.01 meters, and the transition layer sampling attenuation coefficient to 3.5. The experimental data are as follows: Dataset Method Configuration PSNR↑ (dB) SSIM↑ Average depth error ↓ (cm) Rendering time (ms / frame) Family Baseline method (un-densification) 29.8 0.931 2.73 18.4 Family Significant layer compaction 31.2 0.944 2.34 19.1 Family Compactness + Continuity Constraints 32.0 0.950 2.18 19.8 Family All mechanisms (including error feedback) 32.7 0.954 2.07 20.2 Truck Baseline method (un-densification) 30.1 0.927 3.12 16.9 Truck Significant layer compaction 31.5 0.938 2.79 17.3 Truck Compactness + Continuity Constraints 32.1 0.943 2.52 17.9 Truck All mechanisms (including error feedback) 32.4 0.947 2.41 18.0 As can be seen from the table, after introducing densification processing based on color and depth residuals only in the salient layer, the average PSNR of the rendering result improved by about 1.4 to 1.6 dB, indicating that this step has a significant effect on restoring local high-frequency textures and reducing boundary blur. When the continuity constraint of the transition layer is further superimposed, the structural discontinuity between the salient layer and the background layer is significantly reduced, and the depth error is reduced by about 0.2 to 0.3 cm, indicating that the local smooth sampling of the transition layer can effectively eliminate the geometric jumps caused by structural discontinuities.
[0081] During the densification process of the salient layer, the newly added Gaussian elements are mainly concentrated in regions with significant differences in illumination height and abrupt changes in depth, resulting in more continuous local details. A comparison of the salient layer rendering and the actual depth map shows that the method of this invention significantly improves the reconstruction effect in high-frequency geometric regions such as eaves and railings. The distribution of Gaussian ellipsoids at the edges is consistent with the surface normal vectors, eliminating the edge bright spots and voids commonly seen in traditional sputtering methods. The continuity constraint processing of the transition layer plays a crucial role in the structural transition of the densified region. After using radial attenuation sampling, the Gaussian element distribution density smoothly decreases with distance, forming a natural transition layer thickness and avoiding geometric discontinuities between the salient layer and the background layer.
[0082] To further verify the improvement effect of this invention on the densification strategy of salient and transition layers, experimental results of four different densification strategies were compared: Base (basic Gaussian reconstruction, without densification optimization), w / o NOD (densification without normal constraints), w / o CDD (densification without color depth dual difference control), and Full (this scheme). The experiments were conducted under the same input sparse viewpoint and the same hardware conditions, with a rendering resolution of 800×800. See the results below. Figure 6The Base scheme performs poorly in terms of local geometric details and depth consistency, with noticeable light leakage in boundary areas. The lack of normal consistency in salient layers (w / o NOD) leads to ripple artifacts in surface transitions. While w / o CDD maintains the basic shape, it is prone to color drift under complex lighting conditions. In contrast, the Full scheme, through the combined effect of salient layer densification and transition layer continuity constraints, achieves the smoothest and most realistic geometric structure, with an average residual error reduction of 23.4% and a PSNR improvement of 2.1 dB. The rendering results are significantly superior to other schemes in terms of visual continuity and depth stability.
[0083] Visual observations show that the method of this invention has achieved significant improvements in detail restoration, geometric edge continuity, and lighting consistency. In particular, under the synergistic effect of significant layer densification and transition layer, the texture sharpness in high-frequency areas is higher. Background layer error feedback further corrects local exposure unevenness and dark area collapse, ultimately presenting a more natural and stable sparse view reconstruction effect.
[0084] Step 6: Perform volumetric ray propagation and transparency blending on the Gaussian structure updated by the cross-layer control module to obtain the rendered image data corresponding to the target viewpoint. Volumetric ray propagation is not based on density accumulation using a regular voxel grid, but rather on analytical estimation of spatial density based on a continuously distributed Gaussian function, thus achieving smooth volume integration in a non-uniform and sparse Gaussian element field. Simultaneously, transparency blending refers to the dynamic accumulation of transmittance at each sampling point along the ray propagation direction, allowing the radiation contributions from different depths to be naturally superimposed according to physically defined occlusion relationships.
[0085] Specifically, during the rendering process, firstly, for each pixel of the target viewpoint, based on the extrinsic parameter matrix corresponding to that viewpoint... With intrinsic parameter matrix Calculate the direction vector of the light ray. ,in For the two-dimensional coordinates of pixels, This is a unit vector representing the direction of light rays in 3D space. Starting from the origin of the camera coordinate system, this vector extends along the direction from the near clipping plane... Zhiyuan cutting surface Discrete sampling is performed. As one possible implementation, the initial sampling interval can be set to... ,in The preset number of sampling steps is usually 256 or 512 to ensure spatial resolution.
[0086] At each step of volume sampling, the spatial location of the sampling point can be represented as: ,in Let be the camera center position, and s be the current sampling depth. For this sampling point, this invention does not simply look up its voxel, but rather searches for and matches it with the voxel based on the distribution information of Gaussian pixels in three-dimensional space. Several recent Gaussian set Each Gaussian unit From the central position Covariance matrix Visibility parameters and radiation color vector constitute.
[0087] After obtaining the adjacent Gaussian set, it is necessary to calculate the density influence value of the sampling point in the local Gaussian field. This value reflects the combined effect of the contributions of each Gaussian element on the sampling point. Its calculation can be expressed in a weighted Gaussian function form as follows: ,in This represents the visibility weight of the primitive, used to adjust the transmission effect under different directions; The covariance matrix is the inverse of the Gaussian matrix, determining the shape and directional sensitivity of the Gaussian diffusion in space. It's worth noting that when the principal axis of the covariance matrix is parallel to the direction of the light rays, its contribution to the density value is greater, resulting in a realistic transparent gradient effect in the view direction during rendering.
[0088] Subsequently, based on the calculated density influence value, the volume transmittance of the sampling point was determined. Transmittance describes the proportion of energy remaining after light passes through a point, and is calculated using the following formula: ,in This is the absorption coefficient constant, used to control the intensity of light attenuation. The sampling interval is [specified]. By sequentially accumulating the transmittance along the light direction, the depth at which the transmittance is obtained can be calculated. Cumulative transparency at the location .
[0089] Calculate the radiation contribution value at each sampling location. This color is then mixed with the cumulative color of the preceding sample points. The final pixel color output can be expressed in recursive form as follows: ,in This represents the cumulative transparency at the previous sampling position. This formula reflects the dynamic trade-off between the layer-by-layer accumulation of color and transparency as light passes through Gaussian units at different depths.
[0090] To improve computational efficiency, this invention also introduces an adaptive sampling strategy based on visibility gradient. The "visibility gradient" refers to the rate of change of visibility between adjacent sampling points along the ray direction, denoted as... .when Exceeding the set threshold When this occurs, it indicates that there is a significant change in occlusion or a sudden change in transparency in the area, therefore the sampling interval needs to be reduced. ),in This is the sampling step size adjustment coefficient; conversely, when Below the threshold In such cases, the sampling step size can be appropriately increased to reduce redundant calculations, thereby achieving dynamic allocation of computing resources. This mechanism enables rendering to maintain high precision in edges and semi-transparent areas, while automatically accelerating the sampling step speed in smooth areas, achieving a balance between quality and efficiency.
[0091] During the actual rendering process, the system continuously monitors the cumulative transparency along the path of each ray. Time (e.g.) If the light energy is nearly exhausted, the calculation is terminated early to avoid unnecessary redundant sampling. The final accumulated color values will then be used. Output to the corresponding pixels to form the rendered image of the target viewpoint.
[0092] Finally, it should be noted that the mathematical formulas, derivations, symbol definitions, and parameter calculation methods used in this specification are all for the purpose of further clarifying and verifying the technical content of this invention, so that those skilled in the art can more intuitively and accurately understand the working mechanism and technical effects of this invention. These formulas are only used as quantitative expressions or illustrative examples of technical features and do not constitute limiting conditions of the claims of this invention. Those skilled in the art should understand that, without changing the core idea of this invention, the parameter forms, calculation methods, numerical ranges, and even symbol representations involved in the formulas can be equivalently replaced or simplified in engineering according to the actual application environment. The specifics can be determined according to the actual situation, and no limitation is imposed. It should also be emphasized that the formulas in this specification are not theoretical derivations in the style of academic research papers, but rather an engineering description of the embodiments of this invention. Their purpose is to enhance the understandability and implementability of this invention, rather than to increase redundancy and complexity. Those skilled in the art can choose whether to use such quantitative tools when reading this specification, or can achieve the same technical effects through other equivalent methods.
[0093] Furthermore, while specific embodiments of the present invention have been described above, those skilled in the art should understand that these specific embodiments are merely illustrative. Those skilled in the art can omit, substitute, and modify the details of the above methods and systems in various ways without departing from the principles and essence of the present invention. For example, combining the above method steps to perform substantially the same function and achieve substantially the same result according to substantially the same method falls within the scope of the present invention. Therefore, the scope of the present invention is defined only by the appended claims.
Claims
1. A visibility-based layered densification Gaussian sputtering method; characterized in that: include: Step 1: Obtain sparse multi-view image data, construct the initial 3D Gaussian representation of the scene, and determine the spatial location, shape parameters, and radiation properties of each Gaussian element; Step 2: Calculate a multidimensional observability parameter set based on the multi-view projection data of each Gaussian element in the initial three-dimensional Gaussian representation. The multidimensional observability parameter set includes visibility parameters, radiometric consistency parameters, reprojection error parameters, and local texture complexity parameters. Step 3: Perform weighted fusion processing on each Gaussian element to generate comprehensive observability index data; Step 4: Based on the comprehensive observability index data, perform multi-level structural clustering on all Gaussian units to obtain a hierarchical Gaussian structure set containing a salient layer, a transition layer, and a background layer. Step 5: Input the hierarchical Gaussian structure set into the cross-layer control module, perform densification processing on the salient layer based on the comprehensive observability distribution between each layer, perform continuity constraint processing on the transition layer, and update the Gaussian parameters of the background layer based on the error distribution between the salient layer and the transition layer. Step 6: Perform volumetric ray propagation and transparency blending on the Gaussian structure updated by the cross-layer control module to obtain the rendered image data corresponding to the target viewpoint.
2. The visibility-based layered densification Gaussian sputtering method according to claim 1, characterized in that: The working process of step 1 includes: Based on the projection consistency between adjacent viewpoints in sparse multi-view image data, multi-baseline stereo matching is used to obtain the initial depth confidence map corresponding to the scene. Based on the initial depth confidence map and the color gradient map of each view, multi-channel residual estimation is performed to obtain the three-dimensional reprojection error vector corresponding to each pixel. The projection residual field is constructed based on the spatial distribution of the reprojection error, and a three-dimensional Gaussian fitting based on local least squares is performed in the residual field to obtain the center position and covariance matrix of the initial Gaussian elements. By using color gradient consistency constraints and depth confidence weighting constraints, the radiative attribute vectors of each Gaussian element are solved to form an initial 3D Gaussian representation dataset that incorporates observability information.
3. The visibility-based layered densification Gaussian sputtering method according to claim 1, characterized in that: The calculation of the multidimensional observability parameter set includes: For each Gaussian pixel, a direction-weighted visibility function is constructed based on the view distribution of the corresponding pixel in the multi-view projection data. ,in These represent the azimuth and pitch angles in the direction of the view, respectively; visibility parameters. The calculation formula is: in, For the number of views, For the first The directional weight of light rays in the direction of view. The transmission probability of the Gaussian element projection point in the corresponding direction; the ray directionality weight Angle between the viewing angle and the surface normal Confirmed, satisfied σ is the directional dispersion coefficient; The joint observability parameter is calculated by a joint error evaluation method. The joint error evaluation method calculates the radiometric consistency parameter by the consistency of the color response and brightness distribution of the corresponding pixels under each viewpoint, and calculates the geometric reprojection error parameter based on the difference in the projection coordinates of the Gaussian element under different viewpoints. The joint observability parameter is used to describe the cross-viewpoint stability of the Gaussian element in color and geometric dimensions. The pixel region corresponding to each Gaussian pixel in the multi-view projection image is divided into several local windows; gradient direction statistics are performed on each local window to obtain the normalized probability distribution of each direction. Calculate the entropy value of the probability distribution in each direction, and use it as the local texture complexity parameter of the Gaussian element. The formula expression is: in, Indicates the first Local texture complexity parameters of Gaussian units; This indicates that the primitive is in the first... The normalized probability of grayscale changes in each gradient direction; This indicates the total number of directional statistics.
4. The visibility-based layered densification Gaussian sputtering method according to claim 3, characterized in that: The calculation process of the joint error evaluation method includes: S1. For each Gaussian element, extract the color information at the projection position of that Gaussian element from the multi-view image. With depth information ,in For view index; S2. Based on all views, using Gaussian pixel color and multi-view average color. Calculating the difference mean to calculate the cost of color consistency The expression is: in, Number of observable views; For the first View in position Pixel color vector; S3, Based on multi-view depth values and their average depth The relative error mean is used to calculate the geometric consistency cost. The expression is: in, These represent the depth value and the mean depth, respectively; ε is the numerical stability constant. S4. Combine the above two costs to obtain a joint observability index. The calculation relationship is as follows: in These are the weights for color consistency and geometric consistency, respectively.
5. The visibility-based layered densification Gaussian sputtering method according to claim 1, characterized in that: Step 3, the process of weighted fusion of each Gaussian element, includes: For each Gaussian element, obtain a multidimensional observable parameter vector; Calculate the statistical variance of each component in the multidimensional observability parameter vector and the Pearson correlation coefficient between each pair of components to determine the set of weight coefficients for each parameter dimension; The parameters are normalized and weighted according to the set of weight coefficients to obtain a comprehensive observability index value. The normalized and weighted fusion method is as follows: first, the components of each parameter are standardized; then, a linear combination is performed based on the set of weight coefficients to generate a primary observable response value; and a regularization constraint based on the inverse variance function is applied to the primary response value to suppress the influence of high noise parameters; finally, the constrained fusion result is output as the comprehensive observability index data. The set of weight coefficients is dynamically updated based on local spatial consistency and maintains the weighted continuity between adjacent Gaussian elements in the same optimization iteration.
6. The visibility-based layered densification Gaussian sputtering method according to claim 1, characterized in that: The multi-level structure clustering process includes: For each Gaussian element, obtain the comprehensive observability index value. and its coordinate vector in three-dimensional space Construct a joint distance function that includes observability dissimilarity and spatial adjacency. The expression for the joint distance function is: in, They represent the first With the The comprehensive observability index value of each Gaussian element. and These are the observability difference weight and the spatial adjacency weight, respectively, and satisfy the following conditions: ; This represents the Euclidean spatial distance between the center points of two Gaussian elements; Perform multi-stage clustering operations based on hierarchical segmentation on all Gaussian elements, including: In the initial stage, based on the global distribution of comprehensive observability indicators, adaptive thresholds are used. Divide the primary salient clusters into background clusters; In the intermediate stage, for Gaussian elements within the boundary region, the observable gradients between their adjacent elements are calculated. And based on the gradient magnitude falling into the interval The set defines the transition layer cluster; In the termination phase, based on the global variance Dynamically update threshold set The update relationship is as follows: in, Indicates the first The first iteration A threshold, For adaptive learning rate, This is a preset variance reference value used to maintain the stability of the hierarchical structure.
7. The visibility-based layered densification Gaussian sputtering method according to claim 1, characterized in that: The densification process of the salient layer by the cross-layer control module includes: For each pixel in the salient layer, obtain the three-channel color information of the corresponding rendered image pixel value and the actual observed image pixel value, as well as the rendered depth and actual depth information; Color differences are calculated by weighted sum of squares of brightness differences in each channel, and depth differences are calculated by the absolute distance between the rendered depth and the true depth. When the color difference exceeds a first preset threshold and the depth difference exceeds a second preset threshold, the pixel is marked as a densification candidate point, and a back projection operation is performed based on the imaging parameters of the pixel to obtain the three-dimensional spatial coordinates corresponding to the candidate point. At each candidate 3D coordinate, a rotation matrix is constructed based on the local surface normal vector estimation results to determine the principal axis orientation of the Gaussian ellipsoid; Based on the magnitudes of the aforementioned color and depth differences, the scale coefficients of the Gaussian ellipsoid along each principal axis are calculated using a weighted combination method. The calculation formula is as follows: in, This is the initial scale constant. This is the difference response weight, used to control the degree to which color difference and depth difference contribute to scale changes; and These represent the color and depth differences of the pixel, respectively. The covariance matrix of the Gaussian ellipsoid is constructed from the scale coefficient and the rotation matrix, and its expression is: in, Let be a rotation matrix. These are the scale factors along the three principal axes; The visibility parameters of the Gaussian ellipsoid are determined by a weighted sum of the differences. The expression is: in, These are the weighting coefficients controlling the contribution of color and depth differences to visibility, respectively. Used to limit the result to the interval [0,1].
8. The visibility-based layered densification Gaussian sputtering method according to claim 1, characterized in that: The cross-layer control module handles the continuity constraint of the transition layer as follows: Eigenvalue decomposition is performed on the covariance matrix of each transition layer Gaussian element, and the eigenvector corresponding to the minimum eigenvalue is extracted as the local surface normal vector. The GS orthogonalization method is used to generate two orthogonal tangent vectors in the direction of the normal vector, and a local orthogonal plane coordinate system is constructed. Random radial sampling is performed on the local orthogonal plane coordinate system. The position of the sampling point is calculated based on the center position of the parent primitive, and the calculation relationship is as follows: in, Indicates the central position of the father Gorski unit. This represents the radial distance from the center to the sampling point. Indicates the sampling point relative to the reference tangent vector Angle, These are the orthogonal tangent vectors determined by the normal vector; the radial distance... Within the preset interval, according to the density function Weighted sampling is performed, where This is the sampling attenuation coefficient, used to adjust the spatial density distribution of sampling points on the plane; The newly generated Gaussian primitives inherit the visibility parameters and principal axis directions of the covariance matrix from the parent primitives, and adjust the length ratio of the principal axis of the covariance matrix according to the local curvature characteristics. When the Euclidean distance between a newly generated Gaussian primitive and its neighboring Gaussian primitives is less than a preset overlap threshold, the generation of that primitive is suppressed to avoid structural overlap caused by redundant densification.
9. The visibility-based layered densification Gaussian sputtering method according to claim 1, characterized in that: The process by which the cross-layer control module updates the Gaussian parameters of the background layer based on the error distribution of the salient layer and the transition layer includes: For each Gaussian primitive in the salient layer and the transition layer, the residual magnitudes of the rendering result and the real image in terms of color and depth are calculated to form Gaussian error data. The residual information of all primitives is aggregated according to spatial correspondence to construct a comprehensive error matrix of the salient layer and the transition layer, which is used to reflect the reconstruction bias intensity of local areas; For each Gaussian primitive in the background layer, retrieve the set of primitives in the 3D space that are adjacent to the primitives in the salient layer and the transition layer, and take a weighted average of the error values of the set to obtain the error response corresponding to the background layer. An iterative update strategy is used to jointly adjust the visibility parameters and covariance matrix of the background layer. The update relationship of the iterative update strategy is as follows: in, and The background layer is the first The first Gaussian Yuan in the Visibility parameters and covariance matrix at the next iteration For the Gaussian set adjacent to this primitive space, The number of primitives in this set. These are the update rates for visibility and covariance, respectively. These are the weighting coefficients for color residuals and depth residuals, respectively. For the first The residual values of adjacent primitives, These represent the primitive positions of the adjacent layer and the background layer, respectively.
10. The visibility-based layered densification Gaussian sputtering method according to claim 1, characterized in that: The process of step 6 includes: For each pixel of the target viewpoint, the ray direction vector is calculated based on the extrinsic and intrinsic parameter matrices of that viewpoint, and volume sampling is performed step by step in the three-dimensional space along the ray direction from the near clipping plane to the far clipping plane. During the volume sampling process, for each sampling point, a set of multiple neighboring Gaussian elements is determined based on its spatial location, and the spatial location, covariance matrix, visibility parameter and radiation color information of each Gaussian element are obtained. Based on the spatial distance relationship between the sampling point and the center of each Gaussian element and the directional parameter of the covariance matrix, the influence value of the Gaussian element density on the sampling point is calculated, and the volume transmittance of the point is determined in combination with the corresponding visibility parameter. Along the direction of the light's travel path, the transmittance of each sampling point is accumulated forward according to the sampling order to obtain the cumulative transparency of the light at the current depth position; and the radiation color values of each sampling point are weighted and superimposed to generate the cumulative color value of the current light position. The weight of the weighted superposition is obtained based on the density influence value and transmittance of the sampling point. When the cumulative transparency of a ray decreases below a preset threshold, the ray's journey is terminated, and the last accumulated color value is used as the final rendering output for that pixel. During the propagation of volumetric light rays, based on the change in visibility gradient between adjacent sampling points, the sampling interval is reduced when the local visibility gradient is higher than a preset threshold, and increased when the visibility gradient is lower than or equal to the preset threshold.
Citation Information
Cited By
Multi-modal enhanced rendering method based on 3D Gaussian
CN121837483A
A 3D Gaussian-based multi-modal augmented rendering method
CN121837483B
Multi-dimensional scene modeling optimization method and device based on geometric constraint, equipment and storage medium
CN121861216A
